Standalone HAF Apps - Upgradeability

This is a continuation of a previous post where we discussed the idea of Standalone HAF Apps. Check out the post if you haven't.

Why?

Hive offers vast opportunities and possibilities, supported by a decentralized community and governance system. Myriads of apps can be created that greatly benefit from all of this. What is discussed here is how to make developing apps fast, easy, cheap and in a way that developing something for one app can be reused by other apps.

The post does a deep dive into the technical specifics of how blockchain data can be put into databases that support apps. As such, its intended audience is developers. For non-developers, we can summarize that we are discussing approaches to create infrastructure on top of which apps can be built easily, and to furthermore make it easy to change and improve this infrastructure in the future, without such changes requiring apps to do much or anything to still take advantage of the infrastructure.

For Developers

None of the ideas discussed here are final in any way, and your feedback would be essential for creating truly useful infrastructure on top of which apps can be easily built. This is the case regardless of whether the solutions look like standalone HAF apps, or standard HAF apps, or anything else. It is up to all of us to make this blockchain a place where each app is a part of a larger puzzle, a network of apps, where each app greatly benefits the other ones. Such a network holds vastly more possibility than any single app. Creating such networks and larger systems requires that we dedicate some of our attention to the greater whole, while also keeping our attention on our own app.

Technical Discussion

If such standalone HAF apps are made, one of the main issues becomes that if there is any change to the underlying way they work (for example, let’s say we find a better way to get blockchain data into the apps), the app developers then might have to do extra work to make their app compatible with the new approach. Such changes to the underlying infrastructure can take a lot of time from developers. Imagine if there are hundreds of such small standalone HAF apps and they all have to be upgraded because there is a breaking change. This would cause lots of extra work, a lack of desire to keep improving the infrastructure, and would defeat the purpose of letting developers simply focus on their app, without the blockchain getting in their way.

Rather than coming up with the best infrastructure design upfront, the premise here is that it’s even more desirable to make change easy, so that the design can be improved over time.

One way to do this is to decouple the standardized functionalities related to retrieving blockchain data from the functionalities specific to the app (e.g. interpreting of polls data). This will make it so that there is a standard standalone HAF app software that can be upgraded to take advantage of improvements, and on top of it there can be the code for the app’s unique functionalities.

In the more simple sense of separation of code/files, this can be achieved by creating a docker image with a Postgres database that takes care of the data retrieval, with a dedicated folder left for the app’s files (which, presumably, would typically be SQL files containing PL/pgSQL functions). This would allow developers to have a repository containing only their app’s files, and their repository can tie in with the standardized repository for the docker image.

In the more complex sense of separation of functionalities, we can break it down into three main infrastructure elements that can change: the data source, the data retrieval method, and the relay app. Here it is in a schematic presentation:
 

 
Data source refers to where the data for the standalone HAF app is coming from. This would typically be a full HAF server, or a HAF server filled with only some portion of blockchain data. It’s also possible for the source to be some database containing off-chain data such as is planned for the SPK network. Or in the future there could be a p2p network of databases that propagate blockchain data to one another, in a similar fashion to torrents. Or other possibilities.

Data retrieval method refers to how the data gets from the source to the standalone HAF app. The app could pull the data by making queries, or the source could push the data to the app. Or there could be any combination of the two (e.g. push data needed in real time and pull data needed once a day).

Relay app refers to the app that gives notifications to the standalone HAF app that there is new blockchain data of interest. If the source pushes data, then there is no need for a relay app. But if the standalone HAF app pulls data which it needs in real time, then it has to be notified that such data has been added to the blockchain so that it can make a query.

The above three elements describe how the data gets from the blockchain to the standalone HAF app without assuming any particular design. If we come up with a better design and there is a change at any of these three elements, there has to be no work or very little work for the app developers to do to keep their app compatible with this underlying blockchain data retrieval infrastructure.

So how can we do this? How can we make it so that we can change from pulling data to pushing data, or from receiving notifications by a relay app to no relay app being needed, or from getting data from a HAF server to also getting data from a database containing off-chain data, or coming up with a p2p data sharing network, and make any such design improvements without creating extra work for app developers to have to adjust to the changed infrastructure?

Two Main Things To Standardize

To keep it as simple as possible, yet highly flexible, we can standardize two main things in the infrastructure: the table structure of the data source, and the data the standalone HAF app is interested in.

First, the table structure of the data source. The source database contains tables with posts, accounts, upvotes, transfers, etc. Each of these can be standard table structures, no matter the data source. So if at some future time an app wants to include both on-chain and off-chain posts, then the off-chain database can just be added as a data source. No need for any changes to the app’s SQL queries and functions.

Next, the data the standalone HAF app is interested in. The app can have a configuration file that specifies which blockchain operations it wants, along with any filters for them (exactly the same as the filters of hived's sql_serializer plugin), and additionally the frequency at which it wants them (real-time, hourly, daily, etc.). This configuration file would be made publicly accessible, so that the data source can read exactly which data the standalone HAF app is interested in. Let’s do a mental test to see how it holds up. If we change from pulling data to pushing, then the data source can simply read which data it has to push to the app - no need for any work on the app’s end. If we change the data source to off-chain, that db reads the config file just the same. If we want to pull some data and push the rest, still no change required on the app’s end (the relay app reads the configuration file and knows to stop or to start sending notifications). If at any point the app wants to change the data it is interested in, that’s a simple config file change - no need to contact node operators and ask them to make changes on their end. In order words, the standalone HAF app makes publicly known which data it is interested in, which allows for flexibility in the infrastructure.

This also has the added benefit that the information about which blockchain data an app works with becomes programmatically readable. This can allow for some neat capabilities that can help the self-organization of the blockchain community and developers. Let’s say you wanted to see the data that any individual apps or all apps as a whole work with. You would be able to see which apps work with which ops. How could this be useful? It becomes a way to explore/navigate the development happening for the blockchain - you can browse the existing apps by the ops they work with. It becomes like a structured way to explore the development going on. You can see who is doing what and if there is any overlap/similarity between what you are doing and what someone else is doing. New apps will be automatically added and organized, and the info will be automatically updated, instead of manually creating project directories.

This setup also allows for variation in how the standalone HAF apps retrieve data, instead of all of them necessarily having to use the same method. For example, if an app works with derived data (e.g. not the raw custom_json ops but the app’s interpretation of them), then it could choose to query the data source and store only derived data in its db. However, this might turn out a slower method than the data source pushing to the app. If the app chooses the latter method, the data source will push the raw custom_json ops and the app will store them in a table, using the standardized table structure, and then it will additionally interpret and store the derived interpretations. This comes at a potential cost of requiring more database space for the raw ops. The important thing is that the app will have the flexibility to choose whichever method is more suitable for its use case, and to even use different methods for different ops and filters (e.g. it could store raw posts but only interpreted custom_jsons).

All the above has been purely a conceptual development and it will take developing real apps and seeing how they function in practice to assess if a setup like this works and truly has these benefits. Iterations and adjustments on this setup are fully expected.

In summary

Standardizing just two things - the table structure and the publicly accessible config file describing the blockchain data the app is interested in - should allow us to make design changes to the infrastructure without breaking compatibility with already built apps. Feedback from developers is needed in order to think through various pros and cons of all of this prior to making any such apps.