21st update of 2021 on BlockTrades work on Hive software
It’s been a while since I last posted a progress update, so I apologize for the delay.
In the early part of August, I devoted some time to looking for potential bottlenecks in our API node infrastructure, making improvements, then analyzing the results. The positive results of that work were discussed in [Hive core developer meeting #27](https://www.youtube.com/watch?v=8DFXZPK9mQ0), so I won’t repeat the details today, but we did see very substantial gains in API response times. I may make a separate post in the future to show some of the gains across various API calls.
In this post I will focus mainly on some of the coding work done in the past two weeks by the BlockTrades team. As a side note, a lot of smaller tasks were completed during this period, including internal infrastructure improvements (i.e. adding and configuring new servers, gitlab is now hosted on a faster server), testing improvements, etc. that I won’t go into detail about here.
# Hived work (blockchain node software)
As mentioned in the Hive dev meeting, we reduced hived’s memory usage from 20GB to 16GB. As part of this work, we also did some code cleanup and minor efficiency improvements:
https://gitlab.syncad.com/hive/hive/-/merge_requests/248
https://gitlab.syncad.com/hive/hive/-/merge_requests/247
We also completed the refactor of the command-line-interface wallet to reduce the amount of work required when new API methods are added to hived:
https://gitlab.syncad.com/hive/hive/-/merge_requests/170
### Speeding up sql_serializer plugin that writes to HAF database
Most of the recent work in hived has focused on the sql_serializer plugin. This plugin fills a HAF database with the blockchain data used by Hive applications, so it is fundamental to our plan for a scalable 2nd layer app ecosystem.
Since the serializer provides data to HAF apps, the speed at which it can transfer blockchain data to a postgres database sets an upper limit on how fast a new HAF server can be initialized from scratch. Based on benchmarks ran last night, we’ve reduced this time down to where the serializer can write all 50M+ blocks of Hive to a postgres database and initialize the associated table indexes in 7 hours (on a fast machine). That’s more than 2x faster than our previous time for this task.
As impressive as those times are, we should still be able to dramatically reduce this time in many cases, by enabling an option for the serializer to filter out operations that aren’t interesting to the Hive apps hosted on a specific HAF server. For example, a standard social media application such as Hivemind doesn’t need to process most of the blockchain’s custom_json operations, so a lightweight hivemind node could configure its serializer to use a regular expression to capture only the operations it supports.
# Hivemind (2nd layer applications + social media middleware)
### bug fixes
Fix to community pagination with pinned posts: https://gitlab.syncad.com/hive/hivemind/-/merge_requests/496
Only allow referencing permlinks that haven’t been deleted:
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/494
Fixes related to improperly versioned package dependencies:
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/498
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/504
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/506
Restore ctrl-c breaking and sync from block_log database with mock data (part of testing work for HAF):
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/508
### Hivemind optimizations
Improve query planning under postgres 12 (postgres 12 is shipped with Ubuntu 20, so we’re planning to make 12 the recommended version for HAF and hivemind):
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/505
Speedup of post-massive sync cleanup phase:
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/507
Improved storage management during massive reputation data processing to speedup massive sync:
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/509
After consulting with Hive dev apps to be sure no one used it, we are going to eliminate the “active” field from the response returned by some post-related Hive API calls.
# Hive Application Framework (HAF)
We’re currently examining alternatives for injecting “computed data” into a HAF database. As a specific example, currently hived computes “impacted accounts” which are a set of accounts affected by each blockchain operation. Many HAF apps will also need information about impacted accounts. In theory, there are three ways to handle this: 1) have the sql_serializer write the data computed by hived directly to the database, 2) re-use the code from hived in the form of a postgres c++ extension, and 3) have HAF apps recompute this data themselves. Option 3 seems pretty wasteful, so we’re mostly looking at options 1 and 2. Personally, I favor option 1, but we’re trying out option 2 now, to see how it works out.
### HAF-based Account history app
We’ve completed our first example app for HAF. This app replaces the functionality of a hived account history node. This means that future Hive API nodes will be able to operate with a consensus hived node instead of needed to run a heavier weight hived node configured with the account history plugin. The app is located here: https://gitlab.syncad.com/hive/HAfAH/-/commits/develop
Use of this app should also result in much more scalable and responsive account history API calls (currently these are some of the biggest bottlenecks in terms of API performance). I will probably have some benchmarks for this by next progress report.
### HAF-based Hivemind app
We had to make further updates to hivemind to get it to work from tables similar to those used by HAF (we’d done this previously, but later changes to hivemind had to be accounted for). Yesterday we completed a successful massive sync with this new quasi-HAF version of hivemind.
# What’s next?
In the coming week we’ll be focused on the following tasks:
* conversion of hivemind into a HAF app
* testing and benchmarking of HAF account history app
* continued tuning of HAF code based on above work
* begin planning for HF26
See: 21st update of 2021 on BlockTrades work on Hive software by @blocktrades