We’ve spent the past week analyzing the performance of Eclipse hived/hivemind in our API service (api.hive.blog) and making improvements based on real-world loading patterns. It’s been a very productive week and we’ve resolved most of the outstanding issues reported and performance problems we observed. We’ve also identified and fixed several longstanding issues in the hived mutex locking code that prevented peak performance and led to potential data corruption issues. # Hived work (blockchain node software) We made a number of non-consensus changes to improve hived performance and fix old bugs: We reviewed the locking used to protect the chainbase database and found several serious longstanding bugs that showed up under high stress API loading of a hived node. These bugs could lead to data corruption and non-optimal performance. Those fixes are incorporated into this merge request: https://gitlab.syncad.com/hive/hive/-/merge_requests/143 After fixing these problems, we ran several high stress tests using the `siege` tool and were unable to “break” hived’s operation anymore. We were also able to get hived to fully utilize all the server’s cores (48 on our test system), demonstrating that the locking is fairly optimal for realistic loads. As a performance optimization, we enabled filtering of operations returned by one of the most time-consuming API calls, `get_account_history`: https://gitlab.syncad.com/hive/hive/-/merge_requests/136 https://gitlab.syncad.com/hive/hive/-/merge_requests/137 https://gitlab.syncad.com/hive/hive/-/merge_requests/139 We made several changes to API responses returned by hived, mostly in response to reports from apps developers: https://gitlab.syncad.com/hive/hive/-/merge_requests/125 https://gitlab.syncad.com/hive/hive/-/merge_requests/126 https://gitlab.syncad.com/hive/hive/-/merge_requests/133 We also limited the max number of operation of operations to that can be fetched by this call to 1k operations (previously it allowed up to 10k operations) to optimize hived throughput: https://gitlab.syncad.com/hive/hive/-/commit/9a05f4d780fc98b891f51ab4b88ebec89fde467b We corrected account-impacting for several virtual operations created recently: https://gitlab.syncad.com/hive/hive/-/merge_requests/134 ## Hived status We started a full replay and a full sync early this morning to check all the above changes (full replay takes around 18 hours). We don’t expect any issues, but there is a very small chance for issues due to the mutex locking changes, even though no consensus changes were made. With these changes, I think hived’s performance and reliability has been substantially improved over its past condition and it seems quite solid in the face of very substantial API traffic loading. One minor issue still to be resolved is the longstanding issue with hived crashing if it receives API traffic when starting up, so we may tackle that next, since it can be an annoyance when updating several hiveds deployed inside of a API server node configuration. **As this version contains substantial operational improvements, we’ve tagged this version as [v1.24.5](https://gitlab.syncad.com/hive/hive/-/releases/v1.24.5), and it’s a strongly recommended upgrade for all API server nodes. However, there’s no need for exchanges or consensus nodes to upgrade to this version, since the performance improvements mostly come into play when a node is handling significant API traffic.** # Hivemind We made numerous optimizations and bug fixes in hivemind this past week: https://gitlab.syncad.com/hive/hivemind/-/merge_requests/311 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/306 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/312 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/314 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/291 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/280 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/316 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/318 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/323 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/226 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/322 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/289 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/315 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/326 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/321 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/325 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/328 ## Hivemind status (2nd layer social media microservice) By late tomorrow I expect we’ll have resolved all the remaining big issues with hivemind responses and performance, other than reputation support (which we’ve intentionally left out until all other issues were resolved). # Condenser + Condenser wallet (open-source code for **hive.blog**) Merged in support for decentralized lists: https://gitlab.syncad.com/hive/condenser/-/merge_requests/135 This functionality is still waiting one last issue to be resolved in hivemind, so it won’t be fully testable until tomorrow when we deploy a new version of hivemind. We made numerous changes this week to improve condenser’s wallet (corrected some display issues and improved performance of page loads by using newer, faster API calls), including new features in the proposals page: https://gitlab.syncad.com/hive/wallet/-/merge_requests/42 https://gitlab.syncad.com/hive/wallet/-/merge_requests/43 https://gitlab.syncad.com/hive/wallet/-/merge_requests/45 https://gitlab.syncad.com/hive/wallet/-/merge_requests/52 https://gitlab.syncad.com/hive/wallet/-/merge_requests/54 https://gitlab.syncad.com/hive/wallet/-/merge_requests/55 https://gitlab.syncad.com/hive/wallet/-/merge_requests/56 https://gitlab.syncad.com/hive/wallet/-/merge_requests/57 https://gitlab.syncad.com/hive/wallet/-/merge_requests/60 https://gitlab.syncad.com/hive/wallet/-/merge_requests/62 https://gitlab.syncad.com/hive/wallet/-/merge_requests/64 https://gitlab.syncad.com/hive/wallet/-/merge_requests/65 https://gitlab.syncad.com/hive/wallet/-/merge_requests/68 https://gitlab.syncad.com/hive/wallet/-/merge_requests/70 Proposals page changes: https://gitlab.syncad.com/hive/wallet/-/merge_requests/47 https://gitlab.syncad.com/hive/wallet/-/merge_requests/49 https://gitlab.syncad.com/hive/wallet/-/merge_requests/51 ## Condenser status Condenser is looking pretty good at this point, but we’ll continue to test and look for any remaining problems. # Hive-js improvements In order to make hive.blog faster, we modified hive-js to support the new option to filter the operations fetched by the get_account_history call: https://gitlab.syncad.com/hive/hive-js/-/merge_requests/12 https://gitlab.syncad.com/hive/hive-js/-/merge_requests/13 https://gitlab.syncad.com/hive/hive-js/-/merge_requests/14 # What’s next for the week? Tomorrow we’ll hopefully be deploying a new hivemind with all the performance improvements and fixes we made in the last week, after some real world traffic testing on our API node. We’ll also be testing condenser and the wallet and looking for fixes and optimizations we can make. **Assuming deployment of latest hivemind goes well tomorrow, we’ll likely still need to make a few more fixes as issues get reported, but I expect that by the latter part of the coming week we can begin to analyze feasibility for various previously suggested features for HF25 and also intermediate changes that aren’t consensus changes that can be released earlier.**

See: Update on BlockTrades Hive core coding by @blocktrades