It's been a while since I've worked on CoinZdense. More than two months because of personal circumstances. My mariage has been falling apart leading to a semi-homeless status for me in the last few months where I had to often crash at my mom's. At the same time, a new relationship that I put all of my heart into has turned rather complicated in a way that I've found emotionally even harder to handle than the colapse of my mariage and my living conditions. The last two days I've managed to regain some focus, and while it took me some effort to get back into it, I'm setting out to try and spent at least one whole work day on the project each week. In this post I want to talk a little bit about an interesting bit about some new code I was working on these last two days. Before I was forced to pause my efforts on the project, two months ago and a bit, I was reworking the API for the [Python version of coinZdense](https://github.com/pibara/coinzdense-python). I've moved the old API to **unstable** because things overt there kept moving and I wanted to start buikding some tools on top of coinZdense-Python, like the [HIVE CoinZdense Disaster Recovery tools](https://github.com/pibara/hive-coinzdense-disaster-recovery) that I talked about in [this post](https://peakd.com/hive-139531/@pibara/what-if-coinzdense-comes-too-late-part-2--a-working-tool-for-preparing-for-quantum-disaster-recovery). The old API remains available for now, but with a difference in how things are imported. For example: ```python from coinzdense.app import BlockChainEnv from coinzdense.fs import EtcEnv ``` becomes: ```python from coinzdense.unstable.app import BlockChainEnv from coinzdense.unstable.fs import EtcEnv ``` Bit by bit I'm moving functionality from unstable to a part of the API that I intend to keep backward compatible stable. So far, everything in stable has been the layer zero stuff that I needed for building the HIVE CoinZdense Disaster Recovery tools. A coinZdense signing key consists of a small stack of level keys, where every level key consists of a tree of one-time signing keys. The one-time siging keys get used up with usage, and with them, at some point, so do some of the level keys. When a level key gets exausted, the generation of a new level key is quite CPU intensive, and as such, the time needed to sign a single transaction might go up, one signature in many, from significantly less than a second to multiple minutes, and that is something that from a user perspective isn't a consistent user experience. So the idea is that for user exerience, level-keys need to be generated in the background while the user isn't doing anything in particular. Yesterday and today I've been working on the API hooks for that at the lowest level, the one-time signing keys. Working on this subject, I ran into a well known problem with concurency in Python: The GIL or Global Interpreter lock. In short, in most other languages, to speed up the performance of one time signing key creation, the use of threaths would be the obvious choiche. In Python though, in CPU intensive operations not handled by truly GIL-aware libraries like numpy, chances are adding more threads will make CPU intensive tasks slower instead of faster. The coinZdense domain isn't exactly a great fit for numpy, so I started looking at multi process as a option. A really nice abstraction in Python is provided by the ProcessPoolExecutor in concurent.futures. There is also a ThreadPoolExecutor with the same interface, so I decided to use the Executor as a hook into my API. I wanted to test the new API extensions, but I also wanted to benchmark them a bit, so I wrote a tiny benchmark script. In this post I want to walk throug that script to show a bit of the API extension so far. In the future the user won't be exposed to this part of the API at this level as the user normally would use higher level key abstractions that will get a slightly different API, that will abstract away most of the background work in the **Wen3** layer, the Web-3.0 Entropy layer that will implement entropy-as-a-resource management and sub-key management. But now for the low level API. In our script we start off with the imports. ```python import asyncio import time from libnacl import crypto_kdf_derive_from_key as _nacl2_key_derive from coinzdense.layerzero.onetime import OneTimeSigningKey from coinzdense.layerzero.wif import key_from_creds from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor ``` The important ones are the imports from **coinzdense** and from **concurrent**. We import the OneTimeSigningKey and the key_from_creds function from coinzdense. We import the two types of **Executor**, so we can show the GIL problem. And finaly we import a function from libnacl that in the future is going to be abstracted away by the level key. The rest of the code looks like this: ```python print("reference, no executor") key = key_from_creds("coinzdense", "demodemo","secretpassword12345") otsk = [] start = time.time() for index in range(0,1024): salt = _nacl2_key_derive(20, index, "levelslt", key) otsk.append(OneTimeSigningKey(20, 7, salt, key, 1000000*index)) otsk[-1].get_pubkey() reference = time.time() - start print(" done, reference =", int(reference),"sec") for workers in [1,2,3,4]: for usethreads in [False, True]: print("workers =", workers, "use threads =", usethreads) if usethreads: executor = ThreadPoolExecutor(max_workers=workers) else: executor = ProcessPoolExecutor(max_workers=workers) otsk = [] start = time.time() for index in range(0,1024): salt = _nacl2_key_derive(20, index, "levelslt", key) otsk.append(OneTimeSigningKey(20, 7, salt, key, 1000000*index)) otsk[-1].announce(executor) cnt = 0 for index in range(0,1024): while not await otsk[index].available(): cnt += 1 await asyncio.sleep(0.1) measurement = time.time() - start print(" done", int(100*measurement/reference),"%") ``` The first thing the script does is create a master key using the **key_from_creds** function. Then that master key is used to create the equivalent of the level-key salt, after what both are used to create 1024 one time signing keys with 7 bit OTS chains and 20 byte long blake2 hashes. The first part of the script uses the synchonous **get_pubkey** method. OneTimeSigningKey is lazy. On construction the public key isn't calculated yet. But when the pubkey is requested and it hasn't been requested before, the CPU intensive operation is run at that moment. This was already part of the API before. As we see in the script, we use this first run to set ourselves a reference time to measure other outcomes against. Now we get to the core of the script. A double **for** loop that tries out each of the two Executor types with 1, 2, 3 and four workers. We notice the extensions to the API that allow us to use the Executor. ```python for index in range(0,1024): salt = _nacl2_key_derive(20, index, "levelslt", key) otsk.append(OneTimeSigningKey(20, 7, salt, key, 1000000*index)) otsk[-1].announce(executor) ``` The **announce** method will queue the calculation of the public key onto the executor. We basicly start calculation in the background. We do this for each of the 1024 one time signing keys. The next bit of code we wait for all of queued tasks to complete ```python for index in range(0,1024): while not await otsk[index].available(): cnt += 1 await asyncio.sleep(0.1) ``` The async method **available** returns a boolean indicating that the one time key has completed calculating the public key. Available returns without delay with a True or a False. Alternatively, the async **require** method will resolve after the pubkey has been calculated. So now for the output of the script: ![image.png](https://files.peakd.com/file/peakd-hive/pibara/Eo1vWZUfKyby77KnKFDFJHX2YAmymF8YZGNdHGdkCuYZjWScyED9ruv39rXY5TCzHSb.png) Notice what happens? Let's put it into a table | workers | ProcessPoolExecutor | ThreadPoolExecutor | | --- | --- | --- | | 1 | 102% | 130% | | 2 | 60% | 152% | | 3 | 44% | 187% | | 4 | 36% | 186% | There is a tiny for using a single process worker, and a moderate overhead for using a single tread worker. But look at what happens when we scale up. Multi processing makes things speed up, as intended while multi thread makes things slow down. This demonstrates the problems withe Python and the GIL for CPU intensive tasks. When later on we continue with the C++ version of CoinZdense, the code will undoubtedly use threaths rather than processes, but ont so for Python. The next step right now is to start working on integrating the use of this API into the level key code and migrating the resulting code to the stable API as soon as possible. But that is for next time. Getting back into the project again is taking some effort, but there is progress again and that is most important right now. # Support the CoinZdense project As I mentioned in my [talk at HiveFest](https://peakd.com/coinzdense/@pibara/my-hivefest-coinzdense-slides-and-some-personal-shit), CoinZdense is a one-man and **unfunded** project. You can help support the project by buying project support tokens on hive-engine, by helping out looking at the [1995 style project website](https://github.com/pibara/coinzdense), or by advocating with the different target Web3 ecosystem communities. So far the project has received roughly **33 HBD** from sale of [project support tokens](https://hive-engine.com/trade/COINZDENSE) on hive-engine, from a tiny Paypal donation on the [tipping-jar page](https://pibara.github.io/coinzdense/qr.html), and from post upvotes.