• the thing is pretty elegant in its own way tbh, in a league of its own,

  • minimizes lock contention at various levels

  • optmized for high frequency write workloads, as the lock contention is minimized

  • scales up dynamically , scales down dynamically all on its own, what does it means again ? when the load per shard increases, we increase the number of shards in the new engine and migrate current keys to the newer engine

  • can handle abuse like nothing else

  • just run it and leave it

  • pub sub also works in a pretty nice way, for the most part the notifications are almost instant

  • all operations in-memory

  • built grounds up from first principles, I had not a single clue about how these things, just some common sense and grand intuition

  • works in a two engine way

  • almost lock-free for all operations

  • there are two copies of the engine, when one of them is getting too congested, the write load shift to a bigger engine and the data from the old one gets migrated to the new one in background while the old one keeps serving reads all without making any of the current or incoming requests wait in any way

  • in each engine, essentially a big shard manager, it manages multiple shards in kind of a staircase format, and each key gets assigned a shard via a hash function

  • the keys also have a TTL thingy, upon expiration the pub-sub notification is sent instantly for subscribers of the expiration channel and we stop serving that key, while for the main storage engine, the expired keys gets cleaned up for the garbage collector to be removed in the next upscale or downscale migration, at that stage their whole existence gets purged from the system

  • in last benchmarking, got an averge of 900µs write throughput and 500µs read throughput for 10^6 unique keys distributed across 50 clients hammering writes at the same time

  • can be used as a high performance cache layer or as a nosql database by just increasing the upper bound working memory

  • no dependencies, pure golang madness

  • In the benchmark, the system starts from bucket size 1 and beautifully auto scales up

  • The sharp dips seen in the graph are the points where garbage collector kicks in and key migrations happening which affect the throughput temporarily

  • The new engine resizing happens in background and the pointers are replaced when the new engine is ready to take writes which the old one serves reads

Benchmarked on a 8 core fanless M2 air (all cores maxed out under load)

Concurrent clients: 100 Peak Throughput: 115809 ops/sec Average Throughput: 100961.54 ops/sec Dataset size: 1000000 unique Keys Total Operations: 21000000 requests Total Time Tracked: 208 seconds