Using something like Redis saves development time and makes builds and packaging easier as no FFI is needed. However Redis comes at the cost of having a more complicated infrastructure.
On a given node, we maintain a few million sorted sets that see a query and insertion velocity in the orders of millions of operations per second. When we are meticulously measuring operation time in the microsecond range to meet a performance target, a networked solution is out of the question.
Also, rustler handles compiling the rust code, and configuring the nif dynamic library details. I’d wager it’d be a ton easier than Redis infrastructure even if redis worked for the use case. The sheer amount of flexibility in the BEAM is one reason I love working in Elixir.
Really it seems like a “write once and forget” type of nif. Have you had to make a lot of updates after the initial tuning with Rustler?
Though I was curious if you tried ETS or mnesia? They both offer sorted sets. Not sure if they just use ordsets behind the scene.
This data structure has been operating in production for almost a year now, we have not touched it for any modification after writing and deploying it. We had 0 issues deploying it, and thanks to the comprehensive testing suite and benchmarks, we were confident it would meet the performance objectives. It was actually a drop-in replacement for the pure elixir data structure we had written. Deploying it was literally just a find + replace of OrderedSet to SortedSet in our code, running our test suite and then gradually deploying the new code to production.
I wonder if an out of process solution was considered. For instance, Redis supports sorted sets: https://redis.io/commands#sorted_set
Using something like Redis saves development time and makes builds and packaging easier as no FFI is needed. However Redis comes at the cost of having a more complicated infrastructure.