RESP3 looks very interesting, unfortunately I made the mistake of writing my own RESP2 protocol and now I don’t have time to upgrade to RESP3. I guess I’ll move to some third party C++ client library.
RESP3 is a strict superset of RESP2! You can just add to your implementation and end with RESP3 finally :-) It means also that a RESP3 parser can parse any RESP2 reply without issues, in case you use it against a RESP2 server.
I need to pipeline around 100000 commands every 30 seconds as a transaction. So if any of the 100000 commands fail due to database failure, then I need to crash the server and restart.
That is the easy part though, problem is these commands are coming from 12 different cores with possibly that cores will increase to 24 on future. One core might issue 5000 commands, other core might issue 20000 commands, there is no possibility to predict how many commands will come from a specific thread. It is extremely important that this is done with highest performance possible otherwise my system will have bunch of other issues with catching up on other non-db related client-side commands.
So I devised a special Redis pipeline structure where each of the cores can issue commands without any kind of mutex during the course of a tick (server is tick based), mutexes for the threads are locked only at the end of the tick to compile different pipeline commands coming from different threads into one giant pipeline command along with their asynchronous callbacks.
So practically I'm utilizing a giant redis pipeline with 12 threads on Redis client side where I'm locking only 12 mutexes at the end of the tick, this is extremely high performance as far as letting server process other non-db related client requests while all that data is getting written to server.
Truth to be told it is an over-engineered crap and I guess it might be possible to implement something similar with other redis client libraries but this is something I already build so I would like to keep it as it is.
I know you've probably had a look at it and decided it wasn't worth it, but could it be possible to generate a LUA script on the client side and EVAL it once it's completely generated ?
You'd need to periodically SCRIPT FLUSH to not let old scripts linger
I think it can be done through LUA, or even just by using "MULTI" command but I would still have to handle asynchronous replies on the client-side where it would get hairy regardless.
In any case I'm fairly certain there are multiple ways to solve such problems, normally I wouldn't even bother writing an in-house database client library but RESP is so amazing and simple that it doesn't take much time build a client library for Redis.
