Thanks for sharing. Yeah, I think that's a problem with bolted on functionality. Did you end up building a new version with a different language? What kind of problems did you encounter that made the solution "not great"?

We eventually managed to build something in sidekiq that did the job but if your jobs are not unrelated and you want to make sure not to starve any specific resource the options in ruby are just way worse than eg Elixir

Are you able to provide more context and details? It’s always interesting to learn about system design and architecture decisions from real world examples.

The issue was processing messages for live pro esports tournaments. Each game had to be processed and the biggest issue was that messages from the queue were not guaranteed to ordered and because of processing issues upstream you could get a large amount of messages for a specific match in sequence. Thus we decided to run just a light weight worker on the original external queue copying to our datastore which is very fast and can be run massively parallel.

Then run a unique job per running match that does the ordering and processing. This worked well enough but you need to now make sure that each worker gets called again after it finished and originally we had a recursion going on but the best place to do it was eventually deprecated in a new sidekick or unique jobs version update. We eventually settled on a supervisor that would monitor and restart workers. Which is just a bad version of elixir in the end