And here i'm happy with my 8k requests per second Elixir IOT solution.
I use long lived processes and had to come up with some magic to work around the supervisor behavior with high child counts, etc.
Roughly I randomly assign worked to a node in the cluster if they have not yet been assigned. (there is some logic tracking total nodes on cluster and max / target that influences the decision). I verify if the remote (or local) worker is alive or migrating by checking a fragmented process Registry and unique identifier via :rpc (because I do some recovery logic if it's offline and let the caller specify if messages should force a spawn or can be ignored if the process is offline) and then pad the call with some meta information for rerouting so the receiver can confirm it is the same process the message was initially sent to (since processes cycle so frequently that the initial process may have died and a new process may have spawned in the time it took to forward the message).
If the process has changed mid transit or the worker has been flagged for migration the message gets rerouted to the new destination+process. If a process does not yet exist a hash based on the worker type + id and available worker pools is used to select which of the auto generated but named and registered (WorkType.PoolSupervisor_123) worker supervisors spawns the child node.
It's a trip, and needs to be heavily documented. Starting from scratch i'd probably change some things and it probably needs some refinement later this year before the next batch of 250-500k devices get added to the network, but the costs per reporting device are fantastic with plenty of low hanging fruit for improving the cogs further so I'm happy.
I use long lived processes and had to come up with some magic to work around the supervisor behavior with high child counts, etc.
Roughly I randomly assign worked to a node in the cluster if they have not yet been assigned. (there is some logic tracking total nodes on cluster and max / target that influences the decision). I verify if the remote (or local) worker is alive or migrating by checking a fragmented process Registry and unique identifier via :rpc (because I do some recovery logic if it's offline and let the caller specify if messages should force a spawn or can be ignored if the process is offline) and then pad the call with some meta information for rerouting so the receiver can confirm it is the same process the message was initially sent to (since processes cycle so frequently that the initial process may have died and a new process may have spawned in the time it took to forward the message).
If the process has changed mid transit or the worker has been flagged for migration the message gets rerouted to the new destination+process. If a process does not yet exist a hash based on the worker type + id and available worker pools is used to select which of the auto generated but named and registered (WorkType.PoolSupervisor_123) worker supervisors spawns the child node.
It's a trip, and needs to be heavily documented. Starting from scratch i'd probably change some things and it probably needs some refinement later this year before the next batch of 250-500k devices get added to the network, but the costs per reporting device are fantastic with plenty of low hanging fruit for improving the cogs further so I'm happy.
https://github.com/noizu/SimplePool/