I have had great luck using rayon for distributing CPU intensive work across available cores. If you are doing I/O I can see why async is preferable but rayon is an excellent library for parallel work.
I have found that async or even in Go with goroutines, a large amount of small threads are not faster even if they should be in theory when it comes to CPU intensive work.
I use it to generate a 3D universe (at the atomic level) but store the data as enums.
I have found that async or even in Go with goroutines, a large amount of small threads are not faster even if they should be in theory when it comes to CPU intensive work.
I use it to generate a 3D universe (at the atomic level) but store the data as enums.
Here is the repo: https://github.com/selfup/oxidizy
Big help from rayon:
1) https://github.com/selfup/oxidizy/blob/master/crates/unigen/...
2) https://github.com/selfup/oxidizy/blob/master/crates/unigen/...
3) https://github.com/selfup/oxidizy/blob/master/crates/unigen/...
Example perf generating an obscene amount of data on a 32GB machine with 16 threads (3800x):
Atoms: 110592000 Baryons: 26099712000 Quarks: 78299136000
real 0m12.779s user 0m0.000s sys 0m0.000s