What are the resources to look at for designing the architecture of a scheduling system (for coroutines/threads)? Would I first look at how Operating systems implement it? Or how VMs like Erland implement it?
Same question for effects systems, where would I look to understand how they're designed and the trade offs for their design decisions?
It indeed makes sense to look at on OS. You might want to start with a simple one, like FreeRTOS - which is more or less just a task scheduler at core.
I would recommend not to start with coroutines in the beginning if your main focus is scheduling. In the end coroutines and async/await is about building a userspace scheduler on top of a scheduler that already exists in the OS, so you just get twice the amount of logic. However the schedulers used in userspace are often a lot more trivial than the OS ones, since they don't support preemption or priorities. Erlang might be the exception and an interesting thing to look into.
Everyone starts with callbacks, moves on to coroutines, and eventually ends up writing polling loops. Can take decades for each person to work through it.
Erlang's messaging may have potential as another alternative
Go is a modern take on coroutines (M:N threading), when everyone else seems to be going the async/await route. Though, I don't know how simple that scheduler implementation is (or if there are books about the internals).
Since the Linux kernel has pluggable schedulers, that code might be well-structured for reading. Again, I don't know the specifics.
Same question for effects systems, where would I look to understand how they're designed and the trade offs for their design decisions?