To your latter part: This is exactly the point: Using messages and queues makes the flows take whatever time it takes. Settling of the mentioned orders are not time critical - well, at least not in the way a request from a user sitting on his phone logging in to see his holdings is time critical. So therefore, if it takes 1 second, or 1 hour, doesn't matter all that much.
The big point is that none of the flows will fail. They will all pass through as fast as possible, literally, and will never experience any failure mode resulting from randomly exhausted resources. You do not need to make any precautions for this - backpressure, failure handling, retries - as it is inherent in how a messaging-based system works.
Also, if a user logs into the system, and one of the login-flows need a same service as the settling flows, then that flow will "cut the line" since they are marked "interactive".
> You do not need to make any precautions for this - backpressure, failure handling, retries - as it is inherent in how a messaging-based system works.
In reality, there are time limits on fulfillment of requests. Even in your previous example of an order execution / fulfillment system, the orders must execute while the exchange is still open. I can’t think of a system that is truly unbounded in time, but maybe one exists.
This would not work for a exchange. What I work with is an UCITS mutual funds system. Once per day, we get a new price for each fund (the NAV, Net Asset Value). We now need to settle all orders, subscriptions and redemptions, waiting for that NAV. This is of course time critical, but not in the millisecond-sense: As long as it is done within an hour or two, it is all ok.
I believe this holds for very many business processes. If you get a new shipment of widgets, and you can now fulfill your orders waiting for those widgets, it does not really matter if it takes 1 second, or in a very bad outlier day, occasionally 2 hours.
Realize that the point is that this settling, or order fulfillment, will go as fast as possible. Usually within seconds or maybe minutes. However, if you suddenly get a large influx, or the database goes down for a few minutes, this will only lead to a delay - there is nothing you need code up extra to handle such problems. Also, you can scale this very simply, based on what holds you back (services, or database, or other external systems, or IO). It will not be the messaging by itself!
The big point is that none of the flows will fail. They will all pass through as fast as possible, literally, and will never experience any failure mode resulting from randomly exhausted resources. You do not need to make any precautions for this - backpressure, failure handling, retries - as it is inherent in how a messaging-based system works.
Also, if a user logs into the system, and one of the login-flows need a same service as the settling flows, then that flow will "cut the line" since they are marked "interactive".