The secret is to do a hybrid thing. You’re completely multi-tenant but build it so there can be many copies of shared resources and associate a tenant with one specific copy of each.
So customer A is actually on load balancer #1, elasticsearch #3, db cluster #2, app server asg #4, etc.. Then when you need to carve out separate resources for VIP customers you just add a new number and only assign them to it.
Multi-tenant architecture with horizontal scaling!
That can work, but even that can get incrementally complex as you (or your customers' workloads) scale. There are also limitations between those components you mentioned that sometimes necessitate further separation. To deal with all that within the hybrid or multi-tenant architecture, you end up spending a lot more time on tooling just to be able to manage it, and that takes away time from more useful improvements and maintenance.
It's like spoon theory[1]. As a product team, you only have so many spoons. Every bit of difficulty you spend on maintaining a system's operation takes away spoons that could be used for other parts of the product. Regardless of whether you use multi-tenant or hybrid or single-tenant, if it starts to take away all your spoons, you should be ready to try a different model and see if you get some spoons back. (I think this applies to all aspects of a product team, not just operations)
So customer A is actually on load balancer #1, elasticsearch #3, db cluster #2, app server asg #4, etc.. Then when you need to carve out separate resources for VIP customers you just add a new number and only assign them to it.
Multi-tenant architecture with horizontal scaling!