I'm not sure there's a market in LLM "middleware" per se. Look at the market segments:
• B2C: wants vertically-integrated tools that provide "middleware" plus interface. Doesn't want to dick around. Often integrates their own service layer as well (see e.g. "character chat" apps); but if not, aims for a backend-integration experience that "knows about" the quirks of each service+model family, effectively commoditizing them. The ultimate aim of any service-generic app of this type is likely to provide an "subscription store" where you purchase subscriptions to inference services through the app, never visiting the service provider itself.
• B2B (think "using agents to drive pseudo-HFT bots for trades in fancy financial instruments that can't be arbitraged through dumb heuristics"): has a defined use-case and wants to control every detail of both "middleware" and backend together. Vertically integrates their own solution — on-prem inference cluster + custom-patched inference engine + business logic that synthesizes the entire prompt at every step. Doesn't bother with the "chat" abstraction other than as part of several-shot prompting.
• B2B2C: wants "scaling an inference cluster + engine + model deployment" to be Somebody Else's Problem; thinks of an "app agent experience" as the deliverable they want to enable their business customers to achieve through their product or service; and thus thinks of "middleware" as their problem / secret sauce — the thing they will build to enable "app agent experiences" to be created with the least business-customer effort possible. The "middleware" is where these B2B2C businesses see themselves making money. Thus, these B2B2C businesses aren't interested in paying some other middleman for a hosted "generic middleware framework as a service" solution; they're interested in being the only middleman, that captures all of the margin. They're interested in library frameworks they can directly integrate into their business layer.
---
For an analogy, think of the "middleware" of an "easy website builder" service like Squarespace/Wix/etc. You can certainly find vertically-integrated website-builder services; and you can also find slightly-lower-level library components to do what the "middleware part" of these website-builder services do. But you can't find full-on website-builder frameworks (powerful enough that the website-builder services actually use them) — let alone a white-labelable headless-CMS + frontend library "website builder builder" — let alone again, a white-labelable headless-CMS "website builder builder" that doesn't host its own data, but lets you supply your own backend.
Why?
Because B2C businesses just want Squarespace itself (a vertically-integrated solution); B2B businesses don't want an "easy website builder", they want a full-on web-app framework that allows them to control both the frontend and backend; and B2B2C businesses want to be "the Squarespace of X" for some vertical X, using high-ish-level libraries to build the highest-level website-building functionality, while keeping all of that highest-level glue code to themselves, as their proprietary "secret sauce." (Because if they didn't keep that highest-level code proprietary, it would function as a "start your own competitor to our service in one easy step" kit!)
---
The only time when the "refined and knowledge-enriched middleware abstraction layer -as-a-Service — but backend-agnostic!" approach tends to come up, is to serve the use-case of businesspeople within B2B orgs, who want to be able to ask high-level questions or drive high-level operations without first needing to get a bespoke solution built by the engineering arm of said org. This is BI software (PowerBI), ERP software (NetSuite), CRM software (Salesforce), etc.
The weird / unique thing about LLMs, is that I don't think they... need this? The "thing about AI", is precisely that you can simply sit an executive in front of a completely-generic base-model chat prompt, and they can talk their way into getting it to do what they want — without an engineer there to gather + formalize their requirements. (Which is not to say that the executive can get the LLM to build software, correctly, to answer their question; but rather, that the executive can ask questions that invoke the agent's inbuilt knowledge and capabilities to — at least much of the time — directly answer the executive's question.)
For LLMs, the "in-context learning" capability mostly replaces "institutional knowledge burned into a generic middleware." Your generic base-model won't know everything your domain-specialist employees know — but, through conversation, it will at least be able to know what you know, and work with that. Which is usually enough. (At least, if your goal was to get something done on your own without bothering people who have better things to be doing than translating your question into SQL. If your goal is to work around the need for domain expertise, though... well, I don't think any "middleware" is going to help you there.)
In short: the LLM B2C use-case is also the LLM "B2Exec" use-case — they're both most-intuitively solved through vertical integration "upward" into the backend service layer. (Which is exactly why there was a wave of meetings last week, of businesspeople asking whether they could somehow share a single ChatGPT Pro $200/mo subscription across their team/org.)
• B2C: wants vertically-integrated tools that provide "middleware" plus interface. Doesn't want to dick around. Often integrates their own service layer as well (see e.g. "character chat" apps); but if not, aims for a backend-integration experience that "knows about" the quirks of each service+model family, effectively commoditizing them. The ultimate aim of any service-generic app of this type is likely to provide an "subscription store" where you purchase subscriptions to inference services through the app, never visiting the service provider itself.
• B2B (think "using agents to drive pseudo-HFT bots for trades in fancy financial instruments that can't be arbitraged through dumb heuristics"): has a defined use-case and wants to control every detail of both "middleware" and backend together. Vertically integrates their own solution — on-prem inference cluster + custom-patched inference engine + business logic that synthesizes the entire prompt at every step. Doesn't bother with the "chat" abstraction other than as part of several-shot prompting.
• B2B2C: wants "scaling an inference cluster + engine + model deployment" to be Somebody Else's Problem; thinks of an "app agent experience" as the deliverable they want to enable their business customers to achieve through their product or service; and thus thinks of "middleware" as their problem / secret sauce — the thing they will build to enable "app agent experiences" to be created with the least business-customer effort possible. The "middleware" is where these B2B2C businesses see themselves making money. Thus, these B2B2C businesses aren't interested in paying some other middleman for a hosted "generic middleware framework as a service" solution; they're interested in being the only middleman, that captures all of the margin. They're interested in library frameworks they can directly integrate into their business layer.
---
For an analogy, think of the "middleware" of an "easy website builder" service like Squarespace/Wix/etc. You can certainly find vertically-integrated website-builder services; and you can also find slightly-lower-level library components to do what the "middleware part" of these website-builder services do. But you can't find full-on website-builder frameworks (powerful enough that the website-builder services actually use them) — let alone a white-labelable headless-CMS + frontend library "website builder builder" — let alone again, a white-labelable headless-CMS "website builder builder" that doesn't host its own data, but lets you supply your own backend.
Why?
Because B2C businesses just want Squarespace itself (a vertically-integrated solution); B2B businesses don't want an "easy website builder", they want a full-on web-app framework that allows them to control both the frontend and backend; and B2B2C businesses want to be "the Squarespace of X" for some vertical X, using high-ish-level libraries to build the highest-level website-building functionality, while keeping all of that highest-level glue code to themselves, as their proprietary "secret sauce." (Because if they didn't keep that highest-level code proprietary, it would function as a "start your own competitor to our service in one easy step" kit!)
---
The only time when the "refined and knowledge-enriched middleware abstraction layer -as-a-Service — but backend-agnostic!" approach tends to come up, is to serve the use-case of businesspeople within B2B orgs, who want to be able to ask high-level questions or drive high-level operations without first needing to get a bespoke solution built by the engineering arm of said org. This is BI software (PowerBI), ERP software (NetSuite), CRM software (Salesforce), etc.
The weird / unique thing about LLMs, is that I don't think they... need this? The "thing about AI", is precisely that you can simply sit an executive in front of a completely-generic base-model chat prompt, and they can talk their way into getting it to do what they want — without an engineer there to gather + formalize their requirements. (Which is not to say that the executive can get the LLM to build software, correctly, to answer their question; but rather, that the executive can ask questions that invoke the agent's inbuilt knowledge and capabilities to — at least much of the time — directly answer the executive's question.)
For LLMs, the "in-context learning" capability mostly replaces "institutional knowledge burned into a generic middleware." Your generic base-model won't know everything your domain-specialist employees know — but, through conversation, it will at least be able to know what you know, and work with that. Which is usually enough. (At least, if your goal was to get something done on your own without bothering people who have better things to be doing than translating your question into SQL. If your goal is to work around the need for domain expertise, though... well, I don't think any "middleware" is going to help you there.)
In short: the LLM B2C use-case is also the LLM "B2Exec" use-case — they're both most-intuitively solved through vertical integration "upward" into the backend service layer. (Which is exactly why there was a wave of meetings last week, of businesspeople asking whether they could somehow share a single ChatGPT Pro $200/mo subscription across their team/org.)