I agree with your analysis of building your own Agent framework to have some level of control and fewer abstractions. Agents at their core are about programmatically talking to an LLM and performing these basic operations:
1. Structured Input and String Interpolation in prompts
2. Structured Output and Unmarshalling String response to Structured output (This is getting easier now with LLMs supporting Structured output)
3. Tool registry/discovery (of MCP and Function tools), Tool calls and response looping
4. Composability of Tools
5. Some form of Agent to Agent delegation
I’ve had good luck with using PydanticAI which does these core operations well (due to the underlying Pydantic library), but still struggles with too many MCP servers/Tools and composability.
I’ve built an open-source Agent framework called OpusAgents, that makes the process of creating Agents, Subagents, Tools that are simpler than MCP servers without overloading the context. Check it out here and tutorials/demos to see how it’s more reliable than generic Agents with MCP servers in Cursor/ClaudeDesktop - https://github.com/sathish316/opus_agents
It’s built on top of PydanticAI and FastMCP, so that all non-core operations of Agents are accessible when I need them later.
I also recommend this. I have tried all of the frameworks, and deploy some still for some clients- but for my personal agents, its my own custom framework that is dead simple and very easy to spin up, extend, etc.
This sounds interesting. What about the agent behavior itself? How it decides how to come at a problem, what to show the user along the way, and how it decides when to stop? Are these things you have attempted to grapple with in your framework?
2. A way to create specialised subagents that can use their own tool or their own model. The main agent can delegate to subagent exposed as a tool. Subagents don’t get confused because they have their own context window, tools and even models (mix and match Remote LLM with Local LLM if needed)
3. Don’t use all tools of the MCP servers you’ve added. Filter out and select only the most relevant ones for the problem you’re trying to solve
Other than AgentBuilder, CustomTool, HigherOrderTool, MetaTool, SubagentBuilder the framework does not try to control PydanticAI’s main agent behaviour. The high level approach is to use fewer, more relevant tools and let LLM orchestration and prompt tool references drive the rest. This approach has been more reliable and predictable for a given Agent based problem.
I’ve had good luck with using PydanticAI which does these core operations well (due to the underlying Pydantic library), but still struggles with too many MCP servers/Tools and composability.
I’ve built an open-source Agent framework called OpusAgents, that makes the process of creating Agents, Subagents, Tools that are simpler than MCP servers without overloading the context. Check it out here and tutorials/demos to see how it’s more reliable than generic Agents with MCP servers in Cursor/ClaudeDesktop - https://github.com/sathish316/opus_agents
It’s built on top of PydanticAI and FastMCP, so that all non-core operations of Agents are accessible when I need them later.