Author here. We definitely agree these models are really powerful and can figure out a lot on their own. A protocol wouldn't be restrictive to their flexibility, but it'll allow different clients running different models to still be able to interact with the same plugins.

Defining a standard around external memory, authentication, rules-based engines for preprocess/post-processing, and declarative or dynamic chaining of actions can help make plugins model-agnostic, and benefit all of us as developers and users.