I agree with OP on this, YAML seems better suited for such a tool as you can easily add some form validation to it and then serialize it into Go data structures. You really shouldn't have to recompile your codebase in order to modify a diagram.

You don’t recompile the codebase, you build the diagram. You can build it with a tool that converts yaml into svg or golang into svg. The point is that there isn’t much difference as the diagram build process has the exact same high level steps in either case.