It reminds me of the general idea of generative adversarial networks where instead of a teacher and learner, you have a one trying to learn real inputs from fake inputs while the other tries to trick the first with fakes, with both training each other and the ultimate goal is to have the latter create realistic fakes.

I'm sure it's been or is being researched, but since I'm new to LLMs, my immediate thought was having code-aware models where one tries to find and fix vulnerabilities in a codebase while another tries to exploit them where the ultimate goal is to secure a codebase.