Thank you for sharing your deep experience. It's a valid perspective, especially from an expert in the world of testing.
You're right to call out the need for vigilance and to place the burden of proof on those of us who advocate for this tool. That burden is not met by simply trusting the AI, you're right, that would be foolish. The burden is met by changing our craft to incorporate the necessary oversight to not be reckless in our use of this new tool.
Coming from the manufacturing world, I think of it like the transition in metalwork industry from hand tools to advanced CNC machines and robotics. A master craftsman with a set of metal working files has total, intimate control. When a CNC machine is introduced, it brings incredible speed and capability, but also a new kind of danger. It has no judgment. It will execute a flawed design with perfect, precision.
An amateur using the CNC machine will trust it blindly and create "plausibly good" work that doesn’t meet the specifications. A master, however, learns a new set of skills: CAD design, calibrating the machine, and, most importantly, inspecting the output. Their vigilance is what turns reckless use of a new tool into an asset that allows them to create things they couldn't before. They don't trust the tool, they trust their process for using it.
My experience with LLM use has been the same. The "vigilance" I practice is my new craft. I spend less time on the manual labor of coding and more time on architecture, design, and critical review. That's the only way to manage the risks.
So I agree with your premise, with one key distinction: I don’t believe tools themselves can be reckless, only their users can. Ultimately, like any powerful tool, its value is unlocked not by the tool itself, but by the disciplined, expert process used to control it.
You're right to call out the need for vigilance and to place the burden of proof on those of us who advocate for this tool. That burden is not met by simply trusting the AI, you're right, that would be foolish. The burden is met by changing our craft to incorporate the necessary oversight to not be reckless in our use of this new tool.
Coming from the manufacturing world, I think of it like the transition in metalwork industry from hand tools to advanced CNC machines and robotics. A master craftsman with a set of metal working files has total, intimate control. When a CNC machine is introduced, it brings incredible speed and capability, but also a new kind of danger. It has no judgment. It will execute a flawed design with perfect, precision.
An amateur using the CNC machine will trust it blindly and create "plausibly good" work that doesn’t meet the specifications. A master, however, learns a new set of skills: CAD design, calibrating the machine, and, most importantly, inspecting the output. Their vigilance is what turns reckless use of a new tool into an asset that allows them to create things they couldn't before. They don't trust the tool, they trust their process for using it.
My experience with LLM use has been the same. The "vigilance" I practice is my new craft. I spend less time on the manual labor of coding and more time on architecture, design, and critical review. That's the only way to manage the risks.
So I agree with your premise, with one key distinction: I don’t believe tools themselves can be reckless, only their users can. Ultimately, like any powerful tool, its value is unlocked not by the tool itself, but by the disciplined, expert process used to control it.