That's your mistake; it's not a matter of competency and experience. You are not a solo venture, you are a member of a team, and part of that team needs to be a customer representative who can tell you, in real time, what the customer wants.

There is no room for a hero engineer in modern software development. That's not how it works anymore, and it may never have worked that way.

Your countless anecdotes of this working stack only a fraction as high as the countless anecdotes of what you're describing completely failing.

This battle was fought in the 00s and won in the early 2010s. Agile produces results, waterfall generally does not.