For every project I work on, I make sure I understand the problem, define the requirements, design the solution, and validate any necessary design patterns before I begin planning the implementation. How long that takes depends on the scope of the project, but I have never seen a project save time by skipping that bit. I also end up being pretty accurate with my time estimates.
What I have seen several times with that approach is a wonderful plan that needs to get scrapped about half-way through when the requirements suddenly change, for various market-related reasons.
Some sketch of the end-design is always important, thinking about major components and future evolution, but a detailed plan has never been worth it in my line of work.
If you’re making radical changes to your design half way through, then you didn’t understand the problem you were trying to solve to begin with, and possibly didn’t define your requirements properly either. If small changes to your requirements mean you need to do significant redesign, then you didn’t design your solution properly.
The most generous way to view what you described is that you’ve had to cancel your project half way through and start a new one. There’s nothing you can do to make business decisions like that work smoothly. In my experience though, a more likely cause of what you’ve described, is that the project was just planned poorly to begin with.
The problem I was alluding to is exactly one of requirements - requirements aren't always exact, and they may change in unexpected ways as time goes on. In my experience, this is relatively common when building products that have a long time to market: since there are no hard requirements to begin with, just guesses on what features would be useful to have, it is easy for different marketing and product people to have different opinions and change their minds on what is or isn't an important feature.
What your describing is exactly the problem that decent planning solves. I can only imagine two possible reasons for the type of pivot you’re talking about. Either the problem you’re trying to solve has changed (not very likely), or your understanding of the problem has. You can avoid that, and all of the time you’ll waste, but simply investing the effort required to properly understand the problem up front. This is the reason I suspect that many successful founders are solving their own problems. They have some experience in some sector, know what problems it has, and bring a solution to market.
To give some more context, I'm working in a medium-size, well-established company in its field.
The kind of problem we have when launching a new product is that we can have a well-defined list of everything that is required of the new product to be assured of successf - it's going to take 5 years to build that, with at least 20 people.
What can we cut and still have a successful product, that we can launch in say 1 year (or 6 months) with say 10 people? That is much harder to say, and different product leaders have different opinions, based on discussions with different clients. And the consensus opinion may simply change.
So sure, the architecture is generally driven towards that 5 year product idea, but the short term estimates are always going to be geared towards a specific subset of that. And that subset is easily subject to change, from one quarter to another. We won't throw away the design if that happens, but we will definitely throw away any longer-term estimates.
And charting out a 5-year plan that we would only 're-jig' as priorities change would be a recipe for certain failure - teams change, people leave, etc. You can't hope to have an accurate estimate on that time horizon.
Without trying to sound rude, this is just your organisation being bad at business.
> a well-defined list of everything that is required of the new product to be assured of success
Only bad software tries to be everything to everybody (not to shit on bad software, it’s a multi-billion dollar industry). Your customers have a problem that other products weren’t solving, your solution to this problem is the core of your value proposition. It’s the only thing you need to implement to launch your product, everything else can be added iteratively over time. If you don’t know what that problem is, then you’ve failed before you’ve even started. You’re not driving your company forward by writing code as soon as possible, you’re just doing skids in the parking lot.
I’ve seen so many people get sucked into the fiction that their product is by necessity too complex to plan and design properly. In reality I’ve never seen it to be anything other than a cover for not actually knowing how to run a product well. You’re not saving any time by making it up as you go. All of the design decisions you need to make still need to be made, you can choose to make them in a controlled manner, or just do it randomly and hope you pull through. But good design is not an emergent outcome of the resources you devote to implementation.
A key element in inexact requirements is usually the human surprise element. A key to some of my success is a bias on infrastructure which is more about physics rather than messy human clients. Physics is more predictable, and the messy human stuff I have to deal with is whether or not my solutions are palatable by developers.
