Having done hundreds of small to mid sized projects I take comfort in the fact estimating is a very distinct task that even the best of developers learn last.
I've got three tricks in my pocket that at least help me with this:
The first is that I go by the formula mentioned in "The Mythical Man-Month" — which is that the effort in larger projects is distributed as follows:
1/3 planning
1/6 coding
1/4 component test and early system test
1/4 system test, all components in hand
Developers tend to mainly estimate their coding time, which leads exactly to the effects mentioned. It also explains all the annoying situations with all the holdups once the project is "almost done", this is simply a systems test and will add another 25%.
The other is that I simply add 5-10% on top for each person involved into the project (dev, stakeholder, anything) for communication overhead. It usually holds true.
The third trick is meticuluous focus on lean delivery, which involves scoping down, early testing on as much as possible, hidden and pre-deadlines and setting tractable sub-milestones.
Unfortunately, all of these techniques don't accelerate the project, they just help you be more realistic. If you find something that accelerates complex projects, let me know ;-)
Also regarding:
> If you find something that accelerates complex projects, let me know
When possible cut scope and split the project in smaller projects with real deployments (much like you said with "The third trick is meticuluous focus on lean delivery, which involves scoping down, early testing on as much as possible, hidden and pre-deadlines and setting tractable sub-milestones.").
In nearly all my projects I now propose something like this to clients: "Ok we understand your needs and requirements. What do you think about having a V1 with only these core features deployed in a few weeks (which ideally already provide business value), and after that we iteratively improve it?". Works great for smoothly throwing away all the nice to have.
> If you find something that accelerates complex projects, let me know
Developer experience. I've been working as a developer for a living for 18 years & I thought I was pretty hot-shit 10 years ago already, but I can probably do more today in 4 days than I used to in 4 weeks (or 4 months a few years before that).
I've switched sub-fields of development a few times during my career (scientific computing, back-end web, front-end web, desktop applications, embedded, games [various platforms - native desktop/mobile/consoles, html5, flash, etc]) and I found that helped me keep the learning curve steep.
Don't forget the sanity check at the end. If other people also add a lot safety hours then you will end up with an inflated estimation that is hard to sell.
>>If you find something that accelerates complex projects, let me know ;-)
I know this was partly joking but there are two real ways:
Knowing your true critical path and proactively re-plan and manage the risk of those activities that sit on it to clear the way for it to run smoothly... that's of course assuming you planned and estimated it correctly in the first place.
If you have a real schedule that is properly resource and risk loaded, run it through a monte carlo sensitivity analysis and see if you can re-plan those highly sensitive activities or de-risk them.
I've got three tricks in my pocket that at least help me with this:
The first is that I go by the formula mentioned in "The Mythical Man-Month" — which is that the effort in larger projects is distributed as follows:
1/3 planning 1/6 coding 1/4 component test and early system test 1/4 system test, all components in hand
Developers tend to mainly estimate their coding time, which leads exactly to the effects mentioned. It also explains all the annoying situations with all the holdups once the project is "almost done", this is simply a systems test and will add another 25%.
The other is that I simply add 5-10% on top for each person involved into the project (dev, stakeholder, anything) for communication overhead. It usually holds true.
The third trick is meticuluous focus on lean delivery, which involves scoping down, early testing on as much as possible, hidden and pre-deadlines and setting tractable sub-milestones.
Unfortunately, all of these techniques don't accelerate the project, they just help you be more realistic. If you find something that accelerates complex projects, let me know ;-)