This guy gets it. Development is literally the simplest part of the problem set.
The intersection of domain expertise, contracting/procurement creativity, and technical evaluation is the tough part to wrangle. Getting to the point where development can start is 95% of the battle in my experience.
Not to trivialize the actual development portion of the project, but most missteps happen early, dooming projects before anyone sits down to code.
Now think of the problem in reverse, from the government's side. Why are they contracting it out when all of the most difficult aspects involve integrating with government systems, policies and personnel - things that government already knows how to do?
"Because we don't have the expertise to do software development."
The comments in this thread, which I fully agree with by the way, just cement my belief that if government wants to deliver high-quality IT projects they need to start developing the expertise to do it in-house.
Ha, I'm currently a federal government program manager, so I am thinking about it from the government's side right now.
I've also been a consultant cog in a mega-corp and run my own small consulting business, so I've seen it from both sides.
Tech folks underestimate non tech challenges (basically anything outside of their domain), and clients don't actually know what they want. It's an age old problem in lots of industries.
I work in a similar sector but have a strong belief that the software delivery can and should become a capability that occurs within the government.
They should be diligent in what capabilities they choose to support. And ruthless in continually developing and pivoting these capabilities.
This is true of any large organization or enterprise that operates in a software enabled domain, so yes, most industries. The idea that software development is too difficult and should only be contracted out to fleety consultants is one that needs to be properly put to rest.
And to be clear, I'm not saying that consultants have no place. But for the type of technology and delivery as required by the project in the OP - there's no excuse.
I’ve been working in enterprise/government sector for a few years now and my observation is that they need to focus on better systems design. Everything is too convoluted. Trying to make the software fit the poorly thought out and implemented legacy ERP design from yesteryear is where most of the money is wasted.
Oh, my last disappointment of the week was being lectured via 2 screenfuls of Confluence with a “it’s more complex that what you make it” exposure of 2-3 layers of encrusted legacies. The PO was flexing and glowing how their managed to build into the new product this synthesis of bad designs. Which means entire departments get a list of rotating passwords by email every couple weeks. The network of ossified legacies just grew one more sprout.
I've wasted far too much time in my career doing exactly this for federal government customers. My favorite is when someone insists a feature is critical and must be replicated, but when I dig into it I find it *never* actually worked and as such, no one was using it.
Made worse when you have lots of money and even more stakeholders.
Perhaps worst of all, when failure has no true cost, avoiding failure matters far less. The CDC won't go broke because they blew through $44 Mil. Most every other organisation would. That's one of the challenges of government, the lack of creative destruction.
One of the biggest challenges for large orgs is that the knowledge they need to do a given thing is almost always present inside the organization — but it's fragmented and scattered across different teams, roles, and balkanized administrative boundaries such that the big picture is almost never available.
A high-quality consulting partner works to build the relationships and internal lines of communication that are necessary for the client organization to do more of the work — maybe all of it! — in the future. A cash-grab consultancy will just make themselves the organization's "glue" and bill until the money runs dry.
That last dynamic you described is exactly what I saw at a very well known university: upper management knew they weren’t getting the accurate details through the middle management filter, but shied away from fixing the cultural problems. Instead, a major consulting company was charging a considerable amount per hour to have 24 year olds with expensive suits interview the actual workers, summarize what they’d been telling their management for years, and present it directly to the senior level. Fixing the system would have saved millions but required fights they weren’t constitutionally well suited for.
Absolutely, but this is essentially saying "We're okay paying a 2-10% tax on our revenue, forever."
I think that's the reason you tend to see these unhealthy dynamics in extremely conservative, quasi-monopoly industries (government, financial, healthcare), where there's no real question of if they'll be a going concern. And no real rewards / attempts to outperform at the cost of increased risk.
Critically, if someone within the org volunteers to organize and manage the cross departmental knowledge they'll be shut down by people within the company for trying to accumulate power or stepping into someone else's turf.
Only an outsider can safely talk to everyone. They aren't a threat because they aren't part of the company and therefore can't take over.
It cannot be volunteer work. It has to be sponsored from the top. If it is, then a team of qualified FTEs can do this job similar to an external consultancy. But rarely will old-timer well-settled FTEs will leave their current position and take on this temporary risky gig.
Also, the end result and the path to get there isn't clear for FTEs – let's say they did the functional domain requirement gathering work – then who will implement it – they will have to contract out the development work – they may not be empowered to think and see through to the end goal fully.
On the other hand, for the top management folks – it is easier to temporarily empower the external consultants and terminate the engagement with them at any time – but they cannot do the same with FTEs due to employment laws etc.
What if it wasn't temporary, but just part of the CEOs office? The first goal should be to actively prevent these foul-ups by keeping open lines of communication.
I've always thought that a team like this is critical to keeping a company running. Any org that's 1000+ needs a 5-7 person team outside of any department that the CEO can throw at problems, but no org that I'm aware of has adopted this structure.
I can attest that the second modus operandi is the way long time contracts work from my experience.
Nearly every time all the knowledge and skills necessary are hidden and fragmented inside our client's org. With our access through all the management layers and silos we could work to fullfil the contract and also enable change within the client organisation and culture.
But why kill the golden goose? Our higher people' bonuses depend on us being the glue as someone else wrote in another comment. And a lot of my colleagues actually on the client's really do a great job in opening lines of communication. They work their a*es of to help the client.
Sadly helping isn't a monetary long term strategy for upper levels.
Though mostly about ladder climbing and relationship drama, the show 'House of Lies' has some really good scenes where the management consultants do actual work, which is pretty much portrayed as doing the things you've listed. Even as a fictional show, it made it easier for me to understand these situations and not get so flustered when encountering them.
I've been in technical, not management, consulting, but House of Lies felt disturbingly accurate.
Except I usually just stared at bad movies in hotel rooms to decompress after dealing with clients, rather than blow and strippers. Sue me, I'm boring.
Heh. That stuff has always felt like an inexplicably weird trope, like Hackers-inspired visions of computer people as living Mondo 2000 articles.
After 8 hours in a windowless conference room with a parade of stakeholders, making RACI matrices on a stained whiteboard and scarfing down cafeteria sandwiches during "bio-break," I want to _talk to my wife on the phone then stare blankly at the ceiling_, not go out for a night on the town.
I am definitely biased by my past experiences with tech consultants. I’m trying to sort out if it’s a service provider issue or if it’s a customer side issue.
From my limited perspective, it seems management consultants are focused on 80/20 solution approaches. While it seems technical consultants are focused on delivering MVPs that don’t seem to map well to the business space.
The disincentive that tech consulting has is that jobs (at least in my area) were usually bid at fixed price.
I have no idea if that's the way larger firms do it (would expect not).
Consequently, profit = contract price - consultant hours. Every bad happened from minimizing the latter.
Even if you do straight up billable hours with the client, your time was typically scheduled 1-2 projects in advance. This lead to being prematurely (in terms of things being "done") rolled off a project, onto a new one, and finishing / supporting the first from your hotel after-hours.
This was in a pretty cowboy area. I imagine different parts of technical consulting behave differently.
But yes, MVP quality code seems to be a constant. I saw that delivered from a top-tier technical consulting firm (ML solution) to a T100 customer.
To what extent is this due to neo-liberal "small government" policies, where from the highest political levels - through lobbying - it is decided that outsourcing is a win-win that leads to financial savings and budgets that are cut, and later on cut some more, etcetera?
I've witnessed the crazy dance of enterprise consultants in large corporations (not government) and it is astounding how much money is wasted in inefficiency.
I was in a small tech shop and we had just won an RFP at a big corporation, with a very practical solution and for a limited domain. Our down-to-earth approach led to so much enthusiasm with corporate employees, that they sent all their consultants to us to investigate interfaces to ever more back-end systems. Scope ballooned, and we - being a small party - were dictated to support all this stuff. It was utter madness to such extend that we decided to more-or-less graciously wringle out of the contract. Just-in-time before we were committed to a monstrous project that was 100% certain to fail. And that was not the only time we experienced this phenomenon in big corp.
On tasks where building the initial framework is the most expensive part, swapping out contractors only serves to abstract away complexity without reducing its cost.
It's better to have an in-house dev department in these cases even in the complexity side, if you realize that complexity in the contractor side is just as bad as complexity in the basic org.
Exactly, that's why I turn down companies that don't take a tech first approach, if you rely on tech, you ARE a tech company. Clearly the government doesn't see itself as a tech venture of any kind, but tech fails in silo. Tech first is the only way tech can succeed at delivering value to your business. Otherwise, go back to traditional processes, stick to SharePoint, email, excel, and manual logistics. And ya, this is true of the government as well, if it wants to modernize and leverage software innovations it'll need to start approaching things with a tech first mindset.
the UK has done exactly this - to make all Government Digital Services in house. It's been a blockbuster success in my opinion - they have some of the best UX/UI in the world and their websites are ultralight and downgrade well.
There are a few issues with authentication, but there is support.
Everything that's related to dealing with the UK government btw is all in ONE website, where everything is explained to you in Plain English - gov.uk
Coming from a country where this doesn't exist, I was blown away by every interaction I've had with the system. It's simply amazing.
As a foreigner living in the UK, I concur. All .gov.uk are _extremely_ functional. Applying for the pre-settled status (cos Brexit) was actually an enjoyable experience! I had no idea what was going to happen when I started the process, and it was all done in 30min top. It made me _scan my passport with my smartphone_ while flashing weird colors and it worked flowlessly, that absolutely blew my mind.
I've actually looked into getting a job there even though I'm from a start-up background
Yes that was some of their best work. I believe the flashing lights allow for some sort of 3D reconstruction of your face? And scanning the passport with NFC!
One of them is high-tech DYI solution, the other one is super simple yet not used at all by any other govt service I've interacted with.
Every single GDS project I’ve been on is 90% freelancers all via a small handful of companies — it’s not really “in-house” but at least it’s better than absolutely everything going to capgemini, Fujitsu, NTT, ATOS and the like, sure.
Exactly. The UK in-housed their entire website IT team, and it's one of the best set of government websites I've seen worldwide, UX-wise and performance-wise.
I worked for the government and it was pretty damn hard to get further training. We had no funds for conferences, classes, travel, or any such. All the improvement was down to me, what I could learn about R/stats/STATA from forums, from colleagues, and from the 1500 member analyst email list. There were all sorts of forums/libraries "within" the government system, however their search functionality was garbage and they didn't have enough users to make it a useful forum for most topics. Sigh!
The sad, true reason why they don’t is that government programmer jobs will pay 1/2 - 1/3 what a private sector job will pay in the DMV (DC, MD, VA). And the ironic part is that the reason private sector pay is so high is largely attributable to these big federal contracts being handed out. But as things are, the federal government cannot hire enough qualified people in the location (by DC) that they want to.
I’m going to be discrete but I know of a Deloitte project where they are allowed to use AWS Fargate, but the gov’t side is limited to IBM mainframes. Why are they so limited? Who knows, but it’s a deep question, that would require total reinvention. Perhaps, in the end, it might not even be worth the risk because the gov’t is so essential. They can’t move fast and break things. But it does look insane as a result.
FEDRAMP requirements and auditing are difficult and expensive to meet. I've seen the cost of these in bids. We said something like $20k per month for what would be a $50 per month EC2 server. It was a shocking thing to see. I'm guessing Deloitte is charging six figures a month to host this thing.
Does the government, in fact, know how to do those things? How would we know if they did or didn't?
I notice that they spend oddly large amounts of money on anything for which we know what the cost roughly should be -- e.g. CDC web sites -- and don't seem to do so very quickly or very well. Should we assume that they're a lot more competent at things we know less about?
Agencies are less likely to be competent in non core areas. This should not be surprising. There is no career path for you doing software in an agency like the CDC run by bio science people. Same for the Air Force, which is run by fast jet pilots, the Congress by lawyers and liars, Commerce by thieves, etc. Even banks and trading firms, who live and die by software, are run by accountants and economists.
Unless an organization recognizes the core importance of software and carves out a specialisation stream and C level ownership of it, they rarely achieve more than brief or isolated competence.
This is the reason why SV is so appealing -- nowhere else do SWEs hold sway over entire companies and industries.
Govt can do excellent software if it becomes a priority, they could pay competitive wages (there are special pay scales for doctors and lawyers), but they choose not to because of risk aversion from politicians, who mistakenly believe they can outsource the risk.
For something like this, paying Eventbrite to do it wouldn't have been a bad place to start. But the fat cat consulting firms never have domain expertise beyond getting Govt contracts.
There's a couple of reasons government IT projects turn out this way.
In my experience, a lot of the time it's not so much that the contractor is trying to make money by prolonging the problem. It's that the customer doesn't know how to be a competent product owner that prioritizes and scopes things properly.
Then, frequently the software being designed ends up replicating broken/inefficient business processes, so it ends up being difficult to make cleans and efficient software. This becomes worse when there's legacy software in the mix where the customer wants the contractor to port over all of the clunky functionality of the old app. Which of course isn't documented in the first place, so you've got to perform software archaeology to figure out what it did.
Some other show stoppers I've seen are when key stakeholders act as gatekeepers and prevent developers from talking to SMEs directly, so you have to build your requirements based on what the person with the 10,000 foot view knows. So of course you get incomplete and sometimes completely wrong requirements. And you might not find out how wrong they are until after you've spent a lot of time building things
Just those dynamics alone can add an enormous amount of cost or outright doom a project. I could probably fill a blog with all of the dysfunctional stuff I've seen in federal IT, but suffice it to say that these factors can easily balloon costs and timelines to a comical degree. This $44 million failure is actually small compared to some other government IT failures like the billion dollar Air Force accounting system.
There is often the expertise in house, but over years the internal teams get terribly under-funded and the poor service that results is then used to justify spending vastly more amounts on external suppliers.
This always comes down to piss-poor and craven senior management decisions.
I'd never in a million years argue that Deloitte did an awesome job on a large government contract, but it's mind-boggling to watch the regular ritual of HN posters blithely insist that anything more over $20K and a pile of pizzas for government infrastructure is a scam.
Not only do I think thats a strawman (I don't see that base level argument given that often on hn), but even if it wasn't, the truth is there is a lot, and I mean a fuckton, of money waste at the government level on IT projects. Trying to act like just because they aren't all a scam doesn't mean there isn't a systemic problem of waste that still needs to be addressed, and there are plenty of examples in even recent history that prove this point. More than just waste, the intention to fill good ol boy pockets in priority over getting the job done has significantly weakened the function of the projects, for example, the national security implications of abandoning thinthread that William Binney and Thomas Drake talk about... as just one in a myriad of examples. Virginia didn't have multiple counties become the new primary growth centers of millionares because of fair and balanced bids... and anyone pretending otherwise doesn't know how DC works.
The story of Healthcare.gov is pretty much the poster for this. They didn’t need to bring in a bunch of clever people to figure out requirements, that was already done and the site just didn’t work. To fix it they had to bring in a team of tech experts, who largely just did tech things —- competently and at a lower cost than the contractors who did the first iteration. https://www.google.com/amp/s/amp.theatlantic.com/amp/article...
While true, the team that was chosen also did not had the experience with public facing sites.
The contract was given to a team used to do intranet websites in classical JEE format and the letting them lose in such a project, designing as they knew.
> Trying to act like just because they aren't all a scam doesn't mean there isn't a systemic problem of waste that still needs to be addressed
No, of course not. But it's undeniable that these threads on HN (and other engineering-dominated forums) always include a healthy dollop of "I'd have done that better, because I'm a good software engineer" disdain. It's often paired with ignorance of how large-scale enterprise and government contracts and procurement work, what roles are necessary for large-scale projects with a significant discovery component, and what the planning process looks like when a team can't afford to launch-and-get-feedback or fail-fast-and-learn their way to a full specification.
I mean, this Deloitte project and the rollout are clearly a shitshow on a million levels. They got a no-bid contract for an eye-popping sum, then more than doubled it before the building even started! But the approaches boldly prescribed in most of the comments here wouldn't have produced a working system at the scale needed, either — only a cheaper broken one. A cheaper failure is still a failure.
> most missteps happen early, dooming projects before anyone sits down to code
This one is more insightful (in the big non-tech enterprise / government spaces). Because the takeaway for most HN developers who flip between projects should be: (1) learn how to recognize one of these projects & (2) stay away / find a new job when you do.
In addition to what others have said, from my previous experience:
- Does the customer have an IT org capable of fulfilling basic tasks you won't be authorized to do (app / db access, AD / rights modification, firewall adjustment, AV adjustment) in a timely manner?
- Can the customer provision necessary machines in a timely manner? If using VMs, is the virtualization team competent with their chosen platform?
- Does this project directly threaten the job of anyone whose blessing / help is required to successfully deliver the project?
- Is your leadership / your customer champion intelligent and willing enough to say "No" to scope creep when the customer gets excited?
- Do the PMs attached to the project have a basic understanding of how software is built and the systems involved?
- Does their management understand the actual processes being interacted with? (Major red flag! If their management / leaders say things in meetings that demonstrate a basic lack of understanding in what their own people do, run)
That won't save you at all. A lot of federal IT contracts are now written to mandate using Agile or Scrum. Many other forms of dysfunction that doom projects remain.
It also almost certainly applies to whatever every person here works on, no matter what industry you're in or how close to pure tech it is. In fact, I'm not sure I can think of a single field where even the most brutally difficult software tech problem is actually the difference between success and failure. It can be a prerequisite, and it can often be a bottleneck to iteration speed, but it's never the real problem, which is something SWEs need to understand in order to move up into leadership.
What they need to do is flip the script. Instead of only looking at the input side of these systems and how to plug into older systems not designed for it, look at the output side of everything that’s built and make sure it is api first so future projects can integrate much easier.
At the city of antwerp we flipped over in 2015 from a typical assortment of poorly integrated consultant-built monoliths to a microservices api-first approach. It became a hard requirement that all functionality provided by new systems had to be offered by a documented api, and most of the newly built systems dogfood their own api’s from the ui. In 5 years this caused a fundamental shift, where it gave us the ability to set up a covid contact tracing app for schools in weeks, because it was mostly the piecing together of well documented and battle tested api’s.
But most of the rest of it in these huge contracts could just not be done. The myth is that all of the rest is necessary when only a tiny fraction really is.
I have worked in tiny very early stage startups and old dinosaur companies. What comes across loud and clear is that early stage companies could use a bit more organization and dinosaurs could cut huge portions of what they “do” and then both would be in a similar situation to get things done reasonably well.
The bulk of the waste is an attempt to get people who don’t know what they’re doing into a position of getting things done. Sometimes it works.
Whats the prob? Its money changing hands. It flows from the Corp down to developers, managers, etc.
also it is good that corps and governments try to innovate and spend money on it.
Can it be more efficient?
Always(!)
Can it be more focused?
Sure.
But a big innovation takes a lot of factors to be well balanced and that alone is often not guaranteed nor possible given the circumstances. You just might not find the kind of devs it really needs (and not even knowing about it). So you hire the best thats available but it won't work...
Bad luck. At least you tried.
I see it that way. Whether vcs pump millions into 100 startups so that one succeeds or corps pump millions into 100 projects... Who cares? The money is not burned. It hopefully ends in my pocket
people get paid doing this. would you rather have they don't have a job because all of a sudden managers realizie that 90% of what they do is bullshit?
There was just an article on here last week about how software development is a learning process and programs are just artifacts of that process. Big, old, messy, and highly regulated domains will obviously be hard to learn.
If money is left over, apply to the next available item via (logic).
Unless money is sent twice, for two separate sub-items on the same item, in which case if the first amount is insufficient to cover the entirety of the first sub-item, the excess (if any) from the second amount, after being applied to the second sub-item, must be applied to the remainder of the first sub-item.
There are legal ramifications of doing the first way, without the second. And only the actual in-the-trenches people were able to tell us about the second.
Using outsourced process documentation is @&$&ing useless.
not to mention the fact that by definition the software developers know very little about the domain (given that 1/2 of them have less than 5 years of experience)
The project management part, which I agree is the hard bit, could be made tractable too. Other orgs manage this all the time and build sophisticated software which continues to improve.
This is a booking management system for appointments, the initial problem statement is trivial, the difficulties lie in interfacing with other systems, simplifying UI, getting data in and out, providing some customisation but not too much, etc. These are all things you don't know how to do till you start using it.
The process you describe of getting 'domain experts' in a room to decide on a spec without contact with the real users then throwing it over the wall to build is fundamentally broken. You won't get to even 50% done without real users - that's the problem, and it sounds like that's what happened here.
You simply can't build a system like this without frequent contact with real users (old people booking appointments, volunteers entering data in a hurry, pharmacies who need to add extra info for patients), experts on the other hand will often add negative value unless they are trying to use it.
If there was a will within the org commissioning the work to appoint one project lead and then get out of the way apart from demanding weekly builds from week 1 onward this would be tractable. Collect feedback from real users and deliver every week. This is not rocket science and a small team could and should do it if allowed the space to do so.
What you mean is that 95% of the time and money spent is about different bureaucrats fighting over turf wars and trying to get some credit in ,to be able to claim responsibility if it works, but at the same time not exposing themselves too much, just in case it blows badly.
One of my previous employers had a very good software and it sold such software to various other companies around the world. Such software was highly complex and really the top money could buy in the field. A yearly seat for a single user could cost up to around 5-7 k$.
Yet for years the company made most of its money from "installation projects" and "professional services", that is working with the client to understand its needs and implement customisation to the software to make it work well in the customer environment.
On engagements like these most billable hours are spent waiting on the client to make trivial decisions or give access to data/systems. Deloitte doesn’t usually hire dummies. IME it’s always the client being a total swamp creature.
The high bill rate is for your pain and suffering from having to deal with them.
> Deloitte doesn’t usually hire dummies. IME it’s always the client being a total swamp creature.
This has not been my experience, which is really to say that the world is big enough that you can find any combination of blame imaginable somewhere. Once you have misaligned incentives no part of the dynamic can be relied on not to cause problems.
It’s not a great look but it’s usually quite literally what the client is paying them for. I view the companies maximizing revenue the same way I’d view a shark: it’s what they are, and you can always choose not to jump into the water.
Wait! If there was a sane set of APIs and data interchange formats that was standard across the whole system, development would be the simplest part.
It's because they have to shoehorn their solution into a morass of garbage software, that this turned into such a complex task.
The real answer is to dump ALL these huge contractors, and their 'solutions', and make something new and better, with proper engineers directing the process.
Imagine if Deloitte had been contracted to 'invent' the Internet...
This is usually not the solution. You have to understand the requirements/know your actual users. Engineers don’t necessarily do this well. They build with the latest tech but as you see in this example, the primary users are older people who can’t use latest tech
Agreed. This is not a No True Scotsman -- understanding true requirements is a fundamental attribute of engineering. If you aren't doing that, whatever you are doing it isn't engineering.
Ha ha I know, but it isn't like that. APIs are always written as REST but the systems serving those APIs are not REST so everyone starts off building lots of GET APIs and then progress steams to a halt because it's tough to write loads of APIs for the many different types of write data flows in most systems.
The intersection of domain expertise, contracting/procurement creativity, and technical evaluation is the tough part to wrangle. Getting to the point where development can start is 95% of the battle in my experience.
Not to trivialize the actual development portion of the project, but most missteps happen early, dooming projects before anyone sits down to code.