I just don't understand these software deals, the price is so high. At 44 million, you can hire 146 engineers each paid 300k for a full year. Trust me, you need much less to build something like this, and there's no excuse for it to suck so bad.
I'm also curious, anyone know where the development actually happened? Did Deloitte further subcontracted out? Was it outsourced?
These contracts are generally bloated, but if you've never seen how a $10M+ contract for things like this works you're probably thinking about it wrong. The actual development piece is a pretty small portion of the overall work. The bulk of it is figuring out all the requirements, wrangling legacy systems and data into some kind of shape that can either be interfaced with or imported, documentation, support (including phone), client hand-holding, etc. Yes it's a lot of overhead, but this isn't a run of the mill green-field SaaS app a few 22 year olds can crank out in a few weeks.
This guy gets it. Development is literally the simplest part of the problem set.
The intersection of domain expertise, contracting/procurement creativity, and technical evaluation is the tough part to wrangle. Getting to the point where development can start is 95% of the battle in my experience.
Not to trivialize the actual development portion of the project, but most missteps happen early, dooming projects before anyone sits down to code.
Now think of the problem in reverse, from the government's side. Why are they contracting it out when all of the most difficult aspects involve integrating with government systems, policies and personnel - things that government already knows how to do?
"Because we don't have the expertise to do software development."
The comments in this thread, which I fully agree with by the way, just cement my belief that if government wants to deliver high-quality IT projects they need to start developing the expertise to do it in-house.
Ha, I'm currently a federal government program manager, so I am thinking about it from the government's side right now.
I've also been a consultant cog in a mega-corp and run my own small consulting business, so I've seen it from both sides.
Tech folks underestimate non tech challenges (basically anything outside of their domain), and clients don't actually know what they want. It's an age old problem in lots of industries.
I work in a similar sector but have a strong belief that the software delivery can and should become a capability that occurs within the government.
They should be diligent in what capabilities they choose to support. And ruthless in continually developing and pivoting these capabilities.
This is true of any large organization or enterprise that operates in a software enabled domain, so yes, most industries. The idea that software development is too difficult and should only be contracted out to fleety consultants is one that needs to be properly put to rest.
And to be clear, I'm not saying that consultants have no place. But for the type of technology and delivery as required by the project in the OP - there's no excuse.
I’ve been working in enterprise/government sector for a few years now and my observation is that they need to focus on better systems design. Everything is too convoluted. Trying to make the software fit the poorly thought out and implemented legacy ERP design from yesteryear is where most of the money is wasted.
Oh, my last disappointment of the week was being lectured via 2 screenfuls of Confluence with a “it’s more complex that what you make it” exposure of 2-3 layers of encrusted legacies. The PO was flexing and glowing how their managed to build into the new product this synthesis of bad designs. Which means entire departments get a list of rotating passwords by email every couple weeks. The network of ossified legacies just grew one more sprout.
I've wasted far too much time in my career doing exactly this for federal government customers. My favorite is when someone insists a feature is critical and must be replicated, but when I dig into it I find it *never* actually worked and as such, no one was using it.
Made worse when you have lots of money and even more stakeholders.
Perhaps worst of all, when failure has no true cost, avoiding failure matters far less. The CDC won't go broke because they blew through $44 Mil. Most every other organisation would. That's one of the challenges of government, the lack of creative destruction.
One of the biggest challenges for large orgs is that the knowledge they need to do a given thing is almost always present inside the organization — but it's fragmented and scattered across different teams, roles, and balkanized administrative boundaries such that the big picture is almost never available.
A high-quality consulting partner works to build the relationships and internal lines of communication that are necessary for the client organization to do more of the work — maybe all of it! — in the future. A cash-grab consultancy will just make themselves the organization's "glue" and bill until the money runs dry.
That last dynamic you described is exactly what I saw at a very well known university: upper management knew they weren’t getting the accurate details through the middle management filter, but shied away from fixing the cultural problems. Instead, a major consulting company was charging a considerable amount per hour to have 24 year olds with expensive suits interview the actual workers, summarize what they’d been telling their management for years, and present it directly to the senior level. Fixing the system would have saved millions but required fights they weren’t constitutionally well suited for.
Absolutely, but this is essentially saying "We're okay paying a 2-10% tax on our revenue, forever."
I think that's the reason you tend to see these unhealthy dynamics in extremely conservative, quasi-monopoly industries (government, financial, healthcare), where there's no real question of if they'll be a going concern. And no real rewards / attempts to outperform at the cost of increased risk.
Critically, if someone within the org volunteers to organize and manage the cross departmental knowledge they'll be shut down by people within the company for trying to accumulate power or stepping into someone else's turf.
Only an outsider can safely talk to everyone. They aren't a threat because they aren't part of the company and therefore can't take over.
It cannot be volunteer work. It has to be sponsored from the top. If it is, then a team of qualified FTEs can do this job similar to an external consultancy. But rarely will old-timer well-settled FTEs will leave their current position and take on this temporary risky gig.
Also, the end result and the path to get there isn't clear for FTEs – let's say they did the functional domain requirement gathering work – then who will implement it – they will have to contract out the development work – they may not be empowered to think and see through to the end goal fully.
On the other hand, for the top management folks – it is easier to temporarily empower the external consultants and terminate the engagement with them at any time – but they cannot do the same with FTEs due to employment laws etc.
What if it wasn't temporary, but just part of the CEOs office? The first goal should be to actively prevent these foul-ups by keeping open lines of communication.
I've always thought that a team like this is critical to keeping a company running. Any org that's 1000+ needs a 5-7 person team outside of any department that the CEO can throw at problems, but no org that I'm aware of has adopted this structure.
I can attest that the second modus operandi is the way long time contracts work from my experience.
Nearly every time all the knowledge and skills necessary are hidden and fragmented inside our client's org. With our access through all the management layers and silos we could work to fullfil the contract and also enable change within the client organisation and culture.
But why kill the golden goose? Our higher people' bonuses depend on us being the glue as someone else wrote in another comment. And a lot of my colleagues actually on the client's really do a great job in opening lines of communication. They work their a*es of to help the client.
Sadly helping isn't a monetary long term strategy for upper levels.
Though mostly about ladder climbing and relationship drama, the show 'House of Lies' has some really good scenes where the management consultants do actual work, which is pretty much portrayed as doing the things you've listed. Even as a fictional show, it made it easier for me to understand these situations and not get so flustered when encountering them.
I've been in technical, not management, consulting, but House of Lies felt disturbingly accurate.
Except I usually just stared at bad movies in hotel rooms to decompress after dealing with clients, rather than blow and strippers. Sue me, I'm boring.
Heh. That stuff has always felt like an inexplicably weird trope, like Hackers-inspired visions of computer people as living Mondo 2000 articles.
After 8 hours in a windowless conference room with a parade of stakeholders, making RACI matrices on a stained whiteboard and scarfing down cafeteria sandwiches during "bio-break," I want to _talk to my wife on the phone then stare blankly at the ceiling_, not go out for a night on the town.
I am definitely biased by my past experiences with tech consultants. I’m trying to sort out if it’s a service provider issue or if it’s a customer side issue.
From my limited perspective, it seems management consultants are focused on 80/20 solution approaches. While it seems technical consultants are focused on delivering MVPs that don’t seem to map well to the business space.
The disincentive that tech consulting has is that jobs (at least in my area) were usually bid at fixed price.
I have no idea if that's the way larger firms do it (would expect not).
Consequently, profit = contract price - consultant hours. Every bad happened from minimizing the latter.
Even if you do straight up billable hours with the client, your time was typically scheduled 1-2 projects in advance. This lead to being prematurely (in terms of things being "done") rolled off a project, onto a new one, and finishing / supporting the first from your hotel after-hours.
This was in a pretty cowboy area. I imagine different parts of technical consulting behave differently.
But yes, MVP quality code seems to be a constant. I saw that delivered from a top-tier technical consulting firm (ML solution) to a T100 customer.
To what extent is this due to neo-liberal "small government" policies, where from the highest political levels - through lobbying - it is decided that outsourcing is a win-win that leads to financial savings and budgets that are cut, and later on cut some more, etcetera?
I've witnessed the crazy dance of enterprise consultants in large corporations (not government) and it is astounding how much money is wasted in inefficiency.
I was in a small tech shop and we had just won an RFP at a big corporation, with a very practical solution and for a limited domain. Our down-to-earth approach led to so much enthusiasm with corporate employees, that they sent all their consultants to us to investigate interfaces to ever more back-end systems. Scope ballooned, and we - being a small party - were dictated to support all this stuff. It was utter madness to such extend that we decided to more-or-less graciously wringle out of the contract. Just-in-time before we were committed to a monstrous project that was 100% certain to fail. And that was not the only time we experienced this phenomenon in big corp.
On tasks where building the initial framework is the most expensive part, swapping out contractors only serves to abstract away complexity without reducing its cost.
It's better to have an in-house dev department in these cases even in the complexity side, if you realize that complexity in the contractor side is just as bad as complexity in the basic org.
Exactly, that's why I turn down companies that don't take a tech first approach, if you rely on tech, you ARE a tech company. Clearly the government doesn't see itself as a tech venture of any kind, but tech fails in silo. Tech first is the only way tech can succeed at delivering value to your business. Otherwise, go back to traditional processes, stick to SharePoint, email, excel, and manual logistics. And ya, this is true of the government as well, if it wants to modernize and leverage software innovations it'll need to start approaching things with a tech first mindset.
the UK has done exactly this - to make all Government Digital Services in house. It's been a blockbuster success in my opinion - they have some of the best UX/UI in the world and their websites are ultralight and downgrade well.
There are a few issues with authentication, but there is support.
Everything that's related to dealing with the UK government btw is all in ONE website, where everything is explained to you in Plain English - gov.uk
Coming from a country where this doesn't exist, I was blown away by every interaction I've had with the system. It's simply amazing.
As a foreigner living in the UK, I concur. All .gov.uk are _extremely_ functional. Applying for the pre-settled status (cos Brexit) was actually an enjoyable experience! I had no idea what was going to happen when I started the process, and it was all done in 30min top. It made me _scan my passport with my smartphone_ while flashing weird colors and it worked flowlessly, that absolutely blew my mind.
I've actually looked into getting a job there even though I'm from a start-up background
Yes that was some of their best work. I believe the flashing lights allow for some sort of 3D reconstruction of your face? And scanning the passport with NFC!
One of them is high-tech DYI solution, the other one is super simple yet not used at all by any other govt service I've interacted with.
Every single GDS project I’ve been on is 90% freelancers all via a small handful of companies — it’s not really “in-house” but at least it’s better than absolutely everything going to capgemini, Fujitsu, NTT, ATOS and the like, sure.
Exactly. The UK in-housed their entire website IT team, and it's one of the best set of government websites I've seen worldwide, UX-wise and performance-wise.
I worked for the government and it was pretty damn hard to get further training. We had no funds for conferences, classes, travel, or any such. All the improvement was down to me, what I could learn about R/stats/STATA from forums, from colleagues, and from the 1500 member analyst email list. There were all sorts of forums/libraries "within" the government system, however their search functionality was garbage and they didn't have enough users to make it a useful forum for most topics. Sigh!
The sad, true reason why they don’t is that government programmer jobs will pay 1/2 - 1/3 what a private sector job will pay in the DMV (DC, MD, VA). And the ironic part is that the reason private sector pay is so high is largely attributable to these big federal contracts being handed out. But as things are, the federal government cannot hire enough qualified people in the location (by DC) that they want to.
I’m going to be discrete but I know of a Deloitte project where they are allowed to use AWS Fargate, but the gov’t side is limited to IBM mainframes. Why are they so limited? Who knows, but it’s a deep question, that would require total reinvention. Perhaps, in the end, it might not even be worth the risk because the gov’t is so essential. They can’t move fast and break things. But it does look insane as a result.
FEDRAMP requirements and auditing are difficult and expensive to meet. I've seen the cost of these in bids. We said something like $20k per month for what would be a $50 per month EC2 server. It was a shocking thing to see. I'm guessing Deloitte is charging six figures a month to host this thing.
Does the government, in fact, know how to do those things? How would we know if they did or didn't?
I notice that they spend oddly large amounts of money on anything for which we know what the cost roughly should be -- e.g. CDC web sites -- and don't seem to do so very quickly or very well. Should we assume that they're a lot more competent at things we know less about?
Agencies are less likely to be competent in non core areas. This should not be surprising. There is no career path for you doing software in an agency like the CDC run by bio science people. Same for the Air Force, which is run by fast jet pilots, the Congress by lawyers and liars, Commerce by thieves, etc. Even banks and trading firms, who live and die by software, are run by accountants and economists.
Unless an organization recognizes the core importance of software and carves out a specialisation stream and C level ownership of it, they rarely achieve more than brief or isolated competence.
This is the reason why SV is so appealing -- nowhere else do SWEs hold sway over entire companies and industries.
Govt can do excellent software if it becomes a priority, they could pay competitive wages (there are special pay scales for doctors and lawyers), but they choose not to because of risk aversion from politicians, who mistakenly believe they can outsource the risk.
For something like this, paying Eventbrite to do it wouldn't have been a bad place to start. But the fat cat consulting firms never have domain expertise beyond getting Govt contracts.
There's a couple of reasons government IT projects turn out this way.
In my experience, a lot of the time it's not so much that the contractor is trying to make money by prolonging the problem. It's that the customer doesn't know how to be a competent product owner that prioritizes and scopes things properly.
Then, frequently the software being designed ends up replicating broken/inefficient business processes, so it ends up being difficult to make cleans and efficient software. This becomes worse when there's legacy software in the mix where the customer wants the contractor to port over all of the clunky functionality of the old app. Which of course isn't documented in the first place, so you've got to perform software archaeology to figure out what it did.
Some other show stoppers I've seen are when key stakeholders act as gatekeepers and prevent developers from talking to SMEs directly, so you have to build your requirements based on what the person with the 10,000 foot view knows. So of course you get incomplete and sometimes completely wrong requirements. And you might not find out how wrong they are until after you've spent a lot of time building things
Just those dynamics alone can add an enormous amount of cost or outright doom a project. I could probably fill a blog with all of the dysfunctional stuff I've seen in federal IT, but suffice it to say that these factors can easily balloon costs and timelines to a comical degree. This $44 million failure is actually small compared to some other government IT failures like the billion dollar Air Force accounting system.
There is often the expertise in house, but over years the internal teams get terribly under-funded and the poor service that results is then used to justify spending vastly more amounts on external suppliers.
This always comes down to piss-poor and craven senior management decisions.
I'd never in a million years argue that Deloitte did an awesome job on a large government contract, but it's mind-boggling to watch the regular ritual of HN posters blithely insist that anything more over $20K and a pile of pizzas for government infrastructure is a scam.
Not only do I think thats a strawman (I don't see that base level argument given that often on hn), but even if it wasn't, the truth is there is a lot, and I mean a fuckton, of money waste at the government level on IT projects. Trying to act like just because they aren't all a scam doesn't mean there isn't a systemic problem of waste that still needs to be addressed, and there are plenty of examples in even recent history that prove this point. More than just waste, the intention to fill good ol boy pockets in priority over getting the job done has significantly weakened the function of the projects, for example, the national security implications of abandoning thinthread that William Binney and Thomas Drake talk about... as just one in a myriad of examples. Virginia didn't have multiple counties become the new primary growth centers of millionares because of fair and balanced bids... and anyone pretending otherwise doesn't know how DC works.
The story of Healthcare.gov is pretty much the poster for this. They didn’t need to bring in a bunch of clever people to figure out requirements, that was already done and the site just didn’t work. To fix it they had to bring in a team of tech experts, who largely just did tech things —- competently and at a lower cost than the contractors who did the first iteration. https://www.google.com/amp/s/amp.theatlantic.com/amp/article...
While true, the team that was chosen also did not had the experience with public facing sites.
The contract was given to a team used to do intranet websites in classical JEE format and the letting them lose in such a project, designing as they knew.
> Trying to act like just because they aren't all a scam doesn't mean there isn't a systemic problem of waste that still needs to be addressed
No, of course not. But it's undeniable that these threads on HN (and other engineering-dominated forums) always include a healthy dollop of "I'd have done that better, because I'm a good software engineer" disdain. It's often paired with ignorance of how large-scale enterprise and government contracts and procurement work, what roles are necessary for large-scale projects with a significant discovery component, and what the planning process looks like when a team can't afford to launch-and-get-feedback or fail-fast-and-learn their way to a full specification.
I mean, this Deloitte project and the rollout are clearly a shitshow on a million levels. They got a no-bid contract for an eye-popping sum, then more than doubled it before the building even started! But the approaches boldly prescribed in most of the comments here wouldn't have produced a working system at the scale needed, either — only a cheaper broken one. A cheaper failure is still a failure.
> most missteps happen early, dooming projects before anyone sits down to code
This one is more insightful (in the big non-tech enterprise / government spaces). Because the takeaway for most HN developers who flip between projects should be: (1) learn how to recognize one of these projects & (2) stay away / find a new job when you do.
In addition to what others have said, from my previous experience:
- Does the customer have an IT org capable of fulfilling basic tasks you won't be authorized to do (app / db access, AD / rights modification, firewall adjustment, AV adjustment) in a timely manner?
- Can the customer provision necessary machines in a timely manner? If using VMs, is the virtualization team competent with their chosen platform?
- Does this project directly threaten the job of anyone whose blessing / help is required to successfully deliver the project?
- Is your leadership / your customer champion intelligent and willing enough to say "No" to scope creep when the customer gets excited?
- Do the PMs attached to the project have a basic understanding of how software is built and the systems involved?
- Does their management understand the actual processes being interacted with? (Major red flag! If their management / leaders say things in meetings that demonstrate a basic lack of understanding in what their own people do, run)
That won't save you at all. A lot of federal IT contracts are now written to mandate using Agile or Scrum. Many other forms of dysfunction that doom projects remain.
It also almost certainly applies to whatever every person here works on, no matter what industry you're in or how close to pure tech it is. In fact, I'm not sure I can think of a single field where even the most brutally difficult software tech problem is actually the difference between success and failure. It can be a prerequisite, and it can often be a bottleneck to iteration speed, but it's never the real problem, which is something SWEs need to understand in order to move up into leadership.
What they need to do is flip the script. Instead of only looking at the input side of these systems and how to plug into older systems not designed for it, look at the output side of everything that’s built and make sure it is api first so future projects can integrate much easier.
At the city of antwerp we flipped over in 2015 from a typical assortment of poorly integrated consultant-built monoliths to a microservices api-first approach. It became a hard requirement that all functionality provided by new systems had to be offered by a documented api, and most of the newly built systems dogfood their own api’s from the ui. In 5 years this caused a fundamental shift, where it gave us the ability to set up a covid contact tracing app for schools in weeks, because it was mostly the piecing together of well documented and battle tested api’s.
But most of the rest of it in these huge contracts could just not be done. The myth is that all of the rest is necessary when only a tiny fraction really is.
I have worked in tiny very early stage startups and old dinosaur companies. What comes across loud and clear is that early stage companies could use a bit more organization and dinosaurs could cut huge portions of what they “do” and then both would be in a similar situation to get things done reasonably well.
The bulk of the waste is an attempt to get people who don’t know what they’re doing into a position of getting things done. Sometimes it works.
Whats the prob? Its money changing hands. It flows from the Corp down to developers, managers, etc.
also it is good that corps and governments try to innovate and spend money on it.
Can it be more efficient?
Always(!)
Can it be more focused?
Sure.
But a big innovation takes a lot of factors to be well balanced and that alone is often not guaranteed nor possible given the circumstances. You just might not find the kind of devs it really needs (and not even knowing about it). So you hire the best thats available but it won't work...
Bad luck. At least you tried.
I see it that way. Whether vcs pump millions into 100 startups so that one succeeds or corps pump millions into 100 projects... Who cares? The money is not burned. It hopefully ends in my pocket
people get paid doing this. would you rather have they don't have a job because all of a sudden managers realizie that 90% of what they do is bullshit?
There was just an article on here last week about how software development is a learning process and programs are just artifacts of that process. Big, old, messy, and highly regulated domains will obviously be hard to learn.
If money is left over, apply to the next available item via (logic).
Unless money is sent twice, for two separate sub-items on the same item, in which case if the first amount is insufficient to cover the entirety of the first sub-item, the excess (if any) from the second amount, after being applied to the second sub-item, must be applied to the remainder of the first sub-item.
There are legal ramifications of doing the first way, without the second. And only the actual in-the-trenches people were able to tell us about the second.
Using outsourced process documentation is @&$&ing useless.
not to mention the fact that by definition the software developers know very little about the domain (given that 1/2 of them have less than 5 years of experience)
The project management part, which I agree is the hard bit, could be made tractable too. Other orgs manage this all the time and build sophisticated software which continues to improve.
This is a booking management system for appointments, the initial problem statement is trivial, the difficulties lie in interfacing with other systems, simplifying UI, getting data in and out, providing some customisation but not too much, etc. These are all things you don't know how to do till you start using it.
The process you describe of getting 'domain experts' in a room to decide on a spec without contact with the real users then throwing it over the wall to build is fundamentally broken. You won't get to even 50% done without real users - that's the problem, and it sounds like that's what happened here.
You simply can't build a system like this without frequent contact with real users (old people booking appointments, volunteers entering data in a hurry, pharmacies who need to add extra info for patients), experts on the other hand will often add negative value unless they are trying to use it.
If there was a will within the org commissioning the work to appoint one project lead and then get out of the way apart from demanding weekly builds from week 1 onward this would be tractable. Collect feedback from real users and deliver every week. This is not rocket science and a small team could and should do it if allowed the space to do so.
What you mean is that 95% of the time and money spent is about different bureaucrats fighting over turf wars and trying to get some credit in ,to be able to claim responsibility if it works, but at the same time not exposing themselves too much, just in case it blows badly.
One of my previous employers had a very good software and it sold such software to various other companies around the world. Such software was highly complex and really the top money could buy in the field. A yearly seat for a single user could cost up to around 5-7 k$.
Yet for years the company made most of its money from "installation projects" and "professional services", that is working with the client to understand its needs and implement customisation to the software to make it work well in the customer environment.
On engagements like these most billable hours are spent waiting on the client to make trivial decisions or give access to data/systems. Deloitte doesn’t usually hire dummies. IME it’s always the client being a total swamp creature.
The high bill rate is for your pain and suffering from having to deal with them.
> Deloitte doesn’t usually hire dummies. IME it’s always the client being a total swamp creature.
This has not been my experience, which is really to say that the world is big enough that you can find any combination of blame imaginable somewhere. Once you have misaligned incentives no part of the dynamic can be relied on not to cause problems.
It’s not a great look but it’s usually quite literally what the client is paying them for. I view the companies maximizing revenue the same way I’d view a shark: it’s what they are, and you can always choose not to jump into the water.
Wait! If there was a sane set of APIs and data interchange formats that was standard across the whole system, development would be the simplest part.
It's because they have to shoehorn their solution into a morass of garbage software, that this turned into such a complex task.
The real answer is to dump ALL these huge contractors, and their 'solutions', and make something new and better, with proper engineers directing the process.
Imagine if Deloitte had been contracted to 'invent' the Internet...
This is usually not the solution. You have to understand the requirements/know your actual users. Engineers don’t necessarily do this well. They build with the latest tech but as you see in this example, the primary users are older people who can’t use latest tech
Agreed. This is not a No True Scotsman -- understanding true requirements is a fundamental attribute of engineering. If you aren't doing that, whatever you are doing it isn't engineering.
Ha ha I know, but it isn't like that. APIs are always written as REST but the systems serving those APIs are not REST so everyone starts off building lots of GET APIs and then progress steams to a halt because it's tough to write loads of APIs for the many different types of write data flows in most systems.
Bullshit. The broken system Delloit developed is being replaced with Eventbrite in certain states, pen and paper in others. All the users needed was a simple scheduling system which shouldn't cost anywhere near 40M to develop no matter how you spin it. Is Eventbrite works for this, why did no one actually think to just use that (or the many similar systems created for this kind of thing)? Pretty sure Deloit itself just wrapped something like Eventbrite around a skin UI and delivered that.
The only answer I can come up with, even after trying to be generous, is corruption. The fact that there was no bidding should be enough for that contract to be highly suspicious. People in the top are pocketing most of the 40M and everyone involved knows it.
As much as this contract will certainly be a barrel full of pork, and Deloitte can go eat a bag of hammers, I feel this take is a little naive.
A commercial service is optimised for profit, not broadest accessibility. For example Eventbrite won't spend money developing special features for very specific niche user cohorts because of diminishing returns.
Government websites can't work that way, they have to be accessible and usable by all citizens. Think of people with different learning abilities or language proficiency, the likely environments the users might be in, and their general technical abilities.
Here in the UK there are strict rules about use of clear language, well thought out user flows, how to ask for certain types of information etc that has all been backed by years of research. We have a whole government department that covers this and will audit your new service to make sure it is good enough.
Having done this as a web developer for our Government, it was a massive eye opener to have a copywriter pick through the language I used in a form flow, and point out how confusing it could be to some groups of people. Or indicate that the method by which I was asking for an address made no sense for certain remote locations, and it needed to.
A few specific things to point out on Eventbrite in particular. It seems to me that you are likely have a whole bunch of other requirements too. Off the top of my head you'd need more powerful management tools to control: simply setting up so many locations and dates, who can book when (patient cohorts, priority groups), the logistics of vaccines and vaccinators, tracking vaccine batches used for each person.
> Yes it's a lot of overhead, but this isn't a run of the mill green-field SaaS app a few 22 year olds can crank out in a few weeks.
I get your point, but also yes it is. The issue is superficial, it's like when I try to Zoom with my mom, it'll take 10 times longer as I need to explain and bring her along every single step of the way. This is the same for these type of contracts. A lot of the slowdowns come from people just not letting development take its course as it should.
Just look at the article, the states ended up choosing to use a bunch of SaaS startup not even designed for this use case in its place, like eventbrite.
And look at the issues:
> “It won’t work on Internet Explorer; it only works in Chrome. The ‘Next’ button is all the way down and to the right, so if you’re on a cell phone, you literally can’t see it,” says Rowe.
That's not a project management issue, that's a full on development issue, it doesn't even work on mobile in 2021 !!
I worked at some government contractor a few years ago and one of the tasks I was given was to oversee the implementation of a ticketing system for a client. The basic requirements were for something slightly better than using an excel spreadsheet with basic search functionality. After a preliminary sales call with one well known vendor, they told us the next steps to implementing this system was to bring a consultant on-site to assess all the ancillary requirements, at a cost of nearly $100k. I had a little laugh about it with some colleges and my manager and I kind of wept a little inside, because I doubt this is a rare occurrence and it only takes 1 incompetent manager to create a massive amount of useless revenue for some vendor.
I've seen massive government contracts where "Big Consul" has some guy who does nothing else than take all documentation from the previous "Big Consul" and rewrites it according to their own internal format, and they bill €150 an hour for it, for months... And then when their time is up and the new tender starts the next firm does the same thing to rewrite it all in their format...
Even the part you mentioned: 'wrangling data for legacy systems' does not cost that much, it's still a small piece of the pie.
The 'bulk of it' goes to sales, BD, marketing, profits, layers of excess management, specialist consultants, accountants and lawyers.
This contract is an example of 'modern world graft'.
Healtchare.gov cost $1-2 Billion and it was also a mess until Google Engineers came in to save it.
Software is not like Construction where the costs can at least be well estimated and you don't need geniuses.
It's frankly very depressing and this kind of graft and stupidity is all over the system.
There are actually not that many people who are both: 1) technically/project competent enough to figure it out and 2) have enough gravitas on the business side to make proper bids. Partly that's the fault of government.
To be fair, it's also bad buyers who change requirements all over the place and throw projects in dispensary often for what are ultimately political reasons.
And when it goes way over, it's usually easy to identify the problems, it's just often politically infeasible to address them.
With construction - the numbers are on the table.
And FYI - aside from massive project, municipalities and construction is happening all the time. There usually are not massive overruns for such projects.
> 'wrangling data for legacy systems' does not cost that much, it's still a small piece of the pie
Are you sure? Because I think it might possibly involve:
- interfacing with an old COBOL system
- interfacing with an old DB2/ORACLE/(legacy database) with a "very creative" Schema and even more creative indexes (or maybe none - who knows). More bonus points if it's not even ASCII encoded
- Production system runs on a VM serving an old classic ASP or PHP app
- Data can be exported through an XML from Hell but that takes 2h to export.
You forgot the step where you even need to identify that these things already exist, and whats in them.
Are there governance requirements around the data in them? I'm not familiar with US law, but we're talking about PII in this example so what legal and privacy hoops need jumping to copy the data?
Thats before we get to "Can an person with this specific learning disability, or a person who doesn't have a fixed address navigate through this form?"
For many corporate client system developments this is my experience too. Producing the actual LOCs is not the primary cost; it's requirement gathering, figuring out how to plug in the new component into an existing system etc. As an example of the first, you might be digitizing a cross-department workflow. Your job now basically becomes to interview stakeholders from each dept and have them commit to a single online workflow. An example of the second is you might want postgre for your backend, but the client might want a single stop db on whatever they're using today.
Then there are challenges around integrating with existing APIs, workflows etc.
Let's not forget feature creep. Sometimes the client might not be able to visualize the final UX in its entirety (often simply because multiple people are involved),and therefore, they might only begin realizing its impact as and when parts of the new system becomes available.
Essentially, you are not paid to build an isolated system, but to understand well and play nicely with the larger context - human and technological - within which your system must live.
From the article it seems that they don't provide tech support neither to doctors, nor patients. And wrangling legacy systems in the field still may be worked out for much less money.
The real problem is that requirements are written in such a way that very few (in this particular case just ONE) companies can satisfy, so essentially there's no competition - exactly a kind of situation which lets to pump up prices.
And if you asking yourself are those requirements really benefit users, or specifically created for money to go to, you know, "right hands" - you're probably asking the right question.
Why are they even building custom software at all if the target demographic is 70+ ? And why are they making senior citizens opt-in for an appointment ? its idiotic. They should have nationalized the browser/OS vendors to provide a solution, for free if its a national emergency.
Not to mention the terms in these kinds of contracts. The only way to deal with them is to charge enough to be protected. I have sold product (hardware) to places like the Navy Surface Warfare Center and White Sands Missile Test Range. If you are not careful you can come out losing in a big way.
For example, for each day/week/month you deliver past the estimated delivery date they discount 5% from your invoice. Sound like great motivation to deliver on time, right? Well, no, it's a guaranteed 100% (or whatever you can get away with) markup beyond what you would have charged because if anything goes wrong you'll go broke.
That said, this issue of super-expensive government websites done wrong feels all too common. You have to wonder how much cash we burn repeating the same mistakes.
> this isn't a run of the mill green-field SaaS app a few 22 year olds can crank out in a few weeks
It’s probably exactly that. I always strongly wonder if you wouldn’t get a better result if you gave a pair of 22 year olds 1M and a year to develop and figure out all the same things.
Doesn't actually matter how old or young they are, just that they don't actually need more experience than you could get from two top-performing 22-year-olds that could easily make more than a million in progress over a year.
And it would have even been worth it to spend the first year doing nothing but intensively searching for these two so you could have operators so far above average and so well suited to the task, and so well agreed upon over a period of a whole year, that nothing else could even come close. It would probably be best with a pair who are equally motivated whether they would be getting the half million apiece or not, because a lot of that could be spent by now.
When everyone creating the software has the rare extreme performance ability that very few companies ever get to have even one employee that good, the more suitable the performance the fewer operators it should be expected to require.
This just has to be put into perspective compared to what you can get from a typical bureaucratic team for $44 million.
Average bureaucrats have never been good enough for average things.
And then supporting it. For a decade. 44m might be bloated, but, it goes quick and it is work no one really wants to do due to how hairy and legacy it can get.
These contracts often have the hosting and other support fees over some long period built in. I didn’t dig into this one specifically, but it’s quite common.
It sounds like they needed a senior business systems analyst to map out the functional requirements specifications document.
Usually, though not always, this helps to give a road map for the organization to follow.
Bugs will arise of course, but usually, it becomes localized, and can be addressed individually.
But for large systems, with lots of moving parts and data integrations with other systems, then you normally need a good, near-air-tight specifications document.
I work for a megacorp outsourcer, and when we run big projects, we embed Business Analysts in the business, so they are working alongside the key stakeholders/decision makers for the duration of the project.
A recent project was valued at around $10M USD, but for that the customer got a fully-featured mobile app that their entire blue-collar workforce will use every single day, replacing gigantic paper work packs.
The backend was hybrid-cloud, interfacing with multiple on-prem systems, including a legacy ERP system which had started life in the early 80's.
So, it was a big and complex project - but it was also a successful one, because we know how to run projects like this. There were various factors that led to success. The BAs were probably most integral. We also held regular week-long workshops with the customer to work on the backlog, with the architect/tech lead (me) and dev team leads all present too. There was a group of pilot users, that always had access to the latest features. And of course, we used automated tests.
Any big corp, like Deloitte, should be able to deliver a relatively simple website, even if it has multiple integrations with legacy systems. And they should be able to advise and steer the customer if they are woolly on what they actually need - there is only so much blame you can put at the customer's door when systems like this are a disaster.
I’d say getting the contract signed is already 1/3 of the total.
You need to be big enough to apply (requires investment), apply to many government contracts, you need to understand the legal terms, price them, understand the regulations, price them in, then price the software development itself and price any other provider (hardware happens very quickly), and if you guess wrong, like on 1/3rd of the projects, you’re in debt... and you have 100% chance of getting the contract if you price it wrong.
Getting the contract done is another 1/3 of the contract, and the lawyers to deal with disputes and prove that you’ve done the contract, is another 1/3.
When did we have the last nationwide vaccination campaign, whats the "legacy"? Pretty sure that was when IBM still made tabulators.
I think you will find that if only there was someone in charge to put their foot down, you can do without a whole lot of these things. But that's part of the problem of course: the government contracting out is just a bunch of clueless people making up a wish list with a partner on the other side whose only inclination is to say yes to everything.
You’re actually saying it costs millions to hand off data to a legacy system? A system that has probably had to consume something like XML or JSON in the last 20 years?
There is no excuse for this kind of cost for things as simple as tracking g appointments and completions regardless of data transformations and storage. There are no magical new protocols or data formats it’s using, it’s just a lot of people charging too much for them to have week long meetings to talk about their area of expertise. It’s an embarrassment.
Have you seen the vaccination setups? It's just car queues! I'm sure nothing there "flows" without a whole lot of manual data entry.
So now what, you are reporting this adverse event to VAMS who reports it to VAERS? There is no flow, it's all just duplicate effort from architecture astronauts.
Sounds like a lot of $6-700/hr management consultants that don't know much about tech but are happy to channel their inner Steve Jobs to try and architect something.
The 22 year olds might not be able to do it (I beg to differ), but can management consultants/accountants who barely took typing in school who have Excel as the height of their digital literacy be expected to have all the answers?
By the time this gets to technical folks, 70-80% of the budget has been burned.
Edit: Beware Management Consultants and Accountants encroaching into "IT". Their worlds are drying up because tech folks can learn business easier than business folks can learn tech.
There is a difference between "ageism", and "being honest about how hard some problems are to solve".
There are a lot of unbelievably brilliant 22-year-olds.
None of them have been building enterprise-scale architectures that turn messy, vague, and incomplete requirements that model even more messy, vague, and incomplete business processes for 20 years.
There are certainly a lot of 22 year olds that have been led to believe that they’re unbelievably brilliant.
I’m sure some of them are. I’m sure a lot of 22 year olds were brilliant in the 80’s and 90’s too.
Unless your thesis is that there’s been a sudden upsurge in the proportion of superintelligent earthlings in the last 25 years I’m unsure what your point is.
> None of them have been building enterprise-scale architectures that turn messy, vague, and incomplete requirements that model even more messy, vague, and incomplete business processes for 20 years.
Exactly! If you’ve been doing that for 20 years I do not want to work with you, since you are apparently perfectly content with that status quo.
I’ve been doing it for two years and it’s already too much. My highest priority is fixing their chronically useless requirements.
I’m glad you’ve clarified as when you said that it sounded like you’re astonished at the level of apathy and/or incompetence you’re surrounded with and I was worried for a second that you’ve just too little experience to understand that your predecessors have very likely been trying for much of that time to push back against the status quo and have better ideas implemented, likely swimming upstream against resistance you’re wholly unaware of.
But it’s clearly not that after all, which is a relief - I’m glad it’s just a case of everyone else that’s come before you being useless.
Is it just me, or are you being needlessly sarcastic here?
The resistance is ridiculous, but it’s still the single most valuable thing we can be doing to improve our project delivery.
Given the current division of people that care/not care (or are disinclined to act on it?). It is entirely likely that none of my predecessors did.
That said, if they had been doing it (and made any progress), I really wonder what the previous situation was. Maybe there just weren’t any requirements at all?
From experience I would suggest that it's very likely they had a very clear idea of where the problems were and tried to solve them, but found that they were not a priority upstream.
It's still an absurd situation, but I doubt very much it's the result of apathy - or at the very least the result of apathy from your direct predecessors.
Doing something for 20 years does not necessarily make someone better at it than someone who’s done it for 5 years. Arguably doing something for that long could even make you worse.
I don't mean to support any ageism and would be happy to withdraw my age reference.
What that may leave is decision makers who don't always have a high enough level of digital or technical literacy.
Folks who are not competent in technology decisions and implementations being in positions of decision making and spending is the point I was trying to make.
>Beware Management Consultants and Accountants encroaching into "IT". Their worlds are drying up because tech folks can learn business easier than business folks can learn tech.
Could say the exact same thing about tech recruiters
> and use Excel as the height of their digital literacy
In Germany we literally use fax and enter data by hand. A project that successfully moved things to an emailing Excel sheets based workflow would be absolutely revolutionary.
My former business partner back in the Web 1.0 days (hey D) had remarkable experience in scaling large systems. Studied under ast. Rolled his own ETL language to power primary research for a Nobel-prize winning social scientist. Could sound like Werner Herzog on demand. He’s the kind of man anyone shuts up and listens to. While helping him clean up a failed $25mm Deloitte project circa 1996, we got drunk in a Manhattan bar. Sometime around martini #3 he said:
“The next 10 years of software will be very boring. The entire field of CS will be employed turning paper process into digital process, at great cost. Almost all of these projects will fail because Americans are admirably earnest but tragically over-confident. You underestimate paper.”
“Good money for us though. But have a hobby to occupy your mind or you will be miserable. Zum Wohl.”
And that’s how I ended up order an ill-considered 4th martini. He was correct, although the interregnum was perhaps 2-3 years shorter than he predicted. Or 15 years longer depending on how one judges these things.
It doesn’t sound like that at all. “channel their inner Steve Jobs” sounds like everyone is staring at the clouds and practising some blue sky thinking, while the OP you replied to is talking about managing legacy systems, old data and bringing along an existing work force into using a new system. These are all the diametric opposite of “Steve Jobs architecting”.
Apple is about creating usable technology for the masses, not the few.
Apple also simplified access to legacy systems, like buying music and making it digital. Or maybe using your watch or phone to pay.
IPhone is geared for ease of use for anyone to adopt, including folks who arent super tech saavy..
Apple's innovation is charging a premium to create beginners in tech adoption who might not be experts at tech, who want yo feel empowered by everything just working.
As someone that’s been using Apple products for a few decades I don’t think I can agree that they make “for the masses, not the few”.
Their market share figures would show that if that’s their aim then they are failing. IMO Apple makes premium products for a specific kind of customer and have little regard for issues like backwards compatibility. Which is absolutely fine. But probably not a model you’d want the government to follow.
Was the original Macintosh easier for beginners to use than other computers?
Was the Apple Quicktime Pro one of the first digital cameras that anyone could relatively pick up and start using?
Was the iPod the first digital music player that just worked to buy and download music from end to end?
Was iPhone.. despite arriving as a smartphone a decade after Palm, etc, was the first usable smartphone for the masses?
Was iPad the first tablet that also just worked? Seems like anyone who can't use a laptop can use their finger on an iPad.
How about that AppleTV? Can anyone use it? It's easier than my AndroidTV.
What about that Apple watch, how many people who were new to wearables had it as their first? Seems popular with the masses.
Experts are usually the few, not the masses.
There's no doubt apple charges a premium. Their devices on average last 4-6 years, longer than the Android I'm typing this on.
Rather than being first, they do try to get it right for the masses. I'm not a happy Apple user, or fan with the keyboard fiasco. Apple's history though remains what it is.. easier to adopt by beginners.
It's users, as a percentage of mobile phone users (80% of population). So if a family have one iPhone, they can all count as users.
iPhone only bring affordable to richer people who mostly want the latest means lots of second hand items. I'll warrant Android users at the lower end keep their phones longer.
> Edit: Beware Management Consultants and Accountants encroaching into "IT". Their worlds are drying up because tech folks can learn business easier than business folks can learn tech.
Sounds plausible but I sadly only observe the opposite of management consulting “drying up”
This is not a 'software engineering' thing. If you look at it from a software development POV, then the issue is more of not having 'product managers' whose job it is to - understand the requirements, write the specs, basically champion the POV of the end user (after actually identifying the correct end user). Examples
a) The article talked about the software not working on Internet Explorer (IE). Older people still use IE. Software Engs with fewer years of experience tend to optimize for FF, Chrome and latest version of IE.
b) The article talked about 'wrong' placement of buttons (probably the software is not responsive or not optimized for phones or older phones)
c) Input fields seem to be rigid (older people will make mistakes)
d) Once you're registered, you can't switch from patient to staff. This should have been designed like a 2-sided marketplace with the ability of one person to sometimes play the 2 roles
If you don't have the right spec/requirements, the output of the engineers will still be bad
Internet Explorer has less than a 2% market share in the US, so under normal circumstances (given the sacrifices you’d have to make everywhere else) not bothering to support is is often a sensible decision. When it’s an access to healthcare issue though you can’t really exclude 2% of users.
That said, building something that works well on mobile (re your point b) but also works on internet explorer is a pretty tough ask. No reason you couldn’t build a basic bitches version of the site that works on old devices, then route traffic these based on user agent.
Idk, points a through c are basically already solved problems. I’d be disappointed to see problems like that if I was spending $20,000 let alone millions. Issue d though does feel like the sort of thing that you didn’t know was on the horizon up front, would be a pretty major rebuild to add in later on.
> No reason you couldn’t build a basic bitches version of the site that works on old devices, then route traffic these based on user agent.
Why isn’t this just the site?
I love Spartan, ugly, functional sites. That’s what the government should build, always. They should have style guidelines that are basically “no style”.
You’re not competing with anyone, there’s no need to look good. Function, function, function.
One reason that comes to mind is that a lot of people browse on mobile devices. These days a site really ought to be responsive to mobile viewports and a lot of “basic bitch” government sites are not.
Plain old HTML stops being responsive as soon as you try to do anything more than text. A plain HTML table would be horrible on mobile. If you want to reflow anything that isn't text you will need flexbox. There is literally no reason to not use CSS to make a site that works well on mobile and desktop.
I’m not convinced that’s true. Case in point: tabular data. I don’t think presenting something in a table is an attempt to make it look good, I think it’s the most logical and recognisable way of presenting certain kinds of information. But tables don’t scale well on a mobile viewport without extra work.
Part of function is layout. It isn't any more difficult to grab Bootstrap or some other out of the box framework that'll give a good consistent look than it is to go "no style".
There can be a medium. US Gov sites generally look awful. The UK Gov Digital team have done a way better just of standardizing things in ways that look nice.
>>Internet Explorer has less than a 2% market share in the US, so under normal circumstances (given the sacrifices you’d have to make everywhere else) not bothering to support is is often a sensible decision. When it’s an access to healthcare issue though you can’t really exclude 2% of users.<<<
That thinking is exactly the reason why this system turned out to be problematic. Even if it wasn't for healthcare purposes, bottom line is - you have to know who your 'actual users' are. Elderly people (60s and above) are still heavy IE users and so are quite a few government offices. If these 2% of overall browser users are the majority of your users or your early users, then not supporting them becomes a problem.
Detect IE, redirect to a static site with a 1-800 number, have a call centre staffed with people running Chromebooks schedule an appointment over the phone.
Probably cheaper overall, and certainly a better user experience for the tech-challenged seniors.
Something to note about this strategy is that older versions of IE (10 and older) only support TLS 1.0 by default. Below 8 you're stuck with TLS 1.0 and can't even enable anything higher. So if "https://www.cdc.gov" is using modern web security IE users won't even get to request data from the server, so you can't detect their browser.
To add more info. TLS 1.0 is prohibited from usage by many regulations and it will be immediately flagged by any security audit/tools. The government contract most certainly includes clauses about security and following recommended practices.
In short, anything below IE 11 is de-facto out of the picture.
> But the GOV.UK pages are written in simple HTML. They are designed to be lightweight and will work even on rubbish browsers. They have to. This is for everyone.
2% of traffic doesn't mean that 2% of users only have access to IE6. It may limit which computing device they can use but not supporting IE6 isn't blocking millions from your site, it is blocking them from using your site on all of their devices.
Probably not true. It means blocking them from using the site because they only have 1 or 2 devices and those are running IE.
The article talks about the fact that a lot of the seniors get help with their online stuff from the libraries (which are currently closed). What is the probability that folks like these have multiple devices at home and/or they can use them?
Years ago, a lot of my bosses at work (in Tech) still used IE. Lots of our clients used government issued computers which were locked down to IE, sometimes several versions behind.
At the end of the day, it boils down to - know who the actual users are.
never mind all the accessibility rules you have to follow (keyboard navigable, alt text for screenreaders for images, minimum contrast for sight impaired, constrained color scheme for colorblind, etc). . . that greenfield SaaS app made by 5 22 year olds isn't going to cut it here.
> Internet Explorer has less than a 2% market share in the US
This is true but if you are a Government agency in a Country of 300M+ people, that 2% is still several hundred thousands users, who are calling your tech support etc. etc.
If this was a private app then it would be different, IMHO Government websites have accessibility concerns that the rest of us can ignore.
Exactly. Civil Engineering does this better, I think. In a civil engineering project such as a bridge, the requirements may be written down by the customer, but the engineering design of the bridge is entirely the responsibility of a single architect. You can pretty much bet that the city council who bought the bridge doesn't know a thing about physics or engineering, so if the bridge falls down it has to be the architect's fault (assuming that it was built according to the plan). This gives the architect the power to say "no" to the customer.
I agree with you on all those listed points, but I don't know if it's my experience as a software engineer, but all those things you point out in general are something I find developers are cogniciant of, but often the processes in place doesn't allow them to do what's best, instead they are forced to always seek approval and follow stringent specs and requirements even if they don't make sense.
If you let qualified engineers drive those instead, in my experience, the end result is always better.
alot of these issues can be solved by pushing code to UAT or QA environments and having a bug tracker that allows everyone to submit their bugs. It seems like either none of this happened, or it was rolled out too quickly.
I don't think you're really paying for the development per se.
Most of the effort is actually managing the customer. That is, their contracting requirements, the whole requirements process, procurement hoops, change process, optics management, cost of "sales", reporting requirements, various additional compliance requirements...
This is a large part of the truth. Crank out a booking website? Give me a couple weeks. Deal with your byzantine organization, your incoherent standards, your security department coming out of the woodwork with list of requirements, your torturous system for provisioning government services like VMs in secure data centres... and then pile on that integration with a variety of legacy systems across another slew of organizations that bring all their own baggage to the process.
I would blame government, but I'm simultaneously experiencing all these issues with a Fortune 100 company. They're almost indistinguishable in the degree to which their cumulative organizational headwind is a hurricane I'm supposed to fly into. It's simply hard past a certain scale for purely people and procedure reasons.
And let's not forget that 1/3rd of the way through, when you're demonstrating an impressive amount of progress with a working prototype, some decision maker will say "you know, we could really make this a two-for-one if we tied it into this other program...", and start the 1,000 hours of meetings to get to development all over again.
Organisations are like people. Their culture is like someone's personality. Dealing with easygoing people is much smoother than paranoid delusional people with OCD. In the same way, dealing with government agencies is difficult because their corporate culture -- their personality -- is not what most people would categorise as normal.
These deals aren't normal deals. They're the equivalent of specialist treatment for someone with mental problems.
Take for example an organisation like SpaceX: They simply do things. If some manager goes to Elon with a rational but inherently risky proposal, he'll weigh the risk and then more than likely give the goahead. Other companies have a lower risk appetite, but they will take occasional risks. Sometimes huge ones.
Large federal government organisations are the diametric opposite of this kind of agile, low-overhead, no-bullshit private company. Their appetite for risk is precisely zero. That's difficult for people to wrap their head around. It's like a superconductor. It's not just a "really good conductor". It's not a "little better than copper". No, it's something fundamentally different. This kind of zero tolerance squeezes out entire lines of thought, just like a superconductor expels a magnetic field.
Notice how in this article the blame is on Deloitte and the "problem" is that there was a failure. This isn't a fault, this is the system working precisely as designed. Nobody will get fired for picking Deloitte, their purpose all along was to take the blame for the low-low price of a mere $44M so that a government employee making $100K won't have to have a couple of uncomfortable meetings.
The problem isn't that there was a failure, that happens all the time with software projects. The problem is that risk in government agencies is managed by people with essentially infinite source of free money that they can draw down on in order to protect themselves.
If a magic genie gave you a million dollars every week to spend on your own safety with no strings attached, wouldn't you? Would you say "no"? Really?
Reading "Let me try to explain using analogies.", I was worried someone would come up with another car anology.
Seeing orgs as people with characters, oh my, my eyes are opened now. Sounds weird but it makes sense the few times that I have worked on contracted projects.
For me, an added complexity in gauging the other side of the contract is that their personality changes depending on the actual people that are involved. When some higher-ups join the project it in later stages, the personality may turns even more mental than it already is.
A lot of “government contracting” is having people whose job it is to find the people that can authorize a sale and then hand-hold them through all the paperwork needed for them to actually make the sale.
This. Gov contracting is a rare, specialized skill. That's why the same big companies keep getting big contracts over and over. If you're a company without these skills you have no chance at landing one of the giant contracts. The CYA and auditing requirements are too insane to even write the proposal without a full-time staff of contracting specialists and a long track record of prior contracting.
(As a new contractor you might be able to get a small-money contract from e.g. the SBIR but those max out at roughly $100K.)
This is the overview, there are other websites that go more into the details if you google it
But basically it is survival of the most boring and the most detailistic. Then of course you lose the tender because you weren't "in the know", or you missed the requirements of the Toilet Procurement acts of 1974 which specified the thickness and traction resistance of toilet paper used in government purchases.
Oh and if you're "lucky" your Health contract management software will suddenly have to be able to order pizza as well, you know, it's "just an extra thing". But you won't get paid while that doesn't happen (ok this might not apply to the USA specifically, but you get the idea)
I met a manager of this once. She told me Oracle hired anyone with a college degree (hers was English) for her job because “managing developers isn’t that hard, you just have to keep track of milestones etc”
Because governments outsource everything possible and end up with no persistent talent that really understands whats going on so you end up spending all the money on interfacing with contractors.
My recollection of prior HN discussions is that a large part of it was that it involved interfacing to systems in a large number of agencies or departments. Most of those systems were old systems, independently developed systems, that had not been designed to have APIs for other agencies to use (or had not been designed to have APIs at all).
I think it might also have had to interface to state systems in the states that were not running their own ACA exchanges, and to the systems of numerous insurance companies.
As software goes, it's pretty likely that if you hire 146 engineers to build a website that could be built and supported with a team of 8.. you are almost certain to get a buggy mess that takes 146 engineers to maintain and support. After all, the architecture will need to accommodate 18 8 person teams, provide scope for ~20 Staff engineers along with a full layer of middle management.
I wouldn't be surprised if these proposals take into account headcount as a positive for both quality and job creation purposes.
I used to have a few friends who worked at deloitte as analysts I'd often catch up with them for lunch or after work drinks and I can guarantee you this was most likely made by a bunch of coked out 22 year olds working until 3am every night for a couple of months
It reminded me of a university style group project, constantly astonished that they’d turn over a few 10's of thousands of lines of python code that (usually) some government department was paying millions for. Not to smear my friends who worked there who must have to work incredibly hard given the conditions and the stress they were under, I always suspected their code must have been absolutely riddled with bugs, and I shuddered to think what it must have been like for the next poor grads who had to work on it after them.
And it was always some project where I was amazed there wasn't already some mature software out there that accomplished what they were doing anyway.
Many years ago early in my career I worked at an an engineering firm as a third-party contractor doing programming for a large municipal blackwater system.
Despite the fact that I was working on critical code implementing core requirements of the system, I was only paid $20/hr. I believe the firm was charged $40-60.
Long before I started working in the product and before any working software was deployed, the majority of the budget was spent. I believe they had at least three civil engineers, and a hands-off software architect who I only saw once, developing a document with hundreds of pages of studies and requirements.
I remember one severe issue we had was that I was not allowed to test on the real database or one of a similar size. So a main query was tested the day it was deployed (on some extremely expensive Oracle server which was very outdated). And unfortunately my query did not perform adequately on the larger dataset. But the senior programmer decided not to tell me or to try to fix it. He just truncated my query with LIMIT and said nothing. This resulted in the query breaking and showing an error, and totally incomplete and invalid data. It is amazing it actually continued at all from that point but unfortunately it did. I only found out about it several weeks later when the project manager decided to take me on a visit. Strangely, no one ever mentioned the issue, nor addressed my immediate suggestion that the software could not operate without the problem being fixed. I assume the other programmer just said "well that guy is inexperienced" and they had other projects to work on and no budget left so they just left it for quite awhile. Anyway I could not believe that so I left shortly after.
I once hired a company to do some structural repairs. The company usually did contract work for the government. Some weeks went by. I was billed a progress payment of $5000, but the company hadn't actually accomplished anything - just reviews, several site visits, meetings, evaluations, memos, emails and constant reassignments of who was working on it.
I fired them and refused to pay their bill (they hadn't even produced a plan for how they were going to proceed). Hired another outfit, who did the entire job front to back for $5000, and got it done in a couple weeks.
> I just don't understand these software deals, the price is so high.
Don't contractors do everything they can to milk government contracts as much as possible? I'm pretty sure I've even seen people provide advice on how to do this effectively right here on HN. So I can definitely understand how it happens...
The difference is they aren’t paying the engineers 300k they are paying an endless amount of administrator executives with “industry experience” aka 20yrs emailing spreadsheets between other managers at IBM.
I realized this once when I started attending (Canadian) gov ‘innovation’ events and ‘startup’ funding initiatives.
There was hardly anyone I’d consider an engineer that is benefiting from these systems. It’s a giant well organized grift for washed up “industry” execs and others hangers on with friends in the right places.
And I don’t mean to be ageist saying that. That would be over simplifying, but there is a inherent preference to those with safe names in their resume, regardless of relevancy. But there’s plenty of spaces in between for youthful people who understand how these meat factories work. Especially ones who understand their value, usually the unnamed ones working on the edges.
But by the time it comes to actually making the meat most of the money has already been allocated to the aforementioned people and they rush to find outsource the “real” work to some overpaid consulting company who are also experts at fleecing these gov contracts with endless meetings and perfecting the right checkmarks get checked.
I remember going to a Zuckerberg talk around 2010 at YC school at Stanford and he mentioned how he was hiring engineers even for the typical ‘executive’ jobs and I remember feeling like that’s the difference between startup tech companies and these failed hundred million dollar imitations.
Having helped to clean up a couple of those messes IBM left behind in various parts of the Canadian public sector, the only silver lining I see is that governments are starting to take in-house dev talent and the digital service model seriously because nothing (in the best case scenario) gets done if you rely on IBM. Hopefully this trend continues and governments grow some teeth in contract negotiations because of it.
Oh my sweet summer child, thou has not seen the internal working of government contracts.
In order to qualify to even bid on a contract there is a pretty high entry barrier. The amount of time it will take you to cross that barrier would be more than amount of time it will take you to start a real business. Bidding is another expensive process which costs money. If you bid on 10 contracts in a year and win one the cost of bidding on rest 9 contracts must be covered to the won contract.
Then comes the compliance costs of various government regulations. In a startup world you can take up accessibility as a P2 item but in government contracts it is a P0. So there are all these complexities involved which bloat the contract. Government has passed probably thousand regulations on whom you can hire and how you work, where the data is stored and what kind of background verification you have done on your employees.
Of course if you and I were to develop a similar website we could do it for $4M and spare some cash to buy couple of Ferraris.
PS. There is also bribes. Of course these are never called so but you have to spend good amount of time golfing with right people, throwing parties, offering front row seats to NBA finals and those sort of things to get your files moving.
Seems like it’s a few hundred million. IMHO Canada has no better example of idiotic spending than eHealth in Ontario. Literally 1 billion spent, and nothing to show for it. (Though they now have something, after effectively restarting.)
I've no love for Deloitte, but the one benefit they do have here is the ability to mobilize quickly on something like this. It's simply impossible to "hire 146 engineers" within any reasonable time frame, let alone get them all organized in a way that you can manage them effectively (and don't forget the project managers/scrum masters/etc that would need to be hired too). Deloitte, OTOH, has an entire existing business model based on taking vague business requirements and having huge teams of project managers/developers/consultants/advisors/etc ready to go at nearly a moments notice.
$44 million is an egregious number and I don't know how it's that high, but I'm betting at least part of it has to do with the fact that this was a website with an accelerated timeline, which probably necessitated a premium.
I was hired for a particular engagement with Microsoft years ago by a similar firm. Microsoft needed x, the firm had nobody so they put out an urgent call and passed me off like I was one of their long-time employees.
I think it’s safe to say nobody has a bench of 146 people ready to go at a moment’s notice.
Yes, these companies sub-contract, but that doesn't change my comment. Sub-contracting is something that Deloitte et al know how to do, and have existing relationships with staffing agencies, have an existing management structure to manage those subcontractors, have existing contracts so you don't need to waste time negotiating, etc.
And with all that said, companies like Deloitte do have hundreds of people ready to go. They may not have a deep bench for certain niche projects, but I used to work at a Big4 and for something generic like "build us a website for people to sign up", we had entire offices of people around the globe in our offshore delivery centers just waiting around for work like that. Oftentimes we would still subcontract because it works out to be cheaper if 10-20% of your resources on any given project are subcontractors, but they weren't always necessary.
Yeah, but if it got done in time but works so poorly that no one is using it, does that model actually work for anything other than being able to say "we got it done"?
This might be true for your average project, but the pandemic is not average; it’s very high-profile. Put out the word that the government needs help and you would have Googlers taking a leave of absence, like they did to fix the Obamacare website.
There are lots of great people who would rather be doing something meaningful than whatever they’re currently doing and governments need to do better at tapping into this.
It’s a different problem, but for a more recent example, look at how fast vaccinateca.com got going.
I mean it's certainly no secret that the US government suffered from incredible incompetence when it came to managing the pandemic.
With that said, the start of the pandemic was ~1 year ago. Even if it started right then, 12 months for a country-wide, government-run IT project is a rush job.
This isnt that complex, its an appointment system. with the resources the government had for this they should have been able to produce something workable in 12 months. I find it hard to believe there wasnt something in place they could retool to serve this purpose
Without enumerating the criteria for "right", you're probably not going to get a sensible answer.
A lot of the debate about pandemic management boils down to criticisms of, "did it wrong," without saying what would tip the threshold in the other direction.
This is problematic because it inspires a lot of talking past one another, resentment, and miscommunication. Ultimately it turns into a churn of frustration, defensiveness, and offense.
But even if you manage to get a response to agree on what, "right," means, the depth of their example is likely shallow. (e.g. "Such and such had only 12 infections because they such and such.")
This, too, is problematic, because it presupposes a ton of equivalent variables between the two countries. Population demographics, geographic structure, legal structure, culture, economic stratification, etc.
Conflation between a, "did it right," and a, "did it wrong," country is practically impossible, at least in the context of pandemic response. And anyone who tries to suggest otherwise is either ignorant, lazy, or obtuse.
If you get a satisfactory answer to your query, I'll move to the moon.
That is not the point. America fancies itself as an exceptional country (and it certainly is in certain respects) so it is only fair to expect exceptional results from the American leadership.
Can't we throw away the information implied by the contract that resulted in something worthless?
(I'm not taking the dismissive 'this is easy' side of the argument here, but the project failed, the lessons learned need to start there, not with the terms of the contract)
It might sound crazy, but a previous team I worked on would build solutions in this exact space (public sector) in 10 weeks using one PM and 3-5 engineers. And we would do it for free.
Aside from all the other answers, it would cost you millions to hire and onboard 146 qualified engineers. A place like Deloitte can drop them on your doorstep with a week's notice. They also have all the product design, strategy, and organization layers required to stand up the equivalent of mid-size enterprise from scratch really quickly and then tear it all back down a year later. I've worked in this business for smaller agencies and we could absolutely work miracles for stodgy organizations. My biggest account probably took in $20M all told and I'd say we earned every penny and launched a really successful product really quickly. Failures can happen too and they're usually a lot more complicated than just incompetent developers. It's just as likely the client is unable to articulate need accurately and corners get cut.
I work in defense and I always say the govt pays $100 for $1 of actual work. It makes me pretty angry at times as a taxpayer.
The worst part is that we know government software contracts are done completely wrong but the government doesn't want to change. Too much money to be made.
Or you could hire 73 engineers each paid 300k for a year, and have 22 million left over for hosting/operations/overhead costs... Absolutely ridiculous.
You get those 73 engineers in a room, then what? What do they write? You get 15 engineers, 3 DBAs, 5 sys admins + their managers. While that's spinning up, you get a bunch of business analyists to write the requirements. A first version gets written, it doesn't do what people want and the data it works from is really crappy, so you have to write a bunch of custom edge logic to deal with the crappy data. Repeat several times, including massive changes in scope.
Like is there bloat on these projects? 100%, and a lot of it, but saying "just get 73 solid engineers in a room" doesn't really get you very far when the requirements are really really unknown.
Yes, and in the process of that discovery it turns out the "process" is 100 different processes for 100 different people. One person performs data entry by faxing a handwritten list of excel data. One person just tells the current DBA what to put into the DB. One person uses a homegrown webapp that some intern built to automate some of their process.
"This is asinine!" you say. "Just make sure you dont break our existing workflow" they reply.
My first IT job was with a hosting company that noticed how much the goverment contracts were bloated. They then spent ages trying to become an offical supplier so they could undercut companies charging so much more. Literally, one of the issues they had was that so many people were curious as to how the company could make a profit since the prices were so much lower than all the other contractors like Serco.
We also ended up being in contact with Serco about a tender they had to redo one of the goverment websites. We ended up not getting any work from that because Serco was under pressure to stop outsourcing all their contracts.
Goverment contracts just send lots of money to companies who already have lots of money.
And the government can own the code, publish, and as the site finishes up, that team can proceed to apply all they learned and built to other pressing tasks.
A decade of that, and a whole lot of government would be awesome.
Of course, these arguments have been made many times. Money in politics prevents it.
I have presented them. I have watched large donations and a couple nice dinners make the whole idea go away.
I believe Deloitte just PMs the project and farms it out to a sub contractor. This is standard operating procedure.
I understand the Deloitte prefers heavily documented and waterfall development with overseas developers doing the work. Basically top heavy and expensive management and cheap dev resources.
>At 44 million, you can hire 146 engineers each paid 300k for a full year.
You can, but the government has to jump through a lot of hoops. Even government contractors have to jump through a lot of hoops. Sometimes there are regulations affecting the hiring process even for government contractors, e.g. drug tests, citizenship requirements, background checks, location requirements, etc. Every relevant law named after a sympathetic child adds $5B to development costs. And so forth.
I mean, given the results, it seems pretty clear they didn't know what to ask for, how to specify requirements, how to verify they got what they asked for, etc. That sort of thing doesn't get easier if you're hiring your own engineers.
Full disclosure: I have had the distinct "privilege" on having worked with all aspects of large consulting firms, from working at Cap Gemini, to being a sub for a smaller company on a large gov't contract with Lockheed Martin, to actually competing (and winning) against the large companies with my own, small consulting company startup.
In short, the above comment is spot on. Companies like Deloitte, Cap Gemini, PwC, etc. all make their margins purely on their reputation. Just like back in the 80s when the phrase "nobody ever got fired for buying IBM" became popular, I think from the mid-90s all the way through to even 2010s there has been a similar sentiment with large, multinational consulting companies.
These consulting companies know they have a strong reputation for large scale, enterprise software projects, and so they will milk margins by charging high rates, upselling all sorts of additional features, and fulfilling projects with the lowest cost resources, whether it'd be outsourcing overseas, utilizing H1-B workers, or bringing in recent college grads. (Accenture is probably one of the most notorious for this -- not sure if they still do this now, but back in the early 2000s, they were known for bringing in college grads from all sorts of usually non-STEM majors, and they would put them through a 2-week bootcamp (er "Core Analyst Training") out in St. Charles, where more focus was on the partying and hooking up than anything else, but then shortly afterword, they'd be put on projects where they'd be expected to do technical design and even coding, while being billed out at $180+/hour)
Now, in all fairness, for every highly publicized, disastrous rollout that one of these companies gets notoriety for, there are actually still dozens of much less public, relatively successful implementations that gets deployed without any issue or fanfare.
But despite this, the costs for even the successful projects are still ridiculously high. As the main reason for their huge margins are primarily in their reputations, then as their reputations continue to erode from things like this news piece, clients will start to wonder if the large costs can continue to be justified.
And I'm beginning to see this sentiment directly -- my small dozen-person consulting company has not only been able to compete directly against these larger companies, but we have started winning contracts against them as well. For the latest contract we won, we bid at about 20% of what one of the larger companies ended up coming in at -- they actually thought they were a shoe-in because they were the incumbent, having built out several other projects already for the client. But we ended up being selected not just because of our pricing, but because of our overall proposal, favorable terms and expertise on the domain.
What's even more surprising is that even though we offered "bargain basement" pricing relative to theirs, it will still end up being the most profitable project we have had to-date -- it just makes it crazy to think what kind of profits/margins that they can make.
I endorse everything you've said! Also, big consultancies have high costs (hello Booz! great lunch btw) like trade shows, political donations, lobbyists, ex-Agency/Military high muckety mucks on the payroll, etc. At least 50% of the project billings are set aside for this. Projects still need a 20% profit margin too. That might mean they're using budget staff; sometimes this means global software centres. If you think DBAs and coders in Mumbai or Warsaw might have a hard time understanding the organisational and political geography of all the States you'd be right.
I have no experience in this field, but do you think it's more likely they are making crazy profits or that they are just really incompetent? probably a little bit (or a lot) of both I'd imagine.
It can be a combination. Often it honestly depends on both sides of the project.
I've seen projects where bad requirements led to cost overruns and/or delays due to required rework.
I've seen projects where due to bad/strict requirements the customer's product basically became the 'learning platform' for a team to use 'the hot new web ui framework toolchain' of the year. I made sure to completely forget what I 'learned' about react/redux after leaving that gig.
I've also seen projects where the contract house threw bodies on that didn't even know the language used. I've seen horror story houses that are code-genies that overcharge to give you exactly what you didn't want.
But in this case My guess it would be a combination of requirements fustercluckery alongside what sounds like an integration nightmare; from my time working in an environment where we had to do a lot of 'etl loads' from different partners and/or acquisitions, you see a lot of snowflake cases.
From time to time I am trying to crawl government procurement portals for projects, and dropping the idea after a couple of hours again and again: only proposal, initial negotiation and documentation will probably require several people full time, this is even before any project movements..
Mind that, this it's the system where you're bidding for months for a project, and even if you have won and was good at delivery, you will be doing all the rounds for the next one again. So governments, sort of meaning to drive the price down via tenders, in reality are paying for all the losing bids too, and paying a lot.
It's a bit like some of the costs that have been quoted for some of the covid track and trace apps.
Whilst I do realise some of these quoted costs involve more than just the development of the app itself, some of the numbers are utterly ludicrous IMO as a software developer. e.g. Germany's app apparently cost approx €20 million.
Agreed. You can also argue that inflated contract cost in a statement of work leads to artificial labor bloat on the project. This benefits the consulting firm while disadvantaging the client. The unnecessary labor can create collaboration friction and lead to technical decisions like adopting microservices despite not being necessary to fulfill the technical requirements of the project. This kind of complexity doesn't guarantee a higher quality product is delivered to the client. Instead, it's used to justify follow-on work, further extracting dollars from the client.
It's possible there's context we don't have that exponentially increases the complexity of their tech requirements and justifies the price here but I'm skeptical. Looks more like misaligned economic incentives commonly at play in engagements between consulting firms and the gov.
The 44M is most likely over 5 years. Knowing Deloitte generally comes in on the high side of pricing I'd say it's around a 37 person team, cross functional including PMs, BAs, developers, testers, devops, Section 508 compliance, IA, user support.
Your point stands, but a nitpick - don't forget overhead. You should roughly double engineer salary to cover overhead, so it's 146 engs at 150k (still good), or 75 engs at 300.
It's Time and Materials/staff aug contracting - meaning Deloitte isn't selling an end product, they are selling staff on a per hour basis. As with other types of contracting, the hourly rate has a huge markup on it to cover indirect costs as well as profit. They make a lot less profit than you think though. Most of it is indirect costs.
I have no doubt the government got scammed, but often these estimates provision for things like ensuring phone support is available any time the website is up for accessibility requirements.
That's also why you see some government sites operate on a 9-5 basis
> That's also why you see some government sites operate on a 9-5 basis
I’ve always wondered why this was the case. But surely inconveniencing the majority instead of making it more convenient for the minority is a step in the wrong direction. Whatever. Off topic...
These deals are mostly to help out friends and line pockets. Building something that works doesn’t even matter. Government contracting is one of the biggest scams happening in broad daylight.
I'm also curious, anyone know where the development actually happened? Did Deloitte further subcontracted out? Was it outsourced?