This isn't a "bug". The problem is the software was not designed to do this and needs to be updated to support a change in requirements
Bugs are when software does something it isn't supposed to, or doesn't do something it is supposed to. In this case, it's doing exactly what it was intended to do when it was implemented and put into service. Since then things have changed, so the vendor needs to implement the feature request, not "fix the bug", but this takes time.
> They estimated fixing the SB1310 bug would take roughly 2,000 additional programming hours.
wtf?
Guessing they estimated 1 person-year, but that's absurdly high.
1 person-year is pretty much the smallest labor unit when quoting for modifications to government contracts.
If the government wants to change the logo on the login screen, they're gonna pay 1 person-year. Why would I quote less? It'll take over 1 person-year for another contractor to pick up and start supporting the codebase, and we won't support a codebase that another contractor has touched.
It's just the nature of government IT. Lowball the initial quote, then charge massively for any modifications and support now they're locked in.
Once you've executed on enough of these contracts, you develop an idea of what the unknown-unknowns are and plan for them.
It's government work, which automatically means access issues, dozens (or hundreds) of stakeholders requesting meetings, internal politicking, back-and-forth over change requests, etc. All of that costs real money and if you don't plan for it or make contingencies, you're going to be screwed.
These issues are not just limited to government either; any sufficiently large entity will have these same problems. So yeah, one person-year is a reasonable minimum viable contract period unless you have a process in place to fast-track RFP approval.
I heard that relatively smart govs stopped doing stuff like this that the outsource whole projects, but now they "borrow" programmers e.g "give me 20 programmers with higher edu and 5 years of experience" and we will lead this project.
Sounds like a good way to get unmotivated programmers who sit around drinking coffee all day and not doing any work, because hey - the longer this project takes, the more bonus our employer gives us!
This is why I think custom software contracting and vendorization is a “noob-trap”.
Software needs to be updated and maintained and you never know when you start writing it what the real requirements are. If I were a taxpayer in this state I’d be angry that my money would be going to a series of middlemen (e.g. a procurement consultant, program manager on the gov side, a contractor manager, the contracting company’s cut) rather than some state employed software developer.
Unfortunately the pay scale for state software developers never (I almost never say never) has a proper differential over other employees with the same level of experience, meaning they are often below 10th percentile of market rate, and thus the employees that end up in those jobs tend to be among the least skilled and motivated people I've ever dealt with. Even though contractors mess up, you can fire them. What's probably better is to keep a contractor available on an hourly rate for maintenance like this, rather than pay per change order.
I also find the government "requirements" process tends to create situations like this. Rather than build flexible software that puts some degree of trust in the person using it, they tend to overspecify the current bureaucratic process. In many cases, the person pushing for the software is looking to use software to enforce bureaucratic control that they have been unable to otherwise exercise, with the effect of the people the project initiator wants to use the software simply working around it. They then institute all sorts of punishments and controls to insure it must be used. This then results in the kind of insane situation we have here, where you can't do something perfectly legal because "computer says no".
> the employees that end up in those jobs tend to be among the least skilled and motivated
This. The job itself is terrible and as you mention the pay is terrible too. With conditions like that, I'm not surprised the product is buggy, over-budget, and difficult to improve.
If you're planning on breaking out of a certain contract-type or the other will be even more ridiculous, then consider putting out a separate RFP or having some special setup that allows you to just bypass or do the RFP at a higher rate. I'd recommend putting out an RFP for every project that needs it, even though it's a small number of the projects you're doing, and use that for the next project so the future RFPs won't get wasted. Otherwise you'll always run into these problems of a small number of big companies needing to get approvals because they're the one entity.
It gets complicated but there are some simple rules to follow. If you are running a non-competitive RFP then that might be okay because you are running a competitive RFP and so you get these opportunities to leverage your competitors, use one of the other competitive RFP rules for other projects, to use your competitor's RFP, etc. but you need to make sure the RFP you are running only does what the competition isn't allowed to do.
Having worked with a lot of contracted firms that do custom software development and platform customization, let’s not put them too high on a pedestal. What money you save on FTE programmers needs to be funneled right back into management talent to make sure they deliver a quality product (high utility, supportable by next contractor, quality docs, etc). Good management talent is hard to find and not something any government is known for. Consequently, next contractor probably has mondo ramp-up time deciphering what hacked pos the last guys put in with their c-team developers (cuz margins).
I thought they were pitched as a service opportunity. Regardless, it's a small team that only operates at the federal level, state and local budgets are much more limited.
"Salaries at USDS vary, but don’t exceed $170,800, determined by your experience and skills." So, they do have a relatively low top salary, but it is very difficult to do any salary for the government based on skills versus age (disguised as years of experience). At best you might find adjustments for degrees earned, which don't translate well to actual value.
Sure, it sounds like a public-sector employee gives better value but if you want to produce commercial quality then you still need management, consultancy and high quality HR with competitive salaries, otherwise you usually get mediocre developers with mediocre results.
Sometimes it really is better to pay a premium on the "day rate" to get something more quickly and to a higher standard. You also often have access to better support and maintenance.
Are you willing to pay market rates to retain that developer? Or deal with the constant churn as people stay long enough to pad their resume before moving along to the next job?
I’d not be too quick to dismiss the estimate. If you were asked to change a couple nested if statements in a prison’s guest management system, probably written in Natural, RPG, and Cold Fusion or some other obscure combo of tools, you’d be silly not to take a step back and be careful with your bidding.
I mean things probably seem to the outside like “it’s just two lines of code”, but if you’re messing with the business logic of releasing prisoners I’m sure you’d want to document how things currently work pretty thoroughly (inferred requirements), test/validate, have customer sign off on the existing functionality, make your change, re-run all the testing, requirements verification, etc. There’s probably a business analyst involved, a programmer, maybe even dedicated test manager, devops (do they have an existing dev/test system?). People in the prison bureau probably take their jobs seriously on paper, don’t want this change to cause more breakage than it fixes, and expressed that in the RFP.
Not picking on this comment in particular and not a domain expert, but just a few ideas:
- System needs to track sexual offenders and provide updates on their locations to relevant municipalities
- Parole officer check, requirements, and termination system needs to know who, what, and where
- Statistical record-keeping systems for sentence lengths need to be updated
- If felons, voting systems need to be kept updated
All of these sorts of actions would be triggered by someone’s release and probably involve interconnected systems that rely on truth data about the initial release.
One major thing my career in software and systems has taught me is that things are rarely as simple as they seem on the surface. I tend to approach such systems with humility.
Pay each inmate $10k for each day they're unable to leave prison past one week after the end of their sentence. Most inmates probably wouldn't mind staying a couple months longer if it means they get to leave with a million dollars in the bank, and it would correct the state's incentives.
You the taxpayers are the people who would prefer these people stay in prison than release them. You took on the responsibility when you put in place a system to imprison them. You could always campaign for prison abolition if you don’t want the responsibility...
> You the taxpayers are the people who would prefer these people stay in prison than release them.
That's a rather broad blanket statement bordering on collective punishment (which BTW is classified as a war crime by the UN). The status of taxpayer makes you a victim of the state, not necessarily someone with influence over its behavior and certainly not an ardent supporter of all its policies. The vast majority of the taxpayers had nothing to do with the situation and might well be opposed to keeping these people in prison if the facts were explained to them.
Perhaps the matter should be put to a general vote—those actually in favor of keeping the inmates in prison can split the cost of any wrongful-imprisonment suit in the event the state loses.
Luckily I’m not at war with anyone. The vast majority of the population do approve of the prison system though, and shrugging off responsibility for what that means seems perverse.
And yet, the software may save more than a million dollars in admin costs. And as soon as the most of the bug are worked out, it can operate less of those payouts. Don't let perfect be the enemy of good.
I don't understand how people here fail to see that completely. Traffic accidents also happen, doesn't mean we should abolish vehicles or allow only slow moving armored vehicles on the road.
Without the software, how will they know it's time to release?
Also, if you release someone and the system still shows them as being in there, that could lead to a very bad interaction for that person if they have any official interaction with authorities while they are out.
You assume that not only was there a bug in the software (in that it had not been updated to follow the law) but also it was terribly designed in that it could not cope with manual overrides.
None of those problems cited are the problems of the inmate/ex-con. It's the state that is choosing to go ahead with implementing buggy software affecting people's freedom, it's the state's duty to make things right.
I have done some reading about people who got convicted wrongfully. Being released from US prison is an extremely bureaucratic and slow process where nobody seems to be willing to apply some judgement. As long as the process was followed everything is fine even if it’s blindingly obvious that the person is innocent.
Getting into prison is easy but getting out is really hard and will take you years even if everybody agrees a mistake was made.
They might only know that it is not calculating some sentences correctly. That does mean they have figured out little more beyond "we know it is wrong."
This is a somewhat horrific argument for draconian imprisonment.
Maybe we'd have better results if inmates could file paperwork to receive the payments that would normally go to the prison for their imprisonment, starting on the date they should have been released.
The imprisonment data is coordinated between the prison, the police, the prosecution, the public service system and local governments. All of them needs to be up-to-date, or somebody would be screwed in process.
If the list is public, we can at least do the math independently to hold corrections departments accountable (filing suit to release eligible inmates when corrections won’t voluntarily release because of “software enhancements waiting to be built”).
Not that simple. There are behavioral factors that can affect a prisoner's status and that particular information would never be public.
It's a full-on bureaucracy where only the computers actually know the correct full calculation for every prisoner due to the complexity of the formulas, and when the computers can't do that correctly people get screwed with no recourse because it's humanly impossible to keep up with every detail.
I agree it's not simple. I am advocating for activist (engineering, political, etc) efforts against The Bureaucracy. Public data is a component of that, from a transparency and accountability perspective, very similar to FOIAing everything you can get your hands on (ie muckrock.com). You want to be able to "show your work" in broad daylight.
No, I want the presumably thousands of people who work at prisons to each go through their list of prisoners and determine when prisoners should be released, using the assistance of software but not completely deferring to software.
People working at prisons have no incentive to be kind or instill any kind of humanity in their actions, thus I would expect having them go through their list of prisoners and determine when prisoners should be released to have just as cynical a result as relying on broken software.
I don't understand why that's your default presumption. Although I don't have any data about this, my impression is that generally people are aware of how long their prison sentences are and are released from prison when they are supposed to be.
The problem with government contracts is that they are very much the class of customer who does not understand software requirements. They have all the worst aspects of a small company that doesn't understand software and so keeps whip-cracking the architecture, and all of the bureaucracy of a government with institutionalized PTSD about contract fraud and bribery.
If you have a choice between the regional furniture store and your state government, I'd have a hard time advising you on which one will suck less. It's a tossup. If the owners of the furniture store are old enough to have heirs who are late teens to 30-something, take the furniture store, because you can ask them to intervene.
At some point you have to pay the bills. And destress your employees who are constantly having to hurry up and wait.
There is a lot of reasonable opinions on the topic, but I would consider design and requirements bugs that render the software unfit for purpose to be bugs as well.
If they were design and requirements then fine, but you cannot consider something that was not asked or paid for as a bug. As with many cock-ups, somebody did not know enough to ask for the right thing.
On top of that, someone should have been in the room to tell the State Governor that their proposed change was not supported in software and needed something out-of-band to manage it.
It’s still a bug, even if the project managers or software engineers are not responsible for causing the bug. I think people are just using “it’s not a bug” to mean “it’s not my fault as a member of the software engineering team.” But of course there are bugs which are not the software engineering team’s fault.
Seems high, but this is par for the course for this type of work. You would usually estimate this in terms of months for a team. Let's say a team of 4 for simplicity -- that brings it to ~3 months.
I'd also be careful with the term "programming hours". I'm not sure how the news article got that or who said that initially, but it seems like a misrepresentation of the type of work needed. That estimate almost certainly includes everything involved in getting the code to production. You can imagine that means a lot of QA, red tape and holding the code's hand through environments.
It's a rubber stamp like many other things in this development model.
Not saying their QA is useful, but it's not tough to imagine their QA being many layers removed from decision making. ie: Program Manager -> Product Owner -> Project Manager -> Business Analyst -> QA -- not even mentioning reporting lines.
That person in QA at the end of the chain is likely following a script written & approved by the layers above them to a T. It's highly unlikely that they are empowered to go off script or raise tangential issues. Doing so may even cost them their job.
We all know that it takes more skill (and experience and domain knowledge), and more time up-front, to create software that is easier/cheaper to change later in response to changed requirements.
If you got the cheapest possible software up-front that (barely, technically) met the original requirements, you did it by hiring the least skilled people that could barely pull off exactly what would get the contract paid, and asking them to rush and hurry and pile up technical debt.
So.
In this case, the software perhaps shouldn't have even been considered fulfilling the contract in the first place, that's how crappy it was. The crappier the software, the more expensive changes to it are, we all know that.
> One department whistleblower said the number of problems with the ACIS system was unprecedented in their professional experience. “I have never in my life run across an application like this,” they said. “It’s just been one big cluster.”
To be frank, you probably want this solution to be vetted out pretty well. I really don't want to think of cases where the wrong person was let go. The sizing is what 12 people x 40 for 4 weeks? Really not that extreme for large projects.
I also don't understand why they can't identify these people and release them by other means...
I posted under another comment, but this to me is a software-literacy problem. You shouldn't have to hire an engineer to fix or adjust business rules in 2021. Period. The people who know the domain should be able to maintain (and really, create) any automation of that domain themselves.
This is partly a failure of education, partly a failure of organizational structuring, and partly a failure of software accessibility. But it's a gargantuan failure all the same.
>The people who know the domain should be able to maintain (and really, create) any automation of that domain themselves.
This is a recipe for disaster 100% of the time. One needs to know both the domain and software to change software in the domain. People who 'know the domain' but not software, when given the tools to modify software within the domain inevitably create an unmaintainable rats nest of more difficult to change rules, almost invariably in some proprietary WYSYWIG tool that creates more problems than it solves.
Bugs are when software does something it isn't supposed to, or doesn't do something it is supposed to. In this case, it's doing exactly what it was intended to do when it was implemented and put into service. Since then things have changed, so the vendor needs to implement the feature request, not "fix the bug", but this takes time.
> They estimated fixing the SB1310 bug would take roughly 2,000 additional programming hours.
wtf?
Guessing they estimated 1 person-year, but that's absurdly high.