The fact that there is so much B.S. in Scrum doesn't mean that it has nothing of value to offer, though.

Here's what I get out of it:

1. Sprints are a better way to organize than Waterfalls. I've experienced this over and over again personally, so I'm sold on this.

2. It's important to stop what you're doing on a regular basis to evaluate progress and problems. This always seems like a waste in the moment, but failure to do so leads to regret down the line.

3. Ship functional products as frequently as possible. This is better than waiting until everything is done, because you can get feedback early and often from the end user.

Okay, so that's 3 ideas. It's better than most.

One of the especially interesting things to me was that at the beginning there were a variety of different methods. People behind them got together in 2001 to figure out what was common, and that's where the Agile Manifesto came from. Of the Agile Manifesto signatories, I think only two of them were Scrum people. These days, though, most people thing that Scrum is Agile and Agile is Scrum.

I see three reasons for that. One, Scrum was the simplest, arguably the lowest common denominator; it had no technical practices. Two, it could be installed in place at existing waterfall companies doing what is effectively mini-waterfall, so there would be little disruption to the hierarchy. And three, it had a "certification" program, where a) anybody wanting a career bump could spend 2 days to get a "Master" certificate without taking any test or proving any competence, and b) any "consultant" wanting easy money could quickly become a Scrum trainer. Basically, Scrum became the Amway of software processes.

If you go by actual behavior, it turns out the highest priority at most companies is not actually to improve, to get better at making things for users. It's instead to make managers and executives feel like something is being done without disturbing the power hierarchy. Low-end Scrum fills that need adequately, letting you move marginally in the direction of agility, apply some new labels, and declare "mission accomplished".

And my point here isn't "Scrum bad", really. There are some great people in the Scrum world. My point is, "Business models shape outcomes, so be careful which you pick."

I think this is a key insight, and I suspect for many people (myself included), the phrase you use when you say "Low-end Scrum" is just the only type of widespread Scrum we've ever seen practiced anywhere, ever, to the point that it makes us question whether Scrum itself is partly responsible, even in an idealized format.

One of my good friends came up with the idea he called "barriers to anti-quality" to describe this. A good workflow process establishes inviolable norms that act like preventative barriers to doing things in an anti-quality way -- so that you specifically arrange the world such that trying to take a quality shortcut actually costs you more than just doing it in the quality way.

We proposed trying to judge workflow and management practices based on how willing upper management was to give up control of the process in exchange for erecting barriers to anti-quality.

If they wanted to force a team to pivot in some quality-destructive way, just on their whimsical preferences for reprioritizing different tasks according to vague business needs, they could not do it.

Barriers to anti-quality would be like Odysseus pre-committing to be tied to the mast, so then when the Siren song comes, he cannot give in the temptation.

I think a lot of the spirit of early Agile was meant to suggest something like this. Even though the method talks about being flexible and adapting when there are real business needs to adapt to, this gets bastardized when middle and upper management conflate adapting to real business priorities, and begrudgingly paying a cost in terms of quality lost due to pivoting developers, with demanding that people have to adapt to the management flavor of the month, unsubstantiated ideas about business priorities, often stemming from political power struggles, restructurings, quarterly earnings, etc. etc.

What I really wish Scrum or Agile had, was a more unbreakable way to let development teams say no, and decline work or decline pivots. Basically to veto changes made to the backlog, and say that it violates an anti-quality principle for no good reason.

A key failing of modern Agile / Scrum is that it doesn't create any type of stop-increasing-our-entropy-with-unsubstantiated-management-preferences notion of a barrier to anti-quality.

Don't get me wrong, this is hardly a failing that us unique to Agile / Scrum. But I think when people talk about why in principle these methods fail, it's that they lack any mechanism of enforcement for this. As a result, they can be so easily politically subverted to just be a pivoting, bludgeoning tool of misguided or unexplained management choices, that almost always trade-off quality for anything else, and instead of ever looking back to understand when an under-investment in quality actually cost them, they just explain it away as some other team's fault.

In a sense, Agile / Scrum (or 'Low-end Scrum' as you call it), is a tool to create managerial Dutch books. A manager gets to unilaterally disrupt on-going investment in quality without ever being held accountable when that gamble on short-term anti-quality causes losses.

By the end, it makes developers very cranky that they are being jerked around with all this prescriptive and time-wasting overhead of meetings, planning, estimating, etc., only for this political anti-quality nonsense to go uncorrected anyway. If that's going to happen anyway, then what's the supposed value of all that costly Agile-specific overhead?

If your business leaders are prone to changing their minds at the drop of a hat, you probably aren't going to actually change that. Putting process in place to make it painful to do that is just going to cause pain for the people who have the decision imposed upon them - the CEO's just going to tell other people to figure out how to make it work.

Is there a way to bottom-up build something focused instead on reducing that pain from those pivots? Just-in-time planning whenever possible, for instance, to minimize wasted time invested in would-be future projects?

The timeframe matters, here, though. Are the wild changes once a quarter? Once a week? Once a year?

I spend a couple of years at a startup I co-founded; requirements volatility was legitimately very high. We were very experimental, which meant we learned a lot, which meant frequent wild changes. We had a long-term vision, and plenty of theories about the future, but rarely planned more than a couple of weeks out.

This was honestly fine. The product manager (my cofounder) was really good at including us on the necessity of the changes, so it never felt irrational or out of control. Our process was highly tuned for minimum waste. And we were militant about a lot of quality-oriented practices, including unit and acceptance tests, pair programming, paying down technical debt, continuous deployment, frequent releases, and close collaboration with the product manager. That meant that however things changed, we were prepared.

I think if you can do it at a start-up that's shipping every few hours, it definitely works at much lower levels of requirements volatility. To me the real issue at scale is that if agency moves away from the team level and gets concentrated at the executive level, then decisions at least appear irrational, and often are because real-world information becomes ever more removed from the decision makers.

Now, every decision (new or changed) has a cost to be implemented. Some more than others. If you have decision-makers who refuse to recognize that, and think the amount of work they can ask a given staff with a given level of skill to accomplish in a set time period is unbounded -- no process is gonna save you from misery.

I work on 3-12 month projects in an agency, we don't really do Scrum. It's mostly some version of waterfall, and occasionally dual-track agile. Generally the process of determining what to build is squeezed into the cracks. I just recently built about 40% of a prototype for a SaaS in a couple weeks with fractional team input, and then had to lay out the rest of the flows in a week. After that, more needs popped up (I can only imagine the shitshow that development will be). This is the worst case scenario.

In less-worse cases, it seems like this becomes an issue of how to compare a requested change's value to its cost. Without enough data, this can be really hard to actually know well enough to have a grounded position on. It seems like consensus here is to focus on the cost of the change, rather than the potential value.

It also seems like there's little faith in managers. I don't totally know what it's like out there for ya'll though.

Yes, exactly. I know some good people who are dedicated Scrum fans. In their view, the engineering practices are all implied, because if you do have a relatively autonomous team dedicated to continuous improvement, you'll get there. And it works, because they are smart and dedicated people committed to doing the right thing.

So in that sense, the idea of Scrum is a good idea. But Scrum-the-movement and Scrum-the-business have fallen very short of that ideal. So in practice about 98% of Scrum implementations I've seen make me despair for humanity.

> What I really wish Scrum or Agile had, was a more unbreakable way to let development teams say no

One interesting version of that was the Extreme Programming Bill of Rights. One version of it is here (and in XP jargon "customer" is something like "product manager":

http://www.agilenutshell.com/bill_of_rights

It's not perfect, but I think it was a good attempt to make clear that there's a useful balance of power.

Once we got approval for this, it meant that product owner had to focus on communicating to us why the business had certain priorities and desired certain pivots, instead of just nuking our desired plan of work and superseding it "from on high."

It meant everyone had to trust us as the team that we wanted the business to succeed! (Who'd have thought!) And we weren't going to act childish and just demand backlog items and sprint work that satisfied our preferences for some fun project or some back-burner idea. That, as grownups, we would actually consider when business priorities made sense to supersede our intended plans, and act accordingly.

After a while, we exercised our privilege a few times to ignore or decline "urgent" work that the product owner relayed to us. Instead we invested in much needed refactoring stories, fleshed out more serious design documentations, researched detailed trade-offs between approaches or third party tools we would need in the near future.

Over about a year's time span, this worked incredibly well. The team felt happier and more productive. There was less tension with the product owner since he would not circumvent our plans and dictate new priorities all the time. And people learned they could trust us to understand the underlying business considerations and deploy our team accordingly.

In our company, this was very much as "un-Agile" as it gets. It was the farthest thing from what "Agile" methods meant in the company as there could be, and in large part we only got permission to work this way because our output was in a very specialized domain area that had really critical requirements for some large business initiatives. Most teams could not have possibly negotiated this arrangement.

Like clockwork though, the director of our business unit was surprisingly fired and replaced with no explanation or warning, and the new director stopped us from doing this at that point. It reverted to the usual thing where sprint planning is just a Wheel of Fortune gameshow where you learn the radically new and unplanned pivots you'll be forced to make for the next two weeks. Like trying to win a 100m race by dancing on a DDR mat at the starting line.

Anyway, long story short, the barrier to anti-quality principle was that it's totally the team's discretion as to what work is admitted into the backlog. If the team ignores legit business considerations at their own peril, to goof around with fun stuff, then the team owns that failure. But no matter what, they are not told by the product owner what goes in the backlog, and it must be a sacred right for the team to tell the product owner "no" when management proposes to add work or change direction in a clearly stupid way.

If management doesn't buy in and I can operate my own low-overhead, low-formalism workflow, then so be it. I suffer the badness of management, but I don't also pay time waster costs for a formalized process whose sole purpose of better facilitating productivity is categorically disallowed in the first place.

I personally, as a matter of experience and opinion, would take it further to say that certain properties of Agile / Scrum actually add fuel to the fire and allow management's already bad behavior to manifest in even worse ways. But even if someone doesn't agree with that, I think the point stands: why incur the overhead costs of Agile if the same outcomes happen because of mismanagement anyway?

I had a less extreme version that still served our needs. My rule was that a fixed percentage of effort, say 80%, would go to business priorities. The product manager could certainly ask about where the time was going, but they couldn't claim it unilaterally. The rest of it could be ordered and reordered as the product manager thought best. We in turn could ask about that.

This worked well for us, in that it let us keep the code clean and make tooling improvements without having to explain to a non-developer what was going on.

It was also helpful that we were good at expressing technical-sounding things in user terms. E.g., all scaling and reliability work was put into the backlog with titles like, "system handles 1000 simultaneous users" or "system survives web server failure". And a lot of research was covered by the product manager buying information rather than features.

Of course, this was at a startup, where there was no "on high". The product manager was the CEO, so in theory he could have fired us, but mainly we just collaborated on getting shit done.

I think the focus on methodologies is to get a good result from an uneven team. Companies desperately want to treat programmers like standardized workers that can be mixed and matched as needed. The siren song of the methodology is that maybe it can achieve that goal. I have never seen this work in practice.

There are no quick fixes. People can improve but it takes time, dedication, and support.

Of course that's your experience. If you judge teams based on outcomes, then you're going to conclude that teams that succeed are good, and teams that don't must be bad. Unfortunately that offers no guidance in how to become a good team. The goal of software management as a practice is to find patterns of positive behavior that make good teams. Agile, in general, is a collection of those patterns.

At the start, team 2 continues doing awesome, and while team 6 doesn't win, they improve. Next round Team 2 isn't doing quite as well, but still good. Team 6 has improved a bunch. After a couple of rounds, team 2 is falling apart and team 6 is now winning (or 2nd place, I forget).

My experience, admittedly anecdata, is that great teams perform well when they're in an environment where they're allowed to succeed. I've worked with great teams who did terrible work, and it was pretty much always due to a bad environment (e.g. an owner who would redirect their efforts every few days, so no feature ever got to done-done). I've also seen (but not personally led) teams that I had originally somewhat written off, who under good leadership managed to do some pretty damned impressive work.

And don't get me wrong, I'm not devaluing the potential contribution from a single amazing person, or a team full of amazing people. My clients hire me because I have a (locally) rare set of skills that their team doesn't have, and help them get over hurdles that their team can't (currently) tackle on their own. But if you take a great set of individuals and put them together in a shitty environment with a shitty leader, you're going to get what you get.

What if there is no reasonable and reliable way to become a good team, and the only way to go from a bad team to a good team is to replace most of the team?

I suspect the same can happen with software developers.

Of course, there are people that will be bad anywhere and top guys that will be great anywhere.

I don't think that's the world we live in though. I've seen smart, sociable, effective developers fail in misorganised teams, and I've seen teams of ordinary people produce very good software by getting their processes right.

I don't doubt that there are bad developers---I've definitely encountered them. But software development is a team sport, and it makes no more sense to just "recruit good people and leave them alone" in software than it would to do the same in basketball. Great teams have both talented people and winning strategy.

That's surprisingly common across the field. Most developers work on small teams doing small but often very long lived projects. Code staying in production for 30 years is surprisingly common outside of SV.

The assumption of a standard programmer is just never a reality as well. I actually saw a startup try to go full religious agile and only hire "full stack" developers to try to homogenize them so that they would fit the agile mold better. It didn't seem to work very well, people are people and naturally gravitate towards data modeling, or UI, etc.

And honestly that is all that most companies actually want/need. Most teams are mediocre and most projects only need a mediocre result. Getting something as good as a mediocre results out of your mediocre teams would be a massive improvement for a lot of place.

What agile does when executed right is to fully embrace that, and wrap it in a process that ensures that when you reach that point you're able to finish the current iteration, ship what you've got, and move on leaving things in a good enough state. For me that's the core of agile, and in my experience at least you can't pull that off with a mediocre team. You need a team where at least some members have the experience and the pragmatism to know the difference between delivering a minimum viable product and cutting corners in a way that'll hurt in the future.

http://agilemodeling.com/essays/barelyGoodEnough.html

Given the constraints of the estimate we've made for a story that could really go much longer, except that we've only allocated two days... I think the problem you're hinting at is that different people have different ideas of what the Minimum Viable Product should look like.

Accordingly, on my team where we're all under-trained with respect to Project Management and mostly just making this shit up as we go along, we've identified several other points on the curve.

It's suggested that one should make some nominal efforts to identify, and stop for a sniff along the way to JBGE:

"that's definitely not good enough, try again"

"I'll know it when I see it, but I'm sure that isn't it"

"kinda-sorta good enough"

"that's nearly good enough, except for this one thing..."

Identifying some of these points along the way (and making sure you notice when you've arrived at them) in my opinion is a great way to help ensure you're not actually aiming for points somewhere deep on the wrong side of the curve.

The mediocre team also may have trouble getting to Just Barely Good Enough because of overshooting. But in my experience that's usually not the problem. Nobody is working far past JBGE into the wee hours of the "OMG this is too much, I love it but how long did it actually take?"

If anything, the feature has to be shelved or scrapped, or handed off to someone else, because it's taking too long due to an error in calculation of trajectory. Spending too much time on the wrong things. Incremental progress is the goal, and perfect is the mortal enemy of good.

Maybe in Silicone Valley, but most companies are happy if they can just find a programmer that knows how to program.

To combat this, this individual decided to try to reduce the specialization as much as possible. Everyone would do everything was the thought process, so no one could ever hold anything up.

This was a non-technical founder of a now dead startup, but it died of the much more typical problem of market fit- they did launch, and the site worked. It may be interesting to mention he was also using entirely offshore development teams.

My experience is that sprints interrupt my workflow to such an extent that I can no longer get anything done. In other words, they make a good team bad.

I can also see the business point of view in this. You want switchable puzzle pieces. Take this developer from this project and move him to that project. This developer knows X, Y, Z and that project needs X, Y and Z. So it's all nice and well. Only it doesn't seem to work in practice.

I want to be the guy who talks to the rest of the employees, implements things and get direct feedback from them. I dont want anyone in the middle of that. It just complicates everything and makes me less connected and known in the company.

Sadly once a team scales up past a certain size its not practical to have developers also going out, finding out what people want built, and handling all the feedback on that. You need someone (or a team of someones) gathering the often conflicting requests coming in, and prioritising them against each other.

Having said that, I do believe once a piece of work makes it to a point where people are likely to start working on it the developers who will be doing that work should be involved in the process. They should understand not only what is being requested, but also why its being requested, in my experience that often results in them suggesting alternative solutions that either bet solve the problem, or provide a 90% solution with significantly less cost. It also gives them the context needed when deciding on technical trade-offs involved in different design choices.

How exactly does agile prevent you from talking to your teammates and getting feedback from them?

I get it, but it's now how I do my best work, proxying questions and discussions through a gatekeeper.

Regarding decision-making, that can be a problem---some PMs and product owners are micromanagers, hamstringing their team. But they would be micro managers with or without agile.

Scrum processes are just a way to ensure that no stakeholder, including team members, is left out of those talks, left in the dark, or left unheard. Having a clique single-handedly and unilaterally dictating changes without communicating them or listening to any stakeholder doesn't serve anyone's interests. That's how projects fail.

True. Scrum is a floor, not a ceiling. Nothing prevents the team from going above and beyond the daily standup and the sprint review in terms of communication, and I've never been on a team that didn't communicate a lot more than that. But doing that standup every day ensures that there is at least a minimum of communication, every day.

Part of little-a agile is making changes and adapting -- it's literally in the name. If you have a process in place that prevents you talking to the people you need to talk to, change or delete the process.

I am the same way. I much prefer talking to users directly and come up with a solution instead of having a middle man. But I have noticed that a lot of devs much prefer somebody filtering things for them. I guess to each his own...

Here's the common pitfalls as I see them:

1. Sprints offer flexibility but unless you educate the whole company waterfall + gantt charts are what the business side wants

2. Unless you educate all members of a scrum team on what the retrospective is, and let the developers drive it, you won't be effectively improving processes

3. I've always felt conflicted about the agile "working software over comprehensive documentation value". To me it implies that documentation is easier when the product is working and/or the product might change based on feedback, so you don't want to re-write the docs. To people who don't learn scrum/agile it means "don't write documentation"

While design documentation tends to be anti-agile (but sometimes useful to avoid wasting more time on miscommunication and mistakes), end user documentation is, for all practical purposes, part of the valuable software product, as it enables end users to do what they want to do.

Unfortunately, being able to neglect end user documentation makes many outfits treat it as a useless luxury, either because they are only judged on software (e.g. passing a test instead of having satisfied users) or because they don't care (e.g. many introverted hacker/artist open source maintainers). It's purely delusional agility, like driving fast by not stopping for fuel.

That's not true at all. The whole point of organizing projects around scrums is precisely that there is no well defined goal nor can one exist. What exists are the client's needs, and those needs do and will change frequently and radically as projects move on.

Therefore, the classical project structure followed by grey beards does not make any sense in software development. What does make sense is to acknowledge that requirements specified by the client do and will change, and it's better to adapt to the client's needs and get everyone involved in the project. Therefore, projects are organized in small bite-size units of work that gradually contribute to implement a product based on the feedback received by all stakeholders, and that these small bite-size units of work are implemented in small time windows. These units of work are short-lived and the amount of effort invested in planning is limited to the goals set forth for that unit of work, which need to be manageable.

I don't feel this is true at all. The issue is that most clients aren't willing to put in the time or effort to actually see what their needs are.

You're assuming that it's realistic to expect that a requirements gathering process is able to precisely define all requirements, that these requirements will and cannot change, and that software architects have perfect information and are able to make flawless choices regarding the design.

None of these assumptions hold even in conventional engineering projects. The only reason why waterfall processes is used in conventional engineering projects is that it's so expensive to fix problems arising from these sources of failure that it's perfectly fine to deliver working but inadequate solutions.

No, I'm not. I am assuming it's realistic to expect that a company actually put some effort into finding out what they need, and getting us that information so that we can actually plan out a project.

"None of these assumptions hold even in conventional engineering projects."

Part of the reason for that is that companies don't do any of that research. They don't look at what they need; they think about what they want.

One of the biggest reasons people here dislike "Agile" is because management uses it as an excuse not to plan anything, and fly by the seat of their pants every two weeks.

Consulting is the pejorative we gray beards used for that activity.

You reminded me of another pithy throwaway line:

Agile didn't improve outcomes, it just reduced the cost of failure, allowing teams to fail many more times with the same budget.

Those hypothetical gray beards may come up with all the pejorative terms they need, but that only hides the fact that they are entirely oblivious to the reality of running a successful software project, one which actually meets the client's requirements and delivers working products. Therefore, they spend their time coming up with pejorative terms while they insist in wasting their time and effort forcing the proverbial square peg (waterfall) in a round hole (software development projects)

> Agile didn't improve outcomes,

That statement is patently false. No one can make that claim with a straight face in a world where paying customers make it their point to change fundamental requirements on a weekly basis.

In the old timey's waterfall world, a waterfall project that's executed flawlessly is a project that ends up delivering the wrong product that fails to meet the client's basic needs, thus leading to a very unhappy client that may even feel that he has been played.

> it just reduced the cost of failure

A sequence of small failures that converge to the client's needs is a whole lot better than a single unmitigated major failure that's ensured by following Waterfall methodologies.

Wow, amazing that the shoulders of the giants you stand on was never a successful software project.

>In the old timey's waterfall world, a waterfall project that's executed flawlessly is a project that ends up delivering the wrong product that fails to meet the client's basic needs, thus leading to a very unhappy client that may even feel that he has been played.

Waterfall was iterative as well, when the needs changed, so did the project plan. You bought that line, but that's not how it ever was in my experience. You know those version numbers software has? Those are the iterations. We'd cut a release about every 3 months.

Based on your comment, I suspect you've never done anything outside of scrum / agile, so you are comparing it to some mythical way of doing things you heard about (undoubtably from scrum consultants.)

Those giants you're referring to made a lot of mistakes along the way.

One of those mistakes was blindly trying to force project management practices that were driven by the need to allocate material resources into projects where the only resource that's allocatable is man-hours. In projects whose success is determined by how well are material resources spent, redesigning something midway is something that's entirely unthinkable and can even dictate the project's death. That's not the case in software development projects, where the only resource is the fixed amount of man-hours that's at the project manager's disposal. With the development and adoption of a couple of software engineering practices aimed at preserving the project in a deliverable state, redesigning entire modules is essentially free. Therefore we end up with project management challenges that superficiality may appear to be the same but are actually fundamentally different.

Therefore, different types of constraints result in different optimization problems that lead to different solutions and require different approaches.

> Waterfall was iterative as well, when the needs changed, so did the project plan.

The keyword you've used is "needs". You're assuming that changes are exceptional. They are not. In software projects, requirement changes are the norm, not the exception. Changes aren't needed, they are constant. If your goal is to meet the client's needs then you need to meet the client's needs, and not some line in the sand that has no bearing with the paying customer's goals.

> Based on your comment, I suspect you've never done anything outside of scrum / agile, so you are comparing it to some mythical way of doing things you heard about (undoubtably from scrum consultants.)

You suspected wrong. In fact, I have far more years of experience in waterfall projects in real world engineering projects than in scrum/agile. Waterfall projects make sense when the requirements make sense. Software development projects based on basic software engineering practices don't incorporate some fundamental requirements found in engineering projects, thus their efficiency can and does improve by following adequate practices.

re Outcomes, my data is old, I’d love to proven wrong.

I’ve read (elsewhere) that high(est) functioning orgs like google, facebook, netflix have made great progress. But the rest of us are just banging the rocks together.

PS- PMI & critical path, as I’ve done it, is the opposite of waterfall. One hard earned trick is proper ordering of the work. eg, after a project kickoff, first deliverable is the press release, second deliverable is a demo (even if its entirely faked).

My workaround was to do jelly bean estimating. Get guesses from everyone, then use the average. Worked surprisingly well.

We did other things to meet that target date, honor that estimate, of course. But that's a longer story.

Have you ever thought much about why, though? When I was young, I was super optimistic - and I got burned by all of the surprises (including the underspecified or unstated requirements), so I started to estimate pessimistically based on experience. But then I got stuck in this loop: "How long will this take?" "Probably two weeks" "What? Why two weeks? Why not two days? We only have two days. Say it will only take two days, or tell me everything you're going to be doing on an hour-by-hour basis to justify this high estimate!" So now I've just gotten really good at figuring out how long they want me to say it will take, say it will take that long, and shrug my shoulders when it takes longer. Since everybody else is doing the same thing (as they don't have any choice), it doesn't surprise anybody.

I'm told I work too slow, dive too deep. Now I'm trying to figure out how to do just 80%, "good enough".

When I was a kid, I published shareware. After getting a few 3:00am wake up calls from irate customers, I said "never again". I spent 15 years designing, engineering away technical support calls. It mostly worked. Later on... When my team(s) burned a CD, we rarely had to issue patches, hot fixes.

Customers loved us.

I thought that was the pinnacle of human achievement.

Now, I feel like I'm still fighting battles over things no one cares about any more.

In my practical experience, I multiply the amount gathered from the previous statement by two to have sufficient negotiation space for people 'higher up the chain' who have no idea about anything of this and simply pull estimates out of thin air or their rear without any proper experience except "previous time it was xxx so it should be the same" (which it almost always is not).

edit: I prefer to have some time left on larger projects to avoid overrunning, even though that might be common practice. Usually though time is cut short due to delivery constraints or selling a product before asking the development and you get the 'wasn't it done yet?' speech.

Hour Estimate : Volatility Level

    1 - 5 : Little to no volatility.

    5 - 15 : Small volatility (a day, maybe two).

    15 - 30 : Medium volatility (multiple days).

    30 - 60 : Medium to High volatility (days to weeks).

    60+ : High volatility (multiple weeks).

Additionally, as a work item increases in size the amount of "knowable things" increases as well. Of those "knowable things" there are many known knowns, known unknowns, unknown unknowns (thank you DR for that gem). There's a near 0% chance that a team will get all of the requirements in place up front on a 60 hour case.

If a scoping team is told that they have 240 available hours to work with, then they will pick a combination of items that fit within the 240 hour box. If they have four high priority 60 hour estimate items then they may pick all four for the release. In the event that four 60 hour items are chosen I would place a near 100% guarantee that the release will not go out within the 240 hour time frame. I would even wager that will be up to four weeks late.

That's kinda the whole point. Everyone sucks at estimating. Scrum proposes a way to do "what greybeards call project management" in a way that deals with this universal inability.

I didn't explain "the other stuff" for why jelly bean estimating works.

The number doesn't matter. It's the shared hallucination (consensus) for the deadline. It short circuited, resolved the debate about "how long". It secures the team buy-in.

Once we had a deadline, we managed the work to match. Closest analog I can think of is Kickstarter: initial goal and stretch goals. We had the must haves (stuff listed in the draft press release). We also had dozens of nice to haves we'd sneak in, time permitting.

Agile, Scrum, sprint style estimating is too fine grained, the horizon too near, to do effective estimating.

"Evidence Based Scheduling"

https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...

On the dev side, I have never attended so many meetings. Technical debt being addressed is generally against the feedback loop, unless the PO/Leads of the team make it their first priority. Grooming meetings are a mix of people who care, those who don't, and vague stories. To save time, any new story got sent to one of the devs for a sniff test before grooming, to help ask critical questions early. Typically, the request is reasonable, but may run into a contradiction of the system.

I have also found that, forcing scrum (or waterfall) on a product that is running at a difference release methodology is horrible. Got a customer that needs a new thing built by a specific date? That won't fit, unless the date is flexible and customer stake holders can be deeply involved in the process. Waterfall at least forces the customer to define the project up front and allows for calling out changes to the plan and thus changes to the scope.

As with all the above, this sits on how skilled your project manager/scrum master/product owner are at their jobs. Anti patterns abound.

And no, it wasn't just crud apps. It was a distributed security auditing tool. Yes, it had a database component, but it was much more than that.

A simple kanban board seems to give all the same benefits without the constant pressure and weekly death march to meet the end of sprint commitments.

If the sprint is not finished, well it's not a big deal. You move the tasks back to the backlog, or in the next sprint, or delete them if it doesn't make sense anymore. And you decrease the number of tasks in the next sprint. You shouldn't feel a constant pressure to finish sprints if the velocity of your team is taken in account correctly.

More to the point is, setting sprint goals is defacto pressure which many people and orginsations are poor at handling, and we can largely avoid it by not having people doing or managing the work care about end of sprints, but just that the tasks are scoped and are of a reasonble size.

Not everything has to be encapsulated in the TODO list. I mean, a Jira backlog is also just a TODO list.

For larger products with multiple teams working on the same code base it can even work to have a mix. The feature development teams follow Scrum (or something similar) and the maintenance teams follow Kanban.

* Get feedback often from the business owners

* Get feedback often from the developers

* Get feedback often from the users

It's funny how it turned into UML

There was a little overlap; in particular, Martin Fowler wrote an intentionally thin book called "UML Distilled". But the UML world was generally heavy on both up-front architecture and formal, heavyweight design process. The Agile world at the time was energetically opposed to that, thinking that the main value of UML was in having a consistent language for us to all sketch on whiteboards together.

My go to joke: RationalRose is like an 800lb angry gorilla sitting between you and your work.

Alistair Cockburn did a great post mortem about that era.

Characterizing people as non-linear, first-order components in software development [1999] http://alistair.cockburn.us/Characterizing+people+as+non-lin...

During that time, I strongly preferred lo-fi paper prototyping for UI. And I'd scratch out UML diagrams. Worked fine for intra-team. But it just didn't fly with management. We had to make everything more formal and pretty.

Oh well.

Jacobson was a little more method focused. You can see how his introduction of use cases started the idea of talking to your customer about what they need.

I remember seeing Rose and thinking this will make a huge improvement only to be repeatedly confused and frustrated that it seemed to make everything take longer.

At OOPLSA 98, Booch convened the first society of software architects (or some such).

Having read all his writing (ditto Yourdon, Brooks, Rumblaugh, etc), I was super excited.

During the Q&A I finally got to ask “What is software architecture?”

Booch replied “Software architecture is what software architects do.”

Bubble popped.

I gave up seeking the council of the priesthood and blazed my own path.

Every time I see a shop with designated Software Architects who hog the architecture work, it's a fucking mess. Because then Software Architects don't get rewarded in their careers for making software. They get rewarded for writing white papers and coming up with "brilliant" architectures and impressing executives who know less than them about technology. Which is mostly inversely correlated with actually making it easier for developers to get real work done.

But I don't think it was quite as incremental and iterative as all that. One, it was more a "process framework" than an actual process, so iteration length could be whatever the grand poobahs wanted. They were entirely fine with multi-month iterations. Two, as the "RUP hump" chart shows, it was always pretty waterfall-ish. And three, whatever the words in the book, the RUP adopters were generally large-company shops, and he who pays the piper calls the tune.

I definitely remember the RUP people coming to the early Agile conferences and being very suspicious of our wild ways. So I'm not sure even its originators would have called it an Agile process.

This particular comment, the "UML Distilled" reminder, triggered a few hindsight notions.

Much like XML and J2EE tainted Java, I'm now having trouble distinguishing between UML and OOAD. Though I preferred Fusion to UML, I'm now wondering if any OOAD based methodology had a chance.

"...thinking that the main value of UML was in having a consistent language..."

Guilty. For a long time, I very much wanted executable models, roundtripping. I got over it. Now I place more weight on concision, DSLs.

But here's the new (to me) notion:

Maybe UMLs (and RUP) greatest benefit was the idea of having a list of questions to ask. You know, a methodology.

Writing this out, it sounds stupid, so please forgive my stumbling.

But two things stand out from the 90s, which I'm missing terribly today.

I miss Joel Spolsky's 12 steps to better code. What's today's equivalent? I haven't found it.

I miss strategy, strategic thinking. With all this agile stuff, I have no idea what anyone's talking about. Yes, UML & RUP were terrible. But at least we were trying to bring order to chaos. With agile, it feels like we forfeited, that we should just accept the chaos.

The challenge as I see it is that clients usually don't buy into this and they believe that developing software is like making sandwiches or turning buttons on and off when in fact is a discovery process of abstract things. They then demand a fixed budget, fixed time and fixed milestones for a fixed scope ("You are the expert, you should know how much time this takes."). To sell the project the salespeople from software companies deliver this, those who don't are considered incompetent. In this conditions agile methodologies are prone to fail but not for the methodology, but because most of the time those sold promises are done with incomplete information or tailored to be optimistic to sell the project. Metrics from agile methodologies show from the beginning that the project won't meet the expectations and then chaos ensues: they stop collaborating with the client to hide the reality of the project, "managers" pressure the team to do more in each iteration, they stop doing the disciplines altogether, iteration and daily reviews are treated as long meetings where people commiting errors are shamed and scolded and frictions start to emerge between all persons involved. This is where I believe the whole notion of "agile sucks" comes from.

What’s more waterfall than the sales team throwing a poorly baked contract over the wall to the dev team and telling them “make it work”?

This point is of general usefulness and applies to many areas of our lives, not just software development.

Exactly. This is why it's stupid when managers get all hyped up about <present software development trend/> and decide everything needs to be shaken up based on what they learned at a seminar or heard from some consultant/"coach".

For the most part, those 3 points seem like easily understandable ideas, and have been used in some form by other industries, such as manufacturing, for a long time before they were presented for software development. Replace "PM" or "Scrum Master" with "Foreman" and you can get the picture of what activities are involved with a worker managing other workers and the work that gets produced.

There is so much over-complication with the current system of methodologies and nebulous acronyms that accompany it[0]. There's SAFe, Agile, Scrum, Scrum of Scums (recursive scrum??), RAD, LeSS, etc. It gets confusing and messy very quickly. Instead of focusing on doing the work, it's easy for managers to get swept up in the semantics of these frameworks and end up hurting the work by over-thinking and over-applying the principles on the workers. Scrum methodologies are guidelines that can be tweaked to fit the situation, and don't have to implemented if they don't make sense. Instead of thinking "Does this make sense?", the manager/business side tends to think "Well, it worked for X, so it must work for me!". Scrum is an editable template that should help eliminate wasted effort and mindless work, and if that's not the case then it isn't being done right.

[0]https://en.wikipedia.org/wiki/Scrum_(software_development)

I've a SCRUM master certification and still, I follow only what's good for us.

It's totally fine for me to add a task in a current sprint. If you care about finishing the sprint, you just move a task of same points back in the backlog.

It's also totally fine to allow the technical lead to add his own tasks to the sprint. You can do 80/20 for instance (business/technical values). It's not going to kill your business and will make your developers happy (SCRUM is about people, not process, right?).

When the rest of the company saw our productivity, they moved every other team over to Scrum, hired a Scrum coach, etc... And it didn't work because they were extremely strict about the "process" and any changes to the process were like arguing against the bible. They even made our team begin doing daily standups every morning again.

I'm not certified or anything but the entire point of Scrum to me is that you frequently re-evaluate your situation and team and adapt to that. Everything else is just a good baseline to start with.

There are so many people who say, Agile or Scrum, respectively, do not need specs, the specs are just incremented with every sprint as the product itself, leading to software projects with undefined goals and overlong project time

There is definatly a lot of companies peddling hype around scrum but it has a very lean and strong core which is in no way pulling in that direction.

Source: Certified Scrum Master training. Three times. Still have no idea what these charlatans are talking about.

Jokes aside, good call linking that. Super short, super influential, super relevant to the conversation at hand.

One of the problems with the Scrum model is its used for every thing - just like they used to try and do every project with WF and IS9000/BS5750

Also I don't know what IS9000 or BS5750 are but they sound like they had amazing naming committees.

But and its a big but some times you need the rigor a properly done waterfall process provides Air traffic control for example.

You seriously don't understand the trade offs you make with "agile" vs waterfall? I have done both very successfully.

The issue I've had with every other project management approach is the illusion of structure. I'm not an expert on all or any approach. But my experience is that everyone wants me to be able to project, since absolute certainty, what my team can get done in 12, 24, 36 months. And I can't. No one can. It's all bull shiitake.

Scrum does not solve this problem. But it's honest about it.

It says: "We can't tell you what could be done in 2 years. But we can probably tell you fairly accurately what we can do in the next 2 weeks. And if you let us proceed in these honest 2 week intervals, we'll probably get way more done than if we all agreed to the collective insanity of the waterfall lie."

If you've done waterfall successfully, meaning you delivered a project on time, on budget, and on scope based on your original estimates for all three, then congratulations, you're a mythical beast.

I, for one, cannot do that. I've failed every time. Every team I've worked with has failed every time. I've never seen it work.

The idea is that if you have a well defined problem domain and you know exactly what the software is that you have to deliver, you don't need the short cycles, and all the meetings and pokering and whatnot would just be a waste of time. You know what you need to make, so you can just continue working.

I don't have a lot of experience in such areas though, and i can't help but wonder how often that's the case. If my problem domain was well defined, the problem would probably already be solved by someone and I'd use their code. Why reinvent? Am I being too simplistic?

Not necessarily. Look for example at the team that made the code for the central computers of the Space Shuttle. They knew exactly what their code had to do, how fast, etc et.

However the reason there is “bs” (by which I assume you mean the process/meetings/etc) is that most teams suck at being agile. Scrum provides a more rigid framework so that a team can develop agile skills and then, when ready, move beyond scrum.

Most people think Story Points or terminology like Standups comes from Scrum too - they don’t! They’re from XP.

Scrum is exceedingly simple and suffers from being over complicated and dismissed by those who don’t understand it,

The blog post above is no exception.

Despite working in a a Scrum team, I recognise literally nothing of the problems that are described. Our product owner works closely with both commercial and development groups to build the backlog. Pressure to build good technical solutions is reasonably balanced with commercial requirements. The development team is empowered to step in and request product changes when they think it's necessary. This happens because we are a team of skilled professionals who want to deliver working things at reasonable pace, and this applies to both commercial and development groups.

There is a weird underlying assumption in these kinds of articles that implicitly seems to assume that Scrum will somehow turn a bad team into a good one. I have no idea where that comes from. A team using Scrum still needs skilled people who know how to do their jobs effectively – and that includes the scrum master and product owner roles. Introducing it into a good team can then help to reduce the impact of common software development issues. It doesn't work for every company, team, or product – but that doesn't mean its without value.

I'd be more interested in articles that talked a little about better processes that we can use. I'm absolutely open-minded about the idea that there are other legitimate and effective ways to deliver software other than Scrum, because it's obviously the case. "These are the problems with Scrum and this is why they aren't a problem in <other process>" is more valuable than "If you have a bad team then Scrum is bad".

For example, it’s never made sense to me to have a powerful product owner while treating the development team as N interchangeable workers who may have “areas of focus” but have no roles, responsibilities, or organization among themselves, smearing out any sort of power or responsibility for technical quality like butter spread on toast. And yet, the fact that Scrum says to do it makes me wonder if that’s why everyone does it. Reading Scrum’s idea of a development team made me facepalm.

I don’t think the article is attacking a straw man that Scrum can turn a bad team into a good one. It may be pointing out that Scrum is regularly used to turn a good team into a bad one. Because everyone wants to be “agile,” even if they never use the word Scrum, they are implementing bad Scrum or cargo-cult Scrum by default.

What it really means is, "I'm not really responsible for making deadlines anymore."

As SM, it's my job to ensure that not only will the PO listen to concerns from the team, but the team feels empowered to provide that criticism. If neither of those happens, I've failed.

A bad PO will RUIN a team. I've seen them do just what the blog entry claims. I've also worked with POs who don't do any of the bad things claimed.

I think the main issue is that people are doing scrum wrong, applying traditional management to an agile process.

For example, Scrum says responsibility for operations and testing are to be shared responsibilities; you don’t have an ops team or a testing team, or titles, or recognized responsibility. You might have a de facto ops team, but those are just some individuals who have chosen to carry pagers right now. I’ve seen things like this play out and been puzzled by it. I didn’t actually realize until reading this article that people have been giving talks and writing books saying that “subteams” are bad, but it makes sense now.

The point being, not having officially recognized responsibilities is a form of disempowerment.

I feel like that's a bit of an overstatement. While scrum advocates for shared responsibility, it doesn't demand no-titles, no-roles, or no-passions.

It's all about ensuring the entire team possesses 'T' shaped skills. Depth and expertise in some areas, but still generally knowledgable across all areas. My test engineer doesn't write much code, but in a pinch, she can. My PO is fully capable of "running" the team in my absence, as is the test engineer. All my developers know the full DevOps pipeline and can contribute to it, though none of them are DevOps engineers by title or primary responsibility. And I'm an ex-developer-now-manager who can do a bit of everything, though these days I'd rather go to meetings and play office politics so my developers can do their thing.

Scrum recognizes no sub-teams in the Development Team, regardless of domains that need to be addressed like testing, architecture, operations, or business analysis

Maybe T-shaped people are required to make Scrum work, or vice versa? Like if you are running a restaurant and your company process says “there is no distinction between waiters and chefs,” then your waiters need to be decent cooks, and your chefs need to be personable and presentable. “Passion” can be a liability in this environment, for example being passionate about great food or great service. The lack of specialization doesn’t seem directly related to how to do great work, or be really happy at work.

I recently quit a job because I was the only one on the team "capable" (or that felt confident/comfortable) saying "No."

As a "devops" (loosely defined on our team) I felt my focus was uptime and error rates which sometimes is counter to new features (what managers want)...

Then it's no longer Scrum, correct?

This assumption starts with one of the earliest Scrum pioneers, Jeff Sutherland. He writes quite clearly that he doesn't care about an individual's performance, but rather a team's performance. He also talks at length about how poor performing teams can be turned into high performing ones by the implementation of Scrum.

You can agree or disagree with him on this (I'm on the fence), but it's not at all weird that this idea is found within Scrum. It was there from the get go. It's there by design.

That is my key beef with Scrum here. I entirely agree that done well Scrum can be effective. But how it is often done is not well at all (e.g. the two day Scrum master training course.)

I'm criticising Scrum as it's widely implemented and how that is poisoning the groundwater for all agile methodologies.

And definitely I'll get on to how other processes can be better.

Same. This process takes work, requires buy in by all levels of the company and needs to be fully understood including its error cases. It takes time and requires course correct and attention. Its not easy but done right it is effective.

That is one of the problems highlighted in the article though, too. What is the appropriate level of training for a scrum master and scrum team? Should we read the Scrum Guide? Are we reading the Scrum Guide? Do we actually know what Scrum is? Do we even know that we are doing Scrum?

I am working on a young, small team, and we started with an expertly trained scrum-master about 30 sprints ago. She was so well-trained that she was promoted out of our team. Since we started, we've had about 72% turnover.

I'm the only original member of the team who is still on full-time except for the product owner... and reading this article, I am coming to find out that our product owner is not doing the job of product owner (mostly our new project manager is.)

And

> apparently it has to involve something called “Jira”.

hits me right in the feels! We have been working with one project management tool (not Jira) since the beginning, and we found another tool that we like (the dev team likes) better.

Doesn't matter what the tools are. Bear with me. The first tool is used to communicate with our product owner(s) or stakeholders, where we get their acceptance of stories, and the second tool was meant to be a place where we developers can own tasks and "blog" about our progress without adding noise to the stakeholder channels. A task list that we control. Again, neither tool is Jira.

We did a trial, and we quickly found by consensus of opinion and observation that it really helped us a lot to add this tool to our process, all of the developers were buying in! Brought it to management to get it added in the budget... really a nominal sum, but then we were told we had too many tools already and made to abort the trial.

I was devastated, but I played along. It was the wrong decision! I started using slips of paper to fill the void, a place where development issues that are not part of our product backlog can live outside of the backlog and the sprint, and now in my perception at least, we're more disorganized than ever as a team.

I believe you both, that Scrum can work well if done right, and I'm glad it's working well for you, but ... I think at least some of us definitely need to hear these stories, many of them are our own stories, properly articulated so we can understand what's going wrong in our teams.

As for training for the SM (or PO, or anybody else), it's a continuous process. The CSM is a starting point, a basic level of understanding. At my office, many of us have gone on to take the Advanced CSM course. Most of us attend meet-ups, conferences, etc. It's not a once-and-done thing - nothing in a professional career ever is - if you aren't learning and improving every day, you're not doing your job.

"72%" is really misleading... from the original team of 2 developers and one PM/Scrum-Master, the Scrum-Master was promoted to a newly formed team, and the other developer is still with us, but in a mostly consulting-only capacity, as she is focused on other projects in our team.

Nobody is quitting out of frustration. (Not yet anyway!) We've since added two new developers from other groups within our organization, and a new PM.

We're talking about a greenfield project with some inherited/appropriated/shared code from other projects on our (slightly larger) team, a mostly new dev effort, and we're approaching year 3 of development.

For a dev team this small and a project of this scope and duration, I feel like this is almost an appropriate level of turnover. We have to strike a balance between spreading the domain knowledge around, and keeping the expertise near the "most important project," which is obviously the particular one that we're working on. (/s) The big problem is training, or lack of... the SM that left our team was practically the only one of us with any formal training in terms of either development skills or Project Management. The rest of us have learned basically everything "on the job."

We've been exposed to some training opportunities for "business analyst" skills development, which is our organization's take on a little bit like "Scrum-lite" but I am not sure that's for all of us. (I'm working in higher-ed so there are a lot of training opportunities in-house. Just not sure they are the right ones. Most teams in our building just aren't doing product development, so we are special snowflakes.) I've taken some of those BA classes. We really need more technical training. We probably need a larger team too, but hiring is very difficult for us, and in reality I think we're going to have to cope with more of our team members being poached by other departments as we become more successful.

Some members of the team have participated in bootcamp-style training courses for our chosen framework. In every case, it was years ago, some time before I arrived on the scene. I have not had any of this, just some prior on-the-job experience and also have read a book or two to pull myself up by bootstraps! We're all learning on the job, and we have a conference budget so we're getting exposed to plenty of new ideas each year, but that's nothing like a formal training day where we all hammer a particular skill until we all get it, or even a sprint that's dedicated for training.

Second major problem is that we also usually don't have sprints or stories devoted to addressing technical debt. No refactor stories, basically ever. Basically all story cards must be customer-facing and represent some new, testable functionality for some functional user of the product. This part of the article also struck me as poignant:

> The items promoted in the backlog, combined with the schedule pressure set by the product owner and the business mean that the Development Team often are told how to create releases: quickly and exactly how we said. The lack of technical craft or development control means each sprint is launched as soon as possible rather than as soon as prudent, refactoring isn't done, and technical debt builds up, eventually choking the product.

When every story is a feature story or bugfix, this is what you get. When the team says "we need to learn this skill first, so we don't add unnecessary technical debt in the next sprint" and are overridden by the PM's boss and told to stick to the planned release timeline at all costs, we lose.

> As mentioned above, the lack of titles or specific responsibilities in the development team means no-one is empowered to advocate for development priorities, and also contradicts the typical implementation where there certainly are titles and hierarchies within teams

... exactly this. We're told that the development team as a whole is accountable for progress, but when everyone is responsible, nobody is. You can't call someone a technical lead and then confine their decision-making to minutia within the already-planned sprint work. That's not really how I feel about my team, but I feel like sometimes circumstances put the pressure in that direction.

This is getting too personal, but suffice it to say I really agree with this article about a lot of points.

Yeah, lack of buy-in from senior management is probably the biggest single problem with scrum, any other agile framework, and probably even non-agile frameworks.

Interesting that you're in higher ed. I am as well, thought it sounds like you're at a college/university and I'm at a software company selling into that market.

Just goes to show that wherever you look, you're going to find problems.

What is a manager but someone who has control over the controlled? I'm surprised with so many ppl nodding their head at this list of complaints about bad software processes at bad companies, masquerading as analysis. If you think it takes more than 2 days to learn _how_ to be a Scrum master, you don't understand the role. It's not complicated, difficult, and time-consuming enough to require a singular employee...unless your process is already screwed up. Good luck finding whatever magic bullet you think will save you that isn't Scrum-based teams.

Another way of putting it is that there are many good ways of doing project management, as many good ways as their are unique (team, circumstance) pairs.

Most people have an intuitive understanding of why some very low-overhead, low-formalism, "just get stuff done and be reasonable" workflow has worked really well in some specific (team, circumstance) combo situation.

If you want them to override their experiences with how those ad hoc, customized management processes worked well, and instead believe that a company-wide, one-size-fits-all approach that removes tons of workflow autonomy that could just be left up to separate teams to customize for themselves, then you absolutely have to have extremely convincing answers to address widespread bad experiences.

Historically, there are three important sides, and roles, for each project. Product management - takes care what the customer wants to have build. Project management - takes care of what is delivered is on schedule and that there is enough material/personnel to build it. Architect/engineering lead - takes care of whether the thing is technically feasible, what technologies are used and what technical trade-offs are made.

And there was a vast literature and discussion about how these three sides affect the result of the project, and how to solve these problems.

Unfortunately, Scrum, renaming everything, acts as if the history of project management doesn't exist. And if you forget history, how are you supposed to learn from it? That makes it easy to sell snake-oil. I frequently hear, for instance, that scrum master is not a project manager, despite him having some responsibilities in this area.

And as the blog post also expounds, there is no recognition for these three different sides of each project in Scrum (ironically, there is some recognition of that in Scrum derivatives such as Scaled Agile Framework, for instance, modulo nonsensical renaming of the roles).

I don't like renaming things for no reason, but if you want a term to have a precise meaning and not carry any baggage from previous similar-but-subtly-different usages, it's probably best to coin a novel term.

So if Scrum (or any other methodology, for that matter) wants to define a new role (to what end?), they should explain, how they are related to these concerns? What tasks that were traditionally done by the triad are supposed to be done by the new role, and why? And moreover, how are the concerns from the triad dealt with in Scrum?

That's what I miss from Scrum. There is no connection to what was before, aside from using waterfall as a straw man (it's not a methodology to begin with). It's an ideology, its own world.

The result of Scrum being an ideology is, somebody who becomes scrum master in Scrum, and effectively doing project management work (in the better case), by default, has no idea that there is a field called "project management", and that he should study that field. So they don't know, for instance, the classic like Fred Brooks.

That's one way to break down the set of tasks/concerns you've identified, sure - but only one. I don't think that the industry as a whole accepts your categorisation into three, or even your paradigm for which tasks do and don't need addressing in this way.

You're coming across as far more ideological than the thing you're attacking, assuming that your particular breakdown must be the canonical one when it simply doesn't have a level of industry acceptance that would justify that.

And actually, when I started programming professionally, I didn't think that management was necessary, and that we could self-organize and all that. But as I changed projects and bosses, it turned out, I just had a good manager (and it was a waterfall-like method, by the way).

So I think the idea of self-organization is intuitively appealing to many people (especially young without any experience), and if you have a person in the team who can naturally take each of these roles, then I think it can work (that is - sometimes). But today, I don't think it will work in general, and for that reason.

Now I recognize that, for instance, you need somebody well-organized on the team who can make sure things do not fall under the table. Or you need somebody who is process-oriented, interested in improving process. Then self-organization can work.

If you don't have people with specific traits like that, or they don't emerge as natural leaders (gain respect) in their specific areas of interest, then you need to appoint somebody to have that role. Then it makes sense to have explicit project manager, or product manager, or architect, who focus on certain tasks and concerns.

I suspect the fact that sometimes you can do without having these people designated as such can account for some anecdotal success of Scrum (or any other method - I personally had the best experience with waterfall-like approach).

Not really, and I don't think the industry as a whole does either. Everyone has their own set of tasks and concerns; I don't think there's any consensus that says that all the important ones are on your list, or that everything on your list is important.

You probably have a good idea of what a product manager should be doing but someone else in a different company might have a different opinion. Introducing a new term eliminates this source of confusion.

Admittedly, all the jargon you encounter when learning about Scrum can make you feel like you're joining a cult. However, there is a reason behind introducing it.

Mine is "marketing" (biz case, price, feature set, aka what), "engineering" (cost, schedule, aka how), "QA" (owns the release).

Loosely based on the Deming-style "quality circles" used at Ford and else where.

This is the root of the problem in my opinion.

I am not interested in defending "Scrum", I don't like what I know about it, in the few experiments I've been involved in with it (not by choice), I agree it was disempowering to developers, trying to treat developers like commodities.

However, I am a fan of trying to do things agilely (iteratively, figuring out what to do next in relatively short chunks without trying to plan out the next year+).

In the experiences I've had where this _worked_ the product owner was _intimately_ involved with the development team, with lots of communication in both directions, with the development team's info and feedback effecting how the product owner prioritized and determined (and changed, agilely) acceptance criteria.

The product owner had to embrace/accept that this would be a significant time and energy commitment to them, they could not be looking to minimize their time here, and had to accept that "with great power comes great responsibility" -- that they needed feedback from developers to make these decisions properly.

On the flip side, through these good experiences, I also learned that a good product owner is _so important_ -- as a technical decision-maker I don't _want_ to be responsible for determining product priorities or acceptance criteria. I want my feedback to be taken into account, but having someone else (who is good at it) be _responsible_ for it let's me focus on applying technical excellence to achieve the goals set by the PO, and takes _so_ much pressure and anxiety off of me. Especially when I can trust them to know what they are doing it and do it well (just like they can trust me to execute well).

I _do_ think the development team needs a "technical lead" in addition to product owner, not just an amorphous bunch of people "self-organizing".

Backlog grooming is a thing [1]. In fact, I'd say after the Increment, it might be the most important thing. If developers and all the owners are not continually reviewing the items in the backlog together, clarifying them and making sure everybody understands what is being asked for and what the priorities are, breaking down stories that are too big, making sure that, yes, some tech debt gets in there, then all you've accomplished is a kind of mini-waterfall where stories get shoved to the top of the backlog by whoever comes last and then developers hammer away at them frantically at the beginning of each sprint.

> I _do_ think the development team needs a "technical lead" in addition to product owner, not just an amorphous bunch of people "self-organizing".

Scrum consistently fails because everybody thinks in terms of the "who" and the "how." Scrum books are all about the process, the daily meetings, the roles and the responsibility. This a symptom of a much larger issue here, the "technological paradigm" that pervades nearly all corporations. There's an unfailing obsession on organization charts, department boundaries, and processes.

It's all kinda silly.

As somebody who's managed too many scrum teams to remember here's the secret: don't focus on the "who" or the "how", focus on the "what." The what is all that matters. In Scrum there are certain key artifacts: stories, the backlog, the increment, blockers, and retrospectives. Relentless focus on the quality of these artifacts as documents and you will excel at Scrum. But you have to really care about the quality of these documents. Everybody, not just the scrum master, has to keep iterating over these document/artifacts until they are awesome. This means continually refining stories until people know them inside and out. Continually polishing the backlog until it shines. Making sure everybody understands exactly what will be delivered at the end of a sprint, the increment, and when it's delivered having people write feedback and rate the increment. It means investigating and documenting every single time a developer is blocked for more than 15 minutes and then making sure it never happens again.

I've worked with remote/distributed scrum teams and what I've found is that you can even throw out all the ceremony -- the daily stand-ups, the endless sprint review meetings, the endless retrospectives. When the team is distributed between London, Shanghai and Sydney and most of the developers work from home at oddish hours such meetings are impossible. But by focusing on the documents even such a distributed team can fly. So, yeah, at the end of the day this all boils down to effective communication and writing things down. Most knowledge games do.

[1] https://www.agilealliance.org/glossary/backlog-grooming/

It is not only more efficient, but better for morale to have an agreed upon person upon whom the buck will stop. Sometimes when things are "everyones" responsibility they end up actually having nobody take responsibility for them. Such a person with ultimate responsibility and authority for technical decisions should be both competent and not an asshole, they should be in constant dialog and taking feedback from everyone else, and paying real attention to their concerns about technical matters.

But for the same reason you need a Product Owner (instead of just the amorphous group of all stakeholders) to ultimately take responsibility for decisions about priorities and acceptance criteria, you need a technical lead to ultimately take responsibility for technical decisions.

I think that "Scrum" doesn't have this is just part of it's attempt to commoditize developers (something I think is characteristic of "scrum" but not part "agile" generally or at it's best). "Scrum" sometimes seems as if it's designers thought there _are_ no technical decisions to be made, just a bunch of widget-ized, swappable developers churning out code to meet the requirements set by the PO. But, in fact, there are technical decisions to be made, that will effect other people's work and the product as a whole, in the short- and long-terms. Regularly.

Agile implies getting things done and getting things to market. If scrum helps you do that great. If not, ditch it. In my experience, Kanban is a step up on the evolutionary ladder. Both have issues with not prioritizing essential activities related to research, refactoring, etc. that you need to guard against. There's a difference between agile and firefighting.

One worry with scrum is the roles of product manager and scrum master. In my experience these things end up being formal job titles in bigger organizations. This is bad because they are typically on the very low end of the scale. That means you end up with the least experienced people filling these roles and a lot of corporate politics. I've seen more than a few organizations that were hiring accordingly.

Granted, we're very selective in our hiring process, but it means things just work without much need for fancy management practices.

If any dev managers are reading this, I can say, this approach works, and you should give it a try.

However, it was the most happy / least stressed I've ever been in my life. I still get beers almost weekly with over half that team. This "strategy" really built comradery because we were always talking and discussing rather than leaving comments on some Jira ticket (which we did for async or when necessary).

Seriously.... give it a shot if you can

Product owners push customer value. Of course. You should too. Code quality and refactors have business value. Reduced maintenance cost, fewer bugs, faster future dev. If you can't explain that to your product owner then maybe it's not worth doing. I think your big missing piece is the collective ownership aspect. Product owner gets to set priorities, but it should be a collective discussion. Every dev should be allowed to express their opinion and it's the job of the product manager to referee.

The real issue the article brings up is that Scrum has an individual who takes ownership and responsibility for the product backlog, it has an individual who takes ownership and responsibility for the project management, but it has no individual who takes ownership and responsibility for the non visible technical aspects of the product.

In an ideal world, this responsibility would be taken up by the scrum master, who would have technical expertise, but as the article points out, (a) scrum never states this, and in fact, hints in the opposite direction, and (b) this is rarely the case in the real world. The scrum masters are usually as non-technical as you can get.

The team owns it all - the process, the product, the technical artifacts, the code quality. If the team is looking to the PO or the Scrum Master to take responsibility for things, then they are going to fail.

The product owner is there to help the team understand what value they can deliver. The scrum master is there to help the team optimize the process. The team hopefully has developers on it who are there to help the team make well engineered pieces of code Maybe there are designers to to help the team make great user experiences. The team has specialists, but the team works collectively to deliver results.

If your scrum team has a bunch of engineers on it wondering how to get their voice heard over the product owner, you don’t have a scrum team building your software, you have a PO acting as a tech lead running a waterfall project.

The development team!

Well, because more often than not, the answer is management, and there's really no way to fix bad management.

A proper Scrum definition of done requires full test automation for every user story. You can't merge code to the release branch or accept the user story until all the automated tests (unit, functional, integration, system, performance, etc) are working.

The only two types of testing you can't really automate are exploratory testing and usability (UX). Exploratory testing should be done concurrent with product coding, and then the tests that seem useful can be scripted as a permanent part of the test automation suite. Usability testing should be done either on a separate branch, or on the release branch with a feature toggle. It's essentially part of requirements analysis and not part of qualifying a particular build for release to customers.

While Scrum doesn't call for any particular testing approach, in practice it's almost impossible to do Scrum effectively on a large product with multiple agile teams unless you have a very high level of automation. Otherwise development just bogs down and you end up with nonsense like "hardening sprints" (basically waterfall in disguise).

Ya since that is essentially a deal breaker, I wonder why all the scrum consultants fail to mention that.

Automated full regression is very expensive to develop and maintain. I think a lot of people who implement scrum don't realize that they are sacrificing significant levels of testing when they implement it without a testing automation team.

To me, that's just another reason to ditch scrum.

This is absolutely not a reason to ditch Scrum. In fact Scrum is conducive to good quality practices.

Source: Former QA/Test manager.

FWIW, I'm desperately trying to bring "TEST in PROD" style continuous delivery whozeewhatsit to my current team.

When in Rome, join the third millennium, old dog new tricks.

I still can't imagine I can get back to the 90s (quality wise). But if there's a way, I'm gonna find it.

No True Scotsman and all that, but it really was different, back in the day.

I wrote a whole book about "Agile Anti-Patterns" [3]. Most of the book's content is about things that many companies get wrong when they start with agile or lean software development. Because it is very easy to get those things wrong.

Yes, those problems are extremely common. Not only with Scrum - Organizations also face those problems when they try Kanban or SAFe or whatever. Because, change in a traditional organization is hard. And to really implement Scrum, you'd probably need to change more about the organization than the org was willing to change.

But those problems are not Scrum's fault. If the team has no power, it is also not truly self-organized. And the Scrum Master is not doing their job.

You could start to improve by creating awareness about the problems. Blaming Scrum may or may not be a good idea to do that - Because, some people in your org may be invested in Scrum. Some mentoring or coaching for your Scrum Masters, Product Owners and developers and managers might be a better start.

[1] https://www.davidtanzer.net/david%27s%20blog/2014/03/26/no-t...

[2] https://ronjeffries.com/articles/018-01ff/hills-to-die-on/

[3] https://www.quickglance.at/agile_antipatterns.html

Bingo. That is the key point to me. Scrum, and other agile-family methodologies, are great when fully implemented. But what usually happens is that the higher-ups refuse to relinquish a bit of control, and stick tightly to their traditional command-and-control approach, and the dev teams are forced to do something that looks-kinda-like-scrum (or looks-kinda-like-XP or looks-kinda-like-AUP, etc.) while operating in a structure that isn't really compatible with the principles of Scrum, XP, AUP, etc.

Basically, we're forced to try and fit a round peg into a square hole because people higher up the org-chart either don't really understand agile-family methodologies or are actively opposed to truly implementing them.

And there are other organizations who can make any agile method work for them.

So, it's not really Scrum's fault when it fails (at least not always). And it's not really (or not only) Scrum's achievement when it succeeds.

As a friend of mine, Samir Talwar, once said: "To be good at software development, you need an organization that's optimized for software development. Most organizations are optimized for something else."

Anyway, if you implement a process, but ignore even the very short (22 pages) guide that summarizes the process, but instead cherry-pick what you think will work well in your org, then don't blame the process when it does not work.

As I wrote in my book: "But changing Scrum so it works within your company will not make you more agile – It will make Scrum less agile!"

Firstly, it gives you a set of tools to use -- sprints, TDD, product owners, and so on -- that should work well together. To use them well, I think you both need an understanding of how each of those tools supports some idea or principle that the methodology promotes, and whether or not you think that principle is important for your team.

Secondly, a methodology gives you a starting point of how to run a development team. It's easier, although not necessarily better, to start from a pre-existing cohesive set of practices than to build a process from scratch. Especially if a team isn't experienced with many of the ideas, this is not an unreasonable place to start.

I think the key is:

1) knowing what principles your team thinks is important (low defect rate? frequency of releases? developer happiness? empowerment?). This is often hard to articulate. As time goes by, you might discover principles that were previously implicit, or you might find that the principles that are important to you change.

2) knowing how each part of your process maps back to one or more of those principles

3) having a mechanism that allows you to tweak your process over time (for instance, retrospectives), whether that's adding, changing or removing parts of your process

Scrum can be a sensible starting point so long as you're willing to introspect and consider which bits are and aren't working for you, and you're empowered to do something about it.

I've seen this claim multiple times and it's never made sense to me. I do TDD on side projects where there's not even a hint of "agile."

If you think using TDD improves the quality of your code, then it's also important for maintaining high code quality, since that enables some of the other agile practices, such as frequent releases.

I don't worry too much about whether a particular practice is agile or not. If you find TDD useful, then great! It's often included in conversations about agile development, but that doesn't mean you have to be doing agile development in order to use it.

Agile warns against building stuff you don't need. TDD means that test provide the justification for feature work. If your feature work isn't making a failing test past, you don't need that feature work. Obviously this assumes that your tests are justified (ideally there is a stack of tests leading all the way to the end user experiment), but that's much easier to get right than justifying untested feature work.

That said, an entire industry has emerged to sell this methodology, and it is supporting a vast array of jobs: everything from those selling the certifications, to the software used to track developers. There are too many vested interests now, and no company wants to admit that it burned through thousands of dollars, with dubious results, to "train" managers and subscribe to the usual software bundles.

I expect the comments here will also have the usual comments about how Scrum is great, and everyone who is critical of it is just doing it wrong.

In general scrum will work best when 1. The product owner is from technical background with enormous experience under his belt in product development/architecture. The Product owner does not work alone in silos to create backlog but interacts with development team regularly to refine the backlog. In our case, the product owner is someone with 20+ years of experience who has worked in roles like developer/architect.

2. The scrum master is also highly technical person who has been working on the product for several years. In our team, the same person wears different hats as per the need like scrum master/technical leader/individual contributor.

It may not be exactly as per scrum guidelines but worked fine so far for us. Ultimately we need to look for good practices in each methodology and bend it as per our needs.

There are strong philosophies in Scrum and Agile that should be kept as guidelines. The key is being agile (lower case 'a') so you can adapt to changing priorities. Continuous Integration/Continuous Delivery go a long way towards empowering developers.

Two ways to tackle technical debt in projects:

1. Tech debt sprint. If scrum master and product owner can't get onboard then they need to realize somewhere in the near future, a feature they want will either break things or take forever.

2. Break up tech debt into small efforts someone can do in short time boxes. Even do feature flags or something to make slower progress towards a goal rather than tech debt sitting in a backlog for months.

An example of a guideline is estimation with planning poker. We completely ditched estimation activities. The estimates were usually off and not good predictors of completion. The team has a cadence of ticket completion and the 'sizes' of the tickets vary some but you don't need to waste time estimating. Developers (humans) are horrible at estimating. Having a good PM/PO set expectations with the business helps.

We sometimes find that engineers have entirely different concepts of how a particular feature or change needs to be implemented. Estimation – planning poker in particular – is a great way to reveal that and start exploring what the reasons are.

Maybe it works better for small projects/teams, but in my experience it's a massive waste of time.

Disagree with tackling it as a separate activity at all - that's a way to get useless churn as you move towards an architecture you think is better rather than one that meets your actual needs. Do continuous improvement instead: improve the part of the codebase you're working on in a way that also contributes to your immediate, customer-facing task. "Make change easy, then make the easy change".

> An example of a guideline is estimation with planning poker. We completely ditched estimation activities. The estimates were usually off and not good predictors of completion. The team has a cadence of ticket completion and the 'sizes' of the tickets vary some but you don't need to waste time estimating.

You need to give enough of an estimate to let the business prioritise, and to realise when a task is too big for a single ticket. You shouldn't spend a lot of time on estimation (frankly I struggle to imagine how your planning poker ended up taking long enough to matter) except when developers radically disagree, as that's a warning flag for a story that isn't clearly scoped.

> Developers (humans) are horrible at estimating. Having a good PM/PO set expectations with the business helps.

You don't want to rely on one person getting it right. Developers, like most workers, understand their own job better than anyone else; they're the people who should be giving their estimates.

Larger refactorings that impact multiple modules have to be treated as separate user stories and prioritized based on business value just like everything else.

That is a part of the problem. Scrum is very far from Agile. Scrum introduces processes and tools while the Agile Manifesto [1] clearly states:

> Individuals and interactions over processes and tools

That said, Scrum isn't entirely bad. Its just not what many people think it is. As a tool to change the culture of a company that has been shaped by classical project management methods for decades it is very valuable. But the process shouldn't stop there as Scrum isn't the end, its the start.

So companies which managed to implement Scrum should try to move on to more Agile practices to transform their culture over time.

[1]: http://agilemanifesto.org