Why Agile Is Losing Steam

As I am sure you are well aware, companies have been purging themselves of what they now perceive to be niche roles not directly tied to the bottom line. This includes Product Owners, Scrum Masters, Agile Coaches, Internal Consultants, Project Managers, Program Managers, as well as UX, Marketing, Comms, and HR roles. The Agile roles, I will here argue, are in an especially precarious position. Agile itself has long been caught between a rock and a hard place. Here we will discuss what I consider to be its two biggest issues.

1) Agile’s Political Naïveté

Agile teams tend to commit what I have called the error of “tailism”, opposite the top-down executive error of “commandism”. To justify this, they ritualistically overapply the overused Cynefin, as they wave their hands in the air while proclaiming everything “complex”. Thus, upfront planning and contextual research are too often viewed as unnecessarily wasteful.

One instead, it is said, just needs to act(!), to start by poking the system to see what happens and then “iteratively” respond. Sounds reasonable on its face, doesn’t it? Except, there is a huge problem here baked into the terms we routinely fail to operationally define. First, the Agile Manifesto coauthors do not actually agree on what “iteration” even means. Second, and more to the point, what Agile and UX mean by “iteration” tends to be two completely different things.

There is an important analogy here from the world of chess, where if you burn moves just to see what the other person will do, then you are losing “tempo” and strategic advantage. If you make too many ill-advised moves, then you will quickly find yourself in zugzwang (pronounced “zook-zvahng”). This is a position where, no matter what you do next, you can only make things worse for the outcome. Here, however, it is because user patience is a finite resource.

Here you have burned too many moves, users don’t have time for your nonsense and are going to move on no matter what you do next. As a result, you have blown any real chance you had to make an impact. The lesson of zugzwang, applied here, is that you cannot just “iterate” without acknowledging that there are only so many degrees of freedom at play, and eventually the only moves left to you will be bad ones. A drunkard’s walk is no replacement for deep research.

It is perhaps no surprise then that Agile often falls into the trap of “anchoring and adjustment”. The initial proposed solution—whatever it is—typically establishes a frame all future “iterating” remains boxed within. There are multiple reasons for this. Agilists tend to argue that code is easier to change than physical buildings. This ignores, again, the very real political dimension of work. As Jeffrey Pfeffer notes, once a thing is done it then becomes far more difficult to do something else, if only because: 1) the costs and benefits of what was done are now real and not hypothetical; and 2) redoing work, even if it is “code”, requires a certain amount of political will that is often lacking.

As UX desperately tried to counsel Agile (and failed): 1) The name of the game is achieving outcomes that create bidirectional value for users and the business; 2) outcomes are achieved by making smart design decisions; 3) impactful design decisions are typically made well above the level of product teams; and 4) real design work and having an impact are therefore inherently political in nature. Agile’s response, sadly, was to scoff at this, double down on the error of tailism, and insist that the solution is always “more software”.

And here Agile shot itself in the foot. Like the ostrich with its head in the proverbial sand, this left a maddening strategy gap that UXers frantically tried to fill by becoming PMs en masse. As Agile and PM roles fell out of favor, so too did UX work, which always struggled to position itself high enough in the hierarchy to be optimally effective.

Agile also laughed off UX’s warnings about how it defined “value”, or, perhaps more accurately, how it utterly failed to do so. As any UXer worth their salt can tell you, asking users what is of “value” does not predict whether or how they will then use something if it is actually built. Agile thus always fell into the same trap that focus groups do: Users’ stated preferences are simply not predictive of their own future behavior.

If UX research suggests something would be the best solution, but users say they would prefer something else, the standard Agile team would focus on quickly building whatever users tell them is of “value”, ignoring that the optimal solution will often not be something the users are requesting or advocating for. Further, getting to the optimal solution often requires quite a bit of influence and political maneuvering, which Agile teams, in all their stubborn libertarianism, were never very well equipped to do or even take seriously.

And so of course Agile teams tend to be stuck in small frames—they typically never focus on the larger context in the first place. The question of whether more software is really needed to advance optimal outcomes is never even raised. And forget about the strategic research needed to surface what such higher-order optimal outcomes might even be. That’s just not gonna happen!

In this way, Agile has always seemed more similar to Build-Measure-Learn than actual PDCA. But wait, it gets worse! As Pavel Samsonov has noted, Agile typically isn’t real Build-Measure-Learn either. In practice, the “Learn” part is all too often silent, and so it could more accurately be described as “Build-Measure-Build” (which is, of course, customarily sung to the tune of “Burn Baby Burn”).

As I have long argued, this ignores the entire field of Decision Quality, which stresses that the quality of any decision is always independent of its actual outcome. Let that sink in. Product work is a lot like placing bets. If it was a stupid bet, then it’s still a stupid bet even if it ends up paying off. Ignoring such thinking results in overlooking all that can be done to improve the quality and risk profile of decisions before they are made, and this always comes with a large opportunity cost.

As Maxwell Maltz observed, what do people tend to do when placing a bet? Take roulette. First, people make their bet and then, after the wheel is already spinning, they start to sweat the outcome. In other words, they get nervous only after nothing can be done about it. They place the bet and focus on the “aleatory” uncertainty of what comes next. This is like magical thinking. Yes, you can learn from the outcome of a decision and then improve the quality of later ones, but the time to optimize the quality of the decision at hand is before it is made.

Now, this stupid state of affairs isn’t always Agile’s fault, and that brings us to Problem No. 2.

2) What Orgs Really Want

The reality, as noted above, is that most Agile teams do not spend a lot of time revisiting work already done. They just don’t. In practice, it’s usually “Off to the next thing!” In fact, when work is revisited (and therefore iterated the way it should be), in my experience it is often because of people taking the initiative to ask for forgiveness later instead of permission up front. This is in part due to the typical bait-and-switch pulled by large organizations. And this should have come as no surprise.

You see, organizations do not really want increased agility—they only say they do. What they want is Sutherland’s mouthwatering promise of “twice the work in half the time”. “Agile” was originally meant to improve the work life of people coding software. It was then quickly realized, however, that agility could greatly benefit the entire organization…but there’s a catch. Real agility, it turns out, would be an afront to consolidated power. Increasing org agility requires a meaningful redistribution of decision authority. It requires, as Andy Grove noted, that decision authority be pushed down to the lowest level responsibly possible.

Well, guess what? That’s not what most executives want. (Duh.) Most executives want to consolidate power, preserve their fiefdoms, and leverage their current positions to land better ones. That’s a very different game with very different rules. And this points to what I consider to be Rule No. 1 of consulting: Clients do not want you to maximize value as much as they want you to make them look successful—and those are often two completely different things. Consider, again, what any Decision Quality expert typically runs up against:

DQ Expert: “I’m here to help improve the quality of your decisions over time.”

Executive Client: “OK. What I can’t have is you doing anything that leads people to question the decisions I make. So, that’s probably not gonna work.”

Agile, in this perhaps unexpected way, shares interesting similarities with subversive movements. More to the point, what often happens to such movements also explains what happened to Agile. Consider Christianity, which, contrary to current American belief, began as an anti-empire, anti-class movement. To paraphrase theologian Tad DeLay, until the Romans appropriated it into the service of its imperial ideology, Christianity was a very effective plague on the Roman Empire and its goals.

And so it is here. Of course the traditional power hierarchy is going to appropriate Agile and morph it into something that serves its goals. Of course Agile, at the end of the day, is only going to be about “velocity”, because what the org really wants is to get what it already wants done faster and with less. Far from instantiating actual decision agility, what the org wants is for teams to execute to its upfront plans with fewer resources. Agile did not give large orgs what they want, and so now their attention has shifted to the desperate hopes of AI.