Estimating Cost of Delay
In the last post we looked at what cost of delay is and why it is so important. In this post we’ll look at how to facilitate conversations around estimating it. If you’re curious about this concept but don’t know how to apply it, or if you’re a Product Owner looking to up your prioritization game, then this post is for you. If you’re not already familiar with the concept, then you’ll definitely want to read the last post before moving onto this one.
Confidence Intervals and Calibration
Typically, the costs of delay for different options will be very far apart. When there is not a sufficient spread in cost of delay, it might be because you need to find more expensive problems to solve. After all, if what you are doing does not have a large delay cost, then it does not create much value either, so…why are you doing it?
When it does seem the costs of delay would differ greatly, then estimating ranges, even very wide ones, is all you need to do to estimate the value of options. The goal here is to estimate ranges that are simply good enough to allow for a rank ordering of options based on value, not cost. You can then gauge your confidence and tweak your ranges with some calibration exercises.
Take a look at the questions below. You probably don’t know the answers to them. The point is that you don’t need to know the answers in order to state your guess as a 90% confidence interval (CI). This is your range of values you feel 90% confident will contain the correct answer. Go ahead and jot down a low and high value for each. No Googling!
OK. Look at the top question. If you did the exercise, then you wrote down two years, one for the lower and one for the upper limit. These form your CI. Now let’s make a deal. You have a chance to win $1K, but you can only make one of the following bets. Would you rather bet your CI contains the right answer…or would you rather spin the arrow below? With the spinner you have a 90% chance of winning $1K and a 10% chance of winning nothing. Think about which option you would really prefer.
Here’s the trick. If you’re feeling you’d rather spin the arrow, then you are less than 90% confident in your CI. If you would rather bet on your CI, then you are more than 90% confident. Adjust your limits until you’re indifferent between betting on your CI and spinning the arrow. That’s your 90% CI. (This exercise, by the way, is from Douglas Hubbard’s How to Measure Anything.)
This is helping you calibrate. Someone is “calibrated” when they’re right 90% of the time they say they’re 90% sure. Improving calibration makes you a better estimator. Most of us are overconfident. When we give our 90% CI the right answer only falls within it half the time. This is partially due to the bias of anchoring and adjustment. We guess the actual value and then fail to veer from it much in generating our upper and lower limits. As a result, our CI is commonly too narrow.
Now, with cost of delay you’re not answering trivia questions about the lifespans of whales, but you’re still estimating ranges for things you don’t know the actual value of. As you’ll see in the rest of the post, this comes down to facilitating conversations about choices, framed in variety of ways, and aimed at surfacing the assumptions being made. These are the very assumptions that otherwise don’t usually get exposed and discussed.
Facilitated Conversations
Gather a group of people. Five should be fine. As John Cutler puts it, the aim is to help surface assumptions about value and risk, build shared understanding, and stay focused on what you want to achieve. It can help to remind the group of two of Don Reinertsen’s main points:
Intuition is not a good substitute for quantitative estimates.
Quantitative estimates do not need to be accurate to be useful.
Say you have three options, X1, X2, and X3. They each have a respective cost of delay, Y1, Y2, and Y3. You don’t know what the values for Y are, but as Douglas Hubbard would point out, just because you don’t know anything about Y doesn’t mean you don’t know anything about X. After all, if you really knew nothing about the Xs, then you should not be considering them for prioritization in the first place!
The process here is basically to quickly gather what data you can, gather some people with some knowledge about the options being considered (the Xs), and then run the group through a facilitated conversation around choices. The aim is to do “just enough” research. If possible, ask Finance what they would pay to remove a month of cycle time from a project. (Presto! That’s Finance’s approved figure for cost of delay per month!) As Reinertsen stresses, such microeconomics are far more important than go/no-go decisions.
So, gather some people, either your team, or stakeholders, or SMEs, really any individuals with some knowledge of the options you’re considering, and start utilizing some of the techniques and exercises below.
Clean Language: This is a popular technique that came from modeling the work of psychotherapist David Grove. It is now widely used in coaching and research interviewing (and even in Scrum team meetings). Here we’ll just touch on the concept briefly. Below is an adapted set of “clean questions”. You can think of them as questions that explore another person’s mental model and assumptions without imposing your own.
In facilitating the conversation, listen for the metaphors people use in discussing ideas. Use clean questions to unpack and explore them. This is a good way to surface assumptions and desired outcomes. Capture the outcomes that come up. You can then shift to discuss the possible costs of delay for the outcomes captured, asking things like, “And if X was achieved, how would things be different?” “Then what happens?” “And what else?” This encourages people to explore how various options might change the environment in value-adding ways, which plays straight into the concept of cost of delay.
Value Buckets: As the group discusses options and captures outcomes, get them to focus on which value bucket(s) they would primarily fall in. In other words, how does X create value? Again, if you really can’t say, then why would you prioritize it in the first place? (Image adapted from Joshua Arnold.)
Divide the Dollar: List the options or outcomes captured (Xs) in a lefthand column. Give everyone the same amount of Monopoly (or fake) money and have them use it to “buy” the Xs. As they do so, have them jot down their reasons, one per sticky note. Tally how much the group spent on each X in the middle column. What are the winning items? Place all their stickies in the righthand column. The real value here comes in having people flesh out this column. See if there are any clusters. What are the main reasons? What assumptions are being made? What happens when you challenge these assumptions?
Time Travel: Use the questions below to facilitate the group through thinking about their assumptions about outcomes. Make sure to focus on “Why?” (Most of these questions are adapted from Joshua Arnold.)
If you could have X today, what would that be worth to you (in actual dollars)? Why wouldn’t you pay more? (If you said “$0”, then why are you considering X at all?) How would this create value? What else? And how might that create value?
If you could have had X a month ago, what would that be worth to you today? Why? How might that have created value? Which value bucket does it fall in? What are some reasons why that figure might be higher than you think?
If you couldn’t get X until a month from now, how much might that cost the business? In other words, how much value would be lost by delaying X by four weeks? What are some reasons why that figure might be higher?
If you never get X, will you regret it? Why? Will you regret it six months from now? Why? Will you regret it in a year? Two years? Why? Why not? If you find you wouldn’t regret it, then why are you considering this option?
If the organization didn’t have X for another year, what would be the consequences? How much money might that lose the organization? (Notice dividing by 52 would give you a cost of delay per week.)
Drawing a Distribution: This exercise is from the field of Decision Quality. Run through the following questions, keeping in mind that their precise wording is important. Use people’s answers to help them draw a distribution for Y (cost of delay). This is another way to approach the CI conversation discussed above. Feel free to couple it with other activities, such as the equivalence bet test (the spinner).
If you were to draw a distribution of guesses for X’s cost of delay, what do you think the distribution’s median would be? (This is value for which half the distribution would be above it and half below it.)
For X’s cost of delay, what do you think a surprisingly low estimate would be, an estimate for which there is only a 5% chance the real cost of delay would be lower?
What do you think a surprisingly high estimate would be, the estimate for which there is only a 5% chance the real cost of delay is higher?
Do you still like your original estimate for the median? What are some reasons why it might be higher? Write down three reasons it might be higher.
Draw your distribution. What do you think its shape would be? Is it positively or negatively skewed? Why?
Do you think the real value is above or below your distribution’s median? Why?
As we saw in the discussion on calibration above, most distributions will be too narrow. Explicitly getting people to think about “surprisingly low” and “surprisingly high” values helps to broaden the distribution to a more appropriate range.
CI Straw Poll: Capture people’s 90% CIs in a table. (The example below is from John Cutler.) As you walk the group through some of these (or other) exercises, you’ll typically find that people’s answers are very different. Reinertsen has found that a group’s initial cost of delay estimates (for the same option) often differ by a factor of 50:1. (In one workshop, the group’s initial estimates differed by a factor of more than 1000:1!)
Focus the conversation on the range, leveraging it to elicit the different assumptions being made. Challenge assumptions, discuss differences of opinion, and then do another round of estimates. Try to get the difference down to a factor of at least 3:1. You can then average the lower- and upper-limit columns for a more robust overall CI. If you’d rather work with a single value, you can take the midway point between the resulting lower and upper limit as an option’s cost of delay.
After doing this for your different options (Xs), compare their respective CIs for Y (cost of delay). If an option’s CI is, say, $20K to $50K, then maybe you should keep looking. You wouldn’t prioritize this above something with a CI of $500K to $700K or $1MM to $10MM. Another way to say this is that if you’re fighting over a $10K difference, you’re not casting a wide enough net. Again, find more expensive problems to solve!
If, as suggested in the last post, you use T-shirt sizing to estimate duration, you can also then place your cost of delay estimates into three buckets: Small, Medium, and Large. You can then use a matrix like the one shown below to lay out your options.
If you have any questions, feel free to contact me!
Until next time.