Between an idea and a major investment lies a stretch of assumptions, and experimentation is what turns them into knowledge before the organization commits to something that is hard to reverse. Within this horizon we move step by step from ranking assumptions by how critical they are, to the types of tests available, to designing an experiment that yields genuine learning, and we close with the common mistakes made in pilots.
A service organization developed a new appointment booking system for its branches and chose its main branch in the capital to pilot it: the newest branch, the most heavily staffed, and the closest to the project team. Three extra staff members were assigned to the pilot to help visitors, and the development team came in daily to fix any problem on the spot. After three months, the pilot was declared a success: waiting times had fallen and visitor satisfaction had risen. The system was then rolled out to forty branches. In most of them, work was disrupted for weeks: the small branches had no one to help visitors, some had slower networks, and the development team could not be everywhere at once.
The pilot did not fail; it was designed to succeed. The system was tested under the best possible conditions, so the pilot proved that it worked under those conditions and said nothing about the conditions in which it would actually operate. This difference between an experiment that proves and one that teaches is what this chapter is about.
After studying this chapter, the reader should be able to:
In conventional projects, the plan is built on knowledge, and success is measured by adherence to it. In innovation, by contrast, the initial plan rests more on assumptions than on knowledge. McGrath and MacMillan observed that many new ventures fail because their founders treat assumptions as if they were facts, only to discover their errors after everything has been invested. They proposed what they called discovery-driven planning: writing the assumptions down explicitly and sequencing the stages so that each stage tests the most important assumptions before moving on, so that assumptions are gradually converted into knowledge (McGrath & MacMillan, 1995).
Experimental evidence supports this approach. Camuffo and colleagues conducted a randomized trial with 116 start-ups in Italy, half of which received training in making decisions by a scientific method: formulating hypotheses, testing them, and judging the results. The trained start-ups were more likely to change direction or stop when the evidence did not support their idea, and they achieved higher revenues (Camuffo et al., 2020). The experimental approach leaves the odds of the first idea succeeding unchanged; its benefit lies in knowing when the idea needs to change.
This is the logic underlying the gates of Chapter 10 and staged funding: each stage buys knowledge that reduces uncertainty, and the next stage is funded according to what the team has learned. Experimentation is the tool with which the team buys this knowledge. It is consistent with the principle of “managing uncertainty” among the principles of innovation management: dealing with risks by confronting them and learning from them.
Chapter 12 ended with a testable concept that states its critical assumptions, and Chapter 13 noted that the business model itself rests on assumptions. Experimentation begins by gathering all of these assumptions and then ranking them. They can be classified by the four evaluation lenses: assumptions about desirability (Do people want the solution?), feasibility (Can it be built and operated?), viability (Can it be sustained?), and responsibility (Is it fair and safe?).
Testing all the assumptions at once is neither possible nor useful. Priority goes to assumptions that combine two qualities: they are important, meaning that if they are wrong the entire initiative collapses, and the evidence for them is weak. Bland and Osterwalder propose ranking them along these two dimensions (Bland & Osterwalder, 2019), as Figure 14.1 shows.
Teams tend to start with what is easy to test before what must be tested. A technical team tests feasibility because that is what it does well, and postpones the desirability assumption because testing it means going out to people. The result is a solution that works technically and that nobody uses. Ranking assumptions in this way puts risk ahead of convenience.
Tests vary in their cost and in the strength of the evidence they provide. The principle of choice is simple: start with the cheapest test that could disprove the assumption, and move up to a costlier test only if the assumption survives the cheaper one. Figure 14.2 shows a ladder of such tests.
Every rung on the ladder requires a prototype of some kind. Houde and Hill pointed out that the most important question in designing a prototype is not “What are we building?” but “What do we want to find out?” They distinguished among prototypes that test the role of the solution in the user's life, prototypes that test its look and feel, and prototypes that test its technical implementation (Houde & Hill, 1997). Each question has its own prototype: one that answers the feasibility question falls short on desirability, and perfecting a prototype that answers the wrong question is waste.
One of the most widespread concepts in this area is the minimum viable product, the smallest version of a product that allows the team to learn as much as possible about users with the least effort (Ries, 2011). The key word in the definition is “learn”: the initial version is a tool for testing an assumption, not a stripped-down version of the final product that is launched and then added to.
Before your team asks for a budget to build an initial version, ask: Which assumption will this version test? And is there a cheaper test lower down the ladder that could disprove it? Many questions can be answered through conversations or manual delivery of the service before a single line of code is written.
The difference between an experiment that teaches and one that merely proves lies in the design, before the start. A good experiment needs five elements defined in advance:
Comparison is the hardest of these elements and the most often neglected in the public sector. The UK government's Behavioural Insights Team has advocated the use of randomized controlled trials in public policy, showing that in many cases they are less costly and complex than is commonly assumed, and that they allow organizations to learn what actually works before scaling it up (Haynes et al., 2012). Where random assignment is not feasible, comparison with similar branches or periods is still far better than no comparison at all.
At a university, these elements can become part of applied projects in courses: instead of presenting a solution and defending it, students present a hypothesis and test it with real users, and they are assessed on the quality of the experiment and what they learned from it, even if their hypothesis is rejected. This links the course to the innovation system and teaches students the method before it teaches them the result.
The pilot is the highest and costliest rung on the ladder, and it is run once the core assumptions have survived the cheaper tests. Its function is to reveal how the solution works under real operating conditions; it is a tool for learning more than for proof. Certain mistakes recur in pilots:
Every pilot has three possible outcomes, all of them useful if the pilot is well designed: the solution is scaled up, it is modified and tested again, or it is stopped. Chapter 15 addresses the move from pilot to scale.
An experiment that cannot fail teaches nothing. If the team cannot specify before the start the result that would lead it to stop or modify the initiative, then what it is running is a demonstration, even if it is called an experiment.
Organizations that experiment at scale uncover a seemingly discouraging truth: most ideas fail. Kohavi and Thomke reported that only 10 to 20 percent of experiments at major search engines produce positive results, and that small changes whose effect no one anticipated sometimes yielded large returns, such as a tweak to the way ad headlines were displayed that raised one search engine's revenue by about 12 percent (Kohavi & Thomke, 2017). The implication is that experience and intuition, however valuable, still need testing, and that an organization implementing its ideas without testing them implements many unhelpful ideas along with them.
A culture of experimentation is broader than its tools. It requires what Chapter 8 discussed: psychological safety that allows a team to announce that its hypothesis was rejected, a distinction between intelligent failure and avoidable failure, and leadership that asks “What did we learn?” before it asks “Why did we fail?” It also requires respect for the people on whom experiments are conducted: the beneficiary of a public service is not a test subject, and has a right to be informed and protected from harm. Chapter 19 addresses the ethics of experimenting on people.
Write a card for each experiment before it begins, and ask someone outside the team to review it. Once the experiment is over, complete the last two fields without altering anything written above them.
| Item | Question | Example |
|---|---|---|
| Assumption tested | Which critical assumption does the experiment test? | That branches with ordinary resources can run the system without additional support |
| Hypothesis | We believe that... will... because... | We believe that visitors to a mid-sized branch will book their appointments without assistance because the interface is simple |
| Type of test | Which rung on the ladder? Why? | A pilot in two branches: one mid-sized and one small |
| Measure | What behavior will we measure? | The share of bookings completed without staff intervention, and waiting time |
| Success criterion | What figure supports the hypothesis? (Set before the start) | Seventy percent of bookings completed without intervention, and waiting time cut by a quarter |
| Comparison | What do we compare against? | Two similar branches running the old system over the same period |
| Duration and limits | When does it end? To whom does it apply? | Six weeks, in the two branches only, with no additional staff |
| Stopping conditions | What would halt the experiment early? | Waiting time doubling for two consecutive weeks |
| Future owner | Who will adopt the solution if it succeeds? Are they involved? | The branch management department, represented on the pilot team |
| Result | What happened? (After the experiment) | |
| What we learned and the decision | Scale up, modify, or stop? |