In planning and design problems, how much testing should you do?

In a couple of previous posts, I’ve argued that planning and design problems are common and the Model for Improvement is a great framework for tackling them.

The authors of The Improvement Guide pack a lot of good advice and insights into their book.  Chapter 7 is all about testing, a fundamental skill in addressing planning and design problems.

When you have a change in mind, you almost always should try it out before using the change regularly. You should run a test. You face many decisions in running a test; you have to answer the what, who, how questions. One key decision involves the scale of the test. How big should it be?

Here’s a summary presented as a table in Chapter 7 to help you think about the size of a test.

(Table 7.1 G. Langley et al. (2009), The Improvement Guide, 2nd edition, Jossey-Bass, San Francisco © Associates in Process Improvement)

You can see three factors that determine the size of your test:

 (1) your belief in the effectiveness of the proposed change;
 (2) the cost of failure—if the change idea doesn’t yield the expected improvement, what’s the penalty?; and
 (3) organizational commitment to support the change. Do formal and informal leaders agree with the change? Are people ready incorporate the change into training, adapt related systems, and ready to maintain the change into the future?

Now your judgment of these three factors will be approximate. The main point to consider is that too large a test may generate more resistance from your colleagues and create more problems than the value of new knowledge gained from the experiment. Better to do a very small-scale test, with readiness to follow up with a larger test if your change shows promise than to poison the willingness of people in the organization to try something new or to cause a lot of extra work cleaning up after a failure.

What’s a very small-scale test? A starting point is the “rule of 1”: if you have a production process, can you apply the change to one cycle and see what happens? In direct healthcare settings, rule of 1 translates to a test involving your change in working with “one patient, one clinician, one encounter.”

An Advantage of Testing

One reason to test on a very small scale: while you may have a clear idea about what to change, often it is less clear how to make the change stick. Testing helps you learn about the How as well as the What.

There are many examples of changes proposed with good intentions that fail to deliver, like the intervention to improve Newark public schools (e.g. Joe Nocera’s discussion of Dale Russakopf’s new book, The Prize: Who’s In Charge of America’s Schools? in the New York Times, here.) Surely testing aspects of What and How could have helped the reformers learn faster to adapt their approach, with better outcomes and less cost.

What about Changes that Stick, without Multiple Test Cycles?

Table 7.1 helps me understand why sometimes jumping to implementation appears to work—in the sense that the change performs as expected, leading to better results, without multiple tests required.

That may be the case for some changes generated by organizations applying Lean methods within local work units.

A change that emerges from careful study of a work process by the people doing the work can generate a high degree of belief that the change will lead to improvement.

If the people are part of an organization that coaches and encourages staff to monitor and improve their work, there will be strong organizational commitment to support the change.

The third factor—cost of failure—will often be small for many local process changes.

So, successful incorporation of local changes into regular practice without extensive testing need not surprise us nor imply a limitation to the Model for Improvement.

Table 7.1 shows that the complete answer to the question, “should I test my change before putting it into regular practice?” is “it depends.” But Table 7.1 also suggests that in most situations it will pay to work incrementally, learning through testing, about both the What and How of the change.