My API colleagues Lloyd Provost and Jerry Langley pointed me to a 2014 article by Anhøj and Olesen, “Run Charts Revisited: A Simulation Study of Run Chart Rules for Detection of Non-Random Variation in Health Care Processes” (PLOS One, http://dx.doi.org/10.1371/journal.pone.0113825).

Anhøj and Olesen look at the three run chart rules I reviewed in my post “Run Charts in Quality Improvement Work”, published 2 February 2015. They also offer useful guidance on run chart analysis as “Guidelines for Using and Interpreting Run Charts for Health Care Improvement.”

For the shift rule—number of consecutive values on one side of the median of a series of values—Anhøj and Olesen cite M. Schilling (2012), "The Surprising Predictability of Long Runs", Mathematics Magazine. 85, pp. 141–149 (available here). Schilling’s analysis leads to a simple formula for the number of consecutive values on one side of median that would be surprising, relative to series of independent realizations from a single probability distribution.

If you define n as the length of the original series, omitting any points that fall exactly on the median, the formula is: calculate log2(n) + 3 and then round to the nearest integer.

Using the simulation functions developed in R for my 2015 post, here’s the frequency of seeing 6, 7, 8 or 9 consecutive values on one side of a median for series of length n, n between 12 and 48.

Go to this GitHub repository for the R Markdown file that will produce the table shown here.

Restricting run chart analysis to series lengths less than n=20, the table shows that the rule “shift of six consecutive values on one side of the median” proposed by Perla et al. (2011) is a reasonable rule of thumb when looking for confirmation of improvement in short series.

While the Anhøj and Olesen rule doesn’t make sense for a series of length 12--the longest shift on one side of the median is six consecutive values—it looks like a reasonable guide for run chart analysis if you can estimate the critical value, which depends on a base 2 logarithm. For the range of n in the table, just linearly interpolate between powers of 2. E.g. for a series of length n=22, 22 is less than halfway between 16 and 32, so the quick estimate critical value is log216 + 3 = 7. When length of the series is halfway or more to the next power of 2, use the higher power of 2: E.g. for n=48, 48 is halfway between 32 and 64, so the estimate of the critical value is log264 + 3 = 9.

On the other hand, as Lloyd Provost noted last year, for series longer than n=20, a control chart often can provide more insight than a run chart. You can augment the Shewhart “3 sigma rule” with a shift rule based on Anhøj and Olesen, as these authors suggest in the Guidelines section of their paper.


On the first Sunday after the start of the school year, the Quaker Meeting I attend in Madison, WI has a pancake breakfast. The event marks the start of the new year’s First Day School for the Meeting’s children (Quakers traditionally use numbers for days of the weeks and months of the year—First Day is Sunday).

The aim of the pancake breakfast is actually to provide an experience for children and adults to work together to produce and share the meal. Children and a teenager or two pour dollops of batter onto the griddles, monitor the cooking, flip and serve. The adults are in the kitchen, mostly in support.

To cook and serve a few hundred pancakes in the 100 minutes between the end of first Meeting for Worship at 9:30 a.m. and the start of late Meeting for Worship at 11:15 a.m. takes organization and planning.

Bob Newbery, adult leader of the pancake team for 15 years, has implemented a key idea: convert internal set-up to external set-up.

Let me explain.

In pancake production, you assemble and combine a variety of dry and liquid ingredients.

Bob’s standard recipe at left shows that you start by combining the eggs and buttermilk, then add the sifted dry ingredients, followed by the melted butter. In most home settings, it is perfectly all right to proceed by combining the five dry ingredients, measuring out and sifting and then working through the recipe. This takes 10 minutes or more just to work with the dry ingredients, not counting the gathering and opening of containers, cleaning up inevitable spills, and putting things away.

So Bob converted the set-up of dry ingredients to make it "external" to our Quaker pancake cooking process. That is, the only set-up that needs to happen right before cooking on pancake breakfast Sunday is the combining of the wet and dry ingredients. That combining remains as "internal" set-up in our current production system.

Here's the external set-up:  All of the dry ingredients are premixed and assembled in plastic containers of the correct volume to correspond to the recipe. You can see a container of premixed and sifted dry ingredients ready to go in the blue-topped container at the start of this blog.

To prepare the batter takes less than four minutes, even less if the butter has been melted before mixing of ingredients begins. As the production bottleneck is the actual cooking of the batter on the griddles, people like me who are batter makers then can help the kids or wash dishes so that the kitchen is productive and cleaned up by noon. Given our production system, there’s no point of converting more internal set-up to external but as an exercise for the reader, what other set-up now internal could be converted to external?


Shigeo Shingo first distinguished the notion of internal and external set-up. He discussed this concept at length, along with many other insights in his book A Revolution in Manufacturing: The SMED System, Productivity Press, 1985, Cambridge, MA. SMED stands for “single minute exchange of die”, which enshrines the breakthroughs Shingo developed, culminating in the reduction in set-up time for a press at the Toyota Motor main plant in 1969 from hours to under 10 minutes--to single minutes, to change out the die set in the press.

Clinic Application:  The week before Pancake Breakfast, I observed young patients in a dental clinic receiving services. Good dental practice calls for application of sealants to permanent molars for kids with elevated risk for cavities.   The sealant process takes just a few minutes with a reasonably cooperative child. If the dental staff can provide this service while the patient is in the exam room (rather than needing to make a special additional appointment), that’s good for the patient and good for managing the clinic appointment schedule. To find the time for sealants during a standard exam appointment, applying the change concept "convert internal set-up to external set-up" can yield the minutes required.


In The Improvement Guide, 2nd edition, Langley et al. propose this definition of a Plan-Do-Study-Act cycle:

“To be considered a PDSA cycle, four aspects of the activity should be easily identifiable:

1. Plan: the learning opportunity, test, or implementation was planned and included

  • Questions to be answered

  • Predictions of the answers to the question

  • Plan for collection of the data to answer the questions

2. Do: the plan was attempted. Observations are made and recorded, including those things that were not part of the plan.

3. Study: time was set aside to compare the data with the predictions and study the results.

4. Act: action was rationally based on what was learned.”  (p. 98-99)

It’s good to keep this definition in mind when someone tells you they regularly use PDSA. They might be missing a critical piece that will add to the power of their tests.

Of course, as the authors say, not every improvement requires the formality of PDSA as described here; nonetheless, “purposeful improvements in large or complex systems will usually require one or more cycles.” (p. 99).

The authors go on to describe why they insist on including predictions in the definition of the Plan step.

They include this reason: “Prevent hindsight bias (‘I knew it all along.’)” (p. 99)

Hindsight Bias and What to Do About It

Ulrich Hoffrage and Ralph Hertwig argue that hindsight bias is inherent to the way humans interact with the world:

“…it is a by-product of two generally adaptive processes: first, updating knowledge after receiving new information; and second, drawing fast and frugal inferences from this updated knowledge.” (Gerd Gigerenzer; Peter M. Todd; ABC Research Group. Simple Heuristics that Make Us Smart (Evolution and Cognition); Chapter 9: Hindsight Bias: A Price Worth Paying for Fast and Frugal Kindle Locations 2583-2585. Kindle Edition.)

Hoffrage and Hertwig go on to cite Baruch Fischoff’s cogent summary of why hindsight bias is a problem for anyone attempting to learn from experience, including those of us using PDSA cycles:

“When we attempt to understand past events, we implicitly test the hypotheses or rules we use both to interpret and to anticipate the world around us. If, in hindsight, we systematically underestimate the surprises that the past held and holds for us, we are subjecting those hypotheses to inordinately weak tests and, presumably, finding little reason to change them. Thus, the very outcome knowledge which gives us the feeling that we understand what the past was all about may prevent us from learning anything from it.” (Baruch Fischoff, “For those condemned to study the past: Heuristics and biases in hindsight.” In D. Kahneman, P. Slovic, & A. Tversky (Eds.) (1982), Judgment under uncertainty: Heuristics and biases Cambridge, UK: Cambridge University Press, p. 343.)

If Hoffrage and Hertig are right that hindsight bias is a natural result of human evolutionary history, then we should expect hindsight bias in me and you just about every time we try to learn from experience.

Conscious prediction, as part of the Plan step, serves as a strong corrective to hindsight bias, which otherwise can severely limit the value we might extract from tests.

So it looks like The Improvement Guide authors have it right:  prediction is not a nice-to-have feature of a Plan-Do-Study-Act cycle but vital to getting the most out of learning.


I discussed prediction related to PDSA in two previous posts:

"Predictions drive deeper learning:  true for pre-verbal infants as well as for you and me" (here) and "It's tough to make predictions especially about the future but it's worth the effort" (here).

Back to top