Risk of Representation

Risk of Representation

Risk of Representation

Over the past several weeks, I’ve been reflecting on a “risk of representation” in project measurement.

 James Scott’s perspective and David Chapman’s work are helping me to think, see, and act more clearly.

Most of my projects over the past 25 years have gone like this:  

  1. A group of people decides to improve some aspect of healthcare—to close a gap between the current and the desired state.

  2. The group recruits organizations to collaborate in an intentional improvement community.

  3. Each organization picks a project team to participate in the improvement community.

  4. The community works to realize improvements.

My job is to help build and apply measures to guide and track progress in closing the gap.  I’m often engaged to help project teams learn by doing, using simple tests and sometimes more complicated experimental designs. 

Across my projects, details have varied with the scope of the improvement aims, the size of the intentional improvement community, and the time and money available to support the work.  

The Model for Improvement has informed all of these projects—both as a design framework for the community and as a specific method the teams use as they try to make improvements.

What about co-design?

Critically, none of my projects over the past 25 years have demonstrated meaningful co-design of improvements. By co-design, I mean patients and families served by the project’s system work as partners with healthcare leaders, clinical staff, and improvement experts to design and deploy care improvements.

In past projects, co-design was usually aspirational, though a small number of projects have made important choices to involve one or more patients and families in discussions about aims, measures, and changes. My projects to date have not addressed the challenges of integrating patient and family co-designers.  I haven’t experienced co-design work that addresses compensation for participation and scope of responsibility, and we’ve typically failed to have a range of patient and family voices that span the people served by the project’s system of care.

It’s getting easier to build measurement systems

The second question of the Model for Improvement is all about measurement: how will we know that a change is an improvement?

In an improvement community, measuring performance includes collecting data from individual organizations, organizing and sharing it with the community, and displaying it in useful formats. 

In the past, acquiring and sharing data was a major constraint for improvement communities.

How would each organization extract appropriate data from local sources and share it with the community for analysis?   Customized local reports typically required IT resources that were already scarce.  Often, data reporting included manual review or even manual entry into spreadsheets or web forms to share with the community.  

Over the past decade, consolidation in the U.S. electronic health records (EHR) market and federal U.S. adoption of new protocols like Fast Healthcare Interoperability Resources (FHIR) have started to ease the constraint.  When most organizations in an improvement community share a common EHR platform, it becomes much easier to build one reporting tool to extract relevant data.    With appropriate data-sharing agreements in place that address the security of patient-level information, vendors are offering products and services to aggregate data from multiple organizations in ways that meet the technical needs of an improvement community while eliminating local manual data entry.

Measurement system risks in current project work

I welcome a lower burden on individual organizations in an improvement community, including the reduction (or elimination) of manual data entry, and I value the benefits of effective data displays,

Nonetheless, I worry about potential design bias in building a community’s measurement system, especially when the community does not include a range of patient and family co-designers.

 Will we gravitate toward, or even focus exclusively on, numerical measures that appear well-defined in the EHR?   A dashboard of numerical measures, no matter how well-designed and flexible, remains a simplified representation of a real-world system.  

I also worry about representational anchoring.  The faster the dashboard refreshes with up-to-date data gathered with no human effort and the easier it is to drill down and make graphical comparisons, the more attractive the improvement community’s dashboard becomes.    How can we fight the natural tendency to treat the dashboard as a complete picture of system performance rather than an incomplete representation that will be good enough for at least some of our purposes?

Mitigating Measurement System Risks

How?  Here’s the advice I now give myself and my clients: Interact frequently and intentionally with the people experiencing and working in the system the dashboard represents.

Co-design is one tactic. 

The other ways are day-to-day tactics, which serve as a foundation for co-design:

  1. Observe the system in action:  “Go See” the work and “Shadow” patients as they move through care episodes. Previous relevant posts are here, here, and here.

  2. Ask open-ended questions:  “Plus-Delta” is described in this post and a 6.5- minute video presentation featuring Yvonne Fillmore, NP, Eagle View Community Health Center.

Until recently, qualitative analysis of open-ended responses to Plus-Delta questions has required significant time and effort.  It now seems feasible to use large language models to extract themes and patterns from verbal responses.   I’m eager to test this idea.  How willing are patients or staff members to answer our questions?  Put another way, how much trust do patients and staff have in the people asking the questions?  What can we learn about practical details like how and when to respond directly to issues raised, the use of written vs. oral answers, specific choices of language models, synthesis of themes, and management of improvement opportunities?

Reminder from Carlile and Christensen (2005)

Eleven years ago, I wrote about Carlile and Christensen’s working paper, “Practice and Malpractice in Management Research”

Substitute the noun “manager” or “improver” for “researcher” in the next paragraph from their paper, and the advice still holds—everyone involved in improvement, inspired by the Model for improvement, is trying to build a better theory of what works and why, seeking to build better predictions about future behavior.

“The healthiest and most accurate mindset for researchers is that nearly all data – whether presented in the form of large data sample analysis on one extreme, or an ethnographic description of behavior on the other – are subjective. Numerical and verbal data alike are abstractions from a much more complex reality, out of which the researcher attempts to pull the most salient variables or patterns for examination. Whereas the subjectivity of data from field-based, ethnographic research is glaringly apparent, the sources of subjectivity are hidden behind numerical data and cannot be reviewed. There should be no smugness amongst quantitative researchers about the alleged objectivity of their data, and no defensiveness amongst field researchers about the subjectivity of theirs. We are all in the same subjective boat, and are obligated to do our best to be humble and honest with ourselves and our colleagues about where our data comes from as we participate individually within and collectively across the theory building cycle.” 

This excerpt is from version 6.0 of the paper; the version publicly available, 5.0, does not contain this paragraph but does contain the main discussion of theory-building.  I recommend reading version 5.0 if you can’t find 6.0.

Seeing Like A State

Seeing Like A State