Evaluation is a key part of design for health. We can’t claim to be improving people’s lives, health, or wellbeing if we don’t have any means of evaluating what we’ve created once its placed in the world. But what we end up depends highly on what we start with.
Welcome to The Design Loft #65 where today we’re going to talk about the importance of designing for the outcomes you want at the front end of your journey.
Taking account of the perspectives you want reflected in your programs or services ahead of time can save an enormous amount of energy when you come to evaluate it down the road. Let’s look at how we do this.
The Evaluator’s Dilemma
Designers aren’t typically trained in evaluation. Evaluators are rarely trained in design. One of the reasons The Design Loft was conceived of was to address this disconnection (see the post below). We can’t elevate our designs if we have little idea of what they do once put into the world.
Evaluators have their own issues to deal with when dealing with a program or service that bears heavily on how things have been designed. I recently met a colleague for coffee and to commerserate about evaluation issues in health system programs and she we discussed the scenario that both of us have seen played out many times: asking for specific types of analysis in program data. Specifically, we’re speaking about the matter of using a particular lens on program data to inform the findings.
This could involve usng a feminist lens, a health equity perspective, or taking an empowerment perspective, among others. Evaluations of programs often apply lenses that place data in a perspective that is aligned with the goals and mission of the program or service. Many of our designs bring with them or are intended to reflect a particular set of values or principles, which is what an evaluation is intended to reflect.
Where things can start to get unnecessarily complicated is when our designs and the data we’ve gathered to assess them don’t match up. That happens a lot, and it’s one of the many reasons why design and evaluation work need alignment. It creates a dilemma for an evaluator when a program sponsor asks of an evaluation (after the data has been gathered) if we can apply a particular lens or theoretical approach to the data.
For example, we could explore a program or service by taking a decolonization perspective, which might look at how certain values are represented or excuded from the design. That is a far more difficult thing to do if you didn’t collect data with this in mind at the outset. All data gathered reflects values and these values might be expressed subtlety or unconsciously, or in a rather benign manner. Applying a lens post-hoc to data after it’s been gathered might be interesting, but is it useful?
What if one was to design the evaluation with this perspective at the outset? If it was, we might ask different questions and also learn more about our design in a manner that can support improvements. We might uncover certain strengths, weaknesses, gaps, or omissions in an intentional manner that can inform the ongoing development of the program. But if evaluation isn’t part of the discussion at the outset, that’s hard to do.
What Are We Doing?
There are always tensions in health programs. Designing requires we balance needs, resources, pragmatism, and desired outcomes and effects. It is impossible to design for everyone and every context. Designers have to make trade-offs and these extend to the evaluations required to understand the influence these have on the final product. As noted in an earlier issue, linked above:
We can begin to surface these tensions between different design effects by asking: 1) what is [our design] being asked to do? and 2) in what context is it being asked to do it?
Evaluators can help designers in the clarification of what the intention of a program is, and what the implications are. For example, if we take a perspective that is aimed to elevate or amplify certain populations, situations, or abilities in our data collection, we can reveal hidden biases and barriers that might exist in the design that couldn’t be seen from the early prototyping.
Understanding the gap between our aims and context matters not only for our designs (the inputs), but our evaluations (outputs and outcomes).
If we are evaluating the impact of what we create — a program, policy, service, or product — we need to understand the role of different forces in shaping the outcomes. It is possible — and often the case — that we fail to see the desired outcomes from our work because of these countervaling forces that influence our designs.
Our perspectives can, if designed at the front end, shape the questions that allow us to see these forces. It’s because they are made explicit and we can assess what we create based on the lens used to evaluate the creation. If we have to guess, or we are applying it post-hoc, we are more likely to find that our data is poorly set up to reflect what we want.
To use the previous example, if we are using a feminist approach to evaluation, we may ask questions like:
Whose questions are being asked, who is centered in the process, and who holds decision-making power? For example, In what ways do participants co-design or control the data collection, interpretation, and reporting stages of the evaluation?
How do multiple social identities (e.g., race, class, gender identity, ability, age) combine to shape different experiences? For example: In what ways do program experiences and outcomes differ for marginalized or non-binary gender identities compared to dominant groups?
Is the program (design) addressing inequities and challenging inequality or merely treating symptoms? For example: Whose definition of "success" or "merit" is being used, and what competing definitions are being ignored?
Asking evaluation questions in advance can shape design questions and thus, what we end up putting into the world. This is another reason why evaluation isn’t just about accountability or evidence generation, it’s a source of design guidance and a means to ensure that we achieve greater health because of what we make and do.
Practice Notes
Asking evaluation questions in advance can shape design questions and thus, what we end up putting into the world.
My closing notes are to practice evaluation. Begin by starting to ask questions and reflect upon the answers you’re looking for (not the specifics, but the kind of information you want to learn). Evaluation data can be gathered through sophisticated means, or using simple tools like the above example.
Evaluation can be learned and practiced drawing upon many of the methods and tools that we might learn in a research methods course or seminar on social impact. My site Censemaking (see links in menu) has many articles on the application of evaluation, evaluative thinking, design, and design-driven approaches to evaluation and evaluation of design activities.
If you’re a health leader or health promotor, your aim is to create and facilitate programs and services that increase people’s capacity — whether its patients, caregivers or your staff — to thrive. Whether it’s healing sickness, rehabilitation, palliative care, mental health promotion, or organizational performance, we all want to create health and value in what we do. Evaluation is the means to knowing and realizing this value through our designs.
So practice. Ask questions. Start with evaluative thinking and go from there.
Thanks for reading!
Evaluation is the means to knowing and realizing value through our designs.
References
If you’re interested in knowing why and how a feminist approach to evaluation might make sense for certain designs and context, I’d highly encourage you read the book Man Made: How We Designed a World That Leaves Women Out, and How We Can Make It Right by Karen Korellis Reuther