On 20 November I attended a workshop on realistevaluations, organised by IDS Fellow Inka Barnett and co-sponsored by the Centre for Development Impact. The focus was ongoing work in evaluations and how this could be improved. Realist evaluation experts Bruno Marchal and Sara van Belle, who came over especially from Antwerp, provided an excellent overview of the realist evaluation paradigm and actively engaged with workshop participants. Throughout the day it became clear that the realist paradigm is the most detailed and sophisticated of the theory of change / theory based evaluation approaches. These are various theoretical frameworks that aim to look at assumptions that underlie policies, programmes and interventions. These assumptions can then be evaluated to see what works, for whom, how and under which circumstances.
The examples of realist evaluations explored during the workshop were rich and varied. They ranged from using mobile phone technology for nutrition surveillance to a realist synthesis of evaluative evidence on water and sanitation issues, to an evaluation of influencing the Chinese position on global health issues. Participants struggled a bit with some of the jargon: assumptions are especially framed in terms of ‘CMOs’: context-mechanism-outcome configurations that describe how a specific mechanism is thought to bring change (the desired outcome) in a specific context. However, this is the element that makes realist evaluation potentially richer than other theory based approaches.
As an IDS Visiting Fellow I am involved in supporting the Centre for Development Impact, so I was interested in how realist evaluation could be positioned in the range of impact evaluation methodologies that the Centre promotes. For me, the realist evaluation approach scores high on a number of issues. It is the richest and theoretically most satisfying version of the theory based approaches that have been developed over the past decades. The approach accommodates the greatest range of potential evaluation questions. This is immediately clear in the questions that are often used to characterise realist evaluation: what works for whom, how, when and where under which circumstances. The focus on causality, on mechanisms that are supposed to bring change, makes it especially worthwhile for impact evaluations of complex interventions. And a great advantage is that realist evaluation is not dogmatic as regards evidence, counterfactuals and tools and methods, and has a theoretical underpinning for this. In other words: realist evaluators have rigorous justification for the use of data and evidence through the realist evaluation framework.
Some drawbacks also need to be mentioned. The jargon seems somewhat forced – CMOs need to be explained to outsiders. As usual with these frameworks, the need to clearly define what you are talking about may actually turn against you – the best way to prevent this in evaluations is to maintain a strong focus on the questions that need to be answered. If the jargon stands between your question and your answer, the realist framework may not be the best to follow.
A second more profound difficulty is the fact that the realist evaluation framework is all about social change. This is a limitation that becomes a bit of an obstacle when other types of change need to be taken into account. Evaluation is increasingly confronted with change processes in other domains: climate change, environmental degradation, use of natural resources, value chains, and so on. If the follow up to the millennium development goals in 2015 will integrate use of natural resources and environmental issues into new sustainable development goals, the realist framework in its current incarnation will not help them. A new challenge emerges: that of adapting realist evaluation to include non-social processes of change.
The evaluations that were discussed are very relevant to the work of the Centre for Development Impact. Compliments to Inka Barnett for organising this! The Centre for Development Impact should invite these evaluations to become part of a community of practice on impact evaluations, hosted by the Centre, where evaluators may exchange questions, solutions, issues to further explore and learning on new developments. I hope to see more of this in the near future!
Rob D. van den Berg is a Visiting Fellow at the Institute of Development Studies.