How can we create an evidence base for hope?

October 20, 2021

     By Duncan Green     

In their second post on Oxfam’s Inspiring Futures project, Irene Guijt and Ruth Mayne nerd out on the methodology.

If hope is stronger when anchored in evidence, what does it take to create that evidence base? There were plenty of head scratching moments involved in answering our main question:

What evidence exists that transformative and inclusive change at scale is possible in challenging, lower-income contexts and what insights can we gain about how to achieve it?

The first step was to establish the research focus – thematically, geographically and historically.  We wanted evidence of change related to one of three crises – economic inequality, gender injustice or environmental destruction – to inform Oxfam’s work.  One of the biases in the existing evidence base for structural change is its focus on the wealthy OECD contexts. Less is documented on structural solutions to systemic crises in contexts with low income or human development indicators, restricted political rights, extreme inequalities and injustices, or, and conflict. We wanted to start filling this gap. Third we wanted to learn from the recent past, and not rehash iconic historic examples such as slavery or women’s suffrage. What could Oxfam learn about new tactics, new opportunities, and the dynamics of transformational change that was responding to new challenges, such as acute water shortages in Cape Town or transport challenges faced by low-income settlements in large cities?

As our focus was on structural change, we were not looking for tweaks or nudges to existing ways of working, small islands of success, or work that reduces symptoms. We sought evidence of structural changes at meso-scale, going beyond local (micro) successes without duplicating documentation on national level (macro) change. Examples had to focus on structural change that tackled key elements of the system that were locking problems in place. For example, changing practices in female genital mutilation involves many different actors pushing social norm changes, proposing cultural alternatives, advocating for policy change across many countries.

Finally, we wanted examples with sufficiently robust evidence. Much anecdotal evidence exists but we zoomed in on examples of change substantiated by credible sources with underlying data. It was surprising difficult to find examples that described not only the problem and the change but also the transition pathway, obstacles en route and inevitable trade-offs, which limited case study selection.

Collecting evidence for hope was not straightforward

Defining ‘transformative change’ and ‘structural causes’. Just as there are multiple definitions of ‘system’ (e.g. Meadows 1999), there is no consensuson terms such as transformative change or structural causes.We defined transformative change as ‘change that has generated major economic, social, health or environmental benefits for people’s lives or for wider society by changing its fundamental nature’; it helped filter out smaller scale islands of success. Focusing on ‘structural causes or drivers’ of poverty or injustice led us to concrete cases that achieved more than one of the following:

  • Rebalanced power and gender relations
  • Widened political space and strengthened political participation
  • Changed government legislation, policies and practices
  • Transformed corporate policies and practices
  • Innovated and expanded access to appropriate technology
  • Developed and expanded access to inclusive and clean infrastructure
  • Shifted cultural beliefs, social norms and behaviours
  • Strengthened know-how and skills
  • Restored environmental health on which human life depends
  • Transformed markets.

Sufficient scale. Given the size and number of problems that humanity faces, a holy grail for poverty or environmentally focused organizations has been ‘to get to scale’ and to do so quickly. The question of what is enough to count as ‘impact at scale’ was quite a conundrum for us. Is it a certain number of people, a certain geographic coverage, a minimum number of levers? Different domains of change have different scales, for example saving an ecosystem is bounded differently to shifting gender norms. We settled on the meso-level, seeking examples that positively affected populations of at least several hundreds of thousands.

There are downsides to fetishizing scale. It risks valuing impact rather than the journey of change and can risk prioritising big numbers over participation, quality, inclusion and equity. It can ignore what are often considerable previous struggles, which help create the conditions for system shifts. Or it can neglect the need for many smaller efforts to be tried, before seeing which ones can be scaled.

Positive – or negative – bias. Positive bias is a risk, with case study authors over-reporting or not adequately verifying, or ignoring unintended problematic outcomes and impacts. On the other hand, under-reporting of benefits and impacts can also happen, as indirect reach, ripple effects or long-term benefits are rarely included in impact numbers. For example, girls’ schooling (through gender responsive budgeting in Pakistan) is known to generate multiple long-term benefits but is associated purely with advancing gender parity in education. Such benefits include improved health, higher participation in the labour market, higher incomes, later marriage, fewer children and improved health and educational outcomes for their children.

Finding good enough evidence quality. For each case, we asked what constituted sufficient evidence. The quality, reliability and coverage of the evidence varied for each case. While there was often relatively strong evidence for outcomes, data on impact and attribution,  disaggregated data and intersectional analyses were difficult to find. In some cases, impact at scale was inferred from data on reach and outcomes, if and where a wider reliable body of evidence existed to justify this, for example, positive future impacts of increased access to girls’ education or increased government spending on health. The cases were validated by people closely involved and/or peer reviewed.

Accepting no case was perfect. No single case met all the criteria, particularly that of an inclusive impact. For example, is it okay to include case of dramatic decrease of water consumption which safeguarded water supply for the entire population, if it failed to address existing inequalities for access to water for low-income women?

Overvaluing intentionality. There is a risk of over-ascribing intentionality to a story of change that was, in fact, a mix of historical opportunities and intentional change (– and possibly some luck, although we did not systematically assess this). We did not specify whether cases should be about intentional, planned change or spontaneous action. So it was quite surprising that nearly all 18 cases involve intentional rather than spontaneous impact at scale, with some elements of their scaling processes occurring spontaneously. Only the spread of environmentally friendly practices between farmers in the Sahel was mainly spontaneous, even that though with some external NGO support. This suggests that either spontaneous cases of radical change at scale are not well documented or that fewer cases of spontaneous change exist that are transformative and inclusive change at scale. We’d be keen to hear of more examples.

On balance

We are acutely aware that 18 cases are but a small contribution to the existing literature. Nevertheless, the set offers some additions to what is known:

  • Evidence and stories of structural change in institutionally and resource-constrained contexts that went beyond influencing policy.  
  • Highlighting the opportunity to simultaneously mitigate the triple crises (economic inequality, gender injustice and environmental destruction). If we had started with a narrow scope of work, the linkages between crises would not have emerged.
  • Focusing on a meso-level evidence base and why this is valuable.
  • The need for a mix of mutually reinforcing scaling pathways (vertical, horizontal, function) to have impact at scale, rather than pursuing one pathway on its own.

Many more cases exist that could offer insights about transition pathways for systems change.We are particularly interested to stress-test questions about meso-level change with other cases and to flesh out the idea of meso level change itself, as part of transitions.Our limited set has multiple caveats. We were unable to obtain sufficient grounded validation through interviews. Such insights would have given critical detail about the difficulties, struggles and trade-offs, what tactics and strategies were used to overcome obstacles, and what role hope played in sustaining efforts.  We were keen to have conversations with users, especially those who are sceptical of systemic change and whether the triple crisis can be averted, to understand what role evidence might play for them in creating change.

One finding from the case studies was about the efficacy of horizontal scaling – when change happens by people copying and adapting good ideas. So, besides contributing to a wider evidence base of hope, maybe these individual case studies can catalyse further transformative change. Tailoring findings to specific audiences, facilitating spaces for changemakers to share tactics and strategies, and to present their solutions to power holders could then contribute to further transformation.

October 20, 2021
Duncan Green