Propaganda and opinion are easy; establishing the truth is hard (and I speak here as someone once branded Oxfam’s ‘chief opinionator’ – thanks John Magrath). Oxfam has been wrestling with different ways to evaluate impact for decades and in a new paper, a team led by Katrina Barnes ploughed through 67 ‘Effectiveness Reviews’ – rigorous impact evaluations on randomly selected samples of its projects around the world – to see what we’ve learned along the way. Lots to chew over in the findings – see the full paper.
The two sections that stood out for me were on Valuing Women’s Work and Climate Justice. Some extracts:
Valuing Women’s Work
‘We evaluated 36 projects that were tackling issues related to women’s work with communities in Africa and Asia. These projects sought one of two main outcomes:
(1) increasing women’s economic status – specifically to increase household income and wealth; or
(2) increasing various other dimensions of women’s empowerment.
These dimensions include personal (own attitudes, confidence), relational (household decision making, access and control over resources, unpaid care, community organising) and environmental (political engagements, safety). Some projects sought primarily to influence government policies on women’s empowerment.
Findings: ‘80% of projects pursuing broader dimensions of women’s empowerment were found to have a lasting positive impact after project completion. When looking at projects focusing solely on women’s economic status, we were less successful (only 45% of projects increased household income).’
What we Learnt: Women involved in Oxfam projects developed skills (such as agricultural or business know-how) to support their earning capacity. In most projects, these skills enabled greater production, an increase in paid work, and access to markets and credit. However, frequently this did not result in increased household income as we expected.
Community attitudes and narratives about women’s roles, along with government or customary policies, were cited as key reasons why we did not see the intended impact. These societal barriers affect all aspects of women’s work, including the crops they grow, the industries in which they work, their hours of work, their profits, and how dignified, sustainable and reliable their employment is.
Addressing these systemic societal barriers that devalue women’s economic contributions are areas in which we are seeing strong success, with over two-thirds of the projects tackling such barriers showing positive impacts on women’s’ empowerment. We see particularly strong changes around women’s own beliefs and attitudes, for example in joining community groups, taking leadership positions, and raising community awareness about care work or their rights around land ownership.
Working with coalitions of women and supporting women’s collective action were found to be successful approaches. Where we incorporated influencing components, we also saw positive impact.
Our climate justice work so far has focused on building individual and community resilience. We evaluated 28 of these projects, most in rural settings in Africa and Asia.
Findings: The evaluations revealed that Oxfam’s climate-related work has evolved over the past decade. We have moved away from focusing on diversifying livelihoods, and towards strategies that focus on bringing community members together in groups to engage in collective action and push for change to climate policies and financing.
All projects that focused on achieving climate policy change had a positive impact. Our work which sought to build community or individual resilience was also successful – with 85% of projects leading to people being better able to cope with climate shocks and stresses. However, projects that sought to increase household income or wealth remained a challenge, with only 30% of these projects having long term impact.
What we Learnt: One approach to climate justice is to support sustainable and climate-resilient livelihoods. We worked with people to improve skills, alter practices, access credit or diversify crops. However, similarly to the findings under Valuing Women’s Work, we found that whilst this led to increased skills, and in half of the projects led to diversified incomes or greater production, it was insufficient to consistently increase household income. This suggests that, overall, focusing efforts on community resilience had a greater impact than a focus on individual households.
Projects that aimed to raise awareness were successful when targeting policymakers, but we saw less change in communities – possibly as many communities are already very conscious of these changes and have often already developed their own adaptation strategies.
Work supporting coalitions, networks and alliances to influence national governments was largely successful. Many project teams noted that providing support to these networks made the most significant contribution towards achieving project objectives.
Most projects had a duration of between three and five years, yet many evaluations highlight that to achieve resilience and change climate policy, longer timespans are necessary. We are therefore working with others to advocate for and pilot longer-term funding approaches.
Looking to the Future: Building on these evaluation findings we will no longer focus on increasing incomes at a household level, but instead focus on working with partners and communities on systemic solutions. The systems that govern people’s land, agriculture, food, and energy are all integral to how communities adapt – we will work with leaders from the Global South to influence these systems to become more equitable for all.’
Finally, Katrina reflects on how this kind of work is changing the way we think about Oxfam’s role:
‘Two fundamental changes in our understanding of impact have led us to again evolve, and create a new Learning and Accountability Framework. These are:
1. Who defines what should be measured? Defining success and measuring impact has historically been controlled by organisations based in the Global North, rather than by those who experience the changes we are seeking. By focusing on Western understandings of outcomes and ways of knowing, we have not adequately centred communities or in-country partners to decide what changes are meaningful to measure. Nor have we done enough to ensure that this learning is owned and used by the people closest to the changes.
2. Addressing root causes by working on systems change. Sustained change is complex, occurring through the interactions of many actors. Assessing the impact of individual projects fails to capture systemic changes. At Oxfam, we are investing in impact through broad, dynamic portfolios of national and global initiatives. Our approach to learning is therefore moving from examining standalone projects, to one that looks for signs that systems are changing.’
These changes of direction seem right and admirable, but they aren’t going to make the job of measuring impact and learning from it any easier!