Propaganda and opinion are easy; establishing the truth is hard (and I speak here as someone once branded Oxfam’s ‘chief opinionator’ – thanks John Magrath). Oxfam has been wrestling with different ways to evaluate impact for decades and in a new paper, a team led by Katrina Barnes ploughed through 67 ‘Effectiveness Reviews’ – rigorous impact evaluations on randomly selected samples of its projects around the world – to see what we’ve learned along the way. Lots to chew over in the findings – see the full paper.
The two sections that stood out for me were on Valuing Women’s Work and Climate Justice. Some extracts:
Valuing Women’s Work
‘We evaluated 36 projects that were tackling issues related to women’s work with communities in Africa and Asia. These projects sought one of two main outcomes:
(1) increasing women’s economic status – specifically to increase household income and wealth; or
(2) increasing various other dimensions of women’s empowerment.
These dimensions include personal (own attitudes, confidence), relational (household decision making, access and control over resources, unpaid care, community organising) and environmental (political engagements, safety). Some projects sought primarily to influence government policies on women’s empowerment.
Findings: ‘80% of projects pursuing broader dimensions of women’s empowerment were found to have a lasting positive impact after project completion. When looking at projects focusing solely on women’s economic status, we were less successful (only 45% of projects increased household income).’
What we Learnt: Women involved in Oxfam projects developed skills (such as agricultural or business know-how) to support their earning capacity. In most projects, these skills enabled greater production, an increase in paid work, and access to markets and credit. However, frequently this did not result in increased household income as we expected.
Community attitudes and narratives about women’s roles, along with government or customary policies, were cited as key reasons why we did not see the intended impact. These societal barriers affect all aspects of women’s work, including the crops they grow, the industries in which they work, their hours of work, their profits, and how dignified, sustainable and reliable their employment is.
Addressing these systemic societal barriers that devalue women’s economic contributions are areas in which we are seeing strong success, with over two-thirds of the projects tackling such barriers showing positive impacts on women’s’ empowerment. We see particularly strong changes around women’s own beliefs and attitudes, for example in joining community groups, taking leadership positions, and raising community awareness about care work or their rights around land ownership.
Working with coalitions of women and supporting women’s collective action were found to be successful approaches. Where we incorporated influencing components, we also saw positive impact.
Climate Justice
Our climate justice work so far has focused on building individual and community resilience. We evaluated 28 of these projects, most in rural settings in Africa and Asia.
Findings: The evaluations revealed that Oxfam’s climate-related work has evolved over the past decade. We have moved away from focusing on diversifying livelihoods, and towards strategies that focus on bringing community members together in groups to engage in collective action and push for change to climate policies and financing.
All projects that focused on achieving climate policy change had a positive impact. Our work which sought to build community or individual resilience was also successful – with 85% of projects leading to people being better able to cope with climate shocks and stresses. However, projects that sought to increase household income or wealth remained a challenge, with only 30% of these projects having long term impact.
What we Learnt: One approach to climate justice is to support sustainable and climate-resilient livelihoods. We worked with people to improve skills, alter practices, access credit or diversify crops. However, similarly to the findings under Valuing Women’s Work, we found that whilst this led to increased skills, and in half of the projects led to diversified incomes or greater production, it was insufficient to consistently increase household income. This suggests that, overall, focusing efforts on community resilience had a greater impact than a focus on individual households.
Projects that aimed to raise awareness were successful when targeting policymakers, but we saw less change in communities – possibly as many communities are already very conscious of these changes and have often already developed their own adaptation strategies.
Work supporting coalitions, networks and alliances to influence national governments was largely successful. Many project teams noted that providing support to these networks made the most significant contribution towards achieving project objectives.
Most projects had a duration of between three and five years, yet many evaluations highlight that to achieve resilience and change climate policy, longer timespans are necessary. We are therefore working with others to advocate for and pilot longer-term funding approaches.
Looking to the Future: Building on these evaluation findings we will no longer focus on increasing incomes at a household level, but instead focus on working with partners and communities on systemic solutions. The systems that govern people’s land, agriculture, food, and energy are all integral to how communities adapt – we will work with leaders from the Global South to influence these systems to become more equitable for all.’
Finally, Katrina reflects on how this kind of work is changing the way we think about Oxfam’s role:
‘Two fundamental changes in our understanding of impact have led us to again evolve, and create a new Learning and Accountability Framework. These are:
1. Who defines what should be measured? Defining success and measuring impact has historically been controlled by organisations based in the Global North, rather than by those who experience the changes we are seeking. By focusing on Western understandings of outcomes and ways of knowing, we have not adequately centred communities or in-country partners to decide what changes are meaningful to measure. Nor have we done enough to ensure that this learning is owned and used by the people closest to the changes.
2. Addressing root causes by working on systems change. Sustained change is complex, occurring through the interactions of many actors. Assessing the impact of individual projects fails to capture systemic changes. At Oxfam, we are investing in impact through broad, dynamic portfolios of national and global initiatives. Our approach to learning is therefore moving from examining standalone projects, to one that looks for signs that systems are changing.’
These changes of direction seem right and admirable, but they aren’t going to make the job of measuring impact and learning from it any easier!
Nothing will have a serious long-term impact if you do not understand local politics.
Extreme inequality unnecessary suffering!
Our leaders are more concerned with justification than learning.
When we choose to open our minds and hearts to empathy, we gain more tools for strategies that transform our current patterns and achieve success in unforeseen ways.
A change of direction is indeed admirable and makes a lot of sense. Similar to who defines what should be measured, the same applies to who defines ‘capacity gaps’. Decades and millions of $$ are invested in ‘capacity building’, which is often seen and defined through the lens of the Global North. With this mindset, we will continue doing this ad nauseam. While some of this may be needed, we do need a rethink: do we need more capacity exchange or cross-pollination rather than ‘capacity building’? We surely need more creative and dynamic approaches than the traditional western parameters.
Author
Thanks Savio, I’m increasingly a fan of asset-based approaches, where you support what is already working to grow, develop and spread. Seems more respectful, less colonial, and more likely to succeed.
This is so helpful, thanks for sharing. Environmental campaigning groups tackling the climate emergency and injustice have been criticized/excluded by some funders for not measuring their impact effectively, and yet we know that working on systems change and policy development makes it really hard.
What these findings suggest to me is that the impact is definitely there, we just need funders to invest for longer and listen to those ‘on the ground’ as to how to measure it.