The results/value for money steamroller grinds on, with aid donors demanding more attention to measurement of 
impact. At first sight that’s a good thing – who could be against achieving results and knowing whether you’ve achieved them, right? Step forward Ros Eyben, Chris Roche, Irene Guijt and Cathy Shutt, who take a more sceptical look in a new book, The Politics of Results and Evidence in International Development, with a rather Delphic subtitle – ‘playing the game to change the rules?’
The book develops the themes of the ‘Big Push Forward’ conference in April 2014, and the topics covered in one of the best debates ever on this blog – Ros and Chris in the sceptics corner took on two gung-ho DFID bigwigs, Chris Whitty and Stefan Dercon.
The critics’ view is suggested by an opening poem, Counting Guts, by P Lalitha Kumari after she attended a meeting about results in Bangalore, which includes the line ‘We need to break free of the python grip of mechanical measures.’
The book has chapters from assorted aid workers about the many negative practical and political consequences of implementing the results agenda, including one particularly harrowing account from a Palestinian Disabled People’s Organization that ‘became a stranger in our own project’ due to the demands of donors (the author’s skype presentation was the highlight of the conference).
But what’s interesting is how the authors, and the book, have moved on from initial rejection to positive engagement. Maybe a snappier title would have been ‘Dancing with Pythons’. Irene Guijt’s concluding chapter sets out their thinking on ‘how those seeking to create or maintain space for transformational development can use the results and evidence agenda to better advantage, while minimising problematic consequences’. Here’s how she summarizes the state of the debate:
‘No one disputes the need to seek evidence and understand results. Everyone wants to see clear signs of less poverty, less inequity, less conflict and more sustainability, to understand what has made this possible. Development organizations increasingly seek to understand better what works for who and why – or why not. However, disputes arise around the power dynamics that determine who decides what gets measured, how and and why. The cases in this book bear witness to the experiences of development practitioners who have felt frustrated by the results and evidence protocols and practices that have constrained their ability to pursue transformational development. Such development seeks to change power relations and structures that create and reproduce inequality, injustice and the non-fulfillment of human rights.
And yet some of these cases also recognize that the results agenda can, in theory, open up opportunities for people-centred accountability processes, or promote useful debates about value for money, or shed light on power dynamics using theory of change approaches. Some participants at the Big Push Forward event argued that greater emphasis on evidence has led to more intelligent consumption of data.’
Guijt identifies the success factors behind what you could call ‘really useful measurement’: the methods employed must be feasible, useful and rigorous, accompanied by autonomy and fairness, generate time and space for reflection on evidence of results, and agile. She goes on to explain the meaning of those rather motherhood and apple pie terms. A few excerpts:
‘Particularly detrimental is the wasted effort invested in collecting incorrect or unused results data’. She argues that by contrast, ‘soft data’ can be particularly useful, such as programme managers personally listening to the views of children and young people.
What is Rigour?: ‘While much of the rigour debate focuses on whether data is rigorous, we should focus on seeking more rigorous thought processes and method selection and use … the term ‘rigour’ needs to be reclaimed beyond narrow method-bound definitions to encompass better inclusion of less powerful voices and improved analysis of power, politics, assumptions and resource allocation’.
‘Results and evidence approaches should strongly emphasize reflection about what is known and what needs further
inquiry. Asking people to transcribe results data rather than making sense of that data is increasingly seen as an entrenched problem.’
She finishes by setting out a positive agenda of seven strategies
‘Develop political astuteness and personal agency’: ‘people’s ability to use the results and evidence agendas positively makes them activists within their organizations, and with funding agencies and partner organizations’
‘Understand dynamic political context and organizational values’: learn to advocate within your own organization
‘Identify and work with what is positive about the results and evidence agenda’: this is the meaning of the book’s subtitle ‘playing the game to change the rules?’
‘Facilitate front-line staff to speak for themselves’: always powerful as a results focus can technocratize issues and diminish the voices of those on the ground
‘Create space for learning and influence’
‘Advocate for collective action’ – the good guys need to work together to perform the kind of ju jitsu on the results agenda set out in the previous strategies
‘Take advantage of emerging opportunities’: don’t just complain; embrace new converts in the mainstream, despite their irritating habit of claiming that they of ‘doing development differently’ first……
Some previous posts on the measurement debate here, here and here