Payment by Results take 2: what I learned from the response to last month’s rant

A couple of weeks ago I posted a fairly polemical piece about the hype around ‘payment by results’, which prompted quite a response, including a piece by CGD’s Nancy Birdsall and William Savedoff, and an excellent set of comments from a bunch of people who are much more on top of the issue than I am (not difficult, I know).

Nancy argued that the problem with PbR is that it is mostly a severely bastardized version of CGD’s original Cash on Delivery proposal:

‘PbR programs tend to create “deliverables” defined by donors, to be delivered on a preset schedule, which invites creation of “plans” and “result chains” and fixed implementation schedules. That, in turn, diverts donor attention from outcomes to inputs, indulging donor impatience and discouraging the kinds of local initiative and innovation (e.g., Problem-Driven Iterative Adaptation)  that are ultimately the best guarantee of sustained progress.’

Which raises an interesting point that the aid biz seldom thinks about – any clever idea is likely to be mutilated by institutional practice, culture, incentives etc on the way to being implemented. Is half a good idea still good, or is it likely to be even worse than what it is replacing?

What emerged from the comments thread is that there is a lot of experimentation going on, generating a growing understanding of do’s and don’ts, and a few surprises along the way. Here are some of the themes that emerged from the discussion on the blog, and within Oxfam.

Could PbR encourage downward accountability to poor people and communities rather than just upward accountability to donors?

Michael O’Donnell, BOND: ‘Tantalisingly, at a meeting hosted by WorldVision last week, we heard of communities who were involved in verification of results from PbR projects feeling that NGOs were much more accountable to them in that PbR project than in other work. There is scope to do this in a clever way to promote agendas like adaptive management, downwards accountability – but it takes a lot of thought.’

Paul Harvey, ODI:  ‘Are there any examples of payments based on customer satisfaction type ratings? So, in humanitarian aid, you could have a bonus for agencies where disaster-affected people rate the aid provided on measures such as whether it was fairly targeted, delivered with respect to dignity, consulted fully with local people, timely, adequate and appropriate.’

Does PbR encourage learning and better Monitoring and Evaluation?

Michael O’Donnell, BOND talked of ‘the positives that we now hear from organisations implementing PbR. The focus on results is helping improve M&E and performance, but within the narrow scope of whatever results are defined as triggering payments. However, those benefits may not spill over into wider M&E systems and learning beyond the project.’

Oxfam’s Francesco Rigamonti, who is running a huge (£20m) DFID PbR contract in Eastern DRC and Kenya on Water, Sanitation and Hygiene (WASH) is even more positive ‘PBR allowed us to strengthen our M&E and make it more rigorous and this can have positive effects on other programmes implemented with other funding mechanisms.’

But Francesco also sees dangers that PbR will push agencies into ‘counting what can be counted’ rather than ‘counting what counts’. Francesco wrote a nice blog on his experience of PbR, which I had missed.

The PbR contract with DFID has an interesting mix of short term and longer term indicators: ‘In the first phase we had to deliver two or more WASH services to about 850,000 people. Starting from today we have entered a new phase in which our attention will be solely focused on ensuring sustainability and sustained behavioural change for the beneficiaries reached so far. DFID will stop paying us every quarter for the number of people with access to a latrine or safe and clean water.  From now on we will be paid once per year, and the actual level of payment will be based on a survey showing how many people continue to have access to water and healthy sanitation and show signs of sustained behaviour change.’

PbR is a rapidly expanding and diverse field

Donald Menzies of DFID stressed the diversity the sectors in which PbR is now operating, including low carbon energy, girls’ education and agriculture.

Risk of crowding out smaller players

According to Oxfam’s Helen Bushell: ‘We are already seeing a risk of crowding out. At a meeting in Whitehall 2 weeks ago for an upcoming £100 million plus call (WASH 2020 Challenge programme) it was surprising how few agencies were in the room. The combination of the financial risk transfer, cash flow implications, management costs and risk management itself and the costs of learning and adapting are arguably beyond the risk appetite of many agencies. At a time when DFID is looking to promote southern civil society, and advance small and medium NGOs this funding model would seem to be working the other way.’

A Mixed Bag for NGOs?

Oxfam’s Head of Programme Funding Tom Winslow emailed: ‘While I agree with many of the criticisms that you’ve identified in your latest blog, the evidence from our own experience in Oxfam is that it does produce some positive results – greater flexibility in programme spend to achieve outputs (which programme teams really value), greater ability to cover the full costs of implementation (because we charge the donor a price rather than submit a budget), much greater incentive to spend and deliver on time (which can only benefit people in poverty), etc.’

Based on Oxfam’s experience, Emma Feeny pointed to a useful checklist for NGOs considering implementing a Water, Sanitation and Hygiene (WASH) programme on a PbR basis.

My takeaways?

I still think the hype curve is a useful construct for PbR and any other aid fad, but we seem to have all sections of the curve happening at once: the snakeoil salesmen are out there over-selling; the bah humbug types like me are pouring cold water on it; and loads of experimentation and learning is already propelling us towards a more realistic grasp of where/when PbR might be useful and how it needs to evolve (the plateau of productivity), much faster than the comparable curve for, say, microfinance.

What have I missed?

And here’s today’s vlog – lots of advice on my first one last week (thanks to all). Result: I will do these as a supplementary to original thinkpieces (not book reviews or guest posts, obvs) and don’t worry, they are not going to replace the written version. Any more feedback v welcome – done like this, they don’t take much time, but I’m still not sure how much they add to text blogging.