How realtime evaluation can sharpen our work in fragile states

Pity the poor development worker. All the signs are that their future lies in working in ‘fragile and conflict-affected states’ (FRACAS) – the DRCs, Somalias, Afghanistans and Haitis. As more stable countries grow and lift their people out of poverty, that’s where an increasing percentage of the world’s poor people will live. And (not unconnected) they are the hardest places to get anything done – states that are unwilling, unable or both; dilapidated infrastructure; social and organized violence; enfeebled institutions.

So NGOs and aid donors have their work cut out, and are going to need new ways of working to get results. Last week I got a fascinating briefing about one thing Oxfam is trying out in its ‘Within and Without the State’ (WWS) programme, which aims to help strengthen civil society capacity and social contracts between citizen and state in Yemen, South Sudan, OPTI (Occupied Palestinian Territories/Israel) and Afghanistan.

The experiment is realtime evaluation (RTE). The experience of long term development work in FRACAS bears a lot of similarities to emergencies work – rapidly changing contexts, responding to the unexpected, thinking on your feet etc. The rather cumbersome process of monitoring and evaluation, often at the end of a project, doesn’t help much. So the WWS team decided to borrow and adapt humanitarian RTEs and try them out in South Sudan.

Here’s the summary from the internal guidance note (RTE methodology for WWS_South Sudan_Mar13 – keep clicking).

What is a Real Time Evaluation (RTE)? A RTE is an internal rapid review usually carried out early on in a humanitarian response (usually between six weeks to two months after the onset of an emergency) in order to gauge effectiveness, and to adjust or correct the manner in which the project is being carried out. It should be seen as “a snap-shot in time” or a time to “step back and reflect.” Real time evaluation is closer to monitoring, getting quick feedback on operational performance, and identifying systemic issues.

RTE is a learning exercise:

• Carried out by internal evaluators who are not involved with the project
• Addresses all staff involved in the project both at head office and the field and with implementing partners.
• Interactive and flexible.
• Uses traditional evaluation tools such as interviews, focus groups and observation.
• Rarely takes more than 2 weeks – usually 8-10 days (a week in the field and a couple of days in the country office)
• Findings are shared with staff before leaving the field, preferably during a “day of reflection” workshop.
• Has more emphasis on lessons identified and a small number of action points, with a heavy emphasis on how these will be taken forward.
• It is not a replacement for an impact evaluation or a technical review that is usually held at a midpoint in the project life or before closure.
• A draft report should ideally be completed within a week of the RTE and sent for comments
• The report should be no more than 15 pages, plus annexes

The experience in South Sudan was pretty positive – the focus on learning and adapting meant people felt less defensive, and more open to acknowledging problems/challenges and finding ways round them. Practical steps that emerged from the RTE included deciding to introduce community scorecards to improve WWS’ accountability, developing ways to measure improved levels of trust between citizens and state, and making sure a key staff person working on community engagement did not spend his whole time in the office covering for high levels of staff turnover. Most of the recommendations were implemented within weeks of the RTE.

Some wider points:

The RTE preceded the political meltdown in South Sudan, raising the issue of whether they should be scheduled or reactive. Can a small number of people in the programme be given the ability (and supporting resources) to trigger an RTE in response to a major contextual shift?

Staff appreciated the chance to stand back from the day to day chaos of working in a fragile state, and reflect on the processes and impacts to date, but can we make this a continuous part of the day job, rather than a response to the arrival of the evaluator from HQ?

Doing it fast really helps – got to avoid this becoming a long drawn out affair. But there are costs to keeping it quick: it would have been good to have someone from another FRACAS involved, and there could have been more direct engagement with communities. Maybe as prep ask each member of staff and partner involved in the RTE to go and sit down in advance with a community and pick their brains?

And the funders? NGOs often say ‘oh sure, we want to be agile, but the donors won’t wear it – they’ll make us stick to the logframe’. Not so. DFID, which funds the WWS project, has supported the RTE and told us they want to see new and exciting ideas (rather than people whinging about how linear programming doesn’t work). They’ve been happy for us to adapt indicators and targets.

All this fits nicely with the work I’ve been doing on how we can use a ‘power and change cycle’ to guide our work in complex environments – using periodic, low cost sense checks to assess progress and adjust course as we go. V encouraging.

Photo credits: Crispin Hughes