This is the last in a series of three posts on Twaweza, a fascinating NGO doing some pioneering work on accountability in East Africa, whose big navel 
gaze I attended last week. Post one covered Twaweza’s theory of change and initial evaluation results; yesterday I got onto the critique of its thinking and action to date. Today I’m digging deeper into some of the underlying issues.
Given its rethink, Twaweza is now contemplating a shift in direction – while keeping its focus on citizen agency, focus in on education (rather than try and cover education, health and water); reduce the number of partners; do more things on its own (eg research or education programming); expand successful areas such as policy and advocacy; do more experiments to uncover what works and help the organization ‘fail faster’ and so move on to new stuff.
Plenty of good ideas in there, but it also seems to me to mark an intellectual retreat from the initial commitment to finding new ways to achieve change in complex systems. I think there’s a strong case for digging deeper into complexity, rather than retreating from it. One suggestion that moves in the right direction is to set up a ‘positive deviance lab’, dedicated to detecting and then understanding examples of success in citizens’ action across East Africa.
What would digging deeper mean? First, building a deeper understanding of the system, before jumping in with any particular intervention. That requires a much closer scrutiny of power, relationships and all the forces that maintain the status quo. Twaweza doesn’t seem to have spent much time understanding why change doesn’t happen (e.g. through a ‘3i’ combination of ideas, interests and institutions), which turns out to be pretty key as its work has encountered far more inertia than originally expected.
One of the common findings of the evaluators was that where good things have occurred, it is most often down to the charismatic leadership of particular grassroots heroes, rather than systemic interventions by Twaweza or others. So why not, like the Development Leadership Program, try and understand the origins of such ‘Gandhiness’, and see if Twaweza can design programmes to strengthen it or influence future leaders?
Finally of course, critical junctures, aka windows of opportunity. Like much of the rest of Africa, Tanzania is standing on the threshold of a huge resource boom, in this case offshore gas in the South. In terms of the evolution of the wider social contract between citizen and state, it is probably up there with next year’s referendum on the new constitution. Yet neither were mentioned in the two days of debates, suggesting that even Twaweza is a bit too stuck in the Plan, and not looking around it to respond to new opportunities.
An underlying tension throughout the discussion was between Twaweza’s commitment to running experiments/measuring its impact, and its desire to find new ways to work in complex systems. I’ve never spent so much time with a group of academic evaluators and the experimental mind is fascinating: endless discussions over how to measure really subtle, difficult things like agency – signatures on a petition? Going to a meeting? Saying the right thing to a phone interviewer? Every discussion is weighed down by questions of attribution and bias.
But all this heavyweight cleverness ended up just reinforcing my sense of the limits to quantitative measurement in development. Getting to some numbers that can be properly crunched requires levels of abstraction (eg encoding the responses of hundreds of communities to identify a handful of variables suitable for a regression) and agglomeration that can only do violence to reality.
Even if you manage to extract some level of numerical relationship, understanding why activity a leads to outcome b requires ‘sending in the anthropologists’ anyway, who will doubtless (being anthropologists) come back and tell you that the abstractions mean that your results are meaningless and/or useless. Any clarity or certainty that emerges from such an exercise may be comforting, but is likely to be illusory.
To be fair, Rakesh thinks I’m caricaturing here and says ‘We have come to care about measurement deeply, not primarily because we need to cater to some donor need or academic joyride, but because we want to be more effective, we want to make a sharper difference, we want to be more accountable for results to the people we claim to serve.’ I completely understand that impatient desire to know whether your work is having impact, but I think that, at least as currently conceived by Twaweza, giving priority to impact measurement could also be a barrier to working effectively in complex systems.
Maybe Twaweza’s measurement problem (and yes I think it has one) stems from using the wrong kinds of metrics – Kate Dyer of the Accountability Tanzania programme was also in the seminar, and described its attempts to make outcome mapping the foundation for much of its learning, even using it to generate some ‘participatory numbers’ – Twaweza could do worse than draw on that work.
There was much discussion about including the evaluators in the design of new projects, but I was pretty alarmed by that too. Sure they do ask brilliant questions, but their need for numbers inevitably ‘contaminates’ (to use one of their favourite words), their grasp of how to work in complex systems. How to avoid the measurement tail wagging the programme dog? There were several dog metaphors – I (doubtless unfairly) likened some of the evaluators to dogs with a bone, letting their enthusiasm for getting stuck in to extracting numbers cloud the original intention of encouraging citizens to take matters into their own hands.
Twaweza’s growing enthusiasm for experiments also raised some concerns, especially when combined with its hunger for numbers. Is handing tens of thousands of teachers a personal ‘cash on delivery’ bonus for getting their pupils to an acceptable grade (as one Twaweza experiment is currently doing) really compatible with building agency, citizens’ voice etc? Sounds more like an incredibly crude bit of behavioural tweaking, with all sorts of potential unintended consequences for the system (think what the much less immediate incentives of school targets have done to schools in the UK and elsewhere – teaching to the test, excluding those likely to fail etc).
Another notable result of the hunger for number-based certainty is ‘Millennium Village thinking’: if we can’t shift the ecosystem at a national level, why not concentrate all our resources on a few communities? Might not even be a bad idea in this case, but interesting to see how the proposal arises from the initial mindset.
So in conclusion, (as they say): the direction that Twaweza takes over the next few years matters well beyond Tanzania’s shores. It is a global leader among NGOs, heroically intelligent, honest, innovative and yes, courageous in acknowledging failure and trying to grapple with many of the issues that bedevil development wonks everywhere – complexity, scale, what and how to measure impact. If it retreats from its grand ambitions in an understandable search for the achievable and measurable (we all want to know we’ve actually achieved some kind of impact, after all), it will have squandered the chance to do something really innovative and we will all be the poorer.
Last word to another one minute Twaweza video (unless Rakesh and his colleagues exercise their right of reply).