Complexity 101: behind the hype, what do we actually know?

November 5, 2013

     By Duncan Green     

Complexity week continues with this excellent stocktake from the ODI’s Harry Jones (who’s got a new guide out on ‘Managing Projects and Programmes in the Face of Complexity‘). Part two tomorrow.Harry colour adjusted

Seven years ago John Young, Ben Ramalingam and I decided to begin research on complexity theory and international development.  We felt there was really something of use and interest in there but that it might take some time to persuade others of that fact and more time to find out exactly what that value was: although there were already a few people working on complexity, since  then, the blogosphere has come alive with discussion of the ‘C’ word. It’s clear a large number of development professionals see value in complexity theory and international development (for example, here is Owen Barder on complexity in international development, William Easterly’s take, Ben’s blog devoted to the issue, and all the posts on Duncan Green’s blog about complexity and development).

What we don’t seem to have reached is much agreement on exactly what that value is. As we look forwards to this week’s launch of Ben’s book on complexity in development (which I have yet to read), it seems an opportune time to reflect on what we know. In that 2008 exploratory review we set out to examine whether complexity and development was a case of ‘paradigm, hype, or lens. Six years on, here’s my attempt to review the evidence on some straight questions on complexity and development, in two parts.  In this first part, I look at whether development problems are complex and why it matters; in the second part I will look at what to do about that complexity where it exists.

*N.B. to avoid disappearing up my own comments page I will use the popular simple-complicated-complex (bake a cake, make a rocket, raise a child) schema subscribed to by Stacey, Zimmerman, Snowden et al.

1) Are development problems complex?

It is undeniable that many issues, sectors, and challenges have complex characteristics. To name just two examples: on the science-led side Elinor Ostrom used her Nobel prize speech to emphasise the importance of complexity in natural resource management and spent some of her later years attempting to lay solid, shared academic foundations for the study of complex systems.

On the practitioner side, problems of governance and institutional change display the trademark characteristics of messy, unpredictable and adaptive systems.  Alan Fowler argues for the centrality of complex, civic-driven change processes; Frauke de Weijer’s experiences in Afghanistan led her to propose that we rethink change in fragile states along complexity lines; and Robert Chambers found that complexity is the most suitable paradigm to explain the bottom-up change processes he has worked with.

On the other hand, I have not seen any strong evidence to say that all problems faced in development are complex. As Charles Kenny at CGD argued recently, while the success of many development interventions is uncertain and depends strongly on context, there are a number of public investments and institutional lessons that we can say with some certainty will have positive contributions to a country’s development.

So the answer to ‘are development problems complex’ is: yes, some of them. While development problems might be more likely to be complex than others, we must judge on a case by case basis.

2) What does it mean to say development problems are complex?

For me, saying a problem is complex is akin to saying it is economic, or that it is ecological, or anthropological. Complexity works as a lens, and as an area of transdisciplinary academic and practical work that can be called a ‘science’ in itself, or a movement within existing disciplines (e.g. see Beinhocker’s book on what complexity means for economics).

Calling a problem ‘complex’ is a shorthand for saying that it has certain important aspects that we can best understand through a complexity lens, and that we can use explanatory and predictive tools generated by theory and empirical evidence to help address it. For example, social network analysis uses the theory of networks and empirical results on how they function in society to illuminate, explain and predict certain types of phenomena and behaviour.  However, challenges are often termed ‘complex’ as an excuse for inaction or failure. Complex problems and difficult problems overlap, but they are not the same thing.

Dumaguete, Philippines3) Does it matter when development problems are complex?

In a word, yes. Complex problems present challenges for the well-established instruments of public policy and administration. No matter how anyone describes ‘complexity’, I can guarantee that the reason complexity matters comes down to one or more of three challenges:

  1. 1. Distributed capacities: the knowledge and capacities required to tackle problems are spread across actors without strong, formalised institutional links.
  2. 2. Divergent goals: inherent to many problems are divergent interests, competing narratives or conflicting goals.
  3. 3. Uncertain change pathways: it is unclear how to achieve a given aim in a given context, or change processes involve significant, unpredictable forces.

In researching my working paper of 2011, I was struck by the strength of these three themes across such a broad range of bodies of knowledge. None of this is new: uncertainty and divergence have been central elements of most definitions of complexity for decades. Outside complexity science, the terms have been used to classify different types of policy problems by the likes of Hisschemöller and Hoppe (see Hoppe’s recent book on the governance of problems).

The problem of ‘distributed capacities’ is often highlighted separately or assumed to result from the other two, but it represents a strong theme in the work on systems theory, complex adaptive systems and elsewhere.  All three factors have been highlighted as central challenges for guiding change processes under the guise of ‘development as process’, or with relation to learning processes in community development as far back as 1980.

4) Why does this matter? And how much?

It renders many of our tools inappropriate. In many cases our established tools for public policy, programming, and administration are founded on ideas of concentrated capacities, agreed goals, and relatively clear or stable change pathways. When these tools are applied to complex problems, the result is typically one of two effects: formal systems either become meaningless box-ticking exercises, or they actively impede achievement of goals, putting in place perverse incentives and guaranteeing aims are not met. Evidence both can be seen in the reams of research on the log frame, or by a UN-wide evaluation on results-based management that found limited to no evidence of any utility of the system, despite the great amount of money spent.

I think this kind of idea was behind Ros Eyben et al’s ‘big push back’ (now forwards), and Andrew Natsios has argued that both symptoms emerge from the clash of the counter-bureaucracy and development, which sees implementation squeezed and neglected due to overzealous design and process requirements from agency HQs.

It is hard to quantify how much this matters. One could partially estimate this by chalking up the cost of inappropriate agency-wide reforms as a partial waste, but it is impossible to estimate on a system-wide level the value of opportunities missed due to rigid tools. Evidence on management models is rarely conclusive, and importing new ideas into public policy and programming will always need to proceed based on careful assessments of problems with existing systems, but a certain amount of ‘faith’ is required to start with a new approach (for example New Public Management spread like wildfire across the public sector without its impact having been robustly assessed beforehand).

But what principles should this new approach be based on?  Tomorrow, I look at the ‘so what’ question: what do agencies need to do differently?

November 5, 2013
Duncan Green