The political implications of evidence-based approaches
The debate on evidence and results continues to rage. Rosalind Eyben and Chris Roche, two of the organiser’s of next April’s Big Push Forward conference on the Politics of Evidence, kick off a discussion. Tomorrow Chris Whitty, DFID’s Director of Research and Evidence and Chief Scientific Adviser, and Stefan Dercon, its Chief Economist, respond
Distinct from its more general usage of what is observed or experienced, ‘evidence’ has acquired a particular meaning relating to proof about ‘what works’, particularly through robust evidence from rigorous experimental trials. But no-one really believes that it is feasible for external development assistance to consist purely of ‘technical’ interventions. Most development workers do not see themselves as scientists in a laboratory, but more as reflective practitioners seeking to learn how to support locally generated transformative processes for greater equity and social justice. Where have these experimental approaches come from and what is at stake?
The origins and critiques of evidence-based approaches
Evidence-based approaches are pre-occupied with avoiding bias and increasing the precision of estimates of effect. In the UK they spread beyond clinical practice when the government elected in 1997 was keen to demonstrate that its decisions would not be driven by political ideology but rather by objective evidence. Evidence-based approaches became linked to value for money concerns to deliver ‘results’ as efficiently and effectively as possible, by a government recently described as ‘truth junkies’.
Yet, even within medicine, the leap from evidence-based clinical practice into evidence-based policy was challenged. A British Medical Journal article by Nick Black in 2001 drew on an extensive body of contemporary literature on policy processes to argue that policy was shaped by institutional arrangements, values and beliefs and a variety of different sources of information. Opponents of evidence-based education critiqued its positivist assumptions; its linear cause-effect thinking; and the poor understanding of the tensions between scientific and democratic control of educational practice. An OECD report in 2007 on the reasons for the uptake of evidence-based education in Sweden, the UK, Canada, the USA and Australia, noted the increasing pressure for greater accountability of expenditure and effectiveness, an explosion in the search for measurable outcomes, and demands that impacts and effectiveness be given a monetary value. The same report noted that evidence-based approaches were largely absent in OECD countries ‘less used to empirical and quantitative methodologies in the social sciences.’ The de-politicization of policy making is one of the reasons given by development researchers for its neglect in France.
In UK social policy, evidence based approaches with their ‘gold standard’ of experimental or quasi-experimental design, have been criticised as being inapplicable to complex issues.In What Works, Tony Harrison argued that evidence-based approaches can only apply in cases of individual treatment and not at the wider community level where multiple perspectives come into play and no agreement exists about the nature of the problem.This of course is the case with most development programmes, and in particular those that seek transformational change.
Evidence based approaches in development: an anti-politics firewall?
Arguably evidence-based approaches build an anti-politics firewall. Development assistance becomes a ‘technical’ best practice intervention based on rigorous objective evidence, delivering best value for money to domestic taxpayers and recipient country citizens mostly without interfering in that country’s politics. They are the latest manifestation of a certain long-standing approach to development that as Timothy Mitchell wrote in the Power of Development (J. Crush et al 1997), speaks to the sector’s ‘need to overlook its internal involvement in the places and problems it analyses and present itself instead as an external intelligence that stands outside the objects it describes’.
In the 1930’s Africa was seen as ‘a living laboratory’ to achieve improvements in the welfare of the populations . Evidence-based approaches are reviving the development as laboratory idea. In 2012 the World Bank established a Gender Innovation Lab to design ‘innovative interventions to address gender inequality and to develop rigorous research projects in order to produce evidence on what works and what does not’. Jeffrey Sachs’ Millennium Villages have been framed as ‘laboratories to lift people out of poverty’. The most well-known is the J-Pal Poverty Action Lab whose mission is to reduce poverty ‘by ensuring that policy is based on scientific evidence’.
In the absence of political debate, this approach can exacerbate the tendency to see people as subjects requiring treatment, rather than as citizens with political voice. Power silences any challenges to the technical framing of ‘the problem’, foreclosing discussion of the structural causes and consequences of inequity and how these should be tackled.To act ‘technically’ in a politically complex context can make external actors pawns of more powerful vested interests and therefore by default makes them, albeit unintentionally, political actors.
High stakes
Evidence-based technical approaches can therefore deflect attention from the centrality of power, politics and ideology in shaping society. We agree with the view of the Developmental Leadership Program that recent research suggests that the development sector should be ‘at the frontier of a narrative shift between a technical, rational, and scientific approach to development, and a recognition that politics matters; that poverty reduction is not a technical problem but requires significant social change, and that this social change is, and must be, both political and locally led.’ However this has some significant implications for external actors. We need to be self-aware to avoid disempowering others. This requires undertaking power analyses with ourselves factored in – as organisations and individuals who can make a positive or negative contribution, often inadvertently. It means engaging with a wider and more diverse group of policy actors in the state, civil society and the private sector; whenever possible, supporting debate, locally-driven problem solving, and independent research. It means avoiding overly linear project based aid modalities that demand omniscience before they have even begun.
As Michael Sandel has recently argued in his book about the moral limits to markets how we put values (and prices) on things can change their meaning, as well as change the relationship between economic actors. More information, dare we say ‘evidence’, is needed to draw some firmer conclusions about the consequences of evidence-based approaches to designing projects and assessing results. This is why the Big Push Forward is currently seeking to crowd-source more information from development practitioners about how they actually experience the ‘results’ agenda, and why we believe this issue needs more debate.
The debate on evidence and results continues to rage. Rosalind Eyben (left) and Chris Roche (right, dressed for battle), two of the organisers of 
April’s Big Push Forward conference on the Politics of Evidence, kick off a discussion. Tomorrow Chris Whitty, DFID’s Director of Research and Evidence and Chief Scientific Adviser, and Stefan Dercon, its Chief Economist, respond
Distinct from its more general usage of what is observed or experienced, ‘evidence’ has acquired a particular meaning relating to proof about ‘what works’, particularly through robust evidence from rigorous experimental trials. But no-one really believes that it is feasible for external development assistance to consist purely of ‘technical’ interventions. Most development workers do not see themselves as scientists in a laboratory, but more as reflective practitioners seeking to learn how to support locally generated transformative processes for greater equity and social justice. Where have these experimental approaches come from and what is at stake?
The origins and critiques of evidence-based approaches
Evidence-based approaches are pre-occupied with avoiding bias and increasing the precision of estimates of effect. In the UK they spread beyond clinical practice when the government elected in 1997 was keen to demonstrate that its decisions would not be driven by political ideology but rather by objective evidence. Evidence-based approaches became linked to value for money concerns to deliver ‘results’ as efficiently and effectively as possible, by a government recently described as ‘truth junkies’.
Yet, even within medicine, the leap from evidence-based clinical practice into evidence-based policy was challenged. A British Medical Journal article by Nick Black in 2001 drew on an extensive body of contemporary literature on policy processes to argue that policy was shaped by institutional arrangements, values and beliefs and a variety of different sources of information.
Opponents of evidence-based education critiqued its positivist assumptions; its linear cause-effect thinking; and the poor understanding of the tensions between scientific and democratic control of educational practice. An OECD report in 2007 on the reasons for the uptake of evidence-based education in Sweden, the UK, Canada, the USA and Australia, noted the increasing pressure for greater accountability of expenditure and effectiveness, an explosion in the search for measurable outcomes, and demands that impacts and effectiveness be given a monetary value.
The same report noted that evidence-based approaches were largely absent in OECD countries ‘less used to empirical and quantitative methodologies in the social sciences.’ The de-politicization of policy making is one of the reasons given by development researchers for its neglect in France.
In UK social policy, evidence-based approaches with their ‘gold standard’ of experimental or quasi-experimental design, have been criticised as being inapplicable to complex issues. In What Works, Tony Harrison argued that evidence-based approaches can only apply in cases of individual treatment and not at the wider community level where multiple perspectives come into play and no agreement exists about the nature of the problem.This of course is the case with most development programmes, and in particular those that seek transformational change.
Evidence based approaches in development: an anti-politics firewall?
Arguably evidence-based approaches build an anti-politics firewall. Development assistance becomes a ‘technical’ best practice intervention based on rigorous objective evidence, delivering best value for money to domestic taxpayers and recipient country citizens mostly without interfering in that country’s politics. They are the latest manifestation of a certain long-standing approach to development that as Timothy Mitchell wrote in Power of Development, speaks to the sector’s ‘need to overlook its internal involvement in the places and problems it analyses and present itself instead as an external intelligence that stands outside the objects it describes’.
In the 1930s Africa was seen as ‘a living laboratory’ to achieve improvements in the welfare of the populations. Evidence-based approaches are reviving the development-as-laboratory idea. In 2012 the World Bank established a Gender Innovation Lab to design ‘innovative interventions to address gender inequality and to develop rigorous research projects in order to produce evidence on what works and what does not’. Jeffrey Sachs’ Millennium Villages have been framed as ‘laboratories to lift people out of poverty’. The most well-known is the J-Pal Poverty Action Lab whose mission is to reduce poverty ‘by ensuring that policy is based on scientific evidence’.
In the absence of political debate, this approach can exacerbate the tendency to see people as subjects requiring treatment, rather than as citizens with political voice. Power silences any challenges to the technical framing of ‘the problem’, foreclosing discussion of the structural causes and consequences of inequity and how these should be tackled. To act ‘technically’ in a politically complex context can make external actors pawns of more powerful vested interests and therefore by default makes them, albeit unintentionally, political actors.
High stakes
Evidence-based technical approaches can therefore deflect attention from the centrality of power, politics and ideology in shaping society. We agree with the view of the Developmental Leadership Program that recent research suggests that the development sector should be ‘at the frontier of a narrative shift between a technical, rational, and scientific approach to development, and a recognition that politics matters; that poverty reduction is not a technical problem but requires significant social change, and that this social change is, and must be, both political and locally led.’
However this has some significant implications for external actors. We need to be self-aware to avoid disempowering others. This requires undertaking power analyses with ourselves factored in – as organisations and individuals who can make a positive or negative contribution, often inadvertently. It means engaging with a wider and more diverse group of policy actors in the state, civil society and the private sector; whenever possible, supporting debate, locally-driven problem solving, and independent research. It means avoiding overly linear project-based aid modalities that demand omniscience before they have even begun.
As Michael Sandel has recently argued in his book about the moral limits to markets, how we put values (and prices) on things can change their meaning, as well as change the relationship between economic actors. More information, (dare we say ‘evidence’), is needed to draw some firmer conclusions about the consequences of evidence-based approaches to designing projects and assessing results. This is why the Big Push Forward is currently seeking to crowd-source more information from development practitioners about how they actually experience the ‘results’ agenda, and why we believe this issue needs more debate.
And make sure you come back tomorrow for DFID’s counterblast