Because I have one foot in the LSE and one in Oxfam, I sometimes get hauled in as a research ‘user’ (makes me sound like I have a drug problem) to review research funding applications and discuss whether, if approved, the research is likely to have much impact on the real world.
I have to say, that recent experiences have not been very positive. The vast majority of proposals seem to conflate impact with research dissemination (a heroic leap of faith – changing the world one seminar at a time), or outsource impact to partners such as NGOs and thinktanks.
Of the two, the latter looks more promising, but then the funder should ask to see both evidence of genuine buy-in from the partners, and appropriate budget for the work. Bringing in a couple of NGOs as ‘bid candy’ with little money attached is unlikely to produce much impact.
There is plenty written on how to genuinely design research for impact, e.g. this chapter from a number of Oxfam colleagues on its experience, or How to Engage Policy Makers with your Research, an excellent book I reviewed recently. In brief, proposals should:
- Identify the kind(s) of impacts being sought: policy change, attitudinal shifts (public or among decision makers), implementation of existing laws and policies etc.
- Provide a stakeholder mapping of the positions of key players around those impacts – supporters, waverers and opponents.
- How the research plans to target some/all of these different individuals/groups, including during the research process itself (not just ‘who do we send the papers to once they’re published?’).
- Which messengers/intermediaries will be recruited to convey the research to the relevant targets (researchers themselves are not always the best-placed to persuade them)
- Potential ‘critical junctures’ such as crises or changes of political leadership that could open windows of opportunity for uptake, and how the research team is set up to spot and respond to them.
- Anticipated attacks/backlash against research on sensitive issues and how the researchers plan to respond
- Plans for review and adaptation of the influencing strategy
I am not arguing for proposals to indicate specific impact outcomes – most systems are way too complex for that. But an intentional plan based on asking questions on the points above would probably help researchers improve their chances of impact.
Based on the conversations I’ve been having, I also have some thoughts on what is blocking progress.
Impact is still too often seen as an annoying hoop to jump through at the funding stage (and then largely forgotten, at least until reporting at the end of the project). The incentives are largely personal/moral (‘I want to make a difference’), whereas the weight of professional incentives are around accumulating academic publications and earning the approval of peers (hence the focus on seminars).
The timeline of advocacy, with its focus on ‘dancing with the system’, jumping on unexpected windows of opportunity etc, does not mesh with the relentless but slow pressure to write and publish. An academic is likely to pay a price if they drop their current research plans to rehash prior work to take advantage of a brief policy ‘window of opportunity’.
I think there is still some residual snobbery, at least in some disciplines. You still hear terms like ‘media don’, which is not meant as a compliment. I am delighted that my friend Ha-Joon Chang is now an economics professor at SOAS, but what on earth was Cambridge University thinking not making a global public intellectual and brilliant mind into a prof, while he was there?
True, there is also some more justified concern that designing research for impact can damage the research’s objectivity/credibility – hence the desire to pull in NGOs and thinktanks as intermediaries.
But overall, this conversation still feels messy and unresolved, at least in the UK – any suggestions? Have any other countries/funders got it right?
Also, just spotted this 6 week MOOC on Research for Impact, which started this week – anyone doing it?
Many thanks for this, Duncan; very useful. Like you, I am also a ‘user’, especially since my research is focused on drugs and illicit crops. One further area to consider in designing research for impact and plans for the influencing strategy is how to manage findings that are ‘hot potatoes’ that may cause political embarrassment, bring reputational or operational risks, or create unintended consequences. I remember the case of a handful of politicians that we found involved in drug smuggling but couldn’t bring that out because they were also key intermediaries in ongoing peace talks and were key sources of employment in collapsing local economies. I wonder if you or other research practitioners may have a view on this.
Author
I was reading something recently on this Eric, but on corruption – the thinktank in question published a private and public version of the same research, to avoid giving offence to those they were working with. Any other experiences?
In one case, we decided not to publish but circulated the report to those involved — the information became useful for designing subsequent projects. In another case, we waited for prosecutors to file charges, so that we can use the charge/court papers as our source. But they never did. It also appears that it is important to have a sense of the politics behind the evidence (that Steven Rood referred to in his comment here). Thanks Duncan.
Having an ‘intentional plan based on asking questions on the points above would probably help researchers improve their chances of impact’ sounds right.
The FCDO funded ACE – Anti Corruption Evidence – programne calls this ‘nose to tail policy engagement’ meaning that questions/actions to engage policy and practitioners start from the first spark, and continue throughout the research project (and sometimes beyond).
However, while every research project has a plan, not all will get very far down the ‘pathway to impact’. I find that it is easier to then think about impact across a portfolio of research – either across an investment programme (by the funder) or a department or other unit (for researchers). All sub projects have a sensible plan. Some of these will (hopefully) demonstrate high traction and influence (and be ‘crown jewel’ impact case studies), and will help justify the whole investment/programme. The rest will be at least relevant to policy/practice questions, but may take longer to see any influence, or none. I also think that expecting researchers to be ‘policy entrepreneurs’ is often doomed to fail, though some do it well. In directed research programmes (ie thematic investments, not open calls) i think that the funder has strong incentives for getting research ‘taken up’ and so should explicitly take on this role to bridge research and policy worlds (policy entrepreneur, research wrangler, whatever) but without bending the research out of shape. It’s a specific skill/competence though, and needs a strong understanding and experience of how ‘policy’ (whatever that means) gets mashed together, the networks, the influence, as well as research (to me it is a ‘rock/paper/scissors’ of ‘evidence/know how/ know who’).
One of the challenges is that any particular piece of research may be ‘too narrow’ or ‘specific’ to garner sufficient attention/action for impact. We need to create more opportunities to recognize patterns across research that help set an agenda for evidence-based change within the appropriate system (which, also, needs its intentional plan).
The incentives for this does not necessarily lie with the researchers so need to involve intermediaries (funders, NGOs, think tanks etc.) and plan for the long haul. We have done it for enterprise development – eight years to assemble the credible evidence/patterns of success, now another eight years to push adoption! Patience is a virtue:-) Though, I think, we have substantially influenced about a quarter of the global spend in this space to date, so can be done (though we are admittedly not very controversial)!
Author
Like the idea of non-academics as pattern spotters/long term promoters Nicholas – not the usual narrative!
A nice provocative lead in there, Duncan. Something “that is difficult for a lot of academics” hits the nail on the head while being (in my opinion) unfairly judgemental on the academics themselves. It takes a lot of time and a much more collaborative system (on all sides – academics, funders, policymakers) to really build a system fit for impact.
The traditional idea of academics might think of us as lone wolves working independently in our towers. But the chances of impact in research designed and conducted by individuals is slim. And while funders might like us to think that a single piece of research can lead to impact, I’d say this could be possible but again not likely.
To me the biggest chances of impact are the product of long-term bodies of work and collaborative ways of working. So you could look at a piece of research and say “that hasn’t had much impact”, but we’d be missing the point. If it’s building a body of work and a network of connections that keep going beyond it, it’s very much part of the impact horizon. If we don’t view impact as a long game and not just a short term outcome then we’re doing everyone a disservice. The places where I have had real impact as an academic have not been in my research. But in my broader network building, my learning, all the stuff that happens before, through and after the research.
More specifically on the blog I would say that we’ve definitely moved beyond a ‘viewing impact as knowledge dissemination’ place – not everyone buys into it, but that’s okay.
The incentives are moving in the right direction (thanks REF – we don’t say that v often). But nowhere near far enough in a way that views and rewards impact commensurately or sufficiently in terms of how time-intensive impact things can be.
So we can ask “why can’t
Academics design more impactful research?” – but there are a lot of academics that would love to do a lot more and just don’t have the time and resources to do more. That can hurt. And I say this from a University that funds and invests in impact really well.
Author
Thanks Niki, really interesting reflections. At one point (not sure if it’s still happening), the Aussie agency DFAT used to ask for a long term theory of change, and then short term logframe-y type plans within it. Maybe there’s an equivalent in academia – the long term ToC would be about what you call ‘building a body of work and a network of connections that keep going beyond it’. Like it.
I once wrote a report on a sensitive topic for a donor, assuming that a copy might get out and circulate. That affected the style and content. I wasn’t sure a “private” version wouldn’t leak.
I warned the donor that was going to be my style and then extensively briefed them verbally. They seemed satisfied with the result, and I could see the effect on subsequent project designs.