Rethinking monitoring, evaluation and learning in complex systems

June 8, 2022

     By Duncan Green     

Two interesting recent posts on Adaptive Management, complexity etc, which the authors have kindly allowed me to repost here. First up is Søren Vester Haldrup, from UNDP’s Strategic Innovation Unit, wrestling with the issue of measurement and learning. Original post here. Tomorrow Tom Aston provides a great overview of where we’ve got to on adaptive management.

Two years ago, we launched a series of strategic investments focused on helping UNDP and our partners figure out how to better understand and tackle complex systems challenges. We did this with generous support from the Danish Ministry of Foreign Affairs.

We call these investments Deep Demonstrations, as they are meant to show (to both UNDP and our partners) that it is possible to do development differently when facing complex problems and high levels of uncertainty. In these Demonstrations we have supported colleagues in countries across the world to apply systems thinking and portfolio approaches to challenges such as the green transition and the future of work.

In this process, we have learned that using systems and portfolio approaches requires very different ways of doing monitoring, evaluation, and results measurement. This is rooted in the realization that we cannot hope to transform complex systems through business-as-usual approaches founded on linear project-based modes of planning, management, and evaluation. UNDP’s new Strategic Plan emphasizes this and also calls for a shift in how we do M&E.

We therefore need to develop new ways of doing monitoring and evaluation (M&E) that are coherent with the complex nature of the challenges facing the world today. When it comes to M&E, I think at least four issues are worth noting:

  • Need to learn and adapt: Because we don’t know up front how to best help solve complex problems such as transitioning our economy to a circular model, we need to continuously learn and adapt what we do based on learning.
  • Adopt longer time-horizons: We need to better deal with the fact that it takes a long time for substantive change (higher-level results) to materialize, and that we do not necessarily know up-front what such change will look like. This makes it difficult to know if we are on track and whether we should do anything differently.
  • Capture impact in the aggregate: We cannot evaluate individual interventions in isolation because we usually tackle systems challenges through portfolios of interconnected interventions.
  • Focus on contribution over attribution: We should focus on capturing our contribution to bigger change processes rather than seek to directly attribute change to our own work. Coe and Schlangen explain this well: 

In reality, contribution is not singular and additive; instead multiple interacting causes make an effect more likely. Recognizing this, we should shift the lens from the “amount” of contribution a single actor makes to an understanding of the typologies of the different actors and how they combine to contribute to change.

Over the past 12 months I have spoken with many people and organizations across the world who are also trying to tackle or measure complex systems challenges and it turns out that most of them are grappling with these same challenges.

Some are much further ahead than UNDP while others are just beginning their journey.

For instance, the Bill and Melinda Gates Foundation is investing in building an evidence-base around how to document systems change in areas such as food and agriculture and Co-Impact has developed a learning, measurement, and evaluation guidebook for systems change.

Meanwhile, the Cynefin Centre and Climate-KIC offer useful thoughts on developing transformative theories of change for complex systems, while the Small Foundation has developed a framework for measuring and managing impact networks.

Blue Marble Evaluation is rethinking the role and approach of evaluators when it comes to global systems change, and organizations such as UNFPA and the Open Government Partnership are deploying developmental evaluation approaches to help them continuously learn and adapt in the wake of complexity.

Furthermore, the adaptive management community has built a solid evidence base and practice, while a variety of publications such as CEDIL’s Methods Briefs and work by Aston and Colnar discuss complexity appropriate evaluation methods. Lastly, outfits such as the Transformative Innovation Policy Consortium, the Rockwool Foundation’s System Innovation Initiative, and FSG’s Water of Systems Change work offer useful conceptual frameworks for thinking about what to measure when documenting systems change.

Despite encouraging progress within and beyond UNDP, our experience over the past two years has made us painfully aware that our current processes, incentives, and ways of working go against the grain of a stronger focus on systems, learning, and adaptation.

We hear this again and again from colleagues in countries such as Bolivia, Ghana, Palestine, and Serbia who are leading systems and portfolio focused work in our Deep Demonstrations.

For instance, M&E is often seen primarily as an accountability (compliance and reporting) function rather than also being about generating timely and useful learning (Toby Lowe from the Centre for Public Impact discusses this challenge in his piece “Made to measure”).

Similarly, our project management systems and procedures tend to be rigid and lock in programming rather than make it flexible.

Furthermore, structural and organizational incentives in UNDP (and in other organizations) discourage learning and higher-level results measurement in the aggregate (across projects in a UNDP country office), because financing and M&E resourcing is usually project-based (each with a separate donor or source of funding). This tends to lead to activity and output-level reporting within projects rather than a view on how combinations of projects collectively contribute to change (how they add up to more than the sum of the parts).

Join us in the Sandbox

UNDP has set up an “M&E Sandbox” to nurture and learn from innovative M&E efforts that we hope can help address these challenges.

The Sandbox was originally intended as an internal (corporate) space for experimentation to support M&E innovations already emerging across UNDP. However, we decided to progressively open-up the space for others to join as we experienced the strong appetite among partners for this type of community.

In the M&E Sandbox we collectively explore approaches that privilege continuous learning and adaptation, see learning as a result in itself, take a systems view, and measure progress and aggregate results across a portfolio.

The Sandbox is a structured process of exploration where UNDP teams and variety of partners can get support for, collaboratively explore and design, and test approaches that are more suitable for a portfolio and systems programming logic. Under this initiative we will explore questions such as:

  1. How do we document / measure change in complex real-world systems?
  2. How do we measure progress and results in the aggregate across a portfolio of our own interventions or across what many different actors are doing? How do we know if change is really occurring and whether we are impacting the intended beneficiaries?
  3. What capabilities, tools and approaches are useful for this type of work? What counts as good (enough) evidence and data?
  4. How do we design and implement M&E frameworks and practices that allow us and others to continuously learn from, adapt, and accelerate our efforts to transform complex real-world systems? How do we track the frequency whereby our portfolios adapt and (re)generate interventions?
  5. How do we design and implement accountability frameworks, metrics or practices that allow for emergence, flexibility, and adaptation, and that shifts focus from individual interventions as the primary unit for accountability towards portfolios? How do we ensure these recognize learning as a result in itself?

This global initiative is open for everyone interested in pushing the boundaries on M&E and we are actively looking for allies, collaborators, and partners.

Organizations, groups, and individuals are free to join the initiative with varying degrees of involvement depending on their needs, bandwidth, and interests.

A range of organizations have already expressed interest in being part of this initiative, including UN agencies, bilateral donor agencies, philanthropic foundations, as well as civil society and private sector outfits.

In the coming months we will launch a series of events to kick-off this effort. Please reach out to me (soren.haldrup@undp.org) if you would like to learn more or if you are interested in getting involved!

June 8, 2022
 / 
Duncan Green
 / 

Comments