Changing mindsets to measure outside the box.
Gino Bianco – Program Officer, Monitoring, Evaluation, and Reporting at the World Wildlife Fund
Eliminating deforestation from supply chains means changing a complex system which includes companies, financial institutions, and governments. Monitoring and evaluating progress in projects that target these elements is challenging because strategies and goals tend to shift quickly and so project objectives are overtaken by events.
Cause and effect is also often unclear, because other organizations and projects will also be working with these actors to achieve similar goals, so who took the action that caused the change? Or is it a combination of actions that we should be looking at? In this blog we look at “outcome harvesting” which may offer practitioners of complex supply chain interventions an alternative approach to monitoring and evaluation with more flexibility and richer details.
Instead of working forwards from an intervention towards its intended result, outcome harvesting looks at what changes have happened and tracks back to the cause.
…………………………
The logical framework (logframe for short) approach is the most common foundation for monitoring and evaluation (M&E) in a variety of projects and programs, and for good reason. When properly developed, the logframe clearly shows how project inputs lead to outputs, outputs to outcomes, and outcomes to impacts. It also summarizes how the various components of a project fit together, outlines indicators for each output and outcome, and describes how each indicator is verified. Someone unfamiliar with the project can easily glance over the logframe and quickly understand how it is supposed to work and what assumptions it relies on. The logframe approach to project M&E is so popular because it’s so, well, logical.
Like most large environmental projects, the WWF-led and Global Environment Facility (GEF)-funded “Generating Responsible Demand for Reduced-Deforestation Commodities” Project (Demand Project), used a logframe approach to guide its work. Part of the Good Growth Partnership, the Demand Project sought to encourage sustainable production by driving demand for sustainably produced beef, palm oil and soy among companies, investors, policy makers and consumers. Considering that the four-year project had six executing partners operating in three continents, and several sub-grantees and consultants, the logframe approach was a way of keeping all partners organized and working towards the same goals. The logframe and accompanying M&E strategy allowed all organizations to see which strategies were working, and which needed to be adjusted.
In the final year of the project, however, the limitations of the logframe approach became more apparent. Years of engaging companies, investors, governments, and other NGOs around deforestation and conversion issues led to unexpected changes to their policies and actions. Some companies were revising policies based on training received from the project; government agencies were requesting training on traceability tools; and investors were joining task forces on eliminating deforestation after attending workshops. The logframe indicators were mostly showing adequate project results, but important outcomes were falling through the cracks because the project was not designed to capture them. Furthermore, several indicators relied on surveys to measure impact, which were dependent on completion rates. Project partners were seeing impressive results from their efforts through observation, but they could not claim partial credit in the logframe.
To meet these challenges, I worked with project partners to pilot a complementary M&E approach known as outcome harvesting to measure impact in the final year of the project, reflecting back on the past four years. Unlike the logframe approach, outcome harvesting does not rely on predetermined outcomes against which to measure progress. Instead, it challenges project stakeholders to identify observed changes, then work backwards to understand how the project contributed to those outcomes. I was drawn to the idea of outcome harvesting because it was specifically intended for use in complex contexts where the “relationship between cause and effect is not always clear”. In addition, outcome harvesting is ideal in contexts where stakeholder strategies and goals tend to shift quickly and are often out of the control of NGOs implementing specific projects. Outcome harvesting involves six steps, but flexibility in sequencing is encouraged, depending on the context. The outcomes “harvested” through this method consist of four main components: a description of the observed change (the outcome), its significance to the project goals and objectives, how the project contributed to the outcome, and a source to verify it.
© Six key Steps in Outcome Harvesting / Image from www.betterevaluation.org
While outcome harvesting was a promising approach to M&E in the Demand Project, it took some practice for partners to adjust. I found that many stakeholders (myself included!) were stuck in the logframe mindset. Some of the outcomes we harvested were just reiterations of the planned outcomes from the logframe. This isn’t surprising – international development and environmental professionals are almost universally taught the logframe approach. It took some prompting and reflection to think outside of the box, but it eventually revealed fascinating outcomes both related to the project’s goals and outside of them.
Outcome harvesting may offer projects that seek to influence an ecosystem of organizations a complementary tool to understand the broad impact. However, there are a few caveats to keep in mind. First, outcome harvesting cannot completely replace the logframe. Besides the fact that most donors require it, the logframe helps partners understand how the project components theoretically fit together and work toward an over-arching goal. In a project as complex as the Demand Project, the logframe can act as an “anchor” to keep the outcome harvest more focused. Second, outcome harvesting is probably more effective when introduced at the start of the project. This way, stakeholders are trained to look out for unexpected outcomes along the way, rather than trying to recall outcomes they were not planning for. Finally, outcome harvesting should not be a one-time exercise for project stakeholders, but rather a continuous process. This helps to keep observed outcomes fresh, but also facilitates adaptive management of the project.
Eliminating deforestation from supply chains requires that projects create change at the organizational level among companies, financial institutions, and governments. Designing and managing projects that operate in these spaces needs a flexible approach to monitoring and evaluation to truly understand impact. Outcome harvesting, when combined with more traditional methods, offers one such solution. M&E practitioners for this type of project should continue to experiment with this and other approaches moving forward.
‘In Changing Systems through Collaborative Action (CSCA) we aim not only to transform the elements of the system but also to change the way they relate to each other, so Gino’s image of important outcomes “falling through the cracks” really spoke to me. So much complex system change is unpredictable and unexpected that we would be foolish to monitor and evaluate only what we predicted would happen. Gino’s findings encourage us to look down the other end of the telescope to truly capture the richness and diversity of the changes we are seeing.’
Andrea Bina, Monitoring, Evaluation and Learning Specialist, Good Growth Partnership and UNDP Green Commodities Programme.
