How can evaluation promote better results?

Published on Thursday, 14 March 2013

Published by Vinod Thomas on Thursday, 14 March 2013

In development, as in matters of health, prevention is better than cure. Had policymakers acted boldly to avert well-recognized economic imbalances before 2008, the financial crisis may have been avoided. Conversely, bold disaster preparedness measures taken in cyclone-prone Bangladesh over the past decades—such as early warning systems—have significantly reduced death tolls during the monsoon season.

Both examples are cited in Multilateral Banks and the Development Process: Vital Links in the Results Chain. It was launched in February at a knowledge-sharing event that the Independent Evaluation Department co-hosted with the Asia Pacific Evaluation Association at the University of the Philippines’ National College of Public Administration and Governance. In writing the book with Xubei Luo, senior economist in the Independent Evaluation Group, World Bank, we drew on evaluation findings that seemed especially valuable for decision makers and practitioners. 

When looking at the results chain of the book’s title, it appears to be a fairly straightforward process. Evaluation considers intervention goals, assesses their adequacy, and tries to measure the actual effects. But it’s rather more complicated than that: which results should we focus on, how should we measure them, and how can we use the findings to improve them?

Eight crucial evaluation lessons help answer these questions; the first two focus on identifying results.

1. Let not the urgent divert the important—That is, do not let attention to immediate concerns crowd out what’s most important

Dramatic results can be attained where long-term prevention is emphasized in response to natural disasters. In the case of Bangladesh again: while a powerful storm in November 1970 killed 300,000 people, one of similar magnitude in May 1997 killed 188.

2. Connect links that strengthen results—In other words, development gains are impressive when interventions capitalize on vital links among related areas. 

In health, better education improves the outcomes of interventions. Better-educated women are more likely to use disease-prevention measures such as vaccines and mosquito nets.

The next two lessons look at how to measure results.

3. Composite indicators can mislead—These appeal in their simplicity (particularly GDP as a measure), but whether an indicator delivers what it aims to depends on the soundness of its premises.

Arbitrary clustering and weights, and converting ratings to rankings in surveys can obscure findings. In the World Bank’s Doing Business survey: the Philippines ranked 138th in the 2013 survey, way below Malaysia in 12th place. The flaw in this measure lies in its partial focus only on the cost to business of having the regulation, ignoring any benefit to society of having the regulation. 

4. Go from averages to targeted sectors—Focusing on averages can hide vulnerable population segments. Some measures across major segments are essential for policy responses. 

In Indonesia, the Philippines, and Viet Nam there is a three to fourfold difference in education performance between national averages and the poorest quintiles. 

Director General Vinod Thomas with IED staff and commentators on his book, Social Watch president and retired UP NCPAG professor Leonor Briones, UP School of Economics professor Rupert Alonzo, and Asia Pacific Evaluation Association vice president Romeo Santos

5. Align intermediate with final goals—Many promising intermediate development outcomes are necessary, but are not sufficient for achieving final goals.The remaining four lessons deal with how findings can be used to improve development effectiveness.

Education interventions can improve student enrollment. But the ultimate goals of education are learning and improving employment opportunities, and measures are often missing that can turn better enrollment into improved learning.

6. New challenges require shifts in direction—And particularly for the rapidly increasing challenges from climate change, migration to coastal cities, and declining water quality: all call for new approaches to water scarcity.

7. Seize opportunities—Win-win interventions are essential when there are difficult trade-offs, such as environmental protection and promoting economic growth.

Programs to rehabilitate mangrove forests and coral reefs protect against storms and shoreline erosion, as well as provide livelihoods through sustainable lumber, firewood, and charcoal provision, and through fishing industries, tourism, and other sectors.

8. Tarry not for timing is almost everything—It is crucial to learn faster what works and what doesn’t, to focus on results at the right time, and to link evaluation findings to development actions.

In climate change, we’re already witnessing an alarming rise in the number of intense hydrometerological events, up fivefold from the 1970s to the last decade.

Evaluation can make a difference but it must provide decision makers timely information, consider changing circumstances, and encourage fresh thinking.