Top 5 surprising independent evaluation results

Top 5 surprising independent evaluation results

Contractors build a cable tunnel for the new switchyard at the Nurek Hydroelectric Power Plant in Tajikistan.

By Vinod Thomas

Although evaluation findings often confirm strongly held and highly intuitive views in areas like project design, sometimes there are unexpected results.

Evaluation findings often confirm strongly held and highly intuitive views. For instance, some studies show that quality of project design or investment in institutions aids project success. But sometimes evaluation also gives surprising results, as illustrated in the five cases below.

1. Simple is not always better.

Complexity, which many depict as a Christmas tree approach, has come to have a bad name. But the same way there is bad and good cholesterol in the human body, there is a distinction between bad and good complexity in development interventions. Too many bells and whistles can slow a project without clear benefits, but good complexity—for example linking to the work of partners—can improve results, as shown in an evaluation in Tajikistan where linking Japan-financed and ADB-financed flood management made the project more complex but improved impact. 

An evaluation of Asian Development Fund operations over 2005−2014 found a high share of successful agriculture and natural resources and multi-sector interventions (such as in community development or economic recovery), which typically tend to be complex. An ongoing evaluation of partnerships is finding that despite the complexities of mobilizing funds, cofinanced projects have a 12% higher success rate than sole-financed ones (controlling other factors).

Corporate goals, for example environmentally sustainable growth, could add to complexity – but do they add value? The Lanzhou bus rapid transit (BRT) project in the People’s Republic of China (PRC) with multiple goals, seems to do so. A bus-based mass transit system similar to a light rail or metro system but costing 10 times less, it also helps lower emissions by encouraging commuters to use public transportation. Supporting certification under the Clean Development Mechanism for carbon reduction, it includes 3.5- to 5-meter-wide bicycle access, and 280 existing trees into the BRT corridor.

2. The likely benefits of safeguards exceed their costs.

Development professionals sometimes find safeguards to be burdensome and costly, especially when they do not sufficiently differentiate the risks across types of projects. Indeed for an individual project, the cost may seem unnecessarily high if safeguards prompt excessive scrutiny. But damages avoided (i.e. the benefits of having the system) across projects can more than offset the cost of having safeguards in place. The costs of safeguards are easy to estimate, but the benefits are hard to quantify, although individual cases suggest potential damages avoided. For example, in the Viet Nam Song Bung 4 hydropower project, safeguards supported relocating and restoring the livelihood of the affected Co Tu minority communities. They were also key for Weinan municipal government in the PRC to mitigate impacts on biodiversity. In a North-South road program in Armenia, safeguards helped preserve archeological sites.

4. Development effectiveness does not hurt private sector profitability.

It is often said that private sector projects play by different rules. After all, private investments bring money into an institution, and that drive for profitability ought not to be constrained by concerns of greater inclusion or sustainability. However, review of projects at the International Finance Corporation and at ADB found that this concern is overstated. In a sample of 94 ADB non-sovereign projects that have been independently evaluated, a majority have both higher development effectiveness (including socio-environmental effects) and profitability, or neither. Only in a small percentage of cases do development effectiveness and profitability not have a direct correlation.

4. Project success does not always mean sectoral or thematic success.

The portfolio of projects is a solid basis to assess progress in a county or a sector. But sometimes despite good project ratings, the performance of a sector or theme can be vastly strengthened. An evaluation of regional cooperation and integration found that projects involving more than one country had an average success rate of 81%, compared to the ADB average of 61%, but underperformance at the strategic level called for recommendations to deepen and broaden involvement on this theme. By the same token, there may be grounds—despite a poorly rated portfolio—to do more and differently. An evaluation of governance noted lower than average ratings of projects, but  nevertheless it called for more involvement in this area, stressing the high potential of some of the successful interventions, and noting that success rates had increased from 44% in the 1990s to 55% in the 2000s. 

5. Evaluation ratings do not adequately account for innovation.

While the bulk of the ratings from evaluation likely supports projects that deserve to be encouraged, there nevertheless is a bias. For example, repeatedly tried projects like roads score higher on the standard evaluation criteria, even as they might fail to contribute to innovative program delivery needed under changing conditions. Innovation might be a high risk-high reward option that evaluation criteria inadequately appreciate. Doing so, in turn, would support a new technology or a new financing technique that goes beyond existing approaches. Independent evaluation recently viewed favorably a departure from the original plan and drawing in the private sector during the implementation of a Philippines energy project. It upgraded the self-assessment of sustainable to highly sustainable, recognizing market adoption in shifting to greater energy efficiency.