Just how effective is the Millennium Village Project at reducing child deaths?

•  A version of this post appeared on the blog "A View From the Cave." The views expressed are the author's own.

A study published in The Lancet says the Millennium Village Project (MVP) is responsible for a decline in child mortality three times faster than comparable villages that are not a part of the intervention. The authors say the positive gains are the result of a successful program. However, critics have responded to the study citing serious errors in the analysis.

The MVP was established as a part of the Millennium Development Goals, a set of global targets aimed at reducing poverty and improving the quality of life for people in the world’s poorest nations. Columbia professor Jeffrey Sachs lead the development of the MVP based on the theory that increasing the level of spending per person through a comprehensive set of interventions will help lift people out of poverty.

The MVP is still a pilot program and runs in 9 different villages across Africa. The recent study shares data from the first three years of the MVP. According to the researchers, the under-five death rates in control villages declined an average of 2.6 percent each year over the course of a decade. The MVs averaged of 7.8 percent rate of decline per year. The data published in The Lancet study (click here) show promise for the success of the MVP.

Soon after publication, a few individuals raised some concerns about the study’s findings. An article in Nature quoted researcher Michael Clemens of the Center for Global Development. Using figures in the paper, Clemens calculated that the study authors can be confident only that the annual rate of decline for child mortality in the Millennium Villages lies between 1.4 percent and 14.3 percent. “If you claim to triple rates of decline you must have the evidence to back this up,” he said.

For Clemens, this means that the study’s claim that, “the average annual rate of reduction of mortality in children younger than 5 years of age was three-times faster in Millennium Village sites than in the most recent 10-year national rural trends” is not true.

Further complicating the issue is that the evaluation is done by the people who implement the MVP. Clemens and others have criticized this decision in previous studies. For critics, a new MVP established in Ghana will be evaluated by DfID (the British Department for International Development). Though it is likely all parties will not be satisfied after the independent evaluation, an outside perspective on the project in cooperation with the MVP will quiet some of the criticisms.

Lee Crawfurd further points out the observed differences in 10 of the 18 indicators in the study are statistically insignificant. For example, the increase in the asset-based wealth index for households are nearly identical between the MVP and comparison villages.

Further, Matt Collin analyzes the data from the MVP study and compares it to recent country-level data on under-five mortality published in The Lancet just a few days prior to the MVP study. (See his chart here.)

He determines that the rate of decline in the MVPs are on pace with national averages over the same period. An even closer look at the MVP study by Gabriel Demombynes, author of the national averages study, and Espen Beer Prydz in the World Bank blog uncovers some serious errors by the MVP authors.

In short, the comparison data should be calculated from the midpoint of the observation, rather than the end.

The shift to the middle four years for the MVs and the comparison villages yields very different results. The MVPs are slightly behind national averages for that period, but are so close that it is statistically insignificant. In other words the MVs are reducing under-five mortality at the same rate as the rest of the country.

The discussion is important, say the critics, because of the popularity and cost of the MVP. According to the study, it costs $116 per person per year. For the 30,000 people in the MVs that is an annual cost of $3.48 million per year. “Diverting scarce resources towards this project's approach means that other, competing uses of that money will not occur. Such mass diversion of resources can only be done responsibly when it is shown that the new use of the resources does more to end poverty than the alternative use of the resources,” says Clemens. “The burden of proof lies on the project to demonstrate that the diversion does more to end poverty than other uses of that money.”

This morning, more than a week since the paper’s release, the authors of the paper issued a correction saying, “We thank Demombynes and others for pointing out these corrections, and as noted we have withdrawn the erroneous claims as a result.  To improve the performance of the project going forward, we are inviting international public health experts along with critics of the project to join an independent expert committee to review the data collection and help improve its accuracy.”