Monday, March 11, 2013

Key Elements to Rigorous Program Evaluation

Purnima Menon spoke last week at Johns Hopkins Bloomberg School of Public Health on the early results of the Alive and Thrive initiative taking place in Bangladesh, Vietnam, and Ethiopia.  Dr. Menon works on the project in Bangladesh through her position as a Research Fellow at the International Food Policy Research Institute.

Alive and Thrive is a Bill and Melinda Gates Foundation supported initiative that is looking to improve infant and young child feeding practices through a combination of mass media, journalist advocacy, social mobilization, mother's forums, and home visits by front line and health care workers.

One of the topics that Dr. Menon discussed was a review she and her colleagues had done on defining the key elements to a rigorous program evaluation.  This review was essential because Alive and Thrive is not just about quantifying improvements in child health.  Alive and Thrive is also about identifying the factors that led to the success or failure of the program so that the program can be either scaled or improved.

Dr. Menon stated that there are four essential elements to rigorous program evaluation:
1) Pre and Post assessments of the intervention require a quantitative comparison with a control.  "The Double Difference Approach."
Showing that the study group who received an intervention is better off than they were before they received the intervention is not sufficient evidence.  You need an appropriate counterfactual group (a control group) that did not receive the intervention in order to show that the intervention works.

One example of where this is important is in the Alive and Thrive's finding a dramatic reduction in child stunting within the first few years.  The double difference approach is needed to determine whether or not this reduction is due to: 1) the intervention, or 2) improvements in food supply availability.

2) Rigor in Design
Randomization where possible is the gold standard in research design.  Randomization reduces chances of systematic bias.

3) Rigor in Measurement
The right indicators must be measured.  Data need to be collected on immediate drivers of the outcome, as well as contextual factors.  Finally, program exposures need to be captured.  That is, intervention efficacy is reliant on exposure of the population to the intervention.  A quantitative analysis of that exposure is needed to determine whether failures occur due to an ineffective intervention or failure to reach the target population.

4) Theory Driven Process Evaluation
When assessing whether or not an intervention works, you also need a theoretical framework for the mechanism by which the intervention works.  If you can evaluate the mechanism simultaneous to evaluating the outcome, you can the "why?" of whether or not the intervention was successful.  One recent example is from a study on HIV infections in women provided with pre-exposure prophalyaxis (PrEP) to prevent infection.  In the study, enrolled women claimed to be taking PrEP drugs and very few returned unused drugs.  However, blood work showed that the compliance among participants was actually low.  By including measurements of blood work, the researchers were able to identify the failure in the study.  Specifically, the researchers identified human behaviour, not drug efficacy, as the barrier to successful PrEP implementation.

Dr. Menon's overarching theme for this portion of her talk was that effort needs to be spent on program evaluation.  Furthermore, that effort should be just as rigorous (if not more so) as effort devoted to defining and implementing the programs in the first place.

No comments: