Monday, March 11, 2013

Key Elements to Rigorous Program Evaluation

Purnima Menon spoke last week at Johns Hopkins Bloomberg School of Public Health on the early results of the Alive and Thrive initiative taking place in Bangladesh, Vietnam, and Ethiopia.  Dr. Menon works on the project in Bangladesh through her position as a Research Fellow at the International Food Policy Research Institute.

Alive and Thrive is a Bill and Melinda Gates Foundation supported initiative that is looking to improve infant and young child feeding practices through a combination of mass media, journalist advocacy, social mobilization, mother's forums, and home visits by front line and health care workers.

One of the topics that Dr. Menon discussed was a review she and her colleagues had done on defining the key elements to a rigorous program evaluation.  This review was essential because Alive and Thrive is not just about quantifying improvements in child health.  Alive and Thrive is also about identifying the factors that led to the success or failure of the program so that the program can be either scaled or improved.

Dr. Menon stated that there are four essential elements to rigorous program evaluation:
1) Pre and Post assessments of the intervention require a quantitative comparison with a control.  "The Double Difference Approach."
Showing that the study group who received an intervention is better off than they were before they received the intervention is not sufficient evidence.  You need an appropriate counterfactual group (a control group) that did not receive the intervention in order to show that the intervention works.

One example of where this is important is in the Alive and Thrive's finding a dramatic reduction in child stunting within the first few years.  The double difference approach is needed to determine whether or not this reduction is due to: 1) the intervention, or 2) improvements in food supply availability.

2) Rigor in Design
Randomization where possible is the gold standard in research design.  Randomization reduces chances of systematic bias.

3) Rigor in Measurement
The right indicators must be measured.  Data need to be collected on immediate drivers of the outcome, as well as contextual factors.  Finally, program exposures need to be captured.  That is, intervention efficacy is reliant on exposure of the population to the intervention.  A quantitative analysis of that exposure is needed to determine whether failures occur due to an ineffective intervention or failure to reach the target population.

4) Theory Driven Process Evaluation
When assessing whether or not an intervention works, you also need a theoretical framework for the mechanism by which the intervention works.  If you can evaluate the mechanism simultaneous to evaluating the outcome, you can the "why?" of whether or not the intervention was successful.  One recent example is from a study on HIV infections in women provided with pre-exposure prophalyaxis (PrEP) to prevent infection.  In the study, enrolled women claimed to be taking PrEP drugs and very few returned unused drugs.  However, blood work showed that the compliance among participants was actually low.  By including measurements of blood work, the researchers were able to identify the failure in the study.  Specifically, the researchers identified human behaviour, not drug efficacy, as the barrier to successful PrEP implementation.

Dr. Menon's overarching theme for this portion of her talk was that effort needs to be spent on program evaluation.  Furthermore, that effort should be just as rigorous (if not more so) as effort devoted to defining and implementing the programs in the first place.

Monday, March 4, 2013

Slum upgrading

A Cochrane review on the impact of slum upgrading was published last month, looking specifically at the impact of improving physical environments and infrastructure. The most common interventions were improvements in water supply, sanitation, drainage, roads, paved footpaths.

The main finding is a reduction in incidence of diarrhoea due to improved water connections, with mixed results on parasitic infections or other communicable diseases and insufficient evidence on many other impacts. They also call for involvement of independent researchers early on in implementation (since most studies found were post-hoc evaluations) and for more studies to include process evaluations and qualitative evidence.

Find a great summary of it here.

Friday, March 1, 2013

Sequestration Day

Today, the spending cuts across the United States federal government start to take place in the well publicized process of sequestration.  My understanding is that the two political parties in control of the executive and legislative branches agreed that the United States budget needs an overhaul, disagreed on what budget priorities should be, and so agreed that they needed a stick to force them into some sort of agreement.  That stick was sequestration, an across the board spending reductions of approximately $85 billion in 2013. Between 2013 and 2021, sequestration is supposed to reduce national deficit by $1.2 trillion in spending and interest payments.  "Across the board" refers to the concept that spending reductions need to be evenly distributed at the "program, project, and activity level".  The National Park Service can't, for example, close down the Washington Monument while hiring new people in the Everglades.  Or maybe they can, I'm no sequestration expert.  But anyway, apparently the stick to force agreement was not threatening enough for either side.  And so Congress went home yesterday without reaching an agreement.  Side note: I don't understand why Congress doesn't have to work on Friday.  Are they starting sequestration-induced furloughs now?

So why am I writing about sequestration for a blog devoted to global water and sanitation issues?  Because reduced U.S. government spending under the sequestration impacts the research and development projects in the WASH sector.  First, there's cuts in foreign aid.  Approximately 5.3% cost reductions in foreign aid for FY2013, followed by a decade of reductions estimated at $50 billion.  $50 billion is, approximately, what the U.S. spends on total foreign aid every year.  Including diplomacy.  The money goes to $1.5 billion line item to support the USAID (big proponents of WASH funding), $7.9 billion for the Global Health Initiative, $470 million toward building resiliency in the face of Global Climate Change, and $1.3 billion in development in poorest countries.  Reduce those numbers by approximately 5% this year and you have an idea about international impacts of sequestration on foreign aid.

Second, there's cuts to U.S. science funding.  I've heard anecdotally that NIH and NSF, the top two research funding agencies  handle cuts differently.  The NIH attempts to keep the number of research grants consistent at the expense of the funding size of each grant.    I don't know if that's true.  But what I have seen is that the sequester means a reduction in NIH's budget of 5.1%, corresponding to approximately $1.6 billion.  That's huge.  That's over 3000 years of R01 grant funding (at approximately $500k per year).  I just divided those numbers, so that's probably an overestimate (no indirect costs, NIH admin support, etc.).  And also, NIH can't just cut R01 funding (remember the "across the boards" thing from earlier).  But you get the idea.  That's huge.  And NIH is the single largest source of global research and development funds, providing an estimated 62% of total global basic research and development.  That same link suggests the NIH is providing research funds to 90 countries around the world.  So a 5.1% reduction in NIH spending from sequestration will reduce total global health basic R&D by 3.1%.  Did I mention how huge this is?

I've heard (and just found this article that says) that the NSF keeps the size of the grants, and cuts the number of grants.  That article is estimating that NSF will award 1000 fewer grants under sequestration this year.  That's hundreds of projects and careers set back by lack of funding.

So I was hoping to write an in depth analysis about the impacts of sequestration on international WASH research, and instead provided a really rough overview.  But that's mostly because there remains a lot of uncertainty about what sequestration will mean.  In the meantime, if you are an American citizen, write your local representative asking them to get to work coming up with a more targeted approach to fixing the U.S. Budget mess.  They listen.  They might respond with a form letter that clearly misses the point of what you were trying to say.  But they listen.