Olivier Ecker & Jef Leroy ã 2016 IFPRI Egypt Seminar Series: What is the Role of Agriculture & How do we Document Impact?

Please download to get full document.

View again

All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
 5
 
  1. Improving Nutrition in Egypt What is the Role of Agriculture and How do we Document Impact? Jef Leroy & Olivier Ecker IFPRI, Washington, DC IFPRI Egypt Seminar…
Share
Transcript
  • 1. Improving Nutrition in Egypt What is the Role of Agriculture and How do we Document Impact? Jef Leroy & Olivier Ecker IFPRI, Washington, DC IFPRI Egypt Seminar Series Cairo, 16 Nov 2016
  • 2. Overview • Part 1: Nutrition-sensitive agriculture (Ecker) • Part 2: Key challenges and solutions in evaluating the impact of nutrition-sensitive agricultural programs (Leroy)
  • 3. Overview • Part 1: Nutrition-sensitive agriculture (Ecker) • Part 2: Key challenges and solutions in evaluating the impact of nutrition-sensitive agricultural programs (Leroy)
  • 4. Agriculture-nutrition linkages: Theory and evidence Key message: Agricultural interventions can improve nutrition, but they do not automatically lead to improved nutrition. Agricultural programs and projects can improve diets and nutritional outcomes of farmers and their families through many different pathways: 1. Increasing farm incomes ( productivity gain) 2. Improving food consumption from own production ( food quantity and diet diversity) 3. Employment and control over household resources ( e.g. women empowerment)
  • 5. Agriculture-nutrition linkages: Theory and evidence • Growing evidence—from rigorous impact evaluations— suggest that agricultural interventions can improve (child) nutrition, if they … 1. Have specific nutrition goals and actions, and 2. Focus on women. • Many factors influence individual nutritional status (incl. diets, health, care). Agriculture interventions are part of the solution, but more is needed. – E.g., in addition to increasing incomes and improving food availability and access, behavioral change to promote adequate nutrition and health practices is needed.
  • 6. Pathways linking agriculture to nutrition Adapted from: Gillespie et al.. 2012. The Agriculture-Nutrition Disconnect in India: What Do We Know? IFPRI Discussion Paper 1187. IFPRI, Washington DC.
  • 7. Agriculture-nutrition linkages: Knowledge gaps and research needs • Evidence on the impact of agricultural interventions on nutritional outcomes is still scarce. • Existing evidence is largely based on small-scale programs in fairly controlled environments (e.g. home gardens). • Evidence from large-scale, complex agricultural interventions—especially for promotion of agricultural commercialization (“agribusiness”)—is missing so far.
  • 8. Agriculture and nutrition In summary (Lancet, 2013): • Agriculture has enormous potential to improve nutrition, but this potential is yet to be unleashed. • Lack of critical evidence of impact of agriculture on nutrition is likely due to: – Weaknesses in program design, targeting, implementation, and lack of nutrition goals and interventions, or – Poor evaluations – design, methods, indicators.
  • 9. Overview • Part 1: Nutrition-sensitive agriculture (Ecker) • Part 2: Key challenges and solutions in evaluating the impact of nutrition-sensitive agricultural programs (Leroy)
  • 10. Part 2: Key challenges and solutions 1. Challenges in evaluating the impact of nutrition-sensitive agricultural programs 2. Rigorous evaluation approach to evaluate: a) Impact b) Impact pathways, and c) Cost 3. Recommendations on how to address key challenges of carrying out sound evaluations implemented under real life conditions.
  • 11. Part 2: Key challenges and solutions 1. Challenges in evaluating the impact of nutrition-sensitive agricultural programs 2. Rigorous evaluation approach to evaluate: a) Impact b) Impact pathways, and c) Cost 3. Recommendations on how to address key challenges of carrying out sound evaluations implemented under real life conditions.
  • 12. What are some key challenges? • Complexity of nutrition-sensitive agricultural programs • Long impact pathways and time frames • Trade-off between implementation constraints and evaluation rigor
  • 13. What are some key challenges? • Complexity of nutrition-sensitive agricultural programs • Long impact pathways and time frames • Trade-off between implementation constraints and evaluation rigor
  • 14. Program complexity
  • 15. Complexity nutrition-sensitive agricultural programs • Complex in design and implementation: – Multiple goals, multiple inputs, multiple pathways of impact, multiple outcomes and impacts; – Address both the underlying and direct causes of undernutrition; – Span across different sectors (e.g. health, agriculture, education), requiring coordination and integration. • Within each program intervention potential variability in: – Delivery (quantity and quality); – Utilization; – Adherence to the program protocol.
  • 16. What are some key challenges? • Complexity of agricultural programs aimed at improving nutrition • Long impact pathways and time frames • Trade-off between implementation constraints and evaluation rigor
  • 17. Long impact pathways and time frames • Long time frames for implementation and evaluation: – Program development and implementation at desired quality – Long pathways from program inputs to effects: E.g.: installing garden beds, preparing the soil, sowing, planting and harvesting; setting up and implementing the BCC strategy, improving maternal knowledge through repeated BCC sessions, and achieving changes in practices; – Meaningful effect on biological outcomes such as child anthropometry may require as long as 1,000 days of program exposure. – Time to design a rigorous impact evaluation: Building the program theory framework, developing the evaluation and sampling design, designing and pre-testing the data collection instruments, training and standardizing enumerators, planning the logistics of the field work, enroll the necessary number of study participants, and seasonality. • … vs often short time frames imposed by donors
  • 18. What are some key challenges? • Complexity of agricultural programs aimed at improving nutrition • Long impact pathways and time frames • Trade-off between implementation constraints and evaluation rigor
  • 19. Implementation constraints vs. evaluation rigor Treatment 1 Treatment 2 Treatment 3 Control Regular program Cluster randomized controlled design T24 T18 TNFP Control Regular program
  • 20. Part 2: Key challenges and solutions 1. Challenges in evaluating the impact of nutrition-sensitive agricultural programs 2. Rigorous evaluation approach to evaluate: a) Impact b) Impact pathways, and c) Cost 3. Recommendations on how to address key challenges of carrying out sound evaluations implemented under real life conditions.
  • 21. Part 2: Comprehensive evaluation approach PROGRAM Maternal & child health & nutrition What is the impact of the program?
  • 22. Impact evaluation Part 2: Comprehensive evaluation approach: embrace complexity What is the impact of the program? Process + impact evaluation How and why does the program (not) have an impact?Cost study Cost of the program?
  • 23. What is the impact of the program? • impact=(N|with program) - (N|without program) • Problem: (N|with program) and (N|without program) never both “observable” • The key challenge to impact evaluation: what would have happened in the absence of the program = counterfactual
  • 24. Creating a valid counterfactual INTERVENTION CONTROL
  • 25. Experimental designs • Experimental (or randomized) designs considered gold standard for impact evaluations. • Randomization: individual or group (cluster) level. • *If* done well, one can assume: – that both groups are comparable; – that the only difference between the groups is the program; – that the control group provides a valid counterfactual for the intervention group exposed to the program. differences found in the outcomes of interest attributable to the program.
  • 26. Experimental designs • Not always feasible • Often require creative thinking (oversubscription, public lottery, delayed intervention, stepped wedge design, etc.) • Alternatively: quasi-experimental designs – use statistical techniques to create a valid comparison group – E.g.: propensity score matching (PSM), double difference (or difference-in-difference) approach, regression discontinuity, instrumental variable regressions, etc. • Stay away from before-and-after or with-and- without designs!
  • 27. Invalid counterfactuals 68.2 74.8 64.1 64.3 2010 2014 68.2 74.8 64.7 63.0 2010 2014 68.2 74.8 62.3 64.1 2010 2014 68.2 74.8 64.6 65.7 2010 2014 -5.5 -7.1 -5.2 -4.0 -10.0 -5.0 0.0 60.0 65.0 70.0 75.0 80.0 2010-2014 2010-2014 2010-2014 2010-2014 All T24 T18 TNFP Control Treatment Impact (95% CI) Impactonstunting(pp) Wave 68.2 74.8 64.1 64.3 2010 2014 68.2 74.8 64.7 63.0 2010 2014 68.2 74.8 62.3 64.1 2010 2014 68.2 74.8 64.6 65.7 2010 2014 -5.5 -7.1 -5.2 -4.0 -10.0 -5.0 0.0 60.0 65.0 70.0 75.0 80.0 2010-2014 2010-2014 2010-2014 2010-2014 All T24 T18 TNFP Control Treatment Impact (95% CI) Impactonstunting(pp) Wave Stunting(%)Impact(pp) 0.0 -5.0 Randomized design -5.5 pp 68.2 74.8 64.1 64.3 2010 2014 68.2 74.8 64.7 63.0 2010 2014 68.2 74.8 62.3 64.1 2010 2014 68.2 74.8 64.6 65.7 2010 2014 -5.5 -7.1 -5.2 -4.0 -10.0 -5.0 0.0 60.0 65.0 70.0 75.0 80.0 2010-2014 2010-2014 2010-2014 2010-2014 All T24 T18 TNFP Control Treatment Impact (95% CI) Impactonstunting(pp) Wave Before-and- after + 0.2 pp 68.2 74.8 64.1 64.3 2010 2014 68.2 74.8 64.7 63.0 2010 2014 68.2 74.8 62.3 64.1 2010 2014 -5.5 -7.1 -5.2 60.0 65.0 70.0 75.0 80.0 2010-2014 2010-2014 2010-2014 All T24 T18 Control Treatment Impactonstunting(pp) Unadjustedstunting(%) Wave With-and- without - 10.5 pp
  • 28. Impact evaluation Part 2: Comprehensive evaluation approach: embrace complexity What is the impact of the program? Process + impact evaluation How and why does the program (not) have an impact?Cost study Cost of the program?
  • 29. How and why does the program (not) have an impact? • Two ways to assess how impact is achieved : 1. Measurement of intermediary measures (outcomes) in impact study 2. Process evaluation study • Solid understanding of program theory and program impact pathways is key: – Key program components, – Factors that affect optimal delivery or utilization of each component, – Assumptions associated with each of the components, and – How the components are expected to be linked (pathways of impact) • Understanding the pathways to impact: – Critical to improving program delivery and effectiveness – Identify what is needed to scale up and to adapt the program in other settings – Help interpret (lack of) impact
  • 30. Designing the process evaluation • Examines the primary inputs, processes, outputs and outcomes along each of the primary program impact pathways • Addresses five key questions: – Are program services being implemented and provided according to the program design (inputs and processes)? – Are program services being utilized as intended (outputs)? – What is the quality of the program inputs and services (inputs, processes and outputs)? – What are the barriers and facilitators to optimal service delivery and utilization (inputs and processes)? – Is the program on track to have the desired effect on improving intermediary outputs and outcomes, such as improvements in knowledge?
  • 31. Impact evaluation Part 2: Comprehensive evaluation approach: embrace complexity What is the impact of the program? Process + impact evaluation How and why does the program (not) have an impact?Cost study Cost of the program?
  • 32. What is the cost of the program? • Objectives: – Estimate the overall cost of the program, the cost of the main program components, and the program’s cost-effectiveness. – Estimate savings or cost associated with adding, changing or dropping program components, adding beneficiaries or scaling up the program. • Method : Activity Based Costing Ingredients (ABC-I) approach. • Steps: – Using the program impact pathways: detailed description of all program activities. – Identify the program’s main activities and “ingredients” needed for each. – Identify the different types, quantities and costs of the “ingredients” necessary for each activity. – Calculate cost of each program activity and of full program.
  • 33. Part 2: Key challenges and solutions 1. Challenges in evaluating the impact of nutrition-sensitive agricultural programs 2. Rigorous evaluation approach to evaluate: a) Impact b) Impact pathways, and c) Cost 3. Recommendations on how to address key challenges of carrying out sound evaluations implemented under real life conditions.
  • 34. Two keys to success • Solid evaluation framework: In-depth understanding of the program, program theory, program impact pathways, rigorous comprehensive evaluation design • Strong partnership and collaboration between implementers and evaluators
  • 35. Strong partnership between implementer and evaluator • Establish collaboration at the program design phase and maintain throughout. • Align potentially differing priorities, expectations, incentives and time frames. • Program implementers: Share updates and challenges on program roll out and service delivery • Evaluators: Provide regular updates on goals, methods and findings from their evaluation activities.
  • 36. In summary • Evidence of what works, how and at what cost is extremely limited. • Guidance for future investments requires strong evidence from rigorous, theory-based comprehensive evaluations of different nutrition-sensitive program models that bring together interventions from a variety of sectors (e.g. health, education, agriculture, social protection, women’s empowerment, water and sanitation, etc.) • This presentation focused on: – How to design and carry out rigorous impact, process, and cost evaluations. – How to address some of the perceived insurmountable challenges that have prevented investments in rigorous evaluations of such programs in the past.
  • 37. Exciting recent and ongoing studies • Completed: – Homestead food production: • Burkina Faso • Improved women’s and child’s health, nutrition • Improved women’s empowerment – HarvestPlus biofortified orange-fleshed sweet potato (OSP): • OSP vines were given to 24,000 households in Uganda and Mozambique from 2006 to 2009 • project included agriculture component and nutrition behavior change • Project increased vitamin A intake of young children
  • 38. Exciting recent and ongoing studies • Ongoing: – Improve chicken value chains (Burkina Faso) – Improve quality of milk sold by informal vendors (Nairobi, Kenya) – Reduce aflatoxin in maize (Kenya) – Improve milk contracts for nomadic farmers (Senegal) – USAID-funded, large-scale agribusiness program in horticulture and a basic health care program (Upper Egypt)
  • Related Search
    Similar documents
    View more
    We Need Your Support
    Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

    Thanks to everyone for your continued support.

    No, Thanks
    SAVE OUR EARTH

    We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

    More details...

    Sign Now!

    We are very appreciated for your Prompt Action!

    x