Virtual Training on Rigorous Impact Evaluation (RIE)

For Stakeholders of the German Development Cooperation
3 days virtual workshop, December 15th – 17th, 2020

 

In Short:

An interactive workshop on rigorous impact evaluation (RIE) conducted by a team of experienced RIE experts for interested stakeholders of the German development cooperation, who either have some experience or pure interest in the methods and their practical application. We will provide the core intuition of why RIEs are important and guide the participants through core aspects – from an idea of what should be evaluated, to the intuition behind different choices of evaluation methods. Further topics are timelines of an RIE, the prerequisites, opportunities to learn and save valuable resources as well as potential challenges to consider.

The workshop contains many interactive elements and does not require any prior RIE knowledge. Modules build up on each other, therefore we ask for participation in all sessions. The group size is limited to allow an interactive exchange of ideas, feedback, polling, case studies and lively examples as well as discussions in smaller groups. On the third day, we offer the opportunity to book a slot with an RIE expert for individual feedback on particular evaluation ideas. Information on session details and the registration process can be found below.

 

Why a Workshop on rie?

While German development cooperation focuses on generating positive impacts in partner countries, generation and use of rigorous impact evaluation (RIE) seems to be rare. Few rigorous impact evaluations have been conducted or initiated by or with the participation of German development cooperation actors and compared to other donor countries, these activities are rather unsystematic, Furthermore, existing rigorous evidence is not always systematically synthesized and used in development programming and policy-making.

The goals of the virtual training on RIE are (1) capacity-building and community-building through interactive and targeted lectures on RIE methods; (2) hands-on support and consultation to the German development cooperation sector, incl. the German Federal Ministry (BMZ), implementing organizations, and civil society organizations on how to practically implement and oversee RIEs in ongoing or planned projects; and (3) incorporation and presentation of examples from current projects using RIE in Germany.

 

WHO should participate

This workshop is drafted for individuals with no or little prior knowledge of quantitative impact evaluation methods but a high curiosity for such methods. This includes anyone who plans or oversees development programs, sets up (log)frameworks, would like to invest into institutional and program learning, or would like to better understand and assess different types of evaluations.

After setting the core methodological foundations, using examples and setting up interactive groups, we will discuss the day-to-day challenges faced by policy-makers and program managers when integrating RIE methods. The workshop will enable implementers on how to better plan programs and monitoring systems in a way that impact evaluation can be embedded naturally. They will learn how RIE can make a difference in the assessment of their work, allowing for learnings within their project and organization and for increased accountability. The workshop will for example cover when and how to select a control group, and how RIE resources can be cost-effectively allocated and integrated in the overall monitoring budgets.

The training is organized by Dr. Alexandra Avdeenko. Apart from herself, the workshop will include input from Prof. Dr. Markus Frölich, Dr. Esther Heesemann, Verena Himmelreich, Dr. Markus Olapade, and will be supported by a team of experienced RIE experts from C4ED.

 

WORKSHOP FORMAT

This training is divided into six virtual lecture sessions, including interactive breakout sessions for brainstorming and group work. The sixth session consists of an individual consultation session. All training and consultation will be held virtually via a video conferencing platform (Zoom). A team of RIE experts will be present at all times to guide the interactive elements of the sessions. All sessions are basic and do not require any prior knowledge. The core thing to bring along is curiosity and openness.

 

CONTENT

SESSION 1: WHY RIE? DISENTANGLING PROGRAM NEEDS, THEORY OF CHANGE AND EVALUATION QUESTIONS

Identification of sources of existing evidence; Why “rigorous” impact evaluation?; Identification of potential RIE-opportunities and clear objectives of what an RIE can achieve; Identification of suitable evaluation questions for impact evaluations; Identification of key bottlenecks in the theory of change; Presentation and discussion of case studies to illustrate the relevance of assumptions for RIE, including beneficiary selection, implementation plan and stakeholders.

SESSION 2: EVALUATION DESIGN – RIE METHODS

Discussion of (quasi-)experimental impact evaluation methods, including the need to set up a comparison group. RIE designs: What is a comparison group? Which RIE methods exist, why, and when are they suitable? What is meant by “non-experimental” approach? We will illustrate and discuss what is needed to apply the following methods (assumptions, requirements, limitations): Difference-In-Differences (DiD, with matching); Propensity score matching (PSM); Regression discontinuity design (RDD). What is meant by “experimental” approach? RCT (What is the general idea? What is a random assignment and how does it help with inferring causality?).

SESSION 3: EVALUATION DESIGN – RCT: WHY A NOBEL PRIZE?

In this session, we will go in-depth into the question what is an RCT. We will present the richness of the method and explore different methods: Cluster Randomized Trial and Individual-Level RCT; methods of randomization (a) Simple lottery, (b) Randomization in the “bubble”, (c) Randomized phase-in, (d) Rotation, (e) Encouragement design. Also, limitations will be outlined, and several case studies discussed.

SESSION 4: SETTING THE EXPECTATIONS RIGHT – DATA (SOURCES, INDICATORS, AND SIZE) AND TIMELINES

This session shall help understanding the data needs for an RIE (a) based on sample size and power calculations and (b) based on the existence and incorporation of different data sources. Core will be an understanding on how to set realistic expectations towards measurable impacts. Furthermore, we discuss how to ideally time an RIE.

SESSION 5: EVIDENCE UPTAKE, MIXED METHODS (qualitative and quantitative) and Q&A

We start with an outline of potential sources for evidence dissemination and importance of evidence uptake. We continue with an introduction to the benefits of adding qualitative research to RIE (a so-called mixed methods approach) to understand how this approach can help answer questions of “what works (or not)”, “how does it work” and “why does it (not) work”. We close the session with an open round of questions that participants can raise.

SESSION 6: INDIVIDUAL CONSULTATION SESSIONS

The participants have the unique opportunity to book individual consultation slots with the RIE specialists. The specialists will provide feedback on how an RIE could be conducted in the specific context of the participant and will also give feedback on evaluation questions, expectations, timelines, and data availability to further illustrate and guide on whether and how an RIE could be conducted.

 

SCHEDULE

 

REGISTER

Please register on events@c4ed.org until Thursday, December 3rd 2020, stating

  • your affiliation
  • your position
  • your knowledge level of RIE (no prior knowledge is necessary)

Note that the number of participants is limited to 30. There is no fee for this workshop.

All information gathered throughout this registration process will be shared with DEval’s RIE-team (exclusively) for coordination purposes. Gathered data will be deleted after the training has been conducted. You can exercise your right of objection at any time without stating reasons and withdraw from the registration process in text form with effect for the future. Your personal data obtained throughout this registration process will then be deleted by us.

This workshop is jointly given and organized by the German Institute for Development Evaluation (DEval) and the Center for Evaluation and Development (C4ED) and led by Dr. Alexandra Avdeenko. For questions concerning the content of the workshop, please contact Dr. Alexandra Avdeenko. The organization of the event will be supported by Carolin Hoffmann.

 

INSPiRING READINGS on RIE (Entirely optional)

  • Davies, R. (2018) Representing theories of change: Technical challenges with evaluation consequences. Journal of Development Effectiveness, 10(4), 438-461.
  • FAO / UNDP (2018) Using impact evaluation to improve policymaking for climate change adaptation in agriculture. Briefing Note, March 2018.
  • Frölich, M./ S. Sperlich. Impact evaluation. Cambridge University Press, 2019.
  • Glennerster, R. and K. Takavarasha (2013). Running randomized evaluations: A practical guide. Princeton University Press.
  • Gugerty, M. K. and D. Karlan (2018). The Goldilocks challenge: Right-fit evidence for the social sector. Oxford University Press.
  • Oliver, S., Roche, C., Stewart, R., Bangpan, M., Dickson, K., Pells, K., … & Gough, D. (2018). Stakeholdengagement for Development Impact Evaluation and Evidence Synthesis. CEDIL Inception paper No, 3.
  • J. Gertler, S. Martinez, P. Premand, L. B Rawlings and C M J Vermeersch (2011) Impact evaluation in practice, The World Bank.
  • Reichardt, C. S., & Mark, M. M. (2004). Quasi-experimentation. Handbook of Practical Program Evaluation, 19, 126.
  • Rogers, P. (2014). Theory of Change: Methodological Briefs – Impact Evaluation No. 2. UNICEF.
  • Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin. Chapter 8 – Randomized experiments.
  • White, H., & Raitzer, D. A. (2017). Impact evaluation of development interventions: A practical guide. Asian Development Bank.
  • White, Howard. The twenty-first century experimenting society: the four waves of the evidence revolution. Palgrave Communications 5.1 (2019): 47.

Past events

Introduction to Systematic Reviews
in International Development
Mannheim, Germany | December 11th-12th, 2019

 – This two-day hands-on workshop introduces participants to the principles of systematic reviews –

BACKGROUND on the WORKSHOP

The number of research publications doubles every nine years. More than 6,000 new journals are started every year. And policy makers do not read academic journals anyway. So how are we to stay on top of the current state of academic literature, what can we learn and how can consensus positions emerge from a vast array of differing studies in different contexts with different findings, and how can policy be informed by evidence?

The adoption of evidence-based medicine was driven by systematic reviews. Systematic reviews summarize all available high-quality evidence addressing a specific question. Non-systematic reviews, including traditional literature reviews, are more prone subject to bias from missing studies and selective reporting. Well conducted reviews deliver clear policy messages.

The evidence-based policy movement promotes the adoption of the systematic review approach as a standard methodology in social and economic analysis.

WHO

For early to mid-career researchers (incl. PhD students) interested in undertaking a systematic review. Prior experience in conducting a review is not necessary. Participants should: (1) have a good grounding in statistics/ econometrics, and be familiar with approaches to estimating effects/impact in the presence of selection bias; (2) be familiar with R or Stata. Participants should bring their own laptop to the workshop.

 

PRELIMINARY SCHEDULE

REGISTER

events@c4ed.org stating your affiliation and position until 2nd December 2019. Note that the number of participants is limited.

More information: c4ed.org/events

no workshop fee!

 

This workshop is given by the Campbell Collaboration. It is organized by the Center for Evaluation and Development (C4ED, Dr. Avdeenko) and University of Mannheim (Prof. Dr. Markus Frölich, SFB 884 and Chair of Econometrics).

WORKSHOP FORMAT

The workshop has an interactive format of lectures followed by hands-on sessions. There will be a short quiz at the end of each day. The hands-on sessions are partly group work and partly individual exercises. The group work will be based on developing a research question using the Campbell Collaboration Title Registration Form. On the second day, each statistical lecture is followed by individual exercises, where participants perform meta-analytical analyses using R or Stata.

BACKGROUND READING

White (2019): Four Waves of the Evidence Revolution.  https://www.nature.com/articles/s41599-019-0253-6;

Saran/ White (2018) Evidence and Gap Maps: A Comparison of Different Approaches https://onlinelibrary.wiley.com/doi/10.4073/cmdp.2018.2

Waddington et al (2012): How to do a good systematic review of effects in international development: a tool kit Journal of Development Effectiveness https://www.tandfonline.com/doi/full/10.1080/19439342.2012.711765