Mastering Impact Evaluation: Strategies, Methods and Tools

In the past, evaluation was focused to a large extent on evaluating the mechanisms of delivering programmes or projects. In other words, they focused on the implementation process. Relatively little attention was paid to evaluating the impact of interventions that were being financed. While over the last few years substantial efforts have been made to promote and disseminate counterfactual impact evaluation practice (quantitative), there is much less sharing of good practice concerning so-called theory-based impact evaluation. However, while counterfactual impact evaluation can tell us how much of an impact an intervention has, theory-based impact evaluation is crucial to increasing our understanding of ‘why’ certain interventions work or do not work. Without this understanding, it is not possible to improve interventions. In addition, in many cases counterfactual impact evaluation is not actually a viable method, and only theory-based impact evaluation will be able to shed light on whether or not the intervention works.

The seminar will therefore focus on this theory-based impact evaluation, drawing on the latest advances made in terms of methodology. The most cited methods – Process tracing, Congruence Analysis, QCA, Pawson and Tilley’s Realist evaluation, Mayne’s Contribution Analysis, Scriven’s General elimination method – will all be covered and put into perspective. Several cases will be used to exemplify these methods. The participants will also execute many exercises to better grasp the methods. An important focus of the course will be to match the rigor associated with counterfactual impact evaluation when making causal claims.