The new insights and approaches from the European Commission’s Guide on Project Evaluation.
A new, long-awaited Guide
A new Guide for Project Evaluation was published a few months ago by the European Commission (DG International Partnerships). The previous edition was from 2006, so this is a long-awaited document for “insiders,” in this case from the specific field of evaluation and development cooperation. Evaluation is indeed an activity that in part crosses project boundaries, and is the responsibility of “evaluation professionals” appointed by the European Commission, or its dedicated staff.
This guide is intended to provide the procedural, operational and methodological guidance for these professionals so that they operate on the basis of common standards, processes and methods. However, its reading gives very useful guidance to anyone who wants to learn more about the concept of evaluation as applied to European projects. In particular, its third part (devoted to approaches, methods and tools) traces the whole path from the elaboration of the intervention logic to the evaluation of the project and the estimation of its impact, providing interesting methodological insights and tools.
What we have described is largely reminiscent of what was outlined in the dedicated chapter of our Guide(Managing the project: monitoring and evaluation). There are, however, some additional aspects, which we have incorporated into our discussion and summarize below.
The “missing link” for impact estimation
The most interesting aspect, which we have integrated in our dedicated chapter, concerns the analysis of the methods used to estimate impact from the intervention logic (or theory of change).
We focused mainly on nonexperimental methodologies, thus different from “scientific” and counterfactual evaluation, which is difficult to implement in a project. These methodologies formalize a process that attributes (or not) the observed results to a particular project or intervention. We mention the ones that seemed most interesting to us.
- Contribution Analysis focuses on gathering evidence to validate the attribution by researching, identifying and excluding (where possible) complementary factors and alternative explanations;
- The Global Elimination Methodology (GEM) further systematizes this approach by identifying lists of causes and related “modus operandi” for each outcome of interest;
- Realist Evaluation aims to embrace the plurality of mechanisms, interactions and dynamics related to actors and context by analyzing a series of hypothetical “mini-theories of change,” and contextualizing project outcomes within them;
- Qualitative Comparative Analysis (QCA) compares different cases that produced a similar outcome. It analyzes the factors present in the different cases to define their degree of relevance and contribution to the observed impact;
- Process Tracing is a methodology for examining a specific causal link through various types of “tests” to assess whether the evidence is sufficient to draw a conclusion about causation;
- Outcome Mapping (qualitative and participatory methodology) assesses the effects of a project or intervention by analyzing changes in the behavior, relationships, activities, and actions of people, groups, and organizations with whom it has directly worked.
Another interesting aspect (and one that we have supplemented) of DG International Partnerships’ new guide, concerns the analysis of the most frequent biases (biases) in the data collection and treatment process: “confirmation” bias, “empathy” bias, “self-censorship” bias, “strategic” and “manipulation” bias, “apophenia” (or “spurious correlation”) bias, “anchoring” bias or “halo effect” bias. Since project monitoring and evaluation are based on data, it is important to be aware of them and be able to recognize them.
New sources of insights on evaluation
We have included this new guide from the European Commission as one of the recommended sources for learning more about project impact assessment.
We also recommend the Capacity4Dev platform ‘s dedicated page full of articles and insights on various aspects of the subject: evaluation criteria, evaluation methods and tools, indicators, internal monitoring, results framework-just to name a few.
Finally, we point out what is probably the most comprehensive, exhaustive and “scientific” source on the subject: the BetterEvaluation platform, dedicated to the improvement and dissemination of methodologies and tools for good evaluation of projects, interventions and public policies. It proposes in particular a tool(RainbowFramework) to better define and manage one’s own evaluation system, a selection of dedicated Guides and a detailed description (with examples and an extensive review of in-depth links) of methods and approaches used in the world of evaluation.
You can find out all this through the links provided in this article, and in the dedicated chapter of our Guide.
