Managing the project: monitoring and evaluation

Why and how to talk about monitoring and evaluation in European projects

This chapter discusses how the Logical Framework, a basic tool for formulating European projects, is used for project management, that is, as a monitoring and evaluation tool.

As we will find out, there are in fact few differences between the Logical Framework (especially in its “extended” and official version ) and a “monitoring and evaluation framework,” which is essentially a more detailed and operational version of what is already reported in the Logical Framework.

Proper use of the Logical Framework and good management of a European project require effective monitoring and evaluation. These aspects constitute the most important component of project management activity, and few official “instructions for use” exist on them: as well as other specific aspects covered in this section (e.g., partnership management and reporting), they are normally learned as part of specialized courses or through practice.

The following therefore attempts to fill this gap by providing an initial operational and non-exhaustive framework on this topic. It deepens and relates what has already been explained in the previous chapters:

How to structure a project: the process and tools | How to structure a project: a practical example and other tools | Developing and managing a project

Monitoring and evaluation are an integral part of every project activity. To be such, a “project” is in fact subject to constraints that require forms of control:

Achievement constraints (a project is aimed at achieving results, products and objectives, obtained through a series of interrelated activities);
Time constraints (a project takes place within a defined time frame), economic and resource constraints (a project has specifically allocated economic, material and human resources);
Organizational constraints (a project is organized into phases with specific date-limits and includes a division of tasks and responsibilities).

Monitoring and evaluation are forms of controlling these constraints.

This control affects both those who implement the project (the one who must keep its execution within the required constraints) and those who finance it (the one who defines much of those constraints). The European Commission and managing authorities of European funds also respond to constraints and are required (themselves) to carry out monitoring and evaluation activities on their programs and projects. Knowing how to communicate what they have accomplished with the funds entrusted to them is an integral part of their responsibility to the citizenry.

Achieving the set goals, within the given timeframe and with the available resources, therefore, is at the heart of the link between the person submitting a European project and the managing authority funding it. For this reason, well-presented (at the proposal stage) and well-implemented (at the implementation stage) monitoring and evaluation is a key component for those involved in europlanning and a condition of success for their projects.

Even when the notice does not explicitly require the development of a Logical Framework or Monitoring and Evaluation Framework, the requirement is implied in various fields and criteria of the application form (description of the project logic, expected project impact, project monitoring system, etc.). These fields and criteria can carry considerable weight in the project selection process and should be presented in concrete terms that are consistent with the proposal as a whole. This is possible if the project is supported by a good Logical Framework and a good monitoring and evaluation framework.

Monitoring and evaluation are two similar activities; they support each other and both serve to understand “how the project is going” (or “how it went”). However, they are distinct:

By timing. Monitoring is carried out continuously and systematically during the course of activities and is necessary to guide the operational life of the project; evaluation takes place at specific times (especially at the end of activities, but also mid-project or time after project completion) and has strategic implications (did the project produce the effects it intended? What lessons to be learned from it? How to orient new programs and projects accordingly?);
By intent. Monitoring focuses on the “lower” levels of the Logical Framework, the day-to-day management of project activities, risks, time and resources and outputs, and operational dimensions (Efficiency and Effectiveness); evaluation focuses on the “higher” and strategic levels of the Logical Framework (results, objectives, outcome and impact of the project) and invests in depth all 5 criteria of Relevance, Efficiency, Effectiveness, Impact and Sustainability;
For instruments. Evaluation may use “slower” and more complex tools (e.g., questionnaires, interviews, focus groups and other data collection methods), while monitoring uses faster and more immediate data collection and analysis tools, or direct observation of a few key variables. In addition, monitoring is usually carried out by those in charge of project activities, while evaluation is often delegated to third parties;
By relevance (for those involved in European projects). Of the two activities, monitoring is the one to which those implementing a European project are most directly accountable, and the one that most directly impacts project execution. Nevertheless, evaluation remains an important exercise, providing strategic guidance (when carried out, for example, mid-project) and a set of concluding reflections and orientations (often required in final project reporting).

The effort required for monitoring and evaluation extends from the design phase of a project to its completion, and beyond. It consists of the following steps, developed in the next sections of this chapter:

Define the intervention logic;
Define the indicators;
Define the monitoring and evaluation framework;
Implement the monitoring and evaluation plan, through appropriate tools and data collection methods;
Estimating the “impact” of a project.

Defining the logic of intervention: the Logical Framework and the Theory of Change

The starting point for a good measure of project performance is a clear identification of its intervention logic, that is, the (logically concatenated) sequence of objectives, outcomes and activities that it is intended to achieve.

A formulation of the intervention rationale that is inaccurate, clear and consistent (in definition and logical link between its parts) makes the subsequent preparation of the monitoring and evaluation framework more complex, less effective and less meaningful. In fact, if you do not define precisely what you intend to achieve from a project, it is difficult to make an assessment of it.

The intervention logic is in turn the basis of the Logical Framework, which is the most substantial core of a monitoring and evaluation framework. In its “extended” version, the Logical Framework in fact includes almost all the elements of a monitoring and evaluation framework (indicators, verification sources, baselines, targets and progressive data referring to the measurement of indicators).

More specific directions for defining the intervention logic and the Logical Framework are provided in a dedicated chapter (accompanied by a practical example).

To support the formulation of good intervention logic, there is an alternative and complementary tool to the Logical Framework (and the Problem/Solution Tree), called the Theory of Change.

The Theory of Change is based on a similar (though not identical) process and terminology and identifies (like the Logical Framework) various logical steps that unite project activities to the “ultimate change” that is desired. It is a more dynamic and flexible tool than the Logical Framework. It involves the following steps:

The definition from the long-term “change” that the project intends to bring about;
The mapping of the conditions necessary to bring about such change;
The verification of the necessity and comprehensiveness of the identified conditions;
The formulation of hypotheses to exclude some of the identified conditions from the scope of intervention;
The identification of interventions necessary to achieve the remaining conditions, which consequently constitute the scope of the project;
The development of indicators to verify that all conditions necessary for the change initially identified have been realized.

You can learn more about the Theory of Change, with explanations and examples, on a dedicated site.

The visualization of a project’s Theory of Change takes on more varied and flexible forms than the (rigid and structured) table typical of the Logical Framework. A Theory of Change can be visualized through graphs with boxes, arrows and nonlinear paths, and is therefore better suited to capture the complexity of the scope in which a project intervenes.

On the other hand, the more structured form of the Logical Framework makes it possible to define more precisely a “hierarchy” of indicators, which are associated depending on the “level” of reference either with monitoring activities or with evaluation activities.

While the Theory of Change is very useful for defining the intervention logic and structuring the project, the Logical Framework is the baseline for subsequent monitoring and evaluation activities.

Other useful tools to support the development of a project’s intervention logic are “gap analysis” and “scenario planning,” in which the following are respectively analyzed: (a) the “gaps” to be filled regarding certain parameters, the target parameters and the actions and resources needed to achieve them; (b) the possible “scenarios” (positive and negative) for the evolution of a given sector or context, the conditions and factors (positive and negative) associated with each type of evolution, and the path necessary to reach the “scenario” considered most favorable and feasible with the available resources.

Defining indicators: "SMART" criteria, linearity and simplification

The next step after defining the intervention logic is the definition of indicators.

As indicated in the Logical Framework matrix, at least one indicator should be included for each line of the intervention logic (General Objective, Specific Objectives and Results). For the purpose of effective and timely monitoring, it is advisable to provide simple indicators or operational checklists also for the implementation of Activities or groups of Activities (Work Packages or Components).

The definition of indicators should follow the following criteria, which make up the acronym “SMART.”

Specific (must be tangible, precisely defined, uniquely identifiable);
Measurable (must be objectively detectable, qualitatively or quantitatively, with available resources);
Achievable (must refer to content that is actually achievable with available resources);
Time-bound (must be associated with a timeline for their achievement and measurement).

In operational terms, a good definition of indicators requires both linearity and precision.

Linearity and simplification. A good evaluation framework relies on a small number of relevant, specific and well-measured key-indicators: too many indicators are an unnecessary overload on project resources. Each indicator must have its own relevance to define the success (or otherwise) of the project. Indicators that are not really relevant in defining project success or performance, that are difficult to measure, or that are too costly to measure can be avoided: by revising the intervention logic, by using other more easily measurable indicators, or by using “proxy” indicators (which do not directly measure the desired variable but give a reliable estimate of it) ¹ .

The complexity of the indicators depends on the level of intervention logic to which they relate. Activity or outcome indicators should be able to be measured immediately (as part of day-to-day project monitoring), while outcome or impact indicators (related to project goals) may require more in-depth evaluation, as they measure more general and long-term change. The goals and results themselves should be formulated in a simple and possibly “empirical” manner to reduce the number and complexity of indicators needed to measure them.

Linearity and simplification are also guiding principles in the formulation of indicators: better to use indicators that are already known, existing and used by “insiders” (e.g., Invalsi data in a certain Region) than to invent new ones (e.g., similar data obtained through own tests); better to define indicators in concise terms rather than to include too many details (timing, targets and other aspects should be specified in dedicated fields of the scoreboard anyway).

Specificity and precision. Simplification should not come at the expense of specificity and precision, particularly when these are within the reach of one’s resources or can be achieved by simple expedients. In particular, the specificity and accuracy of indicators are a consequence of goals and results formulated in a simple, empirical and directly observable way.

The parameters must give the actual measure of what is defined as goal and outcome. They must be uniquely formulated so that their measurement is not subject to variation given by discretion. For example, if necessary, it can be defined on the side (e.g., parameters to be considered when calculating a percentage).

Specificity and precision can also be applied to qualitative indicators when these are needed to measure a desired change. In such cases, the criterion of accuracy requires defining the scale and situation to which the qualitative parameters used refer (e.g., what ratings such as “excellent” or “good” indicate) and/or specifically defining the ways through which such an indicator is appreciated (e.g., unambiguous wording of the questions administered in a questionnaire or focus group).

Indicators, and in particular their target-values, must be practically feasible within the project and must be associated with timeframes, time of detection, and the other aspects of the monitoring and evaluation framework.

Define the monitoring and evaluation framework

The monitoring and evaluation framework is a large table that cross-references each indicator with each of the following, which specify how and when each indicator will be verified:

Starting values (baseline) and target values (targets) of indicators,
Sources, ways and material means for their verification,
Timing of verification and reporting,
Responsibility and division of labor in the verification process.

The breadth of the monitoring and evaluation framework and the effort it requires must be realistic, not oversized, commensurate with the nature of the project, the resources available, and the actual operational capacities of those managing it and its beneficiaries.It is better to monitor and evaluate a smaller number ofkey indicators (key performance indicators or KPIs), possibly selected from those defined in the Logical Framework, than to carry out no monitoring and evaluation activities at all because of the complexity of the system and the volume of activities it requires.

For complex projects, it is also advisable to develop a monitoring and evaluation plan (e.g., in the form of a Gantt chart) that summarizes the activities to be carried out, the timeline and the highlights of the monitoring and evaluation process.

1. Baseline and target. Establishing a baseline, i.e., a starting figure for the indicators considered, is the first real “evaluation” activity carried out as part of a project and is called a “baseline study.”

Defining a baseline can require data, time and resources, similar to what happens when conducting monitoring and evaluation activities. The effort required to identify a baseline and corresponding target values is proportional to the effort that will be required later, during monitoring and evaluation. Difficulties at this stage (due to lack of data, time and resources) may indicate that the monitoring and evaluation framework is outsized compared to one’s ability to put it into practice.

It is important to provide a timeline for achieving the target values. In addition to the end date of the project, it is advisable to indicate baseline values for a mid-project evaluation (“mid-term review”) and possibly additional intermediate values, to understand before the end of the project whether you are on track to achieve the indicators.

Indicators that quantify specific actions or products to be implemented as part of the project normally have a starting value of zero, which grows as the activities are carried out. The target values of the indicators must obviously be quantified realistically.

2. Sources, ways and means of verification. This part of the monitoring and evaluation framework answers the key-question of “how” the indicators will be measured.

First of all, it is necessary to define the source (document or physical medium) through which the data can be collected: an internal project source (the project “creates” the data through its own questionnaires, research or observations) or an external source (publications, databases and statistical yearbooks of regional, national or international institutions, universities, companies or organizations active in the field, beneficiary organizations, other projects or parallel actions carrying out data collection and processing, etc.).

Along with the source, ways and means of verification should also be verified, considering the following parameters:

The consistency of the data with the definition adopted by the project and its constancy over time (data calculated with the same parameters with the required frequency, from the time of the baseline to the time of the last evaluations);
The reliability of the source (authoritative source, which can also be verified by third parties, or rigorous and verifiable methodology as the data is developed in-house);
Its actual availability (need for agreements or subscriptions, possible costs, uncontrollable factors that could affect access to the data);
The ways in which it is accessed (online, library, media purchase, own databases, need for further data processing).

Next, it is necessary to define how the data will be processed to produce useful information for the project, how it will be stored (type of format and medium), and how it will be accessed, checked, updated, and modified.

The information thus obtained should be passed on to those who can draw useful conclusions (and possible corrective measures) from it, namely those who manage the project, those who exercise a decision-making role in it, those who finance it, and those who have title and interest in following its development. This is done through analysis and reporting (the periodicity and structure of which need to be defined) and specific communication channels (the nature, mechanisms and beneficiaries of which need to be defined).

Given the amount of work that can be involved in data collection and processing, again the simplest, most cost-effective and straightforward and time-saving options should be chosen, such as limiting the extent and complexity of data “production” by the project and its beneficiaries (through questionnaires, surveys, focus groups, etc.) to the bare minimum.

This can be explored in more detail in the following section on how data is collected and processed.

3. Timing of verification and reporting. This part of the monitoring and evaluation framework answers the key question of “when” the indicators will be measured, or more specifically, how much time to devote to it and how often. The choice depends on the indicators, data, previously identified modalities, the commitment they require from the beneficiaries, and the human and material resources available.

The timing also depends on the “level” at which the indicator is located: as mentioned earlier, indicators related to activities and results (outputs) are usually simpler and can be checked frequently, as part of project monitoring; indicators related to outcomes (outcomes) and project objectives (impacts) are measured as part of an ad hoc evaluation activity. If an evaluation activity is counted on being carried out, it should not be relegated to the end of the project, but carried out in parallel with project preparation and implementation, so that the process of structuring and collecting data (e.g., on a target group and a possible “control group”) is properly organized.

Finally, the identified timeline must include ways and times to report the information gathered to those who have the ability to draw conclusions and implement corrective actions. The most critical indicators, i.e., those whose non-achievement poses the greatest risk to project performance, deserve constant attention throughout the life of the project. In fact, risk analysis and management are an integral part of monitoring activities.

You can learn more about how indicators are verified and risks are managed in the following section on monitoring and evaluation tools.

4. Responsibilities and duties. This part of the monitoring and evaluation framework answers the key-question of “who” measures the indicators. Identifying responsibilities in a precise way is helpful in not leaving out important aspects of project management, especially at times when deadlines and current activities absorb a lot of time and energy.

“Raising its head” from time to time, even if only to analyze a few crucial “KPIs,” allows the project to stay in the right direction. This can be done directly by the project staff and its project manager, although for large projects (and particularly for evaluation activities) specialized external consultants may be provided (to ensure process neutrality).

To this end, it may be useful to set up a simple calendar that reminds who and at what times is to carry out the main monitoring and evaluation activities, in the form of a reminder or Gantt chart; also including, as a key aspect, activities related to project reporting. Some of the key operational controls (timing, activities and deliverables, budget and resources) should be implemented automatically by the team during daily project management activities.

Monitoring and evaluation tools

Good monitoring is an indication of good project management. In fact, monitoring-and follow-up actions-is at the heart of project management activities. Many methods and tools have been developed around project management; each project manager and each project adopts the combination that best suits their organization and needs. The European Commission has produced its own “review” of tools, organized in a project management method called “OpenPM2”: we have devoted an article and a brief discussion to it. Thelatest edition of the manual, available free in Italian, is dated April 2023. OpenPM2:

It defines Monitoring and Control as a continuousactivity that extends throughout the life of the project, from its Initiation (phase 1), through its Planning and Execution (phases 2 and 3) to its Closure (phase 4).
Identifies 11 major components of the Monitoring and Control activity: Manage procurement, Manage risk, Manage attention points and decisions, Control cost, Control time, Monitor performance, Manage quality, Accept products, Manage stakeholders, Manage project changes, Manage transition.
It proposes various tools typical of the project management discipline, of which many are presented or cited in this Guide (RACI Matrix and its variants, Stakeholder Matrix, Project Planning and Budgeting, Risk and Attention Point Register, Checklist, Gantt Chart…).

Without claiming to be exhaustive, we present below two of the most widely used tools for project management, monitoring, control and evaluation, in addition to what has already been explained in this and previous chapters. The “dashboard” of the project. One of the most popular and effective ways to monitor the progress of a project is to create a “dashboard,” or “dashboard” or “control panel” that allows certain key parameters to be kept under control, including in particular:

individual activities, the outputs they produce, and their timelines, with a focus on what are called project “ milestones,” or the “milestones” that mark its major milestones (e.g., target dates for the definition, approval, initiation, and completion of a training plan). This part of the “dashboard” can be effectively represented in the form of a Gantt Chart.
The main indicators identified for the project, particularly the simpler and “operational” indicators (those related to activities and results) and those that have been defined as “KPIs” (key indicators) to be monitored on a regular basis. In this case, the “dashboard” can present, for each indicator, an expected value for each scheduled control date, to be flanked by its measurement to highlight any discrepancies.
A resource utilization plan, again with expected values of utilization over time of each resource (funds, personnel, materials) and periodic “checkpoints” to be compared with the progress status of the indicators.

To each measurement it is advisable to give each of these parameters a synthetic “label,” possibly associated with a symbol and color code (as in the example):

▶

◼

During the monitoring activity, it is also advisable to maintain:

a record of discrepancies recorded, corrective actions taken and any changes to the project structure (reshaping of activities and resources) implemented to respond to challenges encountered during its implementation;
A risk matrix and a register of related attention points (see below).

The elements of a “dashboard” can be found in some software specifically dedicated to project management, or they can be created (in a relatively simple form) through appropriately set up spreadsheets shared over a network or cloud among project team members.

The risk matrix. Project management and monitoring activities aim to prevent, mitigate and correct risks that may occur during project execution. The project “Dashboard” is therefore normally supplemented by dedicated tools for the risk concept. Risk management in turn consists of four main moments:

Identification of potential risks that could impact the project, through brainstorming activities, checklist definition and analysis of experience from other projects. It is necessary to consider all relevant risks, which may come from different sources: the organization itself and its i partners (internal risks), target groups and beneficiaries, other organizations that are part of the project’s scope, institutions, the political, economic and financial conjuncture, disruptive social or technological events, natural events, etc;
Analysis of risks, aimed at determining their likelihood and possible impact. Risks can be entered into a matrix, which, by assigning a value of 1 to 3 to probability (P) and impact (I), divides their severity (G = P x I) into mild (G = 1 or 2, compatible with normal project operation), moderate (G = 3 or 4, requires specific countermeasures) and high (G = 6 or 9, capable of undermining project feasibility);
Planning for a risk response, which may consist of various solutions aimed at avoiding the risk (by changing the design, or by specific arrangements), transferring the risk (entrusting a third party to mitigate it, e.g., through insurance), mitigating the risk (reducing its severity to a “mild” level) or accepting the risk (assuming its consequences but providing an elaborate “back-up plan” in case it arises);
Implementation of a risk monitoring system, through more or less continuous updating of the risk matrix and its response plan (in a manner similar to what has been done with the indicators within the project “dashboard”). Similarly to KPIs, risks with higher levels of severity constitute the attention points to be monitored more regularly and thoroughly.

Methods of data collection and processing

Monitoring and evaluation activities rely on data that must be collected (i.e., somewhat observed and recorded), managed (i.e., securely and uniformly maintained and stored in a medium, usually electronic), processed (i.e., transformed into useful project information through analytical and statistical tools) and presented (i.e., be integrated into a reporting system).

The data are divided into various types:

Primary data are directly collected as part of the project from the population to be observed, while secondary data are collected through existing sources. Primary data are more specific with respect to the measurement of project effects and impact, but less comparable with more general data and certainly more challenging to produce; secondary data, on the other hand, are more readily available and comparable but tend to measure project contribution less specifically;
Quantitative data correspond to numerical values directly collected at the time of measurement, are usually structured in a specific format, and are mathematically representable through equations and formulas. Qualitative data, on the other hand, refer to subjective dimensions and cannot be directly summarized in a numerical value, but are usually organized into categories to be more easily managed through mathematical and statistical tools. Quantitative data can be handled more easily with mathematical tools and may appear more “objective” because of this, but may capture the complexity of a phenomenon less effectively than qualitative data.

The science and methods that exist for data collection and processing are varied, broad and complex and are beyond the intent of this Guide. Without any claim to completeness and rigor, we attempt to provide a quick review of them, as a starting point for later study of other, more specialized sources. The various methods complement each other and are normally used in various combinations to support a project’s monitoring and evaluation activities.

1. Documentary research and use of secondary data, which can be obtained, for example, from government documents, academic research, research and analysis by others in the field or other projects, online sources such as social media, blogs, or conversations among users.

2. Direct observation, obtained by directly surveying data related to the project in various ways (e.g., number of media produced, number of participants in a training event, attendance lists, traces and feedback recorded on the IT tools created by the project, feedback obtained directly during project activities, observations recorded by practitioners during the execution of activities, etc.).

3. Surveys, used to collect data from a substantial number of people through online tools, telephone surveys, or paper questionnaires, including those administered during project activities (e.g., at the end of an event). They can allow the collection of quantitative responses, qualitative responses organized into categories, and qualitative feedback in free format (open-ended responses).

4. Interviews, i.e., face-to-face conversations between a researcher and a participant, with a question-and-answer system that can be free-form, semi-structured (with fixed questions and answers that can range) or structured (questions with shorter or predefined answers) as appropriate.

5. Focus groups, which are moderated group discussions in which participants are invited to share opinions and ideas on a given topic. It allows deeper analysis and integration of various viewpoints. It can follow a more or less structured path (such as an interview), depending on the needs and dynamics in the group of participants.

6. Crowdsourcing, which is a system (usually supported by special digital platforms) in which users of a service or beneficiaries of a project collectively contribute to the collection of data, information and feedback.

7. Case studies, used to gain an in-depth understanding of a particular phenomenon and subject. It involves the collection and analysis of multiple sources, such as interviews, surveys and direct observations to reconstruct in detail the context, factors and dynamics behind the phenomenon, which is useful for deriving trends, dynamics, example patterns, solutions, recommendations and lessons learned.

8. Narrative inquiry, qualitative approach that uses interviews, field notes, and other forms of data collection to uncover the stories of groups or individuals and analyze their experiences, perspectives, dynamics, and lessons for future action.

9. Audio and video collection. The widespread use of digital video and photography, and the growing habit of users to share them by developing their own “narrative” with them, can be used to tell the story of a project’s activities and highlights. While this is a highly narrative, participatory and qualitative mode of data collection, it can be effective in terms of the richness of the information gathered and its communicative impact.

8. “Most significant change,” a more structured variant of the previous approaches, which consists of asking each participant to describe the most significant change that has occurred in his or her experience with regard to a particular area or parameter, and to explain its significance. The collected “stories” can be deepened, further selected and commented on as a group.

10. Qualitative trend analysis, which uses tables and visual aids to collect and represent qualitative data on the changes produced about a group and over a given time frame. It can be represented in various ways, with the help of the reference group: 1) Cartesian representation on a scale of 1 to 5 of the trend of a certain parameter over the years. Multiple lines can be used to represent multiple parameters, succinctly indicating the phenomena associated with reaching certain “peaks” and “valleys” in the graph; 2) tabular representation, with the various benchmarks, a rating of them from 1 to 5 over the years and a summary rating (+ / -) of the trend of each parameter over the years; 3) visual representation, in which the group’s path on a real “road” is drawn, indicating with drawings and comments the main events, phenomena and changes that have marked its evolution over the years.

11. Benchmarking (or comparative analysis), a method used to compare and contrast two or more different cases by identifying similarities and differences between two or more actors or phenomena. Data collection should be done in parallel with the same parameters and methods in the two cases to be compared to ensure full comparability.

12. Statistical analysis, which processes quantitative data through statistical methods to describe and summarize information (descriptive statistics) or to make predictions (inferential statistics). Methods of descriptive statistics include, for example, the measurement of central tendency (mean, median and fashion), correlation (which assesses the existence of a positive or negative relationship between variables) and distribution (which describes through functions, data and curves how data are distributed over a field). Methods of inferential statistics include, for example, regression (which allows a trend to be delineated from a set of data), time series analysis, and other methods of building predictive models. Statistical analysis uses the concept of a “sample,” the size of which depends on the size of the total population to which it refers and the desired confidence level and margin of error.

13. Text mining and data mining, which involve online data collection using document sets and large databases, from which useful information, trends, and patterns can be extracted through specific computer tools, algorithms, and applications of Artificial Intelligence.

When setting up a data processing system, particularly when data are collected and processed “in-house” (e.g., surveys through telematic tools), the organization must pay attention to current legislation and regulations regarding personal data(GDPR). Personal data are defined as “information that identifies or makes identifiable, directly or indirectly, a natural person and that can provide information on his characteristics, habits, lifestyle, personal relationships, health status, economic situation, etc.” (here the Privacy Guarantor‘s reference page). It is advisable to use as little personal data as possible, to keep a record of the data collected, to collect data implemented as anonymously as possible (or when not possible, an “anonymization” of the data collected), and to identify a person responsible for the processing of personal data, based on current legislation.

The most frequent biases (biases) in the data collection and processing process are:

“Confirmation” bias, which is the tendency to look for evidence consistent with expected effects while overlooking evidence that might contradict them or potentially problematic aspects;
The “empathy” bias and the “self-censorship” bias, which is the tendency to answer positively to please the questioner, or to avoid expressing opinions contrary to the current opinion or that of one’s hierarchy;
The “strategic” bias and the “manipulation” bias, i.e., that induced in respondents by the desire to guide the assessment based on their own goals, or by the way the questions were phrased, which can result in a bias in the responses;
The bias from “apophenia” or “spurious correlation,” that is, the bias induced by observing a correlation between variables that does not really exist, however, or is not determined by any causal relationship;
The “anchoring” or “halo effect” bias, which is the tendency to interpret all information based on the first information gathered, or based on a single variable.

Some tricks help limit the impact of these biases, such as asking open-ended questions, mixing positive and negative questions, keeping questions focused on facts, ensuring anonymity, and cross-referencing data from different sources (triangulation).

From indicators to impact

Impact, a complex concept. The result of monitoring and evaluation activity is clearly and directly appreciable when it concerns operational aspects: activities and outputs realized, beneficiaries reached, and immediate and visible consequences of a project that has just been completed. These data, which are mainly obtained through monitoring activities, are also the most important for project implementers, who are held directly accountable.

It is, on the other hand, more complex to answer broader and more strategic questions typical of evaluation work: what was (and what will be) the impact of the project? Has the project achieved its ultimate goal, that is, its overall objective? Has it produced the effects and change it was intended to when it was developed?

Answering these questions with objective evidence requires collecting data in a period after the project has ended and using resources that may go beyond what is made available under a single European project.

Moreover, impact is a conceptually and statistically-mathematically complex concept because many factors contribute to its realization: it is not easy to “isolate” the project’s contribution from a plurality of other concomitant factors. For example: how appreciable are the effects of a poverty reduction project on a community, and how do we isolate them from a plurality of other factors (positive or negative) such as the effects of the economic situation, industrial policies, other parallel projects, and the initiative of community members?

However, measuring impact remains a legitimate concern: impact is an integral part of the project’s rationale and its monitoring and evaluation framework; it is the starting and ending point for anyone implementing or funding a project; it is what defines in the broadest terms the actual success of the project.

Again, the following discussion makes no claim to scientific rigor or comprehensiveness, but aims to translate the concept of “impact” into some insights that may be “within the reach” of European project implementers.

Impact as counterfactual analysis. Counterfactual analysis defines impact as the difference between data collected at the end of an intervention (“factual” data) and data collected in a situation characterized by no intervention (“counterfactual” data). This is the most “scientific” approach to impact evaluation: in fact, it is used in medical research, which compares “subject to treatment” groups with time series or control groups.

This approach is difficult to use in the social field, as it assumes:

The existence of indicators that can be unambiguously verified with analytical tools and that have an equally unambiguous and long-term link to the dimension they are intended to measure;
Or, the possibility of identifying a “control group” with characteristics and dynamics fully comparable with those of the project’s target group.

These are not easy conditions for many projects involving “human” and social aspects, where the correlation between data and measured phenomenon may be more or less strong, but is hardly unique and depends on the intervention of multiple factors (especially given the small scale of a normal project); and where the situations of groups and communities are very varied, complex and difficult to compare with the limited resources of a normal project.

Despite its operational limitations in the context of European projects, counterfactual analysis remains a useful “ideal reference” for measuring impact. It can be carried out with three mode modifications:

The one called “Pre-post design”, which estimates impact from discrepancies between long time series of data (and resulting projections) and what is observed after project implementation. This mode requires a time span and scope of intervention that are difficult to match with the data and accomplishments of a project;
The one called “control group design”, which estimates impact by analyzing the difference between the group of project beneficiaries and a “control group” with the same characteristics. This mode involves identifying (and analyzing) a control group that is “other” than the project activities;
The one called “Difference in differences”, which combines the previous two modes and analyzes the double variation of a variable: over time (before, after, ex-post”) and between subjects (recipients and nonrecipients).

Impact as a change in a trend. Counterfactual analysis can be used in a mitigated form, adopting a less rigorous approach to dealing with project data and baseline trends, with the degree of complexity proportionate to the project’s ambitions and available resources. After all, if the group of “treated subjects” (i.e., project beneficiaries) concerns only (as is often the case) a few dozen people, a serious counterfactual impact assessment loses its meaning and scientific relevance.

The “trajectory change” in the estimated trend will therefore have a narrative and indicative value. However, it can be useful in producing some suggestions at the conclusion of a project, in combination with the more substantive elements of a report, related to more easily and objectively verifiable aspects (activities and outputs realized, beneficiaries reached, follow-up data on beneficiaries). To make impact estimation more realistic and truthful, we recommend:

Circumscribe the scope of the phenomenon being measured to that to which the project contributed most strongly and directly (thus increasing the level of correlation between the indicator and the objective being measured);
Compare the evolution recorded by project data against reference points as “close” as possible to the project population-target (a “quasi-counterfactual” situation);
Combine different and complementary comparison references, if possible, by “triangulating” different data and viewpoints to increase the reliability of results;
Include in the analysis, if possible, multiple moments of measurement (to set a trend), including “follow-up” measurements (e.g., after one, two or three years after the conclusion of the project);
Accompany the analysis with an assessment of the factors (positive or negative) that may have influenced the data and “trends” of the project and the references used.

For example, on a project dedicated to job placement for young people in the 15-24 age group, residing in an urban local organisations subject to social problems, changes in employment data of young people in the 15-24 age group recorded can be compared:

From the project on its beneficiaries (baseline vs. final data: “factual” data).
In the project intervention area (or in another urban local organisations subject to social problems), during the same period (a “quasi-counterfactual” figure).

A more detailed and specific example is provided at the end of theMonitoring and Evaluation Framework example.

The choice of comparative metric (or the simultaneous use of multiple comparative references) may vary depending on the availability of data. Differences between “factual data” and “quasi-counterfactual data” can be analyzed (and possibly weighted, or corrected) in light of other factors and variables that may have affected the two reference populations:

Positive factors – e.g., positive results from parallel initiatives in the local organisations (e.g., vocational courses, support for internships, tools for “matching” labor supply and demand…).
Negative factors-for example, economic difficulties of enterprises in the local organisations or worsening enabling conditions (e.g., decrease in resources allocated by government to education or social welfare).

The Theory of Change can assist in this weighting activity, as it provides a “mapping” of all the conditions necessary to bring about a desired change.

In cases where this is possible and the resources are available to do so, it is possible to initiate data collection activities within the scope of the project either on the group of beneficiaries (e.g., youth 15-24 from certain technical colleges) or on control groups that are as similar as possible (e.g., youth 15-24 from the same colleges, similarly distributed in terms of age, class, grade point average, gender, and household income of origin, or youth 15-24 from other technical colleges of the same type and size, located in neighborhoods with similar socioeconomic conditions).

Non-experimental impact assessment methodologies. There are various methodologies that are not strictly quantitative for measuring the impact of a project from its intervention logic, or its theory of change. These are non-experimental methods, which thus have less ambition for “scientificity” than counterfactual methods, but are aimed at formalizing the process of attributing change or observed outcomes to a given project or intervention. We mention in particular:

Contribution Analysis. This methodology starts with a reasoned and plausible causal theory of how change is believed to occur (intervention logic or theory of change). It then focuses on gathering evidence that validates that theory: looking for other factors that may have contributed to the observed outcomes, identifying and excluding (where possible) alternative or complementary explanations that may have led to those outcomes. To do so, it embraces different perspectives (e.g., context, other interventions, actors’ narratives, beneficiary feedback) designed to suggest different factors or “causal pathways” that may have led to or contributed to the outcome;
The Global Elimination Methodology (GEM) is based on the systematic identification of possible alternative explanations that may have led to the observed outcomes by drawing up lists of causes and the “modus operandi” linked to them, for each outcome of interest. It is then determined which of the causes occurred and which of the “modus operandi” were observed. Causes for which the relevant “modus operandi” was not observed are discarded, leaving only those causal explanations that have a genuine causal link. From this, the true effectiveness of the intervention can be more accurately contextualized and evaluated;
Realist Evaluation starts from the consideration that the effectiveness of an intervention depends not only on the cause-effect relationships of what a project or intervention produces, but on a plurality of mechanisms related to the context, the set of actors and a community, their historical, cultural, economic, human and social interactions and dynamics. In different settings and in different communities an intervention will indeed have different outcomes and effectiveness. Evaluation (which draws from quantitative and qualitative sources) thus seeks to map a set of hypothetical “mini-theories of change” that can capture the different dynamics and contextualize in this the intervention and its effectiveness;
Qualitative Comparative Analysis (QCA) compares different cases (configurations of possible different causal conditions), but which produced a similar outcome. It therefore analyzes clusters of factors present in different cases, but which produced the outcomes and impacts of interest, to define the degree of significance of each condition in achieving the desired impact, and to identify the simplest set of conditions that can explain all the observed outcomes, as well as their absence;
Process Tracing is a methodology for examining and testing a specific causal link, to assess whether the evidence is sufficient to draw a conclusion about the cause. It applies four types of tests of increasing “power”: the “straw-in-the-wind” test, which allows one to define whether or not there are conditions to exclude a priori a causal link; the “circle” test, which allows one to exclude a causal link, when the condition analyzed is present, but the expected result is not present; the “smoking gun” test, which, on the contrary, confirms the co-presence (and tendentially, the causal link) of the analyzed condition and the expected outcome; the “doubly conclusive” test allows (in very rare cases) to exclude the existence of other causal links besides the one identified.

Impact as “stories” of change. What has been illustrated so far follows a logical and structured pattern, more or less quantitative, based on the concept of “measuring” the change achieved with respect to and using more or less refined statistical techniques (varying according to the ambitions of “scientificity” of the evaluation process undertaken).

In some projects this scheme may be complex or insufficient to correctly and fully illustrate qualitative changes, unexpected phenomena and effects not defined in the initial metrics. For this reason, there are broader qualitative or untethered methods of measuring impact against initial “goals” (e.g. “goal-free” evaluation).

Again, a comprehensive, exhaustive and rigorous treatment of the topic is beyond the ambitions of this Guide. However, it is important to draw attention to the importance of qualitative and less structured aspects in measuring the impact of a project.

In operational terms, this means asking the following questions: how have the lives of beneficiaries (or beneficiary organizations) changed as a result of the project? What role did the project play in their evolution, their “history,” and their individual experience? In the perception of the beneficiaries (or beneficiary organizations), what would their lives and history have been like without the project intervention? Can these small individual “stories” in turn produce new small and striking “stories of change”? Through individual “stories” and points of view, is it possible to draw a line that identifies the project’s parameters of success and its weaknesses?

“Stories” can be collected and evaluated through various methods of qualitative analysis, already mentioned in the previous sections: interviews and focus groups; case study writing and narrative surveys; and more specific methods, such as“most significant change“ and systems for analyzing and graphing trends and qualitative changes.

This type of analysis adopts an empirical approach based on “induction,” that is, formulating general conclusions from particular cases. It should not be considered a “plan B” compared to other methodologies, as it may be able to capture different, deeper or at least complementary elements than more structured systems of analysis.

Another widely used qualitative assessment approach is theOutcome Mapping, which estimates the effects of a project or intervention through analysis of changes in the behavior, relationships, activities and actions of people, groups and organizations with whom it has directly worked. It is based on the assumption that all social change ultimately depends on changes in human behavior; and that the sustainability of all change depends on relationships among people. From this philosophy flows the object of the relevant evaluation monitoring system: 1) the vision to which the intervention is intended to contribute and (most importantly) the changes in behavior, action, and relationship that are intended to be achieved; and 2) the “boundary partners” of the intervention, i.e., individuals, groups, and organizations with whom the program directly interacts and over whom it has direct opportunities for influence. The resulting analysis is qualitative in nature and focuses on observing (and sometimes self-assessing) the behavior change of those working with the project. Although less rigorous, it can therefore be simpler and more applicable than other approaches.

An analysis through “change stories” of various kinds makes it possible to develop communication and dissemination material that is interesting and usable by a broad audience of specialists (by virtue of its depth of analysis), by partners and stakeholders (who can in turn make it their own and disseminate it), and by the broader audience of laypersons.

These aspects are relevant and appreciated in the context of European projects. Reporting and communication are interrelated aspects that respond to a common goal of accountability and transparency (accountability) towards institutions, citizens and its target community.

Deepen concepts and approaches on impact. For those who wish to address impact measurement and management methodologies in more depth and from alternative and complementary viewpoints, we recommend:

1. BetterEvaluation, a knowledge platform and global community dedicated to the improvement and dissemination of methodologies and tools for good evaluation of projects, interventions and public policies. It proposes in particular a tool(RainbowFramework) to better define and manage one’s own evaluation system, a selection of dedicated Guides, and a detailed description (with examples and an extensive review of in-depth links) of methods and approaches used in the world of evaluation;

2. The Evaluation Manual produced by the European Commission (DG INTPA). It focuses specifically on evaluation procedures and processes used on European projects in third countries, but contains valuable references to how to manage the evaluation process starting with defining the intervention rationale (third chapter, on approaches, methods and tools);

3. TheIPSEE (Inventory of Problems, Solutions, and Evidence on Effects), a dedicated platform curated byASVAPP, the Association for the Development of Public Policy Evaluation and Analysis. It offers a review of the problems that public policies should address, the solutions adopted, and the existing empirical evidence on their effects, accompanied by an impact glossary and an overview of the methods used;

4. Anextensive review of guides and tools produced by specialized organizations in the field of impact investing, to which we have devoted a separate discussion.Impact investing is characterized by a methodical and conscious mobilization of resources to achieve measurable impact in areas where there is a shortage of it (principles of intentionality, measurability, and additionality);

5. A guide developed as part of CIVITAS, a European Union initiative dedicated to urban mobility. Although with examples dedicated to the specific sector, it provides a very clear, comprehensive and general treatment of the topic of project and program evaluation.

6. A “user friendly” project evaluation manual developed in the U.S. (government agency National Science Foundation), which has a systematic, comprehensive and scientific approach to the subject of project evaluation.

Handbook

Programs

Insights

Contact us