Tools to design an evaluation

In order to design and subsequently carry out an evaluation, appropriate data (based on the goals of the intervention and needs within the context) must be collected. Such data fulfils an important role, as it can be used to evaluate a crime prevention intervention in a systematic way. This means that a certain framework exists so that evaluation is a recurrent phenomenon.

Evaluations produce very useful results that can be used to improve existing practices. They contribute to better planning, more efficient resource allocation, lessons learned and identification of areas of improvement. At the same time, evaluations provide necessary information to support crucial decision-making about a intervention as it demonstrates whether or not a intervention has any impact, which is especially relevant when public money is concerned.

Practical templates linked to each of the steps can be found at the bottom of the page

1. Problem analysis

The problem analysis is an essential step within the preparation of a prevention intervention as well as its evaluation. It ensures you have a clear understanding of the crime issue at hand. This analysis delves into the problem for which your intervention seeks to provide a preventative solution and describes the nature of the crime or its harms. It include information on the scale, size, distribution, trends, perpetrators, victims that can be found in, amongst others, statistical or socio-economic data or through case studies.

A specific tool that can be used during the problem analysis, is the crime analysis triangle (see template). The crime analysis triangle is a tool that creates a systematic overview of both the risk- and protective factors of a crime issue/phenomenon. This triangle portrays the main factors for crime: a motivated perpetrator, a suitable victim/target and a location with little to no supervision. 

2. Develop objectives

The following step is deciding what your intervention wants to achieve by developing goals (i.e. objectives). This will be the key marker for the success of your intervention as objectives help to guide an intervention and set a benchmark for its evaluation.

A target tree is a useful tool to create a visual overview of the relationships between different goals of an intervention (see template). As an intervention rarely has only one objective, a target tree usually consists of a wide-ranging main objective and multiple sub-objectives.

The main objective is often too general, so we want to break it down into more specific and measurable sub-objectives. Sub-objectives are often short-term and include specific activities that are needed to create a difference. Achieving these goals is necessary to achieve the main objective. Finally, it can be helpful to include which actors are or should ideally be responsible for each sub-objective or activity.

In order to generate measurable objectives, they have to be SMART. Each letter of SMART represents a way of making objectives as specific as possible:

1. S = Specific: the goals must be clear and specific (what, where, how?) so they are not open to interpretation.
2. M = Measurable: the goals must be measurable either in a quantitative (numbers, percentages) or qualitative (opinions, perceptions or behaviours) manner.
3. A = Acceptable: Ensure that all stakeholders involved agree with the objective. An agreed-on and accepted objective has the advantage that it is supported by all those involved, which will motivate the parties to achieve the result that they have agreed on.
4. R = Realistic: It should be possible for those implementing the intervention to achieve the objectives. A goal that is too demanding will be discouraging. It therefore remains important to find a good balance between realism and ambitiousness.
5. T = Time-bound: set a clear start and end point at which achievement of the goal can be measured. Remember that in crime prevention, especially social crime prevention, effects can take several months or even years to manifest themselves.

3. Programme theory

When designing an intervention and its evaluation, it is important to understand how the intervention works. The programme theory will explain how and why a intervention is supposed to lead to the intended outcomes.

All interventions have an underlying programme theory (see template). Each initiative is based on an idea of how and why it will solve the problem (i.e. problem analysis) and how and why it will achieve its goals (i.e. SMART objectives)

4. Logic model

A logic model represents the relationship between the intervention’s key activities and the intended outcomes in a way that shows the underlying logic behind the intervention (see template). It outlines the necessary resources of the intervention (inputs), its actions that should lead to the intended outcomes (activities), the products or services created by the activities (outputs) and finally the changes created by the intervention (early and later outcomes).

Process evaluation

Evaluations can be carried out during different phases of a crime prevention intervention. A process evaluation examines how (well) the intervention is being implemented and whether everything is going according to plan (e.g. whether the target group is being reached or how the key actors are working together). This evaluation therefore (ideally) happens once an intervention is up and running.

Examples of key process evaluation questions include:

• Are the interventions executed according to the plan? Does every actor know what to do and how to do it properly? If actors work together, is the collaboration going smoothly?
• How many activities were successfully executed?
• Is the target group being reached? If not, how can this problem be tackled?
• Were the inputs sufficient to cover all planned activities?

Outcome evaluation

An outcome evaluation focuses on the effect of the prevention intervention and whether it meets the predetermined objectives and sub-objectives (e.g. whether the interventions reached their purpose, which interventions worked best and how the interventions can be improved in future prevention initiatives). It is usually carried out when the intervention is completed, so both the short-term and long-term effects can be investigated (see template).

Examples of key outcome evaluation questions include:

• What is the effect of the intervention on participants’ knowledge, attitudes, skills or behaviour? Are these outcomes sustained over time?
• What changes did the intervention introduce to the geographic area that was targeted?
• Were there any unintended consequences or outcomes from the intervention?
• What external factors positively or negatively influenced the effectiveness of the intervention and the outcomes that were delivered?
• What changes could be made to the intervention to improve its overall effectiveness?