Category: measuring your impact

6 Reasons to plan your evaluation EARLY

unravelIt is not uncommon for Project Managers to contact me halfway through their project or worse, near the end. Unfortunately evaluation is often an after thought – perhaps something that was hastily included in the funding application and then forgotten about until much later. Don’t panic if this is you – You can still create a useful evaluation, but you have missed some valuable opportunities. Planning your evaluation early enables you to:

1.   Compare your ‘Before & After’

If you want to measure the impact of your project or intervention then you should ideally take a benchmark measure first, so you can understand what changes your project brings about. e.g By finding out people’s knowledge about a certain heritage site before they come and visit, compared with what they know after their visit.

2. Adopt a participatory approach

Participatory evaluation is when your project stakeholders or participants work together as a group to develop the project’s measures of success. They are actively involved in assessing and make decisions, and can even collect data. It might be facilitated by an outsider but is very much focused in serving the project community. This approach cannot be taken retrospectively and needs to be planned in from the beginning.

3. Use your findings to improve the project delivery

By planning your evaluation early you can build in formative evaluation questions to help you assess your progress during your development or early stages of the project. This can provide great feedback to make tweaks to your practice and approach, to better achieve your project aims. e.g Heritage Educators might like to know which of their workshop activities are most engaging and fun so they can build in what will be popular with their audience.

4. Ask the right questions, get the right data

By the end of the project you have missed some great opportunities to collect evidence from your participants. By asking questions about peoples experience while they are having the experience you often get more realistic feedback. Response rates to surveys fall the more the time has past since the event. If you know your key research questions early you can include meaningful ways to collect the evidence you need to answer these at each stage of the delivery.

5. Get valuable insight to use in other areas of your organisation

Evaluating a project often prompts organisations to gather really useful information that they otherwise might not gather. Evaluation findings can feed into your other core areas such as audience development, market research, partnership building, fundraising, budgeting and promotion. E.g. Quotes from audience feedback can make convincing content for newsletters, or findings can be incorporated into grant proposals to prove your impact. By planning early you can help your evaluation to fertilize other areas of your work.

6. Ensure you have the right resources for the task

Evaluation should ideally be budgeted for when the program budget is being planned. Evaluation is not a quick and fast endeavor. Collecting and analyzing data across a robust range of indicators can be time consuming, Such as scrutinizing hundreds of questionnaires. It can also take specialist skills, such as leading a focus group or interpreting arts based outputs such as drawings. By building consensus in your project team about the value of evaluation early on you can help secure the finances and resources it needs to succeed.

Contact me now to see how you can get the most out of your project evaluation by planning it alongside your project planning.

Evaluation basics – a refresher

baseEvaluation is a way of proving and then improving. It involves asking research questions about your service or activities, collecting evidence to answer those questions, and then analysing and acting on the results.  If it’s been a while then here is a little reminder of the steps involved:

Step 1. Clarify your objectives

Evaluation always begins by deciding on a specific project to evaluate. What do you want to prove or improve and what will you do with the information? For example, you might want to find out how to improve your art workshops to build a bigger audience, or to understand and prove how people are benefitting from your heritage site beyond their visit. What resources do you have to evaluate?

Step 2. Choose your evaluation tools

The way you go about gathering evidence will be dictated by the questions you ask and the kinds of information you want to collect. They can include traditional techniques such as interviews, focus groups, surveys and observations. Arts and cultural organisations often gather evidence from work produced as part of the project, such as collaborative works, photographs and films. Do you want a participatory approach where you service users help guide the evaluation? There are pros and cons for each tool. Ideally you’ll need a robust sampling strategy and mix of tools, which often includes both qualitative and quantitative information.

Quantitative data is numerical or statistical data and helps you get an overall picture of what is happening from a large sample of people. For example what percentage of people would recommend your museum to their friends. Qualitative data is much more detailed information gathered from a smaller number of people in greater depth. It provides information on people’s motivations, experiences and their attitudes. It can explain the decisions behind people’s behavior.

Step 3. Collect your evidence

There are numerous methods of collecting evidence for evaluation. You will need to know that your capacity to collect and collate evidence is manageable. What time and money and skills do you have to collect evidence? Who will you collect evidence from, what methods are appropriate? Are you collecting evidence that can be interpreted effectively? You will need to ensure proper permissions as well as know your responsibilities under the Data Protection Act.

Step 4. Interpret the evidence and draw conclusions

Making sense of your evidence is the fun part, but it can require specific skills. Can you collate numerical data into tables, graphs or charts so it is easy to understand? Can you look for patterns, correlations and trends to answer your research questions without bias. You may need to use statistical analysis software or understand how to carry out thematic analysis of qualitative material. Have you answered your original research questions? Do you need to further research?

Step 5. Share your findings and act on the results

Evaluation is worthless if you do not use the results. Once you’ve shared your findings with your colleagues and stakeholders discuss what changes you can make to your service and delivery. Agree on a plan to incorporate your findings into your practice. Your changes may be slight tweaks or a radical improvement in the way your organisation meets the needs of your audience. If you can’t act on the findings yourself who do you need to influence? Your evaluation should provide an evidence base that helps argue the case for more resources to implement needed improvements. And of course, you should plan to evaluate how successful any new changes have been! When you become familiar with evaluation and see the benefits it will become a way of thinking and an integrated part of what you do.

Contact me now to see how you can get the most out of your project evaluation