Category: Evaluation Arts Culture Heritage

How to select the right External Evaluation Consultant (and avoid the wrong one)

Leave a comment

imagesPlanning an evaluation for your project is a significant step, whether it’s driven by your desire to assess your work or a requirement from funders. Often, bringing in external expertise ensures impartiality, credibility, and a professional approach, especially if in-house resources, skills, or time are limited.

Here’s a guide to help you choose the best evaluator for your project:

Preparation: Understanding Your Needs

Before you begin, ensure you’re clear on:

  • Purpose: What do you want the evaluation to achieve for your organisation?
  • Audience: Who will you share the findings with, and how will they be used?
  • Funder Requirements: What does your funder expect from the evaluation?

Key Questions to Ask When Selecting an Evaluator

1. Evaluator’s Experience

  • Have they worked in your sector or industry? What is their area of expertise?
  • Can they provide examples of previous evaluations, demonstrating the scientific rigor of their work? References are essential.
  • Are they experienced in engaging with your target audience? For instance, working with children, families, or older adults often requires specific sensitivities.
  • Have they evaluated similar projects that sought similar outcomes? Do they specialise in qualitative or quantitative methods?
  • Can they demonstrate strong written communication and reporting skills?
  • Do they understand the policy and operational context you work within?

2. Their Approach

  • What is their evaluation philosophy? Do they value collaboration with stakeholders or take a more external, “fixer” approach?
  • How do they propose tailoring their methodology to your project? Avoid evaluators with a “one-size-fits-all” approach.
  • What do they consider hallmarks of a strong evaluation? Do they emphasise using diverse, comprehensive measures to draw valid conclusions?
  • Are they committed to delivering actionable insights and fostering learning and improvement?

3. Building a Productive Relationship

  • Do you feel a strong rapport with the evaluator after initial discussions? Are they personable, adaptable, and communicative?
  • Be ready to share background materials such as funding applications, strategies, organisational policy documents and project plans to help them understand your work.
  • Ensure your organisation supports open communication during the evaluation process, including access to key team members.

Practical Considerations

1. Budget and Timeline

  • Share your evaluation budget and ask how they would structure their approach within it. In many sectors, evaluation budgets are approximately 5 – 7% of project costs.
  • Discuss timeframes: Will formative feedback be available during delivery, or will it be a summative report post-project? How else could findings be communicated?

2. Confidentiality and Ownership

  • Do they adhere to GDPR standards and maintain participant confidentiality? Will they sign a confidentiality agreement if needed?
  • Clarify who owns the findings and intellectual property rights. Typically, clients own the evaluation, but shared rights can sometimes apply.

3. Professional Standards

  • Confirm their professional indemnity and public liability insurance coverage.
  • Formalise the relationship with a written agreement detailing roles, responsibilities, and copyright arrangements.

If you’d like assistance commissioning an evaluator, refining your evaluation brief, or discussing your project’s needs, feel free to get in touch. Contact me at melhumphrey@me.com.

For further resources, consider the UK Evaluation Society’s guidelines on evaluation good practice. Download your copy at www.evaluation.org.uk.

6 Reasons to plan your evaluation EARLY

unravelIt is not uncommon for Project Managers to contact me halfway through their project or worse, near the end. Unfortunately evaluation is often an after thought – perhaps something that was hastily included in the funding application and then forgotten about until much later. Don’t panic if this is you – You can still create a useful evaluation, but you have missed some valuable opportunities. Planning your evaluation early enables you to:

1.   Compare your ‘Before & After’

If you want to measure the impact of your project or intervention then you should ideally take a benchmark measure first, so you can understand what changes your project brings about. e.g By finding out people’s knowledge about a certain heritage site before they come and visit, compared with what they know after their visit.

2. Adopt a participatory approach

Participatory evaluation is when your project stakeholders or participants work together as a group to develop the project’s measures of success. They are actively involved in assessing and make decisions, and can even collect data. It might be facilitated by an outsider but is very much focused in serving the project community. This approach cannot be taken retrospectively and needs to be planned in from the beginning.

3. Use your findings to improve the project delivery

By planning your evaluation early you can build in formative evaluation questions to help you assess your progress during your development or early stages of the project. This can provide great feedback to make tweaks to your practice and approach, to better achieve your project aims. e.g Heritage Educators might like to know which of their workshop activities are most engaging and fun so they can build in what will be popular with their audience.

4. Ask the right questions, get the right data

By the end of the project you have missed some great opportunities to collect evidence from your participants. By asking questions about peoples experience while they are having the experience you often get more realistic feedback. Response rates to surveys fall the more the time has past since the event. If you know your key research questions early you can include meaningful ways to collect the evidence you need to answer these at each stage of the delivery.

5. Get valuable insight to use in other areas of your organisation

Evaluating a project often prompts organisations to gather really useful information that they otherwise might not gather. Evaluation findings can feed into your other core areas such as audience development, market research, partnership building, fundraising, budgeting and promotion. E.g. Quotes from audience feedback can make convincing content for newsletters, or findings can be incorporated into grant proposals to prove your impact. By planning early you can help your evaluation to fertilize other areas of your work.

6. Ensure you have the right resources for the task

Evaluation should ideally be budgeted for when the program budget is being planned. Evaluation is not a quick and fast endeavor. Collecting and analyzing data across a robust range of indicators can be time consuming, Such as scrutinizing hundreds of questionnaires. It can also take specialist skills, such as leading a focus group or interpreting arts based outputs such as drawings. By building consensus in your project team about the value of evaluation early on you can help secure the finances and resources it needs to succeed.

Contact me now to see how you can get the most out of your project evaluation by planning it alongside your project planning.

Evaluation basics – a refresher

baseEvaluation is a way of proving and then improving. It involves asking research questions about your service or activities, collecting evidence to answer those questions, and then analysing and acting on the results.  If it’s been a while then here is a little reminder of the steps involved:

Step 1. Clarify your objectives

Evaluation always begins by deciding on a specific project to evaluate. What do you want to prove or improve and what will you do with the information? For example, you might want to find out how to improve your art workshops to build a bigger audience, or to understand and prove how people are benefitting from your heritage site beyond their visit. What resources do you have to evaluate?

Step 2. Choose your evaluation tools

The way you go about gathering evidence will be dictated by the questions you ask and the kinds of information you want to collect. They can include traditional techniques such as interviews, focus groups, surveys and observations. Arts and cultural organisations often gather evidence from work produced as part of the project, such as collaborative works, photographs and films. Do you want a participatory approach where you service users help guide the evaluation? There are pros and cons for each tool. Ideally you’ll need a robust sampling strategy and mix of tools, which often includes both qualitative and quantitative information.

Quantitative data is numerical or statistical data and helps you get an overall picture of what is happening from a large sample of people. For example what percentage of people would recommend your museum to their friends. Qualitative data is much more detailed information gathered from a smaller number of people in greater depth. It provides information on people’s motivations, experiences and their attitudes. It can explain the decisions behind people’s behavior.

Step 3. Collect your evidence

There are numerous methods of collecting evidence for evaluation. You will need to know that your capacity to collect and collate evidence is manageable. What time and money and skills do you have to collect evidence? Who will you collect evidence from, what methods are appropriate? Are you collecting evidence that can be interpreted effectively? You will need to ensure proper permissions as well as know your responsibilities under the Data Protection Act.

Step 4. Interpret the evidence and draw conclusions

Making sense of your evidence is the fun part, but it can require specific skills. Can you collate numerical data into tables, graphs or charts so it is easy to understand? Can you look for patterns, correlations and trends to answer your research questions without bias. You may need to use statistical analysis software or understand how to carry out thematic analysis of qualitative material. Have you answered your original research questions? Do you need to further research?

Step 5. Share your findings and act on the results

Evaluation is worthless if you do not use the results. Once you’ve shared your findings with your colleagues and stakeholders discuss what changes you can make to your service and delivery. Agree on a plan to incorporate your findings into your practice. Your changes may be slight tweaks or a radical improvement in the way your organisation meets the needs of your audience. If you can’t act on the findings yourself who do you need to influence? Your evaluation should provide an evidence base that helps argue the case for more resources to implement needed improvements. And of course, you should plan to evaluate how successful any new changes have been! When you become familiar with evaluation and see the benefits it will become a way of thinking and an integrated part of what you do.

Contact me now to see how you can get the most out of your project evaluation