5 Hats of an Evaluator

We were recently asked an interesting question about evaluation. How does the role of the evaluator differ from carrying out the evaluation? We thought we would share our answer in case it helped other people.

When an evaluator carries out an evaluation on a project, there will be a series of exercises, pieces of work and actions that the evaluator does in order to understand what they are evaluating, gather the data they need and evaluate the data according the outcomes associated with that project. Usually this will include the creation of the theory of change and evaluation framework, the creation of data capture resources, the data gathering processes, the reading of that data and then the production of the evaluation report at the end. Carrying out an evaluation is rather task orientated.

However, the role of the Evaluator is less task orientated and more purpose orientated.  We describe it as wearing a series of ‘hats’ throughout the evaluation process.

When the Evaluator wears the Travel Agent hat, their purpose is to do all the planning associated with the forthcoming evaluation. The client will know where they want to go and why and the ‘Travel Agent’ must bring to life all the finer details.  This includes what that destination actually looks and feels like, who they will bring on the journey, how they will get there, when they will travel, etc – it all forms the travel itinerary.

In the same way, the Travel Agent hat in evaluation teases out the need for the project, the outcomes and success indicators, the target audiences methodology for evaluation and an evaluation timeline – it all comes together in an evaluation framework. 

The Hunter Gatherer hat is worn by the Evaluator when they are looking for and collecting data, opinions, views, statistics and other evidence that will eventually prove whether (or not) a project has actually achieved the things it was meant to achieve. Some of these hunter gatherer techniques might include looking at reports, combining lengthy data sets, reading comment-books, creating and managing online surveys, conducting in-person field work, creating hoop-toss activities or drawing workshops for children, gathering blogs & vlogs, running focus groups, examining final end products for hidden values & meanings, or just simply, having conversations with people.  

At Flying Geese, we are particularly experienced in developing data capture techniques for people where a form is not suitable. Children, people with neurodiversity and people where English is a second language are often unable to use data capture forms so instead, we plan clay modelling exercises, Lego workshops, and vlogging activities to make sure that we hear from everyone. 

Once data starts coming in, it is time for the Evaluator to wear the Interpreter hat. This is where they read the data to understand what it is saying, identify any trends that are emerging & predict what might happen if those trends continue and make conclusions. Thorough evaluation will see the Interpreter combine and read multiple sources of data to draw together insight over a larger scale.

The Anchor and the Lighthouse hat is worn by only the most special of Evaluators and it is the hat that we take the most pride in wearing at Flying Geese. For us, there is nothing worse than an evaluation that says “If only the project had….. something amazing could have happened.” This isn’t helpful to anyone. Instead we prefer to be able ale to say “Because ….. happened, something amazing did happen!” 

The Anchor hat sees the evaluator communicate with the client and the project at all times. It gives real time updates about the data that is coming in, what it says and most importantly what can be done in real time, to make sure that the project maximises on the best results, minimises the bad results and is generally optimised from start to finish. It enables the client to make real-time changes to make their project better, on the go. The Lighthouse hat, sees the evaluator keep in mind the final objectives and outcomes of the project, to ensure that the little changes that they advise along the way ensure that the project comes into dock safely.

This is the most famous hat of them all, it sees the evaluators draw together all the data collected across the project and make conclusions as to whether or not the project has achieved what it set out to achieve at the start of the project. A thorough evaluator will not only read what the data says, but they will dig deeper to understand whether the mechanisms of change (the activated processes that made the outcomes happen) were optimised as best as they possibly could, and whether all risks and barriers were removed to ensure that the outcomes had their best chance of being achieved. They will also identify what else might have been achieved that was above and beyond what was initially hoped for. Usually the evaluator will bring all this information together in a glossy showcase report which clients can share, but sometimes videos or evaluation podcasts are produced instead.

Because we are also strategic business consultants at Flying Geese, we go to town when we wear our Welcome Home Party hat. We also identify short, medium and long term recommendations that will help shape their future, and then we work with the client to ensure they have the capacity and capability to put these recommendations into place.  As we have roots in audience development and marketing too, we often create celebration documents, which turn our evaluation reports into a series of digital content which enables the client to share their success stories with their audiences and stakeholders.

And so there you have it, the difference between the role of the Evaluator and actually carrying out the evaluation. If you would like to talk to us about how we might be able to help you in your evaluation, please email beckie.smith@flyinggeese.co.uk