Why we need to use the F word in evaluation

As an independent consultant in the cultural sector, I primarily focus on evaluation and research. My approach encourages critical reflection and surfacing learning throughout a project to inform its development – a formative rather than summative method. This requires multiple voices and lenses combined with honesty and openness. 

The notion of failure is pivotal to this approach. Yet, as an evaluator, I used to avoid the F word thinking it generated negativity, defensiveness and blame. When reporting on the outcomes of an evaluation, I was often encouraged to play down the things that hadn’t worked in favour of highlighting those that had. But I always felt that this failure to talk about failure was itself a failure. 
 
As Leila Jancovich and David Stevenson argued in their recent article, if we don’t learn from what we do, we repeat mistakes and things don’t change. Only by addressing failures can we do things differently and take more risks. This realisation has shaped my work as an evaluator. 

Escaping the trap of ‘project mentality’

There is a tendency in our sector to focus on the evaluation of a distinct project or programme rather than the impact of learning across an organisation. This mitigates against learning from failure. 

During the pandemic, there was a shift away from face-to-face evaluation towards capacity-building support and training. At the same time, I became interested in how we might move from project-based evaluation to organisation-wide learning. 

By adopting and embedding whole organisation approaches we were able to shift the dominant paradigm of overstatement, feel-good narratives and tick boxes and could begin to change the discourse of monitoring and accountability to one of critical reflection and organisational learning. This new approach would support a culture in which talking about failure was normalised and welcomed.

This shift in my thinking coincided with the outcomes of the Failspace research project and the work being carried out by the Centre of Cultural Value (the Centre) to develop a set of Evaluation Principles. I was interviewed for the research for the former (and later became a Failspace Champion to encourage conversations about failure), and I was a member of the advisory group for the latter. 

The co-created Evaluation Principles suggested that evaluation should be beneficial, people-centred, robust and connected, and these principles resonate with the Failspace findings. Both share a commitment to learning and change, the need to be open-minded and confident in acknowledging failure and, importantly, increasing the range and number of viewpoints, voices and narratives.

Avoiding absolutes 

In reflecting on failure and success, we are robust and open-minded which leads to honest and balanced evaluation without masking negative opinions or things that didn’t work. Further, it represents different narratives and perspectives, ensuring a more rounded picture of the programme of work. 

The Evaluation Principles emphasise the need for evaluation to be many-voiced. The view of an artist leading a project may be very different from that of the participant in it – both stories are valid. Good evaluation avoids absolutes and embraces complexity and multiple perspectives.  

Likewise, projects can succeed in one way and fail in another. Success to the marketing and communications team may not be seen in the same way by the local community. We are familiar with defining success factors when planning projects, but what if we were to consider failure factors too? 

I have encountered many projects and programmes that set out with unrealistic expectations and overclaim the impact. Arguably, this is encouraged by funders so can lead to a lack of honesty when reporting to them. We need to be braver and more realistic when setting outcomes, ensuring they are attainable and grounded in the capacity to deliver. That is not to say that we shouldn’t be ambitious. Still, we need to be honest about what’s possible within the resources available. 

How do we achieve this shift? 

It requires planning our evaluation work at the onset of project and programme development and establishing clear baselines against which to measure success and failure. It requires a mixed method approach to gathering data that balances qualitative with quantitative and explores how and why outcomes are attained rather than simply claiming them. It requires trust and the creation of safe spaces for reflection to ensure all voices are heard and valued.  

Above all, it requires us to resist the default of telling positive stories and be brave enough to confront the failures. Those leading organisations need to model it, and those of us supporting evaluation can encourage it. Using the Failspace resources and the Centre’s Evaluation Principles can provide practical tools to support this shift. 

The courage to disclose and share stories of failure can benefit us all. Success is to be celebrated. But being honest and open about our failures can help organisations and the sector learn, develop and grow. 

Susanne Burns is an independent consultant in the cultural sector and a Visiting Professor at the University of Sunderland. 
[email protected]
 www.culturalvalue.org.uk
@valuingculture

You can find the FailSpace tools on our website. If you have any stories, thoughts or feedback about the success or failure of using these resources or the Evaluation Principles, please let us know.

This article, sponsored and contributed by the Centre for Cultural Value, is part of a series supporting an evidence-based approach to examining the impacts of arts, culture and heritage on people and society.