Honest lessons

As a former University lecturer who taught market research for nearly 15 years and has published books on the subject, I reckon I can distinguish between a reasonably robust piece of research-based evaluation and a load of self-congratulatory flannel, which in my experience is the most common type of evaluation report presented to funders. It was therefore refreshing to find that the evaluation by consultants Wafer Hadley Ltd  of the Artists Taking the Lead (ATTL) project in Yorkshire falls into the first of these categories, shedding light on some of the hard lessons that were learned through the implementation of this consortium-based project, as well as presenting some (arguable… as these things always are) estimates of audience figures and reactions to the work.
Collaborative working is, of course, a very topical theme, and the more we can learn about its advantages and pitfalls the better. Grant schemes these days often include a requirement for collaboration, yet evidence of the pros and cons of this style of working in arts settings is difficult to come by – or maybe I should say evidence of the ‘cons’. When project case studies are written and reports submitted to funders, it is in neither party’s interests to wash their dirty linen in public, so the interpersonal and organisational traumas that can beset attempts by independent organisations to work together for the first time tend to go unreported and lessons that could be learned lie dormant. The particular strength of the Wafer Hadley report is that it doesn’t shy away from making some very valuable learning points – which is exactly what evaluation reports should do.

The importance of artists buying into the publicity machine needed to promote their work to a mass audience is one good example. Five days before the Leeds-based ATTL event was due to take place, the Yorkshire Post’s arts correspondent Nick Ahad wrote a piece entitled ‘The £500,000 arts spectacular we still don’t know about’,  bemoaning the fact that “no-one really knows what it is” and revealing “there are increasingly sceptical voices being heard even from within the arts community.” Ahad reported that the concept of the lead artists, the Quay Brothers, was to “keep things secret and then have a big reveal and have that great big impact with everything appearing at once”. It’s just the sort of pre-launch PR strategy that leaves arts marketers in despair, and surely wouldn’t have been countenanced by a venue working on a box office split. The report also mentions some serious leadership and branding issues associated with collaborative work – both of which should cause others about to embark on consortium projects to pause for thought.

In my opinion, this type of information is of far more value to AP readers than set of figures documenting how many people attended an event. Such figures rarely stand up to close scrutiny. In this case, the estimated audience over three days of live performances was 20,000, but in the absence of comparators, the figure is meaningless. Respondents on average gave the event a four-star rating (out of five), but 37% gave it 3 stars or less. Is that good or bad? Unless benchmarks are available, who’s to say? The words ‘good’, ‘interesting’ and ‘different’ were used by the audience to describe the work, but so too were the words ‘weird’, ‘strange’ and ‘confusing’. Does that matter? The event was “successful in reaching young people, but it was less successful in winning their wholehearted approval”. Is that a cause for celebration or concern? It’s all a matter of interpretation, and actually I don’t know enough about the aims of the project to pass judgement on this.

So Wafer Hadley are right when they say AP had an agenda for the selection of material for our news piece about their report. The agenda of ArtsProfessional has always been and will always be to share information that will help the arts sector manage its meagre resources as effectively as possible. I think the Yorkshire ATTL report is well worth sharing, and that the authors should be congratulated for a piece of work that is both robust and honest. It’s concerning that this useful report has been published deep in the bowels of the Leeds Canvas website under the ‘Blogs’ tab, where no casual surfer will ever find it; and disappointing that the authors think our coverage of it was misleading (but we did link to and reference the full report online, for those who want more details). I think the authors should be proudly standing up for a quality piece of evaluation work that contributes a lot to our understanding of consortium working: only by sharing lessons such as these will arts organisations be able to avoid the pitfalls that tripped up their predecessors.