EXCLUSIVE: £300k grant sidesteps ACE procurement rules

A fledgling private research company being funded to assess and benchmark artistic quality will be charging at least £2k a year for organisations to use its system when a £300k trial comes to an end, AP has discovered.

Arts Council England (ACE) is currently spending £300k from grant in aid to give 150 of its National Portfolio Organisations (NPOs) the chance to trial a new digital ‘platform’, Culture Counts, and a set of standardised questions – referred to as ‘Quality Metrics’ – which are intended to assess artistic quality. ACE is “considering the possibilities for integrating [this system] into [its] monitoring and decision making processes” and is using the trial to “explore the feasibility of rolling the framework out across the wider portfolio” in the future. But if the system were to be adopted across the NPO portfolio, arts organisations would be faced with an annual fee to be involved unless ACE were to subsidise some or all of the £1.3m a year it would cost to give all 663 organisations access to it.

Public investment in the Culture Counts system has already topped £700k: £75k to a 13-strong Manchester-based consortium of arts organisations to develop a set of UK metrics to provide a “meaningful measure” of artistic quality; £27.5k to dance company Re:Bourne to develop metrics suitable for participatory work by with and for children and young people; £300k from the Nesta-led Digital R&D fund – of which ACE was the primary funder – to develop the Culture Counts approach to the point that it could be used to try out a set of survey questions; and the most recent £300k for a wider test among NPOs.

In response to AP’s request for a breakdown of the grant offer, ACE chose to respond under Freedom of Information rules. It has now released the project proposal to AP but details of the budget have been redacted. ACE has denied AP permission to publish or share the proposal document, saying: “Though we are obliged by the Freedom of Information Act to provide this information we are not under any obligation to grant permission to reproduce it. We feel the granting of copyright would be unfair on our applicant and inappropriate.” This response appears to be at odds with Chief Executive Darren Henley’s view that ACE has an “obligation to explain how we’re investing… to be as open as we possibly can; to make sure the processes we go through are as careful and as fair and as structured as we possibly can”. It means that only those who submit their own FOI request to ACE will be able to read the project proposal for themselves.

Despite its significant investment, ACE retains none of the intellectual property associated with the system it is trialling and appears to have no licensing rights over the expertise being developed – only the actual research questions being asked. The intellectual property belongs to the private company Culture Counts, which has a worldwide patent pending on the underlying system design and business model. This means that, unless other agreements are in place, ACE will be unable to recoup any of its investment in the system and will have little if any legal control over the way it is offered to arts organisations in the future, or what they are charged for its use when the trial ends.

Arts organisations involved in the trial must collect survey data from the public and from arts sector peers, together with their own self-evaluations. In return for the opportunity to participate in the trial, they must share their survey data with others and with ACE, and permit the research company to aggregate their anonymised data for benchmarking purposes. Organisations taking part in the trial are told that they “will not be required to pay for the licence”, but neither the publicity materials and correspondence issued by ACE to encourage them to take part, nor, according to one NPO involved in the trial, any subsequent briefings, have made any mention of the ultimate price tag when the trial is over.

When questioned about future fees for using Culture Counts, ACE was reticent, telling AP: “As this is still in a test phase, no decisions have been made about what might happen once it is completed.” But the proposal document obtained under Freedom of Information reveals that ACE was aware of the company’s pricing structure before it agreed its grant.

Under the grant conditions of its Digital R&D funding, Culture Counts committed to making “reasonable endeavours" to deliver a free trial to ACE NPOs during the 12-month period after grant-end. The trial was to have been implemented when the suppliers felt that “the system, and our support processes, had reached a level of maturity, robustness and user interface simplicity” to enable arts organisations to use it unaided. According to them, however, this point has still not been reached. Applying to ACE for more money to run a wider test, they claimed the system still needs “maturing” considerably, and ACE is paying for the current trial so that Culture Counts can “continue to make improvements” before a second trial is set up next year to fulfil the conditions of the Digital R&D grant. Any “disappointed NPOs” who were not successful at being included in the first trial will be invited to take part in this.

Culture Counts describes its system as being based on the controversial premise that “a standard set of transparent intrinsic impact dimensions, developed and tested with the sector” could be used “instead of the loosely defined, inconsistent and subjective assessment criteria currently used” to evaluate artistic quality. ACE feels that the Quality Metrics approach could “offer arts and cultural organisations greater insights into what people value about their work, as well as allowing them to benchmark against similar organisations”. But it could also be used by ACE “to make better decisions on how to fund, and what kind of organisations to fund” and to “review funded organisations based on concrete evidence”.

ACE is clearly committed to the concept, apparently dismissing concerns raised in a literature review it commissioned, which questioned the plausibility of aggregating survey data across organisations and artforms due to the “highly personal and situational nature of impact, and because of differences across the forms themselves”. One senior arts professional with specific expertise in this field shares this concern, telling AP: “Whether the results for a show in totally different markets with different audiences and different demographics would be valid for comparison and used for inferences would be a question…”

ACE’s money is being invested through a start-up company, Counting What Counts Ltd, whose directors are John Knell and Michael Chappell. Their respective companies have been active partners in the design and implementation of the Quality Metrics project from the start. Knell is a long-standing adviser to ACE, who has undertaken many projects for its senior team over the years. The two evaluation reports he co-authored in relation to the metrics work concluded that the approach had “great potential”, the most recent suggesting that “further refinement and testing of the metrics and Culture Counts system” was needed. Apparently unconcerned by any potential conflict of interest, ACE concurs, and has concluded that “the results of the work to date have been sufficiently encouraging to want us to invest in a sustained trial at scale”.

The current trial is being supported with grant in aid through ACE’s ‘managed funds’. These funds are not openly available. ACE allocates them at will, including to commercial organisations, with neither an application procedure nor a tendering process being a prerequisite. By giving a grant, rather than issuing a contract, ACE has side-stepped its own procurement policy which requires it to put all but its smallest contracts out to tender. It’s an approach that ACE defends on the grounds that: “The Quality Metrics project has been led by and created for the benefit of the sector. As such it is being funded through a grant rather than a tender for services.” By doing so, however, it runs the risk of a legal challenge, as State Aid rules prevent taxpayer-funded grants of this magnitude being used to give one or more commercial organisations an advantage over others.

ACE is the only UK arts council to allocate funds in this way. A spokesperson for the Arts Council of Wales told AP: “If we were to consider funding a commercial organisation we would need to be absolutely convinced that this offered the best prospect for an outcome that was of the necessary quality, provided public benefit and represented value for money… All of our grant funding (whether grant in aid or Lottery) is carefully scrutinised and assessed via a competitive application process.” At Arts Council Northern Ireland, commercial organisations can only apply for support if the proposed programme is primarily for the benefit of the public. A spokesperson said: “we are very tightly regulated and require a formal application/tender process.”

Competitive tendering processes are designed to ensure that a commercial supplier chosen to fulfil a contract is the one that can offer the most cost effective solution from a range of options. In fact Culture Counts is a relatively late entrant to the market, and other organisations and sectors appear to have more experience of data aggregation. In terms of technology, other brands of tried and tested survey software tools, web-based systems for mobile data capture and dashboard systems for delivering real-time visual data summaries – some of which Culture Counts has yet to develop – have been widely available for many years, including some specifically tailored to the needs of cultural organisations.

What’s more, technologies capable of doing what Culture Counts promises in the future – namely to “combine ‘instrumental’ data, such as financial information (earned income, such as box office takings, fees etc) with their returns on the core metrics for quality” – have been available in the UK for some years, with some NPOs already using them as part of their regular evaluation systems. Many ticketing system suppliers have experience in this area. At the Royal Shakespeare Company, for example, data from its box office system is merged with data gathered using online surveys, iPad interviews, survey apps, and pen-and-paper questionnaires to generate more meaningful information about its audiences. Jack Rubin, President of international CRM, ticketing, and fundraising software supplier, Tessitura Network, Inc., explained that the Tessitura system enables venues to conduct a survey post-purchase and/or post-performance, and then combine the data with other activity information, such as frequency of attendance or monetary value – or with other behavioral or personal data that is held by the box office. An analytics dashboard is provided to every licence holder.

ACE has made it clear that, should it decide to make the use of Quality Metrics a requirement for all NPOs, as it has with the audience development tool Audience Finder, then it “will identify a supplier through a formal tender process”. But it is difficult to see how this would introduce a level playing field for competitors to Culture Counts, who will not have had the benefit of ongoing ACE investment to refine and improve their system. One research company AP spoke to commented: “Any software tool designed for continuous use creates barriers to exit for clients. The longer they use it, the higher the barrier to exit.” If an invitation to tender were to be published at the end of the Culture Counts trials, they believe “other suppliers would be at a significant disadvantage”.