Referendum as research

The EU referendum result is too close to call. Why?

Well, if we think of the referendum as a research project to investigate the propensity of the British people to remain in or want to leave the EU, then some golden rules were broken – both in terms of the question asked and the sampling.

Golden rule 1: Never ask a binary question when the answer is most likely to be ‘it depends’

If you were to ask someone ‘would you like to go to see Hamlet’, you could offer them a yes/no answer and in the event of a ‘yes’ walk away with a self-satisfied smile that Shakespeare is still a huge crowd-puller. But of course, in reality, the answer is ‘it depends’. It depends on which day of the week it will be / how much the tickets will cost / whether David Tennant will be in the cast / which venue it will be at / whether it will be live or screened, etc.

The complexity of remain/leave issues is such that the question should never have been asked of the public

Binary questions may be useful for the simplest of classification questions, but can be extremely misleading in relation to complex decisions.

The Government’s public ‘consultation’ (described as a referendum) on whether to leave the EU asked respondents to choose between two polar opposites. Suppose instead they had been asked to use a five-point scale from ‘definitely not’ to ‘definitely’ with a ‘don’t know’ option. How would that have affected the result?

Or suppose they had been asked to respond to a series of more nuanced alternatives, like:

• Leave the EU within two years at any cost
• Remain unless the EU refuses to… (take your pick)
• Leave if £350m a week will be invested in the NHS
• Remain if £350m a week will be invested in the NHS

How many would have responded ‘definitely’ to the first of these, I wonder? And how many of them would have preferred the last on the list?

Golden rule 2: Never ask a question when respondents have insufficient knowledge to give a considered response

How would a director of a gallery or museum feel if the public were to be asked ‘Should all museums and galleries be free of charge to enter?’ and then s/he was forced to respond to the will of ‘the people’, but would get no more money to make it happen? The director would no doubt be extremely keen for ‘the people’ to be hugely well informed about the financial and logistical challenges, and potential consequences that such a move would entail before rushing to a self-interested conclusion that something for nothing sounds like a great deal.

The complexity of remain/leave issues is such that the question should never have been asked of the public, the majority of whom were able only to give an emotional response to it.

It seems that the dishonesty on an industrial scale that characterised both Remain and Leave campaigns (but especially Leave), the agenda promoted by popular press, and the continual rubbishing of independent expert opinion made it almost impossible for voters to extract the facts on which they could have based an informed decision.

Golden rule 3: A self-selected sample cannot be deemed representative of a population with any confidence at all

With a random sample – where every member of a population has an equal chance of responding to a survey – it is possible to predict with a known level of accuracy, what a whole population of interest would have responded if we had asked them all.

But the 72% of the registered electorate who mobilised themselves to vote in the referendum is not a random sample of the UK population – far from it. So should any Government be assuming the will of the entire population on a single life-changing issue based on a self-selected sample?

Given the known likelihood of older people voting, compared with their younger compatriots, if voters were survey respondents they would be selected using a stratified sample based on age. If the referendum result was similarly adjusted for age, this would have undoubtedly tipped the Leave result into Remain territory.

But more than this, the Government is responsible for running the UK on behalf of the entire country – not just the self-selected sample. A disproportionate number of those whose lives will be most affected by it were completely unrepresented – the many millions of those aged under 18 who weren’t eligible to vote at all. They are not even recognised in the population of interest. The Government has a responsibility to look after their interests too. When the public elects a Government, that is their expectation. At what point did that change so that policy decisions that will determine their life opportunities will be taken only in the interests of the self-selected sample?