True or false: 91% of surveys about Bitcoin and crypto are totally wrong

When Tony Richards, the Head of Payments Policy at the Reserve Bank of Australia (RBA), read the recent survey results from Finder’s Crypto Report saying that almost one in five Australians owned crypto, he didn't believe it for a second.

However, the results had already been widely published around the country, gracing headlines for weeks. They even made their way into the recent Senate Committee on Australia as a Technology and Financial Center’s final report in October.

Welcome to the statistically dubious world of cryptocurrency surveys — an easy way for companies to get publicity by hawking survey results, but not necessarily a great way to stay informed.

The Finder survey from August claimed that 17% of Australians own at least one cryptocurrency — 9% own Bitcoin, 8% own Ether and 5% own Dogecoin.

Is the figure plausible?

Richards called these figures into question in his address to the Australia Corporate Treasury Association on Nov. 18, saying that he finds them “somewhat implausible.”

“I cannot help thinking that the online surveys they are based on might be unrepresentative of the population,” he said.

The Reserve Bank of Australia’s head of payments policy Tony Richards said that the growth of crypto in 2021 was “no doubt fuelled by influencers and celebrity tweets.”

— CoinzProfit (@CoinzProfit) November 19, 2021

He referenced “important segments of the population” including the elderly, people living in regional areas, and those without reliable access to the internet, that online survey panels “do not capture well.”

His point echoes a similar sentiment outlined by Dr. Chittaranjan Andrade in his 2020 report for the Indian Journal of Psychological Medicine, where he claims online survey samples are often unrepresentative, regardless of the subject.

Online surveys are completed only by people who are “sufficiently biased to be interested in the subject; why else would they take the time and trouble to respond?” he wrote.

But the Head of Consumer Research at Finder, Graham Cooke defended the methodology, telling Cointelegraph:

“The respondents are selected based on age, gender and location to create a sample which fairly reflects the results that would be expected from a full national survey.”

“We are confident that this produces a trustworthy sample which is representative of the population,” he added.

In the 15-page report summarizing survey results, there are only a few lines at the end to explain methodology/ It says: “Finder’s Consumer Sentiment Tracker is an ongoing nationally representative survey of 1,000 Australians each month, with more than 27,400 respondents between May 2019 and July 2021.”

The survey is conducted by Qualtrics, a Systems Applications and Products in Data Processing (SAP) company. Qualtrics’ website boasts, "in just ten weeks Finder lifted brand awareness 23 percent," but there was no additional information regarding survey methodology, and did not provide any on request from Cointelegraph.

A Finder spokesperson was able to confirm to Cointelegraph that “Qualtrics collects respondents from various panels and can be incentivized in different ways. Some are paid a small fee for their participation, some earn a charity donation, for example.”

Different surveys have estimates 2M people apart

This is not to single out Finder’s survey for particular criticism: There appears to be a new survey every day and often their findings are at odds with one another.

Take the YouGov survey commissioned by Australian crypto exchange Swyftx, which found that the number of Australians who hold crypto is closer to 25%. The July survey collected responses from 2,768 adult Australians, and the figures were weighted using estimates from the Australian Bureau of Statistics. This survey was found to be compliant with the Australian Polling Council Code.

However, both surveys can’t be correct. The population of Australia is 25.69 million. This means that Finder’s 17% of the Australian population equates to roughly 4.37 million people. Meanwhile, Swyftx’s 25% is about 6.42 million people. 

The difference between the two estimates translates to just over two million people — that’s more than the entire population of South Australia.

The numbers also don’t appear to be reflected on local platforms. Crypto trading platform Binance Australia told Cointelegraph that it had 700,000 users, Easy Crypto Australia said it had around 15,000 users, Swyftx has 470,000 users (many from overseas). BTC Markets has over 330 000 Australian users and Independent Reserve’s site claims 200,000 users.

Digital Surge, eToro, Coinspot, and Coinmama did not respond with user numbers.

Not all Australians use a local exchange to trade their crypto of course, but on the other hand, a significant proportion of users are signed up to multiple local exchanges. There appears to be a mismatch of hundreds of thousands if not millions between survey results and exchange accounts.

That said, Jonathon Miller, Australian managing director Kraken exchange, said that his platform came up with similar figures to Finder in YouGov market research in May.

The sample in that survey included 1,027 Australians aged 18 years and older, the data weighted by age, gender and region to reflect the latest ABS population estimates.

It found that ​​one in five (19%) Aussies have owned or currently own a cryptocurrency, and 14% (2.78 million) who currently have a crypto portfolio.

Speaking to the Finder survey, Miller said: “I don’t think it’s going to be that far off. The point is that these surveys are probably representative.”

“If those numbers aren’t exactly right today, they will be tomorrow. I think it’s true that one in five Australians have crypto.”

How many BTC do I need to pay you to say you trust BTC? 

One issue that could be affecting the results of crypto related surveys is that respondents to some of these surveys are actually being paid in crypto. 

On Nov. 18, a Premise Data survey of 11,000 participants across 76 countries claimed that 41% of people globally trust Bitcoin (BTC) over local currencies.

The catch was, a separate survey of Premise’s “contributors” two months earlier reported 23% of its contributor base have been paid in BTC, and since 2016, the data collection company has paid out over $1 million in Bitcoin via Coinbase to survey participants in 137 countries globally. 

Principal Research Fellow at the Melbourne Institute of Applied Economic and Social Research Nicole Watson told Cointelegraph that “paying someone Bitcoin to complete a survey about cryptocurrency would bias the result.”

“People who know what Bitcoin is and want some would be more likely to take part,” she said. In short,  they’re not going to be reflective of the wider population.

Cointelegraph reached out to Premise about its survey methodology, but received no response.

What makes a trustworthy survey? 

In Watson’s opinion, online-only surveys are not representative of the wider population.

“Recruiting a sample online is likely to bias the sample towards people who spend more time online, visit certain websites, or use certain apps, depending on where the invitation to participate placed and who might see it.”

She explained that someone’s participation in a survey could be influenced by who is running it, what it is about, how long it will take, and what (if any) incentives are offered — all of which may bias the results.

“For a new technology like cryptocurrency, you can see how many of these factors could lead to a biased result.”

For research conducted in Australia, a good way to tell whether the findings are trustworthy is by checking whether it has been issued an “Australian Polling Council Quality Mark.” In the UK, you can look to see whether the polling company is a member of the British Polling Council (BPC), and in the U.S. the National Council on Public Polls.

The Australian Polling Council says that any survey or poll worth its weight should include a “long methodology statement,” including additional information like weighting methods, effective sample size and margin of error.