There are so many credentialed people on the internet with sufficiently differing views that it sometimes seems as if we could find an expert somewhere to support almost any harebrained idea. So how does a non-expert figure out the truth?

Most of us lack the time, training, and inclination to investigate most subjects sufficiently so we are often urged to adopt the consensus opinion. While an individual expert may have wild and crazy ideas, the consensus will most likely be an average informed view.

But it’s not that simple. Most of the time it is impossible for the public to determine the consensus opinion. What is usually labeled as consensus opinion is what media believe it to be. And the media are experts neither in the subject nor the sampling methodology needed.

But, even if we knew the consensus, a recent open access study put out by the AMISTAD lab at Harvey Mudd College in California points out an additional requirement for a valid consensus: The consensus must have been reached independently, and without coercion.

Bayes’ Theorem

That seems intuitively right, of course. But it’s more than intuition. The researchers provided calculations, based on Bayes’ Theorem, to demonstrate the process. Dependence between experts (that is, when experts base their judgements on the judgement of another expert) or any sort of coercive treatment causes the relationship between consensus and truth to fall apart quickly.

This study starts off by showing that, given unfavorable observer conditions, an emerging consensus gives a strong signal in favor of the hypothesis. One of the issues, however, is that in many cases, even with strong evidence, no consensus will form. But when a consensus does form, it is very trustworthy.

However, as soon as dependence (whether through influence or coercion) is added into the model, even in small amounts, the ability of a consensus to serve as an indicator of truth drops considerably. A consensus can easily form around a false idea if there are dependencies between the participating individuals.

This study is in line with previous work on the subject, which shows that oftentimes observational judgements tend to average out towards truth — but only when the observations are independent of each other.

The results also mirror what professor Jay Richards has suggested about consensus. Using climate change as an example, Richards gives us a list of 12 things to think about, to see if a consensus should be considered trustworthy. For example, he advises,

“Science says” is a weasely claim. “Science,” after all, is an abstract noun. It can’t speak. Whenever you see these phrases used to imply a consensus, it should trigger your baloney detector.

Jay Richards, “Politics Disguised as Science: When to Doubt a Scientific ‘Consensus’” at The Stream

Richards, like Allen et al., suggests that the process by which the consensus formed is of critical importance. While not infallible, a consensus that is arrived at by independent testimony of independent experts should be considered good evidence. But when people inside or outside try to apply political muscle to the process at any point, the consensus fails to be meaningful.

You may also wish to read: Can an 18^th century statistician help us think more clearly? Distinguishing between types of probability can help us worry less and do more. Bayes’ Rule can help us distinguish between types of probability if, for example, we have a positive screening test for a disease but no symptoms.