Sunday, 17 November 2013

The Art and Science of Crap Detection

One of the ironies of modern culture is that, despite the remarkable advances of science and medicine, many people continue to believe utter nonsense.

The sea of dubious information has clearly been fed by the emergence of social media.  Anyone who uses Twitter or Facebook will be confronted with a continual stream of bold claims, shocking controversies, and supposed secrets for improved health, wealth and happiness.  Some of these claims come from genuine experts, some come from people trying to make a fast buck, and some come from fools.

Telling the difference can be frustratingly difficult.

Two examples.  I've just been reading a report in a newspaper about advances in cancer treatment. The same issue also includes an article about the powers of Chinese "internal healing".  Another source, this time an online magazine, juxtaposes a piece about recent advances in brain science with a quiz inviting readers to find out whether they are "left brain" or "right brain" thinkers.

In case you are not clear yourself (and who could blame you?), there is no compelling evidence in favour of either internal healing or left/right brain thinking.  They might be attractive ideas. They might conceivably turn out to be true at some point in the future. But, in the words of the late, great Carl Sagan, "extraordinary claims require extraordinary evidence".  And, at the moment, the evidence is missing.

The genius of social media is that it has democratised knowledge. Information is no longer the preserve of a select few.  We can all access vast amounts of information.  But democracy cannot work without an educated population.  It assumes that people have sufficient knowledge, skills, and understanding to make informed judgements about the decisions confronting them.

The simple fact is that the vast majority of people have not been educated to make such judgements.

The writer Ernest Hemingway famously said, "The most essential gift for a good writer is a built-in, shockproof, shit detector."  That inspired Neil Postman to claim:
As I see it, the best things schools can do for kids is to help them learn how to distinguish useful talk from bullshit.
Most schools do not do anything that remotely resembles this. Consequently, they send young people out into the world inherently vulnerable to bullshit.  And that seems rather shameful.

Of course, it is not just schools who have failed to prepare their charges.  Many universities force their students to complete 'learning styles' assessments, despite the fact that it is well-known that most experts on learning and the brain deny that such styles exist at all.  Even worse, an informal review of sports coaching education programmes in the UK by my colleagues and me revealed that all of them taught questionable or erroneous theories as scientific fact.

Remember Power Balance?  The bracelets that used "hologram technology" to "resonate with and respond to the natural energy field of the body", resulting in the improvement of sporting performance.  Sounds implausible?  Not so much that it prevented tens of thousands of sports coaches, athletes, and normal people from from around the world buying one, just in case they actually worked.

People bought the devices even after scientists demonstrated that they had no effect at all.  Television news items even demonstrated, with the most basic of tests, that the bracelets were useless.  But people continued to buy them, right up until the company that produced them went out of business, mainly due to a court ruling that the company should stop making unsubstantiated claims and that dissatisfied customers should be given a full refund.

Anyway, the company has relaunched under a new name, and people have started buying the bracelets again.

What is to be done?

Surely, the ultimate solution must be to change education programmes to better prepare people to do with the barrage of bullshit with which they will inevitably be confronted during their lives. This is as much to do with a sceptical mindset as specific techniques.

For a start, though, consider this list of questions from the writer Michael Shermer, which make up his "Baloney Detection Kit".


  1. How reliable is the source of the claim?
  2. Does the source make similar claims?
  3. Have the claims been verified by somebody else?
  4. Does this fit with the way the world works?
  5. Has anyone tried to disprove the claim?
  6. Where does the preponderance of evidence point?
  7. Is the claimant playing by the rules of science?
  8. Is the claimant providing positive evidence?
  9. Does the new theory account for as many phenomena as the old theory?
  10. Are personal beliefs driving the claim?

Crap detection can be framed even easier than this, as one simple but profound question:
How do you know?
If every student was taught to ask this whenever confronted by a new idea - no matter who offered it - they would be making a giant step towards being genuinely educated.


  1. Great post! I'm a school librarian with experience trying to push this message, and was shocked to learn that a lot of teachers are as ignorant as students when it comes to online information. I was just thinking today about the irony of the 'information age' - greater access to information has never been experienced yet modern benefactors barely scratch the surface, believing the first thing they see, and are also losing 'crap detection' skills their ancestors have/had.

  2. Excellent post Richard. The one thing that drives me most crazy on Facebook is the constant 'sharing' of advice or warnings, with no attempt to verify it first. They have no idea of the dangers of some of the claims. And it's the same people who do it all the time, it doesn't matter how many times they are shown places where they can verify information.

    As I understand it (and perhaps you can clarify from your knowledge), it goes against human instinct to be skeptical - at its most simplistic people have to think "There's a tiger, run!" for survival, not "There's a tiger, or is it? Maybe it's something that looks like a tiger, but actually is perfectly harmless, I'm going to go and check it out...". So it's a constant battle with ourselves and others, to question what is presented to us.

    A classic example was in the summer, I fell off my bike into a pile of stinging nettles, and my partner Neil (who is generally very skeptical and questioning of things) ran and got some dock leaves and we rubbed them over all the stings without thinking about it. Later, a thought struck me, and I asked him whether there was actual evidence that dock leaves help stinging nettle stings, and he said "Well all I know is that it's always helped when my kids have been stung." And then he realised he was himself saying the very type of thing that he judges users of alternative treatments for saying! It was a good lesson.

  3. A Popperian response:
    I do not know: my assertion was merely a guess. Never mind the source, or the sources, from which it may spring - there are many possible sources and I may not be aware of half of them; and origins or pedigrees have in any case little bearing upon truth. But if you are interested in the problem which I tried to solve by my tentative assertion, you may help me by criticising it as severely as you can; and if you can design some experimental test which you think might refute my assertion, I shall gladly, and to the best of my powers, help you refute it (Popper, 1972: p. 27).

  4. Thanks for your comments, Rob. I think you are correct that teachers are as ignorant (more ignorant?) than the students about social media.

    Add to that the generally uncritical style of teacher training they have experienced, and we are faced with a large and influential group that is very vulnerable to manipulation by increasingly sophisticated users of social media.

  5. Hi Vanessa, thanks for your comment. I suspect you are right. We are social animals, and with that comes a degree of gullibility. The social group is premised on large amounts of consensus and shared practices, and childhood, in particular, can be considered an extended period of extreme gullibility!

    According to Karl Popper, who is mentioned by Dave in the comment below yours, critical thinking is a relatively recent phenomenal in human evolution. We can see how recent it is by its rarity.

  6. As you know Dave, I am constitutionally inclined to agree with Popper!

    My favourite quotation of his, which should be tattooed on the forearm of every teacher, lecturer, and coach, is:

    "I may be wrong, and you may be right. Together we will get nearer the truth."

    (Source: Conjectures and Reputations, I think)

  7. The good thing about homeopathy is that you can't overdose. Well, you can fucking drown.

    I'd take priest, astrologers etc and put them all in a sack and hit them all with sticks.

    So, if someone gives you some bullshit easy answer to life's most difficult problems and you ask them 'do you have any evidence for that' and they respond, 'well there's more to life than evidence' (lift an eyebrow and pause for effect…) 'get in the fucking sack'

    All ripped straight from Dara O'Briain in a year sometime in the past (I think that's Harvard referencing)

    A telling insight as to where we pull our critical insights from Dave Piggott ;p

  8. Thanks for sharing the video and thoughts here, Richard. It's unfortunate that many feel that teachers generally don't understand social media as that is not my personal experience (full disclosure; I'm a teacher among other things). By all means, the journey for truth is more likely to be successful with others involved, particularly those in a position of leadership.

    I'm curious as to the link to "Coach MisEducation" here - I saw your post on Linked In that appeared to link this blog entry with coaching. However, the link is unclear to me after reading this blog entry. Is there more information you can share to connect the points you've made with the coach miseducation? I'm interested to see your take on this given your informative previous commentary. Thanks for listening, Richard.


  9. Outstanding blog and comments. Myth One sport year round for children helps get a DI scholarship. 2-8 times more likely to get injured. $1.2 BILLION spent on overuse injuries last year.Balance, health and being a good teammate are more important. Only 1% of all children who go to a 4 year college play at the DI level and only half of those play for free.Teach to get the children to think and be creative on their own in a safe environment. You cant microwave their development, it must eb slow cooked.

  10. Thanks for your comments, everyone.

    Beth, in response to your question about coach miseducation, I do refer specifically to examples of inappropriate content in national governing body programmes. Reviewing the content of these programmes was actually the motivation for this exercise in letting off steam! For example, we found one major sport organisation (sorry, I cannot name it for obvious reasons) whose coach education programme included almost every one of our list of identified dodgy practices. This is morally unacceptable, as people come on courses to learn how to coach, not to be peddled bullshit.

    The same is true, unfortunately, of teacher education programmes. I know from personal experience that many courses teach uncritically ideas that are simply not true, particularly ideas about the brain (Disclosure: I was a teacher, and then a teacher educator). Numerous studies around the world have shown teachers operate with many questionable or plain incorrect ideas about teaching and learning, and many of these must be traced to their preparation universities.

    As you might gather, this is probably a topic to which I will return!

    Thanks your contributions. It really is very much appreciated.


  11. This is the first time I am visiting your blog. I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here!