Junk Science is Everywhere

You would really expect better from Prevention Magazine

You would really expect better from Prevention Magazine (image lifted from the linked article on io9)

Perhaps you remember the headlines a while back: “Eat Chocolate to Lose Weight!” Every week we learn about a new study that shows that X helps you lose weight. And right there is the first problem:

A study.

Singular. Let’s get something straight right now: A single study has never proven anything, ever. This is a fundamental part of science. When someone makes a discovery, it’s exciting. When enough other people confirm that discovery, it’s knowledge. “A study” is useful to guide future research and to provide fun anecdotes on “Wait, Wait, Don’t Tell Me”. But that’s all.

Back to the chocolate. The finding that chocolate helped weight loss was discovered in a laboratory study with the proper protocols, and published in a peer-reviewed journal. So, that’s real science, right? Even if it hasn’t been independently reproduced, isn’t it still important health news? You can’t blame the health press for jumping on something as sexy as “chocolate makes you thin”.

But then the people who did the study came forward and told the world that it was all bullshit. They’d done it to prove how easy it is to get junk science into the mainstream. Even they had not imagined how easy it would be.

Let’s start with the scientific study itself. It’s generally considered scientifically significant if the result of the test is less than 5% likely to be the result of random chance. Yep, it’s considered acceptable that one in twenty scientific experiments is incorrect just based on random chance. Madness? Not really, when you consider that all the studies in a field eventually have to work into an interlocking puzzle that forms a bigger picture. The studies that were incorrect either by blind bad luck or poor procedures get weeded out when others cannot reproduce the results.

But what if you test twenty things at the same time? Statistically now you’re very likely to hit a false positive. To quote the article:

Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result.

In the jargon of the junk-science industry, this is called “p-stacking”. An incredible number of the health claims you read are likely the result of this film-flam.

“But,” you might ask, “aren’t there systems to filter this shit out before it goes mainstream?”

Well… yes, but those systems are pretty much broken. First off, science is a discourse, and all new ideas have to run a gauntlet of “peer review”. Ideally, the peers may not agree with the conclusions, but you damn well better dot your i’s and cross your t’s. If you take shortcuts in your process, your peers will keep you out of the journals. In the major journals, the reviewers take their work really seriously.

But now there are journals that, for a price will publish whatever twaddle you wish to sell. While they claim to be peer-reviewed, the peers seem only to be reviewing whether your check clears, and have little interest in the scientific validity of your study.

Academia may not be fooled, but the fifth estate certainly is. Journalists who are trusted to sort through the garbage and bring important health information to their readers instead just blare the sexiest headlines. In some cases, the online comments by the readers of those articles ask the questions the so-called journalist should have asked before even running the story.

In the chocolate scam, they recognized another important fact: if the press release is actually written as an article fit for a magazine, even fewer questions are asked. It’s jut cut, paste, and print.

The press is making hay selling junk science to you and me. We trust them to vet the information they bring us, and they are doing a terrible job. It’s not just health science, but that’s where most of the crap seems to be flying these days.

So if what passes for journalism these days won’t ask the hard questions, we have to. Don’t change your diet because of “a study”. Even honest studies are found to be false later on, and damn few of the health articles we read are based on honest studies. (That “damn few” assertion is totally baseless. I have no statistics to back it up. But you were right there with me, weren’t you?)

For your homework assignment, I’d like you to Stop And Think when you see something on Facebook, especially in the health industry. Maybe do five minutes of research on the people making the claim. Then CALL THEM ON IT. Say, “Hey! I call Junk Science on you!”

Get double-serious when you read the shit in magazines. Let’s publicly shame the so-called journalists who dump this stuff out without asking the hard questions first. Demand footnotes. Check sources. Someone has to teach those bozos their jobs.

3

3 thoughts on “Junk Science is Everywhere

  1. I’m tempted to pull the Gell-Mann Amnesia Effect out of a hat, but I think even that doesn’t go far enough. Journalists are ill-suited for science reporting. The scientific method is alien logic in journalism, because good science is about numbers, replicability, and a systemic process. Journalists think in terms of anecdotes, black swans, and intentions.

    That was great news because it would have been awesome if true, and that’s pretty much the end of what made it great news. Take out the study—say some scientist theorized the same thing, or some new Jared lost two hundred pounds eating chocolate—and I’m pretty sure it still would have been news.

    • I didn’t want to recapitulate the whole article, but even magazines with fact checkers and some credibility completely dropped the ball. It’s a sorry state of affairs.

  2. Read about the study of studies which resulted in us moving schools from under “dangerous” EMF-generating power lines. Basically, they looked at 10,000 health effects from living near power lines, with a high confidence interval. But it was so many studies, that 2 of them showed health benefits from living near power lines, and the childhood leukemia and brain cancer scares turned out to be statistical flukes. Later studies showed busing kids longer distances to these schools were causing net deaths. Science works when conducted by scientists.

Leave a Reply

Your email address will not be published. Required fields are marked *