Red Flags of Bad Science

Like in any other domain of human endeavor, there is a spectrum of quality and accuracy in science.

• For 30 years "science" and physicians promised the public that smoking cigarettes was good for you.
• For 40 years "science" and the FDA put out the ridiculous food pyramid recommending the public eat a diet mostly of bread, pasta, and dairy.

Red Flags of Bad Science - Limitless Mindset

Because dentists never lie...

Sugar for will power? WTF!

A hundred years old?

Smoking is good for you!

"A health cigar"

"No curative power is claimed..."

Would you like some lung cancer with that?

#FAKENEWS

It must be healthy because... Protein!

Feeling lucky?

If she smokes, she...

As long as they're fresh...

Aspirin AND Heroin

Heroin for sale by Bayer

Vitamin... Donuts?

Cocaine for toothaches

Asbestos! That's what you really want in your house...

Has science cleaned up its act in recent times?

In 2015 Richard Horton, editor-in-chief of Lancet, the UK’s leading medical journal, stated:

“The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Science has taken a turn toward darkness.”

In 2010 Dr. Marcia Angell of the prestigious New England Medical Journal was equally disparaging of the state of science:

“It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of the New England Journal of Medicine”

I'm not saying that science is useless but that it should be scrutinized and just because something is labeled "science" you should not turn off your critical thinking skills.

Go to the Source

Probably the most common mistake that the layperson makes is reading an article online about science and assuming that they are doing meaningful research. A well-meaning person will Google a subject like...

• Climate change
• Anti-aging
• IQ
• Or nutrition

...they will read the first few articles they find and give themselves a pat on the back for acquiring a modicum of expertise on the subject.
While not totally useless, this information consumption really doesn't count as researching the science on a topic. The same applies to listening to podcasts or watching YouTube videos.
Researching the science means actually reading scientific studies, meta-analysis papers, books, and PubMed abstracts. Which are full of...

• Stale jargon
• Superfluous acronyms
• Scientific terminology
• And not exactly inspiring verbal prose

...they are not fun to read but if you want to understand a scientific topic, it's crucial to do.

Lifehack: Next time you find yourself in a debate with someone over some scientific topic. Just ask them when was the last time that they read a book specifically on the topic of contention.
Really? You don't think IQ is a meaningful scientific measurement of intelligence? What was the last book that you read on IQ?

Almost always they will have NOT read a book on the topic. So then you can point out how uninformed they are on the topic. As a rule in life:
If you haven't read a book on a topic, you really don't know what you're talking about in regards to it.
They have so little commitment to veracity and empiricism that they are not even willing to commit a couple of solid hours to understand the topic.
Then you can point out that their position is really just an arbitrary preference and that you guys debating this issue is tantamount to merely arguing whether chocolate or vanilla ice cream is better.
Obviously, this argument only works if you have read a book on the topic. So better read up on the topics you anticipate debating!

Mainstream Media Misinformation

You may think that the New York Times, Huffington Post, or The Guardian have high standards when it comes to fact-checking and accuracy but I'll suggest that after reading a New study finds... type of story in these publications go and read the study itself.
Often it's just a Google search away. If you take the time to do this often you'll find a blatant error or misrepresentation of the science in the mainstream publication, particularly if there are political or financial implications of the science.
The job of journalists should be, to tell the truth, but more often they lie. The excellent book Trust Me, I'm Lying by Ryan Holiday explains why...

Information Profiteers

There's a necessary class of merchants of science or science writers that people like myself, Dave Asprey, and Steven Kotler fall into. Our job is to make science more understandable and useful to the public but we make mistakes from time to time, our fact-checking is not perfect and our incentives are often not aligned with our earnest desire to educate people on how to use science to live better. I call us a necessary class because scientific publications are quite boring and nearly indecipherable to the general public, without marketing-minded people repackaging science it would not make it into the mainstream. I think we can all agree that we would like science to be available and understandable to all so that it can be a democratizing force in the world.

• Affiliate sales of products
• Sponsorship
• Advertising revenues

Often these income sources create a perverse incentive. If it's clear that a writer has a financial incentive that makes them partial to a particular product or supplements you should further scrutinize their portrayal of the science surrounding it.

Ads. Online advertising creates bad incentives for writers to put out a great quantity of work instead of quality research. If you're reading a science article on a website that has a bunch of ads in the sidebar and header and below the article that is further reason to question it.

Ghostwritten. Writers who know what they are talking about almost always put their real name and face on their work. They are proud of their work and stand behind it. There are a lot of quintessential content farm websites that churn out a great quantity of articles about health, science, and supplementation. Invariably, the writers are invisible, they are often outsourced digital laborers working for a few dollars an hour in a place like the Philippines or India.

Lack of author credentials. Writers who know what they are talking about will have some relevant credentials and experience.

Scientists will often spend decades myopically focused on a singular scientific niche, they'll understand it profoundly but often they'll lack holistic knowledge of how that science could be useful to laypeople. The writer's job is to be a scientific polymath and a lot of times we can articulate applications of the science that can help people.
However, if you're reading an article online that contains one or more of these red flags that should trigger your skepticism. If you really want to understand something, if you're making a crucial decision based on science, go to the source.

Bad Science

However, sometimes the source publication itself is bad science. Consider the case of the recent (hilarious!) hoax paper...
The conceptual penis as a social construct
...that actually passed the peer review process and was published in a mainstream social science journal, Cogent Social Sciences.

The authors specifically wrote the least scientific paper that they could. They specifically wrote a paper that should have been rejected by a scientific journal. A few examples of the absurdities it postulated:

Manspreading — a complaint levied against men for sitting with their legs spread wide — is akin to raping the empty space around him.

We conclude that penises are not best understood as the male sexual organ, or as a male reproductive organ, but instead as an enacted social construct that is both damaging and problematic for society and future generations. The conceptual penis presents significant problems for gender identity and reproductive identity within social and family dynamics, is exclusionary to disenfranchised communities based upon gender or reproductive identity, is an enduring source of abuse for women and other gender-marginalized groups and individuals, is the universal performative source of rape, and is the conceptual driver behind much of climate change.

Destructive, unsustainable hegemonically male approaches to pressing environmental policy and action are the predictable results of a raping of nature by a male-dominated mindset.

The authors intentionally made it as absurd and anti-scientific as possible, from the Skeptic.com article where they admitted their hoax...

After completing the paper, we read it carefully to ensure it didn’t say anything meaningful, and as neither one of us could determine what it is actually about, we deemed it a success.

Not only is the text ridiculous, so are the references. Most of our references are quotations from papers and figures in the field that barely make sense in the context of the text.

Nearly a third of our references in the original paper go to fake sources from a website mocking the fact that this kind of thing is brainlessly possible, particularly in “academic” fields corrupted by postmodernism.

From Cogent's Facebook page (Now closed)  -  Shouldn't scientific rigor be at "the heart of everything"?

Cui bono?

When researching science, it's important to ask who benefits? Who is getting paid?

Many journals like Cogent Social Sciences operate with a pay-to-publish model. The authors of The conceptual penis paid $625, they published under fake names as part of a fake social research group, which Cogent Social Sciences did not catch. So for $625, approximately the same amount you would have to pay to score some cocaine, get drunk on champagne, and bang a hooker in a chintzy hotel room in Vegas, anyone's nonsensical opinions can be transformed into "science."

How many other papers that have appeared within the pages of Cogent Social Sciences are similar pseudo-scientific nonsense? I would wager probably most of them.

Moralizing Should be Scrutinized

You should be very suspicious of science or scientists that make moral statements. Science and morality should not necessarily be separated like the church and state (or matter and anti-matter in the warp drive of the starship Enterprise to use a pseudo-scientific metaphor) but the more moralizing statements that bracket the science the more you should be suspicious of it.

Case in point, The conceptual penis is chuck full of moralizing statements - another major red flag that the peer reviewers at Cogent Social Sciences ignored.

To quote the hoaxers again

That is, we sought to demonstrate that a desire for a certain moral view of the world to be validated could overcome the critical assessment required for legitimate scholarship. Particularly, we suspected that gender studies is crippled academically by an overriding almost-religious belief that maleness is the root of all evil. On the evidence, our suspicion was justified.

Luckily for Biohackers the science of peak performance, health, and antiaging is not very politicized relative to a field like social science which is just a minefield of misinformation and propaganda. Let's keep it that way!

 What (really) is Pseudoscience?

You probably hear the word pseudoscience bandied about quite a lot.
You also hear its cousin pseudointellectual being used.
A whole lot less frequently you may hear the word I like even more than these two, sophistry.
And I'm sure you're already sick of hearing the buzzphrase Fake News, which is definitionally quite synonymous with pseudo-intellectualism.

Accusations of pseudoscience or pseudo-intellectualism almost always are a part of some political or ideological narrative. Next family reunion, bring up a scientific subject and it won't be long before it devolves into an ideological or political standoff over the dinner table.
Whether effective or ineffective, accusing the other side of pseudo-veracity is one of the most commonly used debate tactics.

How exactly would you know the difference between actual science and pseudoscience?

Actually, it's quite simple if we look at the definition of pseudoscience

Pseudoscience purports to be an accurate description of physical law despite either 1) consistently failing experimental verification or 2) being unfalsifiable.

I think that the second part of the definition is most interesting. Pseudoscience or Pseudointellectual movements are fundamentally unfalsifiable.

They make assertions about life, society, or the world that cannot be disproven.

What are some examples?
If I say...
There are exactly 2.4 million rabbits in France at this moment.
...can you disprove it?
No.

Can you disprove that there is NOT an invisible dancing penguin in the sky named Pete who gets very sad if you have sex with the wrong person?
Not really.

Okay, those examples are pretty silly. What are some more practical ones?

I can't disprove that God created the universe 10,000 years ago because God in his infinite power would have the ability to create a universe instantaneously that appeared to be 13 billion years old.

So creationism or intelligent design is Pseudoscience.

As was postulated in a hilariously stupid "scientific" paper published in the journal Cogent Social Sciences...
Can I disprove that I, as a male, am NOT causing climate change by raping the empty space around me?
It's so absurd that I can't disprove it.
So social science is, let's be honest (mostly), Pseudoscience.

Can you disprove that redistributing the wealth of the top 1% of humanity to the rest of the world would create a global utopia?
Well, even though it's been downright disastrous almost every time it's been attempted on a smaller scale, it's never been attempted on a global scale. So you can't disprove it.
Thus the economists and professors that advocate for Globalist Utopian Socialism (or UBI) are Psuedointellectuals.

So next time you find yourself in the middle of a debate on an ostensibly scientific subject or you read in a magazine that some new groundbreaking study discovered something counterintuitive...
Just ask how what they are saying could be disproven.
If disproving it would be extraordinarily difficult or impossible then it's unfalsifiable. It's NOT a scientific or intellectual position, it's just a preference.

 Recency

Newer scientific studies should be viewed as more credible than older ones. In researching the meta-analyses of Nootropics that I write I generally disregard studies that are over 30 years old.

In April of 2017 Pubmed announced they are standing up for science by publishing conflict of interest information on the study abstract page instead of allowing it to be buried behind an academic paywall.

This resulted from 62 scientists and physicians, and five United States senators petitioning the National Library of Medicine & National Institutes of Health to make public the funding sources of studies. This makes me more optimistic about institutions and politicians — in the news, it seems that all we hear about is how corrupt and incompetent institutions and politicians are but sometimes they act in our best interests!

So it's reasonable to view science published after April of 2017 as a little more credible.

🚩 Red Flags of Bad Science

To quote Maciej D. Zatonski on Quora:

Here are some classic examples that indicate that what you are reading is b******t:

- No control group
- No double blinding
- Results not replicated
- No peer review
- Sensationalized headlines
- Bonus: Betteridge’s Law of Headlines says, that if the title has a question, the answer to that question is “No” [1]
- Conflicts of interest
- Misinterpreted results
- Causation assumed when only correlation exists
- Conclusions not drawn from results
- Too small sample or sample not representative
- Selective reporting of data
- Claimed “Galileo’s Gambit” (results from a single study that contradict mainstream science are claimed to be the only correct ones; discredited authors are claimed to be “persecuted” and therefore right)
- Anecdotes used instead of evidence (“I know there is no evidence homeopathy works, but it worked for me!”)
- Changes of conspiracy and collusion
- Stressing or appealing to status or authority of someone (real or fictitious)
- Repetition of discredited arguments
- False dichotomy / failure to realise there are more than only two opposing outcomes
- Cherry picking of data / ignoring majority of evidence (3 studies show that climate change is not real, while ignoring 3000 studies that show otherwise)
- Wishful thinking (there is no evidence for existence of gods, but we wish they were real)
- Appeal to ancient wisdom (“it must be true because many people were saying this for a long time before”)
- Use of pseudoscientific language
- Daily Mail articles / Fox “news”
- Discrediting an argument because it comes from a person you dislike
- Appeal to nature (“it is natural, so must be healthy”; well… earthquakes, scorpions, cyanide and botulinum toxin are natural too!)
- False balance - counter argument to scientific fact claimed to be of equal value/weight to a counter-argument (theory of embryological foetal development vs stork theory of human birth; theory of evolution by natural selection vs creationism, etc)
- Appeal to common sense or alarmism (usually done to receive extra funding for pointless research)

Dr. John Ioannidis - The Bad Science Expert

A Greek-American doctor; over the past few decades, he has very methodically and non-sensationally shown the light on bad science.
An exhaustive article that appeared in The Atlantic interviewed him thoroughly at the University of Ioannina Medical School campus

Ioannidis... delivered what felt like a coup de grâce: wasn’t it possible, he asked, that drug companies were carefully selecting the topics of their studies—for example, comparing their new drugs against those already known to be inferior to others on the market—so that they were ahead of the game even before the data juggling began? “Maybe sometimes it’s the questions that are biased, not the answers,”

On the integrity of the scientific process

We think of the scientific process as being objective, rigorous, and even ruthless in separating out what is true from what we merely wish to be true, but in fact it’s easy to manipulate results, even unintentionally or unconsciously.

The science journal Nature concurred in a 2006 editorial:

“Scientists understand that peer review per se provides only a minimal assurance of quality, and that the public conception of peer review as a stamp of authentication is far from the truth.”

Double-Blind, Randomized, Placebo-Controlled, and Pre-Registered

You've likely heard before that the gold standard of science is the double-blind, randomized, placebo-controlled study.

• This means that approximately half the subjects are given a placebo. This should prove that the drug or supplement being studied outperforms the placebo effect. This is meaningful because the placebo effect is powerful!
• The patients are not told if they are receiving a placebo nor are the researchers told if they are administering the placebo or the real thing. You can imagine if the researchers know they are giving a placebo to an unwitting patient they will act a bit bashfully which the patient may notice, blunting the placebo effect. This is why it's crucial that studies are double-blinded.

The Replication Crisis

As Dr. Rolf Zwaan explained, it was discovered in 2015 that out of 100 recent studies in the fields of cognitive psychology and social science, less than a third were actually replicable.

Dr. John Ioannidis from the paper Why Most Published Research Findings Are False:

"Several methodologists have pointed out... that the high rate of nonreplication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study..."

The best solution to the replication crisis would seem to be Pre-Registration, meaning that...

• The hypothesis, study design, and methodology are Pre-Registered with journals before conducting the study.
• This keeps the researchers focused on determining if the data confirms or denies the hypothesis.
• This prevents researchers from expressing a bias in research or searching for a desired result.

Dr. Ioannidis goes on to clarify the place of Pre-Registration in studies:

"In some research designs, efforts may also be more successful with upfront registration of studies, e.g., randomized trials... Registration would pose a challenge for hypothesis-generating research. Some kind of registration or networking of data collections or investigators within fields may be more feasible than registration of each and every hypothesis-generating experiment. Regardless, even if we do not see a great deal of progress with registration of studies in other fields, the principles of developing and adhering to a protocol could be more widely borrowed from randomized controlled trials."

Pharmaceutical Science

In 2016 Forbes listed big pharma as the most profitable among major industries - beating out banking, financial services, tobacco, and the IT industry. This creates an appreciable (and irresistible?) incentive for these firms to yield their influence to pervert science.
Consider the sordid history of Bayer Pharmaceutical which has recently merged with the globally hated Monsanto.

It would be difficult to call the current iteration of the company evil based solely on their participation in WW2 70 years ago, however many still consider it one of the least benign corporate citizens in the world because of its Factor VIII scandal in the 1980s; when it turned out that a product of their's infected 6,000 hemophiliacs with AIDS they paid out a $600 million settlement and then outrageously proceeded to continue to sell the product in other international markets, infecting unwitting people around the world with the deadly virus.
I don't ascribe to the fringe conspiracy theories about the purposeful evilness of these companies; I think it's a combination of the perverse sticks and carrots that our globalized world offers in combination with the outgroup preference that a corporate culture engenders naturally amongst its employees.

Ben Goldacre an Epidemiologist points out that

Industry funded trials are four times more likely to have a flattering result

The book Adaptogens is especially critical of pharmaceutical science:

“The great majority of published research is so deeply flawed that it should be considered essentially worthless.” So wrote John Ioannidis, PhD, 
(p. 54)
Ioannidis singled out the following types of studies as being particularly likely to lead to a worthless result:
Studies with a small sample size;
Studies that consider a small number of possible effects;
Studies whose outcomes are poorly or subjectively defined;
Studies in which financial conflict of interest is a factor;
Studies in which the researchers are prejudiced by being unduly wedded to a particular outcome;
and Studies of a topic that is currently “hot.”
(p. 55)

John Ioannidis clarified further on conflict of interest in studies

"The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true."

From The Atlantic article:

...drug-company research wasn’t measuring critically important “hard” outcomes for patients, such as survival versus death, and instead tended to measure “softer” outcomes, such as self-reported symptoms (“my chest doesn’t hurt as much today”)

Not everything pharmaceutical is bad some of the best smart drugs are products of pharmaceutical companies, but where the moral hazard is so great you should be skeptical.

 Be Suspicious of Bias-Confirming Science

You want to scrutinize science that underlines the obvious moral position of the author or publisher. Again, a lot of what you see published is opinion masquerading as science. Examples...

• A lefty Liberal arts college produces social science saying that women are better at engineering (or whatever) than men.
• Evangelical Christians produce an Intelligent Design Biology textbook.
• A vegan that publishes nutrition research that eating meat causes inflammation. (I'd be more likely to believe this last one!)

I'm not suggesting that you disregard science that supports a preferred bias.
I'm suggesting that when you are reading a study, Google search the authors, publication, or establishment and just take about 30 seconds to look at their social media.
If their social media has a really strong ideological or moralizing bent that the study coincidentally supports that should be a red flag, I'll encourage you to further scrutinize the study.

1. Is the study focused mostly on the data?
2. Or is it full of a bunch of moralizing opinions?
3. Do the researchers admit in the study that they were particularly surprised by any of the findings?

Conversely, a lot of times you'll find studies or researchers that mention specifically that their findings challenged their biases or preconceptions yet they published them anyway, I would take this as a sign of integrity.

Neuroplasticity is another topic that "settled science" was wrong about for a long time, from Head Strong by the godfather of biohacking...

It wasn’t until later in the twentieth century that scientists discovered what’s called neuroplasticity— the brain’s ability to grow new cells and forge new neural connections throughout your life. Before then, researchers believed that your brain remained static until it degenerated in old age. (p. 4)

Head Strong

 

5.0

Category: Book

Conclusion

On its surface, this topic might seem depressing. Perhaps you're feeling like a little kid who has just found out that Santa Claus doesn't exist. As Biohackers we love science;

• We lionize scientists.
• We relish that discovery feeling of learning new things.
• We get excited about new scientific findings that allow us to live better.

But the truth is that a lot of science has been corrupted by human bias, bad incentives, and bureaucratic institutions.

My takeaways from researching this topic thoroughly are...

• Quality matters a lot more than quantity when it comes to studies. If there is a recent gold-standard double-blind, randomized, placebo-controlled pre-registered study done on a given Nootropic I'll mostly disregard the lower-quality studies done on it.
• I'll more seriously consider statistically significant amounts of the anecdotal accounts and Biohacker reports about a given Nootropic.

For example, recently I caught some flack on Reddit from "skeptics" about a Vitamin B17 meta-analysis podcast I did, which is a bit of a controversial vitamin.
Unsurprisingly, they pointed to some science done many decades ago that declared B17 a hoax, even though there is more recent science done that is more positive about it. The science that is negative about B17 is replete with red flags and it was published by a pair of researchers that had in the 1950s defended big tobacco and declared smoking cigarettes healthy; something that we now know is absurdly anti-scientific.
As I mentioned in my podcast, in my research of B17 I looked through +600 anecdotal reports and did not find one incidence that confirmed the negative B17 "science."
In this kind of case, it's pretty clear that the anecdotal data should be held in higher esteem than the published "science."

I hope you'll practice similarly nuanced critical thinking in your scrutinizing of the science and your personal Biohacking practice. We are at the frontier of an exciting new era of performance-enhancement science. Our generation was born too late to explore the earth, yet too early to explore the stars but we were born at the right time to take on perhaps a far more important project; making better humans with science. Maybe they'll make it to the stars...