Life Hacking Blog & Articles

Red Flags of Bad Science

By Jonathan Roseland  Connect    

Like in any other domain of human endeavor there is a spectrum of quality and accuracy in science. Some double blind, placebo controlled human research is excellent and profoundly helpful to us in living better but I will show here that some "science" is tantamount to a ranty, opinionated blog, some is funded by the pharmaceutical industry and really is just marketing masquerading as science.

Antifragile blocksI'll encourage you to not place blind faith in the establishment of science the way that your ancestors likely placed blind faith in the catholic church or whatever their religious persuasion was.
As Nassim Taleb articulates in Antifragile:
the unconditional belief in the idea of scientific prediction regardless of the domain, the aim to squeeze the future into numerical reductions whether reliable or unreliable. For we have managed to transfer religious belief into gullibility for whatever can masquerade as science.
Information is the most powerful tool we have for living well, bad information really can limit us, science is the best source of helpful information, but it's far from a crystal ball that can be stared into to ascertain absolute truth. If you're willing to devote a little bit of extra attention and cognitive horsepower as a layperson you can easily recognize bad science.

Skeptical of Bad Science?


You may say...
Come'on Jonathan, there's no such thing as bad science! All science is good!
I'll remind you that:
  • For 30 years "science" and physicians promised the public that smoking cigarettes was good for you.
  • For 40 years "science" and the FDA put out the ridiculous food pyramid recommending the public eat a diet mostly of bread, pasta and dairy.

Has science cleaned up it's act in recent times?
In 2015 Richard Horton, editor in chief of Lancet, the UK’s leading medical journal, stated:
“The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Science has taken a turn toward darkness.”
In 2010 Dr. Marcia Angell of the prestigious New England Medical Journal was equally disparaging of the state of science:
“It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of the New England Journal of Medicine”
I'm not saying that science is useless but that it should be scrutinized and just because something is labeled "science" you should not turn off your critical thinking skills.

Go to the Source

Probably the most common mistake that the layperson makes is reading an article online about science and assuming that they are actually doing meaningful research. A well meaning person will Google a subject like...

  • Climate change
  • Anti-aging
  • IQ
  • Or nutrition

...they will read the first few articles they find and give themselves a pat on back for acquiring a modicum more expertise on the subject.
While not totally useless, this information consumption really doesn't count as researching the science on a topic. The same applies to listening to podcasts or watching Youtube videos.
Researching the science means actually reading the scientific studies, meta analysis papers, books and PubMed abstracts. Which are full of...

  • Stale jargon
  • Superfluous acronyms
  • Scientific terminology
  • And not exactly inspiring verbal prose

...they really are not fun to read but if you actually want to understand a scientific topic, it's crucial to do.

Lifehack: Next time you find yourself in a debate with someone over some scientific topic. Just ask them when was the last time that they actually read a book specifically on the topic of contention.
Really? You don't think IQ is a meaningful scientific measurement of intelligence? What was the last book that you read on IQ?
Almost always they will have NOT read a book on the topic. So then you can point out how really uninformed they are on the topic. As a rule in life:
If you haven't read a book on a topic you really don't know what you're talking about in regards to it.
They have so little commitment to veracity and empiricism that they are not even willing commit a couple of solid hours to understanding the topic.
Then you can point out that their position is really just an arbitrary preference and that you guys debating this issue is really tantamount to merely arguing whether chocolate or vanilla ice cream is better.
Obviously, this argument really only works if you yourself have read a book on the topic. So better read up on the topics you anticipate debating!

Mainstream Media Misinformation

You may think that the New York Times, Huffington Post or The Guardian have high standards when it comes to fact checking and accuracy but I'll suggest that after reading a New study finds... type of story in these publications go and read the study itself.
Often it's just a Google search away. If you take the time to do this often you'll find a blatant error or misrepresentation of the science in the mainstream publication, particularly if there is political or financial implications of the science.
The job of journalists should be to tell the truth but more often they lie. The excellent book Trust Me, I'm Lying by Ryan Holiday explains why.

Information Profiteers

There's a necessary class of merchants of science or science writers that people like myself, Dave Asprey and Steven Kotler fall into. Our job is to make the science more understandable and useful to the public but we make mistakes from time to time, our fact checking is not perfect and our incentives are often not aligned with our earnest desire to educate people on how to use science to live better. I call us a necessary class because scientific publications really are quiet boring and nearly indecipherable to the general public, without marketing minded people repackaging science it would not make it into the mainstream. I think we can all agree that we would like science to be available and understandable to all so that it can be a democratizing force in the world.

 
However amongst us there is a real spectrum of genuine educators to charlatans who are just in it to much a buck. How can you spot a disingenuous middleman?

Lack of proper citations. Many science or health articles consist of several hundred words followed by a handful of links to studies that ostensibly support the conclusions presented by the writer of the article. But a lot of times the citations really don't support the conclusions, this is why you should look for direct quotes from the studies or researchers themselves supporting conclusions as opposed to just links.
 
Omissions of unappealing information. Especially in the field of anti-aging products and Biohacking supplements, disingenuous science writers will omit information that hurts the case for the supplement or drug in question.

Clear conflicts of interest. We usually make our money a few different ways:
  • Affiliate sales of products
  • Sponsorship
  • Advertising revenues

Often these income sources create a perverse incentive. If it's clear that a writer has a financial incentive that makes them partial to a particular product or supplement you should further scrutinize their portrayal of the science surrounding it.

Ads. Online advertising creates a bad incentives for writers to put out a great quantity of work in lieu of quality. If you're reading a science article on a website that has a bunch of ads in the sidebar and header and below the article that is further reason to question it.

Ghostwritten. Writers who know what they are talking about almost always put their real name and face on their work. They are proud of their work and stand behind it. There's a lot of quintessential content farm websites that churn out a great quantity of articles about health, science and supplementation. Invariably, the writers are invisible, they are often outsourced digital labors working for a few dollars an hour in a place like the Philippines or India.

Lack of author credentials. Writers who know what they are talking about will have some relevant credentials and experience.

Scientists will often spend decades myopically focused on a singular scientific niche, they'll understand it profoundly but often they'll lack holistic knowledge of how that science could be useful to laypeople. The writer's job is to be a scientific polymath and a lot of times we can articulate applications of the science that can really help people.
However, if you're reading an article online that contains one or more of these red flags that should trigger your skepticism. If you really want to understand something, if you're making a crucial decision based on science, go the source.

Bad Science

However, sometimes the source publication itself is bad science. Consider the case of the recent (hilarious!) hoax paper...
The conceptual penis as a social construct
...that actually passed the peer review process and was published in a mainstream social science journal, Cogent Social Sciences.

The authors specifically wrote the least scientific paper that they could. They specifically wrote a paper that should have been rejected by a scientific journal. A few examples of the absurdities it postulated:

Manspreading — a complaint levied against men for sitting with their legs spread wide — is akin to raping the empty space around him.

We conclude that penises are not best understood as the male sexual organ, or as a male reproductive organ, but instead as an enacted social construct that is both damaging and problematic for society and future generations. The conceptual penis presents significant problems for gender identity and reproductive identity within social and family dynamics, is exclusionary to disenfranchised communities based upon gender or reproductive identity, is an enduring source of abuse for women and other gender-marginalized groups and individuals, is the universal performative source of rape, and is the conceptual driver behind much of climate change.

Destructive, unsustainable hegemonically male approaches to pressing environmental policy and action are the predictable results of a raping of nature by a male-dominated mindset.

The authors intentionally made it as absurd and anti-scientific as possible, from the Skeptic.com article where they admitted their hoax
After completing the paper, we read it carefully to ensure it didn’t say anything meaningful, and as neither one of us could determine what it is actually about, we deemed it a success.

Not only is the text ridiculous, so are the references. Most of our references are quotations from papers and figures in the field that barely make sense in the context of the text.

Nearly a third of our references in the original paper go to fake sources from a website mocking the fact that this kind of thing is brainlessly possible, particularly in “academic” fields corrupted by postmodernism.

Cui bono?

When researching the science it's important to ask who benefits? Who is getting paid?
Many journals like Cogent Social Sciences operate with a pay-to-publish model. The authors of The conceptual penis paid $625, they published under fake names as part of a fake social research group, which Cogent Social Sciences did not catch. So for $625, approximately the same amount you would have to pay to score some cocaine, get drunk on champaign and bang a hooker in a chintzy hotel room in Vegas, anyone's totally nonsensical opinions can be transformed into "science".

How many other papers that have appeared within the pages of Cogent Social Sciences are similar pseudo scientific none-sense? I would wager probably most of them.

Moralizing Should be Scrutinized

You should be very suspicious of science or scientists that make moral statements. Science and morality should not necessarily be separated like the church and state (or matter and anti-matter in the warp drive of the starship Enterprise to use a psuedo scientific metaphor) but the more moralizing statements that bracket the science the more you should be suspicious of it.

Case in point, The conceptual penis is chuck full of moralizing statements - another major red flag that the peer reviewers at Cogent Social Sciences ignored.

To quote the hoaxers again
That is, we sought to demonstrate that a desire for a certain moral view of the world to be validated could overcome the critical assessment required for legitimate scholarship. Particularly, we suspected that gender studies is crippled academically by an overriding almost-religious belief that maleness is the root of all evil. On the evidence, our suspicion was justified.

Luckily for Biohackers the science of peak performance, health and antiaging is not very politicized relative to a field like the social science which is just a minefield of misinformation and propaganda. Let's keep it that way!

Recency

Newer scientific studies should be viewed more credible than older ones. In researching the meta analyses of Nootropics that I write I generally disregard studies that are over 30 years old.

In April of 2017 Pubmed announced they are actually standing up for science by publishing conflict of interest information on the study abstract page instead of allowing it to be buried behind an academic pay wall.

This resulted from 62 scientists and physicians, and five United States senators petitioning the National Library of Medicine & National Institutes of Health to make public the funding sources of studies. This makes me more optimistic about institutions and politicians — in the news it seems that all we hear about is how corrupt and incompetent institutions and politicians are but sometimes they actually act in our best interests!

So it's reasonable to view science published after April of 2017 as a little more credible.

Red Flags of Bad Science

To quote Maciej D. Zatonski on Quora:
Here are some classic examples that indicate that what you are reading is b******t:

- No control group
- No double blinding
- Results not replicated
- No peer review
- Sensationalised headlines
- Bonus: Betteridge’s Law of Headlines says, that if the title has a question, the answer to that question is “No” [1]
- Conflicts of interest
- Misinterpreted results
- Causation assumed when only correlation exists
- Conclusions not drawn from results
- Too small sample or sample not representative
- Selective reporting of data
- Claimed “Galileo’s Gambit” (results from a single study that contradict mainstream science are claimed to be the only correct ones; discredited authors are claimed to be “persecuted” and therefore right)
- Anecdotes used instead of evidence (“I know there is no evidence homeopathy works, but it worked for me!”)
- Changes of conspiracy and collusion
- Stressing or appealing to status or authority of someone (real or fictitious)
- Repetition of discredited arguments
- False dichotomy / failure to realise there are more than only two opposing outcomes
- Cherry picking of data / ignoring majority of evidence (3 studies show that climate change is not real, while ignoring 3000 studies that show otherwise)
- Wishful thinking (there is no evidence for existence of gods, but we wish they were real)
- Appeal to ancient wisdom (“it must be true because many people were saying this for a long time before”)
- Use of pseudoscientific language
- Daily Mail articles / Fox “news”
- Discrediting an argument because it comes from a person you dislike
- Appeal to nature (“it is natural, so must be healthy”; well… earthquakes, scorpions, cyanide and botulinum toxin are natural too!)
- False balance - counter argument to scientific fact claimed to be of equal value/weight to a counter-argument (theory of embryological foetal development vs stork theory of human birth; theory of evolution by natural selection vs creationism, etc)
- Appeal to common sense or alarmism (usually done to receive extra funding for pointless research)

Dr John Ioannidis - The Bad Science Expert

A Greek American doctor; over the past few decades he has very methodically and none-sensationally shown the light on bad science.
An exhaustive article that appeared in The Atlantic interviewed him thoroughly at the University of Ioannina Medical School campus
Ioannidis... delivered what felt like a coup de grâce: wasn’t it possible, he asked, that drug companies were carefully selecting the topics of their studies—for example, comparing their new drugs against those already known to be inferior to others on the market—so that they were ahead of the game even before the data juggling began? “Maybe sometimes it’s the questions that are biased, not the answers,”
On the integrity of scientific process
We think of the scientific process as being objective, rigorous, and even ruthless in separating out what is true from what we merely wish to be true, but in fact it’s easy to manipulate results, even unintentionally or unconsciously.
The science journal Nature, concurred in a 2006 editorial:
“Scientists understand that peer review per se provides only a minimal assurance of quality, and that the public conception of peer review as a stamp of authentication is far from the truth.”

Double Blind, Randomized Placebo Controlled and Pre-Registered

You've likely heard before that the gold standard of science is the double blind, randomized, placebo controlled study

  • This means that approximately half the subjects are given a placebo. This should prove that the drug or supplement being studied outperforms the placebo effect. Which is actually pretty meaningful because the placebo effect is powerful!
  • The patients are not told if they are receiving a placebo nor are the researchers told if they are administering the placebo or the real thing. You can imagine if the researchers know they are giving a placebo to an unwitting patient they will act a bit bashfully which the patient may notice, blunting the placebo effect. This is why it's crucial that studies are double blinded.

The Replication Crisis
As Dr. Rolf Zwaan explains in this interview, it was discovered in 2015 that out of a 100 recent studies in the fields of cognitive psychology and social science, less than a third were actually replicable.
[video]
Dr John Ioannidis from the paper Why Most Published Research Findings Are False:
"Several methodologists have pointed out... that the high rate of nonreplication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study..."
The best solution to the replication crisis would seem to be Pre-Registration, meaning that...

  • The hypothesis, study design and methodology are Pre-Registered with journals before conducting the study.
  • This keeps the researchers focused on determining if the data confirms or denies the hypothesis.
  • This prevents researchers from expressing a bias in research or searching for desired result.

Dr. Ioannidis goes on to clarify the place of Pre-Registration in studies:
"In some research designs, efforts may also be more successful with upfront registration of studies, e.g., randomized trials... Registration would pose a challenge for hypothesis-generating research. Some kind of registration or networking of data collections or investigators within fields may be more feasible than registration of each and every hypothesis-generating experiment. Regardless, even if we do not see a great deal of progress with registration of studies in other fields, the principles of developing and adhering to a protocol could be more widely borrowed from randomized controlled trials."

Pharmaceutical Science

In 2016 Forbes listed pharmaceutical as the most profitable amongst major industries - beating out banking, financial services, tobacco and the IT industry. This creates an appreciable (and irresistible?) incentive for these firms to yield their influence to pervert science.
Consider the sordid history of Bayer Pharmaceutical which has recently merged with the globally hated Monsanto.
[video]
It would be difficult to call the current iteration of the company evil based solely on their participation in WW2 70 years ago, however many still consider it one of the least benign corporate citizens in the world because of it's Factor VIII scandal in the 1980's; when it turned out that a product of their's infected 6,000 hemophiliacs with AIDS they paid out a $600 million settlement and then outrageously proceed to continue to sell the product in other international markets, infecting unwitting people around the world with the deadly virus.
I don't ascribe to the fringe conspiracy theories about the purposeful evilness of these companies; I think it's a combination of the perverse sticks and carrots that our globalized world offers in combination with the outgroup preference that a corporate culture engenders naturally amongst it's employees.
[video]
Ben Goldacre an Epidemiologist points out that
Industry funded trials are four times more likely to have a flattering result
The book Adaptogens is especially critical of pharmaceutical science:
“The great majority of published research is so deeply flawed that it should be considered essentially worthless.” So wrote John Ioannidis, PhD,
(p. 54)
Ioannidis singled out the following types of studies as being particularly likely to lead to a worthless result:
Studies with a small sample size;
Studies that consider a small number of possible effects;
Studies whose outcomes are poorly or subjectively defined;
Studies in which financial conflict of interest is a factor;
Studies in which the researchers are prejudiced by being unduly wedded to a particular outcome;
and Studies of a topic that is currently “hot.”
(p. 55)

John Loannidis clarified further on conflict of interests in studies
"The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true."
From The Atlantic article:
...drug-company research wasn’t measuring critically important “hard” outcomes for patients, such as survival versus death, and instead tended to measure “softer” outcomes, such as self-reported symptoms (“my chest doesn’t hurt as much today”)

Not everything pharmaceutical is bad, in fact some of the best smart drugs are products of pharmaceutical companies, but where the moral hazard is so great you should be skeptical.

Be Suspicious of Bias Confirming Science

You want to scrutinize science that really underlines the obvious moral position of the author or publisher. Again, a lot of what you see published is opinion masquerading as science.
Examples

  • A really lefty Liberal arts college produces social science saying that women are actually better at engineering (or whatever) than men.
  • Evangelical Christians produce an Intelligent Design Biology textbook.
  • A vegan that publishes nutrition research that eating meat causes inflammation. (I'd be more likely to believe this last one!)

I'm not suggesting that you totally disregard science that supports a preferred bias.
I'm suggesting that when you are reading a study, Google search the authors, publication or establishment and just take about 30 seconds to look at their social media.
If their social media has a really strong ideological or moralizing bent that the study coincidentally supports that should be a red flag, I'll encourage you to further scrutinize the study.

  1. Is the study focused mostly on the data?
  2. Or is it full of a bunch of moralizing opinions?
  3. Do the researchers admit in the study that they were particularly surprised by any of the findings?

Conversely, a lot of times you'll find studies or researchers that mention specifically that their findings challenged their biases or preconceptions yet they published them anyways, I would take this as a sign of integrity.

Conclusion

On it's surface this topic might seem depressing. Perhaps you're feeling like a little kid that has just found out that Santa Claus doesn't exist. As Biohackers we really love science;

  • We lionize scientists.
  • We relish that discovery feeling of learning new things.
  • We get excited about new scientific findings that allow us to live better.

But the truth is that a lot of science has been corrupted by human bias, bad incentives and bureaucratic institutions.
My takeaways from researching this topic thoroughly are...

  • Quality matters a lot more than quantity when it comes to studies. If there is a recent gold standard double blind, randomized, placebo controlled pre-registered study done on a given Nootropic I'll mostly disregard the lower qaulity studies done on it.
  • I'll more seriously consider statistically significant amounts of the anecdotal accounts and Biohacker reports about a given Nootropic.

For example, recently I caught some flack on Reddit from skeptics about a meta analysis podcast of Vitamin B17, which is a bit of a controversial vitamin.
Unsurprisingly, they pointed to some science done many decades ago that declared B17 a hoax, even though there is more recent science done that is more positive about it. The science that is negative about B17 is replete with red flags and it was actually published by a pair of researchers that had in the 1950's defended big tobacco and declared smoking cigarettes healthy; something that we now know is absurdly anti-scientific.
As I mention in my podcast, in my research of B17 I looked through +600 anecdotal reports and did not find one incidence that confirmed the negative B17 "science".
In this kind of case, it's pretty clear that the anecdotal data should be held in higher esteem than the published "science".

I hope you'll practice similarly nuanced critical thinking in your scrutinizing of the science and your personal Biohacking practice.

Finally...

This website has +400 pages of information demystifying Lifehacking, Biohacking and Smart Drugs - that's kind of an overwhelming amount of information. As soon as you join the Limitless Mindset Community (It's free!) we give you instant access to an interactive infographic which visually breaks the widely disparate topics we cover in exhaustively in our articles, videos and podcasts.

Join the Limitless Mindset Community for Access to:

Hack Your Reality

Share This!

           

Join the Limitless Mindset Community for Access to:

The AV Association Technique

The AV Association Technique - 4 Part Video Course

A 'mental hack' that master salespeople, politicians, pickup artists and cult leaders use to remember limitless names instantaneously.

Memory Systems Audio Course

Memory Systems - Triple Your Memory Power & Become a 'Human Google'

HumanGoogle

A 'lost art' used by life hackers, scientists and extremely productive individuals. Acquire skills in 1/3rd the time, vanquish absentmindedness & improve creativity.

3 Hour Audio Program

Mind Power Software

Dual N-Back Training Software Demo - Increase the 'RAM' of Your Mind

intelligence IQ software podcast

20 minutes spent daily training on your PC, Mac, iPhone or Android device will creative noticable cognitive gains in 1 month.

Complimentary download: For Limitless Mindset Community Members

Top Rated Mind Power Products

HeartMath emWave2  
Dual N-Back Pro  
Nicotine USP Solution  
Rhodiola Rosea Extract  
Tianeptine by Intellimeds  

Give Us a 'Like'

Connect with Jonathan

JRFranceheadThe inventor of the most clever cocktail toast ever, former participant in a bank robbery, almost died underwater twice, raconteur & smart drug dealer.

25 minute Biohacking consultation

Affiliate Links?

As you may have noticed this website contains affiliate links to products, supplements and software programs. The small commissions we recieve from sales of these products allow us to commit the time necessary to thoroughly researching which products are credible and will give you the biggest bang for your buck when it comes to upgrading your mind.

We have a strict philosophy of only endorsing or recommending products that we've found really work to help you upgrade your mind.

Thank you, sincerely, for your support!