How I learned to doubt a paper
Science should not be a hope-delivery system
My perspective on the science of ME/CFS and Long Covid has gone through a couple of phases since first becoming ill in October 2020.
The first phase was one of confusion, going to doctors to get tests, eventually being dismissed and experimenting with graded exercise and psychosomatic approaches. To no surprise of my fellow patients, those didn’t yield much.
Once I realized almost a year later that I might have Long Covid and ME/CFS and POTS after all, I quickly became pretty obsessed with The Science of it. I devoured every paper I could get my hands on together with other patients. I still recall pretty vividly how excited I was in late 2021 that microclots could explain it all! Now I needed to get my hands on triple anticoagulants! When that hypothesis inevitably disappointed, I moved on to viral persistence.
This phase of obsession lasted about 2.5 years. It was typified by taking most findings at face value and building elaborate theories upon these findings. It was reading Twitter, Cort Johnson, following conferences, and writing many of my own Twitter threads.
Me, taking claims at face value and building elaborate theories.
But slowly, the cracks started to show.
I interacted with patients more scientifically literate than myself, who told me about all the shortcomings in the papers I was taking at face value. I saw findings fail to replicate. Or they implied a certain treatment would work which we already know it doesn’t because many patients have tried it. Or the findings weren’t specific to ME/CFS at all. Or they haven’t been corrected for multiple testing. And I saw Very Serious Scientists claim they had found a biomarker (e.g. cortisol) even though the existing literature already showed that it was at normal levels in other samples! (Let alone methodological issues like timing of sampling etc.)
Nowadays, I’m in the skeptical phase. I am more selective in which papers I engage with. I have a document with methodological notes that I give to Claude and ask it to evaluate any interesting-looking paper that passes by:
Evaluate objectively and rigorously. Be aware of bias, p-hacking, spin, hype, etc. But be well calibrated - when something’s good you can just say so.
Now of course, even with my extensive notes, AI has its flaws. But so do humans, and it’s a great gating mechanism to catch many weaknesses and provide context, and if the paper interests me I can check out the discussion on Science 4 ME.
But in many cases, there are severe shortcomings and the paper is a nothingburger. Occasionally, a good paper like DecodeME (a highly powered Genome-Wide Association Study) or the muscle biopsy post-PEM provocation study add something really valuable to the field.
The irony is that when patients evaluate studies from the psychosomatic paradigm, they act like great scientists. They scrutinize inclusion criteria, because many studies claim to be about ‘chronic fatigue syndrome’ but have the core feature (post-exertional malaise) as optional! They scrutinize the objectivity of outcome measures, because patients of unblinded treatments will report feeling better even when their physical activity has not increased. Like the cognitive psychologist Daniel Kahneman observed, when people defend their worldviews, it’s like they gain 15 IQ points!
Positive results are highly overrepresented in the literature due to an accumulation of biases - do not take them at face value. From De Vries et al. (2018), n = 104 antidepressant RCTs.
What really systematized all this skepticism for me was listening to the book Science Fictions by Stuart Ritchie. I think it’s a must-read for anyone seriously engaging with science. Science, we’re increasingly seeing, is not in a healthy place. Scientists are highly biased towards publishing positive results and getting citations. They have to, to keep their jobs and secure grant money. It’s extremely rare for a scientific team to publish a paper saying “we tested our hypothesis and found nothing interesting”. Naive patients getting swept up in the hype make this much worse. And the more toned down and skeptical my tweets have become, the fewer likes I get. This dynamic turns science into a hope-delivery system for instant gratification, instead of the long-term solution-delivery system it is supposed to be.
We can do better. We need to figure out how we can improve the scientific quality of our field. As I discussed in 3 levers to solve a disease, scientific quality serves as a multiplier for scientific quantity. We don’t get a lot of funding into the field. Let’s make sure that the funding we do get, counts.
I’m Siebe. I write about solving diseases — especially poorly understood ones like ME/CFS and POTS/OI, which have disabled me since 2020. Questions I focus on are:
How can we make clinical trials cheaper and more effective?
What makes for good science and how can we increase the quality of a field?
What makes for effective patient advocacy?
What does the evidence support regarding how ME/CFS works?
Find me on Twitter as @PatientPersists. Everything written with brain fog and limited spoons.
Sorry to hear you have/had long covid, that sucks. I am a similarly skeptical-minded reader — and a researcher myself— and my own checklist for skimming studies is here https://regressiontothemeat.substack.com/p/how-i-read-studies
(Although some of the things you’re describing sound like scientific misconduct and/or like things whose falseness is only evident to people with a lot of local context. Much harder problems to correct against)
Great article. My only caveat would be that the *funding* of science should have an element of hope, meaning there's a worthwhile goal with a plausible and feasible approach. And I would encourage aspiring towards more funding into this field in addition to increasing its quality. But I understand your article is mostly speaking to the reporting of science results after the fact, however it may have been funded.