On the latest Dark Horse podcast, which you can find on Odysee since YouTube's community guidelines forbid discussion of this topic, Brett Weinstein interviewed Tess Lawrie. Dr. Lawrie is an MD and Ph.D. who calls herself a guideline methodologist. She helps to assess evidence and compile it into evidence-to-decision frameworks to help guideline panels make recommendations for medical treatments.
In this discussion, Dr. Lawrie described what she referred to as the current pyramid of scientific evidence. This pyramid is structured with the weakest evidence on the bottom and the strongest evidence at the top.
Bottom Layer 1: Consensus/Opinions
Layer 2: Case Reports (anecdotal evidence)
Layer 3: Case Series
Layer 4: Case Control Studies (non-randomized)
Layer 5: Cohort Studies
Layer 6: Randomized controlled trials
Top Layer 7: Systematic review with meta-analysis
There are many people who state that double-blind randomized controlled trials are the gold standard for scientific evidence, but this is not true. According to Dr. Lawrie, randomized controlled trials are great at showing efficacy (how well does it work) but are not great at showing safety (what is the risk for harm) because of how they are designed. A double-blind trial means that neither the patients nor the clinicians know the treatments that they are getting. What we really need is a quadruple-blind randomized control trial, which means the people assessing the evidence and the people analyzing the evidence don't know which groups received treatment. This would eliminate additional bias that is either intentional or unintentional based on the nature of clinical trials, expected outcomes, and who is paying for the research.
The actual gold standard at the top of the current pyramid is the systematic review with meta-analysis. This review, when conducted by a dispassionate independent group, looks at all of the available research from anywhere in the pyramid. No trial is perfect. By systematically reviewing all the studies on a particular topic, one can reduce the noise among many trials and determine whether a signal exists and how strong the signal is.
In this discussion, Dr. Lawrie described what she referred to as the current pyramid of scientific evidence. This pyramid is structured with the weakest evidence on the bottom and the strongest evidence at the top.
Bottom Layer 1: Consensus/Opinions
Layer 2: Case Reports (anecdotal evidence)
Layer 3: Case Series
Layer 4: Case Control Studies (non-randomized)
Layer 5: Cohort Studies
Layer 6: Randomized controlled trials
Top Layer 7: Systematic review with meta-analysis
There are many people who state that double-blind randomized controlled trials are the gold standard for scientific evidence, but this is not true. According to Dr. Lawrie, randomized controlled trials are great at showing efficacy (how well does it work) but are not great at showing safety (what is the risk for harm) because of how they are designed. A double-blind trial means that neither the patients nor the clinicians know the treatments that they are getting. What we really need is a quadruple-blind randomized control trial, which means the people assessing the evidence and the people analyzing the evidence don't know which groups received treatment. This would eliminate additional bias that is either intentional or unintentional based on the nature of clinical trials, expected outcomes, and who is paying for the research.
The actual gold standard at the top of the current pyramid is the systematic review with meta-analysis. This review, when conducted by a dispassionate independent group, looks at all of the available research from anywhere in the pyramid. No trial is perfect. By systematically reviewing all the studies on a particular topic, one can reduce the noise among many trials and determine whether a signal exists and how strong the signal is.