Evidence is the supposed outcome of scientific studies that make up what we consider to be our understanding of science and within that our medical literature as a whole.
The idea of learning a fact by creating an environment that deletes normally present confounders and biases to cleanly produce an outcome and result that can be considered pure enough to be a fact is not old. It was only after inventing photography that we discovered most of our medical knowledge. Think of the photos from the American Civil War. Cleaning hands before amputation was not a priority. We didn’t even have antibiotics until after WWI, yet we think that we know everything there is to know about the human body by scientific studies over what is basically a little more than a generation of time. We assume these studies are “science” and “prove” something, but that is far from the truth.
We can find studies out there about anything with different outcomes. That’s wherein lies the art of scientific appraisal. I know that I would not be able to tell a good study from a bad one if it pertains to astronomy or theoretical physics, and for that reason I don’t trust myself judging a study or article’s value in the grand scheme of things. But for whatever reason, people seem to think that a study’s outcome authorizes them to use it as proof. When another study with alternative outcomes is used as a rebuttal, suddenly both parties are at a loss if neither can appraise the value of the studies used.
First, I look at the journal it is from. Journal’s themselves are held to a standard and judged using their Impact Factor.
The highest impact factor in medicine is probably held by The New England Journal of Medicine, what I feel is the most important and powerful medical journal on the planet. On the other hand there are journals like the one I published an article in last year. I’m not embarrassed to highlight a low impact factor for the journal of my own article because I wrote it so I know how cutting edge it was. Since it was written a few articles have been published on the subject yet don’t reference me… I know they read it.
Anyway, next I glance at the authors (MD’s? or not… makes a difference) and actually start reading it. Whether or not it is prospective is a big deal for me. Create a plan, find an answer. Don’t use history to search for patterns! That’s important to me.
The type of study it is. “Double blinded” means everything is blanketed in secrecy so no one knows what drug is being given to who until the very end when the results are pooled. That kills off the “placebo” effect that is up to 30% benefit in many studies. Literally telling someone they are getting the new miracle pill will cause positive effects for them after they take it. That’s how magicians and cold readers thrive so well. Thirty percent of their act actually works! Even though it doesn’t! You get the idea.
Third, the amount of people included. Our strongest studies pool data from all over the world, from various types of hospitals and offices and people, leading to hundreds of thousands of patients and data points to work with. That’s power. Taking 5 people and putting them into two groups and analyzing the results from them is totally ludicrous in comparison to our (usually cardiovascular) legendary trials. This needs to be taken into consideration before appraising results.
Next, I consider underlying biases and real-life street-knowledge situations and factors that may have influenced the results. Always in this list is “publication bias” – the fact that negative results cause studies to be less interesting and thus less frequently published and often lost in time forever – and of course bias by the investigators. I even consider patient circumstances, like in a trial of around 20 patients with Crohn’s disease, which I think may have been affected by a patient using personal marijuana in the non-treated group. It would only make sense in another piece or a full discussion but thankfully I’m able to consider such things after reading so many studies over the years and being interested in skepticism.
Finally, what was the result? How statistically significant was it? How much benefit or harm resulted in general? One percent? Two percent? Fifty percent? And was the change “absolute” -overall in the population – or “relative” -compared to other medicines. These things make a difference and are what we learn in statistics but with a little consideration or self-education anyone can grasp these ideas being discussed.