The trouble with evaluating data
Remember the phrase two wrongs don’t make a right? This holds true when try to evaluate scientific data. Or maybe it could be said that just because one person is wrong, that doesn’t make the other person right. There can be a lot of wrong answers to the same question. That’s kind of how I feel about all the latest reports on things like nutrition and environmental issues.
Whether it is information about anything from best gardening practices to epidemiology, we need to be careful to ask foundational questions.
- how was the data collected
- how was it interpreted
I will forever remember reading my first raw archaeologists report. It was on the Hittites. The author had been present at the dig and was amazingly honest about what was found and what could really be known (or in most cases not known) based on the finds. The most that could be said without a doubt for most things was “we found a piece of broken pottery.”
That contrasted so incredibly with various summaries we read that tried to put the information together for us in a meaningful way. That is, they had to make a lot of stuff up to say anything that sounded worth reading. And it was presented as based on scientific findings.
Have you ever come to wrong conclusions?
The brutal fact is that any of us can be looking right at the raw data and come to the wrong conclusions. My story about broccoli and ants is an example of that. And I have a background in biological sciences and research processes.
Even after many years of gardening in the same climate and yard, I came to the wrong conclusion about something I had been seeing with my own eyes for years. This was even after doing some initial research.
As I cover in detail in Ants, Cabbage Maggots, and Broccoli, ants showed up burrowing at the base of the stems of my broccoli and the plants died. Year after year this happened. I concluded the ants were killing my broccoli. I was wrong.
Since nothing I did deterred the ants or saved my broccoli, I continued to research. Finally I found some well-buried information. I tested this and ironically discovered that ants were actually part of the solution. By supporting their activity I ended up with greater broccoli survival rates.
How far away from the original data are you?
The farther away from the original data we are, the harder it is to double-check things. Take bats and mosquitoes, for example. It has been “common knowledge” for a while that bats help control mosquito populations.
It turns out this is based on one very limited study. Further evaluation shows that it is more likely that while some bats eat some mosquitoes, bats are unlikely to put a dent in the mosquito population.
When you consider the diseases that bats carry, you have to wonder at the wisdom of encouraging them to live near your home.
Have you ever slept with a bat?
Our family has enjoyed watching bats while camping. We have also had cringe-worthy encounters with them in a rental cabin.
Unknown to us before we rented the cabin, the owners encouraged hundreds – yes, hundreds, maybe thousands – of bats to live at their lakeside cabin. They specifically asked us not to disturb the bats.
During the day, the bats hung sleeping under the eaves, under the deck, in random footwear, even on top of flip-flops.
One morning our teenage son awoke to an unidentified tickling in his sleeping bag. There was a bat trying to settle in for the day. There were droppings everywhere around the deck and grounds. Some of these inevitably got visibly tracked into the cabin.
What about the mosquitoes? They were thick in the air. If the bats were eating them, I shudder to think how many bats might be needed to keep the mosquitoes under control! That was one of the longest weekends of my life.
Not all experiments are so conveniently illustrated
If you’ve ever had to set up an experiment, you know how hard it is to account for all possible variables. A lot of times isolation in a lab is an attempt at this. Isolation still introduces a kind of negative variable, especially if you are studying a living thing or a behavior.
One such negative variable came up humorously when I was researching my latest children’s book Four Frogs in a Glass Pond. You can read about it in the second section of the book which is about how to win a frog jumping contest.
Understanding the limits of drawing conclusions from data
If you want to make the best decisions you can for you and your family, you need to understand the limits of drawing conclusions from data.
Did you know that, on average, the students pursuing medical related education have an unusually high incidence of seeking medical diagnoses with no prior cause before medical studies?
There is something about becoming aware of possible medical conditions – not to mention all the scary, worst-case photos in the text books – that raises anxiety. This results in people seeking out medical care when they otherwise wouldn’t.
I remember when I came across eczema during my medical rotation in nursing school. The example photo was horrifying! In this case, I had the advantage because I had had eczema for years and it didn’t look nearly that bad. This made me view all other photos and descriptions with some skepticism.
When I heard of some hospitals recently reporting 150% patient loads, I had to wonder several things:
- How many of the patients were going to the hospital because they had heard in the news about health concerns?
- How many people went because someone they knew had gone?
- What was the previous patient load at that hospital?
- What are the demographics of the community around that hospital?
- How many of those people were poorly affected by hospitalization? (isolation, inactivity, poor food, germ infected environs)
- How many of those patients were badly affected by assumed treatment protocols?
- If other hospitals in adjacent areas were basically inactive, why the difference?
I have worked in hospitals. I have been a patient and I have supervised seriously ill family members in hospitals. I know that most medical staff are doing their best to provide care within a system they have little control over.
No matter what, we need to ask important questions about the data and the presentation of that data. It affects our decisions. Decisions we should be allowed to make.
In asking those questions, we need to be careful of our own assumptions. There is no value in assuming the worst and acting accordingly. If we applied that to all of life, we’d have to lock ourselves up forever.
Reasons to begin with skepticism
There are many reasons to doubt how reliable both test and diagnoses are. The general population is led to believe these things are much more reliable and clear cut than they are.
Often, diagnoses are made due to symptom clusters. There are only so many bodily symptoms to go around and many conditions look alike at presentation.
Too frequently diagnoses are made due to pre-determined protocols. There are both politics and crony-capitalism swaying what hospitals can or will report. Sometimes it is fear of lawsuit or licensing boards that pressure doctors.
And tests? Several immunologists have tried to explain ever since HIV became big news that even if they knew for sure what they were testing for, it is often not that obvious what is causing a particular disease. Showing that something exists in the body is not the same as showing it causes a certain disease state.
What about dying people?
Are people dying? Yes, of course they are. Still, there are important questions to ask.
- Are people dying from what we are told they are dying from? We all know politicians lie and news media at least exaggerates. Why wouldn’t they be doing that now?
- Are people dying at higher rates that they died last year?
- Along the same lines, how have other death rates been affected?
Some day there could be a pandemic, but there is no solid evidence of that right now. If that was what was happening, people should be dropping like flies in the street. More politicians would be dying.
What about scientific data then?
Let’s refer back to the study of bats and mosquitoes. It was science, right? It was probably studied with honest interest. Still, we easily got a skewed picture of something apparently straightforward.
Why would we think studies about viruses are easily measured and interpreted? How many times has science been wrong or incompletely understood something? Why would we assume that now enough is known to be indisputably certain? Why are so many people angry if someone questions the narrative?
We can as least see bats and mosquitoes with our own eyes. Not only can we not view viruses without special tools and skills, but viruses are very similar to components found in human cells. It is conveniently difficult for us to verify what we are being told.
It is also useful to understand that most people who have studied these things are only repeating back what they have been told. They haven’t studied the phenomena or the cells or even read the data themselves. They are relying on what others tell them, but they present it like they know things they don’t.
In light of that, we should each be free to read and study the same things and come to our own conclusions. There is nothing about a degree that inherently makes one person’s thinking superior. There is nothing about a government approved license that makes someone a critical thinker.
One man’s snake oil
In every generation there are claims of cures and treatments that do not stand the test of time. Either those treatments are useless or they are harmful. Some people are skeptical about them sooner than other people. Some people are offended or threatened when you don’t agree with their cures.
Some people truly believed in snake oil. Some salesmen probably did to. Just because it is a term of derision today, doesn’t mean it was all an intended hoax then.
From our discussion, you could now call bats a snake oil treatment for mosquito control. They are probably in the somewhat harmful category, since they carry disease agents.
It is likely that masks are not the cure-all people think they are. They may even cause health issues by limiting the flow of cleansing air, as well as creating a warm, moist environment for pathogens to thrive in. Right next to people’s mouths.
We are beginning to hear that it may be the mistaken application of ventilators that killed many people. We will never know if they would have survived without that treatment. Those deaths get counted as evidence of an epidemic, but are they?
The siren call of catastrophe
Catastrophe and emergency sell. In a perverted sort of way, they give people something to be passionate about. The sense of urgency is used to justify actions that would otherwise be rude at best.
Such events, or the creation of them, give news media good ratings. They also give politicians a chance to be dictators under the guise of benevolence. They give some people the reason they need to push for threats of violence against those who don’t react the way they think they should to life.
These are all reasons we should dig deep into the basic assumptions of how scientific data is collected and presented. Scientific data tells us the measure of things. It brings possibilities to mind. It is up to us to carefully sift through the possibilities.
We should do this sifting with the understanding that any given set of data are probably one or more of the following:
- over simplified
- much more varied than reported
- misleading
- presented based on assumptions
- prone to variables not explained or understood
- taken from questionable samples
Whether it is reading the latest scientific discovery on best nutritional practices are or how to fertilize our garden, we should exercise a healthy skepticism. Remember that it takes time for honest research. It takes years for evidence to get properly sifted. And even then, proceed with caution in drawing conclusions with certainty.
Suggested further resources:
Corona: Creating the Illusion of a Pandemic Through Diagnostic Tests
Are Ventilators Killing COVOD patients?
Why Do We Trust Politicians to Run Our Lives?