During my undergraduate program, I had to take Probability and Statistics. As part of our course, we had to create a project, collect data, analyze it, and present. We decided to poll students and look for correlations between parts of the student body and the results we found. Unfortunately, the data showed ZERO correlation between the answers and any group in any way we identified them. With trepidation, we presented this to our instructor who much to our surprise clapped and congratulated us for not trying to find correlation in the data set when none existed.
People like to find meaning and feel like their work matters. Most of the students who passed through this instructor’s classroom had gone to great lengths in order to demonstrate that their data corroborated their preconceived notions, which is precisely the problem. All too often, people are not looking for the truth; they hope the truth will corroborate what they already happen to believe.
They achieve this by messing with the sampling, which takes several forms. Firstly, you can manipulate the data set that you sample and sample only those data sets that are likely to corroborate the claim. Secondly, you can manipulate the data you include to bias the outcome to the one you prefer. Finally, you can manipulate the data’s appearance so that it shows what you want it to say.
You see this all the time when you look at polls, surveys, and anything that comes out of social science. A good person will report the sampling, and inevitably there is bias in whom they ask. You have probably never responded to a poll, but you see them all the time when they claim that “A majority of Americans feel a certain way” but you don’t know anyone in your circle who agrees with the claim of the survey. You must ask yourself whom they decided to ask. Did they go looking for people they expected to answer a certain way? We all have biases. Do the researchers admit their sampling might be biased?
Unfortunately, you see this in classical fields of scientific endeavors as well. For example, the climate stations that collect temperature and rainfall information may not be in the best locations. There is one for example alongside a major highway in Las Vegas close to a red light, which might overreport smog or pollution depending on how many cars are idling there at the light. There is another in a low spot just west of the University of Nevada, Reno which overreports dew because the fog will sit in the basin and overreport precipitation or the rate thereof. Even research science can be biased. Not everything is easy to secure funding to investigate, and even if the scientists are above board sometimes the people that work for them are either unskilled or unaware enough that they omit data or overreport it. If you’re looking for desert tortoises, unless you tag them as you find them how can you be sure you didn’t double count them?
Much of data is full of lies, damn lies, and statistics. Even the journal “Nature” which is a top tier peer reviewed journal for scientific research admits that it knows that there is bias in science. If medical studies are flawed, you can bet your bottom dollar that other polls, surveys, and research endeavors are flawed, faked, or even fraudulent. People who believe that their political party is above board but that the other party is all liars need a reality check. People are people, and anyone can lie. About what we lie, how frequently and to whom is what determines or should determine the veracity of our claims. Everyone has a track record.
So how do you tell? You follow the money. Find out who paid for it. Anyone who claims to be objective is probably a liar. The funding source is key to determining the objectivity of the data. If a winery pays for a study on the health effects of regularly drinking red wine, it’s probably neither correlated nor causal. If a textbook manufacturer funds a study on physical textbooks being superior to online learning, it’s probably a self-licking ice cream cone. Look at the sample sizes, the location of the study, and their statistics. ANOVA tells you things that the pollsters want to keep hidden.
And if they base their claims on outliers, just run. Sir Arthur Conan Doyle criticized this in “A Study in Scarlett” where he warned against forming theories before you collect facts lest you bend facts to fit theories. And that is exactly the problem.
