Walk along this shore in Hobart and you are walking in the footsteps of Charles Darwin. Darwin came here 190 years ago on the final leg of his voyage on the Beagle. Darwin of course then went home to England and became famous for creating the theory of evolution by natural selection.
Of course the muwinina people walked here for thousands of years before Darwin did. The muwinina are now sadly extinct, a victim of colonialisation.
There’s a surprising historical connection between the way we do modern statistics and these events - the discovery of evolution and the atrocities committed by colonial powers.
Its the dirty secret about stats you’ve probably never been taught.
The European nations came to power in time of growing awareness about scientific reasoning. That scientific reasoning gave them technologies that enabled their conquests. It also gave them an excuse for exploration and conquest, that it was in the pursuit of scientific knowledge.
But Darwin’s discovery of evolution by natural selection revealed an awkward reality for the European powers. If all people were descended from the same ape-like ancestors, how could they claim European superiority over other races? The empire struggled to deny the truth in Darwin’s theory. Darwin’s theory came from the same scientific reasoning tools that had given us so many other obvious impressive technological developments.
The solution of course was to show that white Europeans had evolved a superior intellect. That way it could be justified why they deserved to rule the rest of the world.
This colonial racism carries through the 19th century to set the scene for statistical methods that were pioneered in the first half of the 20th century.
These are methods like p-values, t-tests, ANOVAs and maximum likelihood that are still commonly used today. All of these methods are framed under the viewpoint of probabilities as frequencies of events - the frequentist perspective.
Until recently these methods underpinned almost all scientific inferences. Everything from drug approvals to psychological studies of human behaviour to climate change studies.
Frequentist methods and their application in the scientific method were pioneered and advocated for by three English mathematicians and biologists, Galton, Pearson and Fisher. All were active in the eugenics movement. For example Fisher was chairman of a prominent eugenics society. The eugenics movement aimed to improve the genetic quality of the human race, usually through advocating for breeding or sterilisation of parts of society on racial or class grounds.
Fisher for example wrote that the oncoming collapse of civilization was the fault of increased birth rates in lower classes who he thought were of lesser intellectual character. What the empire needed to do was slow those birth rates and promote procreation in the intellectually superior elite classes.
These were contentious ideas. And contentious ideas needed to be supported by a robust scientific logic that was unassailable by critics. These statisticians and biologists needed a theory that above all appeared to be objective. It is hard to argue with a method that claims to objectively show that elites and white people were superior than other people.
These three scientists developed that statistical theory. A theory based on the frequentist viewpoint of probability.
The frequenist viewpoint looks purely objective on the face of it. Probabilities are frequencies of events, what could be more objective that that?
Frequentist methods are very useful, but they have their limits. They are great for analysing well replicated experiments for instance.
But the frequenist logic starts to break down when you want to make decisions from your science. Frequentist methods can misinform you about the best actions to take when presented with surprising new data.
They are also not very useful for phenomena that can’t be easily studied in controlled experiments, such as the link between smoking and lung cancer. Fisher himself was a prominent denier that smoking causes lung cancer.
Rare events are also hard to study, as are events in nature that are not replicated. For instance, climate change. Its hard to use frequentist methods to establish the role of greenhouse gas emissions in climate change. We only have one earth, we can’t do repeated experiments on it.
A solution to these types of problems is to use Bayesian statistical theory. However, the Bayesian viewpoint of probability is that probability represents degrees of belief for something happening. This view makes the subjectivity of data analysis obvious. That is, different people can have different degrees of belief about the probability of the same event happening.
The brilliant statistical minds of the early 20th century needed statistics to look objective above all else.
Even a hint of subjectivity would weaken their arguments for eugenics. They wanted to show for instance that Jewish immigrants had weaker immune systems than the general population, so therefore should not be allowed to immigrate to England. They claimed these were facts in the data, so objective that arguing against this view impossible.
This strong motivation for objectivity also motivated the oppression of alternative statistical methods, particularly the Bayesian viewpoint. So while these statisticians gave us so much in terms of methods, they also held us back in other areas such as Bayesian statistics.
The culture of aggressively promoting the frequentist viewpoint at the cost of alternative methods was still strong when I was a young research in the 2000s (well after Fisher’s death). It persists today in some places.
Right into the 2000s you were risking academic ridicule if you dared to mention Bayesian statistics in a conference presentation. Scientific dialogue still carried the scars of a toxic culture that arose in the 20th century to support a racist ideology, even though the new perpetrators of this statistical cause did not agree with that ideology (we hope).
I think this has been a net loss to many important areas of science. In the environmental sciences we struggle with sample sizes and studying natural phenomena that are hard to manipulate experimentally.
There are many cases of misinterpreting a non-significant p-value to mean ‘no impact of humans of environment’, whereas a Bayesian viewpoint that builds on the accumulation of prior knowledge can detect these effects in small samples.
We also have the repeatability crises in psychology and biology, which comes about from over-emphasis of the publication system on significant p-values.
Bayesian statistics have risen to prominence only in the last 20 years. They should be much more widely used, exactly because they deal with subjectivity and make the views of the analyst transparent to others.
Frequentist methods are not more objective, they just hide the subjectivity underneath a show of statistical rigour and jargon.
I’ve been doing stats for about 20 years now, but only recently learned about this history. Its made me rethink the role of the quantitative sciences in science and society.
Leading statisticians are also making this reckoning only recently. Andrew Gelman, prominent Bayesian statistician and educator has said “Fisher’s personal racist attitudes do not make him special [in the context of his era], but in retrospect I think it was a mistake for us in statistics for many years to not fully recognize the centrality of racism to statistics, not just a side hustle but the main gig.”
If you want to know more about this statistical history I recommend you read Bernoulli’s Fallacy by Aubrey Clayton. And for a broader understanding of the British empire and its ongoing impact, the podcast ‘Stuff the British Stole’ is a great listen (I find the episodes with British experts insisting that the stuff in question was ‘given’ to them particular informative about the ongoing legacy of empire).
Walking in Darwin’s footprints reminds me of the importance of science in our society. But also how science facts can be portrayed in different ways to suit political means.
As a quantitative scientist its important to acknowledge our work will always have subjective elements. We should be transparent about those, not try to hide those under complex methodology.
That’s why I’ve grown to favour Bayesian statistics. I want my assumptions to be transparent.