Drift and Luck

In 1968, a Japanese geneticist named Motoo Kimura published a paper that changed how every evolutionary biologist thought about evolution. It was written in the language of population genetics, which is to say it was full of equations, and its central claim was that most evolutionary changes are not selected, they are neutral. Which meant that most evolutionary change was driven by chance.

He wasn’t talking about the evolution of eyes or wings which he readily agreed had been formed by natural selection. What he was talking about was the fact that the great bulk of the molecular sequence differences between species are selectively neutral. Then neither help nor hinder they just exist. They arise by mutation and they either spread to fixation or they disappear through the process of random fluctuation in finite populations, and this is orthogonal to fitness.

The process is called random genetic drift and 20 years later when I started university it made its way into the curriculum. It is in effect a coin flip repeated across generations and this means that in a population of finite size the frequency of a neutral variant will fluctuate randomly from one generation to the next, simply because not every individual reproduces and the ones that do are not always a perfectly representative sample of that population. Given enough time and enough generations every neutral variant will either drift all the way to fixation which means that everybody has it or it will drift to loss, which means that nobody has it. The important point is that the outcome is not determined by the quality of the variant it is determined entirely by look by chance.

Figure: A simulation of random drift for a neutral allele with an initial frequency of 0.5, over 500 generations. The gene had always either fixed or was lost before th 500th generation in each of these simulations. Simulation done on: https://udel.edu/~mcdonald/evoldrift.html

Kimura called this the neutral theory of molecular evolution, and he spent a lot of his career defending it against a scientific establishment that found it, in a word, offensive.

Darwin’s great insight, which made the theory of evolution so powerful, so satisfying, so world-reordering, was that the apparent design in nature could be explained by the simple, deterministic process of natural selection. Organisms vary and some variants are better able to survive and reproduce than others which causes them to spread overtime and in that way populations become adapted to their environment. This requires no designer nor does it require a plan just the relentless mechanical sifting of the fit from the unfit. The peacock has its tail because peahens preferred them that way. The cheetah is fast because slow cheetahs didn’t catch prey and they starved. All these observations had perfectly acceptable just-so explanations.

But then along Kimura’s neutral theory which says that most of it is not explained in this way. Most of it just happened and the variation you see within any population, and indeed most of the differences you see between you and a chimpanzee are not thanks to natural selection at all. This variation doesn’t exist because natural selection chose it, it exists because there was no selection against it, and in the absence of selection, drift took over, and drift is entirely random.

This was, for many evolutionary biologists, an assault on the explanatory programme that gave their field its power. If most molecular change is neutral, then most molecular change has no adaptive story. You can only describe it statistically, as a consequence of finite population size, mutation rate, and time. Evolutionary biology had spent a century building just-so stories about why everything in nature is the way it is. If you saw a spot pattern on the back of an animal, then you concluded it was for camouflage, and if you saw a particular beak shape on a bird you would conclude that it was optimised for cracking seeds. This new suggestion, that most change has no cause or no reason, was profoundly unwelcome.

Kimura was mostly right, though not about everything, and the exact boundary between neutral and selected change is still debated. But the core insight has been confirmed so thoroughly, by so many independent lines of evidence, that it is now part of the foundation of molecular evolution. The molecular clock, the idea that there is, roughly, a constant rate at which neutral substitutions accumulate in DNA over time, is one of the most useful tools in all of biology. It works (with caveats) because most substitutions are neutral, and they tick over at a rate determined by the mutation rate, not by the vagaries of natural selection, which means they function as a reliable timepiece (again, with caveats) for dating evolutionary divergences.

The strongest support for Kimura’s ideas comes from completed genomes. If natural selection was the primary driver, the instigator, the impetus behind molecular change, we should expect to see most DNA substitutions concentrated in functionally important regions, the bits of the genome that code for proteins, that regulate gene expression, that do something. Instead, the highest rates of substitution are found pseudogenes (broken, non-functional remnants of former genes), introns (non-coding sequences spliced out of messenger RNA before translation), and synonymous sites (positions in coding DNA where a change in the nucleotide doesn’t change the amino acid produced), the parts that are the least important for function. The parts of the genome where a change would make a difference evolve slowly, because selection constrains them. The parts where a change makes no difference evolve fast, because drift is unconstrained.

Imagine two organisms in the same population. One carries a DNA substitution that has no effect on survival or reproduction – a neutral variant. The other carries the original version. Over the next thousand generations, one of these variants will go to fixation and the other will be lost. The outcome will have nothing to do with the quality of the variant because the variant has no distinguishing quality. It is neutral and it doesn’t matter which of these alternatives goes to fixation and which is lost. How long it takes will depend on population size and that’s about it. In a large population the process takes longer and the odds of which one will win are close to even. In a small population drift is faster to get to that decision and more erratic. Variants drift to fixation or they vanish quickly. Kimura was careful to say that his theory applied to molecular evolution, to DNA substitutions, to the invisible churn of the genome.

But the boundaries are blurrier than anyone would like. Populations are often small, especially after bottlenecks, during colonisation events, on islands, at range margins, and in these populations, drift can overpower weak selection. A variant that is slightly beneficial can be lost by drift. And on the other side, a variant that is slightly harmful can be fixed by drift. The smaller the population, the more powerful drift becomes relative to selection, and the more the evolutionary trajectory of the population is determined by what just happened to be present in the survivors.

This means that some (not most, but some) fraction of the traits you see in living organisms are there by demographic accident, and not because they’re useful. A population doesn’t even have to go through a bottleneck for drift to fix a trait, and once it was fixed it became the starting material for subsequent evolution. The history of life is not a clean narrative of adaptive improvement. It’s a noisy, contingent, partly random walk through a space of possibilities, and some of the paths taken were taken for no reason at all.

Humans are, without question, the worst species on Earth at accepting the role of chance in outcomes. We are pattern-detectors of extraordinary, almost pathological sensitivity. We see faces and elephants in clouds, and intentionality and meaning in coincidences and accidents. Frankly, we are a nightmare. Without wishing to introduce my own just-co story, this was probably adaptive once, because it was better to mistake a shadow for a predator than to mistake a predator for a shadow. But it has left us with a cognitive architecture that is profoundly hostile to the concept of things happening for no reason.

The result is that we construct stories and we construct them about everything. The promotion you got was because of your hard work and strategic networking, while the promotion you didn’t get was because of office politics or a biased manager. The company that succeeded had a visionary leader and a superior product, while the company that failed had poor culture and bad strategy. Every narrative confirms that the world is legible and just, that outcomes are earned, that success is deserved, and that failure is explicable.

I am not saying that none of this is true. Natural selection is real, after all. Some outcomes really are determined by quality, effort, and strategy. The cheetah really is fast because natural selection favoured the faster variants. Some companies really do succeed because they’re better. But some of it is drift. Or to put it another way, it is luck. Is the result of which variants happened to be present in a small sample at a critical moment, propagated forward not because they were superior but because they were there.

An experiment run in 2006 by the sociologist Matthew Salganik and his colleagues tells us a lot about randomness and its relationship to outcomes. They created an artificial music market, which was a website where participants could listen to, rate, and download songs by unknown bands. One group of participants saw only the songs, with no social information attached. Eight other groups saw the same songs, but could also see how many times each song had been downloaded by other participants in their group. In the groups with social information, the popular songs became more popular, and the unpopular songs became more unpopular: this is the classic rich-get-richer dynamic. But what which songs became popular varied enormously between groups. A song that was a hit in one group was a nobody in another. The quality of the songs (as measured by the group with no social information) predicted popularity only weakly. What really predicted overall popularity was early luck. Those songs that happened to get a few early downloads would ultimately end up winners because this triggered the cascade of social proof that amplified initial noise into apparent signal.

The difference between these very different outcomes was drift. Small early fluctuations, amplified by positive feedback, producing outcomes that looked, after the fact, like they were determined by the inherent quality of the songs, when in fact they were determined by which way the coin happened to fall in the first few flips.

This is the idea of genetic drift transferred to a music market. And if it operates in music markets, it operates in job markets, in housing markets, in the market for ideas, and in the trajectory of individual lives. Some of what has happened to you happened for reasons, but without doubt, some of it is drift.

There is a famous essay by Stephen Jay Gould in which he asks you to imagine replaying the tape of life from the beginning with the same starting place, planetary changes and so forth. Would you get the same outcome? Would you get humans?

Gould was certain that the answer was no. He argued that is you replayed the tape, the Cambrian explosion would produce different body plans and then different groups would survive the mass extinctions. In the end, mammals never got their chance in this parallel universe, or they got it but radiated differently, or intelligence evolved in a different lineage, or it never evolved at all. The history of life is so saturated with contingency, with drift, with accidents that propagated forward into world-shaping consequences, that the outcome is only reproducible in broad strokes (something will evolve, complexity will probably increase) but not in specifics. You exist because of a particular sequence of events, many of which were not determined by anything more purposeful than a throw of a die.

Simon Conway Morris doesn’t agree with Gould, and has argued that convergence, the tendency for evolution to arrive at similar solutions independently, makes certain outcomes near-inevitable given enough time. Convergence is well-known and so the argument has merit. But even Conway Morris doesn’t dispute the role of contingency at finer scales. The specific lineages that produce the convergent solutions are not predetermined. Evolution may be destined to produce something like an eye, but it was not destined to produce *your* eye, in *your* skull, in a species that walks upright on the African savanna because a particular asteroid hit a particular spot sixty-six million years ago.

In a finite population under drift, some lineages go to fixation and some go to extinction, and the difference between the two is not quality. A perfectly viable lineage that is well-adapted and doing everything right can be eliminated by drift if the population is small enough and luck runs against it for long enough. This is not a theoretical possibility, it is a routine occurrence. It is how the contingency that Gould described actually operates, generation by generation, coin flip by coin flip.

The honest translation of all this is that you can do everything right and still lose. Not because you missed something, or because you weren’t good enough. And it is not because the universe is teaching you a lesson. You exist in a finite population, you are subject to stochastic variation, and sometimes the coin comes up tails enough times in a row that a perfectly good variant (a perfectly good person, a perfectly good idea, a perfectly good company) gets eliminated for no reason at all.

Nihilism says nothing matters, and this is not nihilism. Drift says some things matter, it says that selection is real, quality is real, effort is real, but they operate against a background of irreducible noise, and the noise is louder than anyone building something that is hard to build would like to admit. I’m talking about something like your brand, or your life, or your company.

There is, in the end, something genuinely freeing about knowing the relationship between selection, drift, contingency and extinction.

The freedom comes through in that explanation. Modern life has this relentless, exhausting insistence that every outcome in your life is accounted for, and that if you failed, there must be a reason, and if you succeeded, you must have earned it, and if something terrible happened, it must be the universe teaching you something. Neutral theory doesn’t tell you to stop trying. Selection is real. Adaptation is real. Effort matters. But neutral theory gives you permission to stop narrating. To stop constructing post-hoc stories about why things happened the way they did. To accept that some of the events that shaped your life may have been drift. May have been blind luck. May have had no more meaning than a neutral substitution drifting to fixation in a small population because that’s what neutral substitutions do.

The compulsion to explain everything is not wisdom. It’s actually a cognitive bias called the narrative fallacy. I think evolutionary biology provides the best possible correction for it. Four billion years of history, and a large fraction of it has no explanation beyond chance. Not because the universe is cruel or indifferent, but because that’s how finite populations work. That’s how stochastic processes work. That’s how reality works, whether you narrate it or not.

Kimura published his paper in 1968, and then he spent the next decades fighting for its acceptance. He was right, and the fight was bitter, and the reason it was bitter is that his opponents understood perfectly well what the neutral theory implied. It implied that the great explanatory engine of Darwinian biology, the machine that assigned reasons to every trait, and purposes to every feature, had a limited jurisdiction. It worked where it worked. But outside that jurisdiction, there was only drift. Only the purposeless, coin-tossing, statistically inevitable accumulation of changes that mean nothing.