Never Bet It All

What potatoes, a gambler and your own genome does to survive an unpredictable future

In the summer of 1845, a water mould came across the Atlantic in a shipment of seed potatoes and found, in Ireland, conditions of almost supernatural hospitality. The weather was cool and damp (tell me about it), and at that time, fields were planted edge to edge with a single high-yielding potato variety that had zero resistance to it. The mould is called Phytophthora infestans and it causes potato blight. The crop went black within weeks of the first infections. It failed again in 1846, and again in 1847, which has since become known in Ireland as “black ‘47”, and again in 1848.

The blight destroyed a crop. What killed a million people was a government, because Ireland went on growing enough food to feed everyone. Grain, butter and beef left the country under military escort, past people starving within sight of the quays, while the Whig government in London held that markets should be left to correct themselves and that Irish property should pay for Irish poverty. Relief was made conditional on surrendering all but a quarter-acre of land, and this turned mass starvation into mass eviction.

Monoculture is the explanation for why the arrival of this one pathogen decimated an entire section of the food supply, rather than just damaging some of the potato crop. If a population is forced to stake its subsistence on one genetic solution, one package of traits, resistances and vulnerabilities, then as soon as the environment changes, that solution has no alternatives to call upon. There was no second variety, and no other crop was available to be planted instead. It was a massive bet on a single genotype, and that genotype was wiped out.

Peruvian potato cultivars - Photo attribution unknown.

Harry Markowitz formalised portfolio theory in a 1952 paper¹, work that would win him a Nobel Prize decades later. The problem he was solving is one that bacteria have been solving ever since they came into being. Organismal life faces the same problem an investor faces. The future is uncertain, the environment will change, and the direction of that change is unknowable in advance. You can observe the present and do your best to adapt to it. You can be the best possible organism for the present time. But the situation you face today is temporary and may not be here tomorrow. The climate will change, or a competitor will migrate in, or a parasite will evolve some new trick, or your favourite food source will vanish. The conditions that fitted best with your traits are replaced by conditions that will punish you for having them, and if you make the mistake of committing wholeheartedly and irreversibly to whatever worked yesterday, then there will be no alternative to call upon if you need it tomorrow.

It seems obvious that the best strategy is always to be excellent at what currently works and to put all your resources into the traits that maximise success under present conditions. If conditions remain unchanged, this is the best idea, and it looks like genius and is hugely successful. The specialist outcompetes everyone. The Irish variety of potato in 1843 yielded more potatoes per acre than any other variety, so mixed planting was never considered. Yield was everything. While the returns from specialist varieties are spectacular, they are fragile, because they depend on conditions staying as they are.

The other response is to carry backup plans, to retain genes in your genome that do nothing for you today, and to tolerate variation in your population that does not currently pay. Accept lower returns now in exchange for a broader range of responses to unpredictable futures.

Which strategy wins, the specialist’s or the diversifier’s, depends entirely on the time horizon. Over a few generations, or a single environmental regime, the specialist wins every time, because it is optimised for the here-and-now while the diversifier is carrying dead weight. Over the sort of interval across which environments actually change, the diversifier wins, for the simple reason that it is still present.

If we were to sequence a single cell of Escherichia coli from your gut we would identify something like four to five thousand genes, which is roughly what it takes to run a bacterial cell. The genome will contain genes for metabolism, replication, transcription, translation, membranes, the machinery for sensing what is going on outside, and so forth. Now let’s say we sequence a different strain, from a different gut. Again we find four to five thousand genes, but these won’t be quite the same four to five thousand. Some will be shared between the two genomes, while others will not. The second strain will almost certainly have genes that are absent in the first, and the first will have genes the second won’t have. Repeat this process a thousand times over, catalogue everything you find, and the sum total of all the genes you catalogue will be closer to twenty-five thousand, and that number will keep climbing with every strain you add. For practical purposes the pangenome of E. coli, the totality of the gene content across a collection of organisms, may be open, meaning there is no finite gene count that defines the species. There is always another gene out there, in another strain, that we haven’t seen yet.

Any individual cell therefore carries a fraction of the genes available to its species. The remainder is distributed across other cells in other places, doing other work. The species maintains a library. Each cell borrows a subset, and the subset it borrows determines what it can do: which sugars it can eat, which antibiotics it survives, and where it can live. The library is always larger than any individual borrower.

When conditions change, the population does not have to invent a solution, because some fraction of the cells already has one. That fraction of cells survives, the others do not, the composition of the population changes, and the species persists. No individual cell was specifically prepared for the change.

In 1956, four years after Markowitz, a physicist at Bell Labs named John Kelly Jr. published a paper on how a gambler with an edge should bet². Suppose you have information that puts the odds in your favour, but no knowledge of the outcome of any particular wager. What fraction of your bankroll should you stake? The intuitive answer, which is to bet as much as possible, turns out to be catastrophic. Maximising expected return also maximises the probability of ruin, because a long enough sequence of favourable bets will still contain a losing streak, and any gambler who stakes and loses her entire pot of money does not survive to collect on the favourable odds afterwards.

Kelly’s answer is to calculate a specific fraction, determined by the odds and the size of the edge, and it is always less than everything. Usually far, far less. It underperforms the maximum-bet strategy when you are having a run of luck, though it has the compensating property of almost never reaching zero. Over a long enough horizon the Kelly bettor ends up wealthier than the maximum bettor could ever have become, for the unglamorous reason that the Kelly bettor is still at the table.

This is quite like a bacterial population that maintains an open pangenome by carrying genes it does not currently need. The genome is paying the metabolic cost of synthesising the extra DNA, tolerating variation with no obvious use right now. It is investing less than everything in the current environment. The bankroll is the population, the stake is the genome, the edge is the diversity held in reserve, and the horizon is evolutionary time, which might be just about as long as a horizon can get.

Biologists call this bet-hedging.

Desert annual plants are the classic case of bet-hedging. Rainfall in any given year may be abundant, moderate, or effectively absent, and if the entire seed output of a plant species germinates all at once, this is equivalent to making a single wager on this year’s weather. If the rains come, then it wins enormously, its offspring blooming everywhere. If the rains do not, then every last seedling dies, and the lineage ends there. A plant that germinates only some fraction of its seeds, leaving the rest dormant in the soil, produces fewer seedlings in every good year, and in those years it looks like it is losing. But when the drought arrives, and the drought always arrives, the all-in germinator is wiped from the system and the hedger is hunkering down, waiting for next year.

Some bacteria hedge within a single clonal population, switching stochastically between metabolic states, or between levels of antibiotic tolerance, even though they get no external signal and perhaps nothing outside has changed. To us, this can look like noise, or like sloppy regulation. But we now know that a species that switches into a state that is less-well suited to current conditions is not actually making an error. In some distant future, it will be the ancestor of many descendants. The population is buying insurance, and the premium it pays is reduced average fitness.

That last point rails against the instinct most people have about how they are going to succeed in life, which is to find out what works and then do more, and more, and more of it. The whole genre of motivational videos runs on this instinct: commit, double down, half measures are for losers.

Commitment is not always wrong. In a stable, predictable environment, full specialisation is mathematically optimal, and the diversifier is wasting resources. Everything depends on that word predictable. No living being has ever had that long-term guarantee. Ecosystems, economies, careers and lives are all unpredictable over the long term. It is an intrinsic property of what they are. Under genuine uncertainty, which is different from quantified risk, the optimal stake is always less than everything.

Diversification is expensive. In any given generation, in any given year, the hedger underperforms the all-in specialist. The bacterium carrying genes it does not use pays a metabolic tax at every replication. The desert annual keeping seeds dormant is forgoing reproduction it could have had. The Andean farmer who plants a dozen different potato varieties in one field harvests less per acre than the neighbour who plants only the best cultivar.

Andean farmers are the people who domesticated the potato in the first place. The Quechua and Aymara of Peru and Bolivia cultivated hundreds, even thousands, of varieties across the region, all differing in colour, shape and size, in frost and drought tolerance, in susceptibility to disease. A single field might have a dozen of them planted side by side. This means the yield this year will be lower than if the field carried a monoculture of the single best variety. But look at the aggregate performance of the field, averaged across good years and bad, across pest outbreaks and droughts and the ordinary perturbations of an Andean season, and you will find that this diversified crop brings in the higher yield over time. Several centuries of direct experience of what happens when you put everything on a single variety have led to the current system.

In human terms the cost shows up as a nagging sense of falling behind. The person who keeps friendships outside their industry, pursues hobbies that don’t advance their career, learns a skill with no immediate application (mine is whittling wood, what’s yours?), and declines to commit fully to the most promising career path will, in any given year, be visibly outpaced by the colleague who went all-in. In my line of work, I see the colleague who is publishing more, earning more, rising faster through the university hierarchy. In that year, the colleague is winning. The question is only whether you are optimising for this year or for all of them.

The pangenome of Escherichia coli holds at least fifty times as many genes as any individual E. coli cell. Roughly ninety-five percent of the genetic portfolio of the species is, at any given moment, being carried by somebody else. Most of the solutions to the uncertain future are out there, distributed, in reserve, earning nothing for the cells that carry them. To an efficiency-minded observer this is staggering, perhaps unforgivable, waste. The species should strip out the redundancy, eliminate the genes that are not being used right now, converge every cell on the single best genome for current conditions, and watch how this leaner, streamlined organism goes!

You would also have the potato crop of 1845.

What looks like waste is the difference between persistence and extinction. It is what allows a species to answer environmental shifts that have not happened yet, changes that have never been encountered before. It is the fraction of the bankroll that Kelly says you must never stake. It is the seed the desert annual leaves in the soil. And it is the wood I whittle of an evening, which will never earn me a penny.

---

Note: I am writing a book that will be published in 2027. If you like my content, please consider buying a copy of the book when it is published.

References:

1. Markowitz, H. (1952), PORTFOLIO SELECTION*. The Journal of Finance, 7: 77-91. https://doi.org/10.1111/j.1540-6261.1952.tb01525.x

2. Kelly, J.L., Jr. (1956), A New Interpretation of Information Rate. Bell System Technical Journal, 35: 917-926. https://doi.org/10.1002/j.1538-7305.1956.tb03809.x