Improbable probability

Chance. Fate. Coincidence. Things may happen for a reason – but as Professor David Hand explains, that reason is maths, not magic.

Have you ever had anyone tell you that “things happen for a reason”, by which they mean that there are mysterious, unrecognised causes underlying extraordinary coincidences? Or that thinking of someone just before they called you on the phone meant there was some sort of telepathic link? Or that your dream about an accident on the night before a major road crash meant a connection between your dream and the future?

It has certainly happened to me. And my response is that, yes, these things do happen for a reason. But that reason is not some mystic or supernatural connection. It is, in fact, well understood: the link underlying these coincidences is the mathematics of chance. But looked at from a rather unusual perspective: the perspective that I call the ‘improbability principle’. This principle essentially says that highly improbable events are commonplace.

The improbability principle is not a single law. Rather, it’s a set of laws – rather like Newton’s three laws of motion, or the four laws of thermodynamics. In this case, it’s five laws.

Perhaps the most straightforward of these is the law of inevitability. This says that one of the set of all possible outcomes must occur. If I throw a die, it is inevitable that it will land showing one of the numbers from one to six. Like the other laws of the improbability principle, this law often manifests itself in other and sometimes quite subtle ways.

For example, while there is no guarantee that, in a conference with 1,000 delegates, someone else will have the same birthday as me, it is certain that some two people will have the same birthday.

Indeed, I can guarantee holding the jackpot winning lottery ticket if I buy all combinations of possible numbers – though this might be expensive! In fact, several groups of people have sought to take advantage of this, by buying up all possible lottery ticket combinations when a rollover jackpot became large enough.

For example, the so-called International Lotto Fund aimed to spend $7m to buy all tickets for a $27m rollover jackpot. It works, but it takes some organising – and there’s no guarantee that you won’t have to share the jackpot.

Truly large numbers

The second law of the principle, the law of truly large numbers, says that even an outcome which has a tiny chance of occurring becomes almost certain if you give it enough opportunities.

For example, if you generated a random sequence of eight digits, you’d be amazed if they just happened to form your birth date. But if you look through the infinite number of seemingly random digits in the decimal expansion of the mathematical constant pi, you will find your birth date eventually. For example, my birth date, 30 June 1950 – 30061950 – starts at digit 190,652 (you can try it for yourself).

The law of truly large numbers also explains why extraordinary lottery coincidences have occurred, such as people winning lotteries multiple times. While the chance that it might happen to any one person is microscopically small, there are so many people playing so many lotteries so many times that it becomes almost inevitable.

One way of putting the third of the laws, the law of selection, is that it tells us we can make events as likely as we want if we choose after the event. I can make it certain that the next throw of a die will be no larger than the last one – by choosing to follow up only those throws that show a six.

One of my favourite examples of this law occurred during the First World War. The British military were experimenting with dropping bundles of metal darts from aircraft onto enemy troops. To explore how the darts dispersed, experiments were conducted, dropping the darts onto a field and then putting squares of paper over the darts to show where they had landed. A cavalry officer rode up and, when told that there was a metal dart in the centre of each square, commented: “I didn’t think you could be so accurate from an aircraft.”

Beware the black swan

A fourth law, the law of the probability lever, tells us that if we slightly change the circumstances or assumptions we can dramatically change the probabilities. A classic example of this is the so-called ‘black swan’ phenomenon in the financial sector, where dramatic market crashes seem to occur much more frequently than they should.

But this is only if one calculates their probability making assumptions that don’t accord with reality. For example, if we assume that the sizes of market movements are distributed according to a normal distribution then we seriously underestimate the chance of such events. If, however, we use a so-called ‘fat-tailed’ distribution – in which extreme outcomes are more likely than you might expect – then they occur with the sort of frequency one would expect.

Another manifestation of the law of the probability lever is when we assume that events are independent when in fact the occurrence of one makes the other more likely. Then we can significantly underestimate the probability of coincidences.

This was the case in the trial of the British solicitor Sally Clark in 1999 for the murder of her two children, when the paediatrician Professor Sir Roy Meadow was – subsequently – deemed to have “grossly” underestimated the chance of two sudden deaths within the same family from natural causes.

The final law of the improbability principle is the law of near enough. This says that if you expand what you mean by a coincidence or improbable event sufficiently you can make it as likely as you want. There’s a one in 365 chance that my birthday is on the same date as that of a randomly chosen reader. But a one in 52 chance that it’s in the same week. And a one in 12 chance that it’s in the same month.

When laws come together

Last year, I was invited to give a talk about the improbability principle in Bournemouth, a town I’d lived in when I was a child. By coincidence, the venue was in Coronation Avenue, a road I’d walked along every weekday for some years, on my way from my home to my junior school. I hadn’t been back to Bournemouth since I was a child, and it seemed an amazing coincidence to be invited back to that particular road. But what if instead the venue had been in a street next to the one I’d walked along, or nowhere near it but I’d simply been driven along that street to get to the venue, or ... Not quite as sharp a coincidence as occurred, but near enough in each case to have made me say: “What a coincidence!”.

In Chapter 39 of his novel The Old Curiosity Shop, Charles Dickens writes a wonderful passage in which the mothers of two characters marvel about the coincidences in their lives, discovering “sundry circumstances that tallied with wonderful exactness” but all of which are absurd expansions of the conditions for a match. It’s a perfect illustration of the law of near enough.

As the examples above show, each of the five laws of the improbability principle can make apparently highly improbable events much more likely. But when the laws work together, the result can be even more startling. A bookmaker’s chance of winning any particular bet may be only a tiny amount greater than yours. But with many bets, placed time after time, things add up. They are taking advantage of the law of the lever (their chance of winning each bet is slightly greater than yours) and the law of truly large numbers (their chance of losing over a great many bets is tiny). Do you know any poor bookmakers?

Coincidences are striking because of their personal meaning. Soon after my book about the improbability principle was published, I was contacted by a novelist named John Ironmonger. He’d just had a book called The Coincidence Authority published. This described a researcher who was an expert on coincidences, who lived in London (as I do), with the female protagonist being a University of London academic (as is my wife), and whose birthday was 30 June (as is mine). He said he’d never heard of me. It was just an extraordinary coincidence.

When I give talks about the improbability principle, I’m often asked if understanding the cause of apparently miraculous coincidences makes the magic go away. Does it reduce the world to a grey uniformity?

My answer is that the opposite is the case. When you understand the improbability principle, and appreciate the elegance of the mathematics constituting its five laws, and the inevitability of the coincidences that they imply, then things seem all the more amazing. Rather than making the sense of wonder vanish, it makes the world all the more awe-inspiring and beautiful.

It’s a lottery

Professor Hand explains why we overestimate our chances of winning the lottery – and underestimate the chances of being struck by lightning

Generally, people don’t have a very good feeling for very small probabilities. My guess is that, say, one in one thousand is sufficiently small that people will regard it as equivalent to one in a million. Perhaps both satisfy Borel’s Law – they are so small one should act as if neither will happen.

Of course, that does not explain why people play the lottery. A one in 45 million chance means it’s less likely than getting 25 consecutive heads when tossing a fair coin. I suspect that if you gave people the chance to pay £2 to win £1m if a fair coin came up heads on 25 consecutive tosses, you wouldn’t get many takers. There is also another point. With things like the lottery, or tosses of a fair coin, we can work out the probabilities exactly. But with things like being struck by lightning, or hit by a meteorite, it’s difficult to work them out. And when probabilities are very small (as in these two examples) the estimates of the probability are very rough.