Fooled by Randomness Read online

  TWO: A BIZARRE ACCOUNTING METHOD

  On alternative histories, a probabilistic view of the world, intellectual fraud, and the randomness wisdom of a Frenchman with steady bathing habits. How journalists are bred to not understand random series of events. Beware borrowed wisdom: How almost all great ideas concerning random outcomes are against conventional sapience. On the difference between correctness and intelligibility.

  THREE: A MATHEMATICAL MEDITATION ON HISTORY

  On Monte Carlo simulation as a metaphor for understanding a sequence of random historical events. On randomness and artificial history. Age is beauty, almost always, and the new and the young are generally toxic. Send your history professor to an introductory class on sampling theory.

  FOUR: RANDOMNESS, NONSENSE, AND THE SCIENTIFIC INTELLECTUAL

  On extending the Monte Carlo generator to produce artificial thinking and compare it with rigorous nonrandom constructs. The science wars enter the business world. Why the aesthete in me loves to be fooled by randomness.

  FIVE: SURVIVAL OF THE LEAST FIT—CAN EVOLUTION BE FOOLED BY RANDOMNESS?

  A case study on two rare events. On rare events and evolution. How “Darwinism” and evolution are concepts that are misundderstood in the nonbiological world. Life is not continuous. How evolution will be fooled by randomness. A prolegomenon for the problem of induction.

  SIX: SKEWNESS AND ASYMMETRY

  We introduce the concept of skewness: Why the terms “bull” and “bear” have limited meaning outside of zoology. A vicious child wrecks the structure of randomness. An introduction to the problem of epistemic opacity. The penultimate step before the problem of induction.

  SEVEN: THE PROBLEM OF INDUCTION

  On the chromodynamics of swans. Taking Solon’s warning into some philosophical territory. How Victor Niederhoffer taught me empiricism; I added deduction. Why it is not scientific to take science seriously. Soros promotes Popper. That bookstore on Eighteenth Street and Fifth Avenue. Pascal’s wager.

  EIGHT: TOO MANY MILLIONAIRES NEXT DOOR

  Three illustrations of the survivorship bias. Why very few people should live on Park Avenue. The millionaire next door has very flimsy clothes. An overcrowding of experts.

  NINE: IT IS EASIER TO BUY AND SELL THAN FRY AN EGG

  Some technical extensions of the survivorship bias. On the distribution of “coincidences” in life. It is preferable to be lucky than competent (but you can be caught). The birthday paradox. More charlatans (and more journalists). How the researcher with work ethics can find just about anything in data. On dogs not barking.

  TEN: LOSER TAKES ALL—ON THE NONLINEARITIES OF LIFE

  The nonlinear viciousness of life. Moving to Bel Air and acquiring the vices of the rich and famous. Why Microsoft’s Bill Gates may not be the best in his business (but please do not inform him of such a fact). Depriving donkeys of food.

  ELEVEN: RANDOMNESS AND OUR MIND: WE ARE PROBABILITY BLIND

  On the difficulty of thinking of your vacation as a linear combination of Paris and the Bahamas. Nero Tulip may never ski in the Alps again. Do not ask bureaucrats too many questions. A Brain Made in Brooklyn. We need Napoleon. Scientists bowing to the King of Sweden. A little more on journalistic pollution. Why you may be dead by now.

  TWELVE: GAMBLERS’ TICKS AND PIGEONS IN A BOX

  On gamblers’ ticks crowding up my life. Why bad taxi-cab English can help you make money. How I am the fool of all fools, except that I am aware of it. Dealing with my genetic unfitness. No boxes of chocolate under my trading desk.

  THIRTEEN: CARNEADES COMES TO ROME: ON PROBABILITY AND SKEPTICISM

  Cato the censor sends Carneades packing. Monsieur de Norpois does not remember his old opinions. Beware the scientist. Marrying ideas. The same Robert Merton putting the author on the map. Science evolves from funeral to funeral.

  FOURTEEN: BACCHUS ABANDONS ANTONY

  Montherlant’s death. Stoicism is not the stiff upper lip, but the illusion of victory of man against randomness. It is so easy to be heroic. Randomness and personal elegance.

  PROLOGUE

  MOSQUES IN THE CLOUDS

  This book is about luck disguised and perceived as nonluck (that is, skills) and, more generally, randomness disguised and perceived as non-randomness (that is, determinism). It manifests itself in the shape of the lucky fool, defined as a person who benefited from a disproportionate share of luck but attributes his success to some other, generally very precise, reason. Such confusion crops up in the most unexpected areas, even science, though not in such an accentuated and obvious manner as it does in the world of business. It is endemic in politics, as it can be encountered in the shape of a country’s president discoursing on the jobs that “he” created, “his” recovery, and “his predecessor’s” inflation.

  We are still very close to our ancestors who roamed the savannah. The formation of our beliefs is fraught with superstitions—even today (I might say, especially today). Just as one day some primitive tribesman scratched his nose, saw rain falling, and developed an elaborate method of scratching his nose to bring on the much-needed rain, we link economic prosperity to some rate cut by the Federal Reserve Board, or the success of a company with the appointment of the new president “at the helm.” Bookstores are full of biographies of successful men and women presenting their specific explanation on how they made it big in life (we have an expression, “the right time and the right place,” to weaken whatever conclusion can be inferred from them). This confusion strikes people of different persuasions; the literature professor invests a deep meaning into a mere coincidental occurrence of word patterns, while the economist proudly detects “regularities” and “anomalies” in data that are plain random.

  At the cost of appearing biased, I have to say that the literary mind can be intentionally prone to the confusion between noise and meaning, that is, between a randomly constructed arrangement and a precisely intended message. However, this causes little harm; few claim that art is a tool of investigation of the Truth—rather than an attempt to escape it or make it more palatable. Symbolism is the child of our inability and unwillingness to accept randomness; we give meaning to all manner of shapes; we detect human figures in inkblots. I saw mosques in the clouds announced Arthur Rimbaud, the nineteenth-century French symbolic poet. This interpretation took him to “poetic”Abyssinia (in East Africa), where he was brutalized by a Christian Lebanese slave dealer, contracted syphilis, and lost a leg to gangrene. He gave up poetry in disgust at the age of nineteen, and died anonymously in a Marseilles hospital ward while still in his thirties. But it was too late. European intellectual life developed what seems to be an irreversible taste for symbolism—we are still paying its price, with psychoanalysis and other fads.

  Regrettably, some people play the game too seriously; they are paid to read too much into things. All my life I have suffered the conflict between my love of literature and poetry and my profound allergy to most teachers of literature and “critics.” The French thinker and poet Paul Valery was surprised to listen to a commentary of his poems that found meanings that had until then escaped him (of course, it was pointed out to him that these were intended by his subconscious).

  More generally, we underestimate the share of randomness in about everything, a point that may not merit a book—except when it is the specialist who is the fool of all fools. Disturbingly, science has only recently been able to handle randomness (the growth in available information has been exceeded only by the expansion of noise). Probability theory is a young arrival in mathematics; probability applied to practice is almost nonexistent as a discipline. In addition we seem to have evidence that what is called “courage” comes from an underestimation of the share of randomness in things rather than the more noble ability to stick one’s neck out for a given belief. In my experience (and in the scientific literature), economic “risk takers” are rather the victims of delusions (leading to overoptimism and overconfidence with their underestimation of
possible adverse outcomes) than the opposite. Their “risk taking” is frequently randomness foolishness.

  Consider the left and the right columns of Table P.1. The best way to summarize the major thesis of this book is that it addresses situations (many of them tragicomical) where the left column is mistaken for the right one. The subsections also illustrate the key areas of discussion on which this book will be based.

  The reader may wonder whether the opposite case might not deserve some attention, that is, the situations where nonrandomness is mistaken for randomness. Shouldn’t we be concerned with situations where patterns and messages may have been ignored? I have two answers. First, I am not overly worried about the existence of undetected patterns. We have been reading lengthy and complex messages in just about any manifestation of nature that presents jaggedness (such as the palm of a hand, the residues at the bottom of Turkish coffee cups, etc.). Armed with home supercomputers and chained processors, and helped by complexity and “chaos” theories, the scientists, semiscientists, and pseudoscientists will be able to find portents. Second, we need to take into account the costs of mistakes; in my opinion, mistaking the right column for the left one is not as costly as an error in the opposite direction. Even popular opinion warns that bad information is worse than no information at all.

  However interesting these areas could be, their discussion would be a tall order. There is one world in which I believe the habit of mistaking luck for skill is most prevalent—and most conspicuous—and that is the world of markets. By luck or misfortune, that is the world in which I have operated most of my adult life. It is what I know best. In addition, economic life presents the best (and most entertaining) laboratory for the understanding of these differences. For it is the area of human undertaking where the confusion is greatest and its effects the most pernicious. For instance, we often have the mistaken impression that a strategy is an excellent strategy, or an entrepreneur a person endowed with “vision,” or a trader a talented trader, only to realize that 99.9% of their past performance is attributable to chance, and chance alone. Ask a profitable investor to explain the reasons for his success; he will offer some deep and convincing interpretation of the results. Frequently, these delusions are intentional and deserve to bear the name “charlatanism.”

  If there is one cause for this confusion between the left and the right sides of our table, it is our inability to think critically—we may enjoy presenting conjectures as truth. It is our nature. Our mind is not equipped with the adequate machinery to handle probabilities; such infirmity even strikes the expert, sometimes just the expert.

  Table P.1 Table of Confusion

  Presenting the central distinctions used in the book

  * * *

  The nineteenth-century cartoon character, pot-bellied bourgeois Monsieur Prudhomme, carried around a large sword with a double intent: primarily to defend the Republic against its enemies, and secondarily to attack it should it stray from its course. In the same manner, this book has two purposes: to defend science (as a light beam across the noise of randomness), and to attack the scientist when he strays from his course (most disasters come from the fact that individual scientists do not have an innate undderstanding of standard error or a clue about critical thinking, and likewise have proved both incapable of dealing with probabilities in the social sciences and incapable of accepting such fact). As a practitioner of uncertainty I have seen more than my share of snake-oil salesmen dressed in the garb of scientists, particularly those operating in economics. The greatest fools of randomness will be found among these.

  We are flawed beyond repair, at least for this environment—but it is only bad news for those utopians who believe in an idealized humankind. Current thinking presents the two following polarized visions of man, with little shades in between. On the one hand there is your local college English professor; your great-aunt Irma, who never married and liberally delivers sermons; your how-to-reach-happiness-in-twenty-steps and how-to-become-a-better-person-in-a-week book writer. It is called the Utopian Vision, associated with Rousseau, Godwin, Condorcet, Thomas Paine, and conventional normative economists (of the kind to ask you to make rational choices because that is what is deemed good for you), etc. They believe in reason and rationality—that we should overcome cultural impediments on our way to becoming a better human race—thinking we can control our nature at will and transform it by mere edict in order to attain, among other things, happiness and rationality. Basically this category would include those who think that the cure for obesity is to inform people that they should be healthy.

  On the other hand there is the Tragic Vision of humankind that believes in the existence of inherent limitations and flaws in the way we think and act and requires an acknowledgment of this fact as a basis for any individual and collective action. This category of people includes Karl Popper (falsificationism and distrust of intellectual “answers,” actually of anyone who is confident that he knows anything with certainty), Friedrich Hayek and Milton Friedman (suspicion of governments), Adam Smith (intention of man), Herbert Simon (bounded rationality), Amos Tversky and Daniel Kahneman (heuristics and biases), the speculator George Soros, etc. The most neglected one is the misunderstood philosopher Charles Sanders Peirce, who was born a hundred years too early (he coined the term scientific “fallibilism” in opposition to Papal infallibility). Needless to say that the ideas of this book fall squarely into the Tragic category: We are faulty and there is no need to bother trying to correct our flaws. We are so defective and so mismatched to our environment that we can just work around these flaws. I am convinced of that after spending almost all my adult and professional years in a fierce fight between my brain (not Fooled by Randomness) and my emotions (completely Fooled by Randomness) in which the only success I’ve had is in going around my emotions rather than rationalizing them. Perhaps ridding ourselves of our humanity is not in the works; we need wily tricks, not some grandiose moralizing help. As an empiricist (actually a skeptical empiricist) I despise the moralizers beyond anything on this planet: I still wonder why they blindly believe in ineffectual methods. Delivering advice assumes that our cognitive apparatus rather than our emotional machinery exerts some meaningful control over our actions. We will see how modern behavioral science shows this to be completely untrue.

  My colleague Bob Jaeger (he followed the opposite course of mine of moving from philosophy professor to trader) presents a more potent view of the dichotomy: There are those who think that there are easy clear-cut answers and those who don’t think that simplification is possible without severe distortion (his hero: Wittgenstein; his villain: Descartes). I am enamored of the diffderence as I think that the generator of the Fooled by Randomness problem, the false belief in determinism, is also associated with such reduction of the dimensionality of things. As much as you believe in the “keep-it-simple-stupid” it is the simplification that is dangerous.

  This author hates books that can be easily guessed from the table of contents (not many people read textbooks for pleasure)—but a hint of what comes next seems in order. The book is composed of three parts. The first is an introspection into Solon’s warning, as his outburst on rare events became my lifelong motto. In it we meditate on visible and invisible histories and the elusive property of rare events (black swans). The second presents a collection of probability biases I encountered (and suffered from) in my career in randomness—ones that continue to fool me. The third illustrates my personal jousting with my biology and concludes the book with a presentation of a few practical (wax in my ears) and philosophical (stoicism) aids. Before the “enlightenment” and the age of rationality, there was in the culture a collection of tricks to deal with our fallibility and reversals of fortunes. The elders can still help us with some of their ruses.

  Part I

  •

  SOLON’S

  WARNING

  Skewness, Asymmetry, Induction

  Croesus, King of Lydia, was considered the richest man o
f his time. To this day Romance languages use the expression “rich as Croesus” to describe a person of excessive wealth. He was said to be visited by Solon, the Greek legislator known for his dignity, reserve, upright morals, humility, frugality, wisdom, intelligence, and courage. Solon did not display the smallest surprise at the wealth and splendor surrounding his host, nor the tiniest admiration for their owner. Croesus was so irked by the manifest lack of impression on the part of this illustrious visitor that he attempted to extract from him some acknowledgment. He asked him if he had known a happier man than him. Solon cited the life of a man who led a noble existence and died while in battle. Prodded for more, he gave similar examples of heroic but terminated lives, until Croesus, irate, asked him point-blank if he was not to be considered the happiest man of all. Solon answered: “The observation of the numerous misfortunes that attend all conditions forbids us to grow insolent upon our present enjoyments, or to admire a man’s happiness that may yet, in course of time, suffer change. For the uncertain future has yet to come, with all variety of future; and him only to whom the divinity has [guaranteed] continued happiness until the end we may call happy.”

  The modern equivalent has been no less eloquently voiced by the baseball coach Yogi Berra, who seems to have translated Solon’s outburst from the pure Attic Greek into no less pure Brooklyn English with “it ain’t over until it’s over,” or, in a less dignified manner, with “it ain’t over until the fat lady sings.” In addition, aside from his use of the vernacular, the Yogi Berra quote presents an advantage of being true, while the meeting between Croesus and Solon was one of those historical facts that benefited from the imagination of the chroniclers, as it was chronologically impossible for the two men to have been in the same location.