The Theory That Would Not Die Read online

  the theory that would not die

  other books by

  sharon bertsch mcgrayne

  Prometheans in the Lab: Chemistry and the Making

  of the Modern World

  Iron, Nature’s Universal Element: Why People Need

  Iron and Animals Make Magnets (with Eugenie V.

  Mielczarek)

  Nobel Prize Women in Science: Their Lives, Struggles,

  and Momentous Discoveries

  the theory that would not die

  how bayes’ rule cracked

  the enigma code,

  hunted down russian

  submarines, & emerged

  triumphant from two

  centuries of controversy

  sharon bertsch mcgrayne

  “The Doctor sees the light” by Michael Campbell is reproduced by permission of John Wiley & Sons; the lyrics by George E. P. Box in chapter 10 are reproduced by permission of John Wiley & Sons; the conversation between Sir Harold Jeffreys and D. V. Lindley in chapter 3 is reproduced by permission of the Master and Fellows of St. John’s College, Cambridge.

  Copyright © 2011 by Sharon Bertsch McGrayne. All rights reserved. This book may not be reproduced, in whole or in part, including illustrations, in any form (beyond that copying permitted by Sections 107 and 108 of the U.S. Copyright Law and except by reviewers for the public press), without written permission from the publishers.

  Yale University Press books may be purchased in quantity for educational, business, or promotional use. For information, please e-mail [email protected] (U.S. office) or [email protected] (U.K. office).

  Designed by Lindsey Voskowsky. Set in Monotype Joanna type by Duke & Company, Devon, Pennsylvania. Printed in the United States of America.

  Library of Congress Cataloging-in-Publication Data

  McGrayne, Sharon Bertsch.

  The theory that would not die : how Bayes’ rule cracked the enigma code, hunted down Russian submarines, and emerged triumphant from two centuries of controversy / Sharon Bertsch McGrayne.

  p. cm.

  Summary: “Bayes’ rule appears to be a straightforward, one-line theorem: by updating our initial beliefs with objective new information, we get a new and improved belief. To its adherents, it is an elegant statement about learning from experience. To its opponents, it is subjectivity run amok. In the first-ever account of Bayes’ rule for general readers, Sharon Bertsch McGrayne explores this controversial theorem and the human obsessions surrounding it. She traces its discovery by an amateur mathematician in the 1740s through its development into roughly its modern form by French scientist Pierre Simon Laplace. She reveals why respected statisticians rendered it professionally taboo for 150 years—at the same time that practitioners relied on it to solve crises involving great uncertainty and scanty information, even breaking Germany’s Enigma code during World War II, and explains how the advent of off-the-shelf computer technology in the 1980s proved to be a game-changer. Today, Bayes’ rule is used everywhere from DNA de-coding to Homeland Security. Drawing on primary source material and interviews with statisticians and other scientists, The Theory That Would Not Die is the riveting account of how a seemingly simple theorem ignited one of the greatest controversies of all time”—Provided by publisher.

  Includes bibliographical references and index.

  ISBN 978-0-300-16969-0 (hardback)

  1. Bayesian statistical decision theory—History. I. Title.

  QA279.5.M415 2011

  519.5’42—dc22

  2010045037

  A catalogue record for this book is available from the British Library.

  This paper meets the requirements of ANSI/NISO Z39.48–1992 (Permanence of Paper).

  10 9 8 7 6 5 4 3 2 1

  When the facts change, I change my opinion. What do you do, sir?

  —John Maynard Keynes

  contents

  Preface and Note to Readers

  Acknowledgments

  Part I. Enlightenment and the Anti-Bayesian Reaction

  1. Causes in the Air

  2. The Man Who Did Everything

  3. Many Doubts, Few Defenders

  Part II. Second World War Era

  4. Bayes Goes to War

  5. Dead and Buried Again

  Part III. The Glorious Revival

  6. Arthur Bailey

  7. From Tool to Theology

  8. Jerome Cornfield, Lung Cancer, and Heart Attacks

  9. There’s Always a First Time

  10. 46,656 Varieties

  Part IV. To Prove Its Worth

  11. Business Decisions

  12. Who Wrote The Federalist?

  13. The Cold Warrior

  14. Three Mile Island

  15. The Navy Searches

  Part V. Victory

  16. Eureka!

  17. Rosetta Stones

  Appendixes

  Dr. Fisher’s Casebook

  Applying Bayes’ Rule to Mammograms and Breast Cancer

  Notes

  Glossary

  Bibliography

  Index

  preface and note to readers

  In a celebrated example of science gone awry, geologists accumulated the evidence for Continental Drift in 1912 and then spent 50 years arguing that continents cannot move.

  The scientific battle over Bayes’ rule is less well known but lasted far longer, for 150 years. It concerned a broader and more fundamental issue: how we analyze evidence, change our minds as we get new information, and make rational decisions in the face of uncertainty. And it was not resolved until the dawn of the twenty-first century.

  On its face Bayes’ rule is a simple, one-line theorem: by updating our initial belief about something with objective new information, we get a new and improved belief. To its adherents, this is an elegant statement about learning from experience. Generations of converts remember experiencing an almost religious epiphany as they fell under the spell of its inner logic. Opponents, meanwhile, regarded Bayes’ rule as subjectivity run amok.

  Bayes’ rule began life amid an inflammatory religious controversy in England in the 1740s: can we make rational conclusions about God based on evidence about the world around us? An amateur mathematician, the Reverend Thomas Bayes, discovered the rule, and we celebrate him today as the iconic father of mathematical decision making. Yet Bayes consigned his discovery to oblivion. In his time, he was a minor figure. And we know about his work today only because of his friend and editor Richard Price, an almost forgotten hero of the American Revolution.

  By rights, Bayes’ rule should be named for someone else: a Frenchman, Pierre Simon Laplace, one of the most powerful mathematicians and scientists in history. To deal with an unprecedented torrent of data, Laplace discovered the rule on his own in 1774. Over the next forty years he developed it into the form we use today. Applying his method, he concluded that a well-established fact—more boys are born than girls—was almost certainly the result of natural law. Only historical convention forces us to call Laplace’s discovery Bayes’ rule.

  After Laplace’s death, researchers and academics seeking precise and objective answers pronounced his method subjective, dead, and buried. Yet at the very same time practical problem solvers relied on it to deal with real-world emergencies. One spectacular success occurred during the Second World War, when Alan Turing developed Bayes to break Enigma, the German navy’s secret code, and in the process helped to both save Britain and invent modern electronic computers and software. Other leading mathematical thinkers—Andrei Kolmogorov in Russia and Claude Shannon in New York—also rethought Bayes for wartime decision making.

  During the years when ivory tower theorists thought they
had rendered Bayes taboo, it helped start workers’ compensation insurance in the United States; save the Bell Telephone system from the financial panic of 1907; deliver Alfred Dreyfus from a French prison; direct Allied artillery fire and locate German U-boats; and locate earthquake epicenters and deduce (erroneously) that Earth’s core consists of molten iron.

  Theoretically, Bayes’ rule was verboten. But it could deal with all kinds of data, whether copious or sparse. During the Cold War, Bayes helped find a missing H-bomb and U.S. and Russian submarines; investigate nuclear power plant safety; predict the shuttle Challenger tragedy; demonstrate that smoking causes lung cancer and that high cholesterol causes heart attacks; predict presidential winners on television’s most popular news program, and much more.

  How could otherwise rational scientists, mathematicians, and statisticians become so obsessed about a theorem that their argument became, as one observer called it, a massive food fight? The answer is simple. At its heart, Bayes runs counter to the deeply held conviction that modern science requires objectivity and precision. Bayes is a measure of belief. And it says that we can learn even from missing and inadequate data, from approximations, and from ignorance.

  As a result of this profound philosophical disagreement, Bayes’ rule is a flesh-and-blood story about a small group of beleaguered believers who struggled for legitimacy and acceptance for most of the twentieth century. It’s about how the rule’s fate got entwined with the secrecy of the Second World War and the Cold War. It’s about a theorem in want of a computer and a software package. And it’s about a method that—refreshed by outsiders from physics, computer science, and artificial intelligence—was adopted almost overnight because suddenly it worked. In a new kind of paradigm shift for a pragmatic world, the man who had called Bayes “the crack cocaine of statistics. . . . seductive, addictive and ultimately destructive” began recruiting Bayesians for Google.

  Today, Bayesian spam filters whisk pornographic and fraudulent e-mail to our computers’ junk bins. When a ship sinks, the Coast Guard calls on Bayes and locates shipwrecked survivors who may have floated at sea for weeks. Scientists discover how genes are controlled and regulated. Bayes even wins Nobel Prizes. Online, Bayes’ rule trawls the web and sells songs and films. It has penetrated computer science, artificial intelligence, machine learning, Wall Street, astronomy and physics, Homeland Security, Microsoft, and Google. It helps computers translate one language into another, tearing down the world’s millennia-old Tower of Babel. It has become a metaphor for how our brains learn and function. Prominent Bayesians even advise government agencies on energy, education, and research.

  But Bayes’ rule is not just an obscure scientific controversy laid to rest. It affects us all. It’s a logic for reasoning about the broad spectrum of life that lies in the gray areas between absolute truth and total uncertainty. We often have information about only a small part of what we wonder about. Yet we all want to predict something based on our past experiences; we change our beliefs as we acquire new information. After suffering years of passionate scorn, Bayes has provided a way of thinking rationally about the world around us.

  This is the story of how that remarkable transformation took place.

  Note: Observant readers may notice that I use the word “probability” a lot in this book. In common speech, most of us treat the words “probability,” “likelihood,” and “odds” interchangeably. In statistics, however, these terms are not synonymous; they have distinct and technical meanings. Because I’ve tried to use correct terminology in The Theory That Would Not Die, “probability” appears quite a bit.

  acknowledgments

  For their scientific advice and perspective and for their patience with someone who asked a multitude of questions, I am deeply indebted to Dennis V. Lindley, Robert E. Kass, and George F. Bertsch. These three also read and made perceptive comments about the entire book in one or more of its many drafts. I could not have written the book at all without my husband, George Bertsch.

  For insightful guidance on various crucial threads of my narrative, I thank James O. Berger, David M. Blei, Bernard Bru, Andrew I. Dale, Arthur P. Dempster, Persi Diaconis, Bradley Efron, Stephen E. Fienberg, Stuart Geman, Roger Hahn, Peter Hoff, Tom J. Loredo, Albert Madansky, John W. Pratt, Henry R. (“Tony”) Richardson, Christian P. Robert, Stephen M. Stigler, and David L. Wallace.

  Many other experts and specialists spoke with me, often at length, about particular eras, problems, details, or people. They include Capt. Frank A. Andrews, Frank Anscombe, George Apostolakis, Robert A. and Shirley Bailey, Friedrich L. Bauer, Robert T. Bell, David R. Bellhouse, Julian Besag, Alan S. Blinder, George E. P. Box, David R. Brillinger, Bruce Budowle, Hans Bühlmann, Frank Carter, Herman Chernoff, Juscelino F. Colares, Jack Copeland, Ann Cornfield, Ellen Cornfield, John Piña Craven, Lorraine Daston, Philip Dawid, Joseph H. Discenza, Ralph Erskine, Michael Fortunato, Karl Friston, Chris Frith, John (“Jack”) Frost, Dennis G. Fryback, Mitchell H. Gail, Alan E. Gelfand, Andrew Gelman, Edward I. George, Edgar N. Gilbert, Paul M. Goggans, I. J. “Jack” Good, Steven N. Goodman, Joel Greenhouse, Ulf Grenander, Gerald N. Grob, Thomas L. Hankins, Jeffrey E. Harris, W. Keith Hastings, David Heckerman, Charles C. Hewitt Jr., Ray Hilborn, David C. Hoaglin, Antje Hoering, Marvin Hoffenberg, Susan P. Holmes, David Hounshell, Ronald H. Howard, David Howie, Bobby R. Hunt, Fred C. Iklé, David R. Jardini, William H. Jefferys, Douglas M. Jesseph.

  Also, Michael I. Jordan, David Kahn, David H. Kaye, John G. King, Kenneth R. Koedinger, Daphne Koller, Tom Kratzke, James M. Landwehr, Bernard Lightman, Richard F. Link, Edward P. Loane, Michael C. Lovell, Thomas L. Marzetta, Scott H. Mathews, John McCullough, Richard F. Meyer, Glenn G. Meyers, Paul J. Miranti Jr., Deputy Commander Dewitt Moody, Rear Admiral Brad Mooney, R. Bradford Murphy, John W. Negele, Vice Admiral John “Nick” Nicholson, Peter Norvig, Stephen M. Pollock, Theodore M. Porter, Alexandre Pouget, S. James Press, Alan Rabinowitz, Adrian E. Raftery, Howard Raiffa, John J. Rehr, John T. Riedl, Douglas Rivers, Oleg Sapozhnikov, Peter Schlaifer, Arthur Schleifer Jr., Michael N. Shadlen, Edward H. (“Ted”) Shortliffe, Edward H. Simpson, Harold C. Sox, David J. Spiegelhalter, Robert F. Stambaugh, Lawrence D. Stone, William J. Talbott, Judith Tanur, The Center for Defense Information, Sebastian Thrun, Oakley E. (Lee) Van Slyke, Gary G. Venter, Christopher Volinsky, Paul R. Wade, Jon Wakefield, Homer Warner, Frode Weierud, Robert B. Wilson, Wing H. Wong, Judith E. Zeh, and Arnold Zellner.

  I would like to thank two outside reviewers, Jim Berger and Andrew Dale; both read the manuscript carefully and made useful comments to improve it.

  Several friends and family members—Ruth Ann Bertsch, Cindy Vahey Bertsch, Fred Bertsch, Jean Colley, Genevra Gerhart, James Goodman, Carolyn Keating, Timothy W. Keller, Sharon C. Rutberg, Beverly Schaefer, and Audrey Jensen Weitkamp—made crucial comments. I owe thanks to the mathematics library staff of the University of Washington. And my agent, Susan Rabiner, and editor, William Frucht, were steadfast in their support.

  Despite all this help, I am, of course, responsible for the errors in this book.

  part I

  enlightenment and the anti-bayesian reaction

  1.

  causes in the air

  Sometime during the 1740s, the Reverend Thomas Bayes made the ingenious discovery that bears his name but then mysteriously abandoned it. It was rediscovered independently by a different and far more renowned man, Pierre Simon Laplace, who gave it its modern mathematical form and scientific application—and then moved on to other methods. Although Bayes’ rule drew the attention of the greatest statisticians of the twentieth century, some of them vilified both the method and its adherents, crushed it, and declared it dead. Yet at the same time, it solved practical questions that were unanswerable by any other means: the defenders of Captain Dreyfus used it to demonstrate his innocence; insurance actuaries used it to set rates; Alan Turing used it to decode the German Enigma cipher and arguably save the Allies from losing the Second World War; the U.S. Navy used it to searc
h for a missing H-bomb and to locate Soviet subs; RAND Corporation used it to assess the likelihood of a nuclear accident; and Harvard and Chicago researchers used it to verify the authorship of the Federalist Papers. In discovering its value for science, many supporters underwent a near-religious conversion yet had to conceal their use of Bayes’ rule and pretend they employed something else. It was not until the twenty-first century that the method lost its stigma and was widely and enthusiastically embraced. The story began with a simple thought experiment.

  Because Bayes’ gravestone says he died in 1761 at the age of 59, we know he lived during England’s struggle to recover from nearly two centuries of religious strife, civil war, and regicide. As a member of the Presbyterian Church, a religious denomination persecuted for refusing to support the Church of England, he was considered a Dissenter or Non-Conformist. During his grandfather’s generation, 2,000 Dissenters died in English prisons. By Bayes’ time, mathematics was split along religious and political lines, and many productive mathematicians were amateurs because, as Dissenters, they were barred from English universities.1

  Unable to earn a degree in England, Bayes studied theology and presumably mathematics at the University of Edinburgh in Presbyterian Scotland, where, happily for him, academic standards were much more rigorous. In 1711 he left for London, where his clergyman father ordained him and apparently employed him as an assistant minister.

  Persecution turned many English Dissenters into feisty critics, and in his late 20s Bayes took a stand on a hot theological issue: can the presence of evil be reconciled with God’s presumed beneficence? In 1731 he wrote a pamphlet—a kind of blog—declaring that God gives people “the greatest happiness of which they are capable.”

  During his 40s, Bayes’ interests in mathematics and theology began to tightly intertwine. An Irish-Anglican bishop—George Berkeley, for whom the University of California’s flagship campus is named—published an inflammatory pamphlet attacking Dissenting mathematicians, calculus, abstract mathematics, the revered Isaac Newton, and all other “free-thinkers” and “infidel mathematicians” who believed that reason could illuminate any subject. Berkeley’s pamphlet was the most spectacular event in British mathematics during the 1700s.