Skip to main content
Top
Published in: BMC Medicine 1/2011

Open Access 01-12-2011 | Debate

Significance testing as perverse probabilistic reasoning

Authors: M Brandon Westover, Kenneth D Westover, Matt T Bianchi

Published in: BMC Medicine | Issue 1/2011

Login to get access

Abstract

Truth claims in the medical literature rely heavily on statistical significance testing. Unfortunately, most physicians misunderstand the underlying probabilistic logic of significance tests and consequently often misinterpret their results. This near-universal misunderstanding is highlighted by means of a simple quiz which we administered to 246 physicians at two major academic hospitals, on which the proportion of incorrect responses exceeded 90%. A solid understanding of the fundamental concepts of probability theory is becoming essential to the rational interpretation of medical information. This essay provides a technically sound review of these concepts that is accessible to a medical audience. We also briefly review the debate in the cognitive sciences regarding physicians' aptitude for probabilistic inference.
Appendix
Available only for authorised users
Literature
1.
go back to reference Olser W, Silverman M, Murray T, Bryan C: The Quotable Osler. 2003, Philadelphia: ACP Press Olser W, Silverman M, Murray T, Bryan C: The Quotable Osler. 2003, Philadelphia: ACP Press
2.
go back to reference Horton NJ, Switzer SS: Statistical methods in the journal. The New England Journal of Medicine. 2005, 353 (18): 1977-1979. 10.1056/NEJM200511033531823.PubMed Horton NJ, Switzer SS: Statistical methods in the journal. The New England Journal of Medicine. 2005, 353 (18): 1977-1979. 10.1056/NEJM200511033531823.PubMed
3.
go back to reference Altman DG, Bland JM: Improving doctors' understanding of statistics. Journal of the Royal Statistical Society. Series A (Statistics in Society). 1991, 154 (2): 223-267. 10.2307/2983040. Altman DG, Bland JM: Improving doctors' understanding of statistics. Journal of the Royal Statistical Society. Series A (Statistics in Society). 1991, 154 (2): 223-267. 10.2307/2983040.
4.
go back to reference Windish DM, Huot SJ, Green ML: Medicine residents' understanding of the biostatistics and results in the medical literature. The Journal of the American Medical Association. 2007, 298 (9): 1010-1022. 10.1001/jama.298.9.1010.PubMed Windish DM, Huot SJ, Green ML: Medicine residents' understanding of the biostatistics and results in the medical literature. The Journal of the American Medical Association. 2007, 298 (9): 1010-1022. 10.1001/jama.298.9.1010.PubMed
5.
go back to reference Ioannidis JPA: Why most published research findings are false. PLoS Medicine. 2005, 2 (8): e124-10.1371/journal.pmed.0020124.PubMedPubMedCentral Ioannidis JPA: Why most published research findings are false. PLoS Medicine. 2005, 2 (8): e124-10.1371/journal.pmed.0020124.PubMedPubMedCentral
6.
go back to reference Friedman SB, Phillips S: What's the difference? Pediatric residents and their inaccurate concepts regarding statistics. Pediatrics. 1981, 68 (5): 644-646.PubMed Friedman SB, Phillips S: What's the difference? Pediatric residents and their inaccurate concepts regarding statistics. Pediatrics. 1981, 68 (5): 644-646.PubMed
7.
go back to reference Goodman SN: Toward evidence-based medical statistics. 1: the P value fallacy. Annals of Internal Medicine. 1999, 130 (12): 995-1004.PubMed Goodman SN: Toward evidence-based medical statistics. 1: the P value fallacy. Annals of Internal Medicine. 1999, 130 (12): 995-1004.PubMed
9.
go back to reference Casscells W, Schoenberger A, Graboys TB: Interpretation by physicians of clinical laboratory results. The New England Journal of Medicine. 1978, 299 (18): 999-1001. 10.1056/NEJM197811022991808.PubMed Casscells W, Schoenberger A, Graboys TB: Interpretation by physicians of clinical laboratory results. The New England Journal of Medicine. 1978, 299 (18): 999-1001. 10.1056/NEJM197811022991808.PubMed
11.
go back to reference Eddy DM: Probabilistic Reasoning in Clinical Medicine: Problems and Opportunities. 1982, Cambridge, UK: Cambridge University Press, 249-267. Eddy DM: Probabilistic Reasoning in Clinical Medicine: Problems and Opportunities. 1982, Cambridge, UK: Cambridge University Press, 249-267.
13.
go back to reference Falk R, Greenbaum CW: Significance tests die hard: the amazing persistence of a probabilistic misconception. Theory Psychology. 1995, 5: 75-98. 10.1177/0959354395051004. Falk R, Greenbaum CW: Significance tests die hard: the amazing persistence of a probabilistic misconception. Theory Psychology. 1995, 5: 75-98. 10.1177/0959354395051004.
15.
go back to reference Gill J: The insignificance of null hypothesis significance testing. Political Research Quarterly. 1999, 52 (3): 647-674. Gill J: The insignificance of null hypothesis significance testing. Political Research Quarterly. 1999, 52 (3): 647-674.
16.
go back to reference Gigerenzer G, Murray DJ: Cognition as Intuitive Statistics. 1987, Hillsdale, NJ: L. Erlbaum Associates Gigerenzer G, Murray DJ: Cognition as Intuitive Statistics. 1987, Hillsdale, NJ: L. Erlbaum Associates
17.
go back to reference Gigerenzer G: The superego, the ego, and the id in statistical reasoning. A Handbook for Data Analysis in the Behavioral Sciences: Methodological Issues. Edited by: Keren G, Lewis C. 1993, Hillsdale, NJ: L. Erlbaum Associates, 574. Gigerenzer G: The superego, the ego, and the id in statistical reasoning. A Handbook for Data Analysis in the Behavioral Sciences: Methodological Issues. Edited by: Keren G, Lewis C. 1993, Hillsdale, NJ: L. Erlbaum Associates, 574.
18.
go back to reference Campbell L, Garnett W: The Life of James Clerk Maxwell. With a Selection from His Correspondence and Occasional Writings and a Sketch of His Contributions to Science. 1882, London: Macmillan and Co Campbell L, Garnett W: The Life of James Clerk Maxwell. With a Selection from His Correspondence and Occasional Writings and a Sketch of His Contributions to Science. 1882, London: Macmillan and Co
19.
go back to reference Mumford D: The dawning of the age of stochasticity. Mathematics: Frontiers and Perspectives. 2000, 197-218. Mumford D: The dawning of the age of stochasticity. Mathematics: Frontiers and Perspectives. 2000, 197-218.
20.
go back to reference Oaksford M, Chater N: Bayesian Rationality: The Probabilistic Approach to Human Reasoning. 2007, Oxford: Oxford University Press, [Oxford Cognitive Science Series] Oaksford M, Chater N: Bayesian Rationality: The Probabilistic Approach to Human Reasoning. 2007, Oxford: Oxford University Press, [Oxford Cognitive Science Series]
21.
go back to reference Jaynes ET, Bretthorst GL: Probability Theory: The Logic of Science. 2003, Cambridge, UK: Cambridge University Press Jaynes ET, Bretthorst GL: Probability Theory: The Logic of Science. 2003, Cambridge, UK: Cambridge University Press
22.
go back to reference Cox RT: The Algebra of Probable Inference. 1961, Baltimore: Johns Hopkins Press Cox RT: The Algebra of Probable Inference. 1961, Baltimore: Johns Hopkins Press
23.
go back to reference Jaynes ET: How does the brain do plausible reasoning?. Maximum-Entropy and Bayesian Methods in Science and Engineering. Edited by: Erickson GJ, Smith CR. 1988, Kluwer Academic Publishers Jaynes ET: How does the brain do plausible reasoning?. Maximum-Entropy and Bayesian Methods in Science and Engineering. Edited by: Erickson GJ, Smith CR. 1988, Kluwer Academic Publishers
24.
go back to reference Horn KSV: Constructing a logic of plausible inference: a guide to Cox's theorem. International Journal of Approximate Reasoning. 2003, 34: 3-24. 10.1016/S0888-613X(03)00051-3. Horn KSV: Constructing a logic of plausible inference: a guide to Cox's theorem. International Journal of Approximate Reasoning. 2003, 34: 3-24. 10.1016/S0888-613X(03)00051-3.
25.
go back to reference Kruschke JK: Doing Bayesian Data Analysis: A Tutorial with R and BUGS. 2010, Academic Press Kruschke JK: Doing Bayesian Data Analysis: A Tutorial with R and BUGS. 2010, Academic Press
26.
go back to reference Kruschke JK: Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science. 2010, 1 (5): 658-676. 10.1002/wcs.72.PubMed Kruschke JK: Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science. 2010, 1 (5): 658-676. 10.1002/wcs.72.PubMed
27.
go back to reference Fisher RA: Statistical Methods and Scientific Inference. 1973, New York: Hafner Press, 3, rev. and enl Fisher RA: Statistical Methods and Scientific Inference. 1973, New York: Hafner Press, 3, rev. and enl
28.
go back to reference Kruschke JK: What to believe: Bayesian methods for data analysis. Trends in Cognitive Sciences. 2010, 14 (7): 293-300. 10.1016/j.tics.2010.05.001.PubMed Kruschke JK: What to believe: Bayesian methods for data analysis. Trends in Cognitive Sciences. 2010, 14 (7): 293-300. 10.1016/j.tics.2010.05.001.PubMed
29.
go back to reference Gelman A: Bayesian Data Analysis. 2004, Boca Raton, Fla: Chapman & Hall/CRC, [Texts in Statistical Science], 2 Gelman A: Bayesian Data Analysis. 2004, Boca Raton, Fla: Chapman & Hall/CRC, [Texts in Statistical Science], 2
30.
go back to reference Diamond GA, Kaul S: Prior convictions: Bayesian approaches to the analysis and interpretation of clinical megatrials. Journal of the American College of Cardiology. 2004, 43 (11): 1929-1939. 10.1016/j.jacc.2004.01.035.PubMed Diamond GA, Kaul S: Prior convictions: Bayesian approaches to the analysis and interpretation of clinical megatrials. Journal of the American College of Cardiology. 2004, 43 (11): 1929-1939. 10.1016/j.jacc.2004.01.035.PubMed
31.
go back to reference Berry DA: Bayesian clinical trials. Nature Reviews. Drug Discovery. 2006, 5: 27-36. 10.1038/nrd1927.PubMed Berry DA: Bayesian clinical trials. Nature Reviews. Drug Discovery. 2006, 5: 27-36. 10.1038/nrd1927.PubMed
32.
go back to reference Goodman SN: Introduction to Bayesian methods I: measuring the strength of evidence. Clinical Trials (London, England). 2005, 2 (4): 282-290. discussion 301-304, 364-378 Goodman SN: Introduction to Bayesian methods I: measuring the strength of evidence. Clinical Trials (London, England). 2005, 2 (4): 282-290. discussion 301-304, 364-378
33.
go back to reference Berry DA: Introduction to Bayesian methods III: use and interpretation of Bayesian tools in design and analysis. Clinical Trials (London, England). 2005, 2 (4): 295-300. discussion 301-304, 364-378 Berry DA: Introduction to Bayesian methods III: use and interpretation of Bayesian tools in design and analysis. Clinical Trials (London, England). 2005, 2 (4): 295-300. discussion 301-304, 364-378
34.
go back to reference Louis TA: Introduction to Bayesian methods II: fundamental concepts. Clinical Trials (London, England). 2005, 2 (4): 291-294. discussion 301-304, 364-378 Louis TA: Introduction to Bayesian methods II: fundamental concepts. Clinical Trials (London, England). 2005, 2 (4): 291-294. discussion 301-304, 364-378
35.
go back to reference Spiegelhalter DJ, Myles JP, Jones DR, Abrams KR: Bayesian methods in health technology assessment: a review. Health Technology Assessment (Winchester, England). 2000, 4 (38): 1-130. Spiegelhalter DJ, Myles JP, Jones DR, Abrams KR: Bayesian methods in health technology assessment: a review. Health Technology Assessment (Winchester, England). 2000, 4 (38): 1-130.
36.
go back to reference Spiegelhalter DJ, Myles JP, Jones DR, Abrams KR: Methods in health service research. An introduction to Bayesian methods in health technology assessment. British Medical Journal (Clinical Research Ed.). 1999, 319 (7208): 508-512. Spiegelhalter DJ, Myles JP, Jones DR, Abrams KR: Methods in health service research. An introduction to Bayesian methods in health technology assessment. British Medical Journal (Clinical Research Ed.). 1999, 319 (7208): 508-512.
37.
go back to reference Jacobs RA, Kruschke JK: Bayesian learning theory applied to human cognition. Wiley Interdisciplinary Reviews: Cognitive Science. 2010, 2: 8-21. 10.1002/wcs.80.PubMed Jacobs RA, Kruschke JK: Bayesian learning theory applied to human cognition. Wiley Interdisciplinary Reviews: Cognitive Science. 2010, 2: 8-21. 10.1002/wcs.80.PubMed
38.
go back to reference Chater N, Manning CD: Probabilistic models of language processing and acquisition. Trends in Cognitive Sciences. 2006, 10 (7): 335-344. 10.1016/j.tics.2006.05.006.PubMed Chater N, Manning CD: Probabilistic models of language processing and acquisition. Trends in Cognitive Sciences. 2006, 10 (7): 335-344. 10.1016/j.tics.2006.05.006.PubMed
39.
go back to reference Chater N, Oaksford M: The Probabilistic Mind: Prospects for Bayesian Cognitive Science. 2008, Oxford: Oxford University Press Chater N, Oaksford M: The Probabilistic Mind: Prospects for Bayesian Cognitive Science. 2008, Oxford: Oxford University Press
40.
go back to reference Tenenbaum JB, Griffiths TL, Kemp C: Theory-based Bayesian models of inductive learning and reasoning. Trends in Cognitive Sciences. 2006, 10 (7): 309-318. 10.1016/j.tics.2006.05.009.PubMed Tenenbaum JB, Griffiths TL, Kemp C: Theory-based Bayesian models of inductive learning and reasoning. Trends in Cognitive Sciences. 2006, 10 (7): 309-318. 10.1016/j.tics.2006.05.009.PubMed
41.
go back to reference Xu F, Tenenbaum JB: Word learning as Bayesian inference. Psychological Review (New York). 2007, 114 (2): 245. Xu F, Tenenbaum JB: Word learning as Bayesian inference. Psychological Review (New York). 2007, 114 (2): 245.
42.
go back to reference Steyvers M, Griffiths TL, Dennis S: Probabilistic inference in human semantic memory. Trends in Cognitive Sciences. 2006, 10 (7): 327-334. 10.1016/j.tics.2006.05.005.PubMed Steyvers M, Griffiths TL, Dennis S: Probabilistic inference in human semantic memory. Trends in Cognitive Sciences. 2006, 10 (7): 327-334. 10.1016/j.tics.2006.05.005.PubMed
43.
go back to reference Manning CD, Schütze H: Foundations of Statistical Natural Language Processing. 1999, MIT Press Manning CD, Schütze H: Foundations of Statistical Natural Language Processing. 1999, MIT Press
44.
go back to reference Westover M, O'Sullivan J: Achievable rates for pattern recognition. Information Theory, IEEE Transactions on. 2008, 54: 299-320. 10.1109/TIT.2007.911296. Westover M, O'Sullivan J: Achievable rates for pattern recognition. Information Theory, IEEE Transactions on. 2008, 54: 299-320. 10.1109/TIT.2007.911296.
45.
go back to reference Yuille A, Kersten D: Vision as Bayesian inference: analysis by synthesis?. Trends in Cognitive Sciences. 2006, 10 (7): 301-308. 10.1016/j.tics.2006.05.002.PubMed Yuille A, Kersten D: Vision as Bayesian inference: analysis by synthesis?. Trends in Cognitive Sciences. 2006, 10 (7): 301-308. 10.1016/j.tics.2006.05.002.PubMed
46.
go back to reference Grenander U, Miller M: Pattern Theory: From Representation to Inference. 2007, USA: Oxford University Press Grenander U, Miller M: Pattern Theory: From Representation to Inference. 2007, USA: Oxford University Press
47.
go back to reference Jordan MI, (Ed): Learning in Graphical Models. 1998, The MIT Press, 1 Jordan MI, (Ed): Learning in Graphical Models. 1998, The MIT Press, 1
48.
go back to reference Bishop CM: Pattern Recognition and Machine Learning. 2007, Springer, 1 Bishop CM: Pattern Recognition and Machine Learning. 2007, Springer, 1
49.
go back to reference Mumford D, Desolneux A: Pattern Theory: The Stochastic Analysis of Real-World Signals (Applying Mathematics). 2010, Natick, Mass.: A K Peters Mumford D, Desolneux A: Pattern Theory: The Stochastic Analysis of Real-World Signals (Applying Mathematics). 2010, Natick, Mass.: A K Peters
50.
go back to reference MacKay DJC: Information Theory, Inference & Learning Algorithms. 2002, Cambridge University Press, 1 MacKay DJC: Information Theory, Inference & Learning Algorithms. 2002, Cambridge University Press, 1
51.
go back to reference Frey BJ: Graphical Models for Machine Learning and Digital Communication. 1998, The MIT Press Frey BJ: Graphical Models for Machine Learning and Digital Communication. 1998, The MIT Press
52.
go back to reference Jelinek F: Statistical Methods for Speech Recognition. 1998, The MIT Press Jelinek F: Statistical Methods for Speech Recognition. 1998, The MIT Press
53.
go back to reference Pearl J: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. 1998, San Francisco, Calif: Morgan Kaufmann, [The Morgan Kaufmann Series in Representation and Reasoning], Rev. 2nd printing Pearl J: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. 1998, San Francisco, Calif: Morgan Kaufmann, [The Morgan Kaufmann Series in Representation and Reasoning], Rev. 2nd printing
54.
go back to reference Paté-Cornell E, Guikema S: Probabilistic modeling of terrorist threats: a systems analysis approach to setting priorities among countermeasures. Military Operations Research. 2002, 7 (4): 5-20. Paté-Cornell E, Guikema S: Probabilistic modeling of terrorist threats: a systems analysis approach to setting priorities among countermeasures. Military Operations Research. 2002, 7 (4): 5-20.
55.
go back to reference Forrester M, Pettitt A, Gibson G: Bayesian inference of hospital-acquired infectious diseases and control measures given imperfect surveillance data. Biostatistics. 2007, 8 (2): 383-10.1093/biostatistics/kxl017.PubMed Forrester M, Pettitt A, Gibson G: Bayesian inference of hospital-acquired infectious diseases and control measures given imperfect surveillance data. Biostatistics. 2007, 8 (2): 383-10.1093/biostatistics/kxl017.PubMed
56.
go back to reference de Campos L, Fernández-Luna J, Huete J: Bayesian networks and information retrieval: an introduction to the special issue. Information Processing & Management. 2004, 40 (5): 727-733. de Campos L, Fernández-Luna J, Huete J: Bayesian networks and information retrieval: an introduction to the special issue. Information Processing & Management. 2004, 40 (5): 727-733.
57.
go back to reference Kolmogorov AN: Foundations of the Theory of Probability. 1956, New York: Chelsea Pub. Co, 2d english Kolmogorov AN: Foundations of the Theory of Probability. 1956, New York: Chelsea Pub. Co, 2d english
58.
go back to reference Bayes T: An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society. 1763, 53: 370-418. 10.1098/rstl.1763.0053. Bayes T: An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society. 1763, 53: 370-418. 10.1098/rstl.1763.0053.
59.
go back to reference Gigerenzer G, Swijtink Z, Porter T, Daston L, Beatty J, Kruger L: The Empire of Chance: How Probability Changed Science and Everyday Life (Ideas in Context). 1989, Cambridge [Cambridgeshire]: Cambridge University Press Gigerenzer G, Swijtink Z, Porter T, Daston L, Beatty J, Kruger L: The Empire of Chance: How Probability Changed Science and Everyday Life (Ideas in Context). 1989, Cambridge [Cambridgeshire]: Cambridge University Press
60.
go back to reference Krüger L: The Probabilistic Revolution. 1987, Cambridge, Mass: MIT Press Krüger L: The Probabilistic Revolution. 1987, Cambridge, Mass: MIT Press
63.
go back to reference Browner WS, Newman TB: Are all significant P values created equal? The analogy between diagnostic tests and clinical research. The Journal of the American Medical Association. 1987, 257 (18): 2459-2463. 10.1001/jama.257.18.2459.PubMed Browner WS, Newman TB: Are all significant P values created equal? The analogy between diagnostic tests and clinical research. The Journal of the American Medical Association. 1987, 257 (18): 2459-2463. 10.1001/jama.257.18.2459.PubMed
66.
go back to reference Dixon P: The p-value fallacy and how to avoid it. Canadian Journal of Experimental Psychology = Revue Canadienne De Psychologie Exp'erimentale. 2003, 57 (3): 189-202. Dixon P: The p-value fallacy and how to avoid it. Canadian Journal of Experimental Psychology = Revue Canadienne De Psychologie Exp'erimentale. 2003, 57 (3): 189-202.
67.
go back to reference Frick RW: The appropriate use of null hypothesis testing. Psychological Methods. 1996, 1 (4): 379-390. 10.1037/1082-989X.1.4.379. Frick RW: The appropriate use of null hypothesis testing. Psychological Methods. 1996, 1 (4): 379-390. 10.1037/1082-989X.1.4.379.
68.
go back to reference Hagen RL: In praise of the null hypothesis statistical test. American Psychologist. 1997, 52: 15-24. 10.1037/0003-066X.52.1.15. Hagen RL: In praise of the null hypothesis statistical test. American Psychologist. 1997, 52: 15-24. 10.1037/0003-066X.52.1.15.
69.
go back to reference Killeen PR: An alternative to null-hypothesis significance tests. Psychological Science: A Journal of the American Psychological Society/APS. 2005, 16 (5): 345-353. [PMID: 15869691] Killeen PR: An alternative to null-hypothesis significance tests. Psychological Science: A Journal of the American Psychological Society/APS. 2005, 16 (5): 345-353. [PMID: 15869691]
70.
go back to reference Killeen ACO, jan Wagenmakers E, Grünwald P: A Bayesian perspective on hypothesis testing. Psychological Science. 2006, 17: 10.1111/j.1467-9280.2006.01758.x. Killeen ACO, jan Wagenmakers E, Grünwald P: A Bayesian perspective on hypothesis testing. Psychological Science. 2006, 17: 10.1111/j.1467-9280.2006.01758.x.
71.
go back to reference Loftus GR: Psychology will be a much better science when we change the way we analyze data. Current Directions in Psychological Science. 1996, 5 (6): 161-171. 10.1111/1467-8721.ep11512376. [ArticleType: research-article/Full publication date: Dec., 1996/Copyright © 1996 Association for Psychological Science] Loftus GR: Psychology will be a much better science when we change the way we analyze data. Current Directions in Psychological Science. 1996, 5 (6): 161-171. 10.1111/1467-8721.ep11512376. [ArticleType: research-article/Full publication date: Dec., 1996/Copyright © 1996 Association for Psychological Science]
73.
go back to reference Nickerson RS: Null hypothesis significance testing: a review of an old and continuing controversy. Psychological Methods. 2000, 5 (2): 241-301. 10.1037/1082-989X.5.2.241. [PMID: 10937333]PubMed Nickerson RS: Null hypothesis significance testing: a review of an old and continuing controversy. Psychological Methods. 2000, 5 (2): 241-301. 10.1037/1082-989X.5.2.241. [PMID: 10937333]PubMed
74.
go back to reference Trafimow D: Hypothesis testing and theory evaluation at the boundaries: surprising insights from Bayes's theorem. Psychological Review. 2003, 110 (3): 526-535. 10.1037/0033-295X.110.3.526. [PMID: 12885113]PubMed Trafimow D: Hypothesis testing and theory evaluation at the boundaries: surprising insights from Bayes's theorem. Psychological Review. 2003, 110 (3): 526-535. 10.1037/0033-295X.110.3.526. [PMID: 12885113]PubMed
75.
go back to reference Wagenmakers EJ: A practical solution to the pervasive problem of p values. Psychonomic Bulletin & Review. 2007, 14 (5): 779-804. Wagenmakers EJ: A practical solution to the pervasive problem of p values. Psychonomic Bulletin & Review. 2007, 14 (5): 779-804.
76.
go back to reference Wainer H: One cheer for null hypothesis significance testing. Psychological Methods. 1999, 4 (2): 212-213. 10.1037/1082-989X.4.2.212. Wainer H: One cheer for null hypothesis significance testing. Psychological Methods. 1999, 4 (2): 212-213. 10.1037/1082-989X.4.2.212.
77.
go back to reference Berger JO, Wolpert RL: The Likelihood Principle. 1988, IMS Berger JO, Wolpert RL: The Likelihood Principle. 1988, IMS
79.
go back to reference Royall R: Statistical Evidence: A Likelihood Paradigm. 1997, Chapman & Hall/CRC Royall R: Statistical Evidence: A Likelihood Paradigm. 1997, Chapman & Hall/CRC
80.
go back to reference Sellke T, Bayarri M, Berger J: Calibration of ρ values for testing precise null hypotheses. The American Statistician. 2001, 55: 62-71. 10.1198/000313001300339950. Sellke T, Bayarri M, Berger J: Calibration of ρ values for testing precise null hypotheses. The American Statistician. 2001, 55: 62-71. 10.1198/000313001300339950.
81.
go back to reference Stuart A, Ord J, Arnold S: Kendall's advanced theory of statistics. Vol. 2a: classical inference and the linear model. 1999 Stuart A, Ord J, Arnold S: Kendall's advanced theory of statistics. Vol. 2a: classical inference and the linear model. 1999
82.
go back to reference Goodman SN: p values, hypothesis tests, and likelihood: implications for epidemiology of a neglected historical debate. American Journal of Epidemiology. 1993, 137 (5): 485-496. discussion 497-501PubMed Goodman SN: p values, hypothesis tests, and likelihood: implications for epidemiology of a neglected historical debate. American Journal of Epidemiology. 1993, 137 (5): 485-496. discussion 497-501PubMed
83.
go back to reference Goodman SN: Toward evidence-based medical statistics. 2: the Bayes factor. Annals of Internal Medicine. 1999, 130 (12): 1005-1013.PubMed Goodman SN: Toward evidence-based medical statistics. 2: the Bayes factor. Annals of Internal Medicine. 1999, 130 (12): 1005-1013.PubMed
84.
go back to reference Wasserman L: All of Statistics: A Concise Course in Statistical Inference. 2004, New York: Springer, [Springer Texts in Statistics] Wasserman L: All of Statistics: A Concise Course in Statistical Inference. 2004, New York: Springer, [Springer Texts in Statistics]
85.
go back to reference Gill J: 'S'attaquer a l'Heritage de Fisher: Comment Tester une Hypothese en Science Sociale: Quelques Commentaires Sur Denis.' (Grappling with Fisher's legacy in social science hypothesis testing: some comments on Denis.). Journal de la Société Franc¸aise de Statistique. 2004, 145: 1-9. Gill J: 'S'attaquer a l'Heritage de Fisher: Comment Tester une Hypothese en Science Sociale: Quelques Commentaires Sur Denis.' (Grappling with Fisher's legacy in social science hypothesis testing: some comments on Denis.). Journal de la Société Franc¸aise de Statistique. 2004, 145: 1-9.
86.
go back to reference Matthews JR: Quantification and the Quest for Medical Certainty. 1995, Princeton, NJ: Princeton University Press Matthews JR: Quantification and the Quest for Medical Certainty. 1995, Princeton, NJ: Princeton University Press
87.
go back to reference Marks HM: The Progress of Experiment: Science and Therapeutic Reform in the United States, 1900-1990. 2000, Cambridge, UK: Cambridge University Press, [Cambridge History of Medicine], 1st pbk Marks HM: The Progress of Experiment: Science and Therapeutic Reform in the United States, 1900-1990. 2000, Cambridge, UK: Cambridge University Press, [Cambridge History of Medicine], 1st pbk
88.
go back to reference Porter TM: Trust in Numbers: The Pursuit of Objectivity in Science and Public Life. 1995, Princeton, NJ: Princeton University Press Porter TM: Trust in Numbers: The Pursuit of Objectivity in Science and Public Life. 1995, Princeton, NJ: Princeton University Press
89.
90.
go back to reference Clarke M, Chalmers I: Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents?. The Journal of the American Medical Association. 1998, 280 (3): 280-282. 10.1001/jama.280.3.280.PubMed Clarke M, Chalmers I: Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents?. The Journal of the American Medical Association. 1998, 280 (3): 280-282. 10.1001/jama.280.3.280.PubMed
91.
go back to reference Clarke M, Alderson P, Chalmers I: Discussion sections in reports of controlled trials published in general medical journals. The Journal of the American Medical Association. 2002, 287 (21): 2799-2801. 10.1001/jama.287.21.2799.PubMed Clarke M, Alderson P, Chalmers I: Discussion sections in reports of controlled trials published in general medical journals. The Journal of the American Medical Association. 2002, 287 (21): 2799-2801. 10.1001/jama.287.21.2799.PubMed
92.
go back to reference Ioannidis JPA, Haidich A, Lau J: Any casualties in the clash of randomised and observational evidence? No - recent comparisons have studied selected questions, but we do need more data. British Medical Journal. 2001, 322 (7291): 879-880. 10.1136/bmj.322.7291.879. [PMC1120057]PubMedPubMedCentral Ioannidis JPA, Haidich A, Lau J: Any casualties in the clash of randomised and observational evidence? No - recent comparisons have studied selected questions, but we do need more data. British Medical Journal. 2001, 322 (7291): 879-880. 10.1136/bmj.322.7291.879. [PMC1120057]PubMedPubMedCentral
93.
go back to reference Lawlor DA, Smith GD, Kundu D, Bruckdorfer KR, Ebrahim S: Those confounded vitamins: what can we learn from the differences between observational versus randomised trial evidence?. Lancet. 2004, 363 (9422): 1724-1727. 10.1016/S0140-6736(04)16260-0.PubMed Lawlor DA, Smith GD, Kundu D, Bruckdorfer KR, Ebrahim S: Those confounded vitamins: what can we learn from the differences between observational versus randomised trial evidence?. Lancet. 2004, 363 (9422): 1724-1727. 10.1016/S0140-6736(04)16260-0.PubMed
94.
go back to reference Vandenbroucke JP: When are observational studies as credible as randomised trials?. Lancet. 2004, 363 (9422): 1728-1731. 10.1016/S0140-6736(04)16261-2.PubMed Vandenbroucke JP: When are observational studies as credible as randomised trials?. Lancet. 2004, 363 (9422): 1728-1731. 10.1016/S0140-6736(04)16261-2.PubMed
95.
go back to reference Michiels S, Koscielny S, Hill C: Prediction of cancer outcome with microarrays: a multiple random validation strategy. Lancet. 2005, 365 (9458): 488-492. 10.1016/S0140-6736(05)17866-0.PubMed Michiels S, Koscielny S, Hill C: Prediction of cancer outcome with microarrays: a multiple random validation strategy. Lancet. 2005, 365 (9458): 488-492. 10.1016/S0140-6736(05)17866-0.PubMed
96.
go back to reference Ioannidis JP, Ntzani EE, Trikalinos TA, Contopoulos-Ioannidis DG: Replication validity of genetic association studies. Nature Genetics. 2001, 29 (3): 306-309. 10.1038/ng749.PubMed Ioannidis JP, Ntzani EE, Trikalinos TA, Contopoulos-Ioannidis DG: Replication validity of genetic association studies. Nature Genetics. 2001, 29 (3): 306-309. 10.1038/ng749.PubMed
97.
go back to reference Ioannidis JPA: Contradicted and initially stronger effects in highly cited clinical research. The Journal of the American Medical Association. 2005, 294 (2): 218-228. 10.1001/jama.294.2.218.PubMed Ioannidis JPA: Contradicted and initially stronger effects in highly cited clinical research. The Journal of the American Medical Association. 2005, 294 (2): 218-228. 10.1001/jama.294.2.218.PubMed
98.
go back to reference Colhoun HM, McKeigue PM, Smith GD: Problems of reporting genetic associations with complex outcomes. Lancet. 2003, 361 (9360): 865-872. 10.1016/S0140-6736(03)12715-8.PubMed Colhoun HM, McKeigue PM, Smith GD: Problems of reporting genetic associations with complex outcomes. Lancet. 2003, 361 (9360): 865-872. 10.1016/S0140-6736(03)12715-8.PubMed
99.
go back to reference Ioannidis JPA: Genetic associations: false or true?. Trends in Molecular Medicine. 2003, 9 (4): 135-138. 10.1016/S1471-4914(03)00030-3.PubMed Ioannidis JPA: Genetic associations: false or true?. Trends in Molecular Medicine. 2003, 9 (4): 135-138. 10.1016/S1471-4914(03)00030-3.PubMed
100.
go back to reference Ioannidis JPA: Microarrays and molecular research: noise discovery?. Lancet. 365 (9458): 454-455. Ioannidis JPA: Microarrays and molecular research: noise discovery?. Lancet. 365 (9458): 454-455.
101.
go back to reference Neyman J, Pearson ES: On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character. 1933, 231: 289-337. 10.1098/rsta.1933.0009. [ArticleType: primary article/Full publication date: 1933/Copyright © 1933 The Royal Society] Neyman J, Pearson ES: On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character. 1933, 231: 289-337. 10.1098/rsta.1933.0009. [ArticleType: primary article/Full publication date: 1933/Copyright © 1933 The Royal Society]
102.
go back to reference Strunk A, Bhalla V, Clopton P, Nowak RM, McCord J, Hollander JE, Duc P, Storrow AB, Abraham WT, Wu AHB, Steg G, Perez A, Kazanegra R, Herrmann HC, Aumont MC, McCullough PA, Maisel A: Impact of the history of congestive heart failure on the utility of B-type natriuretic peptide in the emergency diagnosis of heart failure: results from the Breathing Not Properly Multinational Study. The American Journal of Medicine. 2006, 119: 69.e1-11. 10.1016/j.amjmed.2005.04.029. [http://www.ncbi.nlm.nih.gov/pubmed/16431187] Strunk A, Bhalla V, Clopton P, Nowak RM, McCord J, Hollander JE, Duc P, Storrow AB, Abraham WT, Wu AHB, Steg G, Perez A, Kazanegra R, Herrmann HC, Aumont MC, McCullough PA, Maisel A: Impact of the history of congestive heart failure on the utility of B-type natriuretic peptide in the emergency diagnosis of heart failure: results from the Breathing Not Properly Multinational Study. The American Journal of Medicine. 2006, 119: 69.e1-11. 10.1016/j.amjmed.2005.04.029. [http://​www.​ncbi.​nlm.​nih.​gov/​pubmed/​16431187]
103.
go back to reference McCullough PA, Nowak RM, McCord J, Hollander JE, Herrmann HC, Steg PG, Duc P, Westheim A, Omland T, Knudsen CW, Storrow AB, Abraham WT, Lamba S, Wu AHB, Perez A, Clopton P, Krishnaswamy P, Kazanegra R, Maisel AS: B-type natriuretic peptide and clinical judgment in emergency diagnosis of heart failure: analysis from Breathing Not Properly (BNP) Multinational Study. Circulation. 2002, 106 (4): 416-422. 10.1161/01.CIR.0000025242.79963.4C.PubMed McCullough PA, Nowak RM, McCord J, Hollander JE, Herrmann HC, Steg PG, Duc P, Westheim A, Omland T, Knudsen CW, Storrow AB, Abraham WT, Lamba S, Wu AHB, Perez A, Clopton P, Krishnaswamy P, Kazanegra R, Maisel AS: B-type natriuretic peptide and clinical judgment in emergency diagnosis of heart failure: analysis from Breathing Not Properly (BNP) Multinational Study. Circulation. 2002, 106 (4): 416-422. 10.1161/01.CIR.0000025242.79963.4C.PubMed
104.
go back to reference Glimcher PW: Decisions, Uncertainty, and the Brain: The Science of Neuroeconomics. 2003, Cambridge, Mass: MIT Press Glimcher PW: Decisions, Uncertainty, and the Brain: The Science of Neuroeconomics. 2003, Cambridge, Mass: MIT Press
105.
go back to reference Barlow H: Redundancy reduction revisited. Network-Computation in Neural Systems. 2001, 12 (3): 241-253. Barlow H: Redundancy reduction revisited. Network-Computation in Neural Systems. 2001, 12 (3): 241-253.
106.
go back to reference Barlow H: The coding of sensory messages. Current Problems in Animal Behavior. Edited by: Thorpe W, Zangwill O. 1961, Cambridge: Cambridge University Press Barlow H: The coding of sensory messages. Current Problems in Animal Behavior. Edited by: Thorpe W, Zangwill O. 1961, Cambridge: Cambridge University Press
107.
go back to reference Barlow H: What is the computational goal of the neocortex?. Large Scale Neuronal Theories of the Brain. Edited by: Koch C, Davis JL. 1994, MIT Press, 1-22. Barlow H: What is the computational goal of the neocortex?. Large Scale Neuronal Theories of the Brain. Edited by: Koch C, Davis JL. 1994, MIT Press, 1-22.
108.
go back to reference Knill DC, Richards W: Perception as Bayesian Inference. 1996, Cambridge University Press Knill DC, Richards W: Perception as Bayesian Inference. 1996, Cambridge University Press
109.
go back to reference Kahneman D, Slovic P, Tversky A: Judgment Under Uncertainty: Heuristics and Biases. 1982, Cambridge: Cambridge University Press Kahneman D, Slovic P, Tversky A: Judgment Under Uncertainty: Heuristics and Biases. 1982, Cambridge: Cambridge University Press
110.
go back to reference Elstein AS: Heuristics and biases: selected errors in clinical reasoning. Academic Medicine: Journal of the Association of American Medical Colleges. 1999, 74 (7): 791-794. Elstein AS: Heuristics and biases: selected errors in clinical reasoning. Academic Medicine: Journal of the Association of American Medical Colleges. 1999, 74 (7): 791-794.
111.
go back to reference Dolan JG, Bordley DR, Mushlin AI: An evaluation of clinicians' subjective prior probability estimates. Medical Decision Making. 1986, 6 (4): 216-223. 10.1177/0272989X8600600406.PubMed Dolan JG, Bordley DR, Mushlin AI: An evaluation of clinicians' subjective prior probability estimates. Medical Decision Making. 1986, 6 (4): 216-223. 10.1177/0272989X8600600406.PubMed
112.
go back to reference Phelps MA, Levitt MA: Pretest probability estimates: a pitfall to the clinical utility of evidence-based medicine?. Academic Emergency Medicine: Official Journal of the Society for Academic Emergency Medicine. 2004, 11 (6): 692-694. Phelps MA, Levitt MA: Pretest probability estimates: a pitfall to the clinical utility of evidence-based medicine?. Academic Emergency Medicine: Official Journal of the Society for Academic Emergency Medicine. 2004, 11 (6): 692-694.
113.
go back to reference Bornstein BH, Emler AC: Rationality in medical decision making: a review of the literature on doctors' decision-making biases. Journal of Evaluation in Clinical Practice. 2001, 7 (2): 97-107. 10.1046/j.1365-2753.2001.00284.x.PubMed Bornstein BH, Emler AC: Rationality in medical decision making: a review of the literature on doctors' decision-making biases. Journal of Evaluation in Clinical Practice. 2001, 7 (2): 97-107. 10.1046/j.1365-2753.2001.00284.x.PubMed
114.
go back to reference Dawson N, Arkes H: Systematic errors in medical decision making. Journal of General Internal Medicine. 1987, 2 (3): 183-187. 10.1007/BF02596149.PubMed Dawson N, Arkes H: Systematic errors in medical decision making. Journal of General Internal Medicine. 1987, 2 (3): 183-187. 10.1007/BF02596149.PubMed
115.
go back to reference Laplace PS: Thérie Analytique Des Probabilités. 1847, Paris: Imprimerie royale Laplace PS: Thérie Analytique Des Probabilités. 1847, Paris: Imprimerie royale
116.
go back to reference Olshausen BA, Field DJ: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature. 1996, 381 (6583): 607-609. 10.1038/381607a0.PubMed Olshausen BA, Field DJ: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature. 1996, 381 (6583): 607-609. 10.1038/381607a0.PubMed
117.
go back to reference Rao RPN, Olshausen BA, Lewicki MS: Probabilistic Models of the Brain: Perception and Neural Function. 2002, The MIT Press Rao RPN, Olshausen BA, Lewicki MS: Probabilistic Models of the Brain: Perception and Neural Function. 2002, The MIT Press
118.
go back to reference Rieke F: Spikes: Exploring the Neural Code (Computational Neuroscience). 1997, Cambridge, Mass: MIT Press Rieke F: Spikes: Exploring the Neural Code (Computational Neuroscience). 1997, Cambridge, Mass: MIT Press
119.
go back to reference Koch C, Davis JL: Large-Scale Neuronal Theories of the Brain. 1994, Storming Media Koch C, Davis JL: Large-Scale Neuronal Theories of the Brain. 1994, Storming Media
120.
go back to reference Oaksford M, Chater N: The probabilistic approach to human reasoning. Trends in Cognitive Sciences. 2001, 5 (8): 349-357. 10.1016/S1364-6613(00)01699-5.PubMed Oaksford M, Chater N: The probabilistic approach to human reasoning. Trends in Cognitive Sciences. 2001, 5 (8): 349-357. 10.1016/S1364-6613(00)01699-5.PubMed
121.
go back to reference Baker C, Tenenbaum J, Saxe R: Bayesian models of human action understanding. Advances in Neural Information Processing Systems. 2006, 18: 99. Baker C, Tenenbaum J, Saxe R: Bayesian models of human action understanding. Advances in Neural Information Processing Systems. 2006, 18: 99.
122.
go back to reference Griffiths TL, Tenenbaum JB: Optimal predictions in everyday cognition. Psychological Science. 2006, 17 (9): 767-773. 10.1111/j.1467-9280.2006.01780.x.PubMed Griffiths TL, Tenenbaum JB: Optimal predictions in everyday cognition. Psychological Science. 2006, 17 (9): 767-773. 10.1111/j.1467-9280.2006.01780.x.PubMed
123.
go back to reference Edwards AWF: Likelihood. 1992, Baltimore: Johns Hopkins University Press, Expanded Edwards AWF: Likelihood. 1992, Baltimore: Johns Hopkins University Press, Expanded
124.
go back to reference Skellam JG: Models, inference, and strategy. Biometrics. 1969, 25 (3): 457-475. 10.2307/2528899.PubMed Skellam JG: Models, inference, and strategy. Biometrics. 1969, 25 (3): 457-475. 10.2307/2528899.PubMed
125.
go back to reference Azizi F, Ghanbarian A, Madjid M, Rahmani M: Distribution of blood pressure and prevalence of hypertension in Tehran adult population: Tehran Lipid and Glucose Study (TLGS), 1999-2000. Journal of Human Hypertension. 2002, 16 (5): 305-312. 10.1038/sj.jhh.1001399.PubMed Azizi F, Ghanbarian A, Madjid M, Rahmani M: Distribution of blood pressure and prevalence of hypertension in Tehran adult population: Tehran Lipid and Glucose Study (TLGS), 1999-2000. Journal of Human Hypertension. 2002, 16 (5): 305-312. 10.1038/sj.jhh.1001399.PubMed
126.
go back to reference Cowie MR, Struthers AD, Wood DA, Coats AJ, Thompson SG, Poole-Wilson PA, Sutton GC: Value of natriuretic peptides in assessment of patients with possible new heart failure in primary care. Lancet. 1997, 350 (9088): 1349-1353. 10.1016/S0140-6736(97)06031-5.PubMed Cowie MR, Struthers AD, Wood DA, Coats AJ, Thompson SG, Poole-Wilson PA, Sutton GC: Value of natriuretic peptides in assessment of patients with possible new heart failure in primary care. Lancet. 1997, 350 (9088): 1349-1353. 10.1016/S0140-6736(97)06031-5.PubMed
127.
go back to reference Leibniz GW: Dissertatio De Arte Combinatoria, in Qua Ex Arithmeticae Fundamentis Complicationum Ac Transpositionum Doctrina Novis Praeceptis Extruitur, & Usus Ambarum Per Universum Scientiarum Orbem Ostenditur; Nova Etiam Artis Meditandi, Seu Logicae Inventionis Semina Sparguntur apud Joh. Simon Fickium et Joh. Polycarp. Seuboldum, Literis Sporelianis: Lipsiae. 1666 Leibniz GW: Dissertatio De Arte Combinatoria, in Qua Ex Arithmeticae Fundamentis Complicationum Ac Transpositionum Doctrina Novis Praeceptis Extruitur, & Usus Ambarum Per Universum Scientiarum Orbem Ostenditur; Nova Etiam Artis Meditandi, Seu Logicae Inventionis Semina Sparguntur apud Joh. Simon Fickium et Joh. Polycarp. Seuboldum, Literis Sporelianis: Lipsiae. 1666
128.
go back to reference Couturat L: La Logique De Leibniz Dapres Des Documents Inedits. 1901, Collection historique des grands philosophes. Paris: F. Alcan Couturat L: La Logique De Leibniz Dapres Des Documents Inedits. 1901, Collection historique des grands philosophes. Paris: F. Alcan
129.
go back to reference Boole G: An Investigation of the Laws of Thought, on Which Are Founded the Mathematical Theories of Logic and Probabilities. 1961, New York: Dover Publications Boole G: An Investigation of the Laws of Thought, on Which Are Founded the Mathematical Theories of Logic and Probabilities. 1961, New York: Dover Publications
130.
go back to reference Boole G: The Laws of Thought (1854). 1952, La Salle, Ill: The Open Court Pub. Co Boole G: The Laws of Thought (1854). 1952, La Salle, Ill: The Open Court Pub. Co
Metadata
Title
Significance testing as perverse probabilistic reasoning
Authors
M Brandon Westover
Kenneth D Westover
Matt T Bianchi
Publication date
01-12-2011
Publisher
BioMed Central
Published in
BMC Medicine / Issue 1/2011
Electronic ISSN: 1741-7015
DOI
https://doi.org/10.1186/1741-7015-9-20

Other articles of this Issue 1/2011

BMC Medicine 1/2011 Go to the issue