We come finally, however, to the relation of the ideal theory to real world, or "real" probability. … To someone who wants [applications, a consistent mathematician] would say that the ideal system runs parallel to the usual theory: "If this is what you want, try it: it is not my business to justify application of the system; that can only be done by philosophizing; I am a mathematician". In practice he is apt to say: "try this; if it works that will justify it". But now he is not merely philosophizing; he is committing the characteristic fallacy. Inductive experience that the system works is not evidence.
J. E. Littlewood, A Mathematician's Miscellany, 1953.
For millennia Euclidean Geometry was a statement of fact about the world order. Only in the 19th century did it come to be recognized instead as a model system—an “ideal theory”—that worked exceedingly well when applied to many parts of the real world. It then stepped down from a truth about the world to its current place as first among equals as models of the world; the most useful of a cohort of geometries, each of differential service in particular cases—on spherical surfaces and relativistic universes and fractal percolates. In like manner probability theory was born as an explanation of the contingent world—“‘real’ probability”—and, with the work of Kolmogorov among many others, it matured as a coherent model system, inheriting most features of the earlier versions of the probability calculus.
The abstraction of model systems from the world permits their development as coherent, clear and concise logics. But the abstraction has another legacy: The eventual need for scientists to reconnect the model system to the empirical world. That such rapprochement is even possible is amazing; it stimulated Wigner’s well-known allusion to ‘the unreasonable effectiveness of mathematics in describing the world’. Realizing such ‘unreasonably effective’ descriptions, however, can present reasonably formidable difficulties—difficulties that are sometimes overcome only by fiat, as noted by Littlewood, a mathematician of no mean ability, and Kline (1980), a scholar of comparable acuity. The toolbox that helps us apply of the ‘ideal theory’ of probability to scientific questions is called inferential statistics. These tools are being continually sharpened, with new designs replacing old.
Intellectual ontogeny recapitulates its cultural phylogeny. Just as we must outgrow naive physics, we must outgrow naive statistics. The former is an easier transition than the later. Not only must we as students of contingency deal with the gamblers’ fallacies and exchange paradoxes, we must cope with the academics’ fallacies and statistical paradoxes that are visited upon us as idols of our theater, the university classroom. The first step, one already taken by most readers of this volume, is to recognize that we deal with model systems, some more useful than others; not with truths about real things. The second step is to understand the character of the most relevant tools for their application, their strengths and weaknesses, and attempt to determine in which cases their marriage to data is one of mere convenience, and in which it is blessed with a deeper, Wignerian resonance. That step requires us to remain appreciative but critical craftsmen. It requires us to look through the halo of mathematics that surrounds all statistical inference to assess the goodness of fit between tool and task; to ask of each statistical technique whether it give us leverage, or just adds decoration.
This chapter briefly reviews—briefly, because there are so many good alternative sources (e. g., Harlow, Mulaik, & Steiger, 1997; R. B. Kline, 2004)—the most basic statistical technique we use, null hypothesis statistical testing—NHST—and its limits. It then describes an alternative statistic, p(rep), that predicts replicability. We remain mindful of Littlewood’s observation that “Inductive experience that the system works is not evidence [that it is true]”. But then Littlewood was a mathematician, not a scientist. The search for truth about parameters has often befuddled the progress of science, which recognizes simpler goals as well: to understand and predict. If we “Try [a tool, and] it works” that can be very good news, and may constitute a significant advance over what has been. So, try this new tool.
To make calculations of p(rep) simpler, Dr. Killeen has provided this Excel spreadsheet.