Is the widespread usage of two-sided tests a result of a usability/presentation issue?

Author: Georgi Z. Georgiev, Published: Aug 6, 2018

Since I was born in the age of easy access to abundant computing power until recently I did not fully grasp how much statistical applications – both applied and scientific, depended on pre-calculated tables of critical boundaries and probabilities. When I realized the extent to which this was true in a previous era, it downed on me that this might be a plausible factor in the widespread adoption of two-sided calculations despite them being unsuited / inappropriate for supporting most claims made by both applied and scientific researchers.

In short, I believe that a significant reason for the preference for two-sided tests over one-sided ones is how the famous Fisher tables of the T-distribution, Z-distribution and X² distribution were tabulated and presented.

In order to support this claim I will first demonstrate the importance and ubiquitous use of Fisher’s tables, presented for the first time in his 1925 book "Statistical Methods for Research Workers" ^[1]. Then I will explain why I believe there is a major flow in their presentation and how it was detrimental to and discouraged usage of one-sided tests in the early days of statistics. This then carried on to present day where computing power makes usage of such tables redundant.

The Importance of Fisher’s Tables

The fact that the tables of critical values and probabilities for different distributions that Fisher included in his book are of importance is rarely debated, but I believe stressing this point cannot be done enough, so I will lay some facts and citations on the matter by reputable sources to make the point more vivid.

Writing about Fisher’s "Statistical Methods for Research Workers" Vyas & Desai (2015) state ^[2]: "Here, Fisher included tables that gave the value of the random variable for specially selected values of p that were much more compact than Pearson’s detailed tables. An unexpected reason for a new way of tabling is suggested by Jack Good: ‘Kendall mentioned that Fisher produced the tables of significance levels to save space and to avoid copyright problems with Karl Pearson, whom he disliked’. […] According to Conniffe, this work ‘went through many editions and motivated and influenced the practical use of statistics in many fields of study’. Today it is considered the gold standard of applied statistics for scientists in many fields. It seems that Fisher did as much to popularize the use of statistics in other disciplines as he did to contribute to its own development."

How important statistical tables were at the time is apparent by a report in Kendell’s "Ronald Aylmer Fisher, 1890-1962" (1963) ^[3] on the dispute between K.Pearson and Fisher over the reproduction of Pearson’s chi-squared tables that Fisher requested to be included in his book. "This was perhaps not simply a personal matter because the hard struggle which Pearson had for long experienced in obtaining funds for printing and publishing statistical tables had made him most unwilling to grant anyone permission to reproduce. He was afraid of the effect on sales of his Tables for Statisticians and Biometricians on which he relied to secure money for further table publications. It seems, however, to have been this refusal which first directed Fisher’s thoughts towards the alternative form of tabulation with quantiles as argument, a form he subsequently adopted for all his tables and which has become common practice."

To get a sense of how much of a work that was, it has to be said that according to Box (1981) ^[4] it took several years and the usage of two mechanical calculators to compute the tables with a satisfying level of precision. For a full understanding of the monumental work that tabulating the values for the t distribution and others was, I recommend a reading of Box (1981) "Gosset, Fisher, and the t Distribution".

Yet another testament for that can be found in Krishnan (1997) ^[5] - Fishers Contributions to Statistics: "These tables, together with those by Pearson and Hartley, were essential tools of a statistician's trade in those days when a statistical laboratory consisted of manually or electrically operated calculating machines and even in the days of electronic desk calculators." (emphasis mine).

How the tables are presented in the book

Again, according to Box, the tables published by Fisher represented not only something with wide applications, but also a significant improvement over existing ones in terms of presentation, percentiles being presented not only made the tables shorter, but much more readable and easy to use by practitioners.

Yet, likely for reasons of taking less space, the tables presented only two-tailed probabilities for the corresponding values of the X, t and z statistic.

Starting from Table I: "The deviation in the normal distribution in terms of the standard deviation" we see that the values of X are given as positive numbers. So, when observing X of ~1.64 one adds the column header: .10 to the row header: 00 and finds out P=0.1. When observing X of ~-1.64 one has to first take the absolute value, find that in the column and find the same P. Only in the explanatory text below the table is it mentioned that "x is the deviation such that the probability of an observation falling outside the range from -x to +x is P". This corresponds to a test of a point null and a two-sided alternative hypothesis.

Scans of the table are available online. Here is a partial reconstruction of the table to illustrate what Table I: "Table of x" looked like:

	.01.	.02.	...	.08.	.09.	.10.
.00	2.575	2.326	...	1.750	1.695	1.644
.10	1.598	1.554	...	1.340	1.310	1.281
...	...	...	...	...	...	...
0.70	.371	.358	...	.279	.266	.253
0.80	.240	.227	...	.150	.138	.125
0.90	.113	.100	...	.025	.012	.0

The explanatory text below the table in full: "The value of P for each entry is found by adding the column heading to the value in the left-hand margin. The corresponding value of x is the deviation such that the probability of an observation falling outside the range from -x to +10 is P. For example, P = .03 for x = 2.170090; so that 3 per cent of normally distributed values will have positive or negative deviations exceeding the standard deviation in the ratio 2.170090 at least."

There is no explicit guidance on how one is to treat x if they are interested in the probability of an observation falling in the range from 0 to +x or 0 to -x (the two most commonly used one-sided nulls) or any other range, for that matter.

Then we have a Table IV: "Table of t" wherein we are again given only the positive values of t with no guidance that the probabilities listed are those for a two-sided test, nor guidance in how to convert them for at least the simple one-sided case.

Such guidance is offered in a paragraph inside the book, some 40 pages before the table, and it states: "If it is proposed to consider the chance of exceeding the given values of t, in a positive (or negative) direction only, then the values of P should be halved." This is a clear statement on how to handle one-sided questions, but it is, unfortunately, buried many pages into a lengthy and complicated book.

Scans of the table are available online. Here is a partial reconstruction of the table to illustrate what Table IV: "Table of t" looked like:

n.	9.	8.	...	0.05	0.02	0.01
1	.158	.325	...	12.706	31.821	63.657
2	.142	.289	...	4.303	6.959	9.925
...	...	...	...	...	...	...
29	.127	.256	...	2.045	2.462	2.756
30	.127	.256	...	2.042	2.457	2.750
∞	.127	.256	...	1.95996	2.32634	2.57582

Similar issues are present in tables V.A and V.B (correlation coefficients) and Table VI on the values of the z distribution.

Given that pages with the tables were often ripped off the book and kept at a hand’s distance for reference it becomes painfully clear how easy it is to misuse them in favor of using two-sided significance calculations even when such are not warranted or wanted.

How the presentation favors two-sided significance calculations

It should already be easy for a person with experience in graphical design and user experience to spot why the tables as they are presented would favor two-sided calculations, even if said person has minimal statistical knowledge. Let us make this explicit.

1. Aside from table I, there is no relevant information on the page on which the table is printed showing that the P-values are calculated for a null of zero difference. This makes it more likely for one to remain unaware of that fact and makes it easier for people wise to it to forget it.

2. The explanation on how to treat the one-sided case is buried deep in the book and only references one of the several tables. It is therefore likely that many of the users of the tables would be unaware of it and if in need of such guidance they should consult other sources (books, papers, colleauges, etc.).

Imagine for a moment that you are a research worker with deep expertise in your field, but only cursory understanding of a statistical procedure or two which are relevant to the daily tasks you work with. You, unlike the likely reader of this article, have not had the joy of familiarizing yourself with the deep mathematical and philosophical roots of the procedures you apply. How easy it is then to interpret the probabilities as applicable to any null hypothesis you have at hand?

As demonstrated elsewhere ("Examples of improper use of two-sided hypotheses") many researchers do not understand the need to define a statistical null hypothesis that corresponds to their research hypothesis and threat any statistical answer as if it applies to their research hypothesis. As seen, making a directional claim can be quite subtle at times and given the practice of simply attaching a p-value to a number without explicitly specifying the null hypothesis under which it was calculated it is no wonder that we see so many p-values detached from the conclusions they are meant to support.

The above can be shrugged off as conjecture, ex-post-facto explanation, but I believe simple empirical tests can and will confirm this. A way to test this would be to introduce the tables to students who are familiar with what a t or z distribution is, and then to ask them to apply it to a couple of different problems requiring the use of one-tailed tests to see if they will extract the correct p-value from the tables. To make the test more sensitive, it would be recommended to include at least one experiment in which the observed value of the test statistic has a negative value, e.g. z=-1.65.

A control group can use different, one-sided tables. A great example are the tables provided in the US Environmental Protection Agency: "Data Quality Assessment: Statistical Methods for Practitioners" ^[6] guide, for example the Z table (A-1) is split into to sections, one for negative values of z and another for positive. The p-values provided are one-sided. Above each section is a graphical representation of the rejection region. Now to do a two-sided calculation you need to add up the p-values from the two tables:  for -|z| and for |z|, to get the two-sided p-value.

Current situation and conclusions

Given the usage of one-tailed probabilities by Fisher in many of his examples of applications of statistical methods I do not believe that the reporting of two-tailed probabilities in the tables was any kind of deliberate attempt to discourage usage of one-tailed probabilities. It was most likely a convenient or economical decision, or just lack of oversight for the potential consequences.

Similarly, I do not think that nowadays major statistical software vendors are making it deliberately harder to get one-tailed probabilities out of their tools by adopting two-tailed probabilities as defaults. I think it is part custom/convention and part poor UI that does little to help the researcher decide on what is the best statistical test for their research hypothesis or question at hand. The unintended consequence is bias against the usage of one-sided tests and the proliferation of suboptimal matching between research and statistical hypothesis, resulting in incorrect error probabilities being reported.

In conclusion, I find strong reason to believe that lacking user experience of early statisticians and research workers in using the tables provided by Fisher in his "Statistics for Research Workers" book is at least partly responsible for present day misconceptions about one-sided tests as well as the usage of two-sided tests for providing error probabilities to the answers of questions that require the use of one-sided tests. Newer software tools continue to suffer from similar issues, propagating the confusion further into the future.

P.S. In writing the above, I also realized that in a time when these tables where the main tool of the statistician it is not so difficult for one to forget there are null hypotheses different than the nil hypothesis. The step from that to equating the two is small, indeed. Similar, but less pronounced effects are likely present due to the default choices of present day statistical software.

Reference

[1] Fisher R.A. (1925) "Statistical methods for research workers". Oliver & Boyd, Edinburg

[2] Vyas S.A., Desai S.P. (2015) "The Professor and the Student, Sir Ronald Aylmer Fisher (1890-1962) and William Sealy Gosset (1876-1937): Careers of two giants in mathematical statistics.", Journal of medical biography 23(2):98-107; https://doi.org/10.1177/0967772013479482

[3] Kendall M.G. (1963) "Ronald Aylmer Fisher, 1890–1962", Biometrika 50(1-2):1-15; https://doi.org/10.1093/biomet/50.1-2.1

[4] Box J.F. (1981) "Gosset, Fisher, and the t Distribution", The American Statistician 35(2):61-66; http://dx.doi.org/10.1080/00031305.1981.10479309

[5] Krishnan T. (1997) "Fishers Contributions to Statistics", Resonance Journal of Science Education 2(9):32-37

[6] US Environmental Protection Agency (EPA) "Data quality assessment: statistical methods for practitioners", EPA QA/G-9S, issued Feb 2006

Enjoyed this article? Please, consider sharing it where it will be appreciated!