Refining statistical guidelines and requirements for one-sided tests

Author: Georgi Z. Georgiev, Published: Aug 6, 2018

Currently the US Food and Drug Administration (FDA), the European Medicines Agency (EMA) and other regulatory agencies that set standards and produce statistical guidance documents for their relevant industry tend to frame their minimum risk requirements in the form of two-sided tests and some place undue burdens on reporting one-sided tests and intervals. The FDA does so in its "Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests" while the EMA outlines its recommendations in "Statistical Principles for Clinical Trials".

Here I’ll discuss briefly the approaches of FDA, EMA, EFSA and EPA and provide short recommendations on how they can be refined based on work published on OneSided.org.

The FDA guidelines

The FDA guidelines ^[1] do not specify risk explicitly, but they do require the reporting of 95% two-sided intervals "FDA recommends you report measures of diagnostic accuracy (sensitivity and specificity pairs, positive and negative likelihood ratio pairs) or measures of agreement (percent positive agreement and percent negative agreement) and their two-sided 95 percent confidence intervals.". I’ve seen people draw from this that regulatory risk is set at 5% but given that a drug that is harmful or increases risk will never be approved the actual risk is 2.5% when the one-sided statistic corresponding to such claims is calculated. I believe it will be beneficial for all parties if it is expressed as such explicitly.

Doing so will encourage the reporting of one-sided tests where appropriate, meaning almost universally. The medical literature will finally be more at ease in reporting the error probability statistics that match its claims.

The EMA guidelines

The EMA guidelines ^[2] are more explicit as they state that "Conventionally the probability of type I error is set at 5% or less or as dictated by any adjustments made necessary for multiplicity considerations;". They are wise to continue this with "the precise choice may be influenced by the prior plausibility of the hypothesis under test and the desired impact of the results. [...] Alternative values to the conventional levels of type I and type II error may be acceptable or even preferable in some cases.". I read this as a clear acknowledgement that there is nothing special about the 5% boundary, which is a good thing.

Then we can read, regarding one-sided tests: "The approach of setting type I errors for one-sided tests at half the conventional type I error used in two-sided tests is preferable in regulatory settings." after stating that "It is important to clarify whether one- or two-sided tests of statistical significance will be used, and in particular to justify prospectively the use of one-sided tests."

The justification of using one-sided tests is not at all needed, as already discussed. The second sentence basically means that maximum regulatory risk is in fact set at 2.5% for claims of benefit, non-inferiority, equivalence, harm, increased or decreased risk, etc. same as the FDA’s.

My recommendation to EMA would be to make the maximum 2.5% risk explicit and clear and to drop the requirement for special justifications for using one-sided tests as none such precautions are needed. As we know, it is a myth that a one-sided test is somehow biased or would result in more false positives.

The EFSA guidelines

I’ve also looked at the EFSA (European Food Safety Authority) guidelines ("Guidance on Statistical Reporting") ^[3] and I did not find any hint of a universal regulatory risk minimum and barely anything about one-sided vs. two-sided tests:

"For estimation, the interval estimate to be used should be specified and justified (e.g. confidence or credible interval, level of probability, whether one- or two-sided).".

I think EFSA are erring on the cautious side here, but at least both one- and two-sided intervals are put on the same level of need for justification. I believe "justification" is too strong a word and I think so long as the null hypothesis is specified there is no need for any further justification. If the claim being supported is directional and of the same direction as the computed estimate there should be no issues and no need for explicit justification.

The EPA guidelines

The US Environmental Protection Agency (EPA) statistical guidebook is "Data Quality Assessment: Statistical Methods for Practitioners" ^[4]. It almost has the format of a textbook with formulas, examples, etc. They do not set any regulatory risk standards, but they mention that the levels 90%, 95% and 99% might be important for decisions makers.

The guideline contains clear endorsement of using one-sided tests and many examples of doing so using their very well laid out statistical tables. In fact, the only fault I could find with it was a poor attempt to provide an example for when a two-sided test was appropriate since such a test was put forward as supporting a clearly one-sided claim about the efficiency of a treatment for reducing contamination in an area.

I’m sure there are other guidelines in different agencies around the world that can be used as positive examples or as cases in which significant improvement can be achieved. It is certainly not within my interests to cover them all, but I believe the examples and brief notes given here would be helpful in writing better statistical guidelines.

References:

[1] US Food and Drug Administration (FDA): "Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests", drafted in 2003, issued on March 13, 2007.

[2] European Medicines Agency (EMA): "Statistical Principles for Clinical Trials", drafted 1997, issued Mar 1998.

[3] European Food Safety Authority (EFSA) (2014) "Guidance on Statistical Reporting", EFSA Journal 12(12):3908

[4] US Environmental Protection Agency (EPA) "Data quality assessment: statistical methods for practitioners", EPA QA/G-9S, issued Feb 2006

Enjoyed this article? Please, consider sharing it where it will be appreciated!