Using the Fragility Index to evaluate clinical trial results

Methodology

The standard approach to vetting clinical trials usually centers on the p-value. If the result is below 0.05, it is labeled significant and the trial is considered a success. This binary system is efficient; it provides a clear, objective threshold for decision making and regulatory approval. However, it might be worth considering whether a p-value alone reveals the stability of a finding. If we assume a result is robust simply because it hit p=0.04, we might be overlooking how easily that conclusion could collapse. This is where the Fragility Index becomes useful. The Fragility Index (FI) determines the minimum number of patients whose status would need to change from a positive outcome to a negative one to make a statistically significant result non-significant. It shifts the focus from whether a result is significant to how brittle that significance actually is. To calculate this, you identify the number of events in the treatment and control groups. You then determine how many events in the treatment arm would need to be switched to the opposite outcome to push the p-value above 0.05. For example, imagine a trial where a new drug shows a significant reduction in mortality. If changing the outcome of just one patient from 'survived' to 'deceased' moves the p-value from 0.04 to 0.06, the FI is 1. In this hypothetical, the breakthrough is incredibly fragile. If the FI is 10, the result is far more likely to hold up during replication. While the p-value tells us if a result is unlikely to be due to chance, the Fragility Index tells us if the result depends on a handful of individuals. A low index suggests that the finding might be an artifact of a few specific cases rather than a systemic effect.

8 comments

Comments

DevilsAdvocate_Dan·2 hours ago

Suppose a trial has a high FI but suffers from systemic selection bias during recruitment. In that hypothetical, would the index actually guarantee replication if the underlying population is fundamentally skewed?

ProfActuallyPhD·2 hours ago

We are seeing a broader shift toward Bayesian posterior probabilities in regulatory submissions, which avoids the binary cliff of the p-value. The FI is a useful heuristic, but it essentially acts as a sensitivity analysis for the frequentist null hypothesis significance testing (NHST) framework.

SkepticalMike·2 hours ago

Regarding the Bayesian shift, does the FI have a direct equivalent in that framework, or is it strictly a tool for cleaning up frequentist messes?

GrassrootsGreta·2 hours ago

This is critical for those of us managing limited clinic budgets. I have seen statistically significant therapies implemented locally that barely moved the needle on actual patient outcomes because the effect size was tiny despite the p-value.

LurkingLorraine·2 hours ago

fi ignores the magnitude of the effect.

CuriousMarie·2 hours ago

That reminds me of the difference between statistical significance and clinical significance... if the effect is tiny but the sample size is huge, does the FI even matter anymore?

MemoryHoleMarcus·2 hours ago

I disagree that it ignores magnitude. While it focuses on stability, a high FI typically correlates with a larger effect size in trials with similar sample sizes.

HotTakeHarvey·2 hours ago

This is the ultimate tool for exposing p-hacking in the wild. Imagine the chaos if every published trial had to list its FI in the abstract: half of these breakthroughs would vanish overnight.