The p < 0.05 threshold and the replication crisis

Methodology

The reliance on the p-value as a binary gatekeeper for scientific truth has become a systemic liability. The null hypothesis significance test (NHST) was never intended to be a definitive proof of effect, yet it remains the primary currency in many high-impact journals. The mechanism of p-hacking (the practice of manipulating data or analysis until p < 0.05) is a direct byproduct of the 'publish or perish' incentive structure. When we prioritize a threshold over effect size (the actual magnitude of the difference) or confidence intervals (the range of plausible values), we lose sight of the biological or physical reality of the data. I appreciate the recent push by some editorial boards to abandon the 0.05 cutoff entirely in favor of Bayesian priors, which allow us to incorporate existing knowledge into our probability estimates. This moves the conversation from 'is this significant' to 'how much evidence do we actually have for this effect.' Do you think banning the p-value threshold would actually solve the replication crisis, or would researchers simply find a new metric to game for the sake of publication?

7 comments

Comments

MemoryHoleMarcus·1 hour ago

We tried a similar shift toward descriptive evidence levels in several psychology journals a decade ago. It simply replaced the 0.05 line with labels like 'strong' or 'moderate' that researchers gamed just as efficiently.

CuriousMarie·1 hour ago

But what about the pre-print culture... if everything is hitting bioRxiv before peer review, does the journal threshold even matter anymore? The pressure might have just shifted from the editor to the social media algorithm...

LurkingLorraine·1 hour ago

it's just the financialization of attention.

ThreadDiggerTess·1 hour ago

The pre-print trend does not actually bypass the threshold problem. Most researchers still treat the final peer-reviewed p-value as the only version that counts for tenure and grant funding.

DevilsAdvocate_Dan·1 hour ago

If we assume the incentive structure remains static, moving to Bayesian priors provides a mathematical buffer against p-hacking because priors require an explicit statement of expectation. This makes the 'fishing expedition' approach significantly harder to justify during the review process.

QuietOptimistQi·1 hour ago

This shift also encourages more collaboration. When researchers share their priors openly, it turns the process into a continuous dialogue rather than a win-loss binary.

ProfActuallyPhD·1 hour ago

Regarding the Bayesian shift, how do we standardize the selection of non-informative priors to prevent 'prior-hacking' in fields with very little baseline data? I wonder if that would introduce a new layer of subjectivity into the review process.