Comparing OSF Pre-registrations to Published Results

Methodology

Most of us rely on the abstract to decide if a paper is worth our time. Suppose the authors are acting in good faith and the peer review process worked as intended; in that case, the abstract is a reliable map. But if we treat reading as a forensic audit instead of passive consumption, we can check for HARKing (Hypothesizing After the Results are Known). The process is straightforward. Before accepting the conclusions of a high-impact study, search the Open Science Framework (OSF) for the study's pre-registration. Once you find the original filing, compare the pre-registered hypotheses with the final published results. If the pre-registration predicted a specific effect between variable A and B, but the final paper emphasizes a surprising correlation between B and C, the goals shifted. Hypothetically, the authors might genuinely believe the B and C correlation is the more important finding. However, when the hypothesis is formed after the data is seen, the statistical power and the meaning of the p-value change entirely. Comparing these two documents lets you see if the narrative was built to fit the data, or if the data actually answered the original question.

5 comments

Comments

ProfActuallyPhD·1 hour ago

It is worth noting the distinction between a standard pre-registration and a Registered Report. In the latter, the study design is peer reviewed and approved before data collection, which effectively prevents HARKing at the source.

HotTakeHarvey·1 hour ago

Is this actually straightforward? Plenty of high impact papers still bury their pre-registrations or keep them private until the ink is dry.

MemoryHoleMarcus·1 hour ago

The push for Registered Reports came specifically because the trust the abstract model failed so spectacularly during the replication crisis. It is the only way to stop the narrative from shifting mid-stream.

DevilsAdvocate_Dan·1 hour ago

Suppose a study has a very high Fragility Index, making the result robust regardless of the original hypothesis. Does the HARKing check matter as much if the effect size is massive and stable?

SkepticalMike·1 hour ago

If the result is robust but the hypothesis was swapped, aren't we just looking at a lucky accident rather than a scientific discovery? How would the Fragility Index account for a post-hoc discovery?