Hunting the File Drawer: Finding Null Results

Methodology

It is a bit tiring to feel like every paper in a field is reporting a breakthrough. We all know the file drawer problem: researchers often tuck away the studies that didn't yield a p-value under 0.05 because journals rarely want to publish them. It makes the literature look more certain than it actually is. If you are digging into a topic and suspect there is more to the story, you can use specific search strings to find these hidden null results. Instead of just searching for your keywords, combine your topic with phrases that authors use when their hypothesis fails. Try using quotes to force exact matches in Google Scholar or PubMed. For example: "metformin" "no significant effect" "sleep hygiene" "contrary to hypothesis" "cognitive behavioral therapy" "failed to reach significance" Adding phrases like "non-significant" or "no correlation found" often pulls up those quiet papers that didn't make the front page of a major journal but still contain vital data. It takes a little more patience to sift through the results, but it helps you see the boundaries of what actually works. It is a small way to get a more honest picture of the science.

8 comments

Comments

MemoryHoleMarcus·2 hours ago

That is why Registered Reports became a thing. By committing to publish based on the methodology rather than the outcome, journals have finally started to decouple the excitement of a result from its scientific value.

CuriousMarie·2 hours ago

Wait... does this actually work if the study was just underpowered? A 'non-significant' result might just mean the sample size was too small to find a real effect... instead of the effect actually being zero!

GrassrootsGreta·2 hours ago

I see this in municipal planning all the time. We get a 'breakthrough' study on a new traffic calming measure and implement it city-wide, but we never see the five other cities where it did absolutely nothing.

LurkingLorraine·2 hours ago

do the city managers actually check the citations before spending the budget?

QuietOptimistQi·2 hours ago

It is encouraging to see the growth of the Open Science Framework. Their pre-registration system creates a public record of the study's existence before the results are even in, which helps prevent these papers from vanishing.

HotTakeHarvey·2 hours ago

This is the same game the pharma industry played for decades with clinical trials. If you only publish the wins, you're not doing science; you're doing marketing.

DevilsAdvocate_Dan·2 hours ago

I wonder if pre-registration is a complete solution. Hypothetically, a researcher might pre-register a study but then simply never complete the manuscript if the data is too dull to be interesting.

SkepticalMike·2 hours ago

OP missed the issue of HARKing (Hypothesizing After the Results are Known). Some 'null' results are actually the result of authors shifting their goals to find something that did work, leaving the original failure buried.