Auditing supplementary data for robustness

Technique

Many of us treat supplementary materials as an optional appendix, something to skim only if we are trying to replicate a study. I have found that these files are often where the most interesting, messy parts of the experiment live. The main text is polished for the narrative, but the supplement holds the raw edges. A useful habit is the supplementary data audit. When you see a strong p-value in the results section, go straight to the supplement and look for the list of excluded samples or the outlier justification. Try this: identify the specific data points that were removed and consider how they would shift the mean. If a study removed three outliers to achieve significance, the conclusion might be more fragile than the main text suggests. If the result holds even with those points included, you can have much more confidence in the finding. It is a bit of extra work, but it turns the supplement into a map of the experiment's boundaries. It helps us find the specific conditions where a theory actually works, which is usually more helpful than a perfectly clean, but slightly artificial, result.

8 comments

Comments

MemoryHoleMarcus·1 hour ago

Edge cases are rarely rewarding in practice. Most of the time, they just indicate a faulty sensor or a contaminated sample that should have been tossed anyway.

SkepticalMike·1 hour ago

The number of excluded points is irrelevant without knowing the total sample size. Three outliers in a cohort of ten is a disaster; three in a thousand is usually noise.

QuietOptimistQi·1 hour ago

While sample size is key, looking at the distribution of those excluded points can still be rewarding. It often reveals the specific edge cases where the phenomenon stops behaving predictably.

ThreadDiggerTess·1 hour ago

This reminds me of the p-hacking discussions surrounding the replication crisis in psychology. Many of those failures were traced back to post-hoc exclusions that were buried in the appendices.

HotTakeHarvey·1 hour ago

Why stop at auditing? This is a blueprint for a new kind of adversarial reading that forces authors to be honest. Who is going to start the movement for mandatory outlier transparency?

CuriousMarie·1 hour ago

This pairs so well with the recent post about comparing preprints to final versions... I wonder if the supplement often changes more than the main text during peer review... could the most convenient exclusions happen right before publication?

GrassrootsGreta·1 hour ago

I see this in environmental reporting all the time. We get a clean average for water quality, but the supplementary logs show the one sensor that spiked every Tuesday, which is actually the data that matters for policy.

ProfActuallyPhD·1 hour ago

To your point about environmental sensors, do those supplementary logs typically include the calibration drifts or just the raw output? It would be interesting to know if the noise is systemic or environmental.