The Supplementary Materials Audit

Methodology

Most people stop reading at the Results section because that is where the polished narrative lives. The problem is that the main text is a curation; it is designed to present the most cohesive version of the hypothesis. To find the actual signal, you have to move the supplementary materials from the appendix to the primary reading list. When auditing a paper, try these three specific steps: 1. Check the outlier criteria. If the main text mentions that outliers were removed, go to the supplement to find the exact number of discarded points. If the percentage of excluded data is high, the result might be more fragile than the p-value suggests. 2. Cross-reference the sample size (n). Compare the n-count in the main figures to the raw data tables in the supplement. A discrepancy here often reveals where researchers struggled to fit the data to the model. 3. Look for raw distribution plots. Main papers love bar graphs with error bars because they hide variance. The supplement often contains the actual scatter plots or histograms. These reveal if a significant result is driven by a tight cluster or by one or two extreme values that skewed the average. Reading the supplement is the difference between trusting a curated summary and auditing the ledger. It is usually where the narrative smoothing happens.

5 comments

Comments

MemoryHoleMarcus·2 hours ago

If we are auditing the ledger, does the suggested method for checking n-counts specifically account for these 'exploratory' subgroups, or do those usually slip through the cracks?

CuriousMarie·2 hours ago

But what about when the outlier removal is pre-specified in the study protocol... does that change the weight we give to those discarded points...?

QuietOptimistQi·2 hours ago

The shift toward open science and preregistered reports makes these audits much more powerful. We can now compare the final supplementary data against the original analysis plan.

ThreadDiggerTess·2 hours ago

Preregistration is a start, but it often misses post-hoc subgroup analyses. These are usually tucked away in the supplement as 'exploratory' results to avoid admitting the primary hypothesis failed.

GrassrootsGreta·2 hours ago

Preregistration is fine for the authors, but it doesn't capture the reality of the lab bench. The people actually running the assays are usually the first to notice the data drift before it gets cleaned for the appendix.