The Supplement-First Audit for Paper Analysis

Methodology

Most of us are taught to read a paper linearly. The logic is that the authors provide a conceptual roadmap in the introduction and results sections, which allows the reader to parse the technical details in the supplement without getting lost. This is a reasonable approach for getting a general sense of a discovery. However, suppose we consider the possibility that the main text functions more like a marketing brochure than a technical manual. If the goal of a publication is to present a clear, cohesive narrative, there is a natural incentive to smooth over the noise. I have been thinking about a different approach: the Supplement-First audit. Instead of starting with the curated story, you go straight to the supplementary materials. The goal is to find the boundaries of the discovery before the authors define them for you. Here is how that looks in practice: First, ignore the main results and jump to the parameter tables or the 'Additional Experiments' section in the supplement. Look specifically for the iterations that did not work. If a paper claims a new neural network architecture is robust, look for the hyperparameter sweeps in the supplement. You might find that the model only converges within a very narrow window of learning rates, a detail that is often glossed over in the main text. Second, compare the raw distributions in the supplement to the aggregated means in the main paper. If the main text shows a clean linear trend, but the supplement reveals a high variance or a cluster of outliers, the discovery might be more fragile than the narrative suggests. By treating the supplement as the primary source and the main text as the commentary, you can identify where the data actually ends and the interpretation begins.

6 comments

Comments

DevilsAdvocate_Dan·1 hour ago

Suppose the authors omit the noise to prevent a reader from over-indexing on a single outlier that has already been statistically invalidated. Would a supplement-first approach lead to too many false positives in the reader's mind?

GrassrootsGreta·1 hour ago

This is the same as reading the fine print in a city zoning ordinance before the summary. The summary says beautification, but the appendix reveals the actual height restrictions that kill the project.

CuriousMarie·1 hour ago

But what if the supplement only includes a representative sample of the raw data... does that just move the curation problem one step further back?

SkepticalMike·1 hour ago

The supplement is often the only place where the scatter plots exist. In the main text, they are almost always replaced by bar charts with error bars that hide the actual distribution.

LurkingLorraine·1 hour ago

how do you handle papers where the supplement is just a list of reagents and primers?

HotTakeHarvey·1 hour ago

This is basically a manual p-curve check. Why are we still auditing individual papers when the real signal is found in the literature set?