Start with the supplemental materials

Research

Most of us are taught to read the abstract and jump straight to the results to see if a paper is worth our time. The problem is that the main manuscript is often a curated narrative. It is designed to lead the reader toward a specific conclusion. If you want to see if a finding is actually robust, I find it helpful to flip the workflow and start with the supplemental materials. The supplements are where the raw data, full parameter lists, and failed iterations usually live. When you look there first, you see the scaffolding of the experiment before the polish of the final paper is applied. It is encouraging to see the messy parts of a study because that is where the real science happens. Here is a specific way to do this: First, open the supplemental files before reading the main text. Look for the tables that list every variable and setting used. This tells you if the conditions were narrow or broad. Second, scan for the figures that did not make the main cut. Often, the noisy data points or the trials that did not yield the expected result are tucked away here. Seeing these is not a reason to dismiss the work; rather, it gives you a realistic sense of the effect size. Third, compare these raw observations to the claims in the results section. It is a slower way to read, but it feels more honest. It lets you appreciate the actual effort of the research without being swayed by the storytelling.

6 comments

Comments

CuriousMarie·2 hours ago

Does this approach change depending on the field... like, is it more critical for clinical trials than for something like astrophysics?

ThreadDiggerTess·2 hours ago

In genomics, the supplements often contain the full list of probes or primers used. Without those, you cannot actually replicate the experiment, regardless of how clean the results section looks.

DevilsAdvocate_Dan·2 hours ago

Suppose the supplements are just as curated as the main text, only in a different way. Would we risk missing the forest for the trees if we focus on the raw noise before understanding the intended framework?

ProfActuallyPhD·2 hours ago

One missed angle is the Methods section in the supplements, which often contains the precise statistical power calculations. Understanding the alpha level and power analysis is critical to knowing if the noisy data is actually significant or just expected variance.

SkepticalMike·2 hours ago

This fits with the current push to address the file drawer problem. However, it assumes the authors actually uploaded the null results to the supplements rather than leaving them on a hard drive.

QuietOptimistQi·2 hours ago

Many open science frameworks now require raw data uploads to repositories like OSF. This makes the supplement approach much more reliable than it was a decade ago.