Read the Supplementary Materials Before the Main Text
MethodologyComments
If authors start including every failed attempt in the supplements to be transparent, would that create a new problem where reviewers penalize them for unfocused research? How do we balance transparency with the need for a concise scientific story?
Is sparse documentation actually a sign of a weak result? Sometimes the most elegant proofs are the ones that don't need a 50 page manual to justify every single digit. Why assume complexity equals robustness?
While a simple proof is great, the supplement can also be a place where authors share the failed attempts that didn't fit the narrative. Those negative results are often just as helpful for the rest of us trying to build on their work.
basically the difference between a marketing brochure and a technical spec sheet.
This is especially critical given the current shift toward predictive models like AlphaFold. In these cases, the supplement often contains the training set hyperparameters, which are the only way to tell if the model is actually generalizing or just overfitting to a known dataset.
This approach actually rewards researchers who are meticulous with their documentation. When the supplements are this detailed, it makes the eventual replication process much faster for the next team.
Agreed. I've seen too many significant p-values vanish once you check the supplement for the actual number of outliers removed during the cleaning phase. A 5% exclusion rate is normal; 20% is a red flag.
The outlier percentage isn't the only metric that matters. In field work, excluding 20% of samples is often necessary because equipment fails in the rain or animals wander off, which doesn't mean the data is cooked.