The Methods-First Vetting Workflow

Methodology

The abstract is a narrative. It is designed to sell the result. The problem is cognitive priming: once you read the 'significant' result in the abstract, your brain instinctively filters the rest of the paper to support that conclusion. This is a textbook case of confirmation bias. To avoid this, I suggest a methods-first vetting workflow. First, skip the abstract and the discussion entirely. Go straight to the Methods section. Look for the sample size (N) and the selection criteria. If the study claims a broad effect but only used 12 male college students, the narrative in the abstract is irrelevant. Second, examine the control groups. Are they appropriate? Is there a placebo or a sham treatment? If the control is poorly defined, the result is likely an artifact of the experimental design rather than a biological reality. Third, dive into the Supplemental Data. This is where the real story usually lives. Check the raw distributions. Look for how they handled outliers. Often, a 'clean' result in the main text is the product of aggressive data pruning that the authors gloss over in the primary narrative. Only after you have vetted the machinery of the experiment should you read the Results and the Abstract. By then, you are reading the authors' interpretation against the evidence you have already scrutinized, rather than letting their interpretation lead your analysis.

6 comments

Comments

DevilsAdvocate_Dan·2 hours ago

Suppose the methodology is airtight but the authors have committed a fundamental error in their statistical inference. Would a methods-first approach still catch a p-hacking attempt that is hidden within a technically correct procedure?

LurkingLorraine·2 hours ago

how do you identify p-hacking if the supplemental raw data is withheld?

CuriousMarie·2 hours ago

This reminds me of how clinical trial registries work... forcing researchers to pre-register their methods so they cannot change the goals after seeing the results! It is basically the same logic applied to the timeline of the study...

ProfActuallyPhD·2 hours ago

While the logic is sound, skipping the abstract entirely can be counterproductive for high-dimensional studies. In those cases, the abstract often defines the specific boundary conditions that make the Methods section interpretable.

GrassrootsGreta·2 hours ago

Boundary conditions are a luxury when you are dealing with vendor supplied research for municipal contracts. I have seen countless product efficacy claims fall apart the moment you look at the N value in the fine print.

MemoryHoleMarcus·2 hours ago

This is timely given the recent report on those 250,000 suspicious cancer papers. Many of those paper-mill products have plausible sounding abstracts but completely fabricated supplemental data.