Using the GRIM Test to Spot Impossible Data
MethodologyComments
If authors do correct their data after a GRIM test... does that usually change the p-value or the final conclusion of the study? I wonder how often a 'correction' actually flips the result...
This mirrors the process of Benford's Law analysis used in forensics. By examining the frequency distribution of leading digits, analysts can identify anomalies that suggest human intervention rather than natural stochastic variation.
The suggestion that this helps distinguish rounding errors from fabrication is optimistic. I remember a few cases where authors claimed rounding errors for means that were mathematically impossible, only for the papers to be retracted once the raw data was finally demanded.
Suppose a journal now mandates the submission of raw data alongside the manuscript. In that scenario, the GRIM test becomes less of a forensic tool for reviewers and more of a first-pass filter for spotting transcription errors before the data is even audited.
It is worth noting that these discrepancies often lead to the authors providing a corrected dataset. It can actually be a catalyst for better transparency between the researchers and the peer reviewers.
The idea that raw data mandates make this a simple filter is optimistic. In practice, many reviewers don't even open the supplementary files, so the burden of spotting these errors still falls on the few people actually doing the math.
forces journals to actually vet the numbers instead of just the narrative.
This aligns with the SPRITE method for checking variance. When both the mean and the standard deviation are mathematically incompatible with the sample size, the probability of honest error drops significantly.