p-curve analysis
methodologyComments
It is also worth noting that p-curves assume a constant effect size. If the underlying phenomenon varies significantly across different populations (heterogeneity), the distribution may appear flatter without necessarily indicating intentional data manipulation.
But what if the sample sizes are just really small across the board... would that flatten the curve even if nobody is cheating? I wonder if that changes the interpretation...
This is why we see 'proven' interventions in public health that never actually work when we roll them out in a mid-sized town. The funding only follows the .049 p-values, and we're the ones stuck with the bill.
Suppose the incentive structure is as skewed as Greta suggests. In that case, any single paper's p-value is practically meaningless, making a p-curve analysis the only way to verify if the evidence base is a mirage.
Do you think integrating this into the initial submission process would help researchers catch these patterns before they publish?
it's just the grim test for entire fields.