How to estimate the sample size for your next study from a publication with nothing but p-values and Ns
As statisticians, we’re frequently asked to do power and sample size calculations from very thin data. Often, we’re given nothing but an ancient publication containing a few p-values, an occasional mean, and maybe a standard deviation (if we’re lucky). Back calculating something meaningful in these situations can seem intimidating, but it turns out that most standard sample size software can do the work for us when used unconventionally. We’ve outlined 2 such unconventional uses below.
For these examples, we’ll assume that we are dealing with a typical “difference in means” type of test and that t-test sample size estimation is appropriate.
Scenario 1
You’ll need that publication, ancient or otherwise. We want a publication that reports the p-value with an appropriate degree of precision; <0.01 doesn’t do us much good*. You’ll also need sample size software that can solve for delta (aka the difference between the means). Follow these steps:
1) Open up your sample size software of choice, select the difference in means test
2) Put the observed p-value in from the publication into the “alpha” parameter
3) Enter the Ns from the publication; if it’s a 1:1 ratio (or close), you can do the total N. If it’s an unequal ratio, be sure to match the publication.
4) Put in 1 for the standard deviation of the difference in means
5) Put in .5 (or 50%) for the power
6) Have the software solve for the difference in means
The calculated result for the difference in means is an estimate of the effect size (or Cohen’s d) observed in the publication. This can then be entered back into the sample size software with your planned design to calculate the sample size for your chosen power.
*We can use something like <0.01 to get the lower bound on the effect size, but be aware that this may be too conservative for your purposes.
Scenario 2
Another unconventional use along the same lines as the above is to answer the question “What’s the worst we can do and still hit P<0.05?” To do this you will:
1) Open up your sample size software of choice, select the difference in means test
2) Put 0.05 into the “alpha” parameter
3) Enter your study N. If it’s an unequal ratio, be sure to match the study design.
4) Put in 1 for the standard deviation of the difference in means
5) Put in .5 (or 50%) for the power
6) Have the software solve for the difference in means
The calculated result for the difference in means is the smallest effect size that would result in a significant p-value.
If all this sounds daunting, Rho’s statisticians can do the heavy lifting for you. Contact us to speak with one of our biometrics and study design experts.
Bridget Daly, MS, JD, Senior Biostatistician, has eight years of experience providing statistical support for Phase 1-4 clinical trials, with six years of experience providing statistical support for integrated analyses (ISS/ISE). Ms. Daly has supported three NDA submissions, including work on pivotal clinical trials and continued post-submission support through FDA review. Ms. Daly’s experience includes consulting on protocol development, writing detailed statistical analysis plans, preparing CDISC-compliant specifications for analysis (ADaM) databases, creating CDISC-compliant submissions packages, specifying and performing statistical analyses, preparing displays and reports, communicating with sponsors, and providing safety evaluations for data monitoring committees. She has also provided support for a face-to-face FDA meeting with the Division of Anesthesiology, Addiction Medicine, and Pain Medicine (DAAP). She has collaborated with clinical researchers and medical writers on a diverse selection of indications, including acute pain, fibromyalgia, chronic kidney disease-associated pruritus, alopecia, and hyperhidrosis.