Even heroes get it wrong sometimes
I recently read David Sackett's 2004 paper from Evidence-based Medicine “Superiority trials, non-inferiority trials, and prisoners of the 2-sided null hypothesis “ (Evid Based Med 2004;9:38-39 doi:10.1136/ebm.9.2.38). [links don’t seem to be working, will edit later if I can].
In it I found this:
“As it happened, our 1-sided analysis revealed that the probability that our nurse practitioners’ patients were worse off (by ⩾5%) than our general practitioners’ patients was as small as 0.008.”
I’m pretty sure that 0.008 probability isn’t from a Bayesian analysis and is a misinterpretation of a p-value. It isn’t the probability of the null hypothesis being false! It really isn’t! Obviously that got past the reviewers of this manuscript without comment.
Edit: I've got the paper now. It's a result from a one-tailed test for non-inferiority. The null hypothesis is that the intervention group was worse by 5% or more on their measure of function, p=0.008 so they reject the hypothesis of inferiority. But, as usual, that's the probability of getting the data (or more extreme data) if the null hypothesis is true - not the probability of the null hypothesis.