June 17, 2013

Sample size and the Minimum Clinically Important Difference

Performing a sample size calculation has become part of the rigmarole of randomized trials and is now expected as a sign of “quality”. For example, the CONSORT guidelines include reporting of a sample size calculation as one of the items that should be included in a trial report, and many quality scales and checklists include presence of a sample size calculation as one of the quality markers. Whether any of this is right or just folklore is an interesting issue that receives little attention. [I’m intending to come back to this issue in future posts]

For now I want to focus on one aspect of sample size calculations that seems to me not to make much sense.

In the usual idealized sample size calculation, a treatment effect that it is desired to detect is assumed. Ideally this should be the “minimum clinically important difference” (MCID); the smallest difference that it would be worthwhile to know about, or the smallest difference that would lead to one treatment being favoured over the other in clinical practice. Obviously this is not an easy thing to calculate, but leaving practical issues to one side for the moment, in an ideal situation you would have a good idea of the MCID. Having established the MCID, this is used as the treatment effect in a standard sample size calculation, based on a significance test (almost always at the 5% significance level) and a specified level of power (almost invariably 80% or 90%). This gives a number of patients that need to be recruited. This number will give a “statistically significant” difference the specified percentage of the time (power) if the true difference is the MCID.

The problem here is that the sample size calculation is based on finding a statistically significant result, not demonstrating that the difference is larger than a certain size. But if you have identified a minimum clinically important difference, what you want to be able to say with a high degree of confidence is whether the treatment effect exceeds it. However, the standard sample size calculation is based on statistical significance, which is equivalent to finding that the difference that is non-zero. Obviously, the upper confidence limit is likely to be close to zero and will only rarely be far enough from zero to exclude the MCID. Hence the standard sample size may have adequate power to show whether there is a non-zero difference, but has very little power to show that the difference exceeds the MCID. Hence most results will be inconclusive; they will show that there is evidence of benefit, but uncertainty that it large enough to be clinically important.

As an example, imagine the MCID is thought to be a risk ratio of 0.75 (a bad outcome occurs in 40% of the control group and 30% of the intervention group). A standard sample size calculation gives 350 participants per group. So you do the trial and (unusually!) the proportions are exactly as expected: 40% in the control and 30% in the intervention group. The calculated risk ratio is 0.75 but the 95% confidence interval around this is 0.61 to 0.92. So you can conclude that the treatment has a non-zero effect but you don’t know whether it exceeds the minimum clinically important difference. With this result you would only have a 50% chance that the real treatment effect exceeded the MCID.

So sizing a trial based on the MCID might seem like a good idea, but in fact if you use the conventional methods, the result is probably not going to give you much information about whether the treatment effect really is bigger than the MCID or not. I suspect that in most cases the excitement of a “statistically significant” result overrides any considerations of the strength of the evidence that the effect size is clinically useful.

3 comments by 0 or more people

1. tomwilks

Hi Simon

I am currently doing my PhD in clinical based research. We want to use the MCID to determine sample size but I am struggling to get my head around it. I have determined the MCID of muscle mass (which we are trying to improve) at 600g/0.6kg. How can i use this to calucaute sample size? What calcualtors can I use? Where does the MCID fit into a standard sample size calcualtion.

Our sample size at the minute is based on old data (mean change 7.4, SD 9.8)

(The current study is 2 groups – a placebo vs supplement. With our main DV being muscle mass)

Thanks for any help
T

16 Jul 2013, 16:13

2. Chee-Wee Tan

Simon,

I can see where you’re coming from on this. If MCID (in its various guises) is not an optimal way of going about sample size determination, based on your expertise, what is a good way to go round it. As far as I know MCID is one of the commonest ways to determine sample size for trials.

Chee-Wee

05 Aug 2013, 12:54

3. Simon Gates

Hi Tom

Sorry for delay in replying – taken out by family issues then holiday for the last month or so.

The standard sample size calculation needs an estimate of the difference it is desired to “detect”, which the MCID is often used for, and an estimate of the standard deviation. It’s usually easiest to use the mean and standard deviation for each group (intervention and control, or whatever they are); most online calculators work using these as the inputs.

There are a quite a few websites that will do sample size calculations for you; one of my favourites is
http://homepage.stat.uiowa.edu/%7Erlenth/Power/ (you’ll want the two sample t-test option I think)
or you might try
http://www.sealedenvelope.com/power/

But you will deduce from the post here that I think the standard method of doing sample sizecaculations leaves a lot to be desired!

30 Aug 2013, 14:55

You are not allowed to comment on this entry as it has restricted commenting permissions.

June 2013

Mo Tu We Th Fr Sa Su
May |  Today  | Jul
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30

Galleries

• Hi Tom Sorry for delay in replying – taken out by family issues then holiday for the last month or s… by Simon Gates on this entry
• Simon, I can see where you're coming from on this. If MCID (in its various guises) is not an optimal… by Chee-Wee Tan on this entry
• Hi Simon I am currently doing my PhD in clinical based research. We want to use the MCID to determin… by tomwilks on this entry
• I think your comment reveals how nonsensical null hypothesis testing is (and I see from your other p… by matt on this entry
• Thanks for commenting Matt – I do wonder if anyone ever looks at any of this, not that this is a pro… by Simon Gates on this entry