*Systematic*

#
All 2 entries tagged

No other Warwick Blogs use the tag *Systematic* on entries | View entries tagged *Systematic* at Technorati | There are no images tagged *Systematic* on this blog

## December 23, 2014

### The Cochrane diamond

You know, the one at the bottom of your meta-analysis that summarises the pooled result? This one:

Well, I don't like it. Why not? I think it's misleading, because the diamond shape (to me at least) suggests it is representing a probability distribution. It puts you in mind of something like this:

And that seems to make sense - the thick bit of the diamond, where your point estimate is, ought to be the area where the (unknown) true treatment effect would be most likely to be, and the thin points of the diamond are like the tails of the distribution, where the probability of the true value is getting smaller and smaller. That would be absolutely right, if the analysis was giving you a Bayesian credible interval - but it isn't.

It's a frequentist confidence interval, and as lots of people have been showing recently, frequentist confidence intervals do not represent probability distributions. They are just an interval constructed by an algorithm so that, if the experiment were repeated many times, 95% of the intervals would include the true value. They are NOT a distribution of the probability of any value of the treatment effect, conditional on the data, althought that is the way they are almost always interpreted. They don't say anything about the probability of the location of the true value, or even whether it is inside or outside any particular interval.

I think a solid bar would be a more reasonable way to represent the 95% confidence interval.

For more info:

Hoekstra R, Morey, RD, Rouder JN, Wagenmakers EJ. Robust misinterpretation of confidence intervals. Psychon Bull Rev. 2014, DOI 10.3758/s13423-013-0572-3

## October 31, 2012

### The ketogenic diet and dodgy systematic reviews

Writing about web page http://www.ncbi.nlm.nih.gov/pubmed?term=henderson%20CB%20ketogenic%20diet

I recently came across studies of the ketogenic diet for treating childhood epilepsy. This is a high fat, low carbohydrate diet that mimics starvation and increases the level of ketones in the blood, which is thought to reduce seizures in some way. Nobody has a clear idea of how this might work, but the diet is widely used for epilepsy that does not respond to drugs, perhaps largely because of a lack of other treatment options. The origin of the ketogenic diet seems rather murky, as it was first promoted in the 1920s, building on the idea that fasting could control seizures. The lack of theoretical basis doesn't mean it's wrong, of course.

Anyway, what provoked my interest was a systematic review of the evidence for the ketogenic diet (Henderson et al, Journal of Child Neurology 2006; 21(3): 193-198). This apparently found pretty good evidence of the diet's effectiveness; an odds ratio for >50% seizure reduction of 2.25 (95% confidence interval 1.69-2.98). But looking a bit closer reveals some problems: there wasn't a non-ketogenic diet control group, and the comparison reported is between children who stayed on the diet and those who discontinued it. Read on a bit more and it becomes clear that a major reason for discontinuation was treatment failure (i.e. <50% seizure reduction). So the comparison is between a group many of whom had >50% seizure reduction, and a group many of whom had <50% seizure reduction. Not surprisingly, the reduction in seizures was found to be bigger in the former group. Selecting the groups partly by the outcome that they experienced has introduced serious bias into this comparison.

Abstract is copied below:

The evidence base for the efficacy of the ketogenic diet was assessed among pediatric epileptic patients by application of a rigorous statistical meta-analysis. Nineteen studies from 392 abstracts met the inclusion criteria. The sample size was 1084 patients (mean age at initiation 5.78 +/- 3.43 years). The pooled odds ratio, using a random effects model, of treatment success (> 50% seizure reduction) among patients staying on the diet relative to those discontinuing the diet was 2.25 (95% confidence interval = 1.69-2.98). The reasons for diet discontinuation included < 50% seizure reduction (47.0%), diet restrictiveness (16.4%), and incurrent illness or diet side effects (13.2%). The results indicate that children with generalized seizures and patients who respond with > 50% seizure reduction within 3 months tend to remain on the diet longer. Although no class I or II studies have been published regarding the efficacy of the ketogenic diet, this meta-analysis shows that current observational studies reporting on the therapeutic effect of the ketogenic diet contain valuable statistical data. Future observational studies should aim for long-term follow-up, patient dropout analysis, and improved seizure type characterization.

I will update when I've got the full paper.