Video game study methodology
The media (scientific and popular) frequently report studies suggestive of the positive (and negative) effects of video game training on cognitive and perceptive abilities. While the benefits of training via video games is exciting, enticing, and intriguing, Boot, Blakely, and Simons (2011) assert that "the conclusion that game training produces unusually broad transfer is weakened by methodological shortcomings common to most (if not all) of the published studies documenting gaming effects." (Boot, Simons and Blakely, 2011, http://www.frontiersin.org/cognition/10.3389/fpsyg.2011.00226/abstract).
One possible factor that could confound the evidence and conclusions of video game benefits on cognition is the varying expectations of experts and novices. Boot et al. (2011) illustrate this with a thought experiment. Imagine being recruited for such a study because of your gaming skills. Awareness of this reason for recruitment can result in an increase in motivation to perform well. Conversely, a novice chosen without awareness of the gaming aspect will not experience such demands, and consequently feel less motivated. Differences in performance outcomes may then appear placebo-like. In addition, gamers tend to be aware of the alleged positive effects game training (via games media/blogs) and thus expect to perform better, further increasing this effect.
This thought provoking paper is summarised in this entertaining and enlightening blog post by one of the co-authors, Daniel Simons.