Not all Meta-Analyses and Review Papers Adopt the Same Standards
What to believe? Who to trust? Complicated questions with so many available information sources, clamoring for attention. Historically, governments had the last word on nutrition policy but social media is challenging agencies every day.
Because of financial interests, scientific research funded by industry has been viewed suspiciously but people are motivated by many factors other than financial gain. White hat bias is a threat to the integrity of scientific reporting because it distorts interpretation and fuels misunderstanding.
Sometimes research papers are retracted because of errors. Even more unfortunately, some scientists publish faked data. Fraudulent acts, like those of Andrew Wakefield who altered facts about patients’ medical histories, may have been driven by self-interests to identify a new syndrome.
For this reason, guidance on the proper use of individual participant data (IDP) from controlled trials (RCTs) is important. When reading a meta-analysis of RCTs or any review, these questions should be asked:
- Is the meta-analysis part of a systematic review? Were all the eligible trials identified?
- Were IDP obtained from most trials? Was the integrity of the IDP checked?
- Were the analyses prespecified in detail? Was the risk of bias included trials assessed? Did the authors assess randomization, allocation concealment or blinding? Did the study include result from every randomized participant?
- Were all relevant outcomes reported? Did the investigators check the quality of time-to-event outcome data?
- Were the methods of analysis appropriate? Does this extend to overall effects and did they differ by trial or participant characteristics? For example, compliance or nutritional status?
These are important questions. Whether one is assessing the benefits of vitamin D and calcium on bone health or the impact of omega-3 status on heart, eye, or prostate health, it is important to have common assessments of status.
Vitamin D is important for more than bone health. For every 25 nmol/L drop in serum 25-hydroxyvitamin D concentration, Chowdhury and associates reported a 16% increased risk of mortality. The combination of a SINGLE measure of vitamin D status, serum 25-hydroxyvitamin D, across 73 studies involving 850,000 individuals leads to a strong conclusion.
Contrast this with the situation faced in omega-3 research where there is a lack of standard methodology. A 2015 meta-analysis of prospective RCTs had to rely upon both dietary omega-3 fatty acid intake and blood biomarker data to evaluate the impact of omega-3 fatty acids on prostate cancer risk. Not surprisingly, the 12 RCTs assessing only dietary intake were non-conclusive. A reduced risk of prostate cancer was observed in the 9 biomarker studies measuring blood omega-3 concentrations. However, the fact that the RCTs used different biological measures of omega-3 status (plasma phospholipid (PL) fatty acids, serum PL fatty acids, serum cholesterol fatty acids, and red blood cell membrane fatty acids) undermines the robustness of the conclusion.
As Cope and Allison write, we would benefit from a renewed commitment to faithfulness in research reporting . By following the recommendations of Tierney et al, journal reviewers and editors can apply standards to meta-analyses and decision-makers can better recognize those that are well designed and conducted.
Tierney JF, Vale C, Riley R, Smith CT, Stewart L, Clarke M, Rovers M. Individual participant data (IPD) meta-analyses of randomized controlled trials: Guidance on their use. 2015 PLoSMed doi: 10.1371/journal.pmed.1001855
Cope MB, Allison DB. White hat bias: a threat to the integrity of scientific reporting. 2010 Acta Paediatr doi: 10.1111/j.1651-2227.2010.02006.x
Cope MB, Allison DB. White hat bias: examples of its presence in obesity research and a call for renewed commitment to faithfulness in research reporting. 2010 Int J Obes doi: 10.1038/ijo.2009.239
Alexander DD, Bassett JK, Weed DL, Barrett EC, Watson H, Harris W. Meta-analysis of long-chain omega-3 polyunsaturated fatty acids (LCω-3PUFA) and prostate cancer. 2015 Nutr Cancer doi: 10.1080/01635581.2105.1015745
Grey A, Bolland M. Web of industry, advocacy, and academia in the management of osteoporosis. 2015 BMJ doi: 10.1136/bmj.h3170
Chowdhury R, Kunutsor S, Vitezova A, et al. Vitamin D and risk of cause specific death: systematic review and meta-analysis of observational cohort and randomised intervention studies. 2014 BMJ doi: 10.1136/bmj.g1903