Abstract
This paper discusses the outstanding problem of replicability of empirical data in the context of recent work on meta-analysis, especially within the field of evidence-based medicine. Specifically, it deals with the methodological issue of how to determine the degrees of heterogeneity between different collected studies. After critically reviewing the standard measures used to quantify meta-analytical heterogeneity, we argue that they should be revised in such a way to take into account the statistical power of the individual studies. We thus propose some new measures of heterogeneity. Subsequently, we apply them to re-assess concrete case-studies from clinical research, thereby showing explicitly how the relevant values of heterogeneity diverge from those obtained with the original measures.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
See also Fuller (2018) for another relevant discussion present in philosophical literature.
In particular, under the fixed-effect model, one assumes that all studies in the analysis have the same true effect size, and the summary effect is our estimate of this common effect size; under the random-effects model, instead, one assumes that the true effect size varies from study to study, and the summary effect is our estimate of the mean of the distribution of effect sizes (see Borenstein et al. 2011)
It should be stressed, though, that arbitrariness and uncertainty, as well as poor effectiveness for small sample size, are of course limitations that are common to other measures, too.
Let us note that Ioannidis (2008) pointed out that effect sizes of newly discovered true associations are essentially inflated on average if power is not considered. Our approach thus tries to deal with the issue of the magnitude of effect size based on statistical power.
Let us note that, contrary to the original weights wi, summing over the new weights w~i of all k studies does not necessarily yield a value equal to 1, just owing to the presence of the readjusting factor \(\frac {\pi _{i}}{\pi _{0}}\).
For completeness, let us mention that Crins et al. also run their meta-analysis for the cases of graft-loss and death, which we do not address here.
Note that in this case the source of heterogeneity mainly arises from differences in the design of the studies themselves (for instance, only the study by Heffron et al. (2003) is randomized).
References
Bohlin, I. (2012). Formalizing syntheses of medical knowledge: The rise of meta-analysis and systematic reviews. Perspectives on Science, 20(3), 273–309.
Borenstein, M., Hedges, L.V., Higgins, J.P., & Rothstein, H.R. (2011). Introduction to meta-analysis. John Wiley & Sons.
Cochrane, W.G. (1954). The combination of estimates from different experiments. Biometrics, 10, 101–29. https://doi.org/10.2307/3001666.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Lawrence Earlbaum Associates.
Crins, N.D., Rover, C., Goralczyk, A.D., & Friede, T. (2014). Interleukin-2 receptor antagonists for pediatric liver transplant recipients: a systematic review and meta-analysis of controlled studies. Pediatric Transplantation, 18, 839–850. https://doi.org/10.1111/petr.12362.
Fuller, J. (2018). Meta-Research Evidence for evaluating therapies. Philosophy of Science, 85(5), 767–780.
DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177–188. https://doi.org/10.1016/0197-2456(86)90046-2.
Heffron, T.G., & et al. (2003). Pediatric liver transplantation with daclizumab induction therapy. Transplantation, 2003(75), 2040–2043.
Higgins, J.P., & Thompson, S.G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558.
Higgins, J.P.T., Thompson, S.G., Deeks, J.J., & Altman, D.G. (2003). Measuring inconsistency in meta-analyses. BMJ, 327, 557–60. https://doi.org/10.1136/bmj.327.7414.557.
Ioannidis, J., Patsopoulos, N., & Evangelou, E. (2007). Uncertainty In heterogeneity estimates in meta-analyses. BMJ, 2007(335), 914–6. https://doi.org/10.1136/bmj.39343.408449.80.
Ioannidis, J.P. (2008). Why most discovered true associations are inflated. Epidemiology, 19(5), 640–648.
Popper, K.R. (1935). Logik der Forschung. Vienna: Julius Springer Verlag.
Core Team, R. (2019). R: A language and environment for statistical computing R Foundation for Statistical Computing. Vienna, Austria.
Rücker, G., Schwarzer, G., Carpenter, J.R., & Schumacher, M. (2008). Undue reliance on i2 in assessing heterogeneity may mislead. BMC Medical Research Methodology, 8, 79. https://doi.org/10.1186/1471-2288-8-79.
Schriger, D.L., Altman, D.G., Vetter, J.A., Heafner, T., & Moher, D. (2010). Forest plots in reports of systematic reviews: a cross-sectional study reviewing current practice. International Journal of Epidemiology, 39(2), 421–429.
Schuller, S., Wiederkehr, J.C., Coelho-Lemos, I.M., Avilla, S.G., & Schultz, C. (2005). Daclizumab induction therapy associated with tacrolimus-MMF has better outcome compared with tacrolimus-MMF alone in pediatric living donor liver transplantation. Transplant Proceedings, 37, 1151–1152.
Simpson, R.J.S., & Pearson, K. (1904). Report on certain enteric fever inoculation statistics. The British Medical Journal, 2(2288), 1243–1246.
Stegenga, J. (2009). Robustness, discordance, and relevance. Philosophy of Science, 76(5), 650–661.
Stegenga, J. (2011). Is meta-analysis the platinum standard of evidence?. Studies in history and philosophy of science part C: Studies in history and philosophy of biological and biomedical sciences, 42(4), 497–507.
Turner, R.M., Bird, S.M., & Higgins, J.P.T. (2013). The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews. PloS ONE 2013, 8, e59202. https://doi.org/10.1371/journal.pone.0059202.
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metaphor package. Journal of Statistical Software, 36(3), 1–48.
Walker, E., Hernandez, A.V., & Kattan, M.W. (2008). Meta-analysis: Its strengths and limitations. Cleveland Clinic Journal of Medicine, 75(6), 431–439.
Wang, L.L. (2010). Retrospective statistical power: Fallacies and recommendations. Newborn and Infant Nursing Reviews, 10(1), 55–59.
Worrall, J. (2007). Evidence in medicine and evidence-based medicine. Philosophy Compass, 2(6), 981–1022.
Zumbo, B.D., & Hubley, A.M. (1998). A note on misconceptions concerning prospective and retrospective power. Journal of the Royal Statistical Society: Series D (The Statistician), 47, 385–388. https://doi.org/10.1111/1467-9884.00139.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article belongs to the Topical Collection: Philosophical Perspectives on the Replicability Crisis
Guest Editors: Mattia Andreoletti, Jan Sprenger
Rights and permissions
About this article
Cite this article
Berchialla, P., Chiffi, D., Valente, G. et al. The power of meta-analysis: a challenge for evidence-based medicine. Euro Jnl Phil Sci 11, 7 (2021). https://doi.org/10.1007/s13194-020-00321-w
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s13194-020-00321-w