The bounds of meta-analytics and an alternative method
Article information
Abstract
OBJECTIVES
Meta-analysis is a statistical appraisal of the data analytic implications of published articles (Y), estimating parameters including the odds ratio and relative risk. This information is helpful for evaluating the significance of the findings. The Higgins I2 index is often used to measure heterogeneity among studies. The objectives of this article are to amend the Higgins I2 index score in a novel and innovative way and to make it more useful in practice.
METHODS
Heterogeneity among study populations can be affected by many sources, including the sample size and study design. They influence the Cochran Q score and, thus, the Higgins I2 score. In this regard, the I2 score is not an absolute indicator of heterogeneity. Q changes by bound as Y increases unboundedly. An innovative methodology is devised to show the conditional and unconditional probability structures.
RESULTS
Various properties are derived, including showing that a zero correlation between Q and Y does not necessarily mean that they are independent. A new alternative statistic, S2, is derived and applied to mild cognitive impairment and coronavirus disease 2019 vaccination for meta-analysis.
CONCLUSIONS
A hidden shortcoming of the Higgins I2 index is overcome in this article by amending the Higgins I2 score. The usefulness of the proposed methodology is illustrated using 2 examples. The findings have potential health policy implications.
Key Message
An approach to overcome the hidden shortcomings of Higgens I2 in meta-analysis. The approach has potential health policy implications.
INTRODUCTION
The genesis of meta-analysis can be traced to the work of an eminent statistician [1] who compared evidence from several studies on typhoid inoculation. Meta-analysis is intended to identify patterns of similarities and differences among studies with the same aim. Glass et al. [2] and others have discussed it in detail. Meta-analysis has been criticized for averaging the differences of studies with sample data from heterogeneous populations. A systematic review precedes the meta-analysis for the sake of appraising critical evidence in the publications. A meta-analysis is performed in sequential steps. As exemplified by research aiming to establish the impact of vitamin D on protecting patients from coronavirus disease 2019 (COVID-19) [3-23], the steps include focusing on establishing research questions, formulating the population, conducting a literature search for published results, selecting published studies of appropriate quality, and evaluating whether the summary measures in studies are comparable, whether the model to integrate the studies should involve fixed or random effects, and whether the heterogeneity among the study populations is acceptable in order for the findings of the meta-analysis to yield meaningful insights into the issue at hand.
Other noteworthy recent meta-analytic studies include Pearson [24] and more [25-31]. Recently, Hong et al. [14] published an article on the importance of meta-analysis in the journal of the Korean Society of Epidemiology, Epidemiology and Health. Using the inverse of the estimated variance of the studies, the fixed type of meta-analysis provides a weighted average estimate. When populations are heterogeneous, the random type is appropriate, and it is applied with inverse variance as weights or no weights at all. A disadvantage of meta-analysis is that the sources of bias are not accounted for in the calculations of heterogeneity. When the findings of studies lack significance, the results are often not reported in any publication; this phenomenon is known as publication bias (or the “file drawer” problem). The role of publication bias is beyond the scope of this paper. The reader is referred to Borenstein et al. [8] for the role of the Higgins score in relation to the heterogeneity of the sampled populations in meta-analyses and to Chernikova et al. [32] and Blumenfeld [33] for a discussion of simulation-based learning meta-analysis. The probability distribution of the Higgins statistic, I2 ≥0, is assumed to have a chi-squared distribution. However, in some studies, when Q≥df is not true, I2 does not have a chi-squared distribution. Note that Q and df refer to the Cochran Q score and degrees of freedom (df), respectively. In this manuscript, a modified approach is given to rectify this shortcoming in the Higgins statistic-based approach. The approach is illustrated by applying it to 2 examples—cognitive impairment and COVID-19 vaccination—for meta-analysis.
MATERIALS AND METHODS
An alternative meta analytic approach
Epidemiologists, biostatisticians, and investigators in other disciplines utilize the Higgins statistic,
The probability pattern of the Higgins statistic
Consequently,
as the expression (4) follows a chi-squared distribution with 1 df. In other words, the p-value of a data base S2 is
Ethics statement
In the article, 2 publicly available data sets were used for illustrating the usefulness of the proposed methodology. Informed consent was not required.
RESULTS
Example 1
As an illustration, we consider the recent data collected by Chen et al. [34] and Chen et al. [35] on the global prevalence of mild cognitive impairment (MCI) among elder adults living in nursing homes. The occurrence of MCI is caused by aging and/or dementia. The data they analyzed in various studies using the statistical software Stata, compiling the Q-values and the df from 53 published articles in 17 countries, are reproduced in Table 1. They concluded that there is significant heterogeneity in the studies. The Higgins statistic, I2, has been described to follow the chi-squared distribution, whose sample space should be non-negative (that is, I2 ≥0). The values of are negative (Table 1, last column) in the data for Europe and Central Asia and for the upper middle-income category. The negative values of I2 clearly attest that the Higgins statistic does not always follow the chi-squared distribution. Hence, a refined version of the Higgins statistic is a necessity, and such a revised version is our statistic, S2, whose values are displayed in Table 1.
Results of a meta-analysis on the global prevalence of mild cognitive impairment among older adults living in nursing homes
Example 2
Parents were concerned about vaccinating their children with the then-untested COVID-19 vaccine. A combined worldwide study using a meta-analysis was used to probe patterns in these concerns. A total of 98 papers across 69 different countries with 413,590 participants were examined by Alimoradi et al. [4]. The authors found that countries’ income level, location, and data collection methods were significant moderators of parents’ willingness to vaccinate their children against COVID-19. The data collection method was another significant factor influencing parental willingness. Studies collected using phone interviews had the lowest prevalence of willingness. None of the studies were thought to have exhibited heterogeneity.
Once again, the Higgins statistic, I2, exhibited negative values which violate the required non-negative sample space of the chi-squared distribution (see the last column in Table 2) in the data for all groupings. A refined version of the Higgins statistic is, once again, a necessity. For comparison, our revised statistic, I2, is displayed in Table 2.
Data on parental consent for children receiving vaccination (including Q)
DISCUSSION
A word of caution is necessary when interpreting the Higgins I2 value and its impact. There are 3 challenges in using the Higgins score, I2: (1) It is mentioned by Higgins et al. [36] that I2 is the percentage of variation across the studies that is due to heterogeneity rather than sheer chance. Khan [16] commented that “…. The I2 values of 25%, 50%, and 75% indicate low, moderate, and high heterogeneity, respectively, among the population effect sizes. I2 ≤ 25% of studies are considered to be homogeneous.” (2) Corr (Q,Y)= 0 does not imply that Q and Y are independent. (3) The I2 statistic can be negative when Q is less than df. For this situation, it is commented by Higgins et al. [36]: “Negative values of the I2 are put equal to zero so that I2 is between 0% and 100%.” This causes users to doubt the validity of the score and have less confidence in using it. These shortcomings are overcome by our refinement of the Higgins score, which we explain below:
The exact probability structure of the popularly utilized Higgins score in meta-analytic studies to assess the consistency of the findings in various studies about a healthcare topic is derived. With this probability structure, a method of finding the p-value for the Higgins score and its interpretation is devised and demonstrated. The exact new expression (4) for the score S2 is a refined version of the Higgins standardized score, which follows the chi-squared distribution with 1 df. With these new innovative results, meta-analytic researchers do not have to follow the subjective interpretations of the estimated Higgins score. Instead, the researchers could obtain the p-value for the calculated standardized S2 score based on the chi-squared distribution and conduct an objective, exact interpretation. The values of the new score S2 are objective. The authors show both the conditional and unconditional probability structures of the Higgins statistic, including how the correlation between Q and Y is derived and utilized for Q and Y to be uncorrelated and independent.
Had Higgins followed the line of the traditional thinking of statistical discipline, he could have defined the I2 score as the ratio
In conclusion, the Higgins I2 score stochastically follows a chi-squared distribution with (n-1) df, where n is the number of studies considered. In all applications, if the score is less than the df, the difference is simply considered to be 0. This zone of the probability area might not be negligible. The authors overcame this hidden shortcoming of the Higgins I2 statistic by amending it. The usefulness of the proposed methodology is illustrated by using 2 examples: (1) the global prevalence of MCI among elder adults living in nursing homes, and (2) data on vaccinating children by then-untested COVID-19 vaccines. The findings of this article have potential health policy implications.
Supplementary materials
Supplementary materials are available at https://doi.org/10.4178/epih.e2024016.
Supplementary Material 1.
A Derivation of the probability structures of the Higgins statistic
Supplementary Material 2.
A Proof of Corr(Q,Y)=0.
Supplementary Material 3.
Critical value of I2
Notes
Conflict of interest
The authors have no conflicts of interest to declare for this study.
Funding
The project was partially supported through the National Institutes of Health grant, RCMI MD007586, awarded to the Meharry Medical College.
Author contributions
Conceptualization: Shanmugam R, Singh KP. Data curation: Shanmugam R. Formal analysis: Shanmugam R, Tabatabai M, Wilus D. Funding acquisition: Tabatabai M, Wilus D. Methodology: Shanmugam R, Singh KP, Tabatabai M. Project administration: Singh KP. Visualization: Shanmugam R, Tabatabai M, Wilus D, Singh KP. Writing – original draft: Shanmugam R, Singh KP. Writing – review & editing: Shanmugam R, Singh KP, Tabatabai M, Wilus D.
Acknowledgements
The authors are grateful to their home institutions for supporting the research done in the study.