Comparison of methods of optimal cut-point selection for biomarkers in diagnostic medicine: a simulation study with application of clinical data in health informatics

Abstract

Objectives

Several methods of cut-point selection for biomarkers have been suggested in biomedical research but the superiority of them over others was not studied comprehensively under different pairs of distributions, degree of overlap, and the ratio of sample sizes. This simulation study was aimed to compare five popular methods with application of clinical examples.

Results

The data of simulation was generated from the 12 configurations of binormal, bigamma, and biexponential pairs with different sample sizes The results showed that the four popular methods of Youden, Euclidean, Product, and Index of Union (IU) yielded identical optimal cut-point under binormal model with homoscedastic. While, with high AUC, the Youden may produce less bias and MSE, but for moderate and low AUC, Euclidean has less bias and MSE than other methods. The IU yielded more precise findings than the Youden for moderate and low AUC in binormal pairs, but its performance was lower with skewed distributions. In contrast, the cut-points produced by diagnostic odds ratio (DOR) were extremely high with low sensitivity and high MSE and bias. The results of clinical data showed that when AUC > 0.95, the five methods may produce identical cut-point, but DOR yields an extremely high value of cut-point for AUC < 0.95.

Introduction

One of the important applications of ROC curve is to determine the optimal cut-off point for quantitative biomarkers [1, 2]. However, there is no single method for determining the optimal cut-point. Several methods of cut-point selection have been developed based on ROC curve analysis [3,4,5]. A reasonable subset of the most famous of them are Youden, Euclidean, Product, Index of union, and diagnostic odds ratio (DOR). Some of them are widely used in medical research for biomarkers in diagnosis and predicting outcomes. Each of these methods are defined using unique definition based on object function criteria in ROC space. The clinicians need to better understand the accuracy and precision of the proposed methods in clinical practice. Consistency and inconsistency of results of cut-point are possible in some conditions of screening test results [6]. This may depend on underlying distributions of test results in diseased and non-diseased and degree of separation of pairs of distributions. However, limited data is available for this matter and the question is which of the proposed methods determines the optimal cut-point precisely and more accurately? A few studies have been conducted based on the population-based distributions of diagnostic test data [6], as well as simulation study from a limited pair of distributions [7,8,9,10,11,12]. In some studies, inconsistency in determining the cut-off point was shown between some methods [13]. In other simulated studies, limited cases of certain distributions have been mainly addressed, and the impact of the size of the diagnostic accuracy and the inequality of the variances and the inequality of the sample size and the degree of severe skewness in the estimation of bias and MSE have not been widely evaluated.

There are several clinical examples that motivate the topic of biomarkers in early diagnosis of diseases and health-related outcomes in modern medicine. For example, premature rupture of membrane (PROM) refers to the rupture of the ammoniatic sac before labor begins that has been reported in 3–18% of pregnancies [14]. PROM increases the risk of perinatal mortality and accounts for approximately 18–20% of perinatal fetal deaths in the United States [15] and it is the cause of approximately one third of all premature births in America [16]. Its accurate diagnosis is important because failure to recognize it can lead to obstetric complications such as chorioamnionitis, premature birth, maternal and fetal infections, and prolapsed umbilical cord [17]. On the other hand, improper diagnosis of PROM can lead to unnecessary interventions such as hospitalization [18]. Some diagnostic methods such as nitrazine, pooling and Fern test, measuring vaginal diamino oxidase, prolactin, a-fetoprotein, insulin-like growth factor-binding protein 1, fetal fibronectin and placental 1 α-macroglobulin are currently available [19]. Tests such as nitrazine and pooling are expensive, and less used as screening tests. They are considered as our gold-standard. Laboratory biomarkers such as Beta-human chorionic gonadotropin (B-HCG), urea (BUN) and creatinine (Cr) are used as PROM screening tests [20, 21]. There was no a clear clarification of the methods in their cut-point selection. Therefore, the aim of this study is twofold. Beyond the simulation of data from different configurations of pairs of distributions and comparing the different methods of cut-point election, another aim is the clinical application in “Illustrations and applications with clinical examples of data” section.

Methods

Simulation study

Data was generated by R software in pairs of diseased and non-diseased distributions of bi-normal, bi-gamma, bi-exponential with certain parameters shown in Fig. 1 (12 panels A to L) with sample size and degree of accuracy with equal and unequal variance in 1000 runs. In each pair, the certain parameters were deliberately established that the area under the curve (AUC) is in the range of: low (AUC = 60), medium (AUC = 75), high (AUC = 90), which is the degree of overlap between pairs of distributions. The samples were produced in equal sizes of 50/50, 100/100, and 200/200, and unequal sizes of 50/100, 50/150, and 50/200 in the diseased and nondiseased population respectively that is a disease prevalence of 0.33, 0.25, and 0.20 respectively.

Fig. 1

Density plot of pairs of various distributions of nondiseased and diseased groups with different parameters

Statistical methods for the optimal cut-point

We focused on a subset of the five most popular methods, including Youden’s J statistics, Euclidean distance, Product method, Index of Union (IU), and diagnostic odds' ratio (DOR). The full statistical descriptions have been illustrated elsewhere in detail [5]. In brief: (1) C-_Youden = Max (Se (c) + Sp(c) − 1) that maximizes the percent of net classification that is clinically interesting [4]. (2) C-_Euclidean = Min{Sqrt[(1 − Se(c)]² + [1 = Sp(c)]²} that minimizes the Euclidian distance between the point on ROC curve to right corner (1, 0) in ROC space [10]. (3) Liu’s method that maximizes the product of Se (c) and Sp (c) which is also known as Product methods [3]. (4) C-_Union = Min |Se(c) − AUC| +|Sp(c) − AUC|. This criterion minimizes the difference between Se and Sp and also the difference of the sum of Se and Sp by 2 times that AUC [7]. (5) C-_DOR that maximizes the ratio of the positive likelihood to the negative likelihood. The latter index as a ratio metric has more fluctuations and its shape is convex under some distributional assumptions of diagnostic test results [5, 6, 22].

Determining the true optimal cut-points

First, we calculated the true values of cut-points with five methods under the parameters of different pairs of distributions that were presented in Fig. 1 by analytical calculating sensitivity (Se) and specificity (SP), and AUC for all possible cut-off values of decision scale using the Excel 21.0. The optimal cut-points were selected by maximizing or minimizing the related metrics depending on the methods used.

Appraisal of five different methods of cut-point selection

The performance of the estimates of cut-points were assessed by bias, relative bias (RB), mean square errors (MSE), the coverage rate of confidence interval (CI) for true parameter of cut point, and mean length of CI. The average of cut-points was estimated in 1000 runs of datasets. Then the bias, relative bias and MSE were estimated. Their estimates were calculated by their empirical estimators in 1000 runs of data generated. For example, the empirical estimator of bias is determined by the average of estimated cut-points in 1000 runs minus the true value for each method respectively etc. In addition, in order to examine the percentage of coverage of true parameter of cut point, we applied the bootstrap resampling technique. To calculate the bootstrap estimate of cut- point of ĉ and its standard deviation (SD), a random sampling with replacement was drawn to generate 200 bootstrap samples in in all configurations of distributions. Furthermore, to generate a 95% CI for the optimal cut point, the percentile method was applied considering the 2.5th and 97.5th percentiles of the bootstrap distribution of ĉ.

In cross-validation of the findings, first we performed our R code program with the parameters of distributions used by Unal [7] to generate data and then the outputs of our program were compared with those reported by Unal. If our outputs differ from those reported, we reexamined the R code program, repeating until the outputs become similar to those reported.

Ethical considerations

The study protocol was approved by the ethical Board of Babol University of Medical Sciences, Babol, Iran (Ethical code: IR.MUBABOL.HRI.REC.1402.308). The informed consent was obtained from all participants reported in “Illustrations and applications with clinical examples of data” section.

Results of simulation

Bi-normal model

Table 1 presents when data is generated from a homoscedastic binormal model. The least bias was found by Euclidean method and followed by Product, IU, Youden and DOR while the least MSE was observed by IU methods for low and moderate AUC, but Youden and Euclidean for high AUC. In all configurations of binormal model, both bias and MSE, as one expected, declined with higher sample size except for DOR. For a given sample size, the lowest bias and MSE were found by high AUC. The highest bias, relative bias and MSE are related to DOR as well, which is almost unacceptable. The bottom of Table 1 shows the results when data was generated with unbalanced sample size. The IU method has the lowest bias and is followed by product, Euclidean, Youden, and DOR. For unequal sample sizes. Unless the Youden, the MSE and bias of Euclidean, Product, and IU are lower for equal sample sizes than unequal for all configurations of degree of overlap. However, surprisingly, for Youden, the bias of equal sample size appeared to be higher than unequal sample sizes but not for a high AUC. The Youden index produced the less precise estimates of cut-points in particular for low and moderate AUC than the three other methods with relatively higher bias and MSE but not for high AUC. Overall, the highest MSE and bias were found by DOR in all configurations.

Table 1 The MSE, bias and elative bias of different methods of optimal cut-point selection when data generated under binormal distributions with homoscedastic and equal/unequal sample size from diseased (D) and non-diseased (ND) according to degree of overlap or accuracy

Table 2 indicates the results when data was generated by binormal model with non-homoscedastic. The IU method resulted the lowest MSE for low and moderate AUC but the Euclidean index yielded the lowest MSE for high AUC. While, the product method has the lower bias but a similar MSE with IU and Euclidean. The IU method produced the least MSE and bias that were followed by Euclidean and Product method. The results of coverage rate with homoscedastic binormal data, indicated that the four popular methods had a similar coverage rate of CI for true parameter of cut point ranging from 94 to 99% for equal and unequal sample size depending on AUC and sample size used but DOR had very poor coverage rate which none of CI did cover the true value of cut point at all. However, the IU methods had the smaller mean length of CI but the DOR had the highest value of mean length of CI (see Appendix in Table 1.b and Table 2.b).

Table 2 The MSE, bias and relative bias of different methods of optimal cut-point selection when data generated under binormal distributions with heteroscedastic variance and equal/unequal sample size from diseased (D) and non-diseased (ND) distributions according to degree of overlap/accuracy

Bi-gamma distributions

Table 3 presents the findings when data was generated by very skew pairs of distributions of Gamma with equal and unequal sample size. For balanced sample size, the least bias was found by Euclidean and it was followed by Product method, the Youden index, IU, and DOR but among the four popular methods, the greatest MSE was found by the Youden and the least by the Euclidean. For unequal sample size, similarly the greatest bias and MSE were attributed to IU and Youden respectively among the four methods while the Youden index had the least biased. The Euclidean is more precise than Youden index but the Youden was less biased. Similar to other pairs of distributions, the worst appraisal was found by DOR with extremely high MSE, bias and relative bias. The coverage rate of bootstrap CI for true values of cut point ranging from 95 to 98% have been observed using the three methods of Youden, Product, and Euclidean. However, the coverage rate of CI was declined from 77 to 89% for IU method. Meanwhile none of bootstrap CI did cover the true cut point by DOR method (see Table 3.b in Appendix).

Table 3 The MSE, bias and relative bias of different methods of optimal cut-point selection when data generated under bigamma distributions with equal/unequal sample size from diseased (D) and non-diseased (ND) according to degree of overlap/accuracy

Bi-exponential distributions

Table 4 shows the appraisal findings with data of an extremely skew distributions of exponential pairs. The lowest bias was found by the Euclidean that followed by Product, IU, Youden and DOR respectively. The Euclidean has the lowest MSE that were followed by product, IU and Youden index. The Youden had the high MSE but IU had low MSE at low AUC. For unequal sample size, the lowest bias and MSE were observed by the Euclidean method. Similar to other scenarios, an extremely high MSE, and bias were found by DOR for all combinations. Overall, the extremely deviation of binormality, the bias, and MSE of all methods substantially increased. In this case, the least bias and MSE were observed by Euclidean. Moreover, the three methods of Youden, Euclidian, and Product had the high coverage of CI ranging from 95 to 98% while the IU produced the lower coverage rate ranging from 79 to 91% depending on the AUC and sample size used. Meanwhile, the poor performance of DOR has been observed in terms of coverage rate and mean length of CI (see Table 4.b in Appendix).

Table 4 The MSE, bias and relative bias of different methods of optimal cut-point selection when data generated under biexponential distributions with equal/unequal sample size from diseased (D) and non-diseased (ND) according to degree of overlap/accuracy

Illustrations and applications with clinical examples of data

Data

In a case control study of pregnant women in the third trimester of pregnancy suspected of having PROM were included in the study. These pregnant women were referred to the emergency obstetrics and gynecology clinic of Ayatollah Rouhani hospital in Babol, the north of Iran [19]. Based on the gold standard test status, 60 cases with PROM and 60 healthy individuals without PROM were diagnosed. Briefly, first, the informed consent was obtained from all patients. The full description of inclusion and exclusion criteria were described elsewhere [19]. Pregnant women who were diagnosed as negative in one of two gold standard tests of pooling or nitrazine were excluded from the study as suspicious subjects. Mothers who tested positive for both of these two tests were diagnosed with definite PROM (n = 60), and those who tested negative in both tests were considered as true negative (n = 60). The three biomarkers of BHCG, BUN, and Cr, by enzymatic photometry and Jafee methods and the results were recorded in PROM diagnostic database.

Results of cut-point selection of biomarkers

Figure 2 depicted the density function of three biomarkers in pregnant women with and without PROM in panel A, B, and C. The results Wilcoxon rank test showed that the values of biomarkers are significantly higher in PROM than without PROM (P = 0.001) and a higher SD of biomarkers were observed in PROM patients. Figure 3 shows the nonparametric ROC curve for BHCG, BUN, and Cr for diagnosis of PROM with a high diagnostic accuracy. In Fig. 3, the highest AUC (AUC = 0.992, 95%CI: 0.963, 0.998) was found by BHCG and followed by BUN (AUC = 0.975, 95%CI 0.929, 0.991), and Cr (AUC = 0.954, 95%CI 0.904, 0.978). In Table 5, the results show that for BHCG, and BUN, the five methods produced the identical cut-points (BHCG (44 IU/L), and BUN (1.07 mmol/L) while for CR, the DOR resulted in an extremely higher value of cut-points (40.66 µmol/L) with low sensitivity but the cut-point selection of the four other methods are identical (21.22 µmol/L). Figure 4 shows the changes in five metrics of cut-point selection over various cut-off values by different methods that have been shown with different colors in three panels for BHCG, BUN, and Cr.

Fig. 2

The density plot of three biomarkers of BHCG, BUN, and Cr in healthy and PROM groups

Fig. 3

Empirical ROC curve of three biomarkers of BHCG, BUN, and Cr in diagnosis of PROM

Table 5 The diagnostic performance of cut-point of three biomarkers in diagnosis of PROM by different methods

Fig. 4

The changes of five metrics of cut-off selection versus various cut off values of biomarkers BHCG, BUN, and Cr in diagnosis of PROM in panels of A, B and C, respectively

Discussion

Our findings show that the IU method has the lowest bias, relative bias and MSE than other methods when data are generated from binormal model but not for a highly skew distribution of bigamma and biexponential pairs. The part of results related to the pairs of binormal model are in accordance with those reported by Liker Unal [7]. However, we found the poor performance of IU methods when data is generated from bigamma and biexponential pairs that was highly skewed. The IU may have a clinical interpretation in diagnostic appraisal. It simultaneously minimizes the difference between Se and Sp and also minimize the difference of either Se or Sp with AUC. This property might be clinically interesting in terms of diagnostic accuracy for cut-point selection.

On the other hand, the most popular method of Youden index that has a greater clinical interpretation in terms of net classification, the corresponded cut point is less precise especially for low and moderate AUC even under binormal data with equal sample sizes but nor for high AUC, and its bias and MSE are almost higher than Euclidean and product methods. The diagnostic performance of these two latter methods outrages than others with highly skewed distributions of diagnostic test results. While the product method maximizes the product of Se and Sp that might be interested clinically. Based on our findings, the more precise estimate of cut-point is estimated by Euclidean. These results are also in accordance with other reports [7, 8, 10]. Despite the higher precision and less biased of the Euclidean in some scenarios, it has less clinically interpretations.

Among the five methods of cut-point selection in this study, the worst method was DOR in term of extremely high bias and MSE and very low performance of coverage of CI for true cut point in all configurations of distributions studied. In a population-based distributions under different scenarios, it has been reported by Hajian-Tilaki [6] that DOR produced unexpected high cut-point with poor Se because the convex pattern of DOR as ratio metrics [22]. Even under the bilogistic model DOR metric might be noninformative or have a linear trend that has not produced a proper optimal cut-point [6].

As one expected, in our simulation, the bias and MSE of all methods except for DOR, declined with increasing sample sizes and the higher degree of accuracy. From statistical perspective, the amount of data in term of sample sizes provides the more precise estimate and also less biased estimates of cut point. Meanwhile the high degree of separation pairs of distribution (or high AUC) leads to less room for sampling variability in ROC space. Therefore, the more precise estimates of cut-points are estimable in this scenario as our finding demonstrated. In particular, the results of current study showed the higher precise estimates and less biased with a high AUC by the Youden index.

Moreover, we found, the inconsistency in determining true cut-points by different methods in particular with highly skew pairs of distributions. With binormal pairs and homoscedastic variance, the consistency of true cut-points values is possible but not for unbalanced variance and by the DOR. However, in analysis of our clinical example of data of biomarkers for diagnosis of PROM, surprisingly, identical results of cut-point were observed by five different methods for BHCG and BUN with AUC = 0.992 and AUC = 0.975 respectively. These identical results of all investigated methods can be explained by very high diagnostic performance of these two biomarkers. In this scenario, with extremely high diagnostic accuracy, there are less room in ROC space for variation of cut-points by different methods. In contrast, the inconsistency of cut-point by DOR with other methods in our simulation was present because the highest AUC in our simulation was considered as AUC = 0.90 but in our clinical example of detection PROM, the AUC for these two biomarkers were greater than 0.95. While for Cr that its diagnostic performance was lower than BHCG and BUN, the estimated cut-point of Cr by DOR was substantially higher than other methods with low performance of Se. Overall, the four competitive methods yielded identical results of cut-point for Cr as well but not DOR.

To our best knowledge, the design and results of the current simulation study are novel in terms of different configurations of distributions of diagnostic test results. So far, the other published simulation studies have not included the more extreme skew distributions with different degrees of overlapping pairs with five different methods simultaneously as we studied. Further simulation studies with other pairs of distributions may need to explore the performance of the different methods in other conditions.

Conclusion

Despite the clinical interest property of the Youden index, it may not produce a more precise estimate of the optimal cut-point for severe departure from binormality, in particular for low and moderate AUC. The greatest deviation from binormality, the bias and MSE increased substantially in all methods. In a case, data generated from very skewed distributions of bigamma and biexponential, the lowest bias and MSE resulted from the Euclidean index and the highest yielded by DOR and IU, and Youden respectively. The precision and bias in estimating cut points by different methods may depend on the underling distributions of test results and AUC s, and the sample size used. However, the DOR has an extremely poor performance with very high bias and MSE, and very low coverage rate.

Limitations

The various methods for determining optimal cut-points optimize different objective functions and they have their own true cut-points. In many cases, the choice of the objective function is understood to depend on the specific purpose of the study either may focus on more weighting sensitivity or specificity, or the cost of false positives and false negatives or to maximize the sensitivity at a given value of the specificity. The objective function that has been defined as a criterion for cut-point selection has been criticized in the literature [23]. Moreover, our simulation was limited to the prevalence of 0.20, 0.25, 0.33, and 0.50 for diseased based on the ratios of sample sizes were considered in the study. In practice, the prevalence might be less than 0.20. However, the classical accuracy-based methods of cut-point selection are not influenced by the prevalence of disease, whereas its diagnostic performances as positive predicted value and negative predicted value are affected.