Technical Note pubs.acs.org/jpr
PanelComposer: A Web-Based Panel Construction Tool for Multivariate Analysis of Disease Biomarker Candidates Seul-Ki Jeong,*,† Keun Na,‡ Kwang-Youl Kim,§ Hoguen Kim,⊥ and Young-Ki Paik*,†,§ †
Yonsei Proteome Research Center and Biomedical Proteome Research Center, ‡Graduate Program in Functional Genomics, Department of Biochemistry, Department of the Integrated Omics for Biomedical Science (World Class University Graduate Program), and ⊥Deptartment of Pathology, College of Medicine, Yonsei University, Seoul 120-749, Korea §
S Supporting Information *
ABSTRACT: Measuring and evaluating diagnostic efficiency is important in biomarker discovery and validation. The receiver operating characteristic (ROC) curve is a graphical plot for assessing the performance of a classifier or predictor that can be used to test the sensitivity and specificity of diagnostic biomarkers. In this study, we describe PanelComposer, a Web-based software tool that uses statistical results from proteomic expression data and validates biomarker candidates based on ROC curves and the area under the ROC curve (AUC) values using a logistic regression model and provides an ordered list that includes ROC graphs and AUC values for proteins (individually or in combination). This tool allows users to easily compare and assess the effectiveness and diagnostic efficiency of single or multiprotein biomarker candidates. PanelComposer is available publicly at http:// panelcomposer.proteomix.org/ and is compatible with major Web browsers. KEYWORDS: biomarker evaluation, protein expression, multivariate analysis, panel, bioinformatics
■
INTRODUCTION
and, therefore, is often used to compare the performance of two or more tests.5,6 Single component (protein) biomarkers have been widely used for disease diagnosis (e.g., α-fetoprotein for hepatocellular carcinoma (HCC) and prostate-specific antigen for prostate cancer); however, some limitations exist in their performance and applicable range in providing accurate assessments of disease states. The combination of several proteins into a biomarker panel has been proposed as a means to overcome these limitations, as this strategy may increase the diagnostic efficiency compared to that of a single protein.7−10 Several different methods have been reported for determining biomarker panel composition, such as logistic regression,4,11,12 random forest,13−15 and machine learning algorithms.16,17 To combine multiple proteins into a single panel, we employed a logistic regression model because it is easy to interpret in its linear form.18,19 Unlike other methods, logistic regression also does not require the data set to be normally distributed or linearly related, nor does it require equal variance within each group.18 Further, logistic regression models can be easily updated when new or additional data sets become available.19 Logistic regression can model numerical or categorical values into a single variable ranging from 0 to 1
A disease biomarker is defined as a molecular signature that reflects different stages of disease before or after treatment. Biomarkers are often used to monitor prognosis during treatment.1 The goal of biomarker discovery is to develop non-invasive tests that permit early disease detection, patient classification, and disease progression or recurrence monitoring.2 To better identify biomarkers, the efficacy of a differentially expressed protein in patients with a specific disease compared to that in healthy individuals should be thoroughly evaluated. In addition, the diagnostic performance of the differentially expressed proteins should be compared to that of other known marker proteins.3 The receiver operating characteristic (ROC) curve is generally used as a statistical method to assess the performance of a binary classifier that can distinguish two categories, such as disease versus nondisease or moderate versus severe disease states.4 The ROC curve is created by plotting the sensitivity of a test versus its specificity and can illustrate the efficiency of a test at various cutoff points. ROC curves have also been shown to be useful for determining the optimal threshold for a given test. For classifiers or biomarkers, the threshold can be defined as a criterion to distinguish one group from another. The area under the ROC curve (AUC) is the most widely used measure of biomarker performance. The AUC allows assessment of the performance of each test by means of a single measurement © 2012 American Chemical Society
Received: May 12, 2012 Published: November 9, 2012 6277
dx.doi.org/10.1021/pr3004387 | J. Proteome Res. 2012, 11, 6277−6281
Journal of Proteome Research
Technical Note
Figure 1. PanelComposer interface. (1) Submit datasheet: expression data arranged in a compatible format is submitted. (2) Select positive and negative groups. (3) Set cross-validation method. (4) Compute ROCs, AUCs, and p values. (5) AUC and ROC results: results are displayed in a table and ROC graph. (6) Combine proteins in panel: two or more proteins can be easily combined to determine their effectiveness in a panel. (7) Perform a different analysis: positive and negative groups can easily be redefined to yield updated ROC graphs for the selected biomarker candidates.
logistic regression, and ROC analyzer). After the logistic regression model was used to combine multiple biomarkers into one panel, a stepwise selection method to automatically find an optimal biomarker panel was implemented using the “glm” module of the R software package (version 2.9.2). A Javabased (version 1.6.0) program was written to plot ROC curves, calculate the respective AUC values, perform cross-validation (CV), and obtain other statistical results. Details on these methods related to data acquisition, software implementation, and data analysis and the corresponding data sets are described in Supporting Information.
to evaluate the performance of a panel of several candidate proteins by using the ROC methodology.9,10,20 In our previous work, we experienced many difficulties in statistical analyses related to biomarker panel construction.21,22 Several ROC software packages are available; however, they are not freeware,23−25 easy to use,25 or capable of analyzing panels.25,26 In this paper, we present PanelComposer, a novel Web-based tool for composing biomarker panels. We demonstrate its usefulness for evaluating candidate biomarkers that can distinguish HCC from other liver-related diseases as a sample case. PanelComposer both utilizes statistical results from proteomic expression data and validates biomarker candidates based on ROC curves and AUC values by using a logistic regression model.
■
METHODS We analyzed a sample data set containing expression data obtained by multiple reaction monitoring (MRM) for vitamin D binding protein (VDBP), ceruloplasmin (CP), and apolipoprotein A-1 (ApoA1). We used human plasma isolated from healthy controls and patients with various disease conditions (liver diseases, HCC, and other cancers; see Supplementary Table 1 for more details). MRM analysis was performed using Acquity UPLC (Waters, Milford, MA) and 4000 QTRAP LC-MSMS system (AB-SCIEX, Framingham, MA) following the MIDAS data acquisition workflow.27 Quantitative analysis of MRM data was performed using MultiQuant (AB-SCIEX, version 1.1). PanelComposer was implemented on a three-tiered architecture (i.e., Web interface,
PanelComposer performs several steps when analyzing expression data to assess the effectiveness of a biomarker candidate (Figure 1). First, expression data and supplementary information including gene or protein names and diagnostic categories are submitted to the analysis. Expression data must be input in a comma-separated values (CSV) file format satisfying some requirements (see Supplementary Figure 1). Next, the user designates positive and negative categories, according to the disease state. Examples of positive and negative groups are disease versus nondisease, cancerous versus normal samples, or moderate versus severe disease onset. More than one category can be selected as positive or negative (e.g., normal, hepatitis, and cirrhosis can all be selected as negative and HCC chosen as positive). Third, for the internal validation, the user can select one of three CV methods. ROC graphs are
■
RESULTS AND DISCUSSION
Data Analysis Flow and User Interface
6278
dx.doi.org/10.1021/pr3004387 | J. Proteome Res. 2012, 11, 6277−6281
Journal of Proteome Research
Technical Note
Table 1. Representative Cases of the PanelComposer Application protein name casea case A
case B
case C
category
ApoA1
VDBP
CP
ApoA1/VDBP
ApoA1/VDBP/CP
99%
0.03 0.669 (0.663, 0.675) 4.56 (4.08, 5.03)
0.085 0.634 (0.627, 0.640) 2.73 (2.48, 2.98)
0.171 0.606 (0.600, 0.612) 12.2 (11.83, 12.57)
