data shop
Comparing MS/MS search engines C
harged with the goal of cataloging serum and plasma proteins, the organizers of the Human Proteome Organisation’s Plasma Proteome Project (PPP) collected results from 48 laboratories around the world. But researchers did not follow one master protocol. They ran PPP-approved samples on various types of MS instruments with different settings and analyzed the results with different search algorithms. When it was time to make sense of all the data, “what we were really confronted with was: How are we going to be able to compare the data if there’s no standardization?” says Richard Simpson at the Ludwig Institute for Cancer Research (Australia). Thus, Simpson’s group and collaborators at Agilent Technologies, GE Healthcare, the Institute for Systems Biology, Pacific Northwest National Laboratory (PNNL), and the University of Michigan Medical School attempted to get a handle on one of the variables: search algorithms (Proteomics 2005, 5, 3475–3490). Four of the research groups independently analyzed the same LC/MS/ MS data set, which had been generated by the PNNL researchers with an ion trap instrument. Sequest, Mascot, Spectrum Mill, Sonar, and X!Tandem were applied to the data. Three groups used algorithms that they had developed in their own laboratories, setting the parameters to the best values for each algorithm. In addition to analyzing the data with both Sequest and Mascot, Simpson’s group pooled the data and compared the algorithms’ performances. The researchers benchmarked the algorithms by compiling a “truth list” of correctly identified spectra, regardless of threshold values. Sequest correctly identified 526 peptides and was the most © 2006 AMERICAN CHEMICAL SOCIETY
sensitive program. Mascot, however, was the most specific program because it identified the highest number of peptides at a specified false-positive rate. When the researchers constructed plots to compare the sensitivities and speci-
What we were really confronted with was: How are we going to be able to compare the data if there’s no standardization? —Richard Simpson
ficities of the programs, they found that the PeptideProphet rescoring algorithm enhanced Sequest’s performance. On the basis of the truth list, Simpson’s group also compared thresholds for each program at different false-positive rates and found that using a reversedsequence database resulted in similar, but slightly higher, thresholds than using only a forward-sequence database. The size of the searched database was also evaluated. “The larger the database, the better and cleaner the spectrum you have to have to be able to find that peptide,” says Eugene Kapp, who is a researcher in Simpson’s laboratory. “Otherwise, [the algorithm] gets distracted by all of the other peptides [it] is having
to consider.” He says that although all of the algorithms were affected, probabilistic algorithms, such as Mascot, were affected the most because they take into account the database size when performing searches. Kapp explains, “It’s a balancing act choosing the appropriate database. Bigger is better, however, and we would rather sacrifice a few weak spectra for completeness.” Katheryn Resing at the University of Colorado, Boulder, applauds the study’s design because “the people who were best able to run the particular software were asked to run it, and the results were interpreted by a group that doesn’t have a commitment to any of the software programs.” Richard Johnson at Homestead Clinical Corp. and Viral Logic Systems Technology points out, however, that setting different parameters for each algorithm could make it difficult to accurately compare the programs. Johnson also says that using a sample with unknown components, such as plasma, is not the best way to evaluate search algorithms. Both Resing and Johnson suggest adding freely available programs, such as the Open MS Search Algorithm, to future analyses. Simpson and Kapp plan to reevaluate algorithms annually, including new programs and data produced by new instruments. Kapp says, “Next time around, we will have to compare the different probabilities [reported by newer algorithms], but this [initial study] is a step in the right direction, which ultimately should provide a framework for people to share data and to allow data to be looked at in a transparent manner.” The results of the current analysis are freely available at www.ludwig.edu. au/archive. a —Katie Cottingham
F E B R U A R Y 1 , 2 0 0 6 / A N A LY T I C A L C H E M I S T R Y
643
