Name
That Peptide When relying on software to identify tens of thousands of peptides, how can you be certain that all of the answers are correct? Katie Cottingham
B
y now, most of us have glanced at a few proteomics papers. The details behind the experiments may vary, but there’s typically a table or two listing many, many proteins that the researchers have discovered in a particular organism, tissue, or cellular process. Supplementary information on the journal’s website often includes even longer tables that didn’t fit into the published article. There’s no question that these lists are impressive. But can anyone be absolutely certain that all of those proteins have been correctly identified? Data validation, according to Akhilesh Pandey of the Johns Hopkins University School of Medicine, is a difficult issue in the proteomics field. “In many of these instances, no one knows the real truth because that’s what you’re really trying to find,” he says. “The gold standard is missing.” A strategy known as shotgun proteomics, or multidimensional protein identification technology (MudPIT), is currently the dominant proteomics method. A complex mixture of proteins is digested into peptides by a proteolytic enzyme, typically trypsin. The peptides are then subjected to many rounds of LC, followed by tandem MS. Information from the resulting mass spectrum is fed into a search algorithm that combs a database to find the peptide to which the spectrum corresponds. Peptide information is used to determine the proteins in the original sample. But just because two researchers follow the general MudPIT protocol doesn’t mean that they will arrive at the same final list of proteins for a given sample. “People use different instruments, different approaches to the analyses, and then different criteria for the searching,” says Stan Hefta of Bristol-Myers Squibb. Varying any of these factors could lead to discrepancies between experiments on identical samples.
© 2004 AMERICAN CHEMICAL SOCIETY
M A R C H 1 , 2 0 0 4 / A N A LY T I C A L C H E M I S T R Y
95 A
One size fits all? Proteomics researchers use different instruments, experimental protocols, databases, and software. Why not standardize at least one link in the chain? Most researchers balk at the idea, saying that it would never work or that it is simply not good for science. “I think that by setting standards, what you wind up doing is killing innovation,” says John Yates of The Scripps Research Institute. To facilitate comparisons of proteome analyses from different laboratories, the Human Proteome Organization (HUPO) agreed that researchers participating in its liver, brain, and plasma proteome projects should use a specific database, the International Protein Index (www.ebi.ac.uk/IPI/IPIhelp.html). But HUPO has stopped short of a mandate. Rolf Apweiler of the European Bioinformatics Institute (U.K.) and director of HUPO’s Proteomics Standards Initiative (PSI) says, “We don’t want to force people to do something in a special way because there are usually very good reasons to do it a bit differently for experimental purposes.” Instead, PSI’s goal is to develop a data-capturing format to standardize the reporting of experimental parameters. Several organizations and companies are currently collaborating on the project. The National Institutes of Health (NIH) recently announced its intention to explore the topic of proteomics standards. According to the NIH Roadmap document, which was released in October 2003, NIH will hold workshops on standards for proteomics quality and data. Discussions will include NIH scientists, program staff, and outside researchers. At the moment, it is unclear how far NIH intends the standards to reach. But even if standards are recommended by HUPO or NIH, it doesn’t mean that all researchers will go along with them. Some researchers complain that the HUPO data-capturing format will be far too cumbersome for practical use, and they say attempts to specify one type of proteomics protocol or database searching algorithm is never going to work. Only time will tell if the scientific community embraces or ignores these initial steps.
Peptide identification software is an insidious source of variation, according to some proteomics researchers. Experts say that such software can provide different identities for the same peptide. Although sequence redundancy can explain some of the differing answers, other identifications are simply incorrect. Dave Speicher of the Wistar Institute says, “The programs err on the side of identifying things that might be wrong rather than being very conservative.” Several database searching algorithms are available. The most widely used programs are Mascot and Sequest; typically, a lab will only use one of them. “They’re pretty expensive, so that’s probably why people don’t have the luxury of using them both,” says Andrew Keller of the Institute for Systems Biology. “Also, that would lengthen the search time if you search everything twice.” Because researchers have limited access to these tools, however, little is known about this problem, says Pandey.
Sources of error Although a high degree of overlap between identifications from various algorithms is typical, there may be discrepancies in certain instances, acknowledges John Yates of The Scripps Research Institute. “Different algorithms have different selectivities—different aspects of them that make them work better on some types of spectra than other types of spectra,” he says. Spectral features also influence the quality of an identification. “Often, low-scoring peptides are low-scoring because they have low signal to noise or they have incomplete fragmentation,” says Yates. “There are lots of reasons why they get low scores— primarily, it’s because they’re bad spectra.” Another problem occurs when researchers do not specify the proteolytic enzyme used in an experiment, says Natalie Ahn of the 96 A
A N A LY T I C A L C H E M I S T R Y / M A R C H 1 , 2 0 0 4
Howard Hughes Medical Institute and the University of Colorado. This information is sometimes provided to account for phenomena such as nonspecific proteolysis or in-source fragmentation, in which a protein fragments in the ionization source of the mass spectrometer. A peptide generated by in-source fragmentation does not have the typical ends that would result from an enzymatic cleavage. Failure to specify which enzyme was used can increase the effective database size that is searched by an algorithm, which increases the number of incorrect peptide identifications, according to unpublished work from Ahn’s laboratory. In some cases, conflicting answers arise simply because of the different ways in which the programs work. For instance, Yates has recently developed a probability-based program that measures the frequency of fragment ions in a database. “That’s a different kind of measurement than what Sequest does, which is a comparison between the predicted fragment ions and the fragment ions observed in the mass spectrum,” he explains. Thus, to completely understand the search results, one must understand how a particular algorithm generates data, and some groups lack this understanding, he says.
The real McCoy? Researchers contend that validation of peptide identifications is important, even if errors occur infrequently. William Hancock of Northeastern University says that new, improved versions of algorithms that are currently available have error rates of ~0.1–1%. “So in 500 identifications, there may be as many as five wrong,” he points out. “The problem is, you don’t know which five.” In the past, a trained researcher would check the peptide search results against each spectrum that was generated. Despite the human bias in this type of validation, it provided a way
for an experienced researcher to gain some measure of confidence in the identification. Yates emphasizes that experience is the key for manual validation to be successful. “If one takes the time to learn how to interpret spectra, one can improve their confidence,” he says. But these days, proteomics has picked up the pace and highthroughput is the big buzzword. “Everybody wants to do tens of thousands of peptides,” says Pandey. Although Pandey’s own team would like to manually inspect each spectrum, he says that there is no way to do it in a high-throughput scenario. In fact, mass spectrometers in Speicher’s laboratory run 24 hours a day, 7 days a week. Both Pandey and Speicher agree that the proteomics field has become very dependent on database searching algorithms. “One would just like to hope that [the algorithms are] doing the right thing,” says Pandey. According to some researchers, manual validation still has merit. Hefta says you can’t take the human out of the equation. “At some point in time, if you’re going to be making business decisions based on that mass spec result, you’re going to have to hand-curate it,” he says. “You’re going to have to really look at the fragment ions and make sure that you agree with the result of the searching.” At Bristol-Myers Squibb, Hefta and his colleagues have developed their own algorithm for validating peptide identifications. After running their data through this additional program, the researchers manually validate those spectra that the software has flagged as correctly identified. Keller and his collaborator Alexey Nesvizhskii (also at the Institute for Systems Biology) have developed a statistical model called PeptideProphet for validating search results. To arrive at a convenient summary score for the model, Keller and Nesvizhskii trained it on a set of peptides derived from 18 highly purified proteins. After the initial training step, Keller says the model examines each data set anew. “It makes an evaluation and computes the probabilities based upon what it observes for each data set,” he explains. “That ensures that the model computes accurate probabilities despite variations in spectrum quality, spectrometer mass accuracy, and database size.” The researchers also say that applying PeptideProphet to data generated by different search algorithms can provide a basis for comparing results. But Yates finds this model lacking. “I’m not enthusiastic about empirical applications for generating statistical values for your data,” he says. “I think the statistics change depending on the experiment that you do, the database that you look at, [and] the size of your data set.” Yates has an upcoming paper in Analytical Chemistry (doi:10.1021/ac035112y) on his own model, but he would not reveal any details. Perhaps the most common way of establishing validity is to report the threshold values used in each experiment. “[Researchers] choose a score above which they believe the assignment, below which they don’t believe the assignment,” explains Ahn. But she
warns that a lot of good data are lost when these thresholds are set too high. “It forces you to just sample the spectra that usually look good and clean, which [are] usually [those of ] the higher abundance peptides,” she says. Thus, thresholds can introduce bias, correctly identifying peptides for only certain types of spectra. Although Yates concedes that some correct identifications could fall below the set threshold values, he says you still don’t know which is the right answer. “While you may be throwing out some right answers [using thresholds], you may also be keeping some incorrect answers,” he says. Keller suggests that this is where a statistical model comes in handy. When he and Nesvizhskii looked at thresholds with their control data set, they found a lot of false positives. “That’s another advantage to using PeptideProphet . . . actually being able to estimate those error rates in your data set in a truthful manner,” Keller says.
Do your biology! Some proteomics researchers say that regardless of what thresholds or models you use, the ultimate validation tool is biological characterization. “No matter how good you make the mass spec, you’re going to have to go and do some biology—get your hands dirty,” says Hancock. He explains that after all the mass spectral data are in hand, researchers should just select the candidates in which they are most interested for further biological analysis. Hefta says, “You simply cannot rely exclusively on mass spectrometry to make [business] decisions.” For Bristol-Myers Squibb researchers, MS is a filter that helps them focus on a few interesting drug targets or biomarkers of disease. “We’re always going to take a separate approach, generally involving an antibody, an ELISA, or Western [blot], or some other biochemical characterization before we take something forward,” he says. Yates agrees that validating a peptide identity on the basis of MS alone is not sufficient. “You haven’t gone back and shown that these proteins are indeed involved in the processes that you think they are involved in,” he says. “It’s beyond just validating that your data analysis is correct; it is confirming the biology your proteomics data has uncovered.” The true test of a peptide or a protein’s identity does not rest solely on the statistics of a database searching algorithm, say researchers. Biological evidence provided by methods such as immunofluorescence, co-immunoprecipitation, or RNA interference can help pinpoint a protein’s location or function when combined with MS/MS data. Clearly, says Pandey, this is a field that is ripe for multidisciplinary teams. Statisticians, computer scientists, biologists, and chemists all must work together to be able to “name that peptide” with confidence. Katie Cottingham is an associate editor of Analytical Chemistry. M A R C H 1 , 2 0 0 4 / A N A LY T I C A L C H E M I S T R Y
97 A
