Article pubs.acs.org/jcim
GRID-Based Three-Dimensional Pharmacophores II: PharmBench, a Benchmark Data Set for Evaluating Pharmacophore Elucidation Methods Simon Cross,*,† Francesco Ortuso,‡ Massimo Baroni,† Giosuè Costa,‡ Simona Distinto,‡ Federica Moraca,‡ Stefano Alcaro,‡ and Gabriele Cruciani§ †
Molecular Discovery Limited, 215 Marsh Road, Pinner, Middlesex, London HA5 5NE, United Kingdom Laboratory of Computational Medicinal Chemistry, Department of “Scienze della Salute”, University “Magna Græcia” of Catanzaro, Viale Europa, Loc. Germaneto, 88100 Catanzaro, Italy § Laboratory for Chemometrics and Cheminformatics, Chemistry Department, University of Perugia, Via Elce di sotto 10, I-06123 Perugia, Italy ‡
ABSTRACT: To date, published pharmacophore elucidation approaches typically use a handful of data sets for validation: here, we have assembled a data set for 81 targets, containing 960 ligands aligned using their cocrystallized protein targets, to provide the experimental “gold standard”. The two-dimensional structures are also assembled to remove conformational bias; an ideal method would be able to take these structures as input, find the common features, and reproduce the bioactive conformations and their alignments to correspond with the X-ray-determined gold standard alignments. Here we present this data set and describe three objective measures to evaluate performance: the ability to identify the bioactive conformation, the ability to identify and correctly align this conformation for 50% of the molecules in each data set, and the pharmacophoric field similarity. We have applied this validation methodology to our pharmacophore elucidation method FLAPpharm, that is published in the first paper of this series and discuss the limitations of the data set and objective success criteria. Starting from two-dimensional structures and producing unbiased models, FLAPpharm was able to identify the bioactive conformations for 67% of the ligands and also to produce successful models according to the second metric for 67% of the Pharmbench data sets. Inspection of the unsuccessful models highlighted the limitation of this root mean square (rms)-derived metric, since many were found to be pharmacophorically reasonable, increasing the overall success rate to 83%. The PharmBench data set is available at http://www.moldiscovery.com/PharmBench, along with a web service to enable users to score model alignments coming from external methods in the same way that we have presented here and, therefore, establishes a pharmacophore elucidation benchmark data set available to be used by the community.
■
INTRODUCTION In part I of this paper,1 we introduced a new pharmacophore elucidation approach FLAPpharm, that is based on GRID molecular interaction fields (MIFs),2 and also the derivative FLAP approach3 for molecular alignment. FLAPpharm attempts to find the best common superposition of active ligands, without being dependent on any one of them as a template, and by using each ligand’s MIFs to drive the alignment. The resulting alignment model is then used to derive the common pharmacophore, unlike classical pharmacophore approaches where rule-based features are extracted, common features found, and then these are used to drive the alignment. The common pharmacophore extracted by FLAPpharm is a pharmacophoric “pseudomolecule”, which consists of common pharmacophoric interaction fields (PIFs), common atom-centered pharmacophoric pseudofields (pseudoPIFs), and common pharmacophoric points at the centroid of these pseudoPIFs. These three entities directly correspond to the MIFs, pseudoMIFs, and atoms in a ligand; therefore, the FLAPpharm pseudomolecule can be used just as another molecule in FLAP; it can be used as a template for ligand-based © 2012 American Chemical Society
virtual screening or it can be docked into a receptor by FLAP to help validate or disprove the hypothesis. In part I,1 we validated this approach in several ways. First (and primarily), we used the data sets previously reported by Patel et al.4 to gauge how successful the method was at reproducing the target pharmacophore, and in all cases obtained extremely satisfying results. Second, we used the DUD data set5 to test how discriminatory the method was in terms of virtual screening, after automatically constructing pharmacophore hypotheses from the DUD chemotype cluster centroids identified by Good;6 on average, the FLAPpharm approach (starting without three-dimensional information about the ligands or target) performed better than any of the other approaches we previously tested7 (including those using known three-dimensional information about the ligands and target). Finally, we illustrated the use of a FLAPpharm model in explaining the alternative binding modes available to factor Xa, by docking the pharmacophoric pseudomolecule into the receptor. The Received: March 22, 2012 Published: September 12, 2012 2599
dx.doi.org/10.1021/ci300154n | J. Chem. Inf. Model. 2012, 52, 2599−2608
Journal of Chemical Information and Modeling
Article
a ligand. In this way the pdb structure files were obtained for each target. The Ligand Expo database22 of all small molecule structures present in the protein data bank was downloaded, and for each of the targets in our data set, the small molecules present in the files were analyzed and the target structure was filtered if the ligand did not match the following constraints: molecular weight greater than 100 and less than 500; number of rotatable bonds less than 12; number of hydrogen bond donors less than or equal to 5; number of hydrogen bond acceptors less than or equal to 10; logP less than 5.0. The remaining target structures therefore contained small molecule ligands that were “drug-like”. Small molecules were also filtered if their incidence in the Ligand Expo database was higher than 20, as an attempt to automatically remove cofactors and other small molecules that are not relevant. For each target set, the structure files were also removed if the resolution of the PDB structure was greater than 2.5 Å, to leave only higher resolution structures. As an additional filter, we removed structures that did not have electron density deposited at the Uppsala EDS server23 to ensure that if needed, the original experimental data could be queried. As a final filter, each target was filtered unless the number of unique ligands was greater than or equal to three. This left a total of 94 targets to analyze in more detail by manual inspection. To do this, for each target, the protein structures were first aligned using the CE algorithm as implemented in PyMOL,24 the small molecules extracted, and the atom-typing from the Ligand Expo database mapped onto the aligned PDB structures of the ligands to provide the aligned ligand sets for each target. The ligands were then corrected by hand, with particular respect to tautomerism and protonation. Duplicates were removed, and any other structures where there were problems with chemistry or the alignment. Given that some ligands had been removed, we filtered targets that no longer had at least three unique ligands. Finally, we added the five targets from the Patel data set, and a set of high resolution DNA major groove binders given our interest in this area. This yielded a final data set of 81 targets containing 960 ligands, taken from high resolution crystallography data, and aligned by the target protein structure, which we will henceforth refer to as Pharmbench. All ligands were also converted to their equivalent two-dimensional structures using the dbtranslate utility in SYBYL X1.3,25 to remove any bias when using them as input to a pharmacophore elucidation program. Our aim is that this will become a publicly available resource, downloadable at www. moldiscovery.com/PharmBench, to be maintained and updated over time as new structural data becomes available, to enable molecular alignment and pharmacophore elucidation approaches to be compared. Naturally, being version 1.0 of the benchmark set, there are some limitations. The most obvious of these is that currently no diversity filtering has been applied to the PharmBench data sets; therefore, it is possible for there to be “easy” cases where simple analogues are present. However, at this stage, we decided to retain all of the experimental information that passed our preliminary filters and try to set up an objective measure of success that can be used by the community.
“chloro-binding” mode, which was a significant discovery in that it allows the design of neutral and hence potentially more pharmacokinetically desirable compounds,8 was predicted as the third docking pose. Therefore we believe that FLAPpharm shows great promise and, as discussed in part I, does not suffer from the disadvantages of the classical feature-based methods. Among the most widely known of these feature-based methods are Catalyst/HipHop,9 GASP,10 and the more recently described LigandScout,11 PharmID,12 and PHASE.13 The first of these (Catalyst/HipHop) was first published in 1996 and validated using three data sets. The second was first published in 1995 (GASP) and validated using eight data sets. PharmID (2006) was validated using two data sets. LigandScout (2005) differs in that it was introduced as a structure-based pharmacophore method and two validation examples were used, although a follow-up paper examined five case studies.14 The current version of LigandScout also includes a ligand-based approach.15 Recently another computational approach based on the generation of pharmacophore models by GRID maps of complexes (GBPM) was described by some of us, validated in two systems,16 and subsequently successfully applied to HIV-1 reverse trascriptase case studies.17 PHASE (2006) was validated using the Patel data set4 that we also examined in part I of this series. Typically then, only a handful of data sets are used to validate the algorithms, and even these data sets are not consistent across the different publications. Leach et al. also point out that pharmacophore elucidation programs are occasionally validated on series of ligands that any competent modeler could mentally overlay in a few seconds.18 A step forward in this area is described by Jones when evaluating GAPE,19 a follow-up to the GASP method, which uses 13 target sets for validation, and also describes an objective methodology for evaluating the quality of the alignments. In the area of virtual screening, in recent years a number of publications have appeared to address the similar problem of validating these methods. The DUD data set is an admirable attempt to provide a curated benchmark data set with known active ligands against a large number of targets, with more relevant decoys, so as to avoid bias (for example where decoys could be easily distinguished from actives by trivial metrics such as molecular weight). In our opinion, there is clearly a need for a similar benchmark data set in the area of pharmacophore elucidation, and our aim with this publication is to provide such a data set, some objective metrics to evaluate performance, and describe how our own method performs.
■
ASSEMBLING THE PHARMBENCH DATA SET To validate pharmacophore elucidation methods, for each data set the correct answer must be available to compare with the predicted hypotheses. The first limitation is therefore that experimental structural data must be available, and we added the restriction that the ligands themselves must be pharmaceutically relevant. As a first step, we used the online DrugPort20 resource as a starting point, which lists the protein targets for approved drugs and nutraceuticals. Therefore any structural target in the data set would be one for which there is an approved drug or nutraceutical. The full list of ∼1600 targets was downloaded, and the list was filtered to remove targets with fewer than four structures and having at least one drug on the market, to leave 334 targets. The UniProtKB accession number for each target was then used to query the protein data bank,21 with the additional requirement that the structure must contain
■
GENERATING THE HYPOTHESES We have already described the FLAPpharm method in part I of this paper;1 however, some additional options were tested in this work. The first of these concerns the molecular alignment when constructing the model; the alignment is performed using quadruplets formed from combinations of the atoms in the 2600
dx.doi.org/10.1021/ci300154n | J. Chem. Inf. Model. 2012, 52, 2599−2608
Journal of Chemical Information and Modeling
Article
Table 1. Pharmacophore Elucidation Performance by FLAPpharm on the PharmBench Data Seta
a
The table shows the target name, UniProtKb IDs for each target (with the addition of pseudocodes for the Patel data sets and our DNA minor groove binder data set), the number of molecules aligned for that target, and the results of the three benchmark metrics when building models from the X-ray conformations or from the 2D input structures and up to 30 generated conformations. The second AlignScore evaluation metric is colour coded to indicate successful models (50% or more molecules aligned with an rmsd of
