Named Entity Recognition and Normalization Applied to Large-Scale

Subscriber access provided by EKU Libraries

Chemical Information

Named Entity Recognition and Normalization Applied to LargeScale Information Extraction from the Materials Science Literature Leigh Weston, Vahe Tshitoyan, John Dagdelen, Olga Kononova, Amalie Trewartha, Kristin A. Persson, Gerbrand Ceder, and Anubhav Jain J. Chem. Inf. Model., Just Accepted Manuscript • DOI: 10.1021/acs.jcim.9b00470 • Publication Date (Web): 30 Jul 2019 Downloaded from pubs.acs.org on August 5, 2019

Just Accepted “Just Accepted” manuscripts have been peer-reviewed and accepted for publication. They are posted online prior to technical editing, formatting for publication and author proofing. The American Chemical Society provides “Just Accepted” as a service to the research community to expedite the dissemination of scientific material as soon as possible after acceptance. “Just Accepted” manuscripts appear in full in PDF format accompanied by an HTML abstract. “Just Accepted” manuscripts have been fully peer reviewed, but should not be considered the official version of record. They are citable by the Digital Object Identifier (DOI®). “Just Accepted” is an optional service offered to authors. Therefore, the “Just Accepted” Web site may not include all articles that will be published in the journal. After a manuscript is technically edited and formatted, it will be removed from the “Just Accepted” Web site and published as an ASAP article. Note that technical editing may introduce minor changes to the manuscript text and/or graphics which could affect content, and all legal disclaimers and ethical guidelines that apply to the journal pertain. ACS cannot be held responsible for errors or consequences arising from the use of information contained in these “Just Accepted” manuscripts.

is published by the American Chemical Society. 1155 Sixteenth Street N.W., Washington, DC 20036 Published by American Chemical Society. Copyright © American Chemical Society. However, no copyright claim is made to original U.S. Government works, or works produced by employees of any Commonwealth realm Crown government in the course of their duties.

Page 1 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Journal of Chemical Information and Modeling

Named Entity Recognition and Normalization Applied to Large-Scale Information Extraction from the Materials Science Literature L. Weston,1 V. Tshitoyan,2 J. Dagdelen,1 , O. Kononova,2 A. Trewartha,2 K. A. Persson,1 , G. Ceder2 and A. Jain1 1

Lawrence Berkeley National Laboratory, Energy Technologies Area, 1 Cyclotron Road,

Berkeley, CA 94720, United States 2

Lawrence Berkeley National Laboratory, Materials Science Division, 1 Cyclotron Road,

Berkeley, CA 94720, United States Abstract The number of published materials science articles has increased manyfold over the past few decades. Now, a major bottleneck in the materials discovery pipeline arises in connecting new results with the previously established literature. A potential solution to this problem is to map the unstructured raw-text of published articles onto structured database entries that allows for programmatic querying. To this end, we apply text-mining with named entity recognition (NER) for large-scale information extraction from the published materials science literature. The NER model is trained to extract summary-level information from materials science documents, including: inorganic material mentions, sample descriptors, phase labels, material properties and applications, as well as any synthesis and characterization methods used. Our classifier achieves an accuracy (f 1) of 87%, and is applied to information extraction from 3.27 million materials science abstracts. We extract more than 80 million materials-science-related named entities, and the content of each abstract is represented as a database entry in a

1

ACS Paragon Plus Environment

Journal of Chemical Information and Modeling 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

structured format. We demonstrate that simple database queries can be used to answer complex “meta-questions” of the published literature that would have previously required laborious, manual literature searches to answer. All of our data and functionality has been made freely available (https://github.com/materialsintelligence/matscholar), and we expect these results to accelerate the pace of future materials science discovery.

1

Introduction

Presently, the vast majority of historical materials science knowledge is stored as unstructured text across millions of published scientific articles. The body of research continues to grow rapidly; the magnitude of published data is now so large that individual materials scientists will only access a fraction of this information in their lifetime. With the increasing magnitude of available materials science knowledge, a major bottleneck in materials design arises from the need to connect new results with the mass of previously published literature. Recent advances in machine learning and natural language processing have enabled the development of tools capable of extracting information from scientific text on a massive scale. 1–3 Of these tools, named entity recognition (NER) 4 is currently one of the most widely used. Historically, NER was developed as a text-mining technique for extracting information, such as the names of people and geographic locations, from unstructured text, such as newspaper articles. 4 The task is typically approached as a supervised machine learning problem, in which a model learns to identify the key words or phrases in a sentence; in this way, documents may be represented in a structured format based on the information contained within them. In the past decade, NER has become an important feature for text mining within the chemical sciences. 5,6 In addition to entity recognition, a major challenge lies in mapping each entity onto a unique database identifier in a process called entity normalization. This issue arises due to the many ways in which a particular entity may be written. For example, “age hardening” and “precipitation hardening” refer to the same process. However, training a machine to 2


Page 2 of 32

Page 3 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


recognize this equivalence is especially challenging. A significant research effort has been applied to entity normalization in the biomedical domain; for example, the normalization of gene names has been achieved by using external databases of known gene synonyms. 7 No such resources are available for the materials science domain, and entity normalization has yet to be reported. In the field of materials science, there has been significant effort to apply NER to extracting inorganic materials synthesis recipes; 8–11 furthermore, a number of chemistry-based NER systems are capable of extracting inorganic materials mentions. 12–15 Some researchers have relied on chemical NER in combination with lookup tables to extract mentions of materials properties or processing conditions. 16,17 However, there has been no large-scale effort to extract summary-level information from material science texts. Materials informatics researchers often make predictions for many hundreds or thousands of materials, 18,19 and it would be extremely useful for researchers to be able to ask large-scale questions of the published literature, such as: “for this list of 10,000 materials, which have been studied as a thermoelectric, and which are yet to be explored? ”. An experimental materials science researcher might want to ask: “what is the most common synthesis method for oxide ferroelectrics? And, give me a list of all documents related to this query”. Currently, answering such questions requires a laborious and tedious literature search, performed manually by a domain expert. However, by representing published articles as structured database entries, such questions may be asked programmatically, and can be answered in a matter of seconds. In the present report, we apply NER along with entity normalization for large-scale information extraction from the materials science literature. We apply information extraction to over 3.27 million materials science journal articles; we focus solely on the article abstract, which is the most information-dense portion of the article and also readily available from various publisher application programming interfaces (e.g., Scopus). The NER model is a neural network trained using 800 hand-annotated abstracts and achieves an overall f 1 score of 87%. Entity normalization is achieved using a supervised machine learning model

3



that learns to recognize whether or not two entities are synonyms with an f 1 score of 95%. We find that entity normalization greatly increases the number of relevant items identified in document querying. The unstructured text of each abstract is converted into a structured database entry containing summary-level information about the document: inorganic materials studied, sample descriptors or phase labels, mentions of any material properties or applications, as well as any characterization or synthesis methods used in the study. We demonstrate how this type of large-scale information extraction allows researchers to access and exploit the published literature on a scale that has not previously been possible. In addition, we release the following data sets: (i) 800 hand-annotated materials science abstracts to be used as training data for NER, 20 (ii) JSON files containing details for mapping named entities onto their normalized form, 21 and (iii) the extracted named entities, and corresponding digital object identifier (DOI), for 3.27 million materials science articles. 22 Finally, we have released a public facing website and API for interfacing with the data and trained models, as outlined in Section. 5.

2

Methodology

Our full information-extraction pipeline is shown in Fig. 1. Detailed information about each step in the pipeline is presented below, along with an analysis of extracted results. Machine learning models described below are trained using the Scikit-learn, 23 Tensorflow 24 and Keras 25 python libraries.

2.1

Data collection and preprocessing

Document collection. This work focuses on text-mining the abstracts of materials science articles. Our aim is to collect a majority of all English-language abstracts for materialsfocused articles published between the years 1900 and 2018. To do this, we create a list of over 1100 relevant journals indexed by Elsevier’s Scopus and collect articles published in these

4


Page 4 of 32

Page 5 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


FIG. 1: Workflow for named entity recognition. The key steps are as follows: (i) documents are collected and added to our corpus, (ii) the text is preprocessed (tokenized and cleaned), (iii) for training data, a small subset of documents are labeled (SPL = symmetry/phase label, MAT = material, APL = application), (iv) the labelled documents are combined with word embeddings (Word2vec 26 ) generated from unlabelled text to train a neural network for named entity recognition, and finally (v) entities are extracted from our text corpus.

5



journals that fit these criteria via the Scopus and ScienceDirect APIs, 27 the SpringerNature API, 28 and web scraping for journals published by the Royal Society of Chemistry 29 and the Electrochemical Society. 30 The abstracts of these articles (and associated metadata including title, authors, publication year, journal, keywords, doi, and url) are then each assigned a unique ID and stored as individual documents in a dual MongoDB/ElasticSearch database. Overall, our corpus contains more than 3.27 million abstracts. Text preprocessing. The first step in document preprocessing is tokenization, which we performed using ChemDataExtractor. 12 This involves splitting the raw text into sentences, followed by the splitting of each sentence into individual tokens. Following tokenization, we developed several rule-based pre-processing steps. Our pre-processing scheme is outlined in detail in the supporting information (S.1); here, we provide a brief overview. Tokens that are identified as valid chemical formulae are normalized, such that the order of elements and common multipliers do not matter (e.g. NiFe is the same as Fe50Ni50); this is achieved using regular expression and rule based techniques. Valence states of elements are split into separate tokens (e.g. Fe(III) becomes two separate tokens, Fe and (III)). Additionally, if a token is neither a chemical formula nor an element symbol, and if only the first letter is uppercase, we lowercase the word. This way chemical formulae as well as abbreviations stay in their common form, whereas words at the beginning of sentences as well as proper nouns are converted to lowercase. Numbers with units are often not tokenized correctly with ChemDataExtractor. We address this in the processing step by splitting the common units from numbers and converting all numbers to a special token . This reduces the vocabulary size by several tens of thousand words. Document selection. For this work, we focus on inorganic materials science papers. However, our materials science corpus contains some articles that fall outside the scope of the present work; for example, we consider articles on polymer science or biomaterials to not be relevant. In general, we only consider an abstract to be of interest (i.e., useful for training and testing our NER algorithm) if the abstract mentions at least one inorganic material along with

6


Page 6 of 32

Page 7 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


at least one method for the synthesis or characterization of that material. For this reason, before training an NER model, we first train a classifier for document selection. The model is a binary classifier capable of labeling abstracts as “relevant” or “not relevant”. For training data, we label 1094 randomly selected abstracts as “relevant” or “not relevant”; of these, 588 are labeled as “relevant”, and 494 are labeled “not relevant”. Details and specific rules used for relevance labeling are included in the supporting information (S.3). The labeled abstracts are used to train a classifier; we use a linear classifier based on Logistic Regression, 31 where each document is described by a term frequency – inverse document frequency (tf-idf) vector. The classifier achieves an accuracy (f 1) score of 89% based on 5-fold cross validation. Only documents predicted to be relevant are considered as training/testing data in the NER study; however, we perform NER over the full 3.27 million abstracts regardless of relevance. We are currently developing text-mining tools that are optimized for a greater scope of topics (e.g., the polymer literature).

2.2

Named Entity Recognition

Using NER, we are interested in extracting specific entity types that can be used to summarize a document. To date, there have been several efforts to define an ontology or schema for information representation in materials science (see Ref. [32] for a review of these efforts); in the present work, for each document we wish to know what was studied, and how it was studied. To extract this information, we design seven entity labels: inorganic material (MAT), symmetry/phase label (SPL), sample descriptor (DSC), material property (PRO), material application (APL), synthesis method (SMT), and characterization method (CMT). The selection of these labels is somewhat motivated by the well-known material science tetrahedron: “processing”, “structure”, “properties”, and “performance”. Examples for each of these tags are given in the supporting information (S.4), along with a detailed explanation regarding the rules for annotating each of these tags. Using the tagging scheme described above, 800 materials science abstracts are annotated 7



by hand; only abstracts that are deemed to be relevant based on the relevance classifier described earlier are annotated. We choose abstracts to annotate by randomly sampling from relevant abstracts so as to ensure that a diverse range of inorganic materials science topics are covered. The annotations are performed by a single materials scientist. We stress that there is no necessarily “correct” way to annotate these abstracts, however to ensure that the labelling scheme is reasonable, a second materials scientist annotated a subset of 25 abstracts to assess the inter-annotator agreement, which was 87.4%. This was calculated as the percentage of tokens for which the two annotators assigned the same label. For annotation, we use the inside-outside-beginning (IOB) format. 33 This is necessary to account for multi-word entities, such as “thin film”. In this approach, there are special tags representing a token at the beginning (B), inside (I), or outside (O) of an entity. For example, the text fragment “Thin films of SrTiO3 were deposited ”, would be labeled as (token; IOBtag) pairs in the following way: (Thin; B-DSC), (films; I-DSC), (of; O), (SrTiO3 ; B-MAT), (were; O), (deposited; O). Before training a classifier, the 800 annotated abstracts are split into training, development (validation) and test sets. The development set is used for optimizing the hyperparameters of the model, and the test set is used to assess the final accuracy of the model on new data. We use an 80% -10% -10% split, such that there are 640, 80, and 80 abstracts in the train, development, and test sets respectively.

2.3

Neural Network model

The neural network architecture for our model is based on that of Lample et al.; 34 a schematic of this architecture is shown in Fig. 2(a). We explain the key features of the model below. The aim is to train a model in such a way that materials science knowledge is encoded; for example, we wish to teach a computer that the words “alumina” and “SrTiO3 ” represent materials, whereas “sputtering” and “MOCVD” correspond to synthesis methods. There are three main types of information that can be used to teach a machine to recognize which 8


Page 8 of 32

Page 9 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


(a)

B-DSC

I-DSC

O

B-MAT

O

…

O

Conditional Random Fields Backward-LSTM output Forward-LSTM output

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

…

LSTM

…

LSTM

Char-LSTM output Word embeddings

Thin

films

(b)

of

SrTiO3 were deposited

…

Char-LSTM output Backward-char-LSTM output Forward-char-LSTM output

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

Char embeddings

S

r

T

i

O

3

FIG. 2: Neural network architecture for named entity recognition. The colored rectangles represent vectors that are inputs/outputs from different components of the model. The model in (a) represents the word-level bi-directional LSTM, that takes as input a sequence of words, and returns a sequence of entity tags in IOB format. The word-level features for this model are the word embeddings for each word, which are concatenated with the output of the character-level LSTM run over the same word. The character-level LSTM is shown in (b). This model takes a single word such as “SrTiO3 ”, and runs a bi-directional LSTM over each character to encode the morphological properties of each word.

9



words correspond to a specific entity type: (i) word representation, (ii) local (within sentence) context, and (iii) word shape. For (i), word representation, we use word embeddings. Word embeddings map each word onto a dense vector of real numbers in a high-dimensional space. Words that have a similar meaning, or are frequently used in a similar context, will have a similar word embedding. For example, entities such as “sputtering” and “MOCVD” will have similar vector representations; during training, the model learns to associate these word vectors as synthesis methods. The word embeddings are generated using the Word2vec approach of Mikolov et al.; 26 the embeddings are 200-dimensional and are based on the skip-gram approach. Word embeddings are generated by training on our corpus of 3.27 million materials science abstracts. More information about the training of word embeddings is included in the supporting information (S.2). For (ii), context, the model considers a sentence as a sequence of words, and it takes into account the local context of each word in the sentence. For example, in the sentence “The band gap of

is 4.5 eV”, it is quite clear that the missing word is a material, rather than

a synthesis method or some other entity, and this is obvious from the local context alone. To include such contextual information, we use a recurrent neural network (RNN), a type of sequence model that is capable of sequence-to-sequence (many-to-many) classification. As traditional RNNs suffer from problems in dealing with long-range dependencies, we use a variant of the RNN called Long Short-Term Memory (LSTM). 35 In order to capture both forward and backward context, we use a bi-directional LSTM; in this way, one LSTM reads the sentence forwards, and the other reads it backwards, with the results being combined. For (iii), word shape, we include character-level information about each word. For example, material formulae like “SrTiO3 ” have a distinct shape, containing uppercase, lowercase and numerical characters, in a specific order; this word shape can be used to help in entity classification. Similarly, prefixes and suffixes provide useful information about entity type; for example, the suffix “ium”, for example in “strontium”, is commonly used for elemental

10


Page 10 of 32

Page 11 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


metals, and so a word that has this suffix has a good chance of being part of a material name. In order to encode this information into the model, we use a character-level bi-directional LSTM over each word [Fig. 2(b)]. The final outputs from the character-level LSTM are concatenated with the word embeddings for each word; these final vectors are used as the word representations for the word-level LSTM [Fig. 2(a)]. For the word-level LSTM, we use pre-trained word embeddings that have been trained on our corpus of over 3.27 million abstracts. For the character-level LSTM, the character embeddings are not pretrained, but are learned during the training of the model. The output layer of the model is a Conditional Random Fields (CRF) classifier, rather than a typical softmax layer. Being a sequence-level classifier, the CRF is better at capturing the strong inter-dependencies of the output labels. 36 The model has a number of hyperparameters, including the word- and character-level LSTM size, the character embedding size, the learning rate, and drop out. The NER performance was optimized by repeatedly training the model with randomly selected hyperparameters; the final model chosen was the one with the highest accuracy when assessed on the development set.

2.4

Entity Normalization

After entity recognition, the final step is entity normalization. This is necessarily required as each entity may be written in numerous forms; for example, “TiO2 ”, “titania”, “AO2 (A = Ti)” and “titanium dioxide”, all refer to the same specific stoichiometry: TiO2 . For document querying, it is important to store these entities in a normalized format, so that a query for documents that mention “titania”, also returns documents that mention “titanium dioxide”. In order to normalize material mentions, we convert all material names into a canonical normalized formula. The normalized formula is alphabetized and divided by the highest common factor of the stoichiometry. In this way, “TiO2 ”, “titania”, and “titanium dioxide” are all normalized to “O2Ti”. In some cases, multiple stoichiometries are extracted from a 11



single material mention; for example, “Zrx Ti1−x O3 (x = 0, 0.5, 1)” is converted to “O2Ti”, “O4TiZr” and “O2Zr”. When a continuous range is given, e.g, 0 ≥ x ≤ 1, we increment over this range in steps of 0.1. Material mentions are normalized using regular expressions, rule-based methods, as well as by making use of the PubChem look up tables; 37 final validity checks on the normalized formula are performed using the pymatgen library. 38 Normalization of other entity types is also crucial for comprehensive document querying. For example, “chemical vapor deposition”, “chemical-vapour deposition”, and “CVD” all refer to the same synthesis technique, i.e., they are synonyms for this entity type. In order to determine that two entities have the same meaning, we trained a classifier that is capable of determining whether or not two entities are synonyms. The model uses the word embeddings for each entity as features; after performing NER, each multi-word entity is concatenated into a single word, and new word embeddings are trained such that every multi-word entity has a single vector representation. Each synonym consists of an entity pair, and so the features for the model are created by concatenating the word embeddings of the two entities in question. In addition to the word embeddings, which mostly capture the context in which an entity is used, several other hand-crafted features are included (supporting information S.5). To train the model, 10000 entity pairs are labelled as being either synonyms or not (see supporting information S.6). Using this data, a binary random forest classifier is trained to be able to predict whether or not two entities are synonyms of one another. Using the synonym classifier, each extracted entity can be normalized to a canonical form. Each entity is stored as its most frequently occurring synonym (we exclude acronyms as a normalized form); for example, “chemical vapor deposition”, “chemical-vapour deposition”, and “CVD” are all stored as “chemical vapor deposition”, as this is the most frequently occurring synonym that is not an acronym.

12


Page 12 of 32

Page 13 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


3

Results

3.1

NER classifier performance

Using the trained classifier, we are able to accurately extract information from materials science texts. The NER classifier performance is demonstrated in Fig. 3; the model can clearly identity the key information in the text.

FIG. 3: Example predictions of the materials science NER classifier. The highlighting indicates regions of text that the model has associated with a particular entity type. Model performance can be assessed more quantitatively by assessing the accuracy on the development and test sets. We define our accuracy using the f 1 score, which is the harmonic mean of precision (p) and recall (r), tp , tp + fp tp , r= tp + fn p·r f1 = 2 , p+r p=

(1) (2) (3)

where, tp , fp , and fn represent the rate of true positives, false positives, and false negatives, respectively. We use the CoNLL scoring system, which requires an exact match for the entire multi-word entity; 39 for example, if a CMT (characterization method) is labeled “photoluminescence spectroscopy”, and the model only labels the “photoluminescence” portion as a CMT, the entire entity is considered as being labeled incorrectly. The accuracy of our final neural network model is presented in Table 1 (accuracy as a 13



function of training set size is presented in the supporting information S.7). We include the overall f 1 score as well as the score for each entity type for both the development and test sets (a breakdown of precision and recall for each entity type in the test set is presented in the supporting information S.8). The accuracy on the development and test sets is similar, suggesting that the model has not been overfit to the development set during hyperparameter tuning. The overall f 1 score on the test set is 87.04%. This score is fairly close to the current state of the art NER system (90.94%) 34 based on the same neural network architecture (which is trained and evaluated on hand-labelled newspaper articles 39 ). However, we caution against a direct comparison of scores on these tasks, as the modeling of materials science text is expected to be quite different than newspaper articles. The f 1 score of 90.30% for inorganic material extraction is as good or better than previously reported chemical NER systems. For example, Mysore et al. achieved an f 1 score of 82.11% for extraction material mentions; 9 Swain et al. reported an f 1 score of 93.4% for extracting chemical entity mentions, 12 however their model is not specifically trained to identify inorganic materials. The f 1 scores for other entity tags are all over 80%, suggesting that the model is performing well at extracting each entity type. Materials mentions and sample descriptors have the highest scores; this is most likely due to their frequency in the training set and because these entities are often single tokens. We note that this work makes available all 800 hand-labelled abstracts for the testing of future algorithms against the current work.

3.2

Entity normalization

Following entity extraction, the next step is entity normalization. The trained random forest binary classifier exhibits an f 1 score of 94.5% for entity normalization. The model is assessed using 10-fold cross-validation with a 9000:1000 train/test split. The precision and recall of the model are 94.6% and 94.4%, respectively, when assessed on this test set. While these accuracy scores are high, we emphasize that the training/test data are generated in a synthetic manner (see supporting information S.6), such that the test set has roughly 14


Page 14 of 32

Page 15 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


TABLE 1: Accuracy metric f 1 for named entity recognition on the development and test sets. Results are shown for the total, material (MAT), phase (SPL), sample descriptor (DSC), property (PRO), application (APL), synthesis method (SMT), and characterization (CMT) tags. Also shown is the total number extracted for each entity type over the full corpus of abstracts. Accuracy (f 1) Label

Dev. set

Test set

Extracted (millions)

Total

87.09

87.04

81.23

MAT

92.58

90.30

19.07

SPL

85.24

82.05

0.53

DSC

91.40

92.13

9.36

PRO

80.19

83.19

31.00

APL

80.60

80.63

7.46

SMT

81.32

81.37

5.01

CMT

86.52

86.01

8.80

balanced classes. In reality, the vast majority of entities are not synonyms. By artificially reducing the number of negative test instances, the number of false positives is also artificially reduced. In production, we find that the model is prone to making false-positive predictions. In terms of false positives, the two most common mistakes made by the synonym classifier are: (i) when two entities have an opposite meaning but are used in a similar context, e.g. “p-type doping” and “n-type doping”, or (ii) when two entities refer to related but not identical concepts, e.g, “hydrothermal synthesis” and “solvothermal synthesis”. In order to overcome the large number of false positives, we perform a final, manual error check on any normalized entities. If an entity is incorrectly normalized, the incorrect synonym is manually removed, and the entity is converted to the most frequently occurring correct prediction of the ML model.

15



3.3

Text-mined information extraction

We run the trained NER classifier described in Sec. 2.3 over our full corpus of 3.27 million materials science abstracts. The entities are normalized using the procedure described in Sec. 2.4. Overall, more than 80 million materials-science-related entities are extracted; a quantitative breakdown of the entity extraction is shown in Table 1. After entity extraction, the unstructured raw-text of an abstract can be represented as a structured database entry, based on the entity tags contained within it. We represent each abstract as a document in a non-relational (NoSQL) MongoDB 40 database collection. This type of representation allows for efficient document retrieval and allows the user to ask summary-level or “meta-questions” of the published literature. In Sec. 4, we demonstrate several ways by which large-scale text mining allows for researchers to access the published literature programmatically.

4 4.1

Applications Document retrieval

Online databases of published literature work by allowing users to query based on some keywords. In the simplest case, the database will return documents that contain an exact match to the keywords in the query. More advanced approaches may employ some kind of “fuzzy” matching; for instance, many text-based search engines are built upon Apache Lucene, 41 which allows for fuzzy matching based on edit distance. While advanced approaches to document querying have been developed, to date there has been no freely-available “materialsscience aware” database of published documents; i.e., there is no materials science knowledge base for the published literature. For example, querying for “BaTiO3 ” in Web of Science returns 19,134 results, where as querying for “barium titanate” returns 12,986 documents, and querying for “TiBaO3 ” returns only a single document. Ideally, all three queries should

16


Page 16 of 32

Page 17 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


return the same result (as is the case for our method), as all three queries are based on the same underlying stoichiometry. We find that that by encoding materials science knowledge into our entity extraction and normalization scheme, performance in document retrieval can be significantly increased. In Fig. 4, the effectiveness of our normalized entity database for document retrieval is explored. To test this, we created two databases, in which: (i) each document is represented by the normalized entities extracted, and (ii) each document is represented by the non-normalized entities extracted. We then randomly choose an entity of any type, and query each database based on this entity to see how many documents are returned. The process of querying on a randomly selected entity is repeated 1000 times to generate statistics on the efficiency of each database in document retrieval. This test is also performed for two-entity queries, by randomly selecting two entities and querying each database. We note that because the precision of our manually checked normalization is close to unity, this test effectively measures the relative recall (Eq. 2) of each database (by comparing the false-negative rate).

FIG. 4: Recall statistics for document retrieval are generated by performing 1000 queries for each for each database. Results are shown for the entities that have been properly normalized using the scheme described in Sec. 2.4 (Ents. Norm), as well as for non-normalized entities (Ents. Raw). From Fig. 4, it is clear that normalization of the entities is crucial for achieving high recall in document queries. When querying on a single entity, the recall for the normalized database is about 5 times higher than for the non-normalized case. When querying on two entities this is compounded, and the non-normalized database only returns about 2% of the 17



Page 18 of 32

documents that are returned from the normalized database. Our results highlight that a domain-specific approach such as the one we described can greatly improve performance. As described in Sec. 5, we have built a materials-science aware document search engine that can be used to more comprehensively access the materials science literature.

4.2

Material search

In the field of materials informatics, researchers often use machine learning models to predict chemical compositions that may be useful for a particular application. A practical major bottleneck in this process comes from needing to understand which of these chemical compositions have been previously studied for that application, which typically requires painstaking manual literature review. In Sec. 1, we posed the following example question “for this list of 10,000 materials, which have been studied as a thermoelectric, and which are yet to be explored? ”. Analysis of the extracted named entities can be a useful tool in automating the literature review process. In Fig 5, we illustrate how one can analyze the co-occurrence of various property/application entities with specific materials on a large scale. Our web application has a “Materials Search” functionality. Users can enter a specific property or application entity, and the Materials Search will return a table of all materials that co-occur with that entity, along with document counts and the corresponding DOIs. As demonstrated, the results can be further filtered based on composition; in the example shown in Fig 5, all of the known thermoelectrics compositions that do not contain lead are returned. The results can then by downloaded in comma-separated values (CSV) format, to allow for programmatic literature search.

4.3

Guided literature searches and summaries

One major barrier for accessing the literature is that researchers may not know a priori what are the relevant terms or entities to search for a given material. This is especially true for researchers that may be entering a new field, or may be studying a new material for the first 18


Page 19 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


FIG. 5: A screenshot of the materials search functionality on our web application, which allows one to search for materials associated with a specified keyword (ranked by count of hits) and with element filters applied. The application returns clickable hyperlinks so that users can directly access the original research articles. time. In Fig. 6, we demonstrate how “materials summaries” can be automatically generated based on the entities. The summaries show common entities that co-occur with a material; in this way, a researcher can learn how a material is typically synthesized, common crystal structures, or how and for what application a material is typically studied. The top entities are presented along with a score (0 – 100%), indicating how frequently each entity co-occurs with a chosen material. We note that the summaries are automatically generated without any human intervention or curation. As an example, in Fig. 6 summaries are presented for PbTe and BiFeO3 . PbTe is most frequently studied as a thermoelectric, 42 and so many of the entities in the summary deal with the thermoelectric and semiconducting properties of the material, including dopability. PbTe has a cubic (rock salt, f cc) structure as the ground state crystal structure under ambient conditions. 43 PbTe does have some metastable phases such as orthorhombic, 44 and this is also reflected in the summary; some of the less-frequently-occurring phases (i.e., scores less than 0.01) arise simply due to co-occurrence with other materials, indicating a weakness of the co-occurrence assumption. The summary in Fig. 6(b) for BiFeO3 mostly reflects the extensive research into this material as a multiferroic (i.e., a ferromagnetic ferroelectric). 45 The summary shows that BiFeO3 has several stable or metastable polymorphs, including the rhombohedral (hexago19



FIG. 6: Automatically generated materials summaries for PbTe (a) and BiFeO3 (b). The score is the percentage of papers that mention each entity at least once.

20


Page 20 of 32

Page 21 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


nal) and orthorhombic, 46 as well as tetragonal 47 and others; these phases are all known and have been studied in some form. BiFeO3 is most frequently synthesized by either sol - gel or solid state reaction routes, which is common for oxides. The types of summaries described above are very powerful, and can be used by nondomain experts to get an initial overview of a new material or new field, or for domain experts to gain a more quantitative understanding of the published literature or broader study of their topic (e.g., learning about other application domains that have studied their target material). For example, knowledge of the most common synthesis techniques is often used to justify the choice of chemical potentials in thermodynamic modelling, and these quantities are used in calculations of surface chemistry 48 and charged point defects 49 in inorganic materials. We note that there may be some inherent bias in the aforementioned literature summaries that may arise due to the analysis of only the abstracts. For example, more novel synthesis techniques such as “molecular beam epitaxy” may be more likely to be mentioned in the abstract than a more common technique such as “solid-state reaction”. This could be easily overcome by performing our analysis on full text rather than just abstracts (see Sec. 6 for a discussion on the current limitations and future work).

4.4

Answering meta-questions

Rather than simply listing which entity tags co-occur with a given material, one may want to perform more complex analyses of the published literature. For example, one may be interested in multifunctional materials; a researcher might ask “which materials have the most diverse range of applications? ”. To answer such a question, the co-occurrence of each material with the different applications can be analyzed. In order to ensure that materials with diverse applications are uncovered, we first perform a clustering analysis on the word embeddings for each application entity that has been extracted. Clustering on the full 200-dimensional word embeddings gives poor results – 21



FIG. 7: (a) Word embeddings for the 5000 most commonly mentioned applications are projected onto a two-dimensional plane. The coloring represent clusters identified by the DBSCAN algorithm. The applications from one of these clusters are highlighted – all applications in this cluster relate to solid-state memory devices. In (b), the materials with the largest number of applications are plotted; the same is plotted in (c), however the data has been normalized to account for the total number of material mentions.

22


Page 22 of 32

Page 23 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


this is common for clustering of high-dimensional data. Improved results are obtained by reducing the dimensions of the embeddings using Principal Component Analysis (PCA) 50 and t-Distributed Stochastic Neighbor Embedding (t-SNE). 51 It is found that t-SNE provides qualitatively much better clusters than PCA; t-SNE does not necessarily preserve the data density or neighbor distances, however empirically this approach often provides good results when used with certain clustering algorithms. Each word embedding is first reduced to three dimensions using t-SNE, 51 and then a clustering analysis is performed using Density-based Spatial Clustering of Applications with Noise (DBSCAN). 52 The results of this analysis is shown in Fig. 7(a) (the data is reduced to two dimensions for the purposes of plotting). The DBSCAN algorithm finds 244 distinct application clusters. In Fig. 7(a), the applications found in one small cluster are highlighted – all of the entities in this cluster relate to solidstate memory applications. To determine which materials have the most diverse applications, we look at co-occurrence of each material with entities from each cluster. The top 5000 most commonly occurring materials and applications are considered. We generate a co-occurrence matrix, with rows representing materials and columns representing application clusters. A material is considered to be relevant to a cluster if it co-occurs with an application from that cluster in at least 100 abstracts. The results of this analysis is are presented in Fig. 7(b). Based on this analysis, the most widely used materials are SiO2 , Al2O3 , TiO2 , steel and ZnO. In Fig. 7(c) and (d), the most important applications for SiO2 and steel are shown in a pie chart. The labels for each segment of the chart are created by manually inspecting the entities in each cluster (we note that automated topic modeling algorithms could also be used to create a summary of each application cluster, 53 not pursued here). For SiO2 , the most important applications are in catalysis and complimentary metal-oxide semiconductor (CMOS) technologies (SiO2 is an important dielectric in CMOS devices); for steel, the most important applications are in various engineering applications. However, both materials have widely varied applications across disparate fields.

23



Being able to answer complex meta-questions of the published literature can be a powerful tool for researchers. With all of the corpus represented as a structured database, generating an answer to these questions requires only a few lines of code, and just a few seconds for data transfer and processing – this is in comparison to a several-weeks-long literature search performed by a domain expert that would have been previously required.

5

Database and code access

All of the data associated with this project has been made publicly available, including the 800 hand-annotated abstracts used as training data for NER, 20 as well as JSON files containing the information for entity normalization. 21 The entire entity database of 3.27 million documents is available online for bulk download in JSON format. 22 We have also created a public facing API to access the data and functionality developed in this project; the code to access the API is available via the Github repository (https://github.com/materialsintelligence/matscholar). The API allows users to not only access the entities in our data set, but also to perform NER on any desired materials science text. Finally, we have developed a web application, http://matscholar.herokuapp.com/search, for non-programmatic access to the data. The web application has various materials-science aware tools for accessing the published literature.

6

Summary and Discussion

Using the information extraction pipeline shown in Fig. 1, we have performed information extraction on over 3.27 million materials science abstracts with a high degree of accuracy. Over 80 million materials-science-related entities were extracted, and each abstract is represented as a structured document in a database collection. The entities in each document are normalized (i.e., equivalent terms grouped together), greatly increasing the number of relevant results in document retrieval. Text-mining on such a massive scale provides tremendous 24


Page 24 of 32

Page 25 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


opportunity for accessing information in materials science. One of the most powerful applications of text mining is in information retrieval, which is the process of accurately retrieving the correct documents based on a given query. In the field of biomedicine, the categorization and organization of information into a knowledge base has been found to greatly improve efficiency in information retrieval. 54,55 Combining ontologies with large databases of published literature allows researchers to impart a certain semantic quality onto queries, which is far more powerful than simple text-based searches that are agnostic to the domain. We have stored the digital object identifier (DOI) for each abstract in our corpus, such that targeted queries can be linked to the original research articles. In addition to information retrieval, an even more promising and novel aspect of text mining on this scale, is the potential to ask “meta-questions” of the literature, as outlined in Sec. 4.2 and Sec. 3.3. The potential to connect the extracted entities of each document and provide summary-level information based on an analysis of many thousands of documents can greatly improve the efficiency of the interaction between researchers and the published literature. We should note some limitations of the present information extraction system, and the prospects for improvement. First, certain issues arise as a result of only analyzing the abstracts of publications. Not all information about a publication is presented in the abstract; therefore, by considering only the abstract, the recall in document retrieval can be reduced. Second, the quantitative analysis of the literature, as in Fig. 6, can be affected. This can be overcome by applying our NER system to the full text of journal articles. The same tools developed here can be applied to full text, provided that the full text is freely available. A second issue with the current methodology is that the system is based on co-occurrence of entities, and we do not perform explicit entity relation extraction. By considering only co-occurrence, recall is not affected, but precision can be reduced. We do note that all of the other tools available for querying the materials science literature rely on co-occurrence. Despite these limitations, we have demonstrated the ways in which large-scale text mining can be used to develop research tools, and the present work represents a first step towards

25



artificially intelligent and programmatic access to the published materials science literature.

7

Supporting Information

The supporting information contains further details regarding data cleaning and processing, as well as annotation and model training.

8

Acknowledgments

This work was supported by Toyota Research Institute through the Accelerated Materials Design and Discovery program. We are thankful to Matthew Horton for useful discussions. The abstract data was downloaded from Scopus API between October 2017 and September 2018 via http://api.elsevier.com and http://www.scopus.com.

References (1) Voytek, J. B.; Voytek, B. Automated cognome construction and semi-automated hypothesis generation. J. Neurosci. Methods 2012, 208, 92–100. (2) Tshitoyan, V.; Dagdelen, J.; Weston, L.; Dunn, A.; Rong, Z.; Kononova, O.; Persson, K. A.; Ceder, G.; Jain, A. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature 2019, 571, 95. (3) Mukund, N.; Thakur, S.; Abraham, S.; Aniyan, A.; Mitra, S.; Philip, N. S.; Vaghmare, K.; Acharjya, D. An Information Retrieval and Recommendation System for Astronomical Observatories. Astrophys. J. Suppl. S. 2018, 235, 22. (4) Nadeau, D.; Sekine, S. A Survey of Named Entity Recognition and Classification. Lingvist. Investig. 2007, 30, 3–26.

26


Page 26 of 32

Page 27 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


(5) Segura-Bedmar, I.; Mart´ınez, P.; Zazo, M. H. Semeval-2013 Task 9: Extraction of Drug-Drug Interactions from Biomedical Texts (ddiextraction 2013). Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013). 2013; pp 341–350. (6) Eltyeb, S.; Salim, N. Chemical Named Entities Recognition: a Review on Approaches and Applications. J. Cheminformatics 2014, 6, 17. (7) Fang, H.-r.; Murphy, K.; Jin, Y.; Kim, J. S.; White, P. S. Human Gene Name Normalization using Text Matching with Automatically Extracted Synonym Dictionaries. Proceedings of the Workshop on Linking Natural Language Processing and Biology: Towards Deeper Biological Literature Analysis. 2006; pp 41–48. (8) Kim, E.; Huang, K.; Saunders, A.; McCallum, A.; Ceder, G.; Olivetti, E. Materials Synthesis Insights from Scientific Literature via Text Extraction and Machine Learning. Chem. Mater. 2017, 29, 9436–9444. (9) Mysore, S.; Kim, E.; Strubell, E.; Liu, A.; Chang, H.-S.; Kompella, S.; Huang, K.; McCallum, A.; Olivetti, E. Automatically Extracting Action Graphs from Materials Science Synthesis Procedures. arXiv preprint arXiv:1711.06872 2017, (10) Kim, E.; Huang, K.; Jegelka, S.; Olivetti, E. Virtual Screening of Inorganic Materials Synthesis Parameters with Deep Learning. npj Computational Materials 2017, 3, 53. (11) Kim, E.; Huang, K.; Tomala, A.; Matthews, S.; Strubell, E.; Saunders, A.; McCallum, A.; Olivetti, E. Machine-learned and Codified Synthesis Parameters of Oxide Materials. Scientific data 2017, 4, 170127. (12) Swain, M. C.; Cole, J. M. Chemdataextractor: a Toolkit for Automated Extraction of Chemical Information from the Scientific Literature. J. Chem. Inf. Model. 2016, 56, 1894–1904. 27



(13) Rockt¨aschel, T.; Weidlich, M.; Leser, U. ChemSpot: a Hybrid System for Chemical Named Entity Recognition. Bioinformatics 2012, 28, 1633–1640. (14) Leaman, R.; Wei, C.-H.; Lu, Z. tmChem: a High Performance Approach for Chemical Named Entity Recognition and Normalization. J. Cheminformatics 2015, 7, S3. (15) Krallinger, M.; Rabal, O.; Lourenco, A.; Oyarzabal, J.; Valencia, A. Information Retrieval and Text Mining Technologies for Chemistry. Chemical reviews 2017, 117, 7673– 7761. (16) Court, C. J.; Cole, J. M. Auto-generated Materials Database of Curie and N´eel Temperatures via Semi-supervised Relationship Extraction. Scientific data 2018, 5, 180111. (17) Shah, S.; Vora, D.; Gautham, B.; Reddy, S. A Relation Aware Search Engine for Materials Science. Integrating Materials and Manufacturing Innovation 2018, 7, 1–11. (18) Meredig, B.; Agrawal, A.; Kirklin, S.; Saal, J. E.; Doak, J. W.; Thompson, A.; Zhang, K.; Choudhary, A.; Wolverton, C. Combinatorial Screening for New Materials in Unconstrained Composition Space with Machine Learning. (19) Legrain, F.; Carrete, J.; van Roekeghem, A.; Madsen, G. K.; Mingo, N. Materials Screening for the Discovery of new Half-Heuslers: Machine Learning Versus Ab Initio Methods. J. Phys. Chem. B 2017, 122, 625–632. (20) https://doi.org/10.6084/m9.figshare.8184428. (21) https://doi.org/10.6084/m9.figshare.8184365. (22) https://doi.org/10.6084/m9.figshare.8184413. (23) Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830. 28


Page 28 of 32

Page 29 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


(24) Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M. Tensorflow: a System for Large-scale Machine Learning. OSDI. 2016; pp 265–283. (25) Chollet, F. Keras. 2015. (26) Mikolov, T.; Chen, K.; Corrado, G.; Dean, J. Efficient Estimation of Word Representations in Vector Space. arXiv preprint arXiv:1301.3781 2013, (27) https://dev.elsevier.com/. (28) https://dev.springernature.com/. (29) http://www.rsc.org/. (30) https://www.electrochem.org/. (31) Hosmer Jr, D. W.; Lemeshow, S.; Sturdivant, R. X. Applied Logistic Regression; John Wiley & Sons, 2013; Vol. 398. (32) Zhang, X.; Zhao, C.; Wang, X. A Survey on Knowledge Representation in Materials Science and Engineering: An Ontological Perspective. Comput. Ind. 2015, 73, 8–22. (33) Ramshaw, L. A.; Marcus, M. P. Natural language processing using very large corpora; Springer, 1999; pp 157–176. (34) Lample, G.; Ballesteros, M.; Subramanian, S.; Kawakami, K.; Dyer, C. Neural Architectures for Named Entity Recognition. arXiv preprint arXiv:1603.01360 2016, (35) Hochreiter, S.; Schmidhuber, J. Long Short-term Memory. Neural Comput. 1997, 9, 1735–1780. (36) Huang, Z.; Xu, W.; Yu, K. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv preprint arXiv:1508.01991 2015,

29



(37) Kim, S.; Thiessen, P. A.; Bolton, E. E.; Chen, J.; Fu, G.; Gindulyte, A.; Han, L.; He, J.; He, S.; Shoemaker, B. A. PubChem Substance and Compound Databases. Nucleic Acids Res. 2015, 44, D1202–D1213. (38) Ong, S. P.; Richards, W. D.; Jain, A.; Hautier, G.; Kocher, M.; Cholia, S.; Gunter, D.; Chevrier, V. L.; Persson, K. A.; Ceder, G. Python Materials Genomics (pymatgen): A Robust, Open-source Python Library for Materials Analysis. Comp. Mater. Sci. 2013, 68, 314–319. (39) Tjong Kim Sang, E. F.; De Meulder, F. Introduction to the CoNLL-2003 Shared Task: Language-independent Named Entity Recognition. Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4. 2003; pp 142–147. (40) https://www.mongodb.com/. (41) https://lucene.apache.org/. (42) Pei, Y.; LaLonde, A.; Iwanaga, S.; Snyder, G. J. High Thermoelectric Figure of Merit in Heavy Hole Dominated PbTe. Energy Environ. Sci. 2011, 4, 2085–2089. (43) Rabe, K. M.; Joannopoulos, J. D. Ab Initio Relativistic Pseudopotential Study of the Zero-temperature Structural Properties of SnTe and PbTe. Phys. Rev. B 1985, 32, 2302–2314. (44) Wang, Y.; Chen, X.; Cui, T.; Niu, Y.; Wang, Y.; Wang, M.; Ma, Y.; Zou, G. Enhanced Thermoelectric Performance of PbTe Within the Orthorhombic Pnma Phase. Phys. Rev. B 2007, 76, 155127. (45) Wang, J.; Neaton, J.; Zheng, H.; Nagarajan, V.; Ogale, S.; Liu, B.; Viehland, D.; Vaithyanathan, V.; Schlom, D.; Waghmare, U. Epitaxial BiFeO3 Multiferroic Thin Film Heterostructures. Science 2003, 299, 1719–1722.

30


Page 30 of 32

Page 31 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


(46) Arnold, D. C.; Knight, K. S.; Morrison, F. D.; Lightfoot, P. Ferroelectric-paraelectric Transition in BiFeO3 : Crystal Structure of the Orthorhombic β Phase. Phys. Rev. Lett. 2009, 102, 027602. (47) Fu, Z.; Yin, Z.; Chen, N.; Zhang, X.; Zhao, Y.; Bai, Y.; Chen, Y.; Wang, H.-H.; Zhang, X.; Wu, J. Tetragonal-tetragonal-monoclinic-rhombohedral Transition: Strain Relaxation of Heavily Compressed BiFeO3 Epitaxial Thin Films. Appl. Phys. Lett. 2014, 104, 052908. (48) Reuter, K.; Scheffler, M. Composition, Structure, and Stability of RuO2 (110) as a Function of Oxygen Pressure. Phys. Rev. B 2001, 65, 035406. (49) Weston, L.; Janotti, A.; Cui, X. Y.; Stampfl, C.; Van de Walle, C. G. Hybrid Functional Calculations of Point Defects and Hydrogen in SrZrO3 . Phys. Rev. B 2014, 89, 184109. (50) Jolliffe, I. Principal Component Analysis; Springer, 2011. (51) Maaten, L. v. d.; Hinton, G. Visualizing Data using t-SNE. J. Mach. Learn. Res. 2008, 9, 2579–2605. (52) Ester, M.; Kriegel, H.-P.; Sander, J.; Xu, X. A density-based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. Kdd. 1996; pp 226–231. (53) Blei, D. M.; Ng, A. Y.; Jordan, M. I. Latent Dirichlet Allocation. J. Mach. Learn. Res. 2003, 3, 993–1022. (54) M¨ uller, H.-M.; Kenny, E. E.; Sternberg, P. W. Textpresso: an Ontology-based Information Retrieval and Extraction System for Biological Literature. PLoS Biol. 2004, 2, e309. (55) Bodenreider, O. Biomedical Ontologies in Action: Role in Knowledge Management, Data Integration and Decision Support. Yearb. Med. Inform. 2008, 17, 67–79.

31



FIG. 8: For Table of Contents only.

32


Page 32 of 32

Named Entity Recognition and Normalization Applied to Large-Scale

Recommend Documents