ES2993023T3 - Systems and methods for neural language modeling - Google Patents

Abstract

In some aspects, the present disclosure relates to neural modeling of language. In one embodiment, a computer-implemented neural network includes a plurality of neural nodes, each of which has a plurality of input weights corresponding to a vector of real numbers. The neural network also includes an input neural node corresponding to a linguistic unit selected from an ordered list of a plurality of linguistic units, and an embedding layer with a plurality of embedding node partitions. Each embedding node partition includes one or more neural nodes. Each of the embedding node partitions corresponds to a position in the ordered list relative to a focus term, is configured to receive an input from an input node, and is configured to generate an output. The neural network also includes a classifier layer with a plurality of neural nodes, each configured to receive the embedding outputs from the embedding layer and configured to generate an output corresponding to a probability that a particular linguistic unit is the target term. (Machine translation with Google Translate, no legal value)

Description

DESCRIPTION

Systems and methods for neural language modeling

Background

Natural language processing (NLP) systems seek to automate the extraction of useful information from sequences of symbols in human language. Some NLP systems may encounter difficulties due to the complexity and sparsity of information in natural language. Neural network language models (NNLMs) can overcome the performance limitations of traditional systems. An NNLM can learn distributed representations of words and can incorporate a vocabulary into a smaller dimensional linear space that models a likelihood function for sequences of words, expressed in terms of these representations.

NNLMs can generate word embeddings by training a symbol prediction task on a moving local context window. The ordered set of weights associated with each word becomes the dense vector embedding of that word. The result is a vector space model that encodes semantic and syntactic relationships. An NNLM can predict a word given its surrounding context. These distributed representations encode nuances of meaning across their dimensions, allowing two words to have multiple real-valued relationships encoded in a single representation. This feature follows from the distributional hypothesis: words that appear in similar contexts have similar meaning. Words that appear in similar contexts will experience similar training examples, training results, and converge to similar weights.

Once computed, word embeddings based on word analogies can allow vector operations between words that reflect their semantic and syntactic relationships. The analogy "the king is to the queen as the man is to the woman" can be encoded in the vector space by the equation king - queen = man - woman.

Conventional NNLMs may not take into account word morphology and form. However, information about word structure in word representations can be valuable for some speech analysis, word similarity, and information extraction. It is with respect to these and other considerations that aspects of the present disclosure are presented herein. Quoc Le et al: "Distributed Representations of Sentences and Documents", 16 May 2014 (2014-05-16), proposes Paragraph Vector, an unsupervised algorithm that learns fixed-length feature representations from variable-length text fragments, such as sentences, paragraphs, and documents.

Summary

The invention is defined in the appended independent claims. The appended dependent claims define preferred embodiments.

Brief description of the figures

Reference will now be made to the attached figures, which are not necessarily drawn to scale.

Figure 1 depicts a neural node according to one embodiment.

Figure 2 depicts a neural network according to one embodiment.

Figure 3 depicts a partition-free neural network language model, using a continuous bag of words architecture.

Figure 4 depicts a windowed partitioned neural network language model according to one embodiment.

Figure 5 depicts a directional partitioned neural network language model according to one embodiment.

Figure 6 depicts the relative accuracy of each partition in a PENN model as judged by the row-relative word analogy scores.

Figure 7 is a computer architecture diagram of a computer system capable of implementing aspects of the present disclosure in accordance with one or more embodiments.

Detailed description

Although exemplary embodiments of the present disclosure are explained in detail, it is to be understood that other embodiments are contemplated. Accordingly, it is not intended that the scope of the disclosure be limited to the details of construction and arrangement of components set forth in the following description or illustrated in the figures. The disclosure is capable of other embodiments and of being practiced or carried out in various ways.

It should also be noted that, as used in the specification and the accompanying claims, the singular forms "a", "an", "the" and "the" include plural references unless the context clearly dictates otherwise. References to a composition containing "a" constituent are intended to include constituents other than the one mentioned.

Furthermore, in describing example embodiments, terminology will be resorted to for the sake of clarity. Each term is intended to be given its broadest meaning as understood by those skilled in the art and includes all technical equivalents that operate in a similar manner to achieve a similar purpose.

Ranges may be expressed herein as from "about" or "approximately" or "substantially" to a particular value and/or to "about" or "approximately" or "substantially" another particular value. When this range is expressed, other example embodiments include from the particular value and/or to the other particular value.

Herein, the use of terms such as "having", "has", "including" or "includes" are open-ended and intended to have the same meaning as terms such as "comprising" or "comprises" and do not exclude the presence of other structure, material, or acts. Similarly, although the use of terms such as "may" or "can" is intended to be open-ended and to reflect that the structure, material, or acts are not necessary, the lack of use of such terms is not intended to reflect that the structure, material, or acts are essential. To the extent that the structure, material, or acts are currently considered essential, they are identified as such.

It is also to be understood that the mention of one or more method steps does not exclude the presence of additional method steps or intermediate method steps between those steps expressly identified. Furthermore, although the term "step" may be used herein to connote different aspects of the methods employed, the term should not be construed as implying any particular order among the various steps described herein, save and except where the order of the individual steps is explicitly required.

The components described hereinafter that constitute various elements of this disclosure are intended to be illustrative and not restrictive. Many suitable components that would perform the same or similar functions as the components described herein are intended to be encompassed within the scope of this disclosure. Such other components not described herein may include, but are not limited to, for example, similar components that are developed after the development of the subject matter disclosed herein.

To facilitate understanding of the principles and features of the present disclosure, several illustrative embodiments are explained below. In particular, the presently disclosed subject matter is described in the context of NNLM. The present disclosure, however, is not so limited, and may be applicable in other contexts. For example, and without limitation, some embodiments of the present disclosure may enhance other sequence recognition techniques and the like. These embodiments are contemplated within the scope of the present disclosure. Accordingly, when the present disclosure is described in the context of NNLM, it will be understood that other embodiments may take the place of those referenced.

A neural network may comprise a plurality of layers of neural nodes (i.e., "neurons"). In some embodiments, a neural network comprises an input layer, a hidden layer, and an output layer. For ease of explanation, the basic operation of a neural node according to one embodiment will now be explained. However, as would be understood by one skilled in the art, other types of neurons could be used.

Neural networks are implemented by computer. The plurality of layers and nodes may reside in executable program modules (e.g., program modules 714 in FIG. 7) or other software constructs, or in dedicated programmed hardware components. The layers and nodes, and other functional components described herein in accordance with various embodiments for performing aspects of neural language modeling may be stored in memory devices (e.g., memory 704 or mass storage 712) and executable by processors (e.g., processing unit 702) of one or more computers, such as computer 700 shown in FIG. 7. Analysis, data processing, and other functions associated with the operation of the layers and nodes and the performance of the various neural language modeling functions described herein may be triggered by the execution of instructions by one or more of these processors. Training functions, such as model training processes as described herein, may be performed in conjunction with interactions of one or more users with one or more computers, such as computer 700 of Figure 7, and may be operated and configured such that trainable models may be improved based on the users' interaction with the training data and prior models, and may be implemented on various data in accordance with supervised and/or autonomous machine learning.

Figure 1 illustrates a neural node 100 according to one embodiment. In some embodiments, each node 100 may have a plurality of inputs 110, each with a weight 120, and a single output 130. The output of a node is calculated as:

where the output y of the kenth node is calculated as the sum of each input x of the node multiplied by a corresponding weight wkj for that input. The result of the sum is transformed by a transfer function. In some embodiments, the transfer function (p) may be a sigmoid function, represented by:

<>>5(9<=>T+F<

In some embodiments, each input may accept a value between zero and one, although other values may be used. Collectively, the inputs may be represented by the input vector x:

X = [ x QX1 ... Xn]

Each input also has a weight associated with it, with the collective set of input weights comprising the vector:

Figure 2 depicts a generalized neural network. This network comprises an input layer 210 with three nodes, a hidden layer 220 with four nodes, and an output layer 230 with three nodes. Nodes in the input layer 210 output a value, but do not calculate a value themselves. In one embodiment, the output may be a number between zero and one. Each node in the hidden layer 220 receives outputs from every node in the input layer 210, and k = ¥ .1. ^ )

and outputs the result of equation 10 given above. The output of the hidden layer 220 is then fed forward to the output layer 230. Each node in the output layer computes its output as the result >í = lP(X'¥i)

from the equation J 0 , which corresponds to the output of the network. This representation of a neural network is intended to aid in understanding neural networks and does not limit the disclosed technology. That is, a neural network may consist of hundreds, thousands, millions, or more nodes in each of the input, hidden, and output layers. Furthermore, neural networks may have a single hidden layer (as depicted), or they may have multiple hidden layers.

The weights for a layer of nodes can be further represented by the matrix:

w = [w0>0 w0>1 ■■w0jw 1)0 WU ; :WKo -wkJ ]

Therefore, the output for the node group can be calculated as:

y = (p(w*x)

A neural network may be used according to some embodiments to analyze an ordered list of linguistic units. A linguistic unit as defined herein may refer to a phrase, word, letter, or other character or characters used in the language. The neural network is configured to take the ordered list of linguistic units, with one linguistic unit omitted, and predict the omitted linguistic unit. This omitted linguistic unit is referred to as a "focus term." For example, Figure 3 depicts a neural network of one embodiment, which is not covered by the present subject matter of the claims and in which the neural network is analyzing the phrase "SEE SPOT RUN." The input nodes for "SEE" and "RUN" are turned on, to predict the focus term "SPOT." The neural network then properly predicts that the missing word is "SPOT" by returning 100% at the output node "SPOT."

The neural network 200 has an input layer 210 of nodes that encodes this input using "onehot" encoding. That is, there exists an input node for each linguistic unit in a dictionary of linguistic units that could be used. The dictionary need not be a complete dictionary, such as a complete English dictionary, but may exist as a subset of characters, words, or phrases used in the language. For example, if the ordered list of linguistic units is a word in the English language, the dictionary may be the list of letters A-Z, all uppercase (A-Z) and lowercase (a-z) letters, and punctuation marks, or some subset thereof. If the ordered list of linguistic units is an English sentence, the dictionary may include all or some words in the English language. For example, if the system is to be trained on a specific book or corpus of text, English words that are not used in the book or corpus may be omitted. In some embodiments, dictionaries may further include compound terms that include more than one word, such as “San Francisco,” “hot dog,” etc.

The neural network has a layer of hidden nodes 220. This layer of hidden nodes 220 is divided into partitions. When a layer of hidden nodes is divided into partitions, it is known as a partitioned embedding neural network (PENN). Each partition is related to a position in a sentence (a word before the focus term, a word after the focus term, etc.). This can be known as windowed embedding. Figure 4 depicts a windowed partitioned implementation that has two partitions, one for the word immediately before the focus term (p=+1) and one for the word immediately after the focus term (p=-1). The network is shown here, analyzing the sentence "SEE SPOT RUN", where the focus term is "SPOT". Here, three hidden nodes are used for the partition p=+1, and three hidden nodes for the partition p=-1. Again, as a result of inputting "SEE" to partition p=+1, and "RUN" to partition p=-1, the network predicts that the focus term is "SPOT".

In some embodiments, each partition may refer to a direction in the sentence (all words before the focus term, all words after the focus term, etc.). This may be referred to as directional embedding. As would be recognized by one skilled in the art, the approaches may be combined in numerous permutations, such as an embodiment having a partition for the linguistic unit two windows before the focus term, and a partition for all linguistic units after the focus term, and other permutations. Figure 5 depicts an embodiment with directional partitioning having a partition for all words before the focus term (P > 0), and a partition for all words after the focus term (P < 0). Here, the neural network is analyzing the sentence "SEE SPOT RUN FAST" where the focus term is "SPOT". As shown, the word "SEE" is input into the partition P > 0, and the words "FAST" and "RUN" are input into the partition P < 0. Then, the neural network returns the focus term "SPOT".

When partitions are used, a separate set of input nodes is used for each partition. Alternatively, in an embodiment not covered by the present subject matter of the claims, the set of input nodes can be modeled as specialized nodes capable of issuing a separate output to each partition. Both are mathematically equivalent. For ease of understanding, the present disclosure refers to sets of input nodes for each separate partition, but in each case where separate input nodes are used for each partition, a separate set of input nodes feeding different values to different partitions could be used in embodiments not covered by the present subject matter of the claims.

As discussed above, each node in each hidden layer partition will have a set of weights associated with it, represented by a vector x. In some embodiments, the vector x will have as many elements as there are input nodes associated with the partition. In some embodiments, some hidden layer nodes in a given partition may be fed from fewer than all of the input nodes associated with that partition. This can be modeled either as a vector x that has fewer elements than the number of input nodes associated with that partition, or a vector of equivalent length with the omitted term weights set to zero. Each hidden node may also have a bias term, a weight associated with an input node, that is either not multiplied by anything or multiplied by 1 and added to the result. The weights for all of the hidden nodes in a hidden layer partition can be represented by a matrix formed by concatenating the vectors for each node in the hidden layer of nodes. This matrix may be referred to as a "synapse" matrix. Because it is the first set of nodes in the input layer, it is known as synapse 0 (synO). This matrix can also be referred to as an "embedding matrix".

syn0 = [w l ... w k ] =[w<0)0>w<0>1>-w0Jw<1>0>wu : ;wK<0>-wkJ]

In some embodiments, there is only a single hidden layer, although in some embodiments there may be two or more hidden layers. In embodiments that have multiple hidden layers, there may be the same number of partitions as the previous layer, more or less. In some embodiments, an additional hidden layer may have no partitions at all.

The last hidden layer is followed by an output layer. In some embodiments, the output layer may have an output neuron associated with each language unit in a dictionary of language units. In some embodiments, there are fewer sets of output nodes than there are partitions in the previous hidden layer, which takes as inputs the outputs of more than one partition in the last hidden layer of nodes. In some embodiments, there is a single set of output nodes, each node receiving as input the output of all nodes in the previous layer. In some embodiments, there is a separate set of output nodes associated with each partition in the last hidden layer of nodes.

CLOW

In some embodiments, a neural network of the present disclosure may be trained using a continuous word list (CLOW) training style. The CLOW training style under the PENN framework optimizes the following objective function:

arg max¿(w,ÜC) ed -c< j U< c j^0 P(d,Q)(

C1

whereJ is the location-specific representation (partitionj) for the word at window positionj relative to the focus wordw.

In this style of training, the output layer is configured as a single set (or a single partition) of nodes that receive inputs from each node in the hidden layer above. The neural network is then trained on a language corpus, which is one or more sequences of a plurality of language units. One or more training examples are run, where for each training example, words surrounding the focus term are input into their respective input nodes that correspond to the appropriate partitions. For example, if the language corpus includes the phrase "SEE SPOT RUN FAST", and "SPOT" is the focus term, the input nodes, associated with position one ahead of the focus term, would be activated according to the term "SEE", and position one behind the focus term would be activated according to the term "RUN". Those inputs are propagated through the neural network to produce an output. The output of the neural network, which is correlated with the percentage probability that the output word is the focus term, is then compared to a preferred output where the focus term ("SPOT") is 100% (or a corresponding maximum output value) and the output for all other language units is 0% (or a corresponding minimum output value). The actual output is compared to the preferred output, and then backpropagated through the neural network to update the synapse matrices according to one or more known error backpropagation algorithms. This process can be repeated on additional sequences in the language corpus, until the network is sufficiently accurate.

In some embodiments, implementing directional CLOW with very small window sizes (pictured with a window size of 1) can be used to achieve acceptable performance. Directional CLOW is able to achieve a parity score using a window size of 1, in contrast to word2vec which uses a window size of 10 when all other parameters are equal. In some embodiments, this optimization can reduce the number of training examples and overall training time by a factor of 10.

SKIP-GRAM

In some embodiments, a neural network of the present disclosure may be trained using a "skip-gram" training style. The skip-gram training style optimizes the following objective function:

arg maxe((w, ICT)^d ~c< Uj<cj^0 p(c¡;Q)(w,C I)T^d'-c< j<üc j^0p(wj =O|d; <0>))

ci

where J is the location-specific representation (partition J) for the word at window position j relative to the focus word w.

In this style of training, a network is set up that has separate output partitions for each hidden layer partition. Thus, in the example listed above, the neural network would consist of two neural networks, one that models the probability of the focus term being a given value based on the word at position one ahead of the focus term, and one that models the probability of the focus term being a given value based on the word at position one after the focus term. In other words, if the language corpus includes the phrase "SEE SPOT RUN FAST", and "SPOT" is the focus term, one partition will receive as input "SEE", and one partition will receive as input "RUN". The corresponding output nodes will then generate an output. This actual output is compared to a preferred output where the focus term ("SPOT") is 100% (or a corresponding maximum output value) and the output for all other language units is 0% (or a corresponding minimum output value). The actual output is compared to the preferred output, and then propagated back through the neural network to update the synapse matrices according to any error backpropagation algorithm, as known in the art of neural networks. This process can be repeated on additional sequences in the language corpus, until the network is sufficiently accurate. After training, to arrive at a final result, the results of the various output partitions can be summed or otherwise combined to produce a final probability of the focus term.

In some embodiments, the skip-the-diagram training style can be used to train a neural network in parallel. That is, each separate hidden node partition and output node partition can be trained independently of other hidden node partitions and output nodes. Therefore, those embodiments could be trained using multiple threads on a single computer processor, or on multiple computers operating in parallel.

In some embodiments, when the jump diagram is used to model ordered sets of words under the PENN framework, each classifier partition and its associated embedding partitions can be trained in full parallel to achieve the same state as if they were not distributed. In some embodiments, training can be achieved by multiple computers with no intercommunication at all. In some embodiments, the windowed embedding setup can train all window positions in full parallel and concatenate embeddings and classifiers at the end of training. Given the machine/s, the following objective function is optimized:

arg max(ITp(cj;Q)T Ip(cj;Q))

(w,Q<Bd (w,Q<Ec/

where <c>j<;> is the location-specific representation (partition j) for the word at window position j relative to the focus word w.

The concatenation of the weight matrices s and s and s then incorporates the sum over j back into the PENN skip-gram objective function during the forward propagation process, yielding training results identical to a network trained in a single-threaded, single-model PENN skip-gram manner. This style of training achieves parity in training results with current state-of-the-art methods as long as it is trained in parallel on as many separate machines as possible.

DIEM

In some embodiments, syntactic tasks may be performed by a process called dense interpolation embedding (DIEM). In this technique, feature vectors for larger compound linguistic units (such as words) may be computed from embeddings generated on smaller linguistic units (such as letters). Because the structure of words or phrases correlates more with syntax than with semantic meaning, this technique works best on syntactic analogy tasks. To perform this method, feature vectors are first computed for the smaller linguistic units using a neural network according to one embodiment. A feature vector for a larger linguistic unit may be computed by interpolating the embeddings for the smaller linguistic unit onto the larger linguistic unit. For example, where the smallest linguistic unit is a letter, and the largest linguistic unit is a word, feature vectors for words may be generated according to the following pseudocode:

INPUT: word lengthI, list character embeddings (e.g. word)chart, multipleM, character dimensionalityC, vectorvm.

The final embedding size can be selected as a multipleMof the character embedding dimensionalityC, such that the final embedding containsMslices of lengthC. Within each slice of the final embedding, the embedding of each character is summed according to a weighted average. This weighted average is determined by the variabled above. This variable compares the squared percentage distance between the character positioni in the word of lengthI and the slice positionm in the sequence ofMslices. This is calculated such that the leftmost character is at a distance of approximately 0% from the leftmost slice, and the rightmost character is at a distance of approximately 100% from the rightmost slice. 1 minus this distance, squared, is then used to weight the embedding of each respective character in each respective slice. For example, the percentage distance across a word of length 4 is 0%, 33%, 66%, and 100% (from left to right across the word). The percentage distance across a sequence of slices of length 3 is 0%, 50%, and 100%. Therefore, the distance between character 1 and slice 3 is (0% -100%). Squared, this equals 100%. One minus this square is 0%. Therefore, character 1 in slice 3 would have a weight of 0% (nothing). The opposite would be true for character 4 and slice 3, since they both lie in the rightmost position in their respective sequences.

For each character in a word, its index i is first scaled linearly with the size of the final "syntactic" embedding such that s = M * i/I. Then, for each position C of length m (of M positions/sectors) in the final word embedding vm, a squared distance relative to the scaled index is computed such that distance d = pow(1 - (abs(s - j)) / M,2). The character vector for the character at position i in the word is then scaled by d and added element-wise at position j of the vector v.

In some embodiments, a dense interpolation embedding method may be more efficiently achieved by buffering a set of transformation matrices, which are buffered values of dij for variable-sized words. These matrices may be used to transform variable-length concatenated character vectors into fixed-length word embeddings via vector-matrix multiplication.

Syntactic vectors according to some embodiments also provide significant scaling and generalization advantages over semantic vectors. For example, new syntactic vectors can be easily generated for previously unseen words, giving lossless generalization to any word from the initial character training, assuming only that the word is composed of characters that have been seen. Syntactic embeddings can be generated in a completely distributed manner and only require a small vector concatenation and vector-matrix multiplication per word. Furthermore, in some embodiments, character vectors (typically length 32) and transformation matrices (at most 20 or more of them) can be stored very efficiently relative to semantic vocabularies, which may be several million vectors of dimensionality 1000 or more. In some embodiments, DIEM performs optimally using over 6 orders of magnitude less storage space, and over 5 orders of magnitude fewer training examples than word-level semantic embeddings.

The PENN framework models the probability that a word will occur given the words surrounding it. Changing the partitioning strategy modifies this probability distribution by capturing (or ignoring) various synergies. Modeling each permutation of the window position around a focus word approximates the full distribution, or the full conditional probability that the focus word will occur given the position and value of each and every word in the surrounding context.

Modeling each distribution allows all potential synergies to be captured in the combined embedding. The number of possible permutations in this partitioning strategy can easily exceed several hundred given a window size of 10 or more with each trained model containing billions of parameters. However, similar word partitioning strategies yield similar word embeddings, and by extension, similar word analogy quality in each subtask. Therefore, some realizations can approximate the varying perspectives generated using the full co-occurrence distribution with fewer trained models by training sufficiently different models and concatenating them. This method uses different probability distributions to generate word embeddings that model different perspectives on each word in the vocabulary. Character-derived vectors also incorporate a unique perspective by capturing information from the characters of a word rather than the contexts in which a word occurs. Because the embedding represents syntactic relationships between words from a unique perspective, the concatenation method according to the embodiments described above can be generalized to include DIEM vectors.

Analogy tasks

A related aspect of neural networks according to some embodiments is that the embedding matrix Syn0 can be used for analogy tasks. In the same way that each row can represent all the inputs to a specific neuron, each row represents the characteristic "embedding" of a specific word in the neural network. The embedding of a word can be a vector corresponding to all or a subset of the synapse weights for an individual partition, or all or a subset of all partitions, or variations thereof.

These feature word embeddings can be used to perform analogy tasks. For example, the analogy "king is to queen as man is to woman" can be encoded as king - queen = man - woman. That is, each word can be represented as a feature vector of embeddings, and each relation as a difference between the two feature vectors. Using the king/queen example, a neural network according to one embodiment can be used to derive the answer "woman" to the question "king is to queen as man is to what?" (king - queen = man - ?). The problem can be solved by rearranging the operations to man - king queen = ?. This results in a solution vector which can then be converted into an answer word by comparing the solution vector to the embedding vector for each word in the dictionary using cosine similarity. That is:

Looking up the eigenvectors of words in the dictionary, and finding a word with the smallest cosine distance (and therefore the highest degree of similarity), will usually result in finding the answer "woman". Depending on the network structure, the degree of training, other results may be presented, such as "girl" or "woman", however, "woman" is almost always among the most similar words.

The example given above is a semantic task, however syntactical analogies can also be made. For example, the question “running is to running as pruning is to?” can be similarly performed with a disclosure realization. Among the top results will be “pruning”, which shares a common syntax, albeit with a very different semantic meaning.

In some embodiments, word embeddings learned through neural language models can share resources across multiple languages. By pre-training in both languages, semantic relationships between the languages can be encoded in the embeddings. In some embodiments, for example, simple vector arithmetic of stop words between English and Spanish can approximate machine translation:

(“film”) - (“ is”) (“es”) = (“película”)

This vector operation subtracts an English stop word ("is") from an English noun and adds the equivalent Spanish stop word ("es") to the embedding. The equation returns the Spanish word for movie ("película"). This suggests that neural language pre-training maps words into the vector space such that language becomes another dimension to the distributed representations model. In some embodiments, this property of embeddings allows a sentiment model trained in one language to predict in another. In some embodiments, a sentiment trained in one language will better predict in another where there is a shared vocabulary between languages. In some embodiments, during pre-training, neural language models may build a hidden layer using neurons sampled according to vocabulary frequencies. In this way, the hidden layer construction for each language will be similar despite having completely different input layer representations. Consistency in the hidden layer may allow for alignment of semantic concepts in the vector space. This alignment may allow for sentiment prediction across languages.

Implementations and results

Implementations of various aspects of the disclosure and the corresponding results are described below. Some experimental data is presented herein for purposes of illustration and should not be construed as limiting the scope of the present disclosure in any way or excluding any alternative or additional embodiments.

The present inventors performed experiments on word analogy tasks composed of a variety of word similarity tasks. Specifically, the Google Analogy Dataset was used, which contains 19,544 questions, divided into semantic and syntactic sections. Both sections are further divided into subcategories based on the analogy type. Each analogy task is expressed as a question “Is A to B as C is to ?”.

All training was performed on the dataset available on the Google word2vec website (https://code.google.com/p/word2vec), using the packaged word analogy evaluation script. The dataset contains approximately 8 billion words collected from English News Crawl, 1-Billion-Word Benchmark, UMBc Webbase, and English Wikipedia. The dataset used leverages the default data-phrase2.txt normalization throughout training, which includes both individual tokens and phrases. Unless otherwise specified, all parameters for training and evaluation are identical to the default parameters specified in the default large word2vec model.

Figure 6 shows the relative accuracy of each partition in a PENN model as judged by the row-relative word analogy scores. Further experiments indicated that the pattern present in the heatmap is consistent across parameter settings. There is a clear difference in quality between the positions of the forward-predicting windows (left side of the figure) and the positions of the backward-predicting windows (right side of the figure). “Currency” achieves most of its predictive power in short-range predictions, whereas “common capital countries” is a much smoother gradient over the window.

These patterns support the intuition that different window positions perform different functions in different tasks.

Table A

Table A shows the performance of the default CBOW implementation of word2vec relative to the directional configuration. Most notably, PENN outperforms the word2vec implementation using only a window size of 1, whereas word2vec was parameterized with the default value of 10. Furthermore, it can be seen that increasing the dimensionality of the CBOW word2vec from benchmark 500 achieves suboptimal performance. Therefore, a fair comparison of two models should be between the optimal (rather than equal) parameterization for each model. This is especially important since PENN models are modeling a much richer probability distribution, since order is being preserved. Therefore, optimal parameter settings often require higher dimensionality. Additionally, from this table, it can be seen that, unlike the original CBO word2vec, a larger window size is not always better. Larger windows tend to create slightly more semantic embeddings, while smaller window sizes tend to create slightly more syntactic embeddings. This follows the intuition that syntax plays an important role in grammar, which is dictated by rules about which words make sense to occur immediately next to each other. Words that are 5 words apart are grouped based on subject matter and semantics rather than grammar. With respect to window size and overall quality, because partitions divide the global vector for a word, increasing the window size decreases the size of each partition in the window if the global vector size remains constant. Since each embedding is attempting to model a very complex probability distribution (hundreds of thousands of words), the partition size in each partition must remain high enough to model this distribution. Therefore, large window modeling for semantic embeddings is optimal when using the directional embedding model, which has a fixed partition size of 2, or a large global vector size. The optimally parameterized directional model has slightly lower quality than the optimally parameterized window model because the vector average is presented in each window pane.

Table B

Table B documents the change in syntactic analogy query quality as a result of the interpolated DIEM vectors. For the DIEM experiment, each analogy query was performed by first running the query in CLOW and DIEM independently, and selecting the top thousand cosine similarities from CLOW. We summed the squared cosine similarity of each of these top thousand with each associated cosine similarity returned by DIEM, and recursed. This was found to be an efficient estimate of concatenation that did not reduce quality.

The final results show an increase in quality and size over previous models, with a 40% syntactic increase over other results. Within the PENN models, there is a speed-performance trade-off between SG-DIEM and CLOW-DIEM. Unlike word2vec, where SG was slower and of higher quality, CLOW is the slower model and achieves a higher score. In this case, the present inventors achieve a 20x level of parallelism in SG-DIEM relative to CLOW, with each model training partition being 250-dimensional (250*20 = 5000 final dimensionality). A 160 billion parameter network was also trained overnight on 3 multi-core CPUs, however it produced 20000 dimensional vectors for each word and subsequently overfitted the training data. This is because a dataset of 8 billion tokens with a negative sampling parameter of 10 has 80 billion training examples. Having more parameters than training examples overfits a dataset, while 40 billion performs on par with the current state of the art.

Figure 6 shows the relative accuracy of each partition in a PENN model as judged by the row-relative word analogy scores. Further experiments indicated that the pattern present in the heatmap is consistent across parameter settings. There is a clear difference in quality between the positions of the forward-predicting windows (left side of the figure) and the positions of the backward-predicting windows (right side of the figure). “Currency” achieves most of its predictive power in short-range predictions, whereas “common capital countries” is a much smoother gradient over the window. These patterns support the intuition that different window positions play different roles in different tasks.

Table C

Table D

Tables C and D show the performance of the default CBOW implementation of word2vec relative to the directional configuration. Most notably, PENN outperforms the word2vec implementation using only a window size of 1, whereas word2vec was parameterized with the default value of 10. Furthermore, it can be seen that increasing the dimensionality of the CBOW word2vec benchmark past 500 achieves suboptimal performance. Therefore, a fair comparison of two models should be between the optimal (rather than equal) parameterization for each model. This is especially important since PENN models are modeling a much richer probability distribution, since order is being preserved. Therefore, optimal parameter settings often require higher dimensionality. Additionally, from this table, it can be seen that, unlike the original CBOW word2vec, a larger window size is not always better. Larger windows tend to create slightly more semantic embeddings, while smaller window sizes tend to create slightly more syntactic embeddings. This follows the intuition that syntax plays an important role in grammar, which is dictated by rules about which words make sense to occur immediately next to each other. Words that are 5 words apart are grouped based on subject matter and semantics rather than grammar. With respect to window size and overall quality, because partitions divide the global vector for a word, increasing the window size decreases the size of each partition in the window if the global vector size remains constant. Since each embedding is attempting to model a very complex probability distribution (hundreds of thousands of words), the partition size in each partition must remain high enough to model this distribution. Therefore, large window modeling for semantic embeddings is optimal when using the directional embedding model, which has a fixed partition size of 2, or a large global vector size. The optimally parameterized directional model has slightly lower quality than the optimally parameterized window model because the vector average is presented in each window pane.

Table B documents the change in syntactic analogy query quality as a result of the interpolated DIEM vectors. For the DIEM experiment, each analogy query was performed by first running the query on CLOW and DIEM independently, and selecting the top thousand cosine similarities from CLOW. The present inventors summed the squared cosine similarity of each of these top thousand with each associated cosine similarity returned by DIEM, and recursed. This was found to be an efficient estimate of concatenation that did not reduce quality. The final results show an increase in quality and size over previous models with a syntactic increase of 40% over the best published result. Within the PENN models, there is a speed-performance trade-off between SG-DIEM and CLOW-DIEM. Unlike word2vec, where SG was slower and of higher quality, CLOW is the slower model and achieves a higher score. In this case, the present inventors achieved 20x level of parallelism in SG-DIEM relative to CLOW, with each model training partition being 250 dimensions (250*20 = 5000 final dimensionality). A 160 billion parameter network was also trained overnight on 3 multi-core CPUs, however it produced 20000 dimensional vectors for each word and subsequently overfitted the training data. This is because an 8 billion token dataset with a negative sampling parameter of 10 has 80 billion training examples. Having more parameters than training examples overfits a dataset, while 40 billion performs on par with the current state of the art, as shown in Table E.

Table E

The present inventors performed additional experiments to approximate the full conditional distribution of word embeddings. These experiments were also performed on word analogy tasks in the Google Analogy dataset. This dataset contains 19,544 questions asking “is a to b as c is a?” and is divided into 14 subcategories, 5 semantic and 9 syntactic.

Training is performed on the 'data-phrase2.txt' dataset available on the Google word2vec website using the packaged word analogy evaluation script to query each individual model. Because querying each semantic model requires up to 128 GB of RAM, a normalized concatenation approach similar to that in the experiments described above was used. Each analogy was queried against each semantic model in parallel, and the top 1000 results were saved. The score for each word was summed across the various models to create an overall score for each word. The word with the maximum value was selected as the ensemble prediction. In our experiments, each score in each semantic model (PENN model) was raised to the power of 0.1 before being summed. This was found to normalize the models to similar confidence intervals.

For syntactic analogy queries, the top 100 words from the semantic step were selected to create syntactic embeddings. The analogy query was performed using the syntactic embeddings such that cosine distance scores were generated based on syntactic embeddings. These scores were summed for items with semantic scores in the original top 100 words. The top word was then selected as the final analogy response. This step was skipped for analogies coming from semantic categories. Syntactic scores were normalized by raising them to the power of 10.

The experiment described below approximates the full conditional distribution by concatenating embeddings from a variety of configurations. These configurations are shown in Table F below:

Table F

The results of these experiments are shown in Table G:

Table G

It should be noted that adding additional models to the ensemble increased the overall score, even when the model that was added already had a lower score than the ensemble. At a granular level, this is a result of the uniqueness of the model being added. The word analogy failures produced by the model are significantly different from the failures produced by the ensemble, such that their combination achieves a higher score than either one.

The final results show an increase in quality over previous models, with a 59.2% error reduction in syntactic scores and a 40.2% overall error reduction relative to word2vec. Training on semantic word embeddings based on varying probability distributions, normalization, and concatenation is beneficial for word analogy tasks, achieving state-of-the-art performance in semantic and syntactic categories by a significant margin. These results are based on the intuition that different distributions model different perspectives on the same word, the aggregate of which is more expressive than each embedding individually.

Experiments were further conducted using realizations to perform neural language modeling in different languages, all performed on the IMDB movie review corpus. The corpus consists of 100,000 movie reviews that are labeled for sentiment polarity (positive/negative). The reviews are divided into training, testing, and development sections. The training and testing sets have 25,000 reviews each with 12,500 positive and 12,500 negative reviews. The remaining 50,000 documents comprise the development set.

The present inventors used the Google Translate API to curate a corpus of Spanish language movie reviews. Systems such as the Google Translate API have been evaluated by research in the NLP literature and are generally accepted as a mechanism with which to curate labeled datasets in new languages [1][2][3]. For each review in the original English IMDB corpus, an API request was made to the Google Translate API to translate the full text of the review into Spanish. The resulting paragraph-level translations maintained the same polarity label as well as ID in order to maintain individual comparisons between the Spanish and English corpora.

The present inventors used the word2vec skip-word architecture to learn 100 dimensional feature representations for words in the corpus by using hierarchical softmax and negative sampling (with 10 samples) for the focus word representations. Preprocessing consists of lowercasing, separating words from punctuation, and removing HTML tags. After learning the word representations, stop words are filtered using the NLTK stop word corpus [4] and review-level representations are created by averaging the features of each remaining word in the review.

The present inventors used Paragraph Vector's distributed bag-of-words (dbow) configuration to learn 100-dimensional feature representations for reviews by using softmax and negative hierarchical sampling (with 10 samples) for the focus word representations. Preprocessing consists of lower-casing, separating words from punctuation, and removing HTML tags. Stop words are preserved in the reviews.

The present inventors used PENN's skip-gram architecture to learn 100 dimensional feature representations for words in the corpus by using negative sampling (with 10 samples) and no hierarchical softmax for the focus representations. Window position partitions -9 to -4 were used. The preprocessing and averaging steps are identical to the process described in the word2vec setup section.

Model assembly was performed using an augmented implementation of the recent work of [5]. For each corpus (English and Spanish) a validation word model and a full word model were trained. The validation word models (word2vec and PEN<n>) are trained on 20,000 reviews in the training sets (10,000 positive, 10,000 negative) and 50,000 reviews in the development sets. The remaining 5000 reviews in the training sets are used as validation sets. The full word models are trained on all training sets and development sets. Paragraph vector representations are learned using unsupervised pre-training on the entire corpus (training, development, and testing) using the implementation provided with [5].

Single-language models pre-train English and Spanish word models independently of each other. The English and Spanish feature representations are learned in separate unsupervised models. Their evaluations act as a benchmark against which to interpret the multilingual models in subsequent experiments. The revision-level representations are used as features in a support vector machine (SVM). A separate classifier was used for each language. The unsupervised models for English and Spanish never interact. The Spanish models were evaluated against the Spanish test set and the English models against the English test set. The resulting scores for these models are provided in Table H.

Table H

Pre-training Training Target Language Model<Percentage of>AccuracyWord2Vec 88.42

English English English<Paragraph Vector 88,55>

PENN 87.90Set 89.22 Word2Vec 87.31

Spanish Spanish Spanish<Paragraph Vector 85,3>

PENN 82.25 Set 86.63

The multilingual models are pre-trained using English and Spanish to learn representations. The English and Spanish feature representations are learned in the same unsupervised model. The revision-level representations are used as features in an SVM. The classifier is trained on English examples. No training is performed on the Spanish corpus. The features learned in the multilingual pre-training allow feature resources to be shared between both Spanish and English. The resulting models are evaluated on the Spanish corpus. The scores for these models are provided in Table I.

Table I

Pre-training Training Target Language Model<Percentage of>AccuracyWord2Vec 77.54

English Spanish English English<Paragraph Vector 86,02>

PENN 78.33Set 85.11 Word2Vec 80.08

English Spanish English Spanish<Paragraph Vector 78,86>

PENN 77.54 Set 85.51

Bilingual feature models assemble single-language models with multilingual models, the present inventors leverage ID mappings between English and Spanish corpora to assemble each combination of pre-training and training routines mentioned in the previous section. Additionally, n-gram models (without pre-training) were incorporated that are trained on English. The first n-gram model is an Averaged Perceptron with high affinity n-gram features (chosen via Person's as proposed in [6]). The second n-gram model is a Naive Bayes SVM as proposed in [7]. Maintaining IDs across languages allows to obtain multiple predictions for each review: some models predict on the English translation of the review, other models predict on the Spanish translation of the review, creating an assembled review sentiment score. This assembly exceeds the state of the art for IMDB polarity classification. These scores are provided in table J below: Table J

The present inventors' experiments demonstrate the effectiveness of neural language modeling for multilingual sentiment classification. Neural language models encode meaningful relationships between words by mapping them to vector space, and this property of their embeddings helps explain why multilingual models can effectively predict in languages they have not been trained on. The present inventors employ three different techniques for unsupervised pre-training and show that each technique is capable of encoding semantic relationships across languages. The present inventors trained single-language benchmark models to compare multilingual predictions against (Table H). The highest benchmark score for Spanish is 87.31% (word2vec) and the ensemble score is 86.63%. Using the benchmark scores for comparison, the present inventors evaluate multilingual models on the Spanish corpus. Without training in Spanish, the models are able to classify polarity with 80% accuracy, and when these same models are assembled together they classify polarity with 85.51% accuracy (Table I). Thus, the present inventors achieve a 2% margin between the baseline, the single-language models, and the multilingual models. The overall accuracy of the multilingual models increases by 5% when assembled. This behavior suggests that the pre-training for each language captures slightly different patterns and structures that ultimately help the classifier accurately predict labels for those examples near the decision boundary.

In addition to the examples described above for implementing various aspects of the present disclosure in various fields of application, aspects of the present disclosure may be implemented in numerous other fields, including, but not limited to, healthcare, for example, in real-time clinical monitoring and historical reporting on patient populations. Some embodiments described herein may be applied in clinical decision support and other tasks requiring unstructured text summaries. In an example use case, aspects of the present disclosure may be used to model the likelihood of a patient developing an infection, including a full-body systemic infection. More generally, aspects of the present disclosure may be used to predict whether or not a patient will develop a particular health condition. Electronic health records (EHRs) for multiple patients may be used with aspects of the present disclosure as described above to learn representations of node positions and use embeddings to predict whether a patient will develop an infection. The locations of particular clusters in the vector space may be representative of these probabilities of developing certain conditions.

7 is a computer architecture diagram of a computer system. The computer system includes a computer 700 that can be configured to perform one or more functions associated with the present disclosed technology. The computer 700 includes a processing unit 702, a system memory 704, and a system bus 706 coupling the memory 704 to the processing unit 702. The computer 700 further includes a mass storage device 712 for storing program modules 714. The program modules 714 may include computer-executable modules for performing the one or more functions associated with FIGS. The mass storage device 712 further includes a data store 716. The mass storage device 712 is connected to the processing unit 702 through a mass storage controller (not shown) connected to the bus 706. The mass storage device 712 and its associated computer storage media provide non-volatile storage for the computer 700. Although the description of computer-readable storage media contained herein refers to a mass storage device, such as a hard disk or CD-ROM drive, it should be appreciated by those skilled in the art that the computer-readable storage media can be any available computer-readable storage media that can be accessed and read by the computer 700.

By way of example, and not limitation, computer-readable storage media may include volatile and nonvolatile, removable and nonremovable media implemented in any method or technology for storing information such as computer storage instructions, data structures, program modules, or other data. For example, computer-readable storage media include, but are not limited to, RAM, ROM, EPRO<m>, EEPROM, flash memory or other solid state memory technology, CD-ROM, digital versatile disks (“DVD”), HD-DVD, BLU-RAY or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store the desired information and that can be accessed by computer 700. Computer-readable storage media as described herein do not include transient signals.

Computer 700 may operate in a networked environment using logical connections to remote computers via a network 718. Computer 700 may be connected to network 718 via a network interface unit 710 connected to bus 706. It should be appreciated that network interface unit 710 may also be used to connect to other types of remote computer networks and systems. Computer 700 may also include an input/output controller 708 for receiving and processing input from a number of input devices. Input devices may include, but are not limited to, keyboards, mice, styli, touch screens, microphones, audio capture devices, or image/video capture devices. An end user may use these input devices to interact with a user interface such as a graphical user interface to manage various functions performed by the computer 700. The bus 706 may allow the processing unit 702 to read code and/or data to/from the mass storage device 712 or other computer storage media. The computer-readable storage media may represent apparatus in the form of storage elements that are implemented using any suitable technology, including, but not limited to semiconductors, magnetic materials, optics, or the like. The program modules 714 include software instructions that, when loaded into the processing unit 702 and executed, cause the computer 700 to provide functions for neural language modeling in accordance with aspects of the present disclosure described herein with reference to example embodiments.

The program modules 714 may also provide various tools or techniques by which the computer 700 may participate within general systems or operating environments using the components, flows, and data structures discussed throughout this disclosure. In general, the program modules 714 may, when loaded into the processing unit 702 and executed, transform the processing unit 702 and the general purpose computer 700 from a general purpose computing system into a special purpose computing system. The processing unit 702 may be constructed from any number of transistors or other discrete circuit elements, which may individually or collectively assume any number of states. More specifically, the processing unit 702 may operate as a finite state machine, in response to executable instructions contained within the program modules 714. These computer-executable instructions may transform the processing unit 702 by specifying how the processing unit 702 transitions between states, thereby transforming the transistors or other discrete hardware elements that constitute the processing unit 702.

The coding of the program modules 714 may also transform the physical structure of the computer storage media. The specific transformation of the physical structure may depend on various factors, in different implementations of this disclosure. Examples of these factors may include, but are not limited to: the technology used to implement the computer storage media, whether the computer storage media is characterized as primary or secondary storage, and the like. For example, if the computer storage media is implemented as semiconductor-based memory, the program module 714 may transform the physical state of the semiconductor memory, when software is coded thereon. For example, the program modules 714 may transform the state of transistors, capacitors, or other discrete circuit elements that make up the semiconductor memory.

As another example, the computer-readable storage medium may be implemented using magnetic or optical technology. In these implementations, program modules 714 may transform the physical state of the magnetic or optical media, when software is encoded thereon. These transformations may include altering the magnetic characteristics of particular locations within given magnetic media. These transformations may also include altering the physical characteristics or features of particular locations within given optical media, to change the optical characteristics of those locations. It should be appreciated that other physical media transformations are possible without departing from the scope of the present disclosure.

References

[1] Balahur, Alexandra, and Marco Turchi. “Improving Sentiment Analysis in Twitter Using Multilingual Machine Translated Data.” RANLP 2013.

[2] Balahur, Alexandra, and Marco Turchi. ''Multilingual sentiment analysis using machine translation?.” Proceedings of the 3rd Workshop in Computational Approaches to Subjectivity and Sentiment Analysis. Association for Computational Linguistics, 2012.

[3] Wan, Xiaojun. “Using bilingual knowledge and ensemble techniques for unsupervised Chinese sentiment analysis.” Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2008.

[4] Bird, Steven. ”NLTK: the natural language toolkit.” Proceedings of the COLING/ACL on Interactive presentation sessions. Association for Computational Linguistics, 2006.

[5] Mesnil, Gr'egoire, et al. “Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews.” arXiv preprint arXiv:1412.5335 (2014).

[6] Manning, Christopher D. and Schutze, Hinrich. ''Foundations of Statistical Natural Language Processing.” 1999.

[7] Wang, Sida, and Christopher D. Manning. ''Baselines and bigrams: Simple, good sentiment and topic classification.” Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2. Association for Computational Linguistics, 2012.

Claims (11)

1. A computer-implemented method, comprising: receiving, at a plurality of input neural nodes of a computer-implemented neural network, an input comprising an ordered list of a plurality of linguistic units with one linguistic unit omitted, wherein each of the input neural nodes corresponds to a linguistic unit selected from the ordered list of a plurality of linguistic units and the linguistic units comprise phrases, words, letters or other characters used in the language; one-hot encoding, in the plurality of input neural nodes, the input received by the input neural nodes; receiving, at a plurality of neural nodes in each of a plurality of embedding node partitions in an embedding layer of the computer-implemented neural network, an input from a separate set of the input neural nodes, and generating an output by multiplying at least the input of each input neural node by one of a plurality of input weights, wherein each of the embedding node partitions corresponds to a position in the ordered list with respect to a focus term and comprises a plurality of neural nodes, wherein the focus term is a linguistic unit omitted from the ordered list of the plurality of linguistic units; and receiving, at each neural node in a classifier layer of the computer-implemented neural network, the output of each of the neural nodes of the embedding layer, to generate an output by multiplying at least the output of each neural node of the embedding layer by one of a plurality of input weights, and wherein the output corresponds to a probability that a particular linguistic unit is the focus term; where each neural node of a partition of the embedding layer is trained independently of other partitions; and wherein the plurality of input neural nodes, the embedding layer, at least one hidden layer and the classifier layer form the neural network and the method further comprises training the neural network by: train a first partition of the embedding node partitions by: remove the focus term from the ordered list of linguistic units; select a partition of the embeddings of each remaining linguistic unit based on the position of that respective linguistic unit relative to the focus term; and update the weights for each neural node in the classifier layer and the embedding layer such that the probability of the focus term's presence is approximately 100% and other randomly sampled linguistic units are 0%; and train a second partition by: remove the focus term from the ordered list of linguistic units; select a partition of the embedding for each remaining linguistic unit based on the position of that linguistic unit relative to the focus term; and update the weights to model the probability of the presence of the focus term is approximately 100% and other randomly sampled linguistic units are 0%, where the steps of training a first partition and training a second partition are performed in parallel on two different computers. 2. The computer-implemented method of claim 1, wherein the linguistic unit is a character or a word. 3. The computer-implemented method of claim 1, wherein the positions relative to a focus term of the embedding node partitions are window positions relative to the focus term or addresses relative to the focus term. 4. The computer-implemented method of claim 1, wherein the method further comprises training the neural network by performing functions comprising: remove the focus term from the ordered list of linguistic units; select a partition of the embeddings of each remaining language unit based on the position of that language unit relative to the focus term; concatenate the partitions; propagate the partitions through the classifier layer; and update the weights for one or more neural nodes in the classifier layer and the embedding layer such that the probability of the presence of the focus term is approximately 100% and other randomly sampled linguistic units is approximately 0%. 5. The computer-implemented method of claim 1, wherein the particular linguistic units are selected from the ordered list of the plurality of linguistic units and multiple linguistic domains associated with the selected particular linguistic units are modeled in a common vector space, wherein each of the multiple linguistic domains corresponds to a different language. 6. The computer-implemented method of claim 1, wherein the input weights for each embedding node partition are trained independently of the other embedding node partitions. 7. A system that has two computers configured to implement: a plurality of input neural nodes, wherein the plurality of input neural nodes is configured to: receiving an input comprising an ordered list of a plurality of linguistic units with one linguistic unit omitted, wherein each input neural node of the plurality of input neural nodes corresponds to a linguistic unit selected from the ordered list of a plurality of linguistic units and the linguistic units comprise phrases, words, letters or other characters used in the language; and one-hot encoding the input received by the input neural nodes; an embedding layer comprising a plurality of embedding node partitions, wherein each of the embedding node partitions corresponds to a position in the ordered list relative to a focus term and comprises a plurality of neural nodes, wherein the focus term is a linguistic unit omitted from the ordered list of the plurality of linguistic units, the plurality of neural nodes of each of the embedding node partitions configured to: receive an input from a separate set of input neural nodes; and calculating an output by multiplying at least the input of each input neural node by one of a plurality of input weights; and a classifier layer comprising a plurality of neural nodes, each neural node in the classifier layer configured to: receive the output of each neural node of the embedding layer; and generating an output by multiplying at least the output of each neural node of the embedding layer by one of a plurality of input weights, and wherein the output corresponds to a probability that a particular linguistic unit is the focus term; where each neural node of a partition of the embedding layer is trained independently of other partitions; and wherein the plurality of input neural nodes, the embedding layer, at least one hidden layer and the classifier form a neural network, the neural network trained by: train a first partition of the embedding node partitions by: remove the focus term from the ordered list of linguistic units; select a partition of the embeddings of each remaining linguistic unit based on the position of that respective linguistic unit relative to the focus term; and update the weights for each neural node in the classifier layer and the embedding layer such that the probability of the presence of the focus term is approximately 100% and other randomly sampled linguistic units are 0%; and train a second partition by: remove the focus term from the ordered list of linguistic units; select a partition of the embedding for each remaining linguistic unit based on the position of that linguistic unit relative to the focus term; and update the weights to model the probability of the presence of the focus term is approximately 100% and other randomly sampled linguistic units are 0%, where the steps of training a first partition and training a second partition are performed in parallel on the two computers. 8. The system of claim 7, wherein the linguistic unit is a character or a word. 9. The system of claim 7, wherein the positions relative to a focus term of the embedding node partitions are window positions relative to the focus term or addresses relative to the focus term. 10. The system of claim 7, wherein the particular linguistic units are selected from the ordered list of the plurality of linguistic units and multiple linguistic domains associated with the selected particular linguistic units are modeled in a common vector space, wherein each of the multiple linguistic domains corresponds to a different language. 11. The system of claim 7, wherein the input weights for each embedding node partition are trained independently of the other embedding node partitions.