13 minute read

Published:

# Understanding Word2Vec with PySpark

Gabriel Fair

### Goal

I need to use word embeddings to study the evolution of hate speech across social media. I chose to explore Word2Vec in hopes of learning more about it and to begin to probe the field of Natural Language Processing.

Today we are going to look at how Word2Vec incorporates word embeddings to create a numeric vectors to represent meaning of words. This is an important part of natural language processing (NLP). The goal of NLP is to extract meaning from human language, often this is provided in the form of text. And this meaning can be found in many components of language.

• Pragmatics
• Semantics
• Syntax
• Morphology
• Phonology

## Dataset

I am using a Gab.ai dataset of posts submitted to the social platform. Gab.ai prides itself on the values of “free speech” and a lack of censorship. As a result it has become known for attracting trolls, bots, and the socially maligned. Comments and posts made to this site are notorious for being extreme and hate laden. I have collected over 28 million posts and will use a 1 million post sample to train a skip-grams variant of the word2vec word embedding model. The goal is to identify the proximity of two related words in a vector space.

## Distributional Semantic Models

Word embeddings are word representation algorithms used in an NLP. Word embeddings are a subclass of distributional semantic models because they rely on the distributional hypothesis. The distributional hypothesis, created by Zellig Harris in his 1956 paper “Distributional structure” , is assumption that words in the same context tend to proport similar meanings and occur near each other. And thus synonyms have similar representations in a collection of texts. Word embeddings are represented as vector values created as a result of a neural network.

### Example Posts

data.body
"Probably because I see the faint hint of 'horns' holding that halo up... "
https://youtu.be/YMQRFT4bZuc
http://www.epochtimes.de/politik/europa/zahl-der-toten-nach-londoner-hochhausbrand-auf-79-gestiegen-2-a2146594.html
https://t.co/LTMBeXvHrC
"Ps 37:14 Die Gottlosen ziehen das Schwert aus und spannen ihren Bogen, daß sie fällen den Elenden und Armen und schlachten die Frommen.\nPs 37:15 Aber ihr Schwert wird in ihr Herz gehen, und ihr Bogen wird zerbrechen.\n\n"
At least 25 killed in airstrike on market in Yemen – reports\nhttps://www.rt.com/news/392838-saudi-yemen-market-airstrike/ #saudiarabia #yemen


#### Example after cleaning

['data', 'body']

['', 'probably', 'because', 'i', 'see', 'the', 'faint', 'hint', 'of', "'horns'", 'holding', 'that', 'halo', 'up', '', '', '', '', '']
['https', '//youtu', 'be/ymqrft4bzuc']
['http', '//www', 'epochtimes', 'de/politik/europa/zahl', 'der', 'toten', 'nach', 'londoner', 'hochhausbrand', 'auf', '79', 'gestiegen', '2', 'a2146594', 'html']
['https', '//t', 'co/ltmbexvhrc']
['', 'ps', '37', '14', 'die', 'gottlosen', 'ziehen', 'das', 'schwert', 'aus', 'und', 'spannen', 'ihren', 'bogen', '', 'daß', 'sie', 'fällen', 'den', 'elenden', 'und', 'armen', 'und', 'schlachten', 'die', 'frommen', '\\nps', '37', '15', 'aber', 'ihr', 'schwert', 'wird', 'in', 'ihr', 'herz', 'gehen', '', 'und', 'ihr', 'bogen', 'wird', 'zerbrechen', '\\n\\n', '']
['at', 'least', '25', 'killed', 'in', 'airstrike', 'on', 'market', 'in', 'yemen', '–', 'reports\\nhttps', '//www', 'rt', 'com/news/392838', 'saudi', 'yemen', 'market', 'airstrike/', '#saudiarabia', '#yemen']


Now we have to take this tokenized text and use this as our input text

## Building a Vector

To use the distributional hypothesis to build a vector, we have to choose what words being near each other means to us. This value of “nearness” is known as a window. In the image below, taken from Chris McCormick’s tutorial, the target word is highlighted in blue, and the window shown around it as being two words away from the target. This means the window size is equal to 2.

Word pairs are created between the target word and all other words in the window which can extend forwards or backwards. The target is then moved to the next word and the process repeats. Some embedding models treat words to the left of the target word differently than words to the right. But for now we will treat them both equally.

Figure 2: A diagram of the skip-gram model for starting with the target word and trying to predict the context words which are the words in the window: This image was taken from Tomas Mikolov et al’s original paper: Distributed Representations of Words and Phrases

These word pairs become the training samples for the model. We will be using the key of this pair to create what is known as a one-hot vector. Currently they are in the form of (target word, context-word-in-window) and this will be used as the input for a simple 1 hidden layer neural network. In Figure 2 above, the target word is W(t). The projection of the input onto a hidden layer of which has a pre-determined number of neurons that was specified as a hyper parameter. A hyper parameter is the number of hidden layer neurons has a large effect on the accacury and speed of the model’s runtime and 300 is widely used in practice since it was used by word2vec’s creators. This simple neural network is known as a Restricted Boltzmann Machine (RBM).

## Restricted Boltzmann Machines (RBMs)

In the image above, there are three columns that are known in discriptions of neural networks as layers. These diagrams show cause and effect between the layers of a neural network and are read from left-to-right. Each circle represents a neuron and is called a node. A node is where a calculation is preformed to determine if it will send a 0 or a 1 to a node in the next layer, which is to the right. This communication is known as firing and only happens in one direction, left-to-right.

In our restricted boltzman machine, nodes are not linked to, or communicate with, other nodes within the same layer. This restriction gives the RBM its name. And every node in the input layer is linked to each node in the hidden layer. The nodes/neurons in the input layer are considered to be different neurons in the hidden layer, hence why they are in different layers.

I stress this point because this is known as a bipartite graph. But not just any bipartide graph, a complete bipartite graph because these two layers are fully linked. Note, that some texts call this a symmetrical bipartite graph. Also it is important to notice in the graphic above how the hidden layer has fewer nodes than the input or output layers. This is an important quality of RBMs as a feature known as dimensionality reduction.

When a RBM is inalitized, four things are determined in advance and thus are hard-coded into the construction of the neural network. This things are known as hyper parameters.

• Number of nodes in the input layer
• Number of nodes in the hidden layer
• Number of nodes in the output layer
• The weights of the nodes in the hidden layer

With RBMs a special step happens when the hidden layer is created. Each node is randomlly assigned a weight. A wight is the power that node has on the nodes it is linked to in the next layer. This process of randomlly assigning weights is known as Stochastic Gradient Descent. It is called this because stocastic means “random” and these weights provide influence on the node in the next layer they are linked to.

The random assignment of the weights for word2vec is only done on the first round. The entire neural network is trained multiple times. Each one of these training rounds is called an Epoch. The weight matrix from the previous round is used as the initial weight table for the next round.

These weights are important to Word2Vec and like normal RBMs, Word2Vec randomly assigns weights. Word2Vec adjusts these weights over time while the neural network is fed our one-hot vector we created from our word pairs. This is known as training the neural network. Word2vec is similar to an autoencoder, as it encodes each word into a vector. But rather than training against the input words through reconstruction, as a restricted Boltzmann machine does, word2vec trains words against other words that are found in the context window of the input corpus.

Figure 4: This image was adapted from Chris McCormick’s Word2Vec Tutorial - The Skip-Gram Model tutorial

Figure 5: A diagram of how word2vec uses linear matrixes to train its neural network

### Understanding Figure 5, the linear algebra of Word2Vec

Above I explained how the network is trained many times. The same input is fed into the network each time but the weight matrixes from the previous round is used as the inital weight matrix for the next round. In Figure 5, this is matrix W.

So for our first round we will start with

### h=xTW

Since x is a one-hot encoded vector, h is simply doing a lookup of the kth row of the weight matrix W. Each row becomes a hidden layer for each word via the lookup trick provided by the one-hot encoding.

### yc,j= W′Th

so essentally the output element is just the transpose of the weight matrix between the input and hidden layer and the hidden layer and the output layer. so

### yc,j = W′TWTx

But there is one step missing from this equation above. The final output vector needs to be softmaxed. This takes the output layer (Y) and compresses the values into the range between 0 and 1. This allows for Y to act as a probability distribution for the input words (X). The equation for softmax is below:

We are using softmax since it continuous output provides a good use case for multiclass classification

I have trained using the method explained above. I used only 100 nodes in my hidden layer. The model would be more accurate if it was trained on a larger hiden layer. Often the size of this layer is 300 nodes.

## Explaining skip-gram

#### Create list of predicted words for the target word ‘trump’

print(model_vectors['trump'])

[0.012244515, 0.05669574, 0.4243817, -0.13005282, 0.14591245, 0.15754509, 0.12144023, -0.043377433, -0.036471862, -0.17795071, -0.15042417, 0.4285602, -0.16748007, -0.09618644, 0.07635299, 0.021112783, -0.1097202, 0.16649377, 0.31761286, 0.2781521, 0.26321766, -0.35739362, -0.17595355, -0.28173986, 0.2220869, 0.421465, 0.12334027, 0.17061687, -0.16097873, 0.1101991, 0.39143816, -0.10224187, 0.19060156, 0.06379647, -0.055479944, 0.30508712, -0.33523571, -0.3099334, 0.16205992, 0.23172502, 0.12932838, -0.25712037, -0.24778262, -0.41348562, 0.10876833, -0.095286794, -0.12277438, 0.08167293, 0.2416396, -0.29519707, 0.07202256, -0.03740526, -0.08972215, -0.03250894, -0.21824007, 0.04827257, -0.009086915, 0.18352096, -0.10135367, -0.47981852, -0.06576853, 0.021472175, 0.023349164, 0.05336668, -0.37836334, 0.08596835, -0.08231194, -0.09812828, 0.0058923, -0.06080334, 0.15352124, 0.2911331, 0.15038647, 0.15921666, 0.13570379, 0.09163106, 0.0093092015, 0.0024938602, 0.16191821, -0.116921216, 0.37449756, -0.37325835, 0.17355393, 0.22919315, -0.22791475, -0.12990569, 0.15548478, -0.16302991, 0.09529176, -0.124482594, 0.01942392, -0.18610963, -0.43775123, -0.2965226, -0.07572919, 0.2682866, 0.15111415, -0.03312072, 0.023581406, -0.035215713] --- > Number of total posts processed:  1,000,000


Number of words in the model: 116,568

Number of features per word: 100

## Now visualize the vector space using PCA and KMeans

Here I have to specify the number of clusters that Kmeans should use. A good approximation is to take the square root of half the number of words in the vocabulary list.

Size of the Word2vec matrix (words, features) is: (116568, 100)

Number of PCA clusters used: 241

The dimensions of the Word2Vec matrix: (116568, 100)

### Find cosine simularity between each word in the W matrix

Using cosine simularity we have the closeness of the word inauguration with the word trump.

nwords = 100
indexes = np.argpartition(dist,-(nwords+1))[-(nwords+1):]
di = []
for counter in range(nwords+1):
di.append(( words[indexes[counter]], dist[indexes[counter]], labels[indexes[counter]] ) )
print(di[2])

('inauguration/', 0.5486706985071742, 112)


### The top 100 words that are simular according to Word2Vec

ranked_results = unsorted_result.iloc[::-1] # order results from closest to chosen word
print(ranked_results)

                               word  similarity  cluster
100                           trump    1.000000      112
99                          trump's    0.729306      112
98                            elect    0.722010      112
97                            potus    0.705011      112
96                           donald    0.694213      112
95                          trump’s    0.688237      112
94                        president    0.687781      112
93                           trumps    0.686771      112
92                    clintonrussia    0.680471       35
91                           trump/    0.671885      112
90                       unverified    0.666894      128
89                 winner\r\nsearch    0.663887      118
88                           trump?    0.660687      112
87                           peotus    0.656724       35
86                    html\n\ntrump    0.650845       35
85                         remarks/    0.643584       13
84          video/?utm_medium=email    0.641376       35
83                           somers    0.638150      118
82                            pence    0.636276      112
81                    com/appalling    0.634154      118
80                         tv/watch    0.633391      144
79                             soci    0.632695       35
78                          peegate    0.628730       35
77                            obama    0.627831       93
76                            shole    0.627038      221
75                       ‘shithole’    0.624290       13
74                           feeley    0.623910       35
73                         trump?\n    0.618699       35
72                        netanyahu    0.617232      181
71                  schumershutdown    0.616409       35
..                              ...         ...      ...
29                       'president    0.565560       35
28              com/2017/10/winning    0.564972        0
27            com/2018/01/president    0.564370      218
26                          remark/    0.564070       35
25                 com/2017/06/mika    0.563695       17
24                              djt    0.563693      112
23                          sexist/    0.563230      118
22                     erin'strump®    0.562768      118
21          com/2018/01/13/pandoras    0.562444      112
20         com/2017/07/03/president    0.561896       17
19      office/2017/06/19/statement    0.560339      112
18               statements/remarks    0.559158      144
17   com/california/2017/01/07/dear    0.557056        0
16                      \npresident    0.555016      112
15      office/2017/06/29/statement    0.553720       82
14                          pledge/    0.553681      118
13                       com/donald    0.553214       83
12                   trump\n\ntrump    0.553146       35
11                     house/356849    0.552361       35
10                      haiti\nhttp    0.551846        0
9                      ‘resistance’    0.551726       35
8                             call/    0.551359       35
7        com/2017/06/28/condoleezza    0.551023      187
6                                 j    0.550414      112
5                          streep’s    0.550304       35
4                       trump\nhttp    0.549403      218
3              statements/president    0.548829       17
2                     inauguration/    0.548671      112
1                      nevertrumper    0.547763       13
0                     trump\n\nhttp    0.547598      218

[101 rows x 3 columns]


## Visualize Words using PCA

for i, word in enumerate(words):
ax.scatter(model[i, 0], model[i, 1], color='red', marker='o', edgecolors='black')
ax.text(model[i, 0], model[i, 1], model[i, 2], word)
plt.scatter(model[i, 0], model[i, 1], color='red', marker='o', edgecolors='black')
if(i > 50):
break


counter = 0
i = 0
plt.figure(figsize=(18, 16), dpi= 80, facecolor='w', edgecolor='k')
for counter in range(nwords+1):
word_lable = ranked_results.iloc[counter][0]
cosine_sim = ranked_results.iloc[counter][1]
assigned_cluster = ranked_results.iloc[counter][2]

plt.scatter(dist[indexes[counter]], labels[indexes[counter]], color='red', marker='o', edgecolors='black')
plt.annotate(word_lable, (cosine_sim, assigned_cluster))
if(i > 10):
break

plt.show()


## References and Credits

[1] Jorge Castanon at https://github.com/castanan/w2v


Tags: