Vectoring Words (Word Embeddings) - Computerphile

How do you represent a word to a neural net?

• using a set of characters is suboptimal as it uses most of the neural net’s capacity just learning what combinations of chars count as valid words.
  (but this can be solved by giving the net a dictionary as a jump start)
• neural nets “think” about words as a vector of floats.
• ex. to represent an image you would just assign 0 to white and 1 to black and whatever to gradients in between and represent the image as a vector. Is this the same as a bitmap?
• taking the same image and making it brighter is akin to making that vector “longer” or moving it further away from then origin. If I make it darker I would be moving it close to then origin, that is closer to 0 (black). Additionally, adding a little bit of noise would be “jiggling” that vector in space.
• thus two vectors that are close to each other are maybe close images, i.e. this representation preserves the semantic similarity?
• this is similar to how word embeddings work. A one-hot vector specifically takes one word in maybe a 10,000 dimension space and assigns that particular word one and others 0. But here you’re essentially saying just a “yes” or “no” to whether a word is “there” in a vector. Thus no semantic similarity is observed.
• ex. if you trained a model this way (i.e. one-hot vector) and asked it to complete a sentence, say, “I was playing with my pet ---” and ask it to complete it while aiming for “cat”, it may suggest “car” as it’s lexicographically (i.e. in the sense of similarity in dictionary order or letters but not in meaning) close but if it says “dog” you would want to “tell” it that it was close. Thethe input is 10,000 dimensions “tall” and it feeds into a hidden “layer” which is much smaller, i.e. 5 to several hundred. the output is again 10,000 “tall” and you do something like softmax to turn that into a probability distribution to get a “high” on one of the words (i.e. getting the highest value on one of the words). higher value = more confidence. one-hot reprsentation does not account for that.

What does it mean for two words being similar?

• two words are similar if they are used in similar contexts. Ex. “cat”, “dog”, “feed”, “pet”, etc.
• Challenge for word embeddings -> how do you represent words as vectors such that the directions should have some meaning and two similar vectors are two similar words
• The thing people realised was that if you had a language model which gave a good performance of predicting the next word in a sentence and the architecture was that it didn’t have many neurons, it had to be compressing words efficiently.

A simple example: taking a word and trying to guess the next word:

Think about the “hidden layer” here -> we are compressing the incoming word for efficiency. Perhaps trading exact meaning for comparison?
The way that they built this is rather than trying to predict the next word (that would give you word embeddings, but it’s not that good as it’s based on the adjacent word) you look “around” the word and sample from the neighborhood of that word and train from those. The expectation is that when you train it and give a certain input, it should give you a probability distribution over all of the words in the dictionary which is like “how likely are these words to show up within x words of this first word?”

If the system can get really good at this task, then the weights in that “hidden layer” have to encode something meaningful about that input word. So if “cat” comes in, the prob. distribution of surrounding words should look similar to the output for “dog”. So those two words should be closer together.

Rob Miles then goes into how it’s similar to how generative adversarial networks (GAN) does things - but I don’t know anything of it. In the latent space of words here how would adding/subtracting affect anything?

What does adding and subtracting of vectors mean in the latent space of words?

If you take the vector for KING and subtract the vector for MAN and the vector for WOMAN you get another vector and if you find the nearest point in your embeddings for that vector you get QUEEN. There are thus many ways the ideas of gender and what not are “encoded” here which are a cool exploration direction for you.

Refer: Google Colab for testing stuff out.
Rob Miles then shows a really cool example where he uses model.most_similar_cosmul() which I think is a word2vec “access thing”, with positive and negative parameters for adding and subtracting vectors. Ex. “London + Japan - England = Tokyo”