Definition
Take the vector for "king", subtract "man", add "woman", and the nearest word is "queen". In 2013 this simple arithmetic showed that a computer could learn something like meaning just by reading billions of words.
A word embedding represents each word as a dense vector of real numbers, typically 50 to 300 dimensions, learned so that words used in similar contexts get similar vectors. Similarity is measured with cosine similarity. Word2vec (2013, Tomas Mikolov and colleagues at Google), GloVe (2014, Stanford) and fastText (2016, Facebook, which also uses character pieces) are classic methods. They give each word one fixed vector; contextual embeddings from transformers such as BERT give a different vector for each occurrence, so "bank" by a river and a savings "bank" differ.
In simple words
You can guess the meaning of a new word from its neighbours: "I drank a cup of chai". Embeddings use this idea at huge scale, placing words on a map where similar words sit close together: "chai" near "coffee", far from "cricket".
Overview
- Word embeddings are dense vectors, often 100–300 numbers, that represent meaning.
- Words used in similar contexts get similar vectors (the distributional hypothesis).
- Cosine similarity measures how close two words are in meaning.
- Vector arithmetic can capture analogies: king − man + woman ≈ queen.
- Static embeddings give one vector per word; contextual embeddings change with the sentence.
- Embeddings learn biases present in their training text.
Why it matters
Embeddings fix the biggest weakness of bag of words: they know that "great" and "excellent" are related. They power semantic search, recommendation, clustering of documents and the input layer of every modern language model.
Key terms
- Distributional hypothesis
- Words that occur in similar contexts tend to have similar meanings.
- Dense vector
- A short vector of mostly non-zero numbers, unlike sparse bag-of-words vectors.
- Cosine similarity
- The cosine of the angle between two vectors: 1 for the same direction, 0 for unrelated.
- Word2vec
- A method that learns embeddings by predicting a word from its neighbours or the neighbours from the word.
- Contextual embedding
- A vector that depends on the surrounding sentence, produced by models such as BERT.
- Sentence embedding
- A single vector representing a whole sentence or document for search and clustering.
Types
Embeddings come in several types, from early static vectors to contextual and sentence embeddings.
1. Word2Vec
Static embeddings trained by predicting nearby words (Mikolov et al., Google, 2013).
- How it works: Skip-gram predicts context words from a word; CBOW predicts a word from its context; negative sampling makes training fast.
- Example: Vectors where "Delhi" − "India" + "Japan" lands near "Tokyo".
- When to use it: Use it for similarity, clustering and features in classic models.
2. GloVe
Static embeddings from word co-occurrence statistics (Stanford, 2014).
- How it works: It factorises a global matrix of how often words appear together, combining count-based and predictive ideas.
- Example: Pre-trained 300-dimensional GloVe vectors trained on billions of words.
- When to use it: Use it as an alternative to Word2Vec when ready-made English vectors are needed.
3. FastText
Embeddings built from character n-grams (Facebook, 2016).
- How it works: A word's vector is the sum of its subword vectors, so it can embed misspellings and words never seen in training.
- Example: Pre-trained FastText vectors for 157 languages, including Hindi, Bengali and Tamil.
- When to use it: Use it for morphologically rich languages and noisy text.
4. Contextual embeddings (ELMo, BERT)
Embeddings computed for each word in its sentence by a deep model.
- How it works: The same word gets different vectors in different contexts; BERT (2018) uses a transformer encoder.
- Example: "bank" in "river bank" and "bank loan" getting different vectors.
- When to use it: Use them for most modern NLP tasks where meaning depends on context.
5. Sentence and document embeddings
Single vectors for whole sentences or passages.
- How it works: Models such as Sentence-BERT (2019) and modern embedding APIs are trained so that similar meanings give similar vectors.
- Example: Finding which of 5,000 help articles best answers a customer's question.
- When to use it: Use them for semantic search, clustering and retrieval-augmented generation.
Explained in detail
How embeddings are learned
The linguist J. R. Firth wrote in 1957, "You shall know a word by the company it keeps." Word2vec turns this into a prediction game. In the skip-gram version, the model reads a huge corpus and, for each word, tries to predict the words within a small window around it. Words like "chai" and "coffee" appear next to the same neighbours, such as "cup", "hot" and "drink", so the training pushes their vectors to similar positions. GloVe instead factorises a table of how often words co-occur across the corpus. FastText builds each word vector from character n-grams, so it can produce vectors for misspellings and rare forms, which helps for languages with many word endings, and Facebook released fastText vectors for 157 languages, including many Indian ones.
Measuring similarity
Cosine similarity compares directions: cos θ = (a · b) ÷ (‖a‖ × ‖b‖). Related words such as "doctor" and "nurse" might score 0.7, unrelated ones such as "doctor" and "banana" around 0.1. Vector arithmetic captures some relations: king − man + woman lands near queen, and Paris − France + India lands near Delhi. These analogies work only for some relations and should not be over-interpreted.
| Type | One vector per | Example |
|---|---|---|
| Static word embedding | Word type | word2vec, GloVe, fastText |
| Contextual embedding | Word occurrence in its sentence | BERT, IndicBERT, MuRIL |
| Sentence or document embedding | Whole text | Sentence-BERT, LaBSE, embedding APIs |
Bias in embeddings
Because embeddings learn from human writing, they absorb its stereotypes. A 2016 study by Tolga Bolukbasi and colleagues found that word2vec vectors trained on news text completed the analogy "man is to computer programmer as woman is to homemaker". Such associations can leak into search ranking or résumé screening. Researchers measure these biases with association tests and reduce them by debiasing methods or more balanced training data, though no method removes bias completely.
Embeddings are also the foundation of retrieval-augmented generation. A collection of documents is split into passages, each passage is turned into a sentence embedding and stored in a vector database, and a user's question is embedded the same way. The passages whose vectors are closest to the question are retrieved and given to a language model, which writes an answer grounded in them. The quality of the embeddings therefore decides whether the right passages are found at all, so it is worth testing several embedding models on real questions in the languages your users speak.
How it works
- Choose pre-trained embeddings for your language, or a sentence-embedding model.
- Convert words or texts into vectors.
- Use cosine similarity for search and nearest neighbours, or feed vectors to a classifier.
- Check nearest neighbours of key words to see that results make sense.
- Test for harmful associations before using embeddings in decisions about people.
In code
import numpy as np
# Tiny hand-made 4-dimensional embeddings: [royalty, gender(+male), food, sport]
E = {
"king": np.array([0.9, 0.8, 0.0, 0.1]),
"queen": np.array([0.9, -0.8, 0.0, 0.1]),
"man": np.array([0.1, 0.8, 0.0, 0.2]),
"woman": np.array([0.1, -0.8, 0.0, 0.2]),
"chai": np.array([0.0, 0.0, 0.9, 0.0]),
"coffee": np.array([0.1, 0.0, 0.85, 0.05]),
"cricket": np.array([0.0, 0.1, 0.0, 0.95]),
}
def cosine(a, b):
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
print(f"chai vs coffee : {cosine(E['chai'], E['coffee']):.2f}")
print(f"chai vs cricket: {cosine(E['chai'], E['cricket']):.2f}")
target = E["king"] - E["man"] + E["woman"]
ranked = sorted((w for w in E if w not in {"king", "man", "woman"}), key=lambda w: -cosine(target, E[w]))
print("king - man + woman is closest to:", ranked[0])
def sentence_vector(text):
return np.mean([E[w] for w in text.split() if w in E], axis=0)
print(f"'chai' vs 'coffee': {cosine(sentence_vector('chai'), sentence_vector('coffee')):.2f} (bag of words would give 0)")Output
chai vs coffee : 0.99 chai vs cricket: 0.00 king - man + woman is closest to: queen 'chai' vs 'coffee': 0.99 (bag of words would give 0)
Try it yourself
Searching a help centre by meaning
A mobile-banking app's help search fails when users type words that are not in the articles.
Match the method to its idea
What is special about each?
- Word2vec
- GloVe
- fastText
- BERT
Common mistakes
- Using Euclidean distance on unnormalised vectors instead of cosine similarity.
- Expecting one static vector to handle words with several meanings.
- Ignoring stereotypes learned from the training text.
Check yourself
Summary
- Embeddings place words as dense vectors so similar words are close.
- Word2vec, GloVe and fastText give one vector per word; BERT-style models give contextual vectors.
- Use cosine similarity for meaning-based search, and check for learned biases.
Finished reading?
Sources: Mikolov et al. (2013), Efficient Estimation of Word Representations in Vector Space · Jay Alammar: The Illustrated Word2vec · Jurafsky and Martin, Speech and Language Processing (3rd ed. draft)