Embedding

An embedding is a numerical vector representation of data (text, image, audio) that captures semantic meaning. Similar concepts have embeddings that are close together in vector space, enabling AI systems to measure similarity, cluster data, and retrieve relevant information.

What is an Embedding?

An embedding is a fixed-length numerical vector that represents a piece of data — most commonly text — in a high-dimensional space. The key property is that semantically similar items have vectors that are close together, as measured by cosine similarity or dot product distance.

How Text Embeddings are Created

Text embeddings are produced by neural networks trained on massive corpora. Models like OpenAI's text-embedding-3-small, Google's text-embedding-004, or open-source nomic-embed-text process input text and output a vector of typically 768–3072 float values.

Why Embeddings Are Fundamental

Almost every modern AI application relies on embeddings:

  • RAG systems use embeddings to retrieve semantically relevant documents.
  • Recommendation systems find similar items by comparing embedding distances.
  • Classification models use embeddings as input features.
  • Semantic search replaces keyword search with meaning-based search.

Embedding Models vs Generation Models

Embedding models output vectors, not text. They are separate from generative LLMs. You typically use an embedding model to index your data, and a generative LLM to produce the final response.

Dimensions and Trade-offs

Higher-dimensional embeddings capture more nuance but are slower and more expensive to store. Techniques like matryoshka representation learning (used in OpenAI's latest models) allow you to truncate dimensions without significant quality loss.

Example

The sentences 'The cat sat on the mat' and 'A feline rested on a rug' produce embeddings that are very close in vector space, even though they share no exact words.

Related terms

  • Vector Database A vector database stores and retrieves high-dimensional vector embeddings efficiently. It enables semantic similarity search — finding documents, images, or records that are semantically 'nearest' to a query vector. Examples include Pinecone, Weaviate, Qdrant, and ChromaDB.
  • Semantic Search Semantic search retrieves results based on the meaning of a query rather than exact keyword matches. Using vector embeddings, it finds content that is conceptually related to the query, even when no words overlap — solving the vocabulary mismatch problem in traditional search.
  • RAG (Retrieval-Augmented Generation) RAG is a technique that combines an LLM with an external knowledge retrieval system. Instead of relying solely on its training data, the model retrieves relevant documents at query time and uses them as context to generate more accurate, up-to-date responses.
  • Cosine Similarity Cosine similarity measures the angle between two vectors in high-dimensional space. In AI, it quantifies how semantically similar two embeddings are — a score of 1.0 means identical direction (highly similar), 0 means orthogonal (unrelated), -1 means opposite meaning.