Vector search is not semantic search

Vector search is often described as semantic search.

That is useful shorthand, but it is also misleading.

A vector database can find items whose embeddings are close to a query embedding.

That does not mean it understands what the user needs.

And it definitely does not mean the best result will be the nearest vector.

The distinction matters because many retrieval systems fail in exactly this gap.

What vector search actually does

Suppose we embed a query:

How do I cancel my subscription?

The embedding model turns that text into a vector.

Documents in our search index have also been converted into vectors.

Vector search then asks a geometric question:

Which document vectors are closest to the query vector?

Depending on the system, closeness might be measured using cosine similarity, dot product, Euclidean distance, or another metric.

The result is a ranked set of nearby vectors.

That is all the vector search layer knows.

It does not know:

• whether a document is current
• whether it belongs to the right product
• whether the user has permission to see it
• whether it actually answers the question
• whether a different result is more authoritative
• whether the document is duplicated
• whether the most semantically similar passage is useful at all

Vector search gives us similarity.

A search system needs relevance.

Those are not the same thing.

Similarity is only one signal

Imagine our search index contains these passages:

1. "Customers can cancel subscriptions from the account settings page."
2. "Subscription cancellations increased by 12% last quarter."
3. "The cancellation policy for enterprise subscriptions changed in 2024."
4. "To stop email notifications, unsubscribe using the link at the bottom of the message."

All four passages contain concepts related to subscriptions, cancellation, or unsubscribing.

An embedding model may place several of them close to our query.

But the user does not want something merely related to cancellation.

They want instructions.

That difference sounds obvious to a human.

It is not necessarily obvious from vector distance alone.

Retrieval has multiple jobs

A useful retrieval system usually has to answer several questions at once.

Is this result semantically related?

Is it the correct type of content?

Is it current?

Is it authoritative?

Is the user allowed to see it?

Does it match the user's exact intent?

Is it specific enough to answer the question?

These are different dimensions of relevance.

Vector similarity captures only some of them.

A production retrieval pipeline therefore tends to look less like:

query
  ↓
embedding
  ↓
vector search
  ↓
answer

and more like:

query
  ↓
query understanding
  ↓
candidate retrieval
  ├── vector search
  ├── keyword search
  └── metadata filters
  ↓
candidate ranking
  ↓
reranking
  ↓
context selection
  ↓
answer

Each stage removes a different kind of mistake.

Chunking changes what can be found

Before a document can be retrieved, it usually needs to be divided into chunks.

That sounds like an implementation detail.

It is not.

Chunking determines the units that your retrieval system is capable of returning.

Suppose a document contains:

Cancellation

Customers may cancel their subscription at any time.

Enterprise accounts must contact their account manager.

Refund eligibility depends on the billing plan.

If the entire page is embedded as one chunk, the representation mixes several ideas together.

If every sentence becomes its own chunk, each representation may become too narrow and lose important context.

There is no universally correct chunk size.

The useful unit depends on the shape of the content and the questions users are likely to ask.

This means retrieval quality is partly decided before the query ever happens.

Metadata carries information embeddings do not

Consider two documents with nearly identical text.

One belongs to Product A.

The other belongs to Product B.

Their embeddings might be extremely similar.

But if the user is asking about Product A, that product distinction may be the most important signal in the entire search.

Metadata lets us express information such as:

product = "A"
language = "en"
region = "UK"
document_type = "policy"
status = "published"
valid_from = "2026-01-01"

A retrieval query can then require:

semantic similarity
AND product = "A"
AND status = "published"

This is not a workaround for weak embeddings.

It is part of representing the actual search problem.

Some meaning exists in the text.

Some meaning exists in the system around the text.

Keyword search is still useful

Embeddings are good at finding conceptually related language.

Keyword search is good at exactness.

That matters more often than it first appears.

Queries frequently contain:

• product names
• error codes
• identifiers
• acronyms
• API names
• legislation
• account numbers
• feature names

If a user searches for:

ERR_PAYMENT_0042

we probably do not want a model deciding what error code sounds similar.

We want the exact token.

This is why many strong retrieval systems use hybrid search.

Vector search contributes semantic recall.

Keyword search contributes lexical precision.

The candidate sets are then combined and ranked.

The interesting engineering problem is rarely choosing one forever.

It is deciding how much each signal should matter for a particular workload.

Nearest does not mean best

Imagine vector search returns ten candidate passages.

The first result has a cosine similarity of 0.89.

The second scores 0.86.

It is tempting to assume the first result is better.

But those numbers tell us only how the embedding model positioned the vectors.

A three-point difference in similarity is not automatically a meaningful difference in usefulness.

A reranker can examine the query and candidate together and ask a more direct question:

Given this specific query, how relevant is this specific passage?

That is a different task from embedding generation.

The embedding model helps us efficiently find a manageable candidate set.

The reranker performs a more expensive but more precise comparison over those candidates.

Conceptually:

millions of documents
        ↓
cheap candidate retrieval
        ↓
20–100 candidates
        ↓
expensive reranking
        ↓
best few results

The first stage optimises recall and speed.

The second optimises precision.

Retrieval quality is query-dependent

There is another complication.

Different queries need different retrieval behaviour.

Consider:

What is our parental leave policy?

This may benefit from semantic retrieval over policy documents.

Now consider:

What does policy HR-142 say?

Exact lexical matching becomes much more important.

Now:

What changed in the parental leave policy this year?

Recency and document versioning become critical.

Now:

Can contractors in France use this policy?

Region, employment type, and access to authoritative policy data may matter more than raw vector similarity.

A single retrieval strategy will rarely be equally good for all four.

Production systems therefore often need some form of query classification, routing, or adaptive retrieval.

The embedding model matters

Vector search operates on embeddings.

If the embedding space does not represent distinctions important to your domain, no search algorithm can recover information that was never encoded properly.

For example, a general-purpose embedding model may understand that:

cancelling a subscription

and:

terminating a membership

are related.

But your application may need to distinguish:

• personal vs enterprise accounts
• active vs suspended subscriptions
• cancellation vs refund
• UK vs US policy
• internal vs customer-facing guidance

Whether those distinctions appear usefully in the embedding space depends on the model, the data, and the query.

This is one reason retrieval systems need evaluation.

Search needs an evaluation target

A retrieval system cannot be improved meaningfully until we decide what "good retrieval" means.

One useful starting point is a dataset containing:

query
expected relevant documents

For each query, we can then ask:

• Did we retrieve a relevant result?
• Was it in the top 3?
• Did we retrieve irrelevant distractions above it?
• Did filtering remove the correct answer?
• Did reranking improve the order?

Metrics such as recall, precision, mean reciprocal rank, and NDCG can help quantify these behaviours.

The exact metric matters less than having a repeatable way to test changes.

Without that, retrieval development often becomes:

"I tried five queries and this looks better."

That approach does not survive production complexity.

RAG inherits every retrieval mistake

This becomes particularly important when vector search is used inside a RAG system.

The language model can only reason over the context it receives.

If retrieval selects:

• the wrong document
• an outdated document
• an incomplete chunk
• contradictory passages
• irrelevant context

the generation stage starts from a bad premise.

A very capable model can sometimes compensate.

It can also produce a highly convincing answer from the wrong evidence.

That makes retrieval errors particularly dangerous: the final result may still read beautifully.

For many RAG systems, improving retrieval is more valuable than repeatedly changing the generation model.

Semantic search is a system

So what would I call semantic search?

Not vector search alone.

Semantic search is the wider system that tries to match the user's intent with the most useful information.

Embeddings can be an important part of that system.

So can:

• lexical search
• metadata
• filters
• query understanding
• reranking
• freshness
• authority
• permissions
• domain-specific rules
• evaluation

Vector search answers:

Which vectors are nearby?

Semantic search tries to answer:

Which information is most useful for what this person is actually trying to do?

That is a much larger problem.

A useful mental model

Think of vector search as a candidate generator.

Its job is not necessarily to find the final answer.

Its job is to reduce a huge search space into a smaller set of promising possibilities.

Then other parts of the system can apply more expensive, contextual, and domain-specific reasoning.

search space
    ↓
vector / lexical retrieval
    ↓
candidate set
    ↓
filters + reranking
    ↓
relevant context

That framing makes a lot of production retrieval design easier to reason about.

The nearest neighbour is not the answer.

It is a candidate.