Why AI Agents Need Context Beyond RAG and Semantic Search
- Cargo
- Machine Learning
- Shipped
- Shipper
- William Jones
- Readers
- 809
Relevant documents can still produce wrong answers. Agents need context that explains which sources are valid and how to use them.

An agent can retrieve every relevant document and still give the wrong answer. The failure may have nothing to do with embeddings or SQL: the system may lack the context needed to decide which information applies. That distinction matters more as agents act on retrieved answers without a person checking each result.

Consider a forecast for Q4 mobile growth. An agent queries the data and reports an increase of 16%, counting a one-day partner promotion on September 8 as ordinary demand. Exclude that promotion from the forecast while keeping it in reported totals, and the result becomes 6%. These figures are illustrative; the underlying data was present and the SQL was valid. The missing ingredient was a rule for interpreting the spike.
Retrieval finds matches, not authority
Jörg Schad, Pinecone’s VP of Engineering, used a November 4 webinar to trace the move from keyword search through semantic search and retrieval-augmented generation (RAG), then consider what agents need on top. The core limitation is simple: similarity tells a system that two things are related. It doesn’t tell the system which one is current, authoritative, or appropriate for a particular task.
Keyword search uses an inverted index and relevance ranking to find exact terms at scale. The foundations cited in the webinar include IDF, dating to 1972, and BM25, introduced in 1994. This approach can miss meaning expressed with different words: a query for “cancel,” for example, may not match a record that says “terminated subscription.”
Semantic search addresses that vocabulary gap. Methods and systems such as word2vec (2013), HNSW approximate nearest-neighbor search (2016), and BERT (2018) help retrieve content by meaning. But a search for how a feature works might return both a design document from three years ago and a scan of this morning’s code. Both can resemble the query; only one describes the code running today.
RAG, described by Lewis and coauthors in 2020, gives a model access to an external knowledge base at query time. That can ground an answer in retrieved material, but retrieval alone doesn’t verify that the returned material is the right version or that the model actually used it correctly.
Recall doesn’t measure whether the answer is valid
Retrieval systems are often evaluated with Recall@K: the share of relevant documents found among the top K results. Recall@10, for instance, measures whether relevant material appears in the first ten results. It answers an important question—did the information turn up?—but not whether the system can interpret it.
If a newer policy has replaced an older one, a search may return both. A relevance measure can count both as useful, even if an answer based on the superseded version is wrong. A similar gap arises when a document is relevant to a question but the person or agent asking it isn’t permitted to use it. In both cases, retrieval quality can look good while the resulting answer fails.
Organizational rules create another limit that a similarity score can’t resolve. After a merger, one company may define its fiscal year from January through December, while the other runs from February through January. A search can find the right figures from both companies, but it can’t infer how to align their quarters. The missing information is a mapping rule, not another document match.
These examples share a pattern: the index supplies material, while a separate decision determines how that material should be used. A promotion should be excluded from a forecast; a current code scan should take precedence over an old design document; conflicting fiscal calendars need an explicit mapping; and a superseded policy should not determine the answer.
Agents make context quality more consequential
People often notice when a forecast looks implausible and investigate. An autonomous agent may pass the result along or act on it without the same pause. Agents can also make retrieval requests at volumes that would be unusual for human search, increasing the chance that stale or mismatched results reach a downstream task.
One tempting response is to give the model more material and let it sort out what matters. Yet longer input doesn’t guarantee better answers. Chroma’s context rot report evaluated 18 large language models and found that performance became less reliable as input length increased, even on deliberately simple tasks. On LongMemEval, focused prompts averaging about 300 tokens outperformed full prompts averaging around 113,000 tokens. For agents, task-irrelevant information can make outcomes worse.
So the question extends beyond whether the right documents were retrieved. An agent also needs to know whether information is valid, trusted, permitted, and suitable for the task at hand.
A decision layer on top of search
Different tasks call for different search methods, and some benefit from combining them. Pinecone describes using dense vectors, sparse vectors, and full-text search with BM25 ranking in one index. Keyword matching remains useful when exact terms matter; vector search helps with meaning; hybrid search can draw on both. The webinar’s point is that these retrieval methods remain useful, but they don’t supply the rules for resolving conflicting or time-sensitive information.
The proposed addition is a task context: a collection of summaries, entities, policies, rules, and curated knowledge artifacts prepared before a query arrives. That context helps determine which artifacts matter, then directs lexical, vector, or hybrid searches as needed. For the merger example, it could hold the relationship between the two fiscal calendars alongside the relevant records. The same kind of context can identify which policy takes precedence or explain how to treat the promotion in a forecast.
Pinecone built Nexus as a knowledge engine for agents to provide this layer. Its curation process tracks processed sources by content hash, so a run can focus on what has changed. Answers include citations to the file, page, and verbatim span supporting each claim. Context instructions can also specify how to resolve conflicts, such as which policy version should apply.
Nexus won’t suit every workload. For straightforward retrieval, where the top ten results are likely to contain the answer and no complex precedence rules are needed, RAG may be enough. Workloads built on data that changes every second—such as individual purchases in a large online store—or records edited continuously may need a live, dynamic view. The described approach is a better fit for information that changes on a slower schedule, such as contracts and research.
Before spending time tuning embeddings or reranking, check whether retrieval already returned the necessary documents. If it did, the bottleneck may be the rules that decide what those documents mean for the task. The webinar replay is available on YouTube, with further details in the Nexus documentation.