Skip to the yard
Waxsealai

Pinecone Adds Integrated Embedding and Full-Text Search to the Documents API

Shipped
Shipper
Robert Davis
Readers
929

Pinecone’s Documents API embeds text during writes and searches, combining semantic and keyword retrieval in one index and cutting median search latency in its tests.

Pinecone Adds Integrated Embedding and Full-Text Search to the Documents API

Pinecone’s Documents API moves two retrieval tasks into the database: it can embed text during writes and queries, and index that same text for full-text search. Developers send documents and search queries as text, while the schema determines how each field is stored and searched. In Pinecone’s tests, this single-call search path was about 3.5 times faster at the median than embedding a query through a separate endpoint before searching.

Silas Smith

The change reflects a broader product shift: users have often had to build the embedding and retrieval pipeline around a vector database themselves. The Documents API brings more of that work into one interface. It supports native vector search and keyword search in the same index, with integrated embedding now available as a field configuration. That makes vectors an internal representation the service manages for configured fields, rather than data every client must create and pass in.

The performance result has a specific boundary. For a client in the same region as the index, Pinecone measured a median of 26 milliseconds and a 90th percentile of 39 milliseconds for a text search on an embedded field. The path that first called a standalone embedding API and then searched with the resulting vector measured 92 milliseconds at the median and 109 milliseconds at the 90th percentile. Those figures include the benefit of avoiding an extra network round trip; applications running in other regions may see different results.

Documents API roadmap: integrated embedding is live, built on native vectors and full-text search, with more models, fusion, and other capabilities grayed out as coming next.
The Documents API combines native vectors, full-text search and integrated embedding, with further capabilities planned.

From vector plumbing to document fields

Pinecone’s original Vectors API accepts vectors to store and query vectors to search. That design leaves applications responsible for connecting an embedding provider, handling its authentication, rate limits, retries, billing and monitoring, and arranging separate flows for data ingestion and search. Many teams end up building the integration twice: once to embed records as they are loaded, and again to embed each incoming query with the same model.

Manish Talreja

That setup also dictates how data must be represented. Each record has an ID, a vector and metadata in a prescribed format. The text that produced the vector either stays in a separate store or gets placed in metadata. A separate store adds a lookup after search to retrieve the matching text. Metadata avoids that lookup, but it is designed for compact values used in filters, such as categories. Pinecone indexes metadata with Roaring bitmaps; storing long text there adds storage and write costs, consumes part of the per-record metadata limit and does not make the text searchable by its words.

With the Vectors API, your application calls an embedding provider twice, once for writes and once for queries, and keeps the source text in a separate store or in metadata.
The Vectors API leaves embedding and source-text storage to the application.

Pinecone’s effort to take on some of this pipeline work began with two infrastructure changes. In 2024, the company started hosting models through its inference service, with standalone embedding and reranking endpoints. Those let users call a model and the database through one provider and API key, but clients still had to invoke the model before each write and search. Running the models near the database made tighter integration possible.

A second piece was a storage redesign. In early 2025, Pinecone introduced slab architecture for serverless indexes, based on a log-structured design. Incoming writes first enter an in-memory buffer, then flush to immutable file sets called slabs. Background compaction can merge smaller slabs into larger ones. Each slab carries its data, its index and information describing the index type. This made it possible for the database to compose different index types, a foundation for adding full-text search beside vector search.

Gavin Johnson

Rolling the new architecture out across existing indexes also required migrating live data while customers continued to read and write. Pinecone describes that migration as a substantial part of its 2025 work. More detail is available in its guides to how Pinecone works and the slab architecture.

The Records API showed the text-in, text-out pattern

The first integrated-embedding design was the Records API, launched in late 2024. Users select a model when creating an index, then write text records and search with text. Pinecone performs embedding on the way into the index and on the way out for a query. The application handles text, while the database handles the vectors.

That API demonstrated the appeal of the pattern, but its underlying design limited what could follow. The Records API wrapped the Vectors API, so the source text still occupied a metadata field. An index was tied to one model and one text field, and users could not opt out of storing the text. Since the API inherited the vector-centric structure beneath it, it also lacked a natural way to add other search types, including native full-text search.

Amir Ingber

Full-text search required a different index and API

Vector search is useful when relevant results use different wording from the query. Full-text search handles cases where the exact terms matter, such as product codes, identifiers or error messages. Combining the two can cover more query types, but keyword search needs the text itself to be tokenized and indexed. It cannot be added simply by treating raw text as ordinary metadata.

Pinecone’s slab design could support index types beyond vectors, but that capability needed to be put into practice. The API also needed a way to describe documents, configure fields and express searches across different scoring methods. The resulting work involved three elements: native full-text indexing within the slab architecture, a flexible schema for indexing and query configuration, and a document-oriented API with a search interface that could expand over time.

Document indexes define typed fields. The initial field types support strings for full-text search, dense and sparse vectors for native vector search, and metadata for filtering. Search requests can score documents by keyword relevance over a text field or by semantic similarity over a vector field. The API accepts documents using application-defined field names, while the schema specifies how those fields should be indexed.

Harry Scholes

For full-text search, Pinecone stores the raw text once in a slab and builds a Tantivy inverted index alongside it. The inverted index maps terms to the documents and positions where they occur. Search results are scored with BM25, and text-match filters such as $match_phrase can require a word or phrase to appear in a document. Those filters can also constrain vector searches.

This storage model addresses a weakness of putting full text in metadata. The text is available to a search index built for words and phrases, while metadata remains available for filtering. The Documents API was designed around this document schema from the start, with full-text and vector search sharing the same underlying slab architecture. Integrated embedding, hybrid search, reranking and multiple vector fields were part of the planned direction for the interface.

Embedding is configured per string field

The new capability lets a schema configure both full-text search and embedding on a string field. For example, a body field can use stemming and stop-word handling for keyword retrieval, while also specifying the llama-text-embed-v2 model for semantic search:

Silas Smith
pc.indexes.create(
    name="space-missions",
    schema={
        "fields": {
            "body": {
                "type": "string",
                "full_text_search": {"stemming": True, "stop_words": True},
                "embed": {"model": "llama-text-embed-v2"},
            }
        }
    }
)

On a write, the client sends plain text. Pinecone embeds it using the model configured for that field, stores the vector beside the text and builds the full-text index as part of the same operation. If embedding fails, the request does not write its contents. For a semantic search, the client sends a text query and Pinecone embeds it with the same field model. The design puts model selection in the index schema and removes the need for the client to construct vectors for those operations.

One string field in a document index holds the text, its full-text index, and its vector, so text and embed searches both run against it.
A single string field can support both keyword and semantic search.

The API differences can be summarized by what the application writes, who handles embedding and how the text is stored:

CapabilityDocuments APIVectors APIRecords API
Write formatJSON documents with application-defined fieldsID, vector and metadataID, text field and metadata in JSONL
EmbeddingPinecone, configured per fieldApplication, outside PineconePinecone, one model per index
Text storageIts own string fieldSeparate store or metadataMetadata
Search modesFull-text, vector or text query embedded by PineconeVectorText query embedded by Pinecone
ConfigurationSchemaDimension and metricModel and text field

One field, two search behaviors

A space-mission index illustrates how the modes work together. Each document has a body field containing a short description, plus a year and a Boolean crewed value. Because those latter fields are not declared in the schema, they can be retained as filterable metadata. A search for the exact code “STS-61” can use text scoring on the body field and return the matching Hubble repair mission.

Adam Kelch
results = index.documents.search(
    namespace="missions",
    score_by=[{"type": "text", "field": "body",
               "query": "STS-61"}],
    top_k=2,
    include_fields=["*"],
)

A different request can search by meaning. Asked “Who was the first person to float outside a spacecraft?”, semantic search can match the Voskhod 2 description, even though the wording differs. A filter for crewed missions excludes uncrewed entries such as Voyager 1.

results = index.documents.search(
    namespace="missions",
    score_by=[{"type": "embed", "field": "body",
               "query": "Who was the first person to float outside a spacecraft?"}],
    filter={"crewed": {"$eq": True}},
    top_k=2,
    include_fields=["*"],
)

Both requests operate on the same field and index. The client sends text in either case and does not have to manage a query vector. The distinction is in the scoring clause: one matches terms, while the other embeds the query and searches by semantic similarity.

Request handling and consistency

For a semantic query, the Documents API calls the embedding service with the model declared by the field. The embedding client retries with backoff and spreads requests across healthy endpoints, defined in this context by low error rates and latency. Search embeddings are handled as small, latency-sensitive requests and kept separate from the large batches used for writes, so bulk ingestion does not hold up query embedding. Pinecone describes its load-balancing approach separately.

After embedding, the query router consults the index manifest, which is usually cached, to locate the relevant slabs. It sends the search to executors holding those slabs. Each executor keeps slab index data on local NVMe storage and in memory. New slabs are warmed on executors before the manifest includes them, so a search does not have to wait for a newly added slab to become warm.

Search path: the Documents API embeds the query, the query router reads the index manifest, and executors score slabs cached on local SSD and in memory.
The query is embedded before executors score slabs held on local storage and in memory.

The latency improvement comes partly from this request path: the application makes one call, and search can begin once the query embedding returns. In Pinecone’s same-region test, the Documents API search took 26 ms at the median and 39 ms at p90. Calling the standalone embedding API first and then searching took 92 ms and 109 ms, respectively. These are vendor-reported test figures, and the company notes that client and index locations affect results.

Writes follow a parallel path. The Documents API embeds batches of up to 96 documents. Text and vector are recorded together in the write-ahead log under the same log sequence number, then enter the memtable before being flushed into slabs. Those slabs hold the full-text and vector indexes side by side. As a result, both search clauses read the same data and share the same consistency guarantees.

Search latency from a client in the same region: 26 ms median and 39 ms p90 for Documents API search with text, versus 92 ms and 109 ms when embedding first and then searching with a vector.

Combining scores is still a separate step

The Documents API currently scores a search using one method at a time. An application that wants both keyword precision and semantic matching must issue two searches and combine the result lists itself. Their raw scores cannot simply be compared: BM25 and cosine similarity use different scales. In the example, the exact text match for “STS-61” scores 3.38, while the top semantic result for the question scores 0.43.

Teams therefore need to fuse results by rank or send the combined candidates to a cross-encoder reranker. Pinecone’s next planned step is a single search that combines full-text and embedding clauses and fuses their results. The company also plans a wider selection of embedding models, integrated reranking, import with integrated embedding, multimodal search and support for data zone processing. Those capabilities are planned; the available search behavior described here still requires separate calls when both scoring methods are needed.

Documents API roadmap: more embedding models and fusion are up next; reranking, bulk import embedding, multimodal search, and data zone processing are planned.
Planned additions include more embedding models, search fusion, reranking, bulk import embedding and multimodal search.

Choosing an API

For a new application, Pinecone positions the Documents API as the interface to build on: it brings together document schemas, native vectors, full-text search and integrated embedding. The integrated embedding guide covers index creation, document writes and a first search.

Applications that already supply their own vectors can still use them as native vectors in a document index, alongside full-text search over the same documents. Users of the Records API can retain its text-in, text-out interaction pattern while gaining full-text search; Pinecone says document indexes are more efficient for that use and that migration support is in development.

The practical change is a reduction in retrieval plumbing: the client can send documents and queries as text, while the schema connects stored text to both a full-text index and an embedding model. Search fusion remains an application concern for now, and latency depends on deployment geography, but the API puts more of the routine pipeline inside the database. As Pinecone adds fusion and other planned retrieval functions, the schema will determine how much of a production search flow can be configured in one place.

Source: Robert Davis · www.pinecone.io

Same vesselRelated cargo