Skip to the yard
Waxsealai

Migrating Chroma Cloud Collections to Pinecone: A Cutover and Validation Guide

Shipped
Shipper
Michael Davis
Readers
999

A staged migration preserves existing vectors while checking data, filters and search quality before a Chroma Cloud collection moves to Pinecone.

Migrating Chroma Cloud Collections to Pinecone: A Cutover and Validation Guide

Moving a Chroma Cloud collection to Pinecone is a data migration and an application cutover, with different risks at each stage. Chroma Cloud stops accepting writes on November 22, 2026, and stops serving reads on December 22, 2026. A planned write pause gives teams time to export a stable collection, load it into Pinecone, and compare results before sending production traffic to the new index.

Aaron Kao

The process described here moves one collection that already contains dense embeddings, document text and metadata. It reuses those vectors, so it doesn’t embed the stored text again. The migration toolkit is a set of six Python scripts for preflight checks, index creation, export, conversion, loading and validation. It’s available in the migration toolkit repository.

Open-source Chroma remains available under Apache 2.0; the dates above apply to Chroma Cloud. The toolkit is provided as-is, and its results need to be checked against the requirements of the application being migrated. Treat a successful script run as evidence for review, not as a substitute for deciding whether the data and query behavior are acceptable.

Plan the migration around a write pause

The key operational requirement is to stop writes to Chroma before exporting. Reads can continue against Chroma during the migration, but writes need to remain paused until reads and writes can switch together to Pinecone. Without that pause, a record could change during export or a write could land in only one system.

Start by rehearsing the full sequence on a copy or a small collection. Measure export and load times in that rehearsal to estimate how long the production write pause will need to last; the toolkit doesn’t offer a universal duration because it depends on the collection and environment. Create the Pinecone index and prepare the application’s switching code before the pause begins.

The toolkit uses uv to install its pinned dependencies. From the repository root, set up the environment and provide credentials for both services:

uv sync
export PINECONE_API_KEY=...
export CHROMA_API_KEY=... CHROMA_TENANT=... CHROMA_DATABASE=...

The examples use a collection named my_collection, an index named my-index and a namespace named my-collection-v1; substitute your own names. Scripts that access Chroma read the three CHROMA_* variables. To work from a local open-source Chroma copy, pass --chroma-path PATH to the relevant scripts.

Teams using a coding agent such as Claude Code, Cursor or Codex can install the toolkit’s Agent Skills workflow with npx skills add pinecone-field/chromacloud-migration. The skill directs the agent through the same six scripts; the data checks and cutover decisions still need to be reviewed.

Translate the data model before moving records

A Chroma collection maps most naturally to a Pinecone index. When multiple collections have the same vector dimension and distance metric, they can instead share an index through separate namespaces. Chroma tenants and databases correspond broadly to Pinecone organizations and projects.

The stored fields need more attention. Chroma embeddings map to a Pinecone dense_vector field, named embedding in the toolkit’s schema. Documents map to a text field, while metadata becomes top-level document fields that are filterable. Pinecone metadata values must be strings, numbers, booleans or lists of strings; unsupported values and certain arrays require correction or an accepted transformation.

Chroma’s default l2 distance corresponds to Pinecone’s euclidean metric. Chroma cosine maps to cosine, and ip maps to dotproduct. Confirm the dimension and metric in the source collection before creating the index.

Chroma IDs that don’t meet Pinecone’s requirements—1 to 512 ASCII characters, excluding NUL—are replaced with generated IDs. The conversion records the mapping and retains the original ID in a chroma_id field. Metadata keys may also be renamed, especially keys beginning with _ or $, keys longer than 64 bytes, and names reserved for fields such as embedding and text.

Many metadata filters can be translated to Pinecone operators, including $eq, $ne, comparisons, $in, $nin, $exists, $and, $or and $not. For array metadata, Chroma’s $contains and $not_contains correspond to $in and $nin. Chroma document-content filters using where_document, and containment checks on non-array values, have no direct equivalent in this mapping; applications that depend on them need a query redesign. The toolkit README lists the expected data transformations and changes.

Inspect, create and prepare

Run preflight.py first. It scans the collection page by page and records the collection count, vector dimension, distance space, recorded embedding function, metadata keys and value types, empty or missing documents, and IDs Pinecone would reject. Findings are saved in preflight.json and classified as blocking, transformations, warnings or information.

uv run preflight.py --collection my_collection --workdir migration_work
uv run preflight.py --collection my_collection --workdir migration_work --allow-transforms

Review the initial report before accepting transformations. Blocking findings include all-zero vectors, mixed dimensions, metadata larger than 40 KB, unsupported types, and numeric or boolean arrays. Fix these in Chroma before pausing writes. Changes such as generating replacement IDs, renaming metadata keys or dropping None values require --allow-transforms; accepting that flag should be a deliberate decision.

If long document text pushes metadata beyond 40 KB, the toolkit offers a full-text-search path, or the text can be split in the source. If preflight identifies non-string arrays, --array-policy stringify accepts converting them to string lists. To inspect filters used by the application, provide --app-filters FILE containing a JSON list of named where or where_document filters. Use the same applicable policy flags for conversion.

After rehearsal, create the Pinecone index from the preflight results:

uv run create_index.py --index my-index --workdir migration_work

The script reads the vector dimension and metric from preflight.json and creates an index with one dense-vector field. By default, it uses AWS us-east-1; on Standard, --cloud and --region select a different location. Deletion protection can be enabled with --deletion-protection enabled. If the named index already exists, the script checks its schema and exits with an error if it doesn’t match.

For long text, use the full-text-search variant for preflight, index creation and conversion: add --text-mode fts to preflight and conversion, and --full-text-search to index creation. This path changes the index setup and how text is represented, so rehearse it as a complete path rather than mixing it with the standard commands.

At this stage, prepare one application configuration switch that routes both reads and writes to either Chroma or Pinecone. Keep it pointed at Chroma until validation passes. The Pinecone integration uses pc.index(name=...), index.documents.upsert for writes and index.documents.search for searches. Use the field names from preflight.json and account for any remapped IDs.

Export a stable snapshot and load it

When the production window starts, make the application read-only while reads still go to Chroma. Rerun preflight with the accepted transformation and policy flags. This second scan checks the collection immediately before export.

Then run export.py. It records collection.count() and streams pages from collection.get into export.jsonl, keeping the set of IDs in memory. It exits successfully only when exported unique IDs match the recorded count, no ID repeats, and the count stays the same throughout export.

uv run export.py --collection my_collection --workdir migration_work

Those checks can detect many interrupted or inconsistent exports, but comparing counts cannot reveal an update that leaves the count unchanged. The application-level write pause is what protects against that case. Keep export.jsonl as the archive copy.

Next, convert.py streams the export twice and produces the Pinecone documents plus mapping files. documents.jsonl holds one Pinecone document per line. field_map.json records metadata-key renames and why they happened; its mappings are deterministic and reversible. id_map.jsonl records changed IDs, while transforms.jsonl logs every per-record change or drop so nothing disappears silently. If any record fails conversion, the script does not write documents.jsonl.

For rejected IDs, the replacement has the form chroma-sha256-<hash>, with a suffix if needed to avoid a collision. The original ID is stored in chroma_id. Metadata names that need to move are given a meta_ prefix, with truncation and a hash suffix when required to meet the field-name limit or avoid a collision.

Load the converted documents into a fresh namespace:

uv run load.py --index my-index --namespace my-collection-v1 --workdir migration_work

The loader streams batches to Pinecone and sends a batch before it would exceed either 1,000 documents or 1,900,000 bytes, staying below the 2 MB request limit. It retries rate-limit, server, connection and timeout errors with backoff. The namespace must be new or empty. A rerun is allowed only when load_state.json confirms the same index, namespace and documents.jsonl checksum. Because upserts replace records with matching IDs, a permitted rerun retains the same content.

Validate search behavior before switching

Validation covers data integrity and retrieval behavior. Prepare golden queries using the same embedding function the application uses for Chroma queries; the toolkit doesn’t embed text. Store each query’s embedding in golden.jsonl, then pass that file to validate.py. If the application uses metadata filters, pass --app-filters app_filters.json as well.

The validator checks that the Chroma source count still matches the exported count, the live vector dimension and metric match the Pinecone schema, and the namespace reaches the expected count. It also compares the source ID set—after applying the ID map—with the IDs listed in the Pinecone namespace. For field checks, it samples 200 random documents and up to 200 records with per-record transformations, compares their fields with the source and tests whether they can be mapped back.

Search visibility is polled for up to 180 seconds, and the namespace count is polled for up to 600 seconds. Golden-query results are compared by top-k overlap, with tie adjustment: the mean must reach at least 0.9 and no query may fall below 0.6 by default. These are heuristics, configurable with --min-mean-overlap and --min-query-overlap, and should be judged against the application’s relevance needs. The optional --golden-from-sample N creates perturbed stored-vector queries for tests; it isn’t a substitute for real application queries.

The validator also runs a self-retrieval diagnostic: sampled vectors should find their own records. A miss doesn’t fail validation. It can point to duplicate vectors, unnormalized vectors with dot-product distance, close ties, approximate search behavior or data freshness. Filter checks compare translated filters against Chroma’s results; if only this check fails, the records may be sound while the application’s filter translation needs correction.

If a data check fails, stop and fix the issue, then load into a new namespace and validate again. An upsert-only pass for missing IDs cannot remove records that were deleted from Chroma. Once the checks pass, make an explicit cutover decision. If confidence is low, resume Chroma writes and schedule another window. Before the switch, no application writes have gone to Pinecone, so this is the point where returning to Chroma is simplest.

Cut over, monitor and retire the old service

When ready, change the application switch and resume writes together. Sending reads and writes to Pinecone at the same time avoids writing data to a store that the application’s reads can’t see. After the change, watch errors, latency and result quality. A rollback later would require replaying Pinecone writes into Chroma, and Chroma Cloud stops accepting writes on November 22, 2026.

If that date arrives before the application is switched, Chroma Cloud’s write restriction applies automatically. The export, load and cutover steps can still be completed before reads stop on December 22, but application writes will fail until traffic moves.

After validation and monitoring show the migration is working, retire the Chroma Cloud connection before December 22: remove Chroma client code from applications and jobs, confirm nothing calls its endpoints, and rotate or delete Chroma Cloud credentials. Archive export.jsonl together with field_map.json, id_map.jsonl and transforms.jsonl. The mapping files are needed to reverse field renames and ID changes.

Options after the data move

The migration retains the existing embedding model. A later move to Pinecone integrated embedding is a separate schema and data change: create a new index with a text field and an embed block, then reload stored text. Pinecone-hosted options listed for this path include llama-text-embed-v2 and multilingual-e5-large. Integrated embedding handles text at upsert and query time, removing the client-side embedding step, but search results will change. Evaluate relevance rather than expecting identical result IDs.

The described Pinecone Standard trial provides three weeks and $300 in credits, ending after 21 days or when the credits are used, whichever comes first. A trial started in October ends before November 22, so teams need to account for the plan transition when scheduling a cutover. Standard features listed for the trial include bulk import from object storage, backup and restore, role-based access control, cloud and region choice, higher limits and Developer Support. Afterward, continuing on Standard requires a payment method; the stated minimum usage is $50 per month, followed by pay-as-you-go charges. Other listed options are Enterprise or the free Starter plan. See the trial terms and pricing for details.

The practical takeaway is to make the migration decision from your own rehearsal and validation results. The dates set the outer window, but the collection’s transformations, query filters, write-pause needs and measured search quality determine whether the cutover is ready.

Source: Michael Davis · www.pinecone.io

Same vesselRelated cargo