Skip to the yard
Waxsealai

AI Agents Shift Production Costs Toward Data Infrastructure

Shipped
Shipper
Zeus Kerravala
Readers
840

Agent workloads multiply data requests and model calls. Caching, memory and retrieval can reduce waste, but teams need to test them at production scale.

AI Agents Shift Production Costs Toward Data Infrastructure

For AI agents, the model is only part of the production bill. A single task can trigger many queries, gather information from multiple systems, call a model repeatedly and write results back. As teams scale these workflows, the data layer can determine both how quickly agents respond and how much each task costs.

That shift is the premise behind Couchbase’s expansion of its AI Data Plane. The company added native Amazon Bedrock integration with semantic caching, a managed Agent Memory service and Model Context Protocol server on Amazon Web Services, plus vector search in its analytics engine, renamed Operational Insights. The product set reflects a broader operational question: how can teams support agents without paying repeatedly to retrieve and process the same context?

Agent traffic changes capacity planning

Tim Finley, Couchbase’s senior vice president of AI, used queuing theory to explain why agent workloads can strain systems faster than human activity. As demand rises, latency can grow faster than request volume. Agents add pressure because they can fan out into parallel work and issue many requests within a single task.

Finley cited a Cisco Systems study that found agents generate 450% more traffic per task than people performing the same work. He also pointed to Cloudflare data showing machine traffic surpassed human traffic on the web in June, about 18 months earlier than forecast. These figures describe different measures, but both point to a capacity challenge: infrastructure designed around people clicking, waiting and clicking again may behave differently under sustained agent activity.

“Capacity plans built for human growth curves are likely to break,” Finley said. He called agents “probably the nastiest, most demanding user that you can ever imagine when it comes to running a system.” The practical issue is not simply a larger number of users. Agents can issue concurrent queries, combine current and historical information, and continue operating outside the patterns of a human workday. A successful pilot can therefore expose a data bottleneck when usage expands.

Cut repeated work before downgrading models

One consequence of rising usage is pressure to replace a more capable model with a cheaper one. Finley said customers are “actively sacrificing intelligence by downgrading models because of the bill once they hit scale.” That choice can reduce the quality that made a project valuable in the first place.

Some of the expense may come from avoidable repetition: near-identical questions sent to a model, large prompts resent on each turn, or agents rebuilding context they already processed. Couchbase’s new features target those sources of overhead. Semantic caching can return a stored response for a sufficiently similar query instead of making another model call. The company says this cuts large language model calls by 40% to 60%.

Couchbase also claims Agent Memory reduces token usage by 33% to 45% compared with alternatives on the LoCoMo long-term conversational memory benchmark. Its vector-based context retrieval, the company says, produces prompts about 70% smaller than full-context approaches. These are vendor figures, and the savings will depend on the workload. Caching has more room to help when queries repeat; memory and retrieval gains depend on how much context agents would otherwise resend.

The useful measure is cost per completed task, alongside response quality. Teams can inspect how often prompts repeat, how much context travels on each turn and how frequently an agent reconstructs existing state. That evidence helps show whether caching, memory or retrieval can control spending before model changes affect results.

Fewer data hops can shorten the agent loop

Finley distinguished consolidation, which reduces the number of tools an organization pays for, from convergence, where several capabilities operate on shared data. His argument is that a shared data layer can simplify the path between an insight and an action.

Operational Insights’ vector search illustrates the idea. Finley described a phone manufacturer seeking to identify which factories are associated with the most positive battery-life reviews. A typical implementation might involve a relational database, a key-value store, a vector database, an ETL pipeline and a warehouse. Couchbase says its system can answer the question in one SQL query, using “cluster by” in a manner similar to “group by,” then write the result back for agents to retrieve.

“There has been a fallacy for a long time with analytics that a really great insight was the finish line,” Finley said. “We think it’s the action.” For an agent, the path from finding information to using it matters. Each system hop can add delay, require a synchronization job, introduce another access model and leave data out of date. When an agent acts on stale context, its answer may sound confident while being wrong.

Scale claims need workload-specific proof

Postgres with pgvector is a familiar default for many AI teams because it is accessible and widely used. Finley, who previously worked in AWS’ database business, acknowledged that appeal while arguing that the approach can become strained at Fortune 2000 scale, particularly when vector collections reach hundreds of millions. That is a vendor’s architectural position, not a universal threshold: performance depends on data shape, query patterns, concurrency and system configuration.

Couchbase points to deployments at LinkedIn, Amadeus and Mavenir. Brandon Larson, senior vice president and general manager of Cloud, AI & IMS Business Strategy, said Couchbase had supported high availability in Mavenir’s voice services and that the company is now using it to build a context layer for AI voice services, where reliable context retrieval is required.

Teams evaluating these approaches should load-test with agent-like traffic, including parallel requests and sustained bursts, rather than relying on human user counts. They should map every system and synchronization step in a typical task, then measure latency, throughput, retrieval quality and cost at their own data volumes. A result at 10 million vectors does not establish behavior at a billion.

Caches and memory also need the controls expected of production data: retention rules, user isolation, access limits and audit trails. Similarity thresholds require monitoring, since a cache that treats meaningfully different questions as equivalent can return an incorrect answer. Memory services should also work across frameworks such as LangGraph, CrewAI and LlamaIndex, so a framework change does not force a rebuild of accumulated agent context.

Model selection will remain part of AI economics, but agent costs increasingly depend on the work around inference. The next scaling decision should start with the full task: how many requests it creates, how much context it moves and how many systems it crosses. That is where teams can find whether data infrastructure is driving the bill—or holding back the agent.

Source: Zeus Kerravala · siliconangle.com

Same vesselRelated cargo