Zilliz Adds Cost-Aware Benchmarking to VDBBench, the Open-Source Vector Database Benchmark

VDBBench already tests vector databases under real production workloads; The new release lets any team measure what a vector database actually costs to run in production — across data freshness, filtering, multitenancy, and cold starts — not just peak queries per second.

REDWOOD SHORES, Calif.--(BUSINESS WIRE)--Zilliz, a leading AI data infrastructure company and the creator of Milvus, recently announced a major update to VectorDBBench (VDBBench), an open-source, vendor-neutral benchmark for vector databases, adding cost as a first-class dimension alongside performance under production conditions — not just how fast a system performs in an idealized test.

Most benchmarks optimize for a single headline number: peak queries per second (QPS) on static, fully indexed data. But teams rarely choose a vector database on speed alone. They need to know what it costs to hit a target QPS, when newly written data actually becomes searchable, how filters and payload size change the query surface, how a system holds up across thousands of tenants, and how it responds on the first query after sitting idle. VDBBench now measures those dimensions directly.

"Most benchmarks answer one question — how fast can it go on ideal data," said James Luan, VP of Engineering at Zilliz. "But teams buy on total cost and real production behavior. We built VDBBench as an open, reproducible benchmark so any team can see what a vector database will actually cost and how it will behave on their own workload — and decide for themselves."

What's New in VDBBench

Building on VDBBench 1.0, which moved benchmarking closer to production by testing under streaming ingestion, filtering, recall, and concurrency, the latest release adds cost as a first-class dimension through four new cloud-oriented test cases:

• Insert readiness and write cost — separating when a write is accepted, when it becomes searchable, and when it is fully indexed, alongside the cost of loading data.
• Payload-aware search — how response shape (IDs, metadata, or full vectors) and filters change QPS, latency, and recall.
• Multitenant search — sustained throughput across many tenants and namespaces, as in SaaS workloads.
• Cold-start latency — the first query against a collection that has gone idle, versus the warmed path.

The results feed a new Cost Leaderboard with a cost-performance (Pareto) view that models operating cost at target QPS levels, showing where usage-metered serverless pricing is most efficient at low, spiky volume and where provisioned, flat-rate capacity becomes more cost-effective for the sustained traffic typical of production workloads.

Built to Benchmark Any Vector Database

VDBBench is open-source and supports more than 30 vector databases and search systems. To show the new cases in action, the Cost Leaderboard launches with a sample evaluation of several widely used managed vector databases — Pinecone, Turbopuffer, and Zilliz Cloud — chosen to illustrate how differently products can behave on freshness, filtering, multitenancy, and cold starts. The comparison is a demonstration, not a ranking: because the benchmark is open, teams can reproduce every case, swap in any other candidate, and run it against their own production-like data.

In the sample run, for instance, Zilliz Cloud returned newly inserted data as searchable immediately at a single-digit-dollar bulk-load cost, held a near-flat cold-start latency profile, and grew more cost-effective as sustained query volume rose — an illustration of the performance-and-cost balance the new cases are built to reveal. A product tuned for a different workload could show different strengths, which is exactly why every case is open to reproduce.

"This isn't about naming a winner," Luan added. "Different databases are built for different workloads, and the right choice depends on freshness, filtering, tenancy, budget, and many other factors. The results aren't ours to declare — they're yours to reproduce."

Availability

The VDBBench Cost Leaderboard is available now at zilliz.com/vdbbench-leaderboard-v2. VectorDBBench is open source on GitHub, where teams can reproduce the cases or benchmark their own candidates, and share results via GitHub or the Milvus Discord community.

For more details, read the VDBBench blog.

About Zilliz

Zilliz is a leading AI data infrastructure company and the creator of Milvus, the world's most widely adopted open-source vector database, with 45,000+ GitHub stars and over 100 million Docker pulls. Zilliz helps enterprises and AI startups make their unstructured data searchable, analyzable, and governable — turning text, images, audio, video, and more into a strategic asset for production AI.

Zilliz's technology centers on Milvus and Zilliz Cloud. Milvus is an open-source, lake-native vector database purpose-built for 100-billion-scale vector search. Zilliz Cloud extends that foundation into a fully managed Vector Lakebase platform, combining the high-throughput, low-latency serving capabilities of vector databases with the openness, scalability, and economics of multimodal data lakes. Zilliz powers more than 10,000 enterprises and AI-native startups worldwide, including MiniMax, OpenEvidence, Filevine, Exa, Salesforce, and Read AI.

Headquartered in Redwood Shores, California, Zilliz is backed by leading investors, including Aramco's Prosperity 7 Ventures, Temasek's Pavilion Capital, Hillhouse Capital, 5Y Capital, Yunqi Partners, and Trustbridge Partners. Learn more at Zilliz.com.