Lake-Native Vector Database: How Milvus 3.0 Kills the Double-ETL Problem

A lake-native vector database indexes and searches vectors that stay in open table formats on object storage — Parquet, Iceberg, Lance — instead of requiring you to copy those rows into a proprietary store first. The database owns the schema, the indexes and query execution. The lake keeps owning the data. Milvus 3.0, released 29 July 2026, implements this through a feature called External Collection.

If you have ever written a job that reads a warehouse table, embeds it, writes it into a vector store, and then spent a year keeping those two copies in agreement, that is the pipeline this architecture is aimed at.

Where these facts come from. Every version number, date, format identifier, configuration key and limitation on this page was read on 21 September 2026 from Milvus documentation, the Milvus GitHub releases API, Milvus engineering blog posts, or the LF AI & Data Foundation announcement. Nothing here was benchmarked or run by me. These are documentation-derived claims, not field results. Where the sources are silent — and on query latency for External Collections, they are silent — this page says so rather than estimating.

The 3.0 line is moving quickly: v3.0.2 shipped on 20 September 2026, the day before this page was written, and it changed External Collection behaviour. Check the current release before acting on any version-specific statement here.

What does “lake-native” mean for a vector database?

A lake-native vector database treats object storage as the system of record and adds a retrieval layer on top of it, rather than treating its own storage as the system of record and importing from the lake. Vector data stays in open formats on S3-compatible storage; the database manages schema, indexes and query execution over files it does not own.

Milvus states the position directly in its 3.0 announcement: vector data “can remain in open formats on object storage while Milvus provides production-grade indexing, retrieval, and APIs.” The LF AI & Data Foundation, of which Milvus is a graduated project, used the phrase “lake-native retrieval” in its own 29 July 2026 announcement.

The contrast with a standalone proprietary vector index is about ownership of the bytes, not about file-format support.

A distinction worth drawing, because vendors blur it: a vector database that can import Parquet is not lake-native. Nearly every vector store has a bulk loader that accepts Parquet. That loader still produces a second copy in a proprietary format, and that second copy is the thing that has to be kept in sync. Lake-native means the query runs against indexes built over the lake files themselves, with the source files never moving.

The trade that buys is not free, and the section on latency below is where it gets paid.

What is a double-ETL pipeline, and why is syncing into a vector store such a burden?

A double-ETL pipeline is the pattern where data is transformed twice to reach the same consumers: once to land it in the lakehouse or warehouse, and a second time to extract it from there, embed it, and load it into a separate vector store. Both copies then have to be kept in agreement forever. The second pipeline is the one nobody budgeted for and everybody maintains.

The shape is familiar enough that most teams stop seeing it as a problem:

source systems
      │
      ├── ETL #1 ──▶ lakehouse (Iceberg / Parquet on S3)
      │                     │
      │                     └── ETL #2 ──▶ embed ──▶ vector store (proprietary copy)
      │                                                     │
      └──────────────── BI / Spark / Trino read here        └── RAG reads here

Milvus’s own framing of the problem in its External Collection post is that in typical AI data pipelines, terabytes of embeddings and metadata already sit on object storage as Parquet, Lance or Iceberg tables, and copying that data into a vector database “doubles storage cost, adds an ETL pipeline that has to be kept in sync, and shifts data governance away from the customer.” That is a vendor’s description of a problem its product solves, so treat the framing as interested — but the three costs it names are the three that show up in practice.

Here is what each one actually looks like when you are the one on call.

Storage doubles, and the second copy is the expensive one. The lake copy sits in Parquet on object storage at object-storage prices. The vector-store copy carries the raw vectors plus the index structures, and it frequently lives on provisioned disk or in memory because that is what makes search fast. You are not paying twice for the same thing; you are paying once at cheap rates and once at expensive rates.

The sync job becomes a distributed-systems problem you did not want. It starts as a nightly batch. Then someone asks why a document edited at 10am is not retrievable until tomorrow, so it becomes an hourly job, then a CDC stream. Now you own ordering guarantees, deletes-that-must-propagate, retry semantics and a backfill path for the day the embedding model changes. The failure mode is the nasty kind: nothing crashes, retrieval just silently returns stale results, and the first report is a user saying the assistant “seems wrong lately”.

Schema drift arrives on someone else’s schedule. The upstream team adds a column or changes a type. Your embed-and-load job either breaks loudly or, worse, keeps running and quietly stops populating a field the retrieval filter depends on.

Governance splits in two. The lake has row-level access rules, retention policies, audit logging and a deletion path for the compliance team. The vector copy usually has none of that. For teams in India working under the Digital Personal Data Protection framework, a deletion request that must reach every copy of a person’s data is materially harder when one of those copies is an opaque index in a different system. This is an argument about where the data lives, and it is often the one that carries a migration decision further than the storage bill does.

And the second pipeline is load-bearing but unowned. The lake pipeline has a data-engineering team. The embed-and-load job was written by whoever built the RAG prototype, and it is now in production.

A lake-native architecture targets this second pipeline specifically. It does not remove ETL #1 — your lakehouse still has to be populated — and any honest description of the benefit says so.

Tap to enlarge

How does Milvus 3.0’s External Collection query Parquet and Iceberg without copying?

Milvus 3.0’s External Collection feature lets you define a Milvus collection over files that already exist in Parquet, Iceberg, Lance, Vortex or Milvus snapshot format, then query them with the standard Milvus search and query APIs. The source files never move. Milvus builds and serves vector, BM25, JSON and scalar indexes over the external data, and stores only the schema, the indexes and the metadata that maps fields to source columns.

Milvus 3.0.0 was released on 29 July 2026 according to both the GitHub release tag v3.0.0 and the documentation release notes, which state “Milvus 3.0.0 is officially released!” The GitHub API reports the tag with prerelease: false, published 2026-07-29T14:27:06Z. (One minor inconsistency in the sources: the Milvus announcement blog is dated 27 July 2026 and says “Today, we’re releasing Milvus 3.0”, two days before the tag and release-notes date. The release notes and the Git tag are the more precise artifacts, so 29 July 2026 is the date used on this page.)

The release notes describe the feature as letting you “query lakehouse data (Parquet, Lance, Iceberg, Vortex) in place: zero-copy, read-only” and “synced through incremental refresh.”

External Collection did not debut at GA. The v3.0.0 release body describes itself as “Building on the lake-native architecture introduced in 3.0-beta”, and says that in 3.0.0 “External Collection covers more lakehouse workflows”. The v3.0-beta tag was published 9 May 2026 and is marked as a prerelease. So the architecture is roughly four months old in public form, not two — which is mildly reassuring for maturity, and also means that anything you read about External Collection dated before 29 July 2026 describes the beta, not the GA feature.

Run a current patch release, not 3.0.0

The 3.0 line has moved since GA, and the patch releases matter more than usual here because several fixes land directly on External Collection correctness. As of 21 September 2026 the current release is v3.0.2, published 20 September 2026 (v3.0.1 preceded it on 9 September 2026; both are marked prerelease: false).

Changes in those two releases that bear on this feature, from the release bodies:

• v3.0.2 added support for custom S3-compatible endpoints in external collections via the extfs.endpoint_url option (PR #52814). If your object storage is MinIO, Ceph or any non-AWS S3 implementation, this is the release that supports it.
• v3.0.2 fixed “incomplete or incorrect output fields when search and query requests read external table data” (PRs #53372, #53385), and fixed external data file properties being dropped when building segment manifests, “causing external collections to lose source file metadata” (PR #53444).
• v3.0.1 fixed “inconsistent refreshed data during parallel external collection refreshes when a segment’s source files spanned multiple tasks” (PR #51893), and fixed external columns failing to refresh when an updated manifest was reopened (PR #52397).
• v3.0.1 also fixed unauthenticated access through streaming gRPC calls on the external proxy port (PR #52854) — a security fix, and reason enough on its own not to pilot on 3.0.0.

Read that list as a maturity signal in both directions. Correctness bugs in read paths and refresh consistency were being fixed six and seven weeks after GA, which is what a young feature looks like. They were also found and fixed quickly, with the fixes described specifically. Pilot on v3.0.2 or later. A benchmark run on 3.0.0 may be measuring bugs that are already fixed.

The supported format identifiers, exactly as the API expects them

These are the strings the external_spec format field accepts, taken from the Milvus 3.0.x API reference for refresh_external_collection() and the External Collection engineering post:

A note on that casing, because it will cost someone an afternoon: the Milvus 3.0.x API reference and the External Collection blog post both use hyphens (iceberg-table). One documentation page — “Create an External Collection” — rendered the Iceberg identifier as iceberg_table with an underscore when read on 21 September 2026. The two forms cannot both be correct. Use the hyphenated form, which is what the API reference specifies, and verify against the error message if a create call rejects it. This conflict is unresolved in the published documentation and is logged as such.

When the format is iceberg-table, a snapshot_id parameter is available to pin the collection to a specific Iceberg snapshot. That parameter applies only to Iceberg tables.

Prerequisite: Storage V3 must be turned on, and it is off by default

External Collection depends on Storage V3, the manifest-based columnar storage engine introduced in Milvus 3.0 under the internal name Loon, in which data and metadata both live on S3-compatible object storage and each dataset version is captured as an immutable manifest snapshot.

Two facts about Storage V3 that belong in any migration plan:

1. It is disabled by default in open-source Milvus 3.0. You enable it by setting common.storage.useLoonFFI to true in the Milvus configuration. External Collection cannot be used until it is on. Snapshot and TEXT fields depend on it too.
2. The change is effectively one-way. The documentation states that once Milvus writes data in Storage V3, downgrading to a Milvus version that cannot read Storage V3 is not supported, and that disabling it later neither converts existing Storage V3 data nor restores compatibility with older versions.

Point 2 is the one to take seriously. This is not a feature flag you can flip back after a bad week. Plan the rollback before the rollout, on a cluster you can afford to rebuild.

Defining and refreshing an external collection

The schema carries three external-specific fields:

• external_source — the URI of the external data, structured like an S3 object path.
• external_spec — a stringified JSON blob holding the format and the extfs filesystem settings.
• external_field — the mapping from Milvus field names to source file column names.

The extfs block is where credentials go. The documentation describes several authentication paths: AWS access key and secret (access_key_id, access_key_value, region, cloud_provider, use_ssl, use_virtual_host), AWS IAM (use_iam, iam_endpoint, region), reuse of Milvus’s own storage credentials via storage_type: "remote", and an IAM role ARN path (role_arn, external_id, load_frequency).

Refresh is the mechanism that picks up new data, and it is explicit. From the External Collection post: “Refresh is explicit. It is not a streaming synchronization mechanism.” You call it; Milvus does not watch the table for you.

The Python client signature, from the pymilvus 3.0.x reference:

refresh_external_collection(
    collection_name: str,
    external_source: str = "",
    external_spec: str = "",
    timeout: Optional[float] = None,
    **kwargs,
) -> int

It returns an integer job ID — the operation is asynchronous, so the call returning is not the refresh finishing. The documented behaviour is that refresh “scans the data files in the schema-defined external storage and generates metadata files that record their mapping relationship to those data files.”

The incremental part is what makes this cheap to run repeatedly. Per the External Collection post: “Refresh reads the external metadata, resolves the source fragments, updates the manifests” and “Milvus identifies source fragments that have not changed and can reuse their existing segment and index work.” Unchanged fragments are not re-indexed. An append-only Iceberg table that grew by one day’s partition costs you one day’s indexing, not a full rebuild.

Schema evolution is handled the same way rather than automatically: if the external source later contains another field you want, you add the field to the external collection schema and refresh again.

One documented refresh constraint: a refresh that would require removing all active metadata without any insertions is denied.

What External Collection does not do

Stated plainly, because these decide feasibility more often than the benefits do:

• External Collections are read-only. The documentation is unambiguous: “External Collections are read-only. Source changes happen outside Milvus.” You cannot insert, upsert or delete through Milvus. Every mutation goes through your lake writer.
• Write and CDC support are not available in this release.
• Refresh is not streaming sync. Freshness is bounded by how often you call refresh, and you own the scheduling.
• The source must stay reachable. Search, index and refresh behaviour all depend on continued storage access and valid credentials. A rotated key or a changed bucket policy is now a retrieval outage, not a pipeline outage.
• Non-AWS S3 endpoints need v3.0.2 or later. Custom S3-compatible endpoints for external collections arrived in v3.0.2 via the extfs.endpoint_url option. On 3.0.0, MinIO or Ceph as an external source is not supported.
• No metadata catalog integration is documented. The Milvus 3.0 primary sources read for this page name Iceberg tables and an Iceberg snapshot_id, but do not document integration with a REST catalog, AWS Glue or a Hive metastore. If your Iceberg access goes through a catalog rather than a direct table path, verify that against current documentation before planning around it — this page does not claim support either way.

The read-only constraint is the single most decisive fact in this release. It means External Collection is a retrieval layer over data you are already landing in a lake by some other means. If your RAG system’s write path is “the application inserts a document and it must be searchable immediately”, this feature does not serve that path, and no amount of refresh tuning changes that.

How does zero-copy affect query latency and storage costs compared with normal ingestion?

Zero-copy removes the duplicate storage copy and the sync pipeline, and in exchange introduces a remote-read cost on first access to cold data. Milvus has published no query-latency benchmark comparing External Collections against normally ingested collections, so the honest statement of the trade-off is directional, not numeric. The direction is: lower storage cost and less operational surface, paid for with cache-miss latency and a loss of write-path features.

On benchmarks, specifically. As of 21 September 2026 I could not find, in the Milvus documentation, the v3.0.0 release notes, the Milvus engineering blog or the LF AI & Data announcement, any published figure for External Collection query latency versus a native Milvus collection. Milvus 3.0 launch materials do carry performance numbers, and it would be easy to present them as if they answered this question. They do not, and the distinction matters:

Anyone quoting the 135x number as evidence that External Collections are fast is misreading it. It is a storage-format comparison.

The latency cost that is documented

The External Collection post states the cost directly: “A query that hits a cold field or index may pay a remote-read cost on first access.”

That sentence describes a cache-miss penalty. Data lives on object storage; the first query touching a field or index that is not resident locally has to fetch it over the network before it can answer. Object-storage first-byte latency is orders of magnitude worse than a local page cache — this is a property of the storage medium, not of Milvus — so a cold query is materially slower than a warm one. The primary sources do not quantify the penalty, and neither will this page.

Tap to enlarge

The documented mitigations, again verbatim: “Warm-up policies can preload latency-critical fields or indexes, while cache and eviction policies keep less frequently accessed state from occupying local resources indefinitely.”

So the operational reality is a working set, with the usual consequences: warm queries on a well-sized cache should behave like normal Milvus queries because they are served from the same local state, while queries that stray outside the warm set pay the fetch. Your p50 may look unchanged after migration while your p99 gets worse, and a tail-latency SLO is where this will show up first. Load-test the cold path deliberately, because a benchmark run twice in a row measures your cache, not your architecture.

There is also a second-order cost that is easy to miss in a cloud bill: cache misses are object-storage GET requests. High-concurrency, low-locality query patterns turn into request charges and egress that a self-contained vector store would not have generated. That is a real cost moved, not a cost eliminated.

Where External Collection loses

Milvus publishes the negative case, which is unusual enough to be worth quoting in full. A normal Milvus collection is still the better choice when:

• the application continuously inserts or upserts records;
• deletes need to become visible through the online write path;
• the workload depends on collection features unavailable to external schemas;
• the serving design intentionally keeps all required data in memory, avoiding remote cache misses.

That last bullet is the clearest statement of the trade in the entire documentation set. If your design is “hold everything in memory and never touch the network during a query”, zero-copy is not an optimization — it is a regression, because you have reintroduced a network hop that your architecture existed to remove.

The storage-cost side

The storage arithmetic is simpler and moves the other way.

Eliminated: the duplicate copy of source rows and metadata in the vector store’s own storage.

Retained: the index structures. Milvus still builds and stores vector, BM25, JSON and scalar indexes. Indexes are not free, and on high-dimensional vectors the index can be a significant fraction of the data size. “Zero-copy” refers to the source data, not to the total storage footprint.

Added: local cache capacity, sized to your warm set, plus the object-storage request volume noted above.

So the honest version of the storage claim: you remove one full copy of the source data and keep the indexes, which is a real reduction, larger for text-heavy and metadata-heavy collections and smaller for collections that are mostly raw vectors with thin payloads. Milvus’s “doubles storage cost” framing for the copy-based approach is a vendor characterization of the problem; the size of your own saving depends on your payload-to-vector ratio, and it is arithmetic you can do on your own table before you migrate.

One cost that is genuinely eliminated rather than moved: the sync pipeline’s compute and the engineering time that maintains it. If ETL #2 runs embeddings on a schedule, that compute does not disappear — embeddings still have to be produced somewhere — but the extract-and-load half of the job, and the on-call surface attached to it, does.

Should teams migrate existing vector workloads to a lake-native architecture now, or wait?

Recommendation: do not migrate an existing production vector workload to External Collections today. Do pilot it now, on a real dataset, if your retrieval data already lives in Iceberg, Lance or Parquet and your workload is read-heavy and batch-refreshed. The feature is GA and the architecture is sound, but External Collection correctness bugs were still being fixed in v3.0.1 and v3.0.2 through September 2026, the feature is read-only, and it requires a storage-engine change you cannot cleanly reverse. That combination argues for evidence before commitment, not for waiting on the sidelines.

The reasoning, and the conditions that would change the answer.

Why not a production migration yet. Four specific facts, not general caution:

1. The Storage V3 switch is one-way. Once Milvus has written Storage V3 data, downgrading to a version that cannot read it is unsupported, and turning the flag off does not convert the data back. Any migration plan whose rollback step is “disable the flag” is not a plan.
2. Read-only removes the write path, not just a feature. If anything in your system writes to the vector store — live document ingestion, feedback loops, deletion-on-request — that path has to be rebuilt against the lake first. That is a data-engineering project in its own right, and it must land before the retrieval migration, not alongside it.
3. The performance profile is undocumented for this specific comparison. No published benchmark answers “how much slower is my p99 after this change”. You can only find out by measuring your own workload, which is an argument for a pilot rather than an argument for delay.
4. The read path was still being corrected in September 2026. v3.0.2, published 20 September 2026, fixed incomplete or incorrect output fields when search and query requests read external table data. A feature whose result-correctness fixes are seven weeks old is not one to put a production retrieval workload behind this month.

Why not wait, either. The pattern is not speculative. Milvus 3.0 is GA, Milvus is a graduated LF AI & Data Foundation project, and the direction — retrieval engines reading open table formats in place instead of demanding a private copy — follows the same path the query-engine world already took with Iceberg. Teams that pilot now will have real numbers when the write path arrives. Teams that wait will be making the same decision in a year with no more internal evidence than they have today.

Migrate now if all of these hold:

• Your retrieval corpus already lives in Iceberg, Lance, Parquet or Vortex on object storage.
• Your vector workload is read-heavy, and updates arrive in batches rather than continuously.
• Your freshness requirement is measured in minutes or hours, not seconds, so an explicitly scheduled refresh is adequate.
• Your duplicate copy is large enough that removing it is a line item someone would notice.
• You can run Milvus 3.0 with Storage V3 on infrastructure you are willing to rebuild.

Wait if any of these hold:

• Your application writes to the vector store on the request path.
• Deletes must become visible immediately through the vector store.
• You run a latency-critical service with a strict tail-latency SLO and a working set too large to keep warm.
• Your Iceberg access is mediated by a catalog whose support you have not confirmed against current documentation.
• Your team has no capacity to own a storage-engine migration this quarter.

A pilot worth running, in the order that surfaces problems earliest:

1. Pick one read-heavy collection whose source is already an Iceberg or Parquet table.
2. Stand up Milvus v3.0.2 or later on infrastructure you can discard, with common.storage.useLoonFFI enabled. Do not pilot on 3.0.0 — see the patch-release note above.
3. Define the external collection against the real table, at real scale. Small test data will not exercise the cache behaviour that decides this.
4. Measure cold-query latency deliberately — clear or bypass the cache between runs — and measure warm latency separately. The gap between those two numbers is the decision.
5. Run a refresh against a table that has genuinely changed, and time it. Confirm that unchanged fragments are being reused rather than re-indexed.
6. Compare storage: lake bytes plus Milvus index bytes, against your current lake bytes plus full vector-store bytes.
7. Rehearse the failure you will actually have — rotate the storage credential and confirm what a query does when the source becomes unreachable.

Step 7 is the one most pilots skip, and it tests the property that changed most: your retrieval availability is now coupled to object-storage access.

The broader direction here is the same one visible in how agents now query vector stores through native protocol support rather than bespoke integration code — vector infrastructure is maturing beyond static, self-contained indexes toward components that participate in the data platform around them. Lake-native storage and protocol-native access are two faces of that shift.

Frequently asked questions

Where to go from here

The decision in front of most teams is not whether lake-native retrieval is the right direction. It is whether their own write path, freshness requirement and tail-latency budget survive the read-only constraint — and that is answered with a pilot on real data, not with a reading of release notes.

Designing retrieval infrastructure that holds up in production means reasoning about storage layout, refresh semantics, cache behaviour and governance, not just embedding models and prompt templates.