Vector Database Drift: When Embeddings Move And Old Memory Becomes Unaddressable

TL;DR

Every agent with persistent memory eventually has to upgrade its embedding model. The new embedding model produces vectors in a different geometric space than the old one; existing memory entries are still in storage, but their position in the vector index no longer corresponds to anything queries will retrieve. The naive migration (re-embed everything overnight, swap the index) almost always breaks something. The disciplined migration uses a dual-index pattern with a gradual cutover, verification gates at each phase, and a rollback path. This post walks through what causes embedding drift, what the dual-index pattern looks like in practice, the four properties a good migration plan must have, and how to recover when migration goes wrong. The artifact is a Vector Migration Plan template you can adapt this week.

Intro: The night the customer-service agent forgot how to find anything

A SaaS company we worked with last year ran a customer-service agent backed by a vector store containing about two million memory entries: prior tickets, product documentation snippets, customer-specific configuration notes, internal runbooks. The agent's retrieval quality had been steadily improving for eighteen months. The team decided to upgrade their embedding model from a 768-dimensional model to a 1536-dimensional model from a newer provider; benchmarks showed a clear retrieval-quality improvement. The migration plan was straightforward: re-embed the corpus over a weekend, atomically swap the index, restart the agent on Monday morning with the new embeddings.

Monday morning, the agent's first-touch resolution rate dropped from 71% to 38%. Customers reported that responses had become weirdly generic, citing irrelevant past tickets as references. Investigations showed that re-embedding had completed, the new index had been populated, but a substantial fraction of historically high-value retrievals were no longer surfacing the entries that had previously been their best matches. Some entries that the old index had reliably surfaced for specific query patterns were now buried under newer entries with superficially similar content but worse match quality on the actual semantic intent.

The team rolled back to the old embedding model that afternoon, restored Friday's index from snapshot, and spent the next six weeks rebuilding the migration plan with a dual-index architecture that allowed gradual cutover and per-query verification. The second migration succeeded, but the cost of the first attempt was a week of degraded customer experience, a serious internal credibility hit for the team, and a permanent organizational scar around 'embedding upgrades' that took months to heal.

This pattern is not exotic. Every team running a vector store will eventually need to migrate to a different embedding model, and most teams will get burned at least once because the failure mode is non-obvious. The new model is better; the new vectors are mathematically valid; the index is correctly populated; the system runs without errors. The thing that breaks is the retrieval quality, and retrieval quality is much harder to monitor than uptime. The rest of this piece explains what is actually happening and how to migrate safely.

What embedding drift actually is

Embedding drift is the phenomenon that the vector representation of a given piece of content under embedding model A is not the same as its representation under model B. This sounds obvious. The implications are less obvious, and they are what cause migrations to fail.

The most important implication is that the two vectors are not just numerically different; they exist in geometrically different spaces. The notion of 'distance' between two vectors is meaningful only relative to a single embedding model. A query vector embedded under model A and a memory vector embedded under model B cannot be meaningfully compared; the cosine similarity between them is essentially noise. This means that during a migration, you cannot have query vectors from the new model matching against memory vectors from the old model and expect the retrieval to work. The two indexes have to be coherent.

The second implication is that 'better' embedding models are not strictly better at every retrieval. A new model with higher average benchmark scores typically wins on overall retrieval quality but loses on specific patterns the old model had memorized particularly well. The agent's memory, having been built up under the old model, contains entries that were optimized (implicitly) for the old model's retrieval characteristics. The new model's geometry is different; some of those optimizations no longer pay off. Net retrieval quality after migration is usually positive, but the specific gains and losses are non-uniform and unpredictable in advance.

The third implication is that any auxiliary structure built on top of the vector index (clusters, topic models, similarity-based access controls, retrieval shortcuts) becomes invalid the moment the underlying embedding changes. These structures need to be rebuilt against the new index, and the rebuild may produce different cluster assignments, different topic boundaries, and different access patterns than the originals. Teams that rely on these structures for downstream logic need to plan for the rebuild explicitly.

The fourth implication is that the corpus itself may not be unchanged. In any production deployment, new memory entries are being added continuously. A migration that takes a weekend will see hundreds or thousands of new entries arrive while it runs; those new entries need to be embedded under both the old and new model so they end up correctly placed in both indexes. The dual-write pattern during migration is what enables this; without it, new entries arrive only in one index and the cutover becomes incomplete.

The dual-index pattern

The correct migration pattern is dual-index: maintain both the old and new indexes simultaneously for a defined cutover window, route reads to one or both, route writes to both, and verify retrieval quality on the new index against the old as the source of truth. The pattern has five phases.

Phase one is bootstrap. You stand up the new vector store, configure the new embedding model, and re-embed the existing corpus into the new index. This is the bulk of the work. It can take hours or days depending on corpus size and embedding model latency. During bootstrap, the agent continues to read from and write to the old index normally; the new index is being built in the background and is not yet authoritative.

Phase two is dual-write. Once the new index is fully populated, you switch all writes to write to both indexes simultaneously. New memory entries land in both stores under their respective embeddings. Reads still go to the old index. This phase establishes that ongoing operations stay coherent across both indexes. You should run dual-write for at least one full operational cycle before proceeding; the goal is to confirm that the dual-write logic does not have bugs that would silently desynchronize the indexes.

Phase three is shadow read. You begin issuing every read query against both indexes, returning the old index's result to the agent as the live answer, and logging the new index's result for comparison. This phase produces the data that justifies the cutover: a per-query comparison of old-vs-new retrieval quality. You define a quality metric (we use a weighted combination of position-of-correct-answer and per-result relevance) and track its distribution. Shadow read is where you discover the patterns where the new model retrieves worse than the old; you can address them (typically through query reformulation, retrieval reranking, or per-tag fallback rules) before the cutover.

Phase four is gradual cutover. You begin returning the new index's result for some fraction of queries (say five percent), monitor the live quality signals, and ramp the fraction up as quality holds steady. The ramp can be staged by query type, by user segment, or by raw percentage; the right slicing depends on the deployment. The point is to catch quality regressions before they affect a meaningful fraction of users.

Phase five is decommission. Once the new index has been authoritative for all queries for at least one operational cycle without quality regression, you drop dual-write, archive the old index for some retention period (we use ninety days), and finally delete it. The retention window protects against latent issues that surface after cutover; the deletion at the end protects against the storage cost of a parallel index in perpetuity.

This is the pattern. The phases sound elaborate; in practice they are mostly mechanical. The reason teams skip them is that the simplicity of 'just swap the index' is seductive and the failure mode of skipping is invisible until it isn't.

Verification: the metric you cannot skip

The dual-index pattern is only as good as the verification metric used to judge cutover safety. A bad metric will let you cut over to a worse retrieval system and not notice. A good metric catches the regression before it ships.

The metric we recommend has three components. The first is position-of-correct-answer for queries where the correct answer is known. In any deployment with feedback signals (user thumbs-up, escalation rate, follow-up question rate), there is a usable proxy for which retrieved entries were actually useful. Position is a stricter measure than presence because retrieval quality is bounded by ranking; an entry retrieved at position one is much more useful than the same entry at position twenty.

The second component is per-result relevance, which requires a judge. A small fraction of queries (say one in fifty) is sampled; a multi-LLM jury is asked to rate the relevance of each retrieved entry; the per-result scores are aggregated. This is more expensive than position metrics but catches cases where the correct answer is being retrieved but accompanied by a lot of irrelevant noise that hurts the agent's downstream reasoning. Trimming the jury's top and bottom outliers (we trim 20% on each end) makes the score robust to individual judge weirdness.

The third component is downstream behavior. If the agent's outputs (answers, escalation decisions, action choices) are themselves measurable, the migration's effect on those outputs is the ground truth. Position and relevance are leading indicators; downstream behavior is the lagging indicator that confirms the leading indicators were tracking the right thing. A migration that improves position and relevance but degrades downstream behavior has uncovered a misalignment in the metrics, which is itself useful information.

With all three components, you can produce a single composite cutover-readiness score per query class, monitor it over time, and use it as the gate for ramping up cutover percentage. Without it, you are flying blind.

A practical note: do not try to evaluate the metric only at the cutover decision point. Run the metric continuously during shadow read; the data is cheap to collect and the patterns it reveals (which query classes regress, which improve, which stay flat) are essential to building confidence in the migration.

When new entries arrive mid-migration

A migration that takes more than a few minutes will see new memory entries arrive during the migration window. How you handle them determines whether the cutover is clean or messy.

The correct handling has three rules. Rule one: every new entry must be embedded under both the old and new models from the moment dual-write starts. The dual-write logic is responsible for this. The two embedding operations can be parallelized; the entry is not considered persisted until both succeed. If one fails, the write fails atomically; partial writes are the road to a desynchronized index.

Rule two: the dual-write window must extend long enough to cover the worst-case latency between an entry being added and being retrieved. Most entries are retrieved within a day or two of being added; some are retrieved months later. The dual-write window does not need to cover every retrieval, but it should cover the bulk of retrievals from entries added during the migration. We typically run dual-write for at least the duration of bootstrap plus shadow read combined.

Rule three: when an entry is updated mid-migration, the update propagates to both indexes synchronously. Updates are essentially the same problem as inserts and deserve the same atomicity guarantees. A common bug is to handle inserts atomically but treat updates as best-effort; this introduces silent desynchronization on entries that change frequently.

The practical implementation of these rules is straightforward in most vector stores: a small write-path adapter that fans out to both stores. The complexity is operational rather than architectural. The thing that breaks teams is the long tail of edge cases (deletes, soft-deletes, batch updates, schema migrations on metadata fields) where the dual-write logic has to handle each operation type. Build the adapter as a switch statement that explicitly handles every operation type your store supports, and audit it before starting the migration.

Auxiliary structures: the part everyone forgets

A vector index is rarely the only structure built on top of the embeddings. Most production deployments have at least one of: a clustering layer used for navigation or summarization, a similarity-based access control layer, retrieval shortcut caches, topic models built from embedding distributions, or anomaly detection thresholds calibrated against the embedding distribution. Every one of these auxiliary structures becomes invalid when the underlying embedding changes, and every one needs to be rebuilt against the new index before the cutover is complete.

The dependency map between auxiliary structures and the underlying index should be enumerated explicitly before migration starts. Most teams are surprised by how long this list is once they actually look. We routinely find five to fifteen dependent structures in mature deployments; teams that thought there were two or three. Some of the dependents are user-facing (a search UI that depends on a clustering layer for browse navigation); some are internal (an alerting threshold that depends on the embedding distribution); both classes need rebuild plans.

The rebuild plans for auxiliary structures usually mirror the dual-index pattern. You rebuild each structure against the new index in parallel with the existing structure on the old index, run them in shadow mode, verify the outputs are consistent or improved, and cut over alongside the main index cutover. This adds work to the migration but is the only safe approach for any deployment where the auxiliary structures matter.

The failure mode for skipping this is that the main index cutover succeeds, the agent's retrieval quality is fine, and three weeks later someone notices that the navigation UI's clusters look wrong, or the alerting thresholds have been silently miscalibrated, or a similarity-based access control is now letting through queries it should be filtering. The connection back to the embedding migration is not always obvious by then, and the diagnosis can take days. Plan for the auxiliaries from the start.

Rollback: how to abort a migration that is going wrong

Every migration plan needs a rollback path, because every migration occasionally needs one. The rollback path is the answer to the question 'what happens if we discover at fifty percent cutover that the new index has a quality regression we cannot fix?'

The rollback path is built into the dual-index pattern by design. As long as the old index is still being maintained (dual-write is still running, the old index is still receiving updates), rolling back means routing one hundred percent of reads back to the old index. There is no data loss, no schema migration, no manual reconciliation. The cost of maintaining the old index for the duration of the cutover is the cost of buying this rollback path.

The critical decision is when to stop maintaining the old index. The temptation, once cutover is at one hundred percent, is to drop dual-write immediately to save the operational cost. We recommend extending dual-write for a defined period after one hundred percent cutover (we use one operational cycle, typically a week) before dropping it. This buys you a rollback path during the most failure-prone period: the first week of full cutover, when latent issues are most likely to surface.

After dual-write is dropped, rollback becomes much harder. You can restore the old index from snapshot, but any entries written between snapshot and rollback are lost from the old index. The recovery for those is to re-embed them under the old model from the new index's metadata, which is mechanical but introduces a window of inconsistency. Avoid getting into this state by extending dual-write longer than feels necessary.

There is also a rollback decision matrix worth thinking through in advance. What are the conditions that trigger rollback? What is the threshold for each condition? Who has authority to trigger rollback? How quickly can the rollback be executed? Writing this down before the migration starts means the team has clarity in the moment of crisis, when judgment is most likely to be impaired by the pressure.

Cross-tenant migration: when one platform serves many

Most migration discussions implicitly assume a single-tenant store with one operator's corpus. Multi-tenant platforms (one shared store with many tenants, or many physical stores with shared embedding infrastructure) introduce additional complexity that deserves explicit treatment.

The first complication is per-tenant readiness variance. Different tenants will see different quality regression patterns under the new embedding because their corpora and query distributions differ. A single platform-wide cutover decision based on aggregate metrics will hide tenant-specific regressions; the right approach is per-tenant readiness scoring, with cutover decisions made tenant by tenant. Some tenants may cut over within the first week of shadow read; others may need extended shadow read because their corpus has unusual patterns.

The second complication is auxiliary structure rebuild scope. Auxiliary structures that span tenants (a shared clustering used for cross-tenant analytics, a shared similarity model trained on aggregate behavior) need to be rebuilt against the new embedding before any tenant can fully cut over. The cross-tenant rebuild is a coordination point that affects every tenant simultaneously; it should be planned as a separate workstream from per-tenant cutover.

The third complication is per-tenant rollback authority. In a single-tenant deployment, the operator decides when to roll back. In a multi-tenant deployment, individual tenants may have rollback authority over their own slice (especially if isolation is strong enough that per-tenant rollback is technically possible). The rollback decision matrix needs to account for who can trigger rollback and at what scope; mistakes here can create asymmetric situations where some tenants are on the new index and others on the old, with cross-tenant operations behaving inconsistently.

The fourth complication is metering. Multi-tenant platforms typically meter resource consumption per tenant for billing. The migration's resource usage (dual-write, dual-store maintenance, shadow read overhead) has to be either absorbed by the platform or pro-rated across tenants in a defensible way. The decision should be made before the migration starts; surprising tenants with migration-related charges after the fact damages the platform relationship.

The practical recommendation for multi-tenant platforms is to treat the migration as a portfolio of tenant migrations rather than a single platform migration. The dual-index infrastructure is shared; the cutover decisions are per tenant; the verification metrics are per tenant; the rollback paths are per tenant. This is more operational work but produces a much cleaner result than trying to migrate the platform as a single unit.

Embeddings as a cost-of-ownership decision

A related question that often gets coupled to embedding migrations is the long-term cost of ownership of the embedding choice itself. Different embedding providers have different pricing, latency, dimensionality, and quality profiles. The provider you chose at deployment time may not be the right provider eighteen months later, and the migration cost is what makes the choice sticky.

The cost-of-ownership analysis has four components. The first is per-embedding cost: pay-per-call pricing accumulates with corpus growth and query volume. A platform that scales by 10x sees its embedding cost scale with it; the per-call cost matters more at scale than at launch. The second is migration cost: every embedding change requires the dual-index migration described in this post, with all of its operational overhead. Providers with frequent breaking model changes impose higher migration costs over time.

The third component is sovereignty risk: depending on a single embedding provider creates a vendor dependency that may become uncomfortable if the provider's pricing changes, terms shift, or service availability degrades. Some teams hedge this by maintaining the ability to switch providers (not actively switching, but keeping the migration capability warm); the hedge is itself a cost.

The fourth component is performance ceiling: the embedding model's quality places a ceiling on the agent's retrieval quality. Better models typically cost more, both per call and per migration. The right embedding for a high-stakes deployment is rarely the cheapest one; the right embedding for a cost-sensitive deployment is rarely the highest-quality one. The trade-off is workload-specific.

We recommend revisiting the embedding choice annually as part of the platform's broader infrastructure review. The annual revisit asks: is the current embedding still the right choice for our workload, given current pricing, current quality benchmarks, and our current scale? Most years the answer is yes and no migration is needed. Some years the answer is no, and the migration becomes a planned project rather than an emergency.

The alternative to annual revisit is opportunistic migration: switching embeddings whenever a better option appears. This produces too many migrations and burns engineering bandwidth on a moving target. Annual cadence with explicit decision criteria is the discipline; opportunistic migration is the failure mode.

Embedding evaluation methodology: doing your own benchmarks

Providers publish benchmark scores for their embedding models. The scores are useful as a first filter but rarely predict workload-specific quality. The right evaluation methodology is to benchmark against your own corpus and query distribution before committing to a migration.

The methodology has four steps. First, sample your queries: take a representative sample of recent production queries, ideally weighted by importance to the operator (high-stakes queries weighted more heavily). The sample size should be a few hundred to a few thousand queries; small enough to evaluate manually, large enough to produce statistically meaningful results.

Second, define ground truth: for each sampled query, identify the correct retrieval (which memory entries should ideally be returned in the top results). Ground truth can come from existing operator judgment, from the production agent's outputs that have been validated as correct, or from explicit human labeling. Ground truth is the most labor-intensive part of the methodology; budget for it explicitly.

Third, run the candidate embedding: embed the sample queries and the corresponding corpus subset under each candidate embedding model, run retrieval, and capture the results. The candidate set should include the current embedding (as the baseline) and at least two alternatives.

Fourth, score the results: for each query, score each embedding's results against ground truth using the metrics described earlier (position-of-correct-answer, per-result relevance, and any workload-specific metrics). Aggregate the scores across the sample and compare candidates. The candidate that wins on the workload's preferred metric is the recommendation; if no candidate clearly wins, the current embedding is the right choice for now.

The methodology takes about a week of focused work for a small corpus and queries-of-interest sample, longer for larger workloads. The cost is small compared to the cost of migrating to the wrong embedding. We recommend running this methodology any time the embedding choice is being reconsidered, not just before migration; the methodology produces useful baseline data even when the conclusion is to stay with the current embedding.

Counter-argument: 'We will just take a maintenance window and swap'

The most common objection to the dual-index pattern is that it is overkill for deployments that can tolerate a maintenance window. The argument: if the agent can be offline for a few hours, you can take it down, re-embed everything, swap the index, bring it back up, and skip the elaborate ceremony. Total downtime: the duration of re-embedding plus index swap. Total operational complexity: minimal.

The argument is real for some deployments and the maintenance-window pattern is the right answer where it applies. The conditions are: the agent can be down for the full re-embedding duration (often hours for large corpora), no quality regressions are acceptable enough to roll back without a rollback plan, and the auxiliary structures are simple enough to swap atomically with the main index.

The conditions are rarely all met. In practice, even deployments that nominally accept maintenance windows discover during the window that something is wrong (auxiliary structure mismatch, quality regression on a specific query class, dual-write bug they did not realize they had) and have no clean way to recover. The maintenance window becomes an extended outage, the team improvises a rollback under pressure, and the migration ends up costing more than the disciplined dual-index approach would have.

Our recommendation: use the maintenance-window pattern only for very small corpora (under fifty thousand entries) or very low-stakes deployments where a quality regression that takes a week to notice is not material. For everything else, the dual-index pattern's overhead pays for itself in the first migration where you avoid an unrecoverable cutover. The pattern is also reusable: once you have built the dual-write infrastructure, future migrations are dramatically cheaper because the hard part is already in place.

What Armalo does

Armalo's Cortex memory layer is built around the assumption that embedding models will change over time. The hot, warm, and cold tiers each support dual-index operation natively; a migration is a configuration change rather than a custom engineering project. Dual-write logic is part of the standard write path and handles inserts, updates, and deletes atomically across both indexes during the migration window.

The verification metric is computed continuously during shadow read using a multi-LLM jury (with top and bottom 20% outlier trimming) to rate per-query relevance, combined with downstream behavior signals from the agent's outputs. Operators see a per-query-class readiness score that drives the cutover ramp; the system supports staged cutover by query type, by tag, or by raw percentage. Auxiliary structure rebuilds (clusters, topic models, similarity-based access controls) are tracked alongside the main index cutover with their own readiness signals.

Memory entries that influence pact compliance evidence carry their provenance trail across embedding migrations unchanged; the embedding is an addressing concern, not an identity concern, so the trust oracle at /api/v1/trust/ continues to surface verifiable provenance chains for entries that have been re-embedded. Counterparties relying on those chains see no disruption from migration events. Composite trust scores are not affected by migration mechanics; the score decay function (one point per week) and jury trimming continue to operate against the agent's outputs regardless of which index version produced the retrieval.

FAQ

How long should the dual-index window run end to end?

For a typical deployment with a million-scale corpus, plan for two weeks: bootstrap (variable, often two to four days), dual-write soak (two to three days), shadow read (three to five days), gradual cutover (three to five days), post-cutover dual-write extension (one operational cycle, typically a week). Larger corpora or more complex auxiliary structures stretch this proportionally.

What if the new embedding has different dimensionality?

This is the normal case. The two indexes have to support different dimensionalities, which means physically different stores or a vector store that supports per-collection dimensionality. The migration mechanics are unchanged; the storage layout is just slightly different. Plan for the storage cost of running two collections in parallel during the migration window.

Can I skip shadow read if I have benchmark data on the new embedding?

No. Benchmarks are external; your corpus is yours. Embedding quality is highly sensitive to corpus characteristics, and benchmarks rarely predict the regression patterns that matter for a specific deployment. Shadow read on your corpus is the only reliable signal.

Is hybrid retrieval (combining old and new index results) ever a good idea?

It sounds appealing but is usually a trap. The two indexes' similarity scores are not commensurable, so any combination requires arbitrary weighting that you cannot justify. A reranker on top of both can work, but the simpler answer is to commit to one index for live retrieval and use the other only for shadow comparison.

What about embedding model versioning within the same provider?

Minor versions of the same model often produce non-trivially different vectors. Treat them like full migrations unless the provider explicitly guarantees vector compatibility (most do not). The dual-index pattern is the safe default for any embedding change.

How do I handle entries that were soft-deleted before migration?

Soft-deleted entries should be re-embedded into the new index in the same soft-deleted state. Filtering out soft-deletes only at the application layer, not at the index population step, keeps the indexes structurally consistent and makes future undeletes work correctly.

What if my vector store does not support the dual-index pattern natively?

Most do not, in the sense of having a built-in feature. The pattern is implementable on top of any store that supports multiple collections or namespaces. The cost is the dual-write adapter and the routing logic; both are straightforward to build and reusable across future migrations.

Bottom line

Vector database drift is one of the most predictable failure modes in any agent deployment with persistent memory, and one of the most consistently mishandled. The naive migration (re-embed and swap) breaks something almost every time. The disciplined migration (dual-index, dual-write, shadow read, gradual cutover, extended post-cutover dual-write) handles the cutover cleanly and provides a usable rollback path. Build the verification metric early, enumerate auxiliary structures explicitly, plan for new entries arriving mid-migration, and write the rollback decision matrix before the migration starts. The Vector Migration Plan template in this post is the artifact you will return to every time you upgrade an embedding; the cost of building it once is the cost of avoiding repeated incidents thereafter.