Incremental index with an automatic full-rebuild fallback¶
Problem¶
Re-indexing a corpus into a vector store on every run is slow and wasteful when little changed. Indexing only the diff is fast but accumulates drift: a missed deletion, a failed state write, a rename seen as an add without the matching remove, and the index quietly disagrees with the source.
Drift in an index is not loud. Queries keep returning results — just slightly wrong ones — so the failure surfaces as declining answer quality rather than an error.
Technique¶
Track the last-indexed revision per corpus. On each run, diff against it and apply only the changes — with two escapes.
Fall back to a full rebuild when the diff is unusable: no recorded revision, the recorded revision no longer exists (history rewritten), or the diff is larger than a threshold.
The threshold has a reason worth stating, because it is the part people get wrong. Above roughly the point where per-item fetch overhead dominates a bulk read, incremental is slower than rebuilding — the crossover is a property of your fetch cost, not a safety margin. Below it, incremental wins; above it, rebuilding is both faster and self-healing.
Write the state record after the data, and treat a failed write as harmless. If indexing succeeds and the state write fails, the next run sees no revision and rebuilds. Correct, just expensive — so alert on repeated failures for the cost, not for drift.
When it applies¶
Any derived index over a versioned corpus where you can name what changed — a repository, a CMS with revisions, an event-sourced store.
When it does NOT apply¶
Where the source has no reliable change feed. A modification-time heuristic gives an incremental path you cannot trust, and an untrustworthy incremental index is worse than an honest slow rebuild.
Also skip it where a full rebuild is already cheap. The state tracking, threshold and fallback are real complexity, and complexity that saves seconds is a bad trade.
Evidence¶
A documentation index over several repositories used clear-and-rebuild throughout. It was correct and increasingly slow.
The incremental path was introduced with the fallback built in from the start, which is why it has never produced a drifted index — every failure mode routes to rebuild rather than to a subtly wrong result. The state collection stores metadata only and is deliberately not vectorised: it is bookkeeping, and making it searchable would have made it a second source of truth.