Fix TiDB auto-embedding fallback and migration - #136
Open
YoungYo wants to merge 1 commit into
Open
Conversation
👷 Deploy request for mem9 pending review.A Netlify team Owner will need to approve the deploy before you can run your build. Are you a team Owner? Visit the deploys page to approve it → Need more help? Learn more in the Netlify docs →
|
you06
reviewed
Mar 26, 2026
| `ALTER TABLE memories ADD VECTOR INDEX idx_cosine ((VEC_COSINE_DISTANCE(embedding))) ADD_COLUMNAR_REPLICA_ON_DEMAND`) | ||
| if err != nil && !tenant.IsIndexExistsError(err) { | ||
| slog.Warn("failed to add vector index after embedding migration (non-fatal)", "err", err) | ||
| } |
Contributor
There was a problem hiding this comment.
Missing FTS index.
mem9/server/internal/tenant/zero.go
Lines 203 to 209 in 594a24a
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes TiDB auto-embedding schema drift and makes search degrade gracefully instead of surfacing hard 500 errors.
Problem
In the TiDB auto-embedding flow, a tenant can end up with a plain
VECTORcolumn instead of a properEMBED_TEXT()generated column.When that happens:
VEC_EMBED_COSINE_DISTANCE() first argument must be a vector embedding column generated by EMBED_TEXT()So the broken state is subtle: data appears to be written correctly, but recall/search fails later.
What changed
memories.embedding/sessions.embeddingare notEMBED_TEXT()generated columnsWhy rebuild is needed
TiDB does not always allow converting a plain vector column into a generated
EMBED_TEXT()column in place.For that reason, the repair path falls back to:
This keeps the fix practical for already-drifted tenants.
Scope
This PR does not change the overall tenant model or TiDB provisioning flow.
It only repairs schema drift for TiDB auto-embedding tables and prevents search from failing hard while the schema is not yet usable for vector search.
Validation
Validated against TiDB Cloud with auto embedding enabled:
mnemo-serverstarts successfullySTORED GENERATED