Skip to content

Fix TiDB auto-embedding fallback and migration - #136

Open
YoungYo wants to merge 1 commit into
mem9-ai:mainfrom
YoungYo:fix/tidb-auto-embedding-fallback
Open

Fix TiDB auto-embedding fallback and migration#136
YoungYo wants to merge 1 commit into
mem9-ai:mainfrom
YoungYo:fix/tidb-auto-embedding-fallback

Conversation

@YoungYo

@YoungYo YoungYo commented Mar 22, 2026

Copy link
Copy Markdown

Summary

This PR fixes TiDB auto-embedding schema drift and makes search degrade gracefully instead of surfacing hard 500 errors.

Problem

In the TiDB auto-embedding flow, a tenant can end up with a plain VECTOR column instead of a proper EMBED_TEXT() generated column.

When that happens:

  • memory writes may still succeed
  • semantic search can fail at query time with errors like:
    • VEC_EMBED_COSINE_DISTANCE() first argument must be a vector embedding column generated by EMBED_TEXT()

So the broken state is subtle: data appears to be written correctly, but recall/search fails later.

What changed

  • detect when memories.embedding / sessions.embedding are not EMBED_TEXT() generated columns
  • rebuild the affected tables into the correct TiDB auto-embedding shape when in-place migration is not supported
  • recreate vector indexes against the repaired schema
  • degrade search gracefully when auto vector search is unavailable instead of returning a hard failure

Why rebuild is needed

TiDB does not always allow converting a plain vector column into a generated EMBED_TEXT() column in place.

For that reason, the repair path falls back to:

  1. creating a correctly-shaped replacement table
  2. copying the data
  3. swapping the tables in place

This keeps the fix practical for already-drifted tenants.

Scope

This PR does not change the overall tenant model or TiDB provisioning flow.
It only repairs schema drift for TiDB auto-embedding tables and prevents search from failing hard while the schema is not yet usable for vector search.

Validation

Validated against TiDB Cloud with auto embedding enabled:

  • mnemo-server starts successfully
  • memory creation succeeds
  • auto-embedding schema is repaired into STORED GENERATED
  • hybrid search succeeds again after migration
  • OpenClaw -> mem9 -> TiDB flow was verified end to end

@netlify

netlify Bot commented Mar 22, 2026

Copy link
Copy Markdown

👷 Deploy request for mem9 pending review.

A Netlify team Owner will need to approve the deploy before you can run your build.

Are you a team Owner? Visit the deploys page to approve it

Need more help? Learn more in the Netlify docs

Name Link
🔨 Latest commit b849a07

@mashenjun mashenjun added the size/M LOC size label Mar 24, 2026

@you06 you06 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

`ALTER TABLE memories ADD VECTOR INDEX idx_cosine ((VEC_COSINE_DISTANCE(embedding))) ADD_COLUMNAR_REPLICA_ON_DEMAND`)
if err != nil && !tenant.IsIndexExistsError(err) {
slog.Warn("failed to add vector index after embedding migration (non-fatal)", "err", err)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing FTS index.

if p.ftsEnabled {
_, err := db.ExecContext(ctx,
`ALTER TABLE memories ADD FULLTEXT INDEX idx_fts_content (content) WITH PARSER MULTILINGUAL ADD_COLUMNAR_REPLICA_ON_DEMAND`)
if err != nil && !IsIndexExistsError(err) {
return fmt.Errorf("init schema: fulltext index: %w", err)
}
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M LOC size

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants