`@mdxdb/core` implement Payload collections & `embed` logic

[nathanclevenger]

We should put the Payload collections for mdx and embeddings into @mdxdb/core as well as a workflow to handle generating the embeddings on create & update.

embed should use the AI SDK openai provider text-embedding-3-large with 256 dimensions.

we will handle the actual indexing and storage in the fs, sqlite, postgres, and mongo packages separately

[linear]

MDX-47 `@mdxdb/core` implement Payload collections & `embed` logic

[nathanclevenger]

@claude please work on this

[claude]

Claude finished @nathanclevenger's task —— View job • claude/issue-104-20250522_223604 • Create PR ➔

---

Completed implementing Payload collections and embed logic for @mdxdb/core!

Summary

I've successfully implemented:

1. Payload Collections (collections.ts):

   • MdxCollection - for storing MDX documents with fields like slug, title, frontmatter, content, etc.
   • EmbeddingsCollection - for storing vector embeddings with document references and metadata

2. Embedding Logic (embed.ts):

   • generateEmbedding() - Uses AI SDK with OpenAI's text-embedding-3-large model
   • Configured with 256 dimensions as requested
   • chunkDocument() - Splits documents into chunks (full document, frontmatter, sections)
   • embedDocument() - Generates embeddings for all chunks of a document

3. Workflow System (workflow.ts):

   • Event-driven workflow for create/update/delete operations
   • createEmbeddingWorkflow() - Creates a workflow handler that automatically generates embeddings
   • WorkflowRegistry - Manages and executes multiple workflows
   • registerEmbeddingWorkflow() - Easy registration with custom configuration

4. Base Class Integration:

   • Updated MdxDbBase to execute workflows on set() and delete() operations
   • Separated implementation methods from public API to allow workflow execution

5. Dependencies:

   • Added @ai-sdk/openai and ai to package.json

The implementation handles embedding generation automatically when documents are created or updated. Storage of the embeddings will be handled separately in the fs, sqlite, postgres, and mongo packages as requested.

Example Usage

import { registerEmbeddingWorkflow } from '@mdxdb/core'

// Register the workflow
registerEmbeddingWorkflow({
  model: 'text-embedding-3-large',
  dimensions: 256,
  onEmbeddingGenerated: async (embeddings) => {
    // Store embeddings in your database
  }
})

|