diff --git a/Architecture.md b/Architecture.md index 7ef4769..8ebbc67 100644 --- a/Architecture.md +++ b/Architecture.md @@ -28,17 +28,20 @@ Packages join GemStack by **graduating**, one at a time, when they prove framewo - A `vike-*` package moves here only if a genuinely agnostic *core* falls out of it that is useful beyond its framework. In that case the core graduates (e.g. `@gemstack/`) while the framework binding stays `vike-*`. - Because both repos are co-governed and the `vike-*` set sits in the Vike orbit, any such move is decided with the Vike team, when there is brand traction, not unilaterally. -### Graduation candidates already in `vike-data` +### The data layer (graduated — Phase 0, pending brand confirmation) -An audit of `vike-data` shows the agnostic engines are not in the `vike-*` packages (those are bindings) but in the `universal-*` packages, which already carry **zero Vike imports**. These are the real candidates, in priority order after `@gemstack/ai-sdk`: +An audit of `vike-data` showed the agnostic engines are not in the `vike-*` packages (those are bindings) but in the `universal-*` packages, which carry **zero Vike imports**. The data layer is the second graduation after `@gemstack/ai-sdk`: -| Candidate (today) | Would become | Notes | +| Source (in `vike-data`) | Graduated to | Notes | |---|---|---| -| `@universal-orm/core` (+ `@universal-orm/drizzle` / `/memory` / `/rudder`) | `@gemstack/orm` (+ adapters) | The ORM analog of `@gemstack/ai-sdk`. Mature, clearly agnostic. The strongest next candidate; move the core + its adapter family together. | -| `@vike-data/universal-schema` | `@gemstack/schema` | "Usable standalone by any framework or ORM." Agnostic, but currently mis-scoped under `@vike-data`. | -| `@vike-data/kit` | (stays) | Agnostic primitives (`createPort`), but it is the kit for *authoring bindings*, so it belongs with the binding ecosystem, not the engine umbrella, unless GemStack later wants a shared-primitives package. | +| `@universal-orm/core` (+ `/drizzle` / `/memory`) | `@gemstack/orm` (+ `orm-drizzle` / `orm-memory`) | The ORM analog of `@gemstack/ai-sdk`. Mature, clearly agnostic. Core + its adapter family graduated together. | +| `@vike-data/universal-schema` | `@gemstack/schema` | Agnostic, was mis-scoped under `@vike-data`. **Graduated with the ORM** (not later): the ORM test suite builds fixtures with `defineSchema`/`mergeSchemas`, so they cannot split. Still an early-experiment API. | +| `@universal-orm/rudder` | (decision pending) | The one **framework-coupled** adapter (peer `@rudderjs/database`). Excluded from the agnostic move; it is the "living binding" question — graduate as `@gemstack/orm-rudder` or keep it in the Rudder orbit. | +| `@vike-data/kit` | (stays) | Agnostic primitives (`createPort`), but it is the kit for *authoring bindings*, so it belongs with the binding ecosystem, not the engine umbrella. | -Realized fully, GemStack is the unified home for agnostic engines: `@gemstack/ai-sdk` (AI), `@gemstack/orm` (data), `@gemstack/schema` (schema). +**Status:** copied + renamed and green in the gemstack workspace (Phase 0 spike, #65, on `main` as `private` packages). The packages are **not yet published**, and the move is **gated on the brand-consolidation decision** below (#66): the `@gemstack/*` names are provisional until the Vike team confirms consolidation vs keeping `@universal-orm` as a parallel brand. Publishing, the re-export shims at the old names, and repointing dependents all wait on that confirmation. + +Now realized, GemStack is the unified home for agnostic engines: `@gemstack/ai-sdk` (AI), `@gemstack/orm` (data), `@gemstack/schema` (schema). ### The open brand-consolidation question (for the Vike team) diff --git a/README.md b/README.md index 1b2135c..ac9c5c8 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,10 @@ Full documentation lives in [`docs/`](./docs/guide/index.md) (a hosted site is o | [`ai‑autopilot`](./packages/ai-autopilot) | Orchestration: a Supervisor that plans, dispatches subagents (bounded concurrency + budget guardrails), and synthesizes the result. | [Guide](./docs/packages/ai-autopilot.md) | [![npm](https://img.shields.io/npm/v/@gemstack/ai-autopilot)](https://www.npmjs.com/package/@gemstack/ai-autopilot) | | [`ai‑mcp`](./packages/ai-mcp) | The agent/MCP bridge: consume a remote MCP server's tools as agent tools, and expose an agent as an MCP server. | [Guide](./docs/packages/ai-mcp.md) | [![npm](https://img.shields.io/npm/v/@gemstack/ai-mcp)](https://www.npmjs.com/package/@gemstack/ai-mcp) | | [`mcp`](./packages/mcp) | A standalone framework for *authoring* MCP servers: tools, resources, prompts, decorators, OAuth 2.1, a framework-neutral HTTP handler, and a test client. Agent-agnostic. | [Guide](./docs/packages/mcp.md) | [![npm](https://img.shields.io/npm/v/@gemstack/mcp)](https://www.npmjs.com/package/@gemstack/mcp) | +| [`orm`](./packages/orm) | The data engine: a narrow, ORM-free repository (`db.users.upsert(...)`) over a composed schema, plus the adapter contract and a one-adapter registry. The data-layer twin of `ai-sdk`. | [Guide](./docs/packages/orm.md) | [![npm](https://img.shields.io/npm/v/@gemstack/orm)](https://www.npmjs.com/package/@gemstack/orm) | +| [`schema`](./packages/schema) | The shape engine: declare tables once as plain data, merge contributions, derive migrations, compile to Prisma / Drizzle / Rudder. **Preview.** | [Guide](./docs/packages/schema.md) | [![npm](https://img.shields.io/npm/v/@gemstack/schema)](https://www.npmjs.com/package/@gemstack/schema) | +| [`orm‑memory`](./packages/orm-memory) | In-process `Map` adapter for `orm` — tests, demos, zero-config dev. | [Guide](./docs/packages/orm.md#adapters) | [![npm](https://img.shields.io/npm/v/@gemstack/orm-memory)](https://www.npmjs.com/package/@gemstack/orm-memory) | +| [`orm‑drizzle`](./packages/orm-drizzle) | Drizzle adapter for `orm` — real databases. | [Guide](./docs/packages/orm.md#adapters) | [![npm](https://img.shields.io/npm/v/@gemstack/orm-drizzle)](https://www.npmjs.com/package/@gemstack/orm-drizzle) | ### How they fit together @@ -29,10 +33,17 @@ Full documentation lives in [`docs/`](./docs/guide/index.md) (a hosted site is o @gemstack/ai-mcp agent <-> MCP bridge (the "adapter") -> ai-sdk ----------------------------------------------------------------------------------- @gemstack/mcp standalone MCP server framework agent-agnostic, not ai-* +----------------------------------------------------------------------------------- +@gemstack/schema data shape: define tables, merge, derive migrations (preview) +@gemstack/orm runtime data access over a composed schema +@gemstack/orm-memory in-process Map adapter (tests/demos) -> orm +@gemstack/orm-drizzle Drizzle adapter (real databases) -> orm ``` The `ai-` prefix means **"depends on the agent runtime."** `skills`, `autopilot`, and `ai-mcp` all depend on `ai-sdk`; `ai-sdk` depends on none of them, and nothing depends "up." A package about AI that is agent-agnostic (like `@gemstack/mcp`) is a peer of the family, not a member of it. +The **data family** (`orm` + `schema` + adapters) is a second engine family — framework-agnostic data access, parallel to the AI family and independent of it. `orm` is the runtime (read/write without importing an ORM), `schema` is the shape (declare tables once, compile to any ORM), and `orm-memory` / `orm-drizzle` are adapters that bind the repository to a real backend. + See [`Architecture.md`](./Architecture.md) for the full layering, naming rule, and graduation policy. ### Which MCP package do I use? diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index 74b4284..96d70b0 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -53,6 +53,8 @@ export default defineConfig({ { text: 'ai-autopilot', link: '/packages/ai-autopilot' }, { text: 'ai-mcp', link: '/packages/ai-mcp' }, { text: 'mcp', link: '/packages/mcp' }, + { text: 'orm', link: '/packages/orm' }, + { text: 'schema', link: '/packages/schema' }, ], }, { @@ -83,7 +85,7 @@ export default defineConfig({ ], }, { - text: 'The family', + text: 'The AI family', items: [ { text: 'ai-skills', link: '/packages/ai-skills' }, { text: 'ai-autopilot', link: '/packages/ai-autopilot' }, @@ -91,6 +93,13 @@ export default defineConfig({ { text: 'mcp', link: '/packages/mcp' }, ], }, + { + text: 'The data family', + items: [ + { text: 'orm — the data engine', link: '/packages/orm' }, + { text: 'schema — the shape engine', link: '/packages/schema' }, + ], + }, ], }, diff --git a/docs/guide/index.md b/docs/guide/index.md index 8476aa3..6867f2e 100644 --- a/docs/guide/index.md +++ b/docs/guide/index.md @@ -15,6 +15,9 @@ All packages publish under the **`@gemstack/`** scope. | [`ai-autopilot`](/packages/ai-autopilot) | Orchestration: a Supervisor that plans, dispatches subagents (bounded concurrency + budget guardrails), and synthesizes the result. | | [`ai-mcp`](/packages/ai-mcp) | The agent/MCP bridge: consume a remote MCP server's tools as agent tools, and expose an agent as an MCP server. | | [`mcp`](/packages/mcp) | A standalone framework for *authoring* MCP servers: tools, resources, prompts, decorators, OAuth 2.1, a framework-neutral HTTP handler, and a test client. Agent-agnostic. | +| [`orm`](/packages/orm) | The data engine: a narrow, ORM-free repository (`db.users.upsert(...)`) over a composed schema, plus the adapter contract and a one-adapter registry. The data-layer twin of `ai-sdk`. | +| [`schema`](/packages/schema) | The shape engine: declare tables once as plain data, merge contributions, derive migrations, compile to Prisma / Drizzle / Rudder. **Preview.** | +| [`orm-memory`](/packages/orm#adapters) / [`orm-drizzle`](/packages/orm#adapters) | Adapters that bind the `orm` repository to a backend: in-process `Map`s (tests/demos) or Drizzle (real databases). | ## How they fit together @@ -25,9 +28,14 @@ ai-autopilot orchestration / autonomy (the "director") -> ai-sdk (+ skills) ai-mcp agent <-> MCP bridge (the "adapter") -> ai-sdk ----------------------------------------------------------------------------------- mcp standalone MCP server framework agent-agnostic, not ai-* +----------------------------------------------------------------------------------- +schema data shape: define tables, merge, derive migrations (preview) +orm runtime data access over a composed schema +orm-memory in-process Map adapter (tests/demos) -> orm +orm-drizzle Drizzle adapter (real databases) -> orm ``` -`ai-sdk` is the foundation: it owns the single-agent loop, tools, and streaming. `ai-skills` and `ai-autopilot` build on top of it. `ai-mcp` bridges an agent to the Model Context Protocol. `mcp` stands apart - it is for *authoring* MCP servers and knows nothing about agents. +`ai-sdk` is the foundation of the **AI family**: it owns the single-agent loop, tools, and streaming; `ai-skills` and `ai-autopilot` build on top of it, and `ai-mcp` bridges an agent to the Model Context Protocol. `mcp` stands apart - it is for *authoring* MCP servers and knows nothing about agents. The **data family** is a second, independent set of engines: `orm` reads and writes without importing an ORM, `schema` declares tables once and compiles to any ORM, and the adapters bind the repository to a real backend. ## Design principles diff --git a/docs/packages/index.md b/docs/packages/index.md index e7fefc0..e93da04 100644 --- a/docs/packages/index.md +++ b/docs/packages/index.md @@ -9,6 +9,10 @@ All packages publish under the **`@gemstack/`** scope (e.g. `npm install @gemsta | [`ai-autopilot`](/packages/ai-autopilot) | Orchestration: a Supervisor that plans, dispatches subagents (bounded concurrency + budget guardrails), and synthesizes the result. | | [`ai-mcp`](/packages/ai-mcp) | The agent/MCP bridge: consume a remote MCP server's tools as agent tools, and expose an agent as an MCP server. | | [`mcp`](/packages/mcp) | A standalone framework for *authoring* MCP servers: tools, resources, prompts, decorators, OAuth 2.1, a framework-neutral HTTP handler, and a test client. Agent-agnostic. | +| [`orm`](/packages/orm) | The data engine: a narrow, ORM-free repository (`db.users.upsert(...)`) over a composed schema, plus the adapter contract and a one-adapter registry. The data-layer twin of `ai-sdk`. | +| [`schema`](/packages/schema) | The shape engine: declare tables once as plain data, merge contributions, derive migrations, compile to Prisma / Drizzle / Rudder. **Preview.** | +| [`orm-memory`](/packages/orm#adapters) | In-process `Map` adapter for `orm` — tests, demos, zero-config dev. | +| [`orm-drizzle`](/packages/orm#adapters) | Drizzle adapter for `orm` — real databases. | ## How they fit together @@ -19,8 +23,20 @@ ai-autopilot orchestration / autonomy (the "director") -> ai-sdk (+ skills) ai-mcp agent <-> MCP bridge (the "adapter") -> ai-sdk ----------------------------------------------------------------------------------- mcp standalone MCP server framework agent-agnostic, not ai-* +----------------------------------------------------------------------------------- +schema data shape: define tables, merge, derive migrations (preview) +orm runtime data access over a composed schema +orm-memory in-process Map adapter (tests/demos) -> orm +orm-drizzle Drizzle adapter (real databases) -> orm ``` +### Two engine families + +GemStack has two independent engine families plus the standalone MCP framework: + +- The **AI family** (`ai-sdk` + `ai-skills` / `ai-autopilot` / `ai-mcp`) — everything for running agents. +- The **data family** (`orm` + `schema` + adapters) — framework-agnostic data access: `orm` reads and writes without importing an ORM, `schema` declares tables once and compiles to any ORM. Parallel to the AI family, and independent of it — adopt either on its own. + ### Two MCP packages, two jobs `ai-mcp` and `mcp` both touch the Model Context Protocol, but from opposite ends: diff --git a/docs/packages/orm.md b/docs/packages/orm.md new file mode 100644 index 0000000..1dbefe6 --- /dev/null +++ b/docs/packages/orm.md @@ -0,0 +1,108 @@ +# orm + +`@gemstack/orm` is the **runtime** half of the data layer: how code reads and writes its data **without importing an ORM**. It is the runtime twin of [`schema`](/packages/schema) (the shape half) — you declare your tables once with the schema DSL, then talk to a neutral repository, and a per-ORM **adapter** executes the calls against the real database. + +```bash +npm install @gemstack/orm +``` + +Zero framework imports, zero ORM imports. Usable standalone by any framework or ORM. + +## The surface is narrow on purpose + +```js +db.users.insert(row) // -> inserted row +db.users.find(filter) // -> matching rows (array) +db.users.findOne(filter) // -> first match | null +db.users.upsert(row, { onConflict }) // -> upserted row +db.users.update(filter, patch) // -> updated rows (array) +db.users.delete(filter) // -> number of rows deleted +``` + +Filters are simple **equality** or **`in`** conditions — nothing more: + +```js +db.users.find({ active: true }) // equality +db.users.find({ id: { in: ['u1', 'u2'] } }) // membership +db.users.find() // all rows +``` + +Joins, aggregates, ranges, raw SQL — deliberately **out of scope**. Drop to the underlying ORM for those, the same escape hatch as DB-specific column types. This is not a query language (that is Kysely's job); it is the 90%-case repository. + +## Using it + +```js +import { createRepository } from '@gemstack/orm' +import { defineSchema, mergeSchemas } from '@gemstack/schema' + +const { tables } = mergeSchemas([ + defineSchema('users', (t) => { + t.uuid('id').primary() + t.string('email').unique() + t.boolean('active') + }), +]) + +const db = createRepository({ tables }, adapter) // adapter: see below +await db.users.upsert({ id: 'u1', email: 'a@b.com', active: true }, { onConflict: 'email' }) +``` + +Tables and their columns come from the **merged schema** (the output of `mergeSchemas`), the same single source the ORM artifacts are generated from. A typo'd column or an unknown table is a clear error, not a silent no-op. + +## The adapter contract + +The app installs **one** adapter and hands it the connection; consumers never import an ORM. An adapter implements six operations, each taking the table **name** first: + +```js +const adapter = { + insert(table, row), // -> inserted row + find(table, filter, opts), // -> rows[] (opts: { limit, offset, orderBy }) + count(table, filter), // -> number of matching rows + upsert(table, row, { onConflict }), // -> upserted row (onConflict: column names) + update(table, filter, patch), // -> updated rows[] + delete(table, filter), // -> number deleted +} +``` + +`findOne` is not an adapter op; the repository derives it from `find` (with `limit: 1`). + +In-process adapters can reuse the shared filter matcher so every adapter agrees on what a filter means; SQL adapters translate the same shape into a `WHERE` clause: + +```js +import { matchesFilter } from '@gemstack/orm' +matchesFilter(row, { active: true }) +``` + +> No transactions yet: the common operation is a single (atomic) upsert. + +## The adapter registry: one adapter, every consumer + +So the **app picks the backend once** and everything routes through it, the core holds a tiny runtime registry. The app calls `setAdapter(...)` at server start; each consumer reads the same adapter via `getAdapter()` and builds its repository over its own schema. Nothing hardcodes a backend, and nothing imports an ORM. + +```js +// the app, once at server start: +import { setAdapter } from '@gemstack/orm' +import { createDrizzleAdapter } from '@gemstack/orm-drizzle' +setAdapter(createDrizzleAdapter(drizzle(pool), schema)) // or createMemoryAdapter() + +// a consumer, anywhere: +import { getAdapter, createRepository } from '@gemstack/orm' +import { createMemoryAdapter } from '@gemstack/orm-memory' +const adapter = getAdapter() ?? createMemoryAdapter() // app's choice, else zero-config memory +const db = createRepository({ tables }, adapter) +``` + +`getAdapter()` returns `null` until the app sets one, so a consumer falls back to the memory adapter for zero-config dev/demo. `setAdapter` validates the six-op contract up front, and the registry is cached on `globalThis` so pointer-import / HMR double-eval can't fork it. `clearAdapter()` resets it (tests). + +## Adapters + +The repository is backend-agnostic; an adapter binds it to a real store. Two ship today: + +| Package | Backend | Use it for | +|---|---|---| +| [`orm-memory`](/packages/orm) (`@gemstack/orm-memory`) | plain in-process `Map`s | tests, demos, zero-config dev — no database | +| [`orm-drizzle`](/packages/orm) (`@gemstack/orm-drizzle`) | [Drizzle](https://orm.drizzle.team) over your connection | real databases (Postgres, etc.) | + +`orm-memory` reuses the core filter matcher, so it agrees with SQL adapters on what a filter means — the proof that the contract, not the backend, defines behavior. `orm-drizzle` declares `drizzle-orm` as a peer dependency. + +Authoring your own adapter is just implementing the six-op contract above. diff --git a/docs/packages/schema.md b/docs/packages/schema.md new file mode 100644 index 0000000..80da46d --- /dev/null +++ b/docs/packages/schema.md @@ -0,0 +1,79 @@ +# schema + +::: warning Preview / experimental +`@gemstack/schema` is an early spike. The API is still settling and the per-ORM compilers emit *representative* output — they do not yet run against real databases. Use it to explore the shape of the data layer, not in production. +::: + +`@gemstack/schema` is the **framework-agnostic core** of the data layer — the **shape** half, the twin of [`orm`](/packages/orm) (the runtime half). Zero framework imports, zero ORM imports. + +You describe your tables **once** as plain, declarative data; this package merges contributions from independent sources, derives the migration order, and compiles the result to Prisma, Drizzle, or a Rudder-engine artifact. + +```bash +npm install @gemstack/schema +``` + +## What's in the box + +- **A neutral schema DSL** — `defineSchema` / `extendSchema`. No ORM imported; the result is plain data (an IR). +- **Merge + derive** — `mergeSchemas` folds creates and third-party column adds into final tables (and flags column-edit conflicts); `deriveMigrations` produces the ordered migration names from the contributions. +- **Relations / foreign keys** — declare an FK with `.references('table.column', { onDelete })`; `deriveRelations` computes both ends (forward + inverse) so ORMs that model navigation (Prisma relation fields, Drizzle `relations()`) get them. Self-references, per-FK relation-field naming (`as` / `inverseAs`), composite primary keys (`t.primaryKey(a, b)`), and many-to-many via `defineJoinTable(a, b)` are all supported. +- **Per-ORM compilers** — `toPrisma`, `toDrizzle`, `toRudder` (and the `COMPILERS` map) turn one merged table into that ORM's schema. +- **File generation** — `generateArtifacts({ tables, fragments }, orm)` returns `[{ path, contents }]` ready to write to disk, each with a "generated, don't edit" header. Pure: it performs no filesystem access itself. + +## Usage + +```js +import { + defineSchema, + extendSchema, + defineJoinTable, + mergeSchemas, + deriveMigrations, + generateArtifacts, +} from '@gemstack/schema' + +// 1. Declare tables once (this is the source of truth). +const auth = defineSchema('users', (t) => { + t.uuid('id').primary() + t.string('email').unique() + t.timestamps() +}) + +const roles = defineSchema('roles', (t) => { + t.uuid('id').primary() + t.string('name').unique() +}) + +// 2. A different source can ADD columns to a table it didn't create. +const billing = extendSchema('users', (t) => { + t.string('stripe_customer_id').nullable() +}) + +// 3. Many-to-many is the join table that links two tables (two FKs + composite PK). +const userRoles = defineJoinTable('users', 'roles') +// -> `roles_users` { user_id -> users.id, role_id -> roles.id, primaryKey(both) } + +// 4. Merge + derive. +const fragments = [auth, roles, billing, userRoles] +const { tables, conflicts } = mergeSchemas(fragments) +const migrations = deriveMigrations(fragments) + +// 5. Generate committable artifacts for the ORM of your choice. +const files = generateArtifacts({ tables, fragments }, 'prisma') +// -> [{ path: 'prisma/schema.generated.prisma', contents: '// GENERATED ...' }] +``` + +## Design + +- **Declarative (desired-state) is the right shape.** Prisma and Drizzle diff state, and the Rudder engine generates an ordered migration from it, so authoring desired state and deriving everything downstream fits all three. +- **Migrations are an output, not authored by hand.** Schema is the single source of truth; the ORM schema becomes generated output (the usual model, inverted). +- **Framework-free on purpose.** A framework binding (e.g. the Vike one, `@vike-data/vike-schema`) lives in a separate package; this core is the part meant to be reusable anywhere. + +## Scope / deferred (the interesting hard parts) + +- **Types:** `uuid` / `string` / `text` / `integer` / `boolean` / `timestamp` plus `nullable` / `unique` / `primary` / `default` / `timestamps()`. +- **Relations / foreign keys:** single-column FKs, `onDelete`, cross-source reference validation, self-references, relation-field naming, composite primary keys, and many-to-many sugar all ship. Still deferred: composite (multi-column) foreign keys, and one-to-one inference beyond a `unique` FK. +- **Type escape hatches:** DB-specific types (pg arrays, enums, JSON) want a per-adapter override so the neutral layer isn't lowest-common-denominator. +- **Declarative → ordered migration reconciliation:** real diffing/ordering. +- **Column-edit policy:** conflicts are detected; resolution is unspecified. +- Compilers emit representative artifacts; they don't run against real DBs yet.