feat(tables): fractional order keys for O(log n) row insert/delete (flag-gated, default off) (#4890)

* feat(tables): add order_key column, fractional-indexing util, and ordering flag (off)

* feat(tables): write order_key on insert, flag-gate delete reindex + query ordering, add backfill

Flag off (default) = identical behavior. Single-insert assigns a fractional
order_key; queryRows orders by order_key when the flag is on; deletes skip the
O(N) reindex when on. Per-table-atomic backfill script populates existing rows.

* feat(tables): write order_key on all insert paths (batch, upsert, replace, import, create, copilot)

Completes the always-write-keys prerequisite: every row insert now assigns a
fractional order_key consistent with position order, so the flag can be flipped
safely after backfill. Flag off (default) still = identical behavior.

* feat(tables): insert-by-neighbor-id + orderKey on wire + client order-by-key

Inserts express intent as afterRowId/beforeRowId (O(1) key mint via the
(table_id,order_key,id) index); orderKey is returned on every row; client
reconcile/undo place by orderKey (no neighbor bump) with position fallback.
Flag off = unchanged. 205 table tests pass.

* feat(tables): resolve position-based inserts by key ordinal under the flag

Position-based callers (mothership tool, v1 API, undo fallback, transient old
clients) resolve their insert neighbor by order_key ordinal (OFFSET) when the
flag is on — positions are gappy then, so WHERE position=N would miss. Flag off
keeps the indexed position lookup. The mothership tool itself is unchanged.

* test(tables): flag-on coverage — delete skips reindex, insert mints key + no shift

* fix lint

* chore(db): regenerate order_key migration with default drizzle name

* fix(tables): address review — guard neighbor insert + mutual-exclusion + safe reconcile

- resolveInsertByNeighbor throws when the anchor row is missing (was silently
  inserting at the front) and when its order_key is null under the flag.
- insert contract: afterRowId/beforeRowId are mutually exclusive (refine).
- reconcileCreatedRow only key-sorts when every cached row is keyed, so mid-
  backfill un-keyed rows aren't yanked to the front.

* fix(kb): restore non-null guard in storage-key filter (unsafe-lint regression)

* refactor(tables): extract maxOrderKey + thread import append key

- Extract maxOrderKey(executor, tableId) helper; replaces three identical
  max(order_key) selects (single/batch insert append + import).
- Import: read the append anchor once up front and thread each batch's last
  key forward (nextImportStartOrderKey + afterOrderKey) instead of re-scanning
  max(order_key) per batch over a growing table — one scan per import, not
  one per 1k-row batch.

* fix(tables): keep insert body base omittable for v1 contract

The afterRowId/beforeRowId mutual-exclusion .refine() turned the schema into a
ZodEffects, which Zod forbids .omit() on — v1's insertTableRowBodySchema.omit({
position }) threw at module load (runtime-only; tsc misses it). Split the plain
object base out, apply the shared refine on top, and have v1 omit from the base
then re-apply it.

* fix(tables): chunk backfill order-key writes

A single UPDATE … FROM (VALUES …) over a whole large table overflows the JS call
stack while drizzle assembles the VALUES list (and would blow past Postgres's
65535 bound-param limit at ~32k rows) — large tables failed with 'Maximum call
stack size exceeded'. Write in 1000-row chunks inside the same per-table
transaction so keying stays atomic.

* fix(tables): emit orderKey in insert responses

The single-row and batch insert handlers dropped orderKey from the JSON
response even though the service returns it, so reconcileCreatedRow always fell
back to position-sorting and could place neighbor inserts wrong under the
fractional-ordering flag. Serialize orderKey alongside position.

* fix(tables): restore by orderKey, not position, under fractional flag

A saved position is the gappy column value, but under the flag insert reads
position as a visual rank (OFFSET) — so position-based restore misplaces rows.

- create-row redo now goes through the batch path carrying the saved orderKey
  (the single-insert API has no orderKey field); drop the now-unused single
  create mutation.
- resolveBatchInsertOrderKeys appends under the flag instead of feeding gappy
  positions to resolveInsertOrderKey; positions remain the flag-off path.

* perf(tables): backfill writes 5000 rows/chunk (was 1000)

5x fewer round-trips per table; ~10k bound params stays well under Postgres's
65535 ceiling and far below the single-statement size that overflows the stack.

* fix(tables): drop rowNumber from table trigger payload

position is gappy under the fractional-ordering flag, so rowNumber (= row.position)
no longer reflects a contiguous visual rank. Rather than compute-on-read, remove
it from the trigger payload, output schema, and column-execution input.

Also pin isTablesFractionalOrderingEnabled=false in update-row.test.ts so its
flag-off position-shift assertions are deterministic regardless of local env.

* chore(db): format generated 0226 migration metadata

biome check . flagged the drizzle-generated _journal.json and 0226_snapshot.json;
apply the formatter so packages/db lint:check passes in CI.

* docs(triggers): drop rowNumber from table trigger outputs

rowNumber was removed from the table trigger payload; remove it from the
documented output fields to match.

* test(tables): remove flag-on fractional-ordering unit suite

Flag-on behavior is covered by manual large-table verification; the heavily-
mocked DB-chain suite added little signal.

Author: TheodoreSpeaks

apps/docs/content/docs/en/triggers/table.mdx

@@ -38,7 +38,6 @@ Triggers when rows are inserted or updated in a table
 | `changedColumns` | json | List of column names that changed \(empty for inserts\) |
 | `rowId` | string | The unique row ID |
 | `headers` | json | Column names from the table schema |
-| `rowNumber` | number | The position of the row in the table |
 | `tableId` | string | The table ID |
 | `tableName` | string | The table name |
 | `timestamp` | string | Event timestamp in ISO format |

apps/sim/app/api/table/[tableId]/rows/route.ts

@@ -71,6 +71,7 @@ async function handleBatchInsert(
         workspaceId: validated.workspaceId,
         userId,
         positions: validated.positions,
+        orderKeys: validated.orderKeys,
       },
       table,
       requestId
@@ -83,6 +84,7 @@ async function handleBatchInsert(
           id: r.id,
           data: r.data,
           position: r.position,
+          orderKey: r.orderKey ?? undefined,
           createdAt: r.createdAt instanceof Date ? r.createdAt.toISOString() : r.createdAt,
           updatedAt: r.updatedAt instanceof Date ? r.updatedAt.toISOString() : r.updatedAt,
         })),
@@ -162,6 +164,8 @@ export const POST = withRouteHandler(
           workspaceId: validated.workspaceId,
           userId: authResult.userId,
           position: validated.position,
+          afterRowId: validated.afterRowId,
+          beforeRowId: validated.beforeRowId,
         },
         table,
         requestId
@@ -174,9 +178,11 @@ export const POST = withRouteHandler(
             id: row.id,
             data: row.data,
             position: row.position,
+            orderKey: row.orderKey ?? undefined,
             createdAt: row.createdAt instanceof Date ? row.createdAt.toISOString() : row.createdAt,
             updatedAt: row.updatedAt instanceof Date ? row.updatedAt.toISOString() : row.updatedAt,
           },
+
           message: 'Row inserted successfully',
         },
       })

apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/table-grid.tsx

@@ -837,9 +837,12 @@ export function TableGrid({
 
   function handleInsertRow(offset: 0 | 1) {
     if (!contextMenu.row) return
+    const anchorId = contextMenu.row.id
+    // Fractional ordering: express intent by neighbor id, not integer position.
+    const intent = offset === 0 ? { beforeRowId: anchorId } : { afterRowId: anchorId }
     const position = contextMenu.row.position + offset
     createRef.current(
-      { data: {}, position },
+      { data: {}, ...intent },
       {
         onSuccess: (response: Record<string, unknown>) => {
           const newRowId = extractCreatedRowId(response)
@@ -904,7 +907,7 @@ export function TableGrid({
     const sourceArrayIndex = rowsRef.current.findIndex((r) => r.id === contextRow.id)
     closeContextMenu()
     createRef.current(
-      { data: rowData, position },
+      { data: rowData, afterRowId: contextRow.id },
       {
         onSuccess: (response: Record<string, unknown>) => {
           const newRowId = extractCreatedRowId(response)

apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/utils.ts

@@ -302,7 +302,12 @@ export function computeNormalizedSelection(
 export function collectRowSnapshots(rows: Iterable<TableRowType>): DeletedRowSnapshot[] {
   const snapshots: DeletedRowSnapshot[] = []
   for (const row of rows) {
-    snapshots.push({ rowId: row.id, data: { ...row.data }, position: row.position })
+    snapshots.push({
+      rowId: row.id,
+      data: { ...row.data },
+      position: row.position,
+      orderKey: row.orderKey,
+    })
   }
   return snapshots
 }

apps/sim/background/workflow-column-execution.ts

@@ -589,7 +589,6 @@ async function runWorkflowAndWriteTerminal(
         changedColumns: [],
         rowId,
         headers,
-        rowNumber: row.position,
         tableId,
         tableName,
         timestamp: new Date().toISOString(),

apps/sim/hooks/queries/tables.ts

@@ -512,7 +512,13 @@ export function useCreateTableRow({ workspaceId, tableId }: RowMutationContext)
     ) => {
       return requestJson(createTableRowContract, {
         params: { tableId },
-        body: { workspaceId, data: variables.data as RowData, position: variables.position },
+        body: {
+          workspaceId,
+          data: variables.data as RowData,
+          position: variables.position,
+          afterRowId: variables.afterRowId,
+          beforeRowId: variables.beforeRowId,
+        },
       })
     },
     onSuccess: (response) => {
@@ -592,35 +598,47 @@ function reconcileCreatedRow(
       if (!old) return old
       if (old.pages.some((p) => p.rows.some((r) => r.id === row.id))) return old
 
-      const pages = old.pages.map((page) =>
-        page.rows.some((r) => r.position >= row.position)
-          ? {
-              ...page,
-              rows: page.rows.map((r) =>
-                r.position >= row.position ? { ...r, position: r.position + 1 } : r
-              ),
-            }
-          : page
-      )
+      // Use key-ordering only when the new row AND every cached row have an
+      // `orderKey` — then no neighbor bump is needed and order is exact. If any
+      // cached row is un-keyed (mid-backfill), fall back to the legacy `position`
+      // path so un-keyed rows aren't yanked to the front by an empty-string sort.
+      const byKey =
+        row.orderKey != null && old.pages.every((p) => p.rows.every((r) => r.orderKey != null))
+      const sortRows = (rows: TableRow[]) =>
+        byKey
+          ? [...rows].sort((a, b) => (a.orderKey as string).localeCompare(b.orderKey as string))
+          : [...rows].sort((a, b) => a.position - b.position)
+      const fitsAfter = (last: TableRow | undefined) =>
+        last === undefined ||
+        (byKey
+          ? (last.orderKey as string) >= (row.orderKey as string)
+          : last.position >= row.position)
+
+      const pages = byKey
+        ? old.pages
+        : old.pages.map((page) =>
+            page.rows.some((r) => r.position >= row.position)
+              ? {
+                  ...page,
+                  rows: page.rows.map((r) =>
+                    r.position >= row.position ? { ...r, position: r.position + 1 } : r
+                  ),
+                }
+              : page
+          )
 
       let inserted = false
       const nextPages = pages.map((page) => {
         if (inserted) return page
-        const last = page.rows[page.rows.length - 1]
-        const fits = last === undefined || last.position >= row.position
-        if (!fits) return page
+        if (!fitsAfter(page.rows[page.rows.length - 1])) return page
         inserted = true
-        const merged = [...page.rows, row].sort((a, b) => a.position - b.position)
-        return { ...page, rows: merged }
+        return { ...page, rows: sortRows([...page.rows, row]) }
       })
 
       if (!inserted && nextPages.length > 0) {
         const lastIdx = nextPages.length - 1
         const lastPage = nextPages[lastIdx]
-        nextPages[lastIdx] = {
-          ...lastPage,
-          rows: [...lastPage.rows, row].sort((a, b) => a.position - b.position),
-        }
+        nextPages[lastIdx] = { ...lastPage, rows: sortRows([...lastPage.rows, row]) }
       }
 
       const firstPage = nextPages[0]
@@ -655,6 +673,7 @@ export function useBatchCreateTableRows({ workspaceId, tableId }: RowMutationCon
           workspaceId,
           rows: variables.rows as RowData[],
           positions: variables.positions,
+          orderKeys: variables.orderKeys,
         },
       })
     },

apps/sim/hooks/use-table-undo.ts

@@ -5,7 +5,6 @@ import {
   useAddTableColumn,
   useBatchCreateTableRows,
   useBatchUpdateTableRows,
-  useCreateTableRow,
   useDeleteColumn,
   useDeleteTableRow,
   useDeleteTableRows,
@@ -56,7 +55,6 @@ export function useTableUndo({
   const canRedo = useTableUndoStore((s) => (s.stacks[tableId]?.redo.length ?? 0) > 0)
 
   const updateRowMutation = useUpdateTableRow({ workspaceId, tableId })
-  const createRowMutation = useCreateTableRow({ workspaceId, tableId })
   const batchCreateRowsMutation = useBatchCreateTableRows({ workspaceId, tableId })
   const batchUpdateRowsMutation = useBatchUpdateTableRows({ workspaceId, tableId })
   const deleteRowMutation = useDeleteTableRow({ workspaceId, tableId })
@@ -137,11 +135,18 @@ export function useTableUndo({
             if (direction === 'undo') {
               deleteRowMutation.mutate(action.rowId)
             } else {
-              createRowMutation.mutate(
-                { data: action.data ?? {}, position: action.position },
+              // Redo via the batch path so the saved orderKey restores exact placement.
+              // The single-insert API has no orderKey field, and under the fractional-ordering
+              // flag its `position` is read as a rank — a gappy saved position misplaces.
+              batchCreateRowsMutation.mutate(
+                {
+                  rows: [action.data ?? {}],
+                  positions: [action.position],
+                  orderKeys: action.orderKey ? [action.orderKey] : undefined,
+                },
                 {
                   onSuccess: (response) => {
-                    const newRowId = extractCreatedRowId(response as Record<string, unknown>)
+                    const newRowId = response?.data?.rows?.[0]?.id
                     if (newRowId && newRowId !== action.rowId) {
                       patchUndoRowId(tableId, action.rowId, newRowId)
                     }
@@ -165,6 +170,9 @@ export function useTableUndo({
                 {
                   rows: action.rows.map((r) => r.data),
                   positions: action.rows.map((r) => r.position),
+                  orderKeys: action.rows.every((r) => r.orderKey)
+                    ? action.rows.map((r) => r.orderKey as string)
+                    : undefined,
                 },
                 {
                   onSuccess: (response) => {
@@ -187,6 +195,9 @@ export function useTableUndo({
                 {
                   rows: action.rows.map((row) => row.data),
                   positions: action.rows.map((row) => row.position),
+                  orderKeys: action.rows.every((row) => row.orderKey)
+                    ? action.rows.map((row) => row.orderKey as string)
+                    : undefined,
                 },
                 {
                   onSuccess: (response) => {

apps/sim/lib/api/contracts/tables.ts

@@ -147,12 +147,29 @@ export const rowDataSchema = domainObjectSchema<RowData>()
 export const tableDefinitionSchema = domainObjectSchema<TableDefinition>()
 export const tableRowSchema = domainObjectSchema<TableRow>()
 
-export const insertTableRowBodySchema = z.object({
+/**
+ * Plain-object base for the single-row insert body. Kept un-refined so callers
+ * (e.g. the v1 public contract) can `.omit()` fields before applying
+ * {@link rowAnchorMutexRefine} — Zod forbids `.omit()` on a refined schema.
+ */
+export const insertTableRowBodyBaseSchema = z.object({
   workspaceId: z.string().min(1, 'Workspace ID is required'),
   data: rowDataSchema,
   position: z.number().int().min(0).optional(),
+  /** Fractional ordering: insert directly after this row id. Takes precedence over `position`. */
+  afterRowId: z.string().min(1).optional(),
+  /** Fractional ordering: insert directly before this row id. Takes precedence over `position`. */
+  beforeRowId: z.string().min(1).optional(),
 })
 
+/** `afterRowId` and `beforeRowId` are mutually exclusive insert anchors. */
+export const rowAnchorMutexRefine = [
+  (data: { afterRowId?: string; beforeRowId?: string }) => !data.afterRowId || !data.beforeRowId,
+  { message: 'afterRowId and beforeRowId are mutually exclusive' },
+] as const
+
+export const insertTableRowBodySchema = insertTableRowBodyBaseSchema.refine(...rowAnchorMutexRefine)
+
 /**
  * POST `/api/table/[tableId]/rows/upsert` body — insert-or-update keyed by a
  * unique column name. `conflictTarget` is optional (server picks a single
@@ -175,13 +192,18 @@ export const batchInsertTableRowsBodySchema = z
         `Cannot insert more than ${TABLE_LIMITS.MAX_BATCH_INSERT_SIZE} rows per batch`
       ),
     positions: z.array(z.number().int().min(0)).max(TABLE_LIMITS.MAX_BATCH_INSERT_SIZE).optional(),
+    /** Fractional ordering: exact per-row order keys (undo restore). Takes precedence over `positions`. */
+    orderKeys: z.array(z.string().min(1)).max(TABLE_LIMITS.MAX_BATCH_INSERT_SIZE).optional(),
   })
   .refine((data) => !data.positions || data.positions.length === data.rows.length, {
     message: 'positions array length must match rows array length',
   })
   .refine((data) => !data.positions || new Set(data.positions).size === data.positions.length, {
     message: 'positions must not contain duplicates',
   })
+  .refine((data) => !data.orderKeys || data.orderKeys.length === data.rows.length, {
+    message: 'orderKeys array length must match rows array length',
+  })
 
 /**
  * POST `/api/table/[tableId]/rows` body — accepts either a batch payload

apps/sim/lib/api/contracts/v1/tables/index.ts

@@ -4,7 +4,8 @@ import {
   createTableColumnBodySchema,
   deleteTableColumnBodySchema,
   deleteTableRowsBodySchema,
-  insertTableRowBodySchema,
+  insertTableRowBodyBaseSchema,
+  rowAnchorMutexRefine,
   rowDataSchema,
   tableIdParamsSchema,
   tableRowParamsSchema,
@@ -60,7 +61,9 @@ export const v1CreateTableBodySchema = createTableBodySchema.omit({
  * Public API insert row body — no caller-controlled `position`. Server places
  * new rows at the tail; ordering by index is an in-app affordance only.
  */
-export const v1InsertTableRowBodySchema = insertTableRowBodySchema.omit({ position: true })
+export const v1InsertTableRowBodySchema = insertTableRowBodyBaseSchema
+  .omit({ position: true })
+  .refine(...rowAnchorMutexRefine)
 
 /**
  * Public API batch insert body — no `positions`. Same rationale as above.

apps/sim/lib/copilot/request/tools/tables.ts

@@ -13,6 +13,7 @@ import { TraceSpan } from '@/lib/copilot/generated/trace-spans-v1'
 import { withCopilotSpan } from '@/lib/copilot/request/otel'
 import type { ExecutionContext, ToolCallResult } from '@/lib/copilot/request/types'
 import type { RowData } from '@/lib/table'
+import { nKeysBetween } from '@/lib/table/order-key'
 import { buildOrderedRowValues, getTableById } from '@/lib/table/service'
 
 const logger = createLogger('CopilotToolResultTables')
@@ -103,6 +104,8 @@ export async function maybeWriteOutputToTable(
           await tx.delete(userTableRows).where(eq(userTableRows.tableId, outputTable))
 
           const now = new Date()
+          // Replace-all: table was just cleared — mint a fresh contiguous key run.
+          const orderKeys = nKeysBetween(null, null, rows.length)
           for (let i = 0; i < rows.length; i += BATCH_CHUNK_SIZE) {
             if (context.abortSignal?.aborted) {
               throw new Error('Request aborted before tool mutation could be applied')
@@ -113,6 +116,7 @@ export async function maybeWriteOutputToTable(
               workspaceId: context.workspaceId!,
               rows: chunk as RowData[],
               startPosition: i,
+              orderKeys: orderKeys.slice(i, i + BATCH_CHUNK_SIZE),
               now,
               createdBy: context.userId,
               makeId: () => `row_${generateId().replace(/-/g, '')}`,
@@ -246,6 +250,8 @@ export async function maybeWriteReadCsvToTable(
           await tx.delete(userTableRows).where(eq(userTableRows.tableId, outputTable))
 
           const now = new Date()
+          // Replace-all: table was just cleared — mint a fresh contiguous key run.
+          const orderKeys = nKeysBetween(null, null, rows.length)
           for (let i = 0; i < rows.length; i += BATCH_CHUNK_SIZE) {
             if (context.abortSignal?.aborted) {
               throw new Error('Request aborted before tool mutation could be applied')
@@ -256,6 +262,7 @@ export async function maybeWriteReadCsvToTable(
               workspaceId: context.workspaceId!,
               rows: chunk as RowData[],
               startPosition: i,
+              orderKeys: orderKeys.slice(i, i + BATCH_CHUNK_SIZE),
               now,
               createdBy: context.userId,
               makeId: () => `row_${generateId().replace(/-/g, '')}`,