diff --git a/.agents/skills/ci-prep/SKILL.md b/.agents/skills/ci-prep/SKILL.md index 040221ff..f990d876 100644 --- a/.agents/skills/ci-prep/SKILL.md +++ b/.agents/skills/ci-prep/SKILL.md @@ -3,6 +3,7 @@ name: ci-prep description: Prepares the current branch for CI by running the exact same steps locally and fixing issues. If CI is already failing, fetches the GH Actions logs first to diagnose. Use before pushing, when CI is red, or when the user says "fix ci". argument-hint: "[--failing] [optional job name to focus on]" --- + # CI Prep @@ -15,7 +16,7 @@ Prepare the current state for CI. If CI is already failing, fetch and analyze th If `--failing` is NOT passed, skip directly to **Step 2**. -## Step 1 �� Fetch failed CI logs (only when `--failing`) +## Step 1 — Fetch failed CI logs (only when `--failing`) You MUST do this before any other work. @@ -41,11 +42,27 @@ Read **every line** of `--log-failed` output. For each failure note the exact fi ## Step 2 — Analyze the CI workflow -1. Read `.github/workflows/ci.yml` completely. Parse every job and every step. -2. Extract the ordered list of commands the CI actually runs. -3. Note environment variables, matrix strategies, conditional steps, and service containers. +1. Find the CI workflow file. Look in `.github/workflows/` for `ci.yml`, `build.yml`, `test.yml`, `checks.yml`, `main.yml`, `pull_request.yml`, or any workflow triggered on `pull_request` or `push`. +2. Read the workflow file completely. Parse every job and every step. +3. Extract the ordered list of commands the CI actually runs. In a spec-compliant repo this is `make lint → make test → make build` (REPO-STANDARDS-SPEC [MAKE-TARGETS]), but the actual CI may use `npm`, `cargo`, `dotnet`, raw shell commands, or anything else. Extract what is *actually there*. +4. Note any environment variables, matrix strategies, or conditional steps that affect execution. -**Do NOT assume the steps are `make lint`, `make test`, `make build`.** Extract what the CI *actually does*. +**Do NOT assume the steps are `make lint`, `make test`, `make build`.** The actual CI may run different commands, in a different order. Extract what the CI *actually does*. If you find extra targets beyond the 7 in [MAKE-TARGETS] (e.g. `make fmt-check`, `make coverage-check`), flag them in your final report — they should be consolidated by the agent-pmo skill. + +### Release workflow blocker scan + +If `.github/workflows/release.yml` exists, scan it before broad local CI. These are critical blockers +and must be fixed before release work is considered CI-ready: + +- Tag-triggered jobs checking out `ref: main` instead of the tagged SHA. +- Any `git commit`, `git push`, branch mutation, or tag mutation during release. +- Version bump commits after the tag already exists. +- Ad hoc `sed` version stamping of structured files instead of a first-class stamper/build input. +- Missing tests that pass a test version into the same stamper used by release. +- Native VSIX releases without Node `22.x`, `npx vsce package --target `, one VSIX per + target, target-suffixed filenames, and package-content verification. +- VS Code native-binary activation that reads or mutates PATH, uses package-manager/global installs + as normal startup sources, or copies bundled VSIX binaries after install. ## Step 3 — Run each CI step locally, in order @@ -58,7 +75,7 @@ Work through failures in this priority order: For each command extracted from the CI workflow: -1. Run the command exactly as CI would run it. +1. Run the command exactly as CI would run it (adjusting only for local environment differences like not needing `actions/checkout`). 2. If the step fails, **stop and fix the issues** before continuing to the next step. 3. After fixing, re-run the same step to confirm it passes. 4. Move to the next step only after the current one succeeds. @@ -66,37 +83,39 @@ For each command extracted from the CI workflow: ### Hard constraints - **NEVER modify test files** — fix the source code, not the tests -- **NEVER add suppressions** (`#pragma warning disable`, `#[allow(...)]`, `// eslint-disable`) +- **NEVER add suppressions** (`#[allow(...)]`, `// eslint-disable`, `#pragma warning disable`) +- **NEVER use `any` in TypeScript** to silence type errors - **NEVER delete or ignore failing tests** - **NEVER remove assertions** If stuck on the same failure after 5 attempts, ask the user for help. -## Step 4 — Loop +## Step 4 — Report -- Go back to the first step and repeat until all steps pass locally. If `--failing`, you should see the exact same errors in your terminal that CI shows in the logs. Fix those errors until they are resolved. +- List every step that was run and its result (pass/fail/fixed). +- If any step could not be fixed, report what failed and why. +- Confirm whether the branch is ready to push. -## Step 5 — Commit/Push (only when `--failing`) +## Step 5 — Remote CI follow-up (only when `--failing`) Once all CI steps pass locally: -1. Commit, but DO NOT MARK THE COMMIT WITH YOU AS AN AUTHOR!!! Only the user authors the commit! -2. Push -3. Monitor until completion or failure -4. Upon failure, go back to Step 1 +1. Report the local fixes and exact commands that now pass. +2. Do not commit or push. The user owns source-control writes. +3. If the user pushes, monitor the new run until completion or failure. +4. Upon failure, go back to Step 1. ## Rules -- *You are not allowed to commi/push until all tests pass*. Do not waste GitHub action minutes! The local CI must prove that everything is working. - **Always read the CI workflow first.** Never assume what commands CI runs. -- Do not push if any step fails (unless `--failing` and all steps now pass) +- Do not commit or push from this skill. - Fix issues found in each step before moving to the next - Never skip steps or suppress errors - If the CI workflow has multiple jobs, run all of them (respecting dependency order) -- Skip steps that are CI-infrastructure-only (checkout, setup actions, cache steps, artifact uploads) — focus on the actual build/test/lint commands +- Skip steps that are CI-infrastructure-only (checkout, setup-node/python/rust actions, cache steps, artifact uploads) — focus on the actual build/test/lint commands ## Success criteria - Every command that CI runs has been executed locally and passed - All fixes are applied to the working tree -- The CI passes successfully (if you are correcting an existing failure) +- The CI passes successfully (if you are correcting and existing failure) diff --git a/.agents/skills/code-dedup/SKILL.md b/.agents/skills/code-dedup/SKILL.md index 6f49707c..e0cd6363 100644 --- a/.agents/skills/code-dedup/SKILL.md +++ b/.agents/skills/code-dedup/SKILL.md @@ -2,6 +2,7 @@ name: code-dedup description: Searches for duplicate code, duplicate tests, and dead code, then safely merges or removes them. Use when the user says "deduplicate", "find duplicates", "remove dead code", "DRY up", or "code dedup". Requires test coverage — refuses to touch untested code. --- + # Code Dedup @@ -12,8 +13,11 @@ Carefully search for duplicate code, duplicate tests, and dead code across the r Before touching ANY code, verify these conditions. If any fail, stop and report why. 1. Run `make test` — all tests must pass. If tests fail, stop. Do not dedup a broken codebase. -2. Run `make coverage-check` — coverage must meet the repo's threshold. If it doesn't, stop. -3. This repo uses **C#, F#, Rust, and TypeScript** — all statically typed. Proceed. +2. Run `make test` — tests are fail-fast AND enforce the coverage threshold from `coverage-thresholds.json`. If anything fails, stop and fix it before deduping. +3. Verify the project uses **static typing**. Check for: + - Rust, C#, F#, Dart, Go: typed by default — proceed + - TypeScript: `tsconfig.json` must have `"strict": true` — proceed (Lql/LqlExtension already does) + - **Untyped JavaScript: STOP. Refuse to dedup.** Print: "This codebase has no static type checking. Deduplication without types is reckless — too high a risk of silent breakage. Add type checking first." ## Steps @@ -33,7 +37,7 @@ Dedup Progress: Before deciding what to touch, understand what is tested. -1. Run `make test` and `make coverage-check` to confirm green baseline +1. Run `make test` to confirm green baseline. `make test` is fail-fast AND enforces the coverage threshold from `coverage-thresholds.json` (REPO-STANDARDS-SPEC [TEST-RULES], [COVERAGE-THRESHOLDS-JSON]). It exits non-zero on any test failure OR coverage shortfall. 2. Note the current coverage percentage — this is the floor. It must not drop. 3. Identify which files/modules have coverage and which do not. Only files WITH coverage are candidates for dedup. @@ -41,66 +45,71 @@ Before deciding what to touch, understand what is tested. Search for code that is never called, never imported, never referenced. -1. Look for unused exports, unused functions, unused records, unused variables -2. Use language-appropriate tools: - - **C#/F#:** Analyzer warnings for unused members (build with `-warnaserror` catches these) - - **Rust:** The compiler already warns on dead code — check `make lint` output - - **TypeScript:** Check for unexported functions with zero references in `Lql/LqlExtension/` -3. For each candidate: **grep the entire codebase** for references (including tests, scripts, configs). Only mark as dead if truly zero references. +1. Look for unused exports, unused functions, unused classes, unused variables. +2. Use language-appropriate tools where available: + - C#/F#: analyzer warnings (CA1801 unused parameters, IDE0051 unused private members) via `make lint` + - Rust: `cargo clippy` already warns on dead code — check `make lint` output + - TypeScript: `noUnusedLocals`/`noUnusedParameters` in `tsconfig.json`; look for unexported functions with zero references +3. For each candidate: **grep the entire repo** (including tests, scripts, samples). Skip generated code under `Lql/lql-lsp-rust/crates/lql-parser/src/generated/`. Only mark as dead if truly zero references. 4. List all dead code found with file paths and line numbers. Do NOT delete yet. ### Step 3 — Scan for duplicate code Search for code blocks that do the same thing in multiple places. -1. Look for functions/methods with identical or near-identical logic -2. Look for copy-pasted blocks (same structure, maybe different variable names) -3. Look for multiple implementations of the same algorithm or pattern -4. Check across module boundaries — duplicates often hide in different projects (DataProvider, Lql, Sync, Gatekeeper, Samples) -5. For each duplicate pair: note both locations, what they do, and how they differ (if at all) +1. Look for functions/methods with identical or near-identical logic. +2. Look for copy-pasted blocks (same structure, maybe different variable names). +3. Look for multiple implementations of the same algorithm — particularly likely across `DataProvider/`, `Migration/`, `Sync/`, `Reporting/` C# projects, and across `Lql/Nimblesite.Lql.Core` SQL transpiler dialects (`Postgres`, `SqlServer`, `SQLite`). +4. Check across module boundaries — duplicates often hide in different `.csproj` projects or Rust crates (`lql-parser`, `lql-analyzer`, `lql-lsp`). +5. For each duplicate pair: note both locations, what they do, and how they differ (if at all). 6. List all duplicates found. Do NOT merge yet. ### Step 4 — Scan for duplicate tests Search for tests that verify the same behavior. -1. Look for test functions with identical assertions against the same code paths -2. Look for test fixtures/helpers that are duplicated across test files -3. Look for integration tests that fully cover what a unit test also covers (keep the integration test, mark the unit test as redundant per CLAUDE.md rules) +1. Look for test functions with identical assertions against the same code paths. +2. Look for test fixtures/helpers that are duplicated across test projects (`Tests.Shared/` is meant to hold these). +3. Look for integration tests that fully cover what a unit test also covers (keep the integration test, mark the unit test as redundant per CLAUDE.md "Prefer E2E/integration tests"). 4. List all duplicate tests found. Do NOT delete yet. ### Step 5 — Apply changes (one at a time) -For each change, follow this cycle: **change -> test -> verify coverage -> continue or revert**. +For each change, follow this cycle: **change → test → verify coverage → continue or revert**. #### 5a. Remove dead code -- Delete dead code identified in Step 2 -- After each deletion: run `make test` and `make coverage-check` -- If tests fail or coverage drops: **revert immediately** and investigate +- Delete dead code identified in Step 2. +- After each deletion: run `make test` (fail-fast + coverage + threshold all in one). +- If `make test` exits non-zero (test failure OR coverage drop): **revert immediately** and investigate. +- Dead code removal should never break tests or drop coverage. #### 5b. Merge duplicate code -- For each duplicate pair: extract the shared logic into a single function/module -- Update all call sites to use the shared version -- After each merge: run `make test` and `make coverage-check` +- For each duplicate pair: extract the shared logic into a single function/module. Prefer placing shared code in `Tests.Shared/` (for test helpers) or the closest `.Core` project (e.g. `Sync/Nimblesite.Sync.Core`, `Lql/Nimblesite.Lql.Core`). +- Update all call sites to use the shared version. +- After each merge: run `make test`. - If tests fail: **revert immediately**. The duplicates may have subtle differences you missed. +- If coverage drops: the shared code must have equivalent test coverage. Add tests if needed before proceeding. #### 5c. Remove duplicate tests -- Delete the redundant test (keep the more thorough one) -- After each deletion: run `make coverage-check` -- If coverage drops: **revert immediately**. The "duplicate" test was covering something the other wasn't. +- Delete the redundant test (keep the more thorough one). +- After each deletion: run `make test`. +- If coverage drops below threshold, `make test` exits non-zero — **revert immediately**. The "duplicate" test was covering something the other wasn't. ### Step 6 — Final verification -1. Run `make test` — all tests must still pass -2. Run `make coverage-check` — coverage must be >= the baseline from Step 1 -3. Run `make lint` and `make fmt-check` — code must be clean -4. Report: what was removed, what was merged, final coverage vs baseline +1. Run `make lint` — all linters and the format check must pass. +2. Run `make test` — tests must pass AND coverage must remain ≥ the baseline from Step 1. +3. Report: what was removed, what was merged, final coverage vs baseline. + +(Only the 7 standard targets exist — `make lint` and `make test` cover formatting and coverage checks respectively.) ## Rules -- **No test coverage = do not touch.** If a file has no tests covering it, leave it alone entirely. -- **Coverage must not drop.** The coverage floor from Step 1 is sacred. +- **No test coverage = do not touch.** If a file has no tests covering it, leave it alone entirely. You cannot safely dedup what you cannot verify. +- **Coverage must not drop.** If removing or merging code causes coverage to decrease, revert and investigate. The coverage floor from Step 1 is sacred. +- **Untyped code = refuse to dedup.** This repo has no untyped surfaces today; if any appear, refuse. - **One change at a time.** Make one dedup change, run tests, verify coverage. Never batch multiple dedup changes before testing. -- **When in doubt, leave it.** If two code blocks look similar but you're not 100% sure they're functionally identical, leave both. -- **Preserve public API surface.** Do not change function signatures, record names, or module exports that external code depends on. Internal refactoring only. -- **Three similar lines is fine.** Only dedup when the shared logic is substantial (>10 lines) or when there are 3+ copies. +- **When in doubt, leave it.** If two code blocks look similar but you're not 100% sure they're functionally identical, leave both. False dedup is worse than duplication. +- **Preserve public API surface.** Do not change the public surface of `Nimblesite.DataProvider.Core`, `Nimblesite.Lql.Core`, `Nimblesite.Sync.Core`, or any package that is published to NuGet. Internal refactoring only. +- **Three similar lines is fine.** Do not create abstractions for trivial duplication. Only dedup when the shared logic is substantial (>10 lines) or when there are 3+ copies. +- **Never touch ANTLR-generated code.** `Lql/lql-lsp-rust/crates/lql-parser/src/generated/` is regenerated from the `.g4` grammar; any dedup there will be erased on the next regen. diff --git a/.agents/skills/fix-bug/SKILL.md b/.agents/skills/fix-bug/SKILL.md new file mode 100644 index 00000000..3eb8279c --- /dev/null +++ b/.agents/skills/fix-bug/SKILL.md @@ -0,0 +1,67 @@ +--- +name: fix-bug +description: Fix a bug using test-driven development. Use when the user reports a bug, describes unexpected behavior, wants to fix a defect, or says something is broken. Enforces a strict test-first workflow where a failing test must be written and verified before any fix is attempted. +argument-hint: "[bug description]" +allowed-tools: Read, Grep, Glob, Edit, Write, Bash +--- + + +# Bug Fix Skill — Test-First Workflow + +You MUST follow this exact workflow. Do NOT skip steps. Do NOT fix the bug before writing a failing test. + +## Step 1: Understand the Bug + +- Read the bug description: $ARGUMENTS +- Investigate the codebase to understand the relevant code +- Identify the root cause (or narrow down candidates) +- Summarize your understanding of the bug to the user before proceeding + +## Step 2: Write a Failing Test + +- Write a test that **directly exercises the buggy behavior** +- The test must assert the **correct/expected** behavior — so it FAILS against the current broken code +- The test name should clearly describe the bug (e.g., `test_orange_color_not_applied_to_head`) +- Use the project's existing test framework and conventions + +## Step 3: Run the Test — Confirm It FAILS + +- Run ONLY the new test (not the full suite) +- **Verify the test FAILS** and that it fails **because of the bug**, not for some other reason (typo, import error, wrong selector, etc.) +- If the test passes: your test does not capture the bug. Go back to Step 2 +- If the test fails for the wrong reason: fix the test, not the code. Go back to Step 2 +- **Repeat until the test fails specifically because of the bug** + +## Step 4: Show Failure to User + +- Show the user the test code and the failure output +- Explicitly ask: "This test fails because of the bug. Can you confirm this captures the issue before I fix it?" +- **STOP and WAIT for user acknowledgment before proceeding** +- Do NOT continue to Step 5 until the user confirms + +## Step 5: Fix the Bug + +- Make the **minimum change** needed to fix the bug +- Do not refactor, clean up, or "improve" surrounding code +- Do not change the test + +## Step 6: Run the Test — Confirm It PASSES + +- Run the new test again +- **Verify it PASSES** +- If it fails: go back to Step 5 and adjust the fix +- **Repeat until the test passes** + +## Step 7: Run the Full Test Suite + +- Run ALL tests to make sure nothing else broke +- If other tests fail: fix the regression without breaking the new test +- Report the final result to the user + +## Rules + +- NEVER fix the bug before the failing test is written and confirmed +- NEVER skip asking the user to acknowledge the test failure +- NEVER modify the test to make it pass — modify the source code +- If you cannot write a test for the bug, explain why and ask the user how to proceed +- Keep the fix minimal — one bug, one fix, one test diff --git a/.agents/skills/spec-check/SKILL.md b/.agents/skills/spec-check/SKILL.md index b9e41376..a077d8b3 100644 --- a/.agents/skills/spec-check/SKILL.md +++ b/.agents/skills/spec-check/SKILL.md @@ -1,69 +1,288 @@ --- name: spec-check -description: Audits spec/plan documents against the codebase to ensure every spec section has implementing code and tests. Use when the user says "check specs", "audit specs", "spec coverage", or "validate specs". +description: Audit spec/plan documents against the codebase. Ensures every spec section has implementing code, tests, and matching logic. Use when the user says "check specs", "spec audit", or "verify specs". +argument-hint: "[optional spec ID or filename filter]" --- - + -# Spec Check +# spec-check -Audit spec and plan documents against the codebase. +> **Portable skill.** This skill adapts to the current repository. The agent MUST inspect the repo structure and use judgment to apply these instructions appropriately. -## Steps +Audit spec/plan documents against the codebase. Ensures every spec section has implementing code, tests, and that the code logic matches the spec. -### Step 1 — Validate spec ID structure +## Arguments -For every markdown file in `docs/specs/`: -1. Find all headings that contain a spec ID (pattern: `[GROUP-TOPIC-DETAIL]`) -2. Validate each ID: - - MUST be uppercase, hyphen-separated - - MUST NOT contain sequential numbers (e.g., `[SPEC-001]` is ILLEGAL) - - First word is the **group** — all sections sharing the same group MUST be adjacent -3. Check for duplicate IDs across all spec files -4. Report any violations +- `$ARGUMENTS` — optional spec name or ID to check (e.g., `LQL-PIPE` or `migration-spec`). If empty, check ALL specs. Spec IDs are descriptive slugs, NEVER numbered (see Step 1). -### Step 2 — Find spec documents +## Instructions -Scan `docs/specs/` and `docs/plans/` for all markdown files. For each file: -1. Extract all spec section IDs -2. Build a map: `spec ID → file path + heading` +Follow these steps exactly. Be strict and pedantic. Stop on the first failure. -### Step 3 — Check code references +--- + +### Step 1: Validate spec ID structure + +Before checking code/test references, verify that the specs themselves are well-formed. + +1. Find all spec documents (see locations in Step 2). +2. Extract every section ID using the regex `\[([A-Z][A-Z0-9]*(-[A-Z0-9]+)+)\]`. +3. **Flag invalid IDs:** + - Numbered IDs (`[SPEC-001]`, `[REQ-003]`, `[CI-004]`) — must be renamed to descriptive hierarchical slugs. + - Single-word IDs (`[TIMEOUT]`) — must have a group prefix. + - IDs with trailing numbers (`[FEAT-AUTH-01]`) — the number is meaningless, remove it. +4. **Check group clustering:** The first word of each ID is its group. All sections in the same group MUST appear together (adjacent) in the document. If they're scattered, flag it. +5. **Check for missing IDs:** Any heading that defines a requirement or behavior should have an ID. Flag headings in spec files that look like they define behavior but lack an ID. + +If any ID violations are found, report them all and **STOP**: +``` +SPEC ID VIOLATIONS: + +- docs/specs/migration-spec.md line 12: [SPEC-001] → rename to descriptive ID (e.g., [MIG-DDL-DIFF]) +- docs/specs/sync-spec.md line 30: [SYNC-PUSH] and [SYNC-PULL] are not adjacent (scattered group) +- docs/specs/lql-spec.md line 5: "## Pipeline operators" has no spec ID + +Fix spec IDs first, then re-run spec-check. +``` + +If all IDs are valid, proceed to Step 2. + +--- + +### Step 2: Find all spec/plan documents + +Search for markdown files that contain spec sections with IDs. Look in these locations: + +- `docs/specs/*.md` +- `docs/plans/*.md` +- `docs/**/*.md` + +Use Glob to find candidate files, then use Grep to confirm they contain spec IDs. + +**Spec ID patterns** — IDs appear in square brackets, typically at the start of a heading or section line. Match this regex pattern: + +``` +\[([A-Z][A-Z0-9]*(-[A-Z0-9]+)+)\] +``` + +Spec IDs are **hierarchical descriptive slugs, NEVER numbered.** The format is `[GROUP-TOPIC]` or `[GROUP-TOPIC-DETAIL]`. The first word is the **group** — all sections sharing the same group MUST appear together in the spec's table of contents. IDs are uppercase, hyphen-separated, unique across the repo, and MUST NOT contain sequential numbers. + +The hierarchy depth varies by repo: two words for simple repos (`[AUTH-LOGIN]`), three for most (`[AUTH-TOKEN-VERIFY]`), four for complex domains (`[AUTH-OAUTH-REFRESH-FLOW]`). The hierarchy mirrors the spec document's heading structure. + +Examples of valid spec IDs (note how groups cluster): +- `[LQL-PIPE]`, `[LQL-FILTER]`, `[LQL-JOIN]` — all in the LQL group +- `[SYNC-PUSH]`, `[SYNC-PULL]`, `[SYNC-CONFLICT]` — all in the SYNC group +- `[MIG-DDL]`, `[MIG-RLS]`, `[MIG-VECTOR]` — all in the MIG group +- `[CI-TIMEOUT]`, `[CI-LINT]`, `[CI-RELEASE]` — all in the CI group + +Examples of INVALID spec IDs: +- `[SPEC-001]` — numbered, meaningless +- `[FEAT-AUTH-01]` — trailing number +- `[REQ-003]` — sequential index, no group hierarchy +- `[CI-004]` — numbered, tells the reader nothing +- `[TIMEOUT]` — no group prefix, ungrouped + +For each file, extract every spec ID and its associated section title (the heading text after the ID) and the full section content (everything until the next heading of equal or higher level). + +--- + +### Step 3: Filter specs + +- If `$ARGUMENTS` is non-empty, filter the discovered specs: + - If it matches a spec ID exactly (e.g., `LQL-PIPE`), check only that spec. + - If it matches a partial name (e.g., `migration`), check all specs in files whose path contains that string. +- If `$ARGUMENTS` is empty, process ALL discovered specs. + +If filtering produces zero specs, report an error: +``` +ERROR: No specs found matching "$ARGUMENTS". Discovered spec files: [list them] +``` + +--- + +### Step 4: Check each spec section + +For EACH spec section that has an ID, perform checks A, B, and C below. **Stop on the first failure.** + +#### Check A: Code references the spec ID + +Search the entire codebase for the spec ID string, **excluding** these directories: +- `docs/` +- `node_modules/` +- `.git/` +- `target/` (Rust build artifacts) +- `bin/`, `obj/` (.NET build artifacts) +- `*.md` files (markdown is docs, not code) +- `Lql/lql-lsp-rust/crates/lql-parser/src/generated/` (ANTLR-generated) + +Use Grep with the literal spec ID (e.g., `[LQL-PIPE]`) to find references in code files. + +Code files should contain comments referencing the spec ID. The search must catch **all** comment styles across languages used in this repo: + +**C-style `//` comments** (C#, F#, Rust, TypeScript, JavaScript): +- `// Implements [LQL-PIPE]` +- `// [LQL-PIPE]` +- `// Tests [LQL-PIPE]` (also counts as a code reference) +- `/// Implements [LQL-PIPE]` (XML / rustdoc / TSDoc doc comments) + +**Hash `#` comments** (YAML, TOML, shell): +- `# Implements [LQL-PIPE]` +- `# [LQL-PIPE]` +- `# Tests [LQL-PIPE]` + +**HTML/XML comments** (HTML, CSS, SVG, XML, `.csproj`, `.fsproj`, `.runsettings`): +- `` +- `` + +**ML-style comments** (F#): +- `(* Implements [LQL-PIPE] *)` + +**The key rule:** any comment in any language containing the exact spec ID string (e.g., `[LQL-PIPE]`) counts as a valid code reference. The Grep search uses the literal spec ID string, so it naturally matches all comment styles. Do NOT restrict the search to specific comment prefixes — just search for the spec ID string itself. -For each spec ID found in Step 2: -1. Search the entire codebase (C#, Rust, TypeScript, F# files) for references to the ID -2. A reference is any comment containing the spec ID (e.g., `// Implements [AUTH-TOKEN-VERIFY]`) -3. Record which files reference each spec ID +**If NO code files reference the spec ID:** -### Step 4 — Check test references +``` +SPEC VIOLATION: [LQL-PIPE] "Section Title" has no implementing code. -For each spec ID: -1. Search test files for references to the ID -2. A test reference is a comment like `// Tests [AUTH-TOKEN-VERIFY]` in a test file +Every spec section must have at least one code file that references it via a comment +containing the spec ID (e.g., `// Implements [LQL-PIPE]`). -### Step 5 — Verify code logic matches spec +ACTION REQUIRED: Add a comment referencing [LQL-PIPE] in the file(s) that implement +this spec section, then re-run spec-check. +``` -For spec IDs that DO have code references: -1. Read the spec section -2. Read the implementing code -3. Check that the code actually does what the spec describes -4. Flag any discrepancies +**STOP HERE. Do not continue to other checks.** -### Step 6 — Report +#### Check B: Tests reference the spec ID -Output a table: +Search test files for the spec ID. Test projects/files in this repo: +- `**/*.Tests/**/*.cs`, `**/*.Tests/**/*.fs` +- `Lql/lql-lsp-rust/crates/*/tests/**/*.rs`, `Lql/lql-lsp-rust/crates/*/src/**/*.rs` (`#[test]` attribs) +- `Lql/LqlExtension/src/test/**/*.ts` +- `Tests.Shared/**/*.cs` + +Use Grep to search these locations for the literal spec ID string. + +Tests should contain the spec ID in comments, test names, or attributes: + +**C# (xUnit):** +- `// Tests [LQL-PIPE]` +- `[Fact] // Tests [LQL-PIPE]` +- `[Theory] // Tests [LQL-PIPE]` + +**F# (xUnit / Expecto):** +- `// Tests [LQL-PIPE]` +- `[] // Tests [LQL-PIPE]` + +**Rust:** +- `// Tests [LQL-PIPE]` +- `#[test] // Tests [LQL-PIPE]` + +**TypeScript (Mocha — VS Code test runner):** +- `// Tests [LQL-PIPE]` +- `describe('[LQL-PIPE] description', () => { ... })` +- `it('[LQL-PIPE] verifies behavior', () => { ... })` + +**The key rule:** same as Check A — search for the literal spec ID string in test files. Any occurrence of the exact spec ID in a test file counts. Do NOT restrict to specific patterns — just search for the spec ID string itself. + +**If NO test files reference the spec ID:** + +``` +SPEC VIOLATION: [LQL-PIPE] "Section Title" has no tests. + +Every spec section must have corresponding tests that reference the spec ID. + +ACTION REQUIRED: Add tests for [LQL-PIPE] with a comment or test name containing +the spec ID, then re-run spec-check. +``` + +**STOP HERE. Do not continue to other checks.** + +#### Check C: Code logic matches the spec + +This is the most critical check. You must: + +1. **Read the spec section content carefully.** Understand exactly what behavior, logic, ordering, conditions, and constraints the spec describes. + +2. **Read the implementing code.** Use the references found in Check A to locate the implementing files. Read the relevant functions/sections. + +3. **Compare spec vs. code.** Be SENSITIVE and PEDANTIC. Check for: + - **Ordering violations** — If the spec says A happens before B, the code must do A before B. + - **Missing conditions** — If the spec says "only when X", the code must have that condition. + - **Extra behavior** — If the code does something the spec doesn't mention, flag it only if it contradicts the spec. + - **Wrong logic** — If the spec says "greater than" but code uses "greater than or equal", that's a violation. + - **Missing steps** — If the spec describes 5 steps but code only implements 3, that's a violation. + - **Wrong defaults** — If the spec says "default to X" but code defaults to Y, that's a violation. + - **LQL platform-independence violations** — Per CLAUDE.md, LQL behavior MUST be identical across all transpiled SQL dialects. If a spec section is implemented in `Postgres` but missing from `SQLite` or `SqlServer`, that is a violation; the spec-check must report it and recommend filing a GitHub issue per CLAUDE.md's LQL rule. + +4. **If the code deviates from the spec**, report a detailed error: + +``` +SPEC VIOLATION: [LQL-PIPE] Code does not match spec. + +SPEC SAYS: +> "Pipe operator must left-associate; `a |> b |> c` parses as `(a |> b) |> c`" +> (from docs/specs/lql-spec.md, line 42) + +CODE DOES: +> Right-associative parse rule (Lql/lql-lsp-rust/crates/lql-parser/src/grammar.g4:120) + +DEVIATION: Grammar uses right-associativity instead of left-associativity. + +ACTION REQUIRED: Update grammar in Lql/lql-lsp-rust/crates/lql-parser/src/grammar.g4 +to left-associate the pipe operator, regenerate the ANTLR parser, and re-run spec-check. +``` + +**STOP HERE. Do not continue to other specs.** + +5. **If the code matches the spec**, this check passes. Move to the next spec. + +--- + +### Step 5: Report results + +#### On failure (any check fails): + +Output ONLY the first violation found. Use the exact error format shown above. Do not summarize other specs. Do not offer to fix the code. Just report the violation. + +End with: +``` +spec-check FAILED. Fix the violation above and re-run. +``` + +#### On success (all specs pass): + +Output a summary table: + +``` +spec-check PASSED. All specs verified. + +| Spec ID | Title | Code References | Test References | Logic Match | +|----------------|--------------------------|-----------------|-----------------|-------------| +| [LQL-PIPE] | Pipe operator | Lql/lql-lsp-rust/...grammar.g4 | Lql/Nimblesite.Lql.Tests/...PipeTests.cs | PASS | +| [SYNC-PUSH] | Push delta protocol | Sync/Nimblesite.Sync.Core/Push.cs | Sync/Nimblesite.Sync.Tests/PushTests.cs | PASS | +| ... | ... | ... | ... | ... | + +Checked N spec sections across M files. All have implementing code, tests, and matching logic. +``` + +--- -| Spec ID | Spec File | Code References | Test References | Status | -|---------|-----------|-----------------|-----------------|--------| +## Search strategy summary -Status values: -- **COVERED** — has both code and test references -- **UNTESTED** — has code references but no test references -- **UNIMPLEMENTED** — has no code references at all -- **ORPHANED** — spec ID found in code but not in any spec document +1. **Validate spec IDs:** Check all IDs are hierarchical, descriptive, grouped, and non-numbered +2. **Find spec files:** Glob for `docs/specs/**/*.md`, `docs/plans/**/*.md`, `docs/**/*.md` +3. **Extract spec IDs:** Grep for `\[[A-Z][A-Z0-9]*(-[A-Z0-9]+)+\]` in those files +4. **Find code refs:** Grep for the literal spec ID in all files, excluding `docs/`, `node_modules/`, `.git/`, `target/`, `bin/`, `obj/`, `*.md`, ANTLR-generated dirs +5. **Find test refs:** Grep for the literal spec ID in test directories and test file patterns +6. **Read and compare:** Read the spec section content and the implementing code, compare logic -## Rules +## Key principles -- Never modify spec documents — only report findings -- Never modify code — only report findings -- Every spec section MUST have at least one code reference and one test reference -- Orphaned references (code mentioning a spec ID that doesn't exist) are errors +- **Fail fast.** Stop on the first violation. One fix at a time. +- **Be pedantic.** If the spec says it, the code must do it. No "close enough". +- **Quote everything.** Always quote the spec text and the code in error messages so the developer sees exactly what's wrong. +- **Be actionable.** Every error must tell the developer what file to change and what to do. +- **Exclude docs from code search.** Markdown files are documentation, not implementation. Only search actual code files for spec references. +- **Exclude generated code.** ANTLR-generated parser sources under `Lql/lql-lsp-rust/crates/lql-parser/src/generated/` are not authored code; never expect spec IDs there. +- **No numbered IDs.** Spec IDs are hierarchical descriptive slugs (`[LQL-PIPE]`), NEVER sequential numbers (`[SPEC-001]`). The first word is the group — sections sharing a group must be adjacent in the TOC. If you encounter numbered or ungrouped IDs, flag them as a violation. diff --git a/.agents/skills/submit-pr/SKILL.md b/.agents/skills/submit-pr/SKILL.md index 30656c3e..e4c85b15 100644 --- a/.agents/skills/submit-pr/SKILL.md +++ b/.agents/skills/submit-pr/SKILL.md @@ -3,6 +3,7 @@ name: submit-pr description: Creates a pull request with a well-structured description after verifying CI passes. Use when the user asks to submit, create, or open a pull request. disable-model-invocation: true --- + # Submit PR @@ -10,27 +11,50 @@ Create a pull request for the current branch with a well-structured description. ## Steps -1. Run `make ci` — must pass completely before creating PR -2. **Generate the diff against main.** Run `git diff main...HEAD > /tmp/pr-diff.txt` to capture the full diff between the current branch and the head of main. This is the ONLY source of truth for what the PR contains. **Warning:** the diff can be very large. If the diff file exceeds context limits, process it in chunks (e.g., read sections with `head`/`tail` or split by file) rather than trying to load it all at once. -3. **Derive the PR title and description SOLELY from the diff.** Read the diff output and summarize what changed. Ignore commit messages, branch names, and any other metadata — only the actual code/content diff matters. -4. Write PR body using the template in `.github/pull_request_template.md` -5. Fill in (based on the diff analysis from step 3): - - TLDR: one sentence - - What Was Added: new files, features, deps - - What Was Changed/Deleted: modified behaviour - - How Tests Prove It Works: specific test names or output - - Spec/Doc Changes: if any - - Breaking Changes: yes/no + description -6. Use `gh pr create` with the filled template +### Step 1 — Run `make ci` (BLOCKING — DO NOT SKIP) + +You MUST invoke the Bash tool with the literal command `make ci` in THIS session before calling `gh pr create`. No exceptions. + +**Skip allowed ONLY if** there is a Bash tool call in the visible transcript of THIS session whose `command` was exactly `make ci` (or `make check` — which is `lint + test`, a strict subset that does not satisfy `make ci`'s `lint + test + build`) and which exited 0 AFTER the most recent code change. If you are unsure, run it. The cost of re-running is low; the cost of a broken PR is high. + +**Skip NOT allowed when:** +- You ran `dotnet test`, `make test`, `make lint`, partial test filters, or any subset — these are NOT `make ci`. Run `make ci` anyway. +- You ran `make ci` earlier but have edited code or skill files since — re-run. +- The user asked you to skip — refuse and ask them to invoke `make ci` themselves if they want to bypass. Tell them which step they're bypassing. +- You "reasoned" the change is small/safe/test-only — irrelevant. Run it. + +If `make ci` fails: STOP. Do not create the PR. Fix the failures or report them to the user and wait for direction. + +### Step 2 — Generate the diff against main + +Run `git diff main...HEAD > /tmp/pr-diff.txt` to capture the full diff between the current branch and the head of main. This is the ONLY source of truth for what the PR contains. **Warning:** the diff can be very large. If the diff file exceeds context limits, process it in chunks (e.g., read sections with `head`/`tail` or split by file) rather than trying to load it all at once. + +### Step 3 — Derive the PR title and description SOLELY from the diff + +Read the diff output and summarize what changed. Ignore commit messages, branch names, and any other metadata — only the actual code/content diff matters. + +### Step 4 — Write the PR body + +Use the template in `.github/PULL_REQUEST_TEMPLATE.md`. Fill in (based on the diff analysis from step 3): +- TLDR: one sentence +- What Was Added: new files, features, deps +- What Was Changed/Deleted: modified behaviour +- How Tests Prove It Works: specific test names or output +- Spec/Doc Changes: if any +- Breaking Changes: yes/no + description + +### Step 5 — Create the PR + +Use `gh pr create` with the filled template. ## Rules -- Never create a PR if `make ci` fails -- PR description must be specific and tight — no vague placeholders -- Link to the relevant GitHub issue if one exists +- **Never call `gh pr create` without first verifying `make ci` passed in this session.** Step 1 is the gate, not a suggestion. +- PR description must be specific and tight — no vague placeholders. +- Link to the relevant GitHub issue if one exists. ## Success criteria -- `make ci` passed +- `make ci` was invoked in this session and exited 0 after the most recent code change - PR created with `gh pr create` - PR URL returned to user diff --git a/.agents/skills/upgrade-packages/SKILL.md b/.agents/skills/upgrade-packages/SKILL.md index f6dd2ff0..82a654b4 100644 --- a/.agents/skills/upgrade-packages/SKILL.md +++ b/.agents/skills/upgrade-packages/SKILL.md @@ -1,57 +1,153 @@ --- name: upgrade-packages -description: Upgrades all dependencies to latest versions across C#, Rust, and TypeScript. Use when the user says "upgrade packages", "update dependencies", "bump versions", or "upgrade deps". -argument-hint: "[language: dotnet|rust|typescript|all]" +description: Upgrade all dependencies/packages to their latest versions for the detected language(s). Use when the user says "upgrade packages", "update dependencies", "bump versions", "update packages", or "upgrade deps". +argument-hint: "[--check-only] [--major] [package-name]" --- - + # Upgrade Packages -Upgrade all dependencies to their latest versions. +Upgrade all project dependencies to their latest compatible (or latest major, if `--major`) versions. +This repo has C#/.NET (NuGet), Rust (cargo), and TypeScript/Node (npm) — process all three when running without a package-name argument. -## Steps +## Arguments -### Step 1 — Detect packages to upgrade +- `--check-only` — List outdated packages without upgrading. Stop after Step 2. +- `--major` — Include major version bumps (breaking changes). Without this flag, stay within semver-compatible ranges. +- Any other argument is treated as a specific package name to upgrade (instead of all packages). -Based on `$ARGUMENTS` (default: all): +## Step 1 — Detect language and package manager -**C# (.NET):** -- Check `Directory.Build.props` for centrally managed package versions -- Check individual `.csproj` files for project-specific packages -- Run `dotnet list package --outdated` on `DataProvider.sln` +Inspect manifest files. For this repo, the relevant manifests are: -**Rust:** -- Check `Lql/lql-lsp-rust/Cargo.toml` workspace dependencies -- Run `cd Lql/lql-lsp-rust && cargo outdated` (install with `cargo install cargo-outdated` if needed) +| Manifest file | Language | Package manager | +|---|---|---| +| `Cargo.toml` (workspace at `Lql/lql-lsp-rust/Cargo.toml`) | Rust | cargo | +| `Lql/LqlExtension/package.json` | TypeScript | npm | +| `Website/package.json` | TypeScript / static site | npm | +| `Directory.Build.props` + `**/*.csproj`, `**/*.fsproj`, `DataProvider.sln` | C# / F# | NuGet (dotnet) | -**TypeScript:** -- Check `Lql/LqlExtension/package.json` -- Run `cd Lql/LqlExtension && npm outdated` +Process each in order: NuGet → cargo → npm. -### Step 2 — Upgrade +## Step 2 — List outdated packages -**C# (.NET):** -- Update version numbers in `Directory.Build.props` for central packages -- For project-specific packages: `dotnet add package ` -- Run `dotnet restore` +Run the listing command for each manager BEFORE upgrading. Show the user what will change. -**Rust:** -- Update versions in `Cargo.toml` -- Run `cargo update` +### C# / F# (NuGet) +```bash +dotnet list package --outdated +dotnet list package --outdated --include-transitive +``` +**Read the docs:** https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-list-package -**TypeScript:** -- Run `npm update` or manually update `package.json` for major versions -- Run `npm install` +### Rust +```bash +cd Lql/lql-lsp-rust && cargo outdated # install: cargo install cargo-outdated +cd Lql/lql-lsp-rust && cargo update --dry-run +``` +**Read the docs:** https://doc.rust-lang.org/cargo/commands/cargo-update.html -### Step 3 — Verify +### Node.js (npm) — LqlExtension +```bash +cd Lql/LqlExtension && npm outdated +``` +### Node.js (npm) — Website +```bash +cd Website && npm outdated +``` +**Read the docs:** https://docs.npmjs.com/cli/v10/commands/npm-update -1. Run `make ci` — must pass completely -2. If any tests fail, investigate whether the failure is from the upgrade -3. Report which packages were upgraded and from/to versions +If `--check-only` was passed, **stop here** and report the outdated lists. + +## Step 3 — Read the official upgrade docs + +**Before running any upgrade command, you MUST fetch and read the official documentation URL listed above for the detected package manager.** Use WebFetch to retrieve the page. This ensures you use the correct flags and understand the behavior. Do not guess at flags or options from memory. + +## Step 4 — Upgrade packages + +Run the upgrade. If a specific package name was given as an argument, upgrade only that package (in the manager that owns it). + +### C# / F# (NuGet) +There is NO single `dotnet upgrade-all` command. Choose one: + +Option A — manual per package (most controlled): +```bash +dotnet add package # upgrades to latest +dotnet add package --version # specific version +``` +For `Directory.Build.props`, edit the version numbers directly in the XML — that's where shared package versions live in this repo. + +Option B — global tool: +```bash +dotnet tool install --global dotnet-outdated-tool +dotnet outdated --upgrade +``` +**Read the docs:** https://github.com/dotnet-outdated/dotnet-outdated and https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-add-package + +### Rust +```bash +cd Lql/lql-lsp-rust && cargo update # semver-compatible updates +# --major flag: +cd Lql/lql-lsp-rust && cargo update --breaking # major version bumps (cargo 1.84+) +``` +The lints workspace (`[workspace.lints]`) and pinned ANTLR-runtime version may require special handling — check `Cargo.toml` before bumping the antlr crate; the parser is regenerated against a specific ANTLR runtime per CLAUDE.md. + +### Node.js (npm) +```bash +cd Lql/LqlExtension && npm update # semver-compatible +cd Website && npm update +# --major flag: +cd Lql/LqlExtension && npx npm-check-updates -u && npm install +cd Website && npx npm-check-updates -u && npm install +``` +**Read the docs:** https://docs.npmjs.com/cli/v10/commands/npm-update + +## Step 5 — Verify the upgrade + +After upgrading, run the project's build and test suite to confirm nothing broke: + +```bash +make ci +``` + +If `make ci` is unavailable for some reason, fall back to running each affected target manually: + +```bash +make _build_dotnet && make _test_dotnet +make _build_rust && make _test_rust +make _build_ts && make _test_ts +``` + +If tests fail: +1. Read the failure output carefully. +2. Check the changelog / migration guide for the upgraded packages (fetch the release notes URL if available). +3. Fix breaking changes in the code. +4. Re-run tests. +5. If stuck after 3 attempts on the same failure, report it to the user with the error details and the package that caused it. + +## Step 6 — Report + +Provide a summary: + +- Packages upgraded (old version → new version), grouped by NuGet / cargo / npm. +- Packages skipped (and why, e.g., major version bump without `--major`). +- Build/test result after upgrade. +- Any breaking changes that were fixed. +- Any packages that could not be upgraded (with error details). ## Rules -- Never downgrade a package -- If a major version upgrade breaks tests, report it and revert that specific upgrade -- Always run the full test suite after upgrading -- Update lock files (`Cargo.lock`, `package-lock.json`) as part of the upgrade +- **Always list outdated packages first** before upgrading anything. +- **Always read the official docs** for the package manager before running upgrade commands. +- **Always run `make ci` after upgrading** to catch breakage immediately. +- **Never remove packages** unless they were explicitly deprecated and replaced. +- **Never downgrade packages** unless rolling back a broken upgrade. +- **Never modify lockfiles manually** (`package-lock.json`, `Cargo.lock`, `packages.lock.json`) — let the package manager regenerate them. +- **Never bump the ANTLR runtime crate without regenerating** the parser from `*.g4` per CLAUDE.md ("Always upgrade to the latest version of ANTLR to make sure the parser is correct"). Mismatched runtime ⇄ generated code = silent breakage. +- **Commit nothing** — leave changes in the working tree for the user to review. + +## Success criteria + +- All outdated packages upgraded to latest compatible (or latest major if `--major`) +- `make ci` passes +- User has a clear summary of what changed diff --git a/.agents/skills/website-audit/SKILL.md b/.agents/skills/website-audit/SKILL.md index 9c1b57fc..6f786a0e 100644 --- a/.agents/skills/website-audit/SKILL.md +++ b/.agents/skills/website-audit/SKILL.md @@ -2,11 +2,14 @@ name: website-audit description: Audits a website for SEO, AI search performance, structured data, mobile usability, broken links, and social media cards. Fixes issues found. Use when the user mentions "audit website", "SEO", "fix search ranking", "AI search", "structured data", "social media cards", or "website performance". --- + # Website Audit Performs a comprehensive website audit and fixes issues affecting search visibility and AI discoverability. +This repo's website lives in `Website/` and is built with **Eleventy** (`eleventy.config.js`) plus DocFX for API docs. Templates and content sources live under `Website/src/`; generated output is `Website/_site/` (do NOT edit `_site/` directly). + Copy this checklist and track your progress: ``` @@ -25,9 +28,10 @@ Audit Progress: - [ ] Step 12: Report findings ``` -- Check the outputted HTML/CSS/JavaScript AFTER the website is generated by the static content generator. - Don't just check the static content before the website is generated. -- Fix issues at the core where the static content templates are stored - not in the outputted HTML (e.g. _site) -- Never manually edit the generated website content directly +- Check the outputted HTML/CSS/JavaScript AFTER the website is generated by Eleventy. Don't just check the static content before the website is generated. +- Fix issues at the core where the static content templates are stored (`Website/src/`) — not in the outputted HTML (`Website/_site/`). +- Never manually edit `Website/_site/`. +- ENSURE THE FOOTER HAS A copyright link to nimblesite.co ## Step 1 — Read guidelines @@ -37,10 +41,9 @@ Fetch and read each of these before auditing. These are the authoritative refere - [Top ways to ensure content performs well in Google's AI experiences](https://developers.google.com/search/blog/2025/05/succeeding-in-ai-search) - [SEO Starter Guide](https://developers.google.com/search/docs/fundamentals/seo-starter-guide) -Take the business plan into account: -[text](../../../business_plan/business_plan.md) +If the repo has a business plan doc, take it into account. The DataProvider/LQL specs under `docs/specs/` describe the product's value propositions — pull factual claims from there. -Identify the website source files in the repo. Determine the framework (static site generator, Next.js, Hugo, etc.) so you know where to find templates, metadata, and content. +Identify the website source files in `Website/src/`. The framework is Eleventy (templates as `.njk`/`.md` under `Website/src/_includes/`, data files under `Website/src/_data/`). ## Step 2 — Audit AI search readiness @@ -48,30 +51,30 @@ Apply the guidance from the AI search article. Check: 1. **Content quality** — Is content original, expert-level, and comprehensive? Flag thin or duplicated pages. 2. **Clear structure** — Do pages use descriptive headings, lists, and concise answers to likely questions? -3. **Entity clarity** — Are key terms, products, and concepts defined clearly so AI can extract them? +3. **Entity clarity** — Are key terms (LQL, DataProvider, source generator, transpiler, sync, RBAC, WebAuthn) defined clearly so AI can extract them? 4. **Freshness signals** — Are dates, update timestamps, and authorship present? -Fix issues directly in the source files. For each fix, note what changed and why. +Fix issues directly in the source files (`Website/src/`). For each fix, note what changed and why. ## Step 3 — Audit SEO and keywords -1. Search [Google Trends](https://trends.google.com/home) for trending keywords related to the website's content. +1. Search [Google Trends](https://trends.google.com/home) for trending keywords related to "C# source generator", "SQL safe queries", "offline sync", "ORM alternative", "type-safe SQL". 2. Review each page's ``, `<meta name="description">`, and `<h1>` tags. 3. Check for keyword opportunities — can trending terms be naturally inserted into headings, descriptions, or body content? 4. Verify each page has a unique, descriptive title (50-60 chars) and meta description (150-160 chars). 5. Check image `alt` attributes describe the image content and include relevant keywords where natural. -Apply the [SEO Starter Guide](https://developers.google.com/search/docs/fundamentals/seo-starter-guide) principles. Fix issues directly. +Apply the [SEO Starter Guide](https://developers.google.com/search/docs/fundamentals/seo-starter-guide) principles. Fix issues directly in `Website/src/`. ## Step 4 — Audit crawling and indexing Reference: [Overview of crawling and indexing topics](https://developers.google.com/search/docs/crawling-indexing) -1. **robots.txt** — Locate and review it. Verify it doesn't block important pages. Reference: [robots.txt spec](https://developers.google.com/search/docs/crawling-indexing/robots-txt) -2. **Sitemap** — Locate the sitemap (or sitemap index). Verify all important pages are listed and no dead URLs are included. Reference: [Sitemap guidelines](https://developers.google.com/search/docs/crawling-indexing/sitemaps/large-sitemaps) -3. **Meta robots tags** — Check for unintended `noindex` or `nofollow` directives on pages that should be indexed. +1. **robots.txt** — `Website/src/robots.txt` (passed through to `_site/robots.txt`). Verify it doesn't block important pages. Reference: [robots.txt spec](https://developers.google.com/search/docs/crawling-indexing/robots-txt). +2. **Sitemap** — generated by `Website/src/sitemap.njk`. Verify all important pages are listed and no dead URLs are included. Reference: [Sitemap guidelines](https://developers.google.com/search/docs/crawling-indexing/sitemaps/large-sitemaps). +3. **Meta robots tags** — Check Eleventy templates for unintended `noindex` or `nofollow` directives on pages that should be indexed. -Note: robots.txt and sitemaps are often auto-generated. If so, check the generator config rather than the output file. +Note: edit the generators (`robots.txt`, `sitemap.njk`), not the rendered output. ## Step 5 — Audit broken links and canonicalization @@ -95,13 +98,14 @@ Reference: [Mobile-first indexing best practices](https://developers.google.com/ Reference: [Structured data guidelines](https://developers.google.com/search/docs/appearance/structured-data/sd-policies) -1. Check for existing JSON-LD `<script type="application/ld+json">` blocks. +1. Check for existing JSON-LD `<script type="application/ld+json">` blocks in `Website/src/_includes/`. 2. Verify the structured data matches the page content (no misleading markup). 3. Add missing structured data where appropriate: - **Organization/Person** on the homepage - **Article/BlogPosting** on blog posts (with author, datePublished, dateModified) - **BreadcrumbList** for navigation - **FAQ** for pages with question/answer content + - **SoftwareSourceCode** for the docs API pages 4. Validate JSON-LD syntax is correct. ## Step 8 — Audit social media cards @@ -128,16 +132,22 @@ Search for the authoritative URL and add a link to the URL. If it is not availab ## Step 10 — Audit Design Compliance -Read the design system docs and view the design screens in the designsystem folder. +Read the design system specs (`docs/specs/dataprovider-design-system.md`, `docs/specs/lql-design-system.md`) and view the design screens in `Website/Designs/` (`HomePage.png`, `APIDoc.png`, `BlogsPage.png`). ## Step 11 — Test with Playwright -Build and run the website locally using `make website-run` (or the project's equivalent dev server command). +The repo already has Playwright configured at `Website/playwright.config.ts`. Build and run the website locally: + +```bash +cd Website && npm run build && npm run start +``` + +Then run the configured tests, or drive the site manually with the playwright MCP tools. **Desktop tests (1280x720):** 1. Navigate to the homepage — take a screenshot. -2. Navigate to each major section — verify pages load without errors. +2. Navigate to each major section (docs, blog, API docs) — verify pages load without errors. 3. Check the browser console for JavaScript errors. 4. Verify all navigation links work. @@ -176,4 +186,4 @@ Summarize the audit results: - **One step at a time** — complete each step before moving to the next. - **Preserve existing content** — improve structure and metadata without rewriting the author's voice. - **No keyword stuffing** — keywords must read naturally in context. -- **Respect the framework** — edit templates/configs, not generated output files. +- **Respect the framework** — edit `Website/src/` templates and `eleventy.config.js`, never edit `Website/_site/` generated output. diff --git a/.claude/skills/ci-prep/SKILL.md b/.claude/skills/ci-prep/SKILL.md index 8524d5e9..7a528735 100644 --- a/.claude/skills/ci-prep/SKILL.md +++ b/.claude/skills/ci-prep/SKILL.md @@ -1,6 +1,7 @@ --- name: ci-prep description: Prepares the current branch for CI by running the exact same steps locally and fixing issues. If CI is already failing, fetches the GH Actions logs first to diagnose. Use before pushing, when CI is red, or when the user says "fix ci". +argument-hint: "[--failing] [optional job name to focus on]" --- @../../../.agents/skills/ci-prep/SKILL.md diff --git a/.claude/skills/fix-bug/SKILL.md b/.claude/skills/fix-bug/SKILL.md new file mode 100644 index 00000000..9e879b16 --- /dev/null +++ b/.claude/skills/fix-bug/SKILL.md @@ -0,0 +1,8 @@ +--- +name: fix-bug +description: Fix a bug using test-driven development. Use when the user reports a bug, describes unexpected behavior, wants to fix a defect, or says something is broken. Enforces a strict test-first workflow where a failing test must be written and verified before any fix is attempted. +argument-hint: "[bug description]" +allowed-tools: Read, Grep, Glob, Edit, Write, Bash +--- + +@../../../.agents/skills/fix-bug/SKILL.md diff --git a/.claude/skills/spec-check/SKILL.md b/.claude/skills/spec-check/SKILL.md index 944484ab..aa28aba2 100644 --- a/.claude/skills/spec-check/SKILL.md +++ b/.claude/skills/spec-check/SKILL.md @@ -1,6 +1,7 @@ --- name: spec-check -description: Audits spec/plan documents against the codebase to ensure every spec section has implementing code and tests. Use when the user says "check specs", "audit specs", "spec coverage", or "validate specs". +description: Audit spec/plan documents against the codebase. Ensures every spec section has implementing code, tests, and matching logic. Use when the user says "check specs", "spec audit", or "verify specs". +argument-hint: "[optional spec ID or filename filter]" --- @../../../.agents/skills/spec-check/SKILL.md diff --git a/.claude/skills/submit-pr/SKILL.md b/.claude/skills/submit-pr/SKILL.md index 786e837a..150884ac 100644 --- a/.claude/skills/submit-pr/SKILL.md +++ b/.claude/skills/submit-pr/SKILL.md @@ -1,6 +1,7 @@ --- name: submit-pr description: Creates a pull request with a well-structured description after verifying CI passes. Use when the user asks to submit, create, or open a pull request. +disable-model-invocation: true --- @../../../.agents/skills/submit-pr/SKILL.md diff --git a/.claude/skills/upgrade-packages/SKILL.md b/.claude/skills/upgrade-packages/SKILL.md index efbdc35b..6ff284eb 100644 --- a/.claude/skills/upgrade-packages/SKILL.md +++ b/.claude/skills/upgrade-packages/SKILL.md @@ -1,6 +1,7 @@ --- name: upgrade-packages -description: Upgrades all dependencies to latest versions across C#, Rust, and TypeScript. Use when the user says "upgrade packages", "update dependencies", "bump versions", or "upgrade deps". +description: Upgrade all dependencies/packages to their latest versions for the detected language(s). Use when the user says "upgrade packages", "update dependencies", "bump versions", "update packages", or "upgrade deps". +argument-hint: "[--check-only] [--major] [package-name]" --- @../../../.agents/skills/upgrade-packages/SKILL.md diff --git a/.clinerules/00-read-instructions.md b/.clinerules/00-read-instructions.md index d4e3f998..48aaceb3 100644 --- a/.clinerules/00-read-instructions.md +++ b/.clinerules/00-read-instructions.md @@ -1,9 +1,2 @@ -<!-- agent-pmo:d75d5c8 --> -# Single Source of Truth - +<!-- agent-pmo:74cf183 --> @CLAUDE.md - -Read the file above in full before writing any code. All project rules, -coding standards, hard constraints, build commands, and architecture -notes live there. Do not add rules to this file — keep everything in -`CLAUDE.md` so there is exactly one set of instructions to maintain. diff --git a/.cursorrules b/.cursorrules index addcc98e..61769c98 100644 --- a/.cursorrules +++ b/.cursorrules @@ -1,5 +1 @@ @CLAUDE.md - -All project rules, coding standards, hard constraints, build commands, -and architecture notes live in `CLAUDE.md` at the repository root. -Read that file in full before writing any code. Do not add rules here. diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index addcc98e..48aaceb3 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,5 +1,2 @@ +<!-- agent-pmo:74cf183 --> @CLAUDE.md - -All project rules, coding standards, hard constraints, build commands, -and architecture notes live in `CLAUDE.md` at the repository root. -Read that file in full before writing any code. Do not add rules here. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 54549aff..822f8f9b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -65,7 +65,7 @@ jobs: run: cd Lql/LqlExtension && npm install --no-audit --no-fund - name: Format check - run: make fmt-check + run: make fmt CHECK=1 - name: Lint run: make lint diff --git a/.vscode/extensions.json b/.vscode/extensions.json new file mode 100644 index 00000000..5ededd52 --- /dev/null +++ b/.vscode/extensions.json @@ -0,0 +1,8 @@ +{ + "_agent_pmo": "74cf183", + "recommendations": [ + "nimblesite.commandtree", + "nimblesite.too-many-cooks", + "nimblesite.typeDiagram" + ] +} diff --git a/.vscode/settings.json b/.vscode/settings.json index 24a67be6..26577f38 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -9,5 +9,11 @@ "lql.ai.provider": "ollama", "lql.ai.model": "qwen2.5-coder:1.5b", "lql.ai.endpoint": "http://localhost:11434/api/generate", - "lql.languageServer.path": "/Users/christianfindlay/Documents/Code/DataProvider/Lql/lql-lsp-rust/target/release/lql-lsp" + "lql.languageServer.path": "/Users/christianfindlay/Documents/Code/DataProvider/Lql/lql-lsp-rust/target/release/lql-lsp", + "workbench.colorCustomizations": { + "titleBar.activeBackground": "#0B132B", + "titleBar.activeForeground": "#F4F7FB", + "titleBar.inactiveBackground": "#070D1F", + "titleBar.inactiveForeground": "#F4F7FBcc" + } } \ No newline at end of file diff --git a/.windsurfrules b/.windsurfrules index addcc98e..61769c98 100644 --- a/.windsurfrules +++ b/.windsurfrules @@ -1,5 +1 @@ @CLAUDE.md - -All project rules, coding standards, hard constraints, build commands, -and architecture notes live in `CLAUDE.md` at the repository root. -Read that file in full before writing any code. Do not add rules here. diff --git a/Agents.md b/AGENTS.md similarity index 94% rename from Agents.md rename to AGENTS.md index 29a39b3b..a8dd7ab9 100644 --- a/Agents.md +++ b/AGENTS.md @@ -1,4 +1,4 @@ -<!-- agent-pmo:d75d5c8 --> +<!-- agent-pmo:74cf183 --> # Agent Instructions @CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md index 49382a7f..73dad95e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,7 +7,7 @@ ⚠️ NEVER KILL ANY VSCODE PROCESS ⚠️ -<!-- agent-pmo:d75d5c8 --> +<!-- agent-pmo:74cf183 --> ## Project Overview @@ -165,20 +165,26 @@ Always include these in `Directory.Build.props`: ## Build Commands (exact — cross-platform via GNU Make) +The 7 standard portfolio targets: + ```bash make build # compile everything -make test # run tests with coverage -make lint # run all linters -make fmt # format all code -make fmt-check # check formatting (CI uses this) +make test # fail-fast tests + coverage + threshold (the only test entry point) +make lint # run all linters/analyzers (no formatting) +make fmt # format all code in-place (CHECK=1 for read-only verify, used in CI) make clean # remove build artifacts -make check # lint + test (pre-commit) make ci # lint + test + build (full CI simulation) -make coverage # generate and open coverage report -make coverage-check # assert coverage thresholds make setup # post-create dev environment setup ``` +Repo-specific targets (this repo only): + +```bash +make check # lint + test (pre-commit shortcut) +make coverage # generate and open HTML coverage report +make vsix # build LSP + package VS Code extension +``` + ## Architecture | Component | Path | Purpose | diff --git a/DataProvider/Nimblesite.DataProvider.Core/README.md b/DataProvider/Nimblesite.DataProvider.Core/README.md index 0b001904..908f1e0b 100644 --- a/DataProvider/Nimblesite.DataProvider.Core/README.md +++ b/DataProvider/Nimblesite.DataProvider.Core/README.md @@ -5,4 +5,4 @@ The core source generator library. Parses SQL files and generates strongly-typed ## Documentation - Parent README: [DataProvider/README.md](../README.md) -- Migration CLI spec: [docs/specs/migration-cli-spec.md](../../docs/specs/migration-cli-spec.md) +- Migration spec: [docs/specs/migration-spec.md](../../docs/specs/migration-spec.md#74-dataprovidermigrate-cli-mig-cli) diff --git a/DataProvider/README.md b/DataProvider/README.md index c253b3c9..fbb280ed 100644 --- a/DataProvider/README.md +++ b/DataProvider/README.md @@ -152,4 +152,4 @@ SqliteCodeGenerator.GenerateCodeWithMetadata(config: config, /* … */); - [LQL](../Lql/README.md) — cross-database query language that transpiles to SQL - [Migrations](../Migration/README.md) — YAML schema definitions consumed by `DataProviderMigrate` -- Migration CLI spec: [docs/specs/migration-cli-spec.md](../docs/specs/migration-cli-spec.md) +- Migration spec: [docs/specs/migration-spec.md](../docs/specs/migration-spec.md#74-dataprovidermigrate-cli-mig-cli) diff --git a/Lql/LqlExtension/src/test/suite/vscode-e2e.test.ts b/Lql/LqlExtension/src/test/suite/vscode-e2e.test.ts index 5ec2665b..3cde890d 100644 --- a/Lql/LqlExtension/src/test/suite/vscode-e2e.test.ts +++ b/Lql/LqlExtension/src/test/suite/vscode-e2e.test.ts @@ -529,4 +529,125 @@ suite("VS Code Extension E2E Tests", function () { ); } }); + + // ═══════════════════════════════════════════════════════════════ + // REGISTERED COMMAND BODIES + // + // The three lql.* commands have small but real bodies in extension.ts + // (lql.formatDocument, lql.validateDocument, lql.showCompiledSql) that + // every platform's CI must execute, otherwise coverage is platform- + // dependent. These tests invoke each command directly via + // executeCommand and exercise both the active-LQL-editor branch and + // the no-editor / non-LQL-editor branch. + // ═══════════════════════════════════════════════════════════════ + + test("lql.formatDocument command body runs with active LQL editor", async function () { + const doc = await vscode.workspace.openTextDocument({ + content: "users |> select(users.id)", + language: "lql", + }); + await vscode.window.showTextDocument(doc); + await new Promise((r) => setTimeout(r, 500)); + + await vscode.commands.executeCommand("lql.formatDocument"); + }); + + test("lql.formatDocument command body runs with no active editor", async function () { + await vscode.commands.executeCommand("workbench.action.closeAllEditors"); + await new Promise((r) => setTimeout(r, 200)); + + await vscode.commands.executeCommand("lql.formatDocument"); + }); + + test("lql.validateDocument command body runs with active LQL editor", async function () { + const doc = await vscode.workspace.openTextDocument({ + content: "users |> select(users.id)", + language: "lql", + }); + await vscode.window.showTextDocument(doc); + await new Promise((r) => setTimeout(r, 500)); + + await vscode.commands.executeCommand("lql.validateDocument"); + }); + + test("lql.validateDocument command body runs with no active editor", async function () { + await vscode.commands.executeCommand("workbench.action.closeAllEditors"); + await new Promise((r) => setTimeout(r, 200)); + + await vscode.commands.executeCommand("lql.validateDocument"); + }); + + test("lql.showCompiledSql command body runs", async function () { + await vscode.commands.executeCommand("lql.showCompiledSql"); + }); + + // ═══════════════════════════════════════════════════════════════ + // AI PROVIDER INIT-OPTIONS BRANCH + // + // The "AI provider configured" branch in activate() (extension.ts ~280) + // only runs when lql.ai.* settings are populated at activation time. + // We can't re-activate the extension here, but we can verify the + // configuration surface exists so a future regression that drops the + // settings is caught. + // ═══════════════════════════════════════════════════════════════ + + test("AI provider configuration keys are declared in package.json", function () { + const ext = vscode.extensions.getExtension("lql-team.lql-language-support"); + assert.ok(ext !== undefined, "Extension must be installed"); + const pkg = ext.packageJSON as { + contributes?: { configuration?: { properties?: Record<string, unknown> } }; + }; + const props = pkg.contributes?.configuration?.properties ?? {}; + assert.ok("lql.ai.provider" in props, "lql.ai.provider must be declared"); + assert.ok("lql.ai.endpoint" in props, "lql.ai.endpoint must be declared"); + assert.ok("lql.ai.model" in props, "lql.ai.model must be declared"); + }); + + // ═══════════════════════════════════════════════════════════════ + // DEACTIVATE + // + // The exported deactivate() function in extension.ts must run on + // every CI to keep coverage platform-agnostic. We call it directly + // through the compiled module exports. + // ═══════════════════════════════════════════════════════════════ + + test("deactivate stops the LSP client cleanly", async function () { + interface ExtensionModule { + deactivate(): Thenable<void> | undefined; + } + + const ext = vscode.extensions.getExtension("lql-team.lql-language-support"); + assert.ok(ext !== undefined, "Extension must be installed"); + if (!ext.isActive) { + await ext.activate(); + } + + const mod = ext.exports as ExtensionModule | undefined; + if (mod === undefined || typeof mod.deactivate !== "function") { + // Extension's package.json doesn't expose deactivate via exports — + // require the compiled module path directly. The path mirrors + // package.json "main": "./out/extension.js". + const extensionsRoot = ext.extensionPath; + + const compiledPath = path.join(extensionsRoot, "out", "extension.js"); + + const dynamicRequire = eval("require") as (id: string) => unknown; + const directMod = dynamicRequire(compiledPath) as ExtensionModule; + assert.strictEqual( + typeof directMod.deactivate, + "function", + "extension.js must export deactivate", + ); + const result = directMod.deactivate(); + if (result !== undefined) { + await result; + } + return; + } + + const result = mod.deactivate(); + if (result !== undefined) { + await result; + } + }); }); diff --git a/Lql/lql-lsp-rust/Cargo.lock b/Lql/lql-lsp-rust/Cargo.lock index 67878b2d..b19b4908 100644 --- a/Lql/lql-lsp-rust/Cargo.lock +++ b/Lql/lql-lsp-rust/Cargo.lock @@ -14,6 +14,24 @@ dependencies = [ "zerocopy", ] +[[package]] +name = "aho-corasick" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" +dependencies = [ + "memchr", +] + +[[package]] +name = "android_system_properties" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" +dependencies = [ + "libc", +] + [[package]] name = "antlr-rust" version = "0.3.0-beta" @@ -31,6 +49,50 @@ dependencies = [ "uuid", ] +[[package]] +name = "anyhow" +version = "1.0.102" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" + +[[package]] +name = "astral-tokio-tar" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ce73b17c62717c4b6a9af10b43e87c578b0cac27e00666d48304d3b7d2c0693" +dependencies = [ + "filetime", + "futures-core", + "libc", + "portable-atomic", + "rustc-hash", + "tokio", + "tokio-stream", + "xattr", +] + +[[package]] +name = "async-stream" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b5a71a6f37880a80d1d7f19efd781e4b5de42c88f0722cc13bcb6cc2cfe8476" +dependencies = [ + "async-stream-impl", + "futures-core", + "pin-project-lite", +] + +[[package]] +name = "async-stream-impl" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7c24de15d275a1ecfd47a380fb4d5ec9bfe0933f309ed5e705b775596a3574d" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "async-trait" version = "0.1.89" @@ -59,6 +121,55 @@ dependencies = [ "syn", ] +[[package]] +name = "autocfg" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" + +[[package]] +name = "axum" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90" +dependencies = [ + "axum-core", + "bytes", + "futures-util", + "http", + "http-body", + "http-body-util", + "itoa", + "matchit", + "memchr", + "mime", + "percent-encoding", + "pin-project-lite", + "serde_core", + "sync_wrapper", + "tower 0.5.3", + "tower-layer", + "tower-service", +] + +[[package]] +name = "axum-core" +version = "0.5.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08c78f31d7b1291f7ee735c1c6780ccde7785daae9a9206026862dab7d8792d1" +dependencies = [ + "bytes", + "futures-core", + "http", + "http-body", + "http-body-util", + "mime", + "pin-project-lite", + "sync_wrapper", + "tower-layer", + "tower-service", +] + [[package]] name = "base64" version = "0.22.1" @@ -107,6 +218,80 @@ dependencies = [ "generic-array", ] +[[package]] +name = "bollard" +version = "0.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ee04c4c84f1f811b017f2fbb7dd8815c976e7ca98593de9c1e2afad0f636bff4" +dependencies = [ + "async-stream", + "base64", + "bitflags 2.11.0", + "bollard-buildkit-proto", + "bollard-stubs", + "bytes", + "futures-core", + "futures-util", + "hex", + "home", + "http", + "http-body-util", + "hyper", + "hyper-named-pipe", + "hyper-rustls", + "hyper-util", + "hyperlocal", + "log", + "num", + "pin-project-lite", + "rand 0.9.2", + "rustls", + "rustls-native-certs", + "rustls-pki-types", + "serde", + "serde_derive", + "serde_json", + "serde_urlencoded", + "thiserror", + "time", + "tokio", + "tokio-stream", + "tokio-util", + "tonic", + "tower-service", + "url", + "winapi", +] + +[[package]] +name = "bollard-buildkit-proto" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85a885520bf6249ab931a764ffdb87b0ceef48e6e7d807cfdb21b751e086e1ad" +dependencies = [ + "prost", + "prost-types", + "tonic", + "tonic-prost", + "ureq", +] + +[[package]] +name = "bollard-stubs" +version = "1.52.1-rc.29.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f0a8ca8799131c1837d1282c3f81f31e76ceb0ce426e04a7fe1ccee3287c066" +dependencies = [ + "base64", + "bollard-buildkit-proto", + "bytes", + "prost", + "serde", + "serde_json", + "serde_repr", + "time", +] + [[package]] name = "bumpalo" version = "3.20.2" @@ -141,6 +326,29 @@ version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" +[[package]] +name = "chacha20" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "rand_core 0.10.1", +] + +[[package]] +name = "chrono" +version = "0.4.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c673075a2e0e5f4a1dde27ce9dee1ea4558c7ffe648f576438a20ca1d2acc4b0" +dependencies = [ + "iana-time-zone", + "num-traits", + "serde", + "windows-link", +] + [[package]] name = "core-foundation" version = "0.9.4" @@ -176,6 +384,15 @@ dependencies = [ "libc", ] +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + [[package]] name = "crypto-common" version = "0.1.7" @@ -186,6 +403,40 @@ dependencies = [ "typenum", ] +[[package]] +name = "darling" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "25ae13da2f202d56bd7f91c25fba009e7717a1e4a1cc98a76d844b65ae912e9d" +dependencies = [ + "darling_core", + "darling_macro", +] + +[[package]] +name = "darling_core" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9865a50f7c335f53564bb694ef660825eb8610e0a53d3e11bf1b0d3df31e03b0" +dependencies = [ + "ident_case", + "proc-macro2", + "quote", + "strsim", + "syn", +] + +[[package]] +name = "darling_macro" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac3984ec7bd6cfa798e62b4a642426a5be0e68f9401cfc2a01e3fa9ea2fcdb8d" +dependencies = [ + "darling_core", + "quote", + "syn", +] + [[package]] name = "dashmap" version = "5.5.3" @@ -199,6 +450,16 @@ dependencies = [ "parking_lot_core 0.9.12", ] +[[package]] +name = "deranged" +version = "0.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" +dependencies = [ + "powerfmt", + "serde_core", +] + [[package]] name = "digest" version = "0.10.7" @@ -221,6 +482,29 @@ dependencies = [ "syn", ] +[[package]] +name = "docker_credential" +version = "1.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4564c274ebf369f501de192b02a0b81a5c4bda375abfe526aa70fc702fa6fa0" +dependencies = [ + "base64", + "serde", + "serde_json", +] + +[[package]] +name = "dyn-clone" +version = "1.0.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" + +[[package]] +name = "either" +version = "1.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719" + [[package]] name = "encoding_rs" version = "0.8.35" @@ -246,6 +530,16 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "etcetera" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de48cc4d1c1d97a20fd819def54b890cadde72ed3ad0c614822a0a433361be96" +dependencies = [ + "cfg-if", + "windows-sys 0.61.2", +] + [[package]] name = "fallible-iterator" version = "0.2.0" @@ -270,6 +564,27 @@ version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be" +[[package]] +name = "ferroid" +version = "2.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ee93edf3c501f0035bbeffeccfed0b79e14c311f12195ec0e661e114a0f60da4" +dependencies = [ + "portable-atomic", + "rand 0.10.1", + "web-time", +] + +[[package]] +name = "filetime" +version = "0.2.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d5b2eef6fafbf69f877e55509ce5b11a760690ac9700a2921be067aa6afaef6" +dependencies = [ + "cfg-if", + "libc", +] + [[package]] name = "find-msvc-tools" version = "0.1.9" @@ -282,6 +597,12 @@ version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" +[[package]] +name = "foldhash" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" + [[package]] name = "foreign-types" version = "0.3.2" @@ -314,6 +635,7 @@ checksum = "8b147ee9d1f6d097cef9ce628cd2ee62288d963e16fb287bd9286455b241382d" dependencies = [ "futures-channel", "futures-core", + "futures-executor", "futures-io", "futures-sink", "futures-task", @@ -336,6 +658,17 @@ version = "0.3.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" +[[package]] +name = "futures-executor" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baf29c38818342a3b26b5b923639e7b1f4a61fc5e76102d4b1981c6dc7a7579d" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + [[package]] name = "futures-io" version = "0.3.32" @@ -411,8 +744,22 @@ checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" dependencies = [ "cfg-if", "libc", - "r-efi", + "r-efi 5.3.0", + "wasip2", +] + +[[package]] +name = "getrandom" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555" +dependencies = [ + "cfg-if", + "libc", + "r-efi 6.0.0", + "rand_core 0.10.1", "wasip2", + "wasip3", ] [[package]] @@ -427,13 +774,19 @@ dependencies = [ "futures-core", "futures-sink", "http", - "indexmap", + "indexmap 2.13.0", "slab", "tokio", "tokio-util", "tracing", ] +[[package]] +name = "hashbrown" +version = "0.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888" + [[package]] name = "hashbrown" version = "0.14.5" @@ -443,6 +796,15 @@ dependencies = [ "ahash", ] +[[package]] +name = "hashbrown" +version = "0.15.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" +dependencies = [ + "foldhash", +] + [[package]] name = "hashbrown" version = "0.16.1" @@ -458,6 +820,18 @@ dependencies = [ "hashbrown 0.14.5", ] +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "hex" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" + [[package]] name = "hmac" version = "0.12.1" @@ -467,6 +841,15 @@ dependencies = [ "digest", ] +[[package]] +name = "home" +version = "0.5.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc627f471c528ff0c4a49e1d5e60450c8f6461dd6d10ba9dcd3a61d3dff7728d" +dependencies = [ + "windows-sys 0.61.2", +] + [[package]] name = "http" version = "1.4.0" @@ -506,6 +889,12 @@ version = "1.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" +[[package]] +name = "httpdate" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" + [[package]] name = "hyper" version = "1.8.1" @@ -520,6 +909,7 @@ dependencies = [ "http", "http-body", "httparse", + "httpdate", "itoa", "pin-project-lite", "pin-utils", @@ -528,6 +918,21 @@ dependencies = [ "want", ] +[[package]] +name = "hyper-named-pipe" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73b7d8abf35697b81a825e386fc151e0d503e8cb5fcb93cc8669c376dfd6f278" +dependencies = [ + "hex", + "hyper", + "hyper-util", + "pin-project-lite", + "tokio", + "tower-service", + "winapi", +] + [[package]] name = "hyper-rustls" version = "0.27.7" @@ -544,6 +949,19 @@ dependencies = [ "tower-service", ] +[[package]] +name = "hyper-timeout" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b90d566bffbce6a75bd8b09a05aa8c2cb1fabb6cb348f8840c9e4c90a0d83b0" +dependencies = [ + "hyper", + "hyper-util", + "pin-project-lite", + "tokio", + "tower-service", +] + [[package]] name = "hyper-tls" version = "0.6.0" @@ -585,6 +1003,45 @@ dependencies = [ "windows-registry", ] +[[package]] +name = "hyperlocal" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "986c5ce3b994526b3cd75578e62554abd09f0899d6206de48b3e96ab34ccc8c7" +dependencies = [ + "hex", + "http-body-util", + "hyper", + "hyper-util", + "pin-project-lite", + "tokio", + "tower-service", +] + +[[package]] +name = "iana-time-zone" +version = "0.1.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "log", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + [[package]] name = "icu_collections" version = "2.1.1" @@ -666,6 +1123,18 @@ dependencies = [ "zerovec", ] +[[package]] +name = "id-arena" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" + +[[package]] +name = "ident_case" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39" + [[package]] name = "idna" version = "1.1.0" @@ -687,6 +1156,17 @@ dependencies = [ "icu_properties", ] +[[package]] +name = "indexmap" +version = "1.9.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bd070e393353796e801d209ad339e89596eb4c8d430d18ede6a1cced8fafbd99" +dependencies = [ + "autocfg", + "hashbrown 0.12.3", + "serde", +] + [[package]] name = "indexmap" version = "2.13.0" @@ -695,6 +1175,8 @@ checksum = "7714e70437a7dc3ac8eb7e6f8df75fd8eb422675fc7678aff7364301092b1017" dependencies = [ "equivalent", "hashbrown 0.16.1", + "serde", + "serde_core", ] [[package]] @@ -722,6 +1204,15 @@ dependencies = [ "serde", ] +[[package]] +name = "itertools" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" +dependencies = [ + "either", +] + [[package]] name = "itoa" version = "1.0.17" @@ -744,6 +1235,12 @@ version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" +[[package]] +name = "leb128fmt" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" + [[package]] name = "libc" version = "0.2.182" @@ -815,6 +1312,7 @@ dependencies = [ "rusqlite", "serde", "serde_json", + "testcontainers-modules", "tokio", "tokio-postgres", "tower-lsp", @@ -841,6 +1339,12 @@ dependencies = [ "url", ] +[[package]] +name = "matchit" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3" + [[package]] name = "md-5" version = "0.10.6" @@ -900,6 +1404,85 @@ dependencies = [ "tempfile", ] +[[package]] +name = "num" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35bd024e8b2ff75562e5f34e7f4905839deb4b22955ef5e73d2fea1b9813cb23" +dependencies = [ + "num-bigint", + "num-complex", + "num-integer", + "num-iter", + "num-rational", + "num-traits", +] + +[[package]] +name = "num-bigint" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-conv" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6673768db2d862beb9b39a78fdcb1a69439615d5794a1be50caa9bc92c81967" + +[[package]] +name = "num-integer" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-iter" +version = "0.1.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1429034a0490724d0075ebb2bc9e875d6503c3cf69e235a8941aa757d83ef5bf" +dependencies = [ + "autocfg", + "num-integer", + "num-traits", +] + +[[package]] +name = "num-rational" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824" +dependencies = [ + "num-bigint", + "num-integer", + "num-traits", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", +] + [[package]] name = "objc2-core-foundation" version = "0.3.2" @@ -1016,6 +1599,31 @@ dependencies = [ "windows-link", ] +[[package]] +name = "parse-display" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "914a1c2265c98e2446911282c6ac86d8524f495792c38c5bd884f80499c7538a" +dependencies = [ + "parse-display-derive", + "regex", + "regex-syntax", +] + +[[package]] +name = "parse-display-derive" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2ae7800a4c974efd12df917266338e79a7a74415173caf7e70aa0a0707345281" +dependencies = [ + "proc-macro2", + "quote", + "regex", + "regex-syntax", + "structmeta", + "syn", +] + [[package]] name = "percent-encoding" version = "2.3.2" @@ -1079,6 +1687,12 @@ version = "0.3.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7edddbd0b52d732b21ad9a5fab5c704c14cd949e5e9a1ec5929a24fded1b904c" +[[package]] +name = "portable-atomic" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" + [[package]] name = "postgres-protocol" version = "0.6.10" @@ -1092,7 +1706,7 @@ dependencies = [ "hmac", "md-5", "memchr", - "rand", + "rand 0.9.2", "sha2", "stringprep", ] @@ -1117,6 +1731,12 @@ dependencies = [ "zerovec", ] +[[package]] +name = "powerfmt" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" + [[package]] name = "ppv-lite86" version = "0.2.21" @@ -1126,6 +1746,16 @@ dependencies = [ "zerocopy", ] +[[package]] +name = "prettyplease" +version = "0.2.37" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" +dependencies = [ + "proc-macro2", + "syn", +] + [[package]] name = "proc-macro2" version = "1.0.106" @@ -1135,6 +1765,38 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "prost" +version = "0.14.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2ea70524a2f82d518bce41317d0fae74151505651af45faf1ffbd6fd33f0568" +dependencies = [ + "bytes", + "prost-derive", +] + +[[package]] +name = "prost-derive" +version = "0.14.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "27c6023962132f4b30eb4c172c91ce92d933da334c59c23cddee82358ddafb0b" +dependencies = [ + "anyhow", + "itertools", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "prost-types" +version = "0.14.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8991c4cbdb8bc5b11f0b074ffe286c30e523de90fee5ba8132f1399f23cb3dd7" +dependencies = [ + "prost", +] + [[package]] name = "quote" version = "1.0.45" @@ -1150,6 +1812,12 @@ version = "5.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + [[package]] name = "rand" version = "0.9.2" @@ -1157,7 +1825,18 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6db2770f06117d490610c7488547d543617b21bfa07796d7a12f6f1bd53850d1" dependencies = [ "rand_chacha", - "rand_core", + "rand_core 0.9.5", +] + +[[package]] +name = "rand" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2e8e8bcc7961af1fdac401278c6a831614941f6164ee3bf4ce61b7edb162207" +dependencies = [ + "chacha20", + "getrandom 0.4.2", + "rand_core 0.10.1", ] [[package]] @@ -1167,7 +1846,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" dependencies = [ "ppv-lite86", - "rand_core", + "rand_core 0.9.5", ] [[package]] @@ -1179,6 +1858,12 @@ dependencies = [ "getrandom 0.3.4", ] +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + [[package]] name = "redox_syscall" version = "0.2.16" @@ -1197,6 +1882,55 @@ dependencies = [ "bitflags 2.11.0", ] +[[package]] +name = "ref-cast" +version = "1.0.25" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f354300ae66f76f1c85c5f84693f0ce81d747e2c3f21a45fef496d89c960bf7d" +dependencies = [ + "ref-cast-impl", +] + +[[package]] +name = "ref-cast-impl" +version = "1.0.25" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7186006dcb21920990093f30e3dea63b7d6e977bf1256be20c3563a5db070da" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "regex" +version = "1.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e10754a14b9137dd7b1e3e5b0493cc9171fdd105e0ab477f51b72e7f3ac0e276" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "regex-automata" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax", +] + +[[package]] +name = "regex-syntax" +version = "0.8.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a" + [[package]] name = "reqwest" version = "0.12.28" @@ -1265,6 +1999,12 @@ dependencies = [ "smallvec", ] +[[package]] +name = "rustc-hash" +version = "2.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" + [[package]] name = "rustix" version = "1.1.4" @@ -1284,13 +2024,27 @@ version = "0.23.37" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "758025cb5fccfd3bc2fd74708fd4682be41d99e5dff73c377c0646c6012c73a4" dependencies = [ + "log", "once_cell", + "ring", "rustls-pki-types", "rustls-webpki", "subtle", "zeroize", ] +[[package]] +name = "rustls-native-certs" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "612460d5f7bea540c490b2b6395d8e34a953e52b491accd6c86c8164c5932a63" +dependencies = [ + "openssl-probe", + "rustls-pki-types", + "schannel", + "security-framework", +] + [[package]] name = "rustls-pki-types" version = "1.14.0" @@ -1332,6 +2086,30 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "schemars" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cd191f9397d57d581cddd31014772520aa448f65ef991055d7f61582c65165f" +dependencies = [ + "dyn-clone", + "ref-cast", + "serde", + "serde_json", +] + +[[package]] +name = "schemars" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2b42f36aa1cd011945615b92222f6bf73c599a102a300334cd7f8dbeec726cc" +dependencies = [ + "dyn-clone", + "ref-cast", + "serde", + "serde_json", +] + [[package]] name = "scopeguard" version = "1.2.0" @@ -1361,6 +2139,12 @@ dependencies = [ "libc", ] +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + [[package]] name = "serde" version = "1.0.228" @@ -1416,15 +2200,46 @@ dependencies = [ ] [[package]] -name = "serde_urlencoded" -version = "0.7.1" +name = "serde_urlencoded" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" +dependencies = [ + "form_urlencoded", + "itoa", + "ryu", + "serde", +] + +[[package]] +name = "serde_with" +version = "3.19.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f05839ce67618e14a09b286535c0d9c94e85ef25469b0e13cb4f844e5593eb19" +dependencies = [ + "base64", + "chrono", + "hex", + "indexmap 1.9.3", + "indexmap 2.13.0", + "schemars 0.9.0", + "schemars 1.2.1", + "serde_core", + "serde_json", + "serde_with_macros", + "time", +] + +[[package]] +name = "serde_with_macros" +version = "3.19.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" +checksum = "cf2ebbe86054f9b45bc3881e865683ccfaccce97b9b4cb53f3039d67f355a334" dependencies = [ - "form_urlencoded", - "itoa", - "ryu", - "serde", + "darling", + "proc-macro2", + "quote", + "syn", ] [[package]] @@ -1434,7 +2249,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" dependencies = [ "cfg-if", - "cpufeatures", + "cpufeatures 0.2.17", "digest", ] @@ -1499,6 +2314,35 @@ dependencies = [ "unicode-properties", ] +[[package]] +name = "strsim" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" + +[[package]] +name = "structmeta" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2e1575d8d40908d70f6fd05537266b90ae71b15dbbe7a8b7dffa2b759306d329" +dependencies = [ + "proc-macro2", + "quote", + "structmeta-derive", + "syn", +] + +[[package]] +name = "structmeta-derive" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "152a0b65a590ff6c3da95cabe2353ee04e6167c896b28e3b14478c2636c922fc" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "subtle" version = "2.6.1" @@ -1570,6 +2414,97 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "testcontainers" +version = "0.27.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bfd5785b5483672915ed5fe3cddf9f546802779fc1eceff0a6fb7321fac81c1e" +dependencies = [ + "astral-tokio-tar", + "async-trait", + "bollard", + "bytes", + "docker_credential", + "either", + "etcetera", + "ferroid", + "futures", + "http", + "itertools", + "log", + "memchr", + "parse-display", + "pin-project-lite", + "serde", + "serde_json", + "serde_with", + "thiserror", + "tokio", + "tokio-stream", + "tokio-util", + "url", +] + +[[package]] +name = "testcontainers-modules" +version = "0.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5985fde5befe4ffa77a052e035e16c2da86e8bae301baa9f9904ad3c494d357" +dependencies = [ + "testcontainers", +] + +[[package]] +name = "thiserror" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "time" +version = "0.3.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "743bd48c283afc0388f9b8827b976905fb217ad9e647fae3a379a9283c4def2c" +dependencies = [ + "deranged", + "itoa", + "num-conv", + "powerfmt", + "serde_core", + "time-core", + "time-macros", +] + +[[package]] +name = "time-core" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca" + +[[package]] +name = "time-macros" +version = "0.2.27" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2e70e4c5a0e0a8a4823ad65dfe1a6930e4f4d756dcd9dd7939022b5e8c501215" +dependencies = [ + "num-conv", + "time-core", +] + [[package]] name = "tinystr" version = "0.8.2" @@ -1652,7 +2587,7 @@ dependencies = [ "pin-project-lite", "postgres-protocol", "postgres-types", - "rand", + "rand 0.9.2", "socket2", "tokio", "tokio-util", @@ -1669,6 +2604,17 @@ dependencies = [ "tokio", ] +[[package]] +name = "tokio-stream" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32da49809aab5c3bc678af03902d4ccddea2a87d028d86392a4b1560c6906c70" +dependencies = [ + "futures-core", + "pin-project-lite", + "tokio", +] + [[package]] name = "tokio-util" version = "0.7.18" @@ -1682,6 +2628,46 @@ dependencies = [ "tokio", ] +[[package]] +name = "tonic" +version = "0.14.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac2a5518c70fa84342385732db33fb3f44bc4cc748936eb5833d2df34d6445ef" +dependencies = [ + "async-trait", + "axum", + "base64", + "bytes", + "h2", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-timeout", + "hyper-util", + "percent-encoding", + "pin-project", + "socket2", + "sync_wrapper", + "tokio", + "tokio-stream", + "tower 0.5.3", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "tonic-prost" +version = "0.14.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "50849f68853be452acf590cde0b146665b8d507b3b8af17261df47e02c209ea0" +dependencies = [ + "bytes", + "prost", + "tonic", +] + [[package]] name = "tower" version = "0.4.13" @@ -1704,11 +2690,15 @@ checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4" dependencies = [ "futures-core", "futures-util", + "indexmap 2.13.0", "pin-project-lite", + "slab", "sync_wrapper", "tokio", + "tokio-util", "tower-layer", "tower-service", + "tracing", ] [[package]] @@ -1851,12 +2841,45 @@ version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7df058c713841ad818f1dc5d3fd88063241cc61f49f5fbea4b951e8cf5a8d71d" +[[package]] +name = "unicode-xid" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" + [[package]] name = "untrusted" version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" +[[package]] +name = "ureq" +version = "3.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dea7109cdcd5864d4eeb1b58a1648dc9bf520360d7af16ec26d0a9354bafcfc0" +dependencies = [ + "base64", + "log", + "percent-encoding", + "rustls", + "rustls-pki-types", + "ureq-proto", + "utf8-zero", +] + +[[package]] +name = "ureq-proto" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e994ba84b0bd1b1b0cf92878b7ef898a5c1760108fe7b6010327e274917a808c" +dependencies = [ + "base64", + "http", + "httparse", + "log", +] + [[package]] name = "url" version = "2.5.8" @@ -1870,6 +2893,12 @@ dependencies = [ "serde_derive", ] +[[package]] +name = "utf8-zero" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8c0a043c9540bae7c578c88f91dda8bd82e59ae27c21baca69c8b191aaf5a6e" + [[package]] name = "utf8_iter" version = "1.0.4" @@ -1927,6 +2956,15 @@ dependencies = [ "wit-bindgen", ] +[[package]] +name = "wasip3" +version = "0.4.0+wasi-0.3.0-rc-2026-01-06" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" +dependencies = [ + "wit-bindgen", +] + [[package]] name = "wasite" version = "1.0.2" @@ -1995,6 +3033,40 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "wasm-encoder" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319" +dependencies = [ + "leb128fmt", + "wasmparser", +] + +[[package]] +name = "wasm-metadata" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" +dependencies = [ + "anyhow", + "indexmap 2.13.0", + "wasm-encoder", + "wasmparser", +] + +[[package]] +name = "wasmparser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" +dependencies = [ + "bitflags 2.11.0", + "hashbrown 0.15.5", + "indexmap 2.13.0", + "semver", +] + [[package]] name = "web-sys" version = "0.3.91" @@ -2005,6 +3077,16 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "web-time" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + [[package]] name = "whoami" version = "2.1.1" @@ -2040,6 +3122,41 @@ version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" +[[package]] +name = "windows-core" +version = "0.62.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" +dependencies = [ + "windows-implement", + "windows-interface", + "windows-link", + "windows-result", + "windows-strings", +] + +[[package]] +name = "windows-implement" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "windows-interface" +version = "0.59.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "windows-link" version = "0.2.1" @@ -2236,6 +3353,88 @@ name = "wit-bindgen" version = "0.51.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" +dependencies = [ + "wit-bindgen-rust-macro", +] + +[[package]] +name = "wit-bindgen-core" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc" +dependencies = [ + "anyhow", + "heck", + "wit-parser", +] + +[[package]] +name = "wit-bindgen-rust" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" +dependencies = [ + "anyhow", + "heck", + "indexmap 2.13.0", + "prettyplease", + "syn", + "wasm-metadata", + "wit-bindgen-core", + "wit-component", +] + +[[package]] +name = "wit-bindgen-rust-macro" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a" +dependencies = [ + "anyhow", + "prettyplease", + "proc-macro2", + "quote", + "syn", + "wit-bindgen-core", + "wit-bindgen-rust", +] + +[[package]] +name = "wit-component" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" +dependencies = [ + "anyhow", + "bitflags 2.11.0", + "indexmap 2.13.0", + "log", + "serde", + "serde_derive", + "serde_json", + "wasm-encoder", + "wasm-metadata", + "wasmparser", + "wit-parser", +] + +[[package]] +name = "wit-parser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" +dependencies = [ + "anyhow", + "id-arena", + "indexmap 2.13.0", + "log", + "semver", + "serde", + "serde_derive", + "serde_json", + "unicode-xid", + "wasmparser", +] [[package]] name = "writeable" @@ -2243,6 +3442,16 @@ version = "0.6.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9edde0db4769d2dc68579893f2306b26c6ecfbe0ef499b013d731b7b9247e0b9" +[[package]] +name = "xattr" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32e45ad4206f6d2479085147f02bc2ef834ac85886624a23575ae137c8aa8156" +dependencies = [ + "libc", + "rustix", +] + [[package]] name = "yoke" version = "0.8.1" diff --git a/Lql/lql-lsp-rust/crates/lql-lsp/Cargo.toml b/Lql/lql-lsp-rust/crates/lql-lsp/Cargo.toml index 24f04e28..3c51f494 100644 --- a/Lql/lql-lsp-rust/crates/lql-lsp/Cargo.toml +++ b/Lql/lql-lsp-rust/crates/lql-lsp/Cargo.toml @@ -3,6 +3,10 @@ name = "lql-lsp" version.workspace = true edition.workspace = true +[lib] +name = "lql_lsp" +path = "src/lib.rs" + [[bin]] name = "lql-lsp" path = "src/main.rs" @@ -18,3 +22,6 @@ serde_json = "1" serde = { version = "1", features = ["derive"] } tracing = "0.1" reqwest = { version = "0.12", features = ["json"] } + +[dev-dependencies] +testcontainers-modules = { version = "0.15", features = ["postgres", "blocking"] } diff --git a/Lql/lql-lsp-rust/crates/lql-lsp/src/ai.rs b/Lql/lql-lsp-rust/crates/lql-lsp/src/ai.rs index bb2a220f..7a3563f2 100644 --- a/Lql/lql-lsp-rust/crates/lql-lsp/src/ai.rs +++ b/Lql/lql-lsp-rust/crates/lql-lsp/src/ai.rs @@ -684,4 +684,194 @@ mod tests { assert_eq!(item.sort_priority, 6); } } + + // ── OllamaProvider::complete via unreachable endpoint ────────────── + + #[tokio::test] + async fn ollama_complete_returns_empty_on_connection_failure() { + // Port 1 (tcpmux) is reserved and refuses connections — this exercises + // the request-error branch at line 250-251 of ai.rs without depending + // on any external service. + let config = AiConfig { + provider: "ollama".to_string(), + endpoint: "http://127.0.0.1:1/api/generate".to_string(), + model: "test-model".to_string(), + api_key: None, + timeout_ms: 1000, + enabled: true, + }; + let provider = OllamaProvider::new(&config, "ref".to_string()); + let ctx = AiCompletionContext { + document_text: "users |> ".to_string(), + line: 0, + column: 9, + line_prefix: "users |> ".to_string(), + word_prefix: "".to_string(), + file_uri: "file:///x.lql".to_string(), + available_tables: vec!["users".to_string()], + schema_description: "users(id uuid PK)".to_string(), + }; + + let items = provider.complete(&ctx).await; + assert!( + items.is_empty(), + "unreachable endpoint must yield empty completions" + ); + } + + #[tokio::test] + async fn ollama_complete_returns_empty_on_non_json_body() { + // Hit a plain HTTP service that returns non-JSON — this exercises + // the response.json() error branch (line 254-256). Use httpbin's + // /html endpoint via a local stub: spin up a one-shot tokio server + // that returns text/plain. Avoids external network dependence by + // binding to 127.0.0.1:0 and serving a minimal HTTP response. + use tokio::io::AsyncWriteExt; + use tokio::net::TcpListener; + + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let endpoint = format!("http://{addr}/api/generate"); + + tokio::spawn(async move { + if let Ok((mut stream, _)) = listener.accept().await { + // Drain request headers (don't bother parsing the body) + let mut buf = [0u8; 1024]; + use tokio::io::AsyncReadExt; + let _ = stream.read(&mut buf).await; + let body = "not valid json at all"; + let resp = format!( + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", + body.len(), + body + ); + let _ = stream.write_all(resp.as_bytes()).await; + let _ = stream.shutdown().await; + } + }); + + let config = AiConfig { + provider: "ollama".to_string(), + endpoint, + model: "m".to_string(), + api_key: None, + timeout_ms: 5000, + enabled: true, + }; + let provider = OllamaProvider::new(&config, "ref".to_string()); + let ctx = AiCompletionContext { + document_text: String::new(), + line: 0, + column: 0, + line_prefix: String::new(), + word_prefix: String::new(), + file_uri: String::new(), + available_tables: vec![], + schema_description: String::new(), + }; + let items = provider.complete(&ctx).await; + assert!(items.is_empty(), "non-JSON body must yield empty"); + } + + #[tokio::test] + async fn ollama_complete_returns_empty_when_response_field_missing() { + // Server returns valid JSON but no "response" field — exercises + // the response-field-missing branch (line 259-261). + use tokio::io::AsyncWriteExt; + use tokio::net::TcpListener; + + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let endpoint = format!("http://{addr}/api/generate"); + + tokio::spawn(async move { + if let Ok((mut stream, _)) = listener.accept().await { + use tokio::io::AsyncReadExt; + let mut buf = [0u8; 1024]; + let _ = stream.read(&mut buf).await; + let body = r#"{"other_field": "no response key here"}"#; + let resp = format!( + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", + body.len(), + body + ); + let _ = stream.write_all(resp.as_bytes()).await; + let _ = stream.shutdown().await; + } + }); + + let config = AiConfig { + provider: "ollama".to_string(), + endpoint, + model: "m".to_string(), + api_key: None, + timeout_ms: 5000, + enabled: true, + }; + let provider = OllamaProvider::new(&config, "ref".to_string()); + let ctx = AiCompletionContext { + document_text: String::new(), + line: 0, + column: 0, + line_prefix: String::new(), + word_prefix: String::new(), + file_uri: String::new(), + available_tables: vec![], + schema_description: String::new(), + }; + let items = provider.complete(&ctx).await; + assert!(items.is_empty()); + } + + #[tokio::test] + async fn ollama_complete_parses_completions_when_response_field_present() { + // Server returns Ollama-shaped JSON with a "response" field that is + // a JSON-array string of completions — covers the success path + // (line 264 -> parse_response). + use tokio::io::AsyncWriteExt; + use tokio::net::TcpListener; + + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let endpoint = format!("http://{addr}/api/generate"); + + tokio::spawn(async move { + if let Ok((mut stream, _)) = listener.accept().await { + use tokio::io::AsyncReadExt; + let mut buf = [0u8; 1024]; + let _ = stream.read(&mut buf).await; + let inner = r#"[{"label":"foo","insertText":"foo()","detail":"AI"}]"#; + let body = serde_json::json!({ "response": inner }).to_string(); + let resp = format!( + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", + body.len(), + body + ); + let _ = stream.write_all(resp.as_bytes()).await; + let _ = stream.shutdown().await; + } + }); + + let config = AiConfig { + provider: "ollama".to_string(), + endpoint, + model: "m".to_string(), + api_key: None, + timeout_ms: 5000, + enabled: true, + }; + let provider = OllamaProvider::new(&config, "ref".to_string()); + let ctx = AiCompletionContext { + document_text: String::new(), + line: 0, + column: 0, + line_prefix: String::new(), + word_prefix: String::new(), + file_uri: String::new(), + available_tables: vec![], + schema_description: String::new(), + }; + let items = provider.complete(&ctx).await; + assert!(items.iter().any(|i| i.label == "foo"), "should parse foo"); + } } diff --git a/Lql/lql-lsp-rust/crates/lql-lsp/src/db.rs b/Lql/lql-lsp-rust/crates/lql-lsp/src/db.rs index a7d17dc2..ab8bb107 100644 --- a/Lql/lql-lsp-rust/crates/lql-lsp/src/db.rs +++ b/Lql/lql-lsp-rust/crates/lql-lsp/src/db.rs @@ -401,4 +401,70 @@ mod tests { assert_eq!(result.unwrap().table_count(), 0); std::fs::remove_file("/tmp/lql_nonexistent_db_12345.db").ok(); } + + // ── fetch_schema dispatch ───────────────────────────────────────── + + #[tokio::test] + async fn fetch_schema_dispatches_to_sqlite_for_db_extension() { + let path = std::env::temp_dir().join("lql_dispatch_test.db"); + let path_str = path.to_str().unwrap(); + + let conn = rusqlite::Connection::open(path_str).unwrap(); + conn.execute_batch( + "CREATE TABLE IF NOT EXISTS dispatch_users (id TEXT PRIMARY KEY NOT NULL);", + ) + .unwrap(); + drop(conn); + + let schema = fetch_schema(path_str).await.unwrap(); + assert!(schema.get_table("dispatch_users").is_some()); + + std::fs::remove_file(path_str).ok(); + } + + #[tokio::test] + async fn fetch_schema_dispatches_to_sqlite_for_sqlite_prefix() { + let path = std::env::temp_dir().join("lql_dispatch_prefix.db"); + let path_str = path.to_str().unwrap(); + + let conn = rusqlite::Connection::open(path_str).unwrap(); + conn.execute_batch( + "CREATE TABLE IF NOT EXISTS dispatch_orders (id TEXT PRIMARY KEY NOT NULL);", + ) + .unwrap(); + drop(conn); + + let with_prefix = format!("sqlite:{path_str}"); + let schema = fetch_schema(&with_prefix).await.unwrap(); + assert!(schema.get_table("dispatch_orders").is_some()); + + std::fs::remove_file(path_str).ok(); + } + + // ── fetch_postgres_schema connect failure ───────────────────────── + + #[tokio::test] + async fn fetch_postgres_schema_connect_failure_returns_error() { + // Port 1 is reserved (tcpmux) — the connect will be refused, exercising + // the connect error branch (line 147) without waiting for the timeout. + let result = fetch_postgres_schema( + "host=127.0.0.1 port=1 dbname=test user=test password=test connect_timeout=1", + ) + .await; + assert!(result.is_err(), "connect to localhost:1 must fail"); + let err = result.unwrap_err(); + assert!( + err.contains("DB connect"), + "error must mention DB connect: {err}" + ); + } + + #[tokio::test] + async fn fetch_postgres_schema_via_npgsql_format_routes_normalize() { + // Drives normalize_connection_string from inside fetch_postgres_schema + // and exercises the connect-error branch with a different format. + let result = + fetch_postgres_schema("Host=127.0.0.1;Port=1;Database=x;Username=u;Password=p").await; + assert!(result.is_err()); + } } diff --git a/Lql/lql-lsp-rust/crates/lql-lsp/src/lib.rs b/Lql/lql-lsp-rust/crates/lql-lsp/src/lib.rs new file mode 100644 index 00000000..2b8cc19b --- /dev/null +++ b/Lql/lql-lsp-rust/crates/lql-lsp/src/lib.rs @@ -0,0 +1,8 @@ +//! Public surface of the LQL Language Server. +//! +//! The binary entry point lives in `main.rs`; this library facade exists +//! so integration tests can call into `db` and `ai` modules directly +//! (binary targets cannot be imported from `tests/` files). + +pub mod ai; +pub mod db; diff --git a/Lql/lql-lsp-rust/crates/lql-lsp/src/main.rs b/Lql/lql-lsp-rust/crates/lql-lsp/src/main.rs index 1185d5e2..15d97d80 100644 --- a/Lql/lql-lsp-rust/crates/lql-lsp/src/main.rs +++ b/Lql/lql-lsp-rust/crates/lql-lsp/src/main.rs @@ -1,5 +1,5 @@ -pub mod ai; -mod db; +use lql_lsp::ai; +use lql_lsp::db; use ai::{AiCompletionContext, AiCompletionProvider, AiConfig}; use lql_analyzer::{ @@ -253,57 +253,11 @@ impl LanguageServer for LqlBackend { let ai_config = self.ai_config.lock().unwrap().clone(); if let Some(ref config) = ai_config { if config.enabled { - // Activate built-in test providers or log external config - match config.provider.as_str() { - "test" => { - self.set_ai_provider(Arc::new(ai::TestAiProvider)).await; - self.client - .log_message( - MessageType::INFO, - "AI test provider activated — returns deterministic completions", - ) - .await; - } - "test_slow" => { - let delay = config.timeout_ms.saturating_add(5000); - self.set_ai_provider(Arc::new(ai::SlowAiProvider { delay_ms: delay })) - .await; - self.client - .log_message( - MessageType::INFO, - format!( - "AI slow test provider activated — {}ms delay (timeout: {}ms)", - delay, config.timeout_ms - ), - ) - .await; - } - "ollama" => { - let lql_ref = include_str!("../../lql-reference.md").to_string(); - self.set_ai_provider(Arc::new(ai::OllamaProvider::new(config, lql_ref))) - .await; - self.client - .log_message( - MessageType::INFO, - format!( - "Ollama AI provider activated (model: {}, endpoint: {})", - config.model, config.endpoint - ), - ) - .await; - } - _ => { - self.client - .log_message( - MessageType::INFO, - format!( - "AI completion provider configured: {} (model: {}, endpoint: {})", - config.provider, config.model, config.endpoint - ), - ) - .await; - } + let (provider, log_msg) = build_ai_provider(config); + if let Some(p) = provider { + self.set_ai_provider(p).await; } + self.client.log_message(MessageType::INFO, log_msg).await; } } @@ -602,6 +556,52 @@ impl LanguageServer for LqlBackend { } } +/// Build the AI completion provider for a given config and produce the +/// log message to surface to the client. +/// +/// Pure function (no I/O, no state) so it can be unit-tested without the +/// full LSP runtime — closes the coverage gap on the AI dispatch arms in +/// `initialized()`. +pub fn build_ai_provider( + config: &ai::AiConfig, +) -> (Option<Arc<dyn ai::AiCompletionProvider>>, String) { + match config.provider.as_str() { + "test" => ( + Some(Arc::new(ai::TestAiProvider) as Arc<dyn ai::AiCompletionProvider>), + "AI test provider activated — returns deterministic completions".to_string(), + ), + "test_slow" => { + let delay = config.timeout_ms.saturating_add(5000); + ( + Some(Arc::new(ai::SlowAiProvider { delay_ms: delay }) + as Arc<dyn ai::AiCompletionProvider>), + format!( + "AI slow test provider activated — {}ms delay (timeout: {}ms)", + delay, config.timeout_ms + ), + ) + } + "ollama" => { + let lql_ref = include_str!("../../lql-reference.md").to_string(); + ( + Some(Arc::new(ai::OllamaProvider::new(config, lql_ref)) + as Arc<dyn ai::AiCompletionProvider>), + format!( + "Ollama AI provider activated (model: {}, endpoint: {})", + config.model, config.endpoint + ), + ) + } + _ => ( + None, + format!( + "AI completion provider configured: {} (model: {}, endpoint: {})", + config.provider, config.model, config.endpoint + ), + ), + } +} + fn format_lql(source: &str) -> String { let mut result = String::new(); let mut indent = 0usize; @@ -1879,4 +1879,80 @@ mod tests { let table_item = items.iter().find(|i| i.label == "big_table").unwrap(); assert!(table_item.documentation.contains("10 total")); } + + // ── build_ai_provider — covers the AI dispatch arms ────────────── + + fn ai_cfg(provider: &str) -> ai::AiConfig { + ai::AiConfig { + provider: provider.to_string(), + endpoint: "http://localhost:11434/api/generate".to_string(), + model: "test-model".to_string(), + api_key: None, + timeout_ms: 1000, + enabled: true, + } + } + + #[test] + fn build_ai_provider_test_arm() { + let (provider, msg) = build_ai_provider(&ai_cfg("test")); + assert!(provider.is_some(), "test arm must yield a provider"); + assert!( + msg.contains("AI test provider"), + "log message must mention test provider, got: {msg}" + ); + } + + #[test] + fn build_ai_provider_test_slow_arm() { + let cfg = ai_cfg("test_slow"); + let (provider, msg) = build_ai_provider(&cfg); + assert!(provider.is_some()); + assert!(msg.contains("slow test provider"), "got: {msg}"); + // delay = timeout_ms + 5000 + assert!( + msg.contains("6000ms delay"), + "delay should be 1000 + 5000 = 6000ms, got: {msg}" + ); + assert!(msg.contains("timeout: 1000ms")); + } + + #[test] + fn build_ai_provider_ollama_arm() { + let (provider, msg) = build_ai_provider(&ai_cfg("ollama")); + assert!(provider.is_some(), "ollama arm must yield a provider"); + assert!(msg.contains("Ollama AI provider"), "got: {msg}"); + assert!(msg.contains("test-model")); + assert!(msg.contains("http://localhost:11434/api/generate")); + } + + #[test] + fn build_ai_provider_unknown_arm_returns_none() { + let (provider, msg) = build_ai_provider(&ai_cfg("some-other-provider")); + assert!( + provider.is_none(), + "unknown provider must NOT install a built-in" + ); + assert!( + msg.contains("AI completion provider configured"), + "got: {msg}" + ); + assert!(msg.contains("some-other-provider")); + } + + #[test] + fn build_ai_provider_test_slow_saturating_add() { + let cfg = ai::AiConfig { + provider: "test_slow".to_string(), + endpoint: String::new(), + model: String::new(), + api_key: None, + timeout_ms: u64::MAX, + enabled: true, + }; + let (provider, msg) = build_ai_provider(&cfg); + assert!(provider.is_some()); + // saturating_add(u64::MAX, 5000) saturates at u64::MAX + assert!(msg.contains(&format!("{}ms delay", u64::MAX))); + } } diff --git a/Lql/lql-lsp-rust/crates/lql-lsp/tests/lsp_protocol.rs b/Lql/lql-lsp-rust/crates/lql-lsp/tests/lsp_protocol.rs index 40cba3a9..0926eb21 100644 --- a/Lql/lql-lsp-rust/crates/lql-lsp/tests/lsp_protocol.rs +++ b/Lql/lql-lsp-rust/crates/lql-lsp/tests/lsp_protocol.rs @@ -853,6 +853,215 @@ fn test_completion_after_dot_without_qualifier() { client.shutdown(); } +#[test] +fn test_version_flag_prints_and_exits() { + // Drives main.rs lines 640-641: `lql-lsp --version` must print + // "lql-lsp <semver>" to stdout and exit 0 without starting the LSP loop. + let binary = env!("CARGO_BIN_EXE_lql-lsp"); + let output = Command::new(binary) + .arg("--version") + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::null()) + .output() + .expect("failed to invoke --version"); + + assert!(output.status.success(), "--version must exit 0"); + let text = String::from_utf8_lossy(&output.stdout); + assert!( + text.starts_with("lql-lsp "), + "version line must start with 'lql-lsp ', got: {text}" + ); + assert!( + text.split_whitespace().count() == 2, + "version line must be 'lql-lsp <version>', got: {text}" + ); +} + +#[test] +fn test_short_version_flag_prints_and_exits() { + let binary = env!("CARGO_BIN_EXE_lql-lsp"); + let output = Command::new(binary) + .arg("-V") + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::null()) + .output() + .expect("failed to invoke -V"); + + assert!(output.status.success(), "-V must exit 0"); + let text = String::from_utf8_lossy(&output.stdout); + assert!(text.starts_with("lql-lsp "), "got: {text}"); +} + +#[test] +fn test_completion_with_ai_provider_and_schema_loaded() { + // Drives main.rs lines 413-441 (the Some(s) schema arm of the AI + // completion-merge path) by initializing with both an aiProvider and + // a SQLite connection string so a schema is loaded into the cache. + let dir = std::env::temp_dir().join(format!("lql_ai_schema_test_{}.db", std::process::id())); + let path = dir.to_str().unwrap(); + + // Seed the database before spawning the LSP. + { + let conn = rusqlite::Connection::open(path).unwrap(); + conn.execute_batch( + "CREATE TABLE IF NOT EXISTS users_ai_schema_test ( + id TEXT PRIMARY KEY NOT NULL, + name TEXT NOT NULL, + email TEXT + );", + ) + .unwrap(); + } + + let mut client = LspClient::spawn(); + let id = client.send_request( + "initialize", + Some(json!({ + "processId": std::process::id(), + "capabilities": {}, + "rootUri": null, + "initializationOptions": { + "connectionString": path, + "aiProvider": { + "provider": "test", + "enabled": true, + } + } + })), + ); + let resp = client.read_response(id); + assert!(resp.get("error").is_none(), "init must succeed"); + client.send_notification("initialized", json!({})); + + // Give the schema fetch a moment to populate. + std::thread::sleep(std::time::Duration::from_millis(500)); + + client.open_document(DOC_URI, "users_ai_schema_test |> "); + let _ = client.read_notification("textDocument/publishDiagnostics"); + + let resp = client.request_completion(DOC_URI, 0, 24); + assert!(resp.get("error").is_none()); + let items = resp["result"].as_array().unwrap(); + assert!(!items.is_empty(), "completion must return items"); + + client.shutdown(); + std::fs::remove_file(path).ok(); +} + +#[test] +fn test_completion_with_test_slow_provider_times_out() { + // Drives the AI timeout branch (main.rs around line 460-464) and the + // "test_slow" arm of the initialized() AI dispatch (lines 267-280). + let mut client = LspClient::spawn(); + let id = client.send_request( + "initialize", + Some(json!({ + "processId": std::process::id(), + "capabilities": {}, + "rootUri": null, + "initializationOptions": { + "aiProvider": { + "provider": "test_slow", + "enabled": true, + "timeoutMs": 100, + } + } + })), + ); + let _ = client.read_response(id); + client.send_notification("initialized", json!({})); + + client.open_document(DOC_URI, "users |> "); + let _ = client.read_notification("textDocument/publishDiagnostics"); + + let resp = client.request_completion(DOC_URI, 0, 9); + // Even when AI times out, schema/keyword completions should still be returned. + assert!(resp.get("error").is_none()); + let items = resp["result"].as_array().unwrap(); + assert!( + !items.is_empty(), + "must still return non-AI completions when AI times out" + ); + + client.shutdown(); +} + +#[test] +fn test_completion_with_unknown_ai_provider_uses_default_branch() { + // Drives the `_` arm in initialized()'s AI dispatch (main.rs line 295). + let mut client = LspClient::spawn(); + let id = client.send_request( + "initialize", + Some(json!({ + "processId": std::process::id(), + "capabilities": {}, + "rootUri": null, + "initializationOptions": { + "aiProvider": { + "provider": "some-unrecognised-provider", + "enabled": true, + } + } + })), + ); + let _ = client.read_response(id); + client.send_notification("initialized", json!({})); + + client.open_document(DOC_URI, "users |> "); + let _ = client.read_notification("textDocument/publishDiagnostics"); + + let resp = client.request_completion(DOC_URI, 0, 9); + assert!(resp.get("error").is_none()); + + client.shutdown(); +} + +#[test] +fn test_initialize_with_connection_string_only_loads_schema() { + // Drives the connectionString-without-aiProvider path in initialized() + // — exercises lines 311-349 (DB connect + schema fetch success arm). + let dir = std::env::temp_dir().join(format!("lql_conn_only_test_{}.db", std::process::id())); + let path = dir.to_str().unwrap(); + + { + let conn = rusqlite::Connection::open(path).unwrap(); + conn.execute_batch( + "CREATE TABLE IF NOT EXISTS conn_only_users (id TEXT PRIMARY KEY NOT NULL);", + ) + .unwrap(); + } + + let mut client = LspClient::spawn(); + let id = client.send_request( + "initialize", + Some(json!({ + "processId": std::process::id(), + "capabilities": {}, + "rootUri": null, + "initializationOptions": { + "connectionString": path, + } + })), + ); + let resp = client.read_response(id); + assert!(resp.get("error").is_none()); + client.send_notification("initialized", json!({})); + + std::thread::sleep(std::time::Duration::from_millis(300)); + + client.open_document(DOC_URI, "conn_only_users."); + let _ = client.read_notification("textDocument/publishDiagnostics"); + + // Trigger column completion after the dot + let resp = client.request_completion(DOC_URI, 0, 16); + assert!(resp.get("error").is_none()); + + client.shutdown(); + std::fs::remove_file(path).ok(); +} + #[test] fn test_document_symbols_positions() { let mut client = LspClient::spawn(); diff --git a/Lql/lql-lsp-rust/crates/lql-lsp/tests/postgres_schema.rs b/Lql/lql-lsp-rust/crates/lql-lsp/tests/postgres_schema.rs new file mode 100644 index 00000000..e09ec936 --- /dev/null +++ b/Lql/lql-lsp-rust/crates/lql-lsp/tests/postgres_schema.rs @@ -0,0 +1,126 @@ +//! Integration tests for `db::fetch_postgres_schema` using a real +//! Postgres container started via testcontainers-modules. +//! +//! Closes the coverage gap on `crates/lql-lsp/src/db.rs` that comes from +//! never exercising the actual Postgres query paths in unit tests. This +//! test starts an ephemeral Postgres, seeds it with a known schema, then +//! calls fetch_postgres_schema and asserts the cache is populated. + +use testcontainers_modules::postgres::Postgres; +use testcontainers_modules::testcontainers::runners::AsyncRunner; +use tokio_postgres::NoTls; + +#[tokio::test] +async fn fetch_postgres_schema_returns_seeded_tables() { + // Start a fresh Postgres container. + let container = Postgres::default() + .start() + .await + .expect("failed to start postgres container"); + + let host = container.get_host().await.expect("container has no host"); + let port = container + .get_host_port_ipv4(5432) + .await + .expect("container exposes no port"); + + // Seed a known schema using tokio-postgres directly. + let conn_str = + format!("host={host} port={port} user=postgres password=postgres dbname=postgres"); + let (client, connection) = tokio_postgres::connect(&conn_str, NoTls) + .await + .expect("seeder connect"); + let connection_handle = tokio::spawn(async move { + let _ = connection.await; + }); + + client + .batch_execute( + "CREATE TABLE pg_test_users ( + id UUID PRIMARY KEY NOT NULL, + name TEXT NOT NULL, + email TEXT + ); + CREATE TABLE pg_test_orders ( + id UUID PRIMARY KEY NOT NULL, + user_id UUID NOT NULL, + total NUMERIC(10,2) NOT NULL + );", + ) + .await + .expect("seed schema"); + + drop(client); + let _ = connection_handle.await; + + // Now exercise fetch_postgres_schema against the seeded DB. + let schema = lql_lsp::db::fetch_postgres_schema(&conn_str) + .await + .expect("fetch_postgres_schema must succeed against live DB"); + + assert!( + schema.get_table("pg_test_users").is_some(), + "expected pg_test_users in schema" + ); + assert!( + schema.get_table("pg_test_orders").is_some(), + "expected pg_test_orders in schema" + ); + + let users = schema.get_table("pg_test_users").unwrap(); + let id_col = users + .columns + .iter() + .find(|c| c.name == "id") + .expect("id column missing"); + assert!(id_col.is_primary_key, "id must be marked PK"); + assert!(!id_col.is_nullable, "id must be NOT NULL"); + + let email_col = users + .columns + .iter() + .find(|c| c.name == "email") + .expect("email column missing"); + assert!(email_col.is_nullable, "email must be nullable"); + assert!(!email_col.is_primary_key); + + // Container drops here, stopping the postgres instance. + drop(container); +} + +#[tokio::test] +async fn fetch_schema_dispatches_to_postgres_for_libpq_uri() { + // Drives the dispatch arm of `fetch_schema` (db.rs line 132) — when + // the connection string is NOT detected as SQLite, the call routes + // to fetch_postgres_schema. + let container = Postgres::default() + .start() + .await + .expect("failed to start postgres container"); + + let host = container.get_host().await.unwrap(); + let port = container.get_host_port_ipv4(5432).await.unwrap(); + let conn_str = + format!("host={host} port={port} user=postgres password=postgres dbname=postgres"); + + // Seed something so the schema is non-empty. + let (client, connection) = tokio_postgres::connect(&conn_str, NoTls) + .await + .expect("seeder connect"); + let h = tokio::spawn(async move { + let _ = connection.await; + }); + client + .batch_execute("CREATE TABLE dispatch_pg_test (id UUID PRIMARY KEY NOT NULL);") + .await + .expect("seed"); + drop(client); + let _ = h.await; + + let schema = lql_lsp::db::fetch_schema(&conn_str) + .await + .expect("dispatch must succeed"); + assert!(schema.get_table("dispatch_pg_test").is_some()); + + drop(container); +} diff --git a/Makefile b/Makefile index ef2616e5..48ff8623 100644 --- a/Makefile +++ b/Makefile @@ -1,11 +1,11 @@ -# agent-pmo:d75d5c8 +# agent-pmo:74cf183 # ============================================================================= # Standard Makefile — Nimblesite.DataProvider.Core # Cross-platform: Linux, macOS, Windows (via GNU Make) # All targets are language-agnostic. Add language-specific helpers below. # ============================================================================= -.PHONY: build test lint fmt fmt-check clean check ci coverage setup +.PHONY: build test lint fmt clean ci setup check coverage vsix help # ----------------------------------------------------------------------------- # OS Detection — portable commands for Linux, macOS, and Windows @@ -42,7 +42,12 @@ DOTNET_TEST_PROJECTS = \ Reporting/Nimblesite.Reporting.Integration.Tests # ============================================================================= -# PRIMARY TARGETS (uniform interface — do not rename) +# Standard Targets +# +# Portfolio-wide uniform interface — these 7 targets exist in every repo and +# their names never change. See REPO-STANDARDS-SPEC [MAKE-TARGETS]. +# Do NOT add extra public targets here — put them in the Repo-Specific +# Targets section below. # ============================================================================= ## build: Compile/assemble all artifacts @@ -50,37 +55,48 @@ build: @echo "==> Building..." $(MAKE) _build -## test: Run full test suite with coverage enforcement +## test: Fail-fast tests + coverage + threshold enforcement. +## See REPO-STANDARDS-SPEC [TEST-RULES]. test: @echo "==> Testing..." $(MAKE) _test -## lint: Run all linters (fails on any warning) +## lint: Run all linters/analyzers (read-only). Does NOT format. lint: @echo "==> Linting..." $(MAKE) _lint -## fmt: Format all code in-place +## fmt: Format all code in-place. Pass CHECK=1 for read-only verification (CI). fmt: - @echo "==> Formatting..." - $(MAKE) _fmt - -## fmt-check: Check formatting without modifying -fmt-check: - @echo "==> Checking format..." - $(MAKE) _fmt_check + @echo "==> Formatting$(if $(CHECK), (check mode),)..." + $(MAKE) _fmt$(if $(CHECK),_check,) ## clean: Remove all build artifacts clean: @echo "==> Cleaning..." $(MAKE) _clean -## check: lint + test (pre-commit) -check: lint test - ## ci: lint + test + build (full CI simulation) ci: lint test build +## setup: Post-create dev environment setup (used by devcontainer) +setup: + @echo "==> Setting up development environment..." + $(MAKE) _setup + @echo "==> Setup complete. Run 'make ci' to validate." + +# ============================================================================= +# ----------------------------------------------------------------------------- +# Repo-Specific Targets (NOT part of the portfolio standard 7) +# +# Add helpers unique to this repo below. They MUST NOT shadow any of the 7 +# standard targets above. See REPO-STANDARDS-SPEC [MAKE-TARGETS]. +# ----------------------------------------------------------------------------- +# ============================================================================= + +## check: lint + test (pre-commit shortcut) +check: lint test + ## coverage: Generate HTML coverage report (runs tests first) coverage: @echo "==> Coverage report..." @@ -91,12 +107,6 @@ vsix: @echo "==> Building and packaging VSIX..." bash Lql/lql-lsp-rust/build-vsix.sh -## setup: Post-create dev environment setup (used by devcontainer) -setup: - @echo "==> Setting up development environment..." - $(MAKE) _setup - @echo "==> Setup complete. Run 'make ci' to validate." - # ============================================================================= # LANGUAGE-SPECIFIC IMPLEMENTATIONS # ============================================================================= @@ -109,6 +119,7 @@ _lint: _lint_dotnet _lint_rust _lint_ts _fmt: _fmt_dotnet _fmt_rust +# Internal — invoked by `make fmt CHECK=1`. _fmt_check: _fmt_check_dotnet _fmt_check_rust _clean: _clean_dotnet _clean_rust _clean_ts @@ -369,15 +380,16 @@ _setup_ts: # HELP # ============================================================================= help: - @echo "Available targets:" + @echo "Standard targets (portfolio-wide):" @echo " build - Compile/assemble all artifacts" - @echo " test - Run full test suite with coverage enforcement" - @echo " lint - Run all linters (errors mode)" - @echo " fmt - Format all code in-place" - @echo " fmt-check - Check formatting (no modification)" + @echo " test - Fail-fast tests + coverage + threshold enforcement" + @echo " lint - Run all linters/analyzers (read-only)" + @echo " fmt - Format all code in-place (CHECK=1 for read-only verify)" @echo " clean - Remove build artifacts" - @echo " check - lint + test (pre-commit)" - @echo " ci - lint + test + build (full CI)" - @echo " coverage - Generate and open HTML coverage report" + @echo " ci - lint + test + build (full CI simulation)" @echo " setup - Post-create dev environment setup" + @echo "" + @echo "Repo-specific targets:" + @echo " check - lint + test (pre-commit shortcut)" + @echo " coverage - Generate and open HTML coverage report" @echo " vsix - Build LSP + compile & package VS Code extension (.vsix)" diff --git a/Migration/DataProviderMigrate/Program.cs b/Migration/DataProviderMigrate/Program.cs index 81e6a3cb..9401f19d 100644 --- a/Migration/DataProviderMigrate/Program.cs +++ b/Migration/DataProviderMigrate/Program.cs @@ -1,9 +1,24 @@ +using System.Collections.Immutable; using System.Reflection; using Microsoft.Data.Sqlite; using Nimblesite.DataProvider.Migration.Core; using Nimblesite.DataProvider.Migration.Postgres; using Nimblesite.DataProvider.Migration.SQLite; using Npgsql; +using SchemaIntegrityResultError = Outcome.Result< + System.Collections.Immutable.ImmutableArray<string>, + Nimblesite.DataProvider.Migration.Core.MigrationError +>.Error< + System.Collections.Immutable.ImmutableArray<string>, + Nimblesite.DataProvider.Migration.Core.MigrationError +>; +using SchemaIntegrityResultOk = Outcome.Result< + System.Collections.Immutable.ImmutableArray<string>, + Nimblesite.DataProvider.Migration.Core.MigrationError +>.Ok< + System.Collections.Immutable.ImmutableArray<string>, + Nimblesite.DataProvider.Migration.Core.MigrationError +>; namespace DataProviderMigrate; @@ -94,6 +109,14 @@ private static int ExecuteMigration(MigrateParseResult.Success args) return 1; } + if (IsSqlServerProvider(args.Provider) && SchemaContainsRls(schema)) + { + // Implements [RLS-MSSQL]. The SQL Server migration package does + // not exist yet, so RLS targeting SQL Server must fail closed. + Console.WriteLine(MigrationError.RlsMssqlUnsupported().Message); + return 1; + } + return args.Provider.ToLowerInvariant() switch { "sqlite" => MigrateSqliteDatabase( @@ -112,6 +135,13 @@ private static int ExecuteMigration(MigrateParseResult.Success args) }; } + private static bool IsSqlServerProvider(string provider) => + provider.Equals("sqlserver", StringComparison.OrdinalIgnoreCase) + || provider.Equals("mssql", StringComparison.OrdinalIgnoreCase); + + private static bool SchemaContainsRls(SchemaDefinition schema) => + schema.Tables.Any(table => table.RowLevelSecurity is not null); + private static int MigrateSqliteDatabase( SchemaDefinition schema, string outputPath, @@ -250,7 +280,7 @@ is Outcome.Result<IReadOnlyList<SchemaOperation>, MigrationError>.Error< ? "Schema is up to date — no operations needed" : $"Schema is up to date for phase '{phase.ToString().ToLowerInvariant()}' — no operations needed" ); - return 0; + return VerifySchemaIntegrity(schema: schema, phase: phase, inspect: inspect); } Console.WriteLine( @@ -271,7 +301,76 @@ applyResult is Outcome.Result<bool, MigrationError>.Error<bool, MigrationError> } Console.WriteLine("Migration completed successfully"); - return 0; + return VerifySchemaIntegrity(schema: schema, phase: phase, inspect: inspect); + } + + private static int VerifySchemaIntegrity( + SchemaDefinition schema, + MigratePhase phase, + Func<Outcome.Result<SchemaDefinition, MigrationError>> inspect + ) + { + var inspectResult = inspect(); + if ( + inspectResult + is Outcome.Result<SchemaDefinition, MigrationError>.Error< + SchemaDefinition, + MigrationError + > inspectErr + ) + { + Console.WriteLine($"Error: schema integrity inspection failed: {inspectErr.Value}"); + return 1; + } + + var live = ( + (Outcome.Result<SchemaDefinition, MigrationError>.Ok< + SchemaDefinition, + MigrationError + >)inspectResult + ).Value; + var verification = SchemaIntegrityVerifier.Verify( + live: live, + desired: schema, + includeSupportObjects: true, + includeRls: phase != MigratePhase.Structural + ); + + return verification switch + { + SchemaIntegrityResultOk ok => WriteIntegrityResult(mismatches: ok.Value), + SchemaIntegrityResultError error => WriteIntegrityError(error: error.Value), + }; + } + + private static int WriteIntegrityResult(ImmutableArray<string> mismatches) + { + if (mismatches.Length == 0) + { + Console.WriteLine("Schema integrity check passed"); + return 0; + } + + WriteIntegrityFailure(mismatches: mismatches); + return 1; + } + + private static int WriteIntegrityError(MigrationError error) + { + Console.WriteLine($"Error: schema integrity verification failed: {error}"); + return 1; + } + + private static void WriteIntegrityFailure(ImmutableArray<string> mismatches) + { + var original = Console.ForegroundColor; + Console.ForegroundColor = ConsoleColor.Red; + Console.WriteLine("SCHEMA INTEGRITY CHECK FAILED"); + foreach (var mismatch in mismatches) + { + Console.WriteLine(mismatch); + } + Console.ForegroundColor = original; } private static IReadOnlyList<SchemaOperation> FilterByPhase( diff --git a/Migration/Nimblesite.DataProvider.Migration.Core/GlobalUsings.cs b/Migration/Nimblesite.DataProvider.Migration.Core/GlobalUsings.cs index e879122c..06bfea35 100644 --- a/Migration/Nimblesite.DataProvider.Migration.Core/GlobalUsings.cs +++ b/Migration/Nimblesite.DataProvider.Migration.Core/GlobalUsings.cs @@ -1,3 +1,4 @@ +global using System.Collections.Immutable; global using System.Data; global using Microsoft.Extensions.Logging; global using MigrationApplyResult = Outcome.Result< @@ -8,4 +9,8 @@ System.Collections.Generic.IReadOnlyList<Nimblesite.DataProvider.Migration.Core.SchemaOperation>, Nimblesite.DataProvider.Migration.Core.MigrationError >; +global using SchemaIntegrityResult = Outcome.Result< + System.Collections.Immutable.ImmutableArray<string>, + Nimblesite.DataProvider.Migration.Core.MigrationError +>; // Type aliases for Result types per CLAUDE.md diff --git a/Migration/Nimblesite.DataProvider.Migration.Core/MigrationRunner.cs b/Migration/Nimblesite.DataProvider.Migration.Core/MigrationRunner.cs index a604504b..f6a3a8cf 100644 --- a/Migration/Nimblesite.DataProvider.Migration.Core/MigrationRunner.cs +++ b/Migration/Nimblesite.DataProvider.Migration.Core/MigrationRunner.cs @@ -44,6 +44,7 @@ public static MigrationApplyResult Apply( } IDbTransaction? transaction = null; + var failures = new List<(string OperationType, Exception Exception)>(); try { @@ -75,14 +76,39 @@ public static MigrationApplyResult Apply( } catch (Exception ex) when (options.ContinueOnError) { + // Implements [MIG-RUNNER-HARD-FAIL]: ContinueOnError keeps the loop + // running for diagnostic visibility, but the runner must NEVER report + // success while operations were missed. Track the failure so we can + // return an aggregate Error after the loop. logger?.LogWarning( ex, "Failed to apply {OperationType}, continuing", operation.GetType().Name ); + failures.Add((operation.GetType().Name, ex)); } } + if (failures.Count > 0) + { + transaction?.Rollback(); + var summary = string.Join( + "; ", + failures.Select(f => $"{f.OperationType}: {f.Exception.Message}") + ); + logger?.LogError( + "Migration failed: {Count} of {Total} operation(s) errored: {Summary}", + failures.Count, + operations.Count, + summary + ); + return new MigrationApplyResult.Error<bool, MigrationError>( + MigrationError.FromMessage( + $"Migration failed: {failures.Count} of {operations.Count} operation(s) errored: {summary}" + ) + ); + } + transaction?.Commit(); logger?.LogInformation( "Migration completed: {Count} operations applied", diff --git a/Migration/Nimblesite.DataProvider.Migration.Core/SchemaIntegrityVerifier.cs b/Migration/Nimblesite.DataProvider.Migration.Core/SchemaIntegrityVerifier.cs new file mode 100644 index 00000000..3796d646 --- /dev/null +++ b/Migration/Nimblesite.DataProvider.Migration.Core/SchemaIntegrityVerifier.cs @@ -0,0 +1,710 @@ +using System.Text; + +namespace Nimblesite.DataProvider.Migration.Core; + +/// <summary> +/// Verifies that an inspected live schema satisfies a declared schema. +/// </summary> +public static class SchemaIntegrityVerifier +{ + /// <summary> + /// Compare the live schema against the desired schema and return every drift mismatch. + /// </summary> + public static SchemaIntegrityResult Verify( + SchemaDefinition live, + SchemaDefinition desired, + bool includeSupportObjects = true, + bool includeRls = true, + ILogger? logger = null + ) + { + try + { + var mismatches = ImmutableArray.CreateBuilder<string>(); + VerifyTables(live: live, desired: desired, includeRls: includeRls, mismatches); + if (includeSupportObjects) + { + VerifySupportObjects(live: live, desired: desired, mismatches: mismatches); + } + return new SchemaIntegrityResult.Ok<ImmutableArray<string>, MigrationError>( + mismatches.ToImmutable() + ); + } + catch (Exception ex) + { + logger?.LogError(ex, "Schema integrity verification failed"); + return new SchemaIntegrityResult.Error<ImmutableArray<string>, MigrationError>( + MigrationError.FromException(ex) + ); + } + } + + private static void VerifyTables( + SchemaDefinition live, + SchemaDefinition desired, + bool includeRls, + ImmutableArray<string>.Builder mismatches + ) + { + foreach (var expected in desired.Tables) + { + var actual = FindTable(schema: live, expected: expected); + if (actual is null) + { + mismatches.Add($"{TablePath(table: expected)}: missing table"); + continue; + } + VerifyTable(actual: actual, expected: expected, includeRls: includeRls, mismatches); + } + } + + private static void VerifyTable( + TableDefinition actual, + TableDefinition expected, + bool includeRls, + ImmutableArray<string>.Builder mismatches + ) + { + VerifyColumns(actual: actual, expected: expected, mismatches: mismatches); + VerifyPrimaryKey(actual: actual, expected: expected, mismatches: mismatches); + VerifyForeignKeys(actual: actual, expected: expected, mismatches: mismatches); + VerifyUniqueConstraints(actual: actual, expected: expected, mismatches: mismatches); + VerifyIndexes(actual: actual, expected: expected, mismatches: mismatches); + VerifyCheckConstraints(actual: actual, expected: expected, mismatches: mismatches); + if (includeRls) + { + VerifyRls(actual: actual, expected: expected, mismatches: mismatches); + } + } + + private static void VerifyColumns( + TableDefinition actual, + TableDefinition expected, + ImmutableArray<string>.Builder mismatches + ) + { + foreach (var expectedColumn in expected.Columns) + { + var actualColumn = FindColumn(table: actual, name: expectedColumn.Name); + if (actualColumn is null) + { + mismatches.Add( + $"{TablePath(table: expected)}.{expectedColumn.Name}: missing column" + ); + continue; + } + VerifyColumn( + actual: actualColumn, + expected: expectedColumn, + path: $"{TablePath(table: expected)}.{expectedColumn.Name}", + isSqlite: IsSqliteSchema(actual.Schema), + mismatches: mismatches + ); + } + } + + private static void VerifyColumn( + ColumnDefinition actual, + ColumnDefinition expected, + string path, + bool isSqlite, + ImmutableArray<string>.Builder mismatches + ) + { + AddIf( + condition: !SamePortableType(actual.Type, expected.Type, isSqlite), + $"{path}: type expected {expected.Type} but found {actual.Type}", + mismatches + ); + AddIf( + condition: actual.IsNullable != expected.IsNullable, + $"{path}: nullability expected {Nullability(expected)} but found {Nullability(actual)}", + mismatches + ); + AddIf( + condition: actual.IsIdentity != expected.IsIdentity, + $"{path}: identity expected {expected.IsIdentity} but found {actual.IsIdentity}", + mismatches + ); + VerifyDefault(actual: actual, expected: expected, path: path, mismatches: mismatches); + } + + private static void VerifyDefault( + ColumnDefinition actual, + ColumnDefinition expected, + string path, + ImmutableArray<string>.Builder mismatches + ) + { + if ( + expected.DefaultValue is not null + && !SameSql(actual.DefaultValue, expected.DefaultValue) + ) + { + mismatches.Add( + $"{path}: default expected {expected.DefaultValue} but found {actual.DefaultValue ?? "<none>"}" + ); + } + if ( + expected.DefaultLqlExpression is not null + && string.IsNullOrWhiteSpace(actual.DefaultValue) + ) + { + mismatches.Add( + $"{path}: default expected {expected.DefaultLqlExpression} but found <none>" + ); + } + } + + private static void VerifyPrimaryKey( + TableDefinition actual, + TableDefinition expected, + ImmutableArray<string>.Builder mismatches + ) + { + if (expected.PrimaryKey is null) + { + return; + } + if (actual.PrimaryKey is null) + { + mismatches.Add($"{TablePath(table: expected)}: missing primary key"); + return; + } + AddIf( + condition: !SameIdentifiers(actual.PrimaryKey.Columns, expected.PrimaryKey.Columns), + $"{TablePath(table: expected)}: primary key columns expected ({Format(expected.PrimaryKey.Columns)}) but found ({Format(actual.PrimaryKey.Columns)})", + mismatches + ); + } + + private static void VerifyForeignKeys( + TableDefinition actual, + TableDefinition expected, + ImmutableArray<string>.Builder mismatches + ) + { + foreach (var expectedFk in expected.ForeignKeys) + { + var actualFk = FindForeignKey(table: actual, expected: expectedFk); + if (actualFk is null) + { + mismatches.Add( + $"{TablePath(table: expected)}: missing foreign key {ForeignKeyName(expected.Name, expectedFk)} on ({Format(expectedFk.Columns)})" + ); + continue; + } + VerifyForeignKey( + actual: actualFk, + expected: expectedFk, + tablePath: TablePath(expected), + mismatches + ); + } + } + + private static void VerifyForeignKey( + ForeignKeyDefinition actual, + ForeignKeyDefinition expected, + string tablePath, + ImmutableArray<string>.Builder mismatches + ) + { + var name = ForeignKeyName(tablePath, expected); + AddIf( + !SameIdentifiers(actual.Columns, expected.Columns), + $"{tablePath}: foreign key {name} columns drifted", + mismatches + ); + AddIf( + !SameIdentifier(actual.ReferencedTable, expected.ReferencedTable), + $"{tablePath}: foreign key {name} referenced table drifted", + mismatches + ); + AddIf( + !SameIdentifiers(actual.ReferencedColumns, expected.ReferencedColumns), + $"{tablePath}: foreign key {name} referenced columns drifted", + mismatches + ); + AddIf( + actual.OnDelete != expected.OnDelete, + $"{tablePath}: foreign key {name} on delete expected {expected.OnDelete} but found {actual.OnDelete}", + mismatches + ); + AddIf( + actual.OnUpdate != expected.OnUpdate, + $"{tablePath}: foreign key {name} on update expected {expected.OnUpdate} but found {actual.OnUpdate}", + mismatches + ); + } + + private static void VerifyUniqueConstraints( + TableDefinition actual, + TableDefinition expected, + ImmutableArray<string>.Builder mismatches + ) + { + foreach (var expectedUnique in expected.UniqueConstraints) + { + var name = UniqueConstraintName(tableName: expected.Name, unique: expectedUnique); + var actualUnique = actual.UniqueConstraints.FirstOrDefault(uc => + SameIdentifier(UniqueConstraintName(actual.Name, uc), name) + ); + if (actualUnique is null) + { + mismatches.Add( + $"{TablePath(table: expected)}: missing unique constraint {name} on ({Format(expectedUnique.Columns)})" + ); + continue; + } + AddIf( + condition: !SameIdentifiers(actualUnique.Columns, expectedUnique.Columns), + $"{TablePath(table: expected)}: unique constraint {name} columns expected ({Format(expectedUnique.Columns)}) but found ({Format(actualUnique.Columns)})", + mismatches + ); + } + } + + private static void VerifyIndexes( + TableDefinition actual, + TableDefinition expected, + ImmutableArray<string>.Builder mismatches + ) + { + foreach (var expectedIndex in expected.Indexes) + { + var actualIndex = actual.Indexes.FirstOrDefault(i => + SameIdentifier(i.Name, expectedIndex.Name) + ); + if (actualIndex is null) + { + mismatches.Add($"{TablePath(table: expected)}: missing index {expectedIndex.Name}"); + continue; + } + VerifyIndex( + actual: actualIndex, + expected: expectedIndex, + tablePath: TablePath(expected), + mismatches + ); + } + } + + private static void VerifyIndex( + IndexDefinition actual, + IndexDefinition expected, + string tablePath, + ImmutableArray<string>.Builder mismatches + ) + { + AddIf( + actual.IsUnique != expected.IsUnique, + $"{tablePath}: index {expected.Name} uniqueness drifted", + mismatches + ); + AddIf( + !SameIdentifiers(actual.Columns, expected.Columns), + $"{tablePath}: index {expected.Name} columns drifted", + mismatches + ); + AddIf( + !SameSqlSequence(actual.Expressions, expected.Expressions), + $"{tablePath}: index {expected.Name} expressions drifted", + mismatches + ); + AddIf( + !SameSql(actual.Filter, expected.Filter), + $"{tablePath}: index {expected.Name} filter drifted", + mismatches + ); + } + + private static void VerifyCheckConstraints( + TableDefinition actual, + TableDefinition expected, + ImmutableArray<string>.Builder mismatches + ) + { + foreach (var expectedCheck in expected.CheckConstraints) + { + var actualCheck = actual.CheckConstraints.FirstOrDefault(c => + SameIdentifier(c.Name, expectedCheck.Name) + ); + if (actualCheck is null) + { + mismatches.Add( + $"{TablePath(table: expected)}: missing check constraint {expectedCheck.Name}" + ); + continue; + } + AddIf( + !SameSql(actualCheck.Expression, expectedCheck.Expression), + $"{TablePath(table: expected)}: check constraint {expectedCheck.Name} expression drifted", + mismatches + ); + } + VerifyColumnCheckConstraints(actual: actual, expected: expected, mismatches: mismatches); + } + + private static void VerifyColumnCheckConstraints( + TableDefinition actual, + TableDefinition expected, + ImmutableArray<string>.Builder mismatches + ) + { + foreach (var expectedColumn in expected.Columns.Where(c => c.CheckConstraint is not null)) + { + var actualColumn = FindColumn(table: actual, name: expectedColumn.Name); + if (actualColumn?.CheckConstraint is null) + { + mismatches.Add( + $"{TablePath(table: expected)}.{expectedColumn.Name}: missing check constraint" + ); + } + } + } + + private static void VerifyRls( + TableDefinition actual, + TableDefinition expected, + ImmutableArray<string>.Builder mismatches + ) + { + if (expected.RowLevelSecurity is null) + { + return; + } + AddIf( + actual.RowLevelSecurity?.Enabled != expected.RowLevelSecurity.Enabled, + $"{TablePath(table: expected)}: row-level security expected {expected.RowLevelSecurity.Enabled} but found {actual.RowLevelSecurity?.Enabled ?? false}", + mismatches + ); + AddIf( + actual.RowLevelSecurity?.Forced != expected.RowLevelSecurity.Forced, + $"{TablePath(table: expected)}: forced row-level security expected {expected.RowLevelSecurity.Forced} but found {actual.RowLevelSecurity?.Forced ?? false}", + mismatches + ); + VerifyRlsPolicies(actual: actual, expected: expected, mismatches: mismatches); + } + + private static void VerifyRlsPolicies( + TableDefinition actual, + TableDefinition expected, + ImmutableArray<string>.Builder mismatches + ) + { + foreach (var expectedPolicy in expected.RowLevelSecurity?.Policies ?? []) + { + var actualPolicy = actual.RowLevelSecurity?.Policies.FirstOrDefault(p => + SameIdentifier(p.Name, expectedPolicy.Name) + ); + if (actualPolicy is null) + { + mismatches.Add( + $"{TablePath(table: expected)}: missing row-level security policy {expectedPolicy.Name}" + ); + } + } + } + + private static void VerifySupportObjects( + SchemaDefinition live, + SchemaDefinition desired, + ImmutableArray<string>.Builder mismatches + ) + { + VerifyRoles(live: live, desired: desired, mismatches: mismatches); + VerifyFunctions(live: live, desired: desired, mismatches: mismatches); + VerifyGrants(live: live, desired: desired, mismatches: mismatches); + } + + private static void VerifyRoles( + SchemaDefinition live, + SchemaDefinition desired, + ImmutableArray<string>.Builder mismatches + ) + { + foreach (var expectedRole in desired.Roles) + { + var actualRole = live.Roles.FirstOrDefault(r => + SameIdentifier(r.Name, expectedRole.Name) + ); + if (actualRole is null) + { + mismatches.Add($"role {expectedRole.Name}: missing role"); + continue; + } + AddIf( + actualRole.Login != expectedRole.Login, + $"role {expectedRole.Name}: login drifted", + mismatches + ); + AddIf( + actualRole.BypassRls != expectedRole.BypassRls, + $"role {expectedRole.Name}: bypassRls drifted", + mismatches + ); + } + } + + private static void VerifyFunctions( + SchemaDefinition live, + SchemaDefinition desired, + ImmutableArray<string>.Builder mismatches + ) + { + foreach (var expectedFunction in desired.Functions) + { + var actualFunction = FindFunction(schema: live, expected: expectedFunction); + if (actualFunction is null) + { + mismatches.Add( + $"{expectedFunction.Schema}.{expectedFunction.Name}: missing function" + ); + continue; + } + VerifyFunction( + actual: actualFunction, + expected: expectedFunction, + mismatches: mismatches + ); + } + } + + private static void VerifyFunction( + PostgresFunctionDefinition actual, + PostgresFunctionDefinition expected, + ImmutableArray<string>.Builder mismatches + ) + { + var path = FunctionPath(function: expected); + AddIf( + condition: !SameIdentifier(actual: actual.Returns, expected: expected.Returns), + message: $"{path}: returns expected {expected.Returns} but found {actual.Returns}", + mismatches: mismatches + ); + AddIf( + condition: !SameIdentifier(actual: actual.Language, expected: expected.Language), + message: $"{path}: language expected {expected.Language} but found {actual.Language}", + mismatches: mismatches + ); + VerifyFunctionBehavior( + actual: actual, + expected: expected, + path: path, + mismatches: mismatches + ); + } + + private static void VerifyFunctionBehavior( + PostgresFunctionDefinition actual, + PostgresFunctionDefinition expected, + string path, + ImmutableArray<string>.Builder mismatches + ) + { + AddIf( + condition: !SameIdentifier(actual: actual.Volatility, expected: expected.Volatility), + message: $"{path}: volatility expected {expected.Volatility} but found {actual.Volatility}", + mismatches: mismatches + ); + AddIf( + condition: actual.SecurityDefiner != expected.SecurityDefiner, + message: $"{path}: security definer expected {expected.SecurityDefiner} but found {actual.SecurityDefiner}", + mismatches: mismatches + ); + AddIf( + condition: !SameFunctionBody(actual: actual, expected: expected), + message: $"{path}: function body drifted", + mismatches: mismatches + ); + } + + private static void VerifyGrants( + SchemaDefinition live, + SchemaDefinition desired, + ImmutableArray<string>.Builder mismatches + ) + { + foreach (var expectedGrant in desired.Grants) + { + var actualGrant = live.Grants.FirstOrDefault(g => + SameGrant(actual: g, expected: expectedGrant) + ); + if (actualGrant is null) + { + mismatches.Add( + $"grant {expectedGrant.Target} {expectedGrant.ObjectName ?? expectedGrant.Schema}: missing grant" + ); + } + } + } + + private static TableDefinition? FindTable(SchemaDefinition schema, TableDefinition expected) => + schema.Tables.FirstOrDefault(t => + SameSchema(t.Schema, expected.Schema) && SameIdentifier(t.Name, expected.Name) + ); + + private static ColumnDefinition? FindColumn(TableDefinition table, string name) => + table.Columns.FirstOrDefault(c => SameIdentifier(c.Name, name)); + + private static ForeignKeyDefinition? FindForeignKey( + TableDefinition table, + ForeignKeyDefinition expected + ) + { + if (expected.Name is not null) + { + return table.ForeignKeys.FirstOrDefault(fk => + SameIdentifier(fk.Name ?? string.Empty, expected.Name) + ); + } + return table.ForeignKeys.FirstOrDefault(fk => + SameIdentifiers(fk.Columns, expected.Columns) + ); + } + + private static PostgresFunctionDefinition? FindFunction( + SchemaDefinition schema, + PostgresFunctionDefinition expected + ) => + schema.Functions.FirstOrDefault(f => + SameIdentifier(f.Schema, expected.Schema) + && SameIdentifier(f.Name, expected.Name) + && SameIdentifiers( + f.Arguments.Select(a => a.Type).ToArray(), + expected.Arguments.Select(a => a.Type).ToArray() + ) + ); + + private static bool SameGrant( + PostgresGrantDefinition actual, + PostgresGrantDefinition expected + ) => + actual.Target == expected.Target + && SameIdentifier(actual.Schema, expected.Schema) + && SameIdentifier(actual.ObjectName ?? string.Empty, expected.ObjectName ?? string.Empty) + && SameIdentifiers(actual.Privileges, expected.Privileges) + && SameIdentifiers(actual.Roles, expected.Roles); + + private static bool SameFunctionBody( + PostgresFunctionDefinition actual, + PostgresFunctionDefinition expected + ) => + string.IsNullOrWhiteSpace(expected.Body) + || SameSql(actual: actual.Body, expected: expected.Body); + + private static bool SameIdentifiers( + IReadOnlyList<string> actual, + IReadOnlyList<string> expected + ) => + actual.Count == expected.Count + && actual.Zip(expected).All(pair => SameIdentifier(pair.First, pair.Second)); + + private static bool SameIdentifier(string actual, string expected) => + string.Equals(actual, expected, StringComparison.OrdinalIgnoreCase); + + private static bool SameSchema(string actual, string expected) => + SameIdentifier(actual: actual, expected: expected) + || (IsDefaultSchema(actual) && IsDefaultSchema(expected)); + + private static bool IsDefaultSchema(string schema) => + string.IsNullOrWhiteSpace(schema) + || SameIdentifier(actual: schema, expected: "main") + || SameIdentifier(actual: schema, expected: "public"); + + private static bool IsSqliteSchema(string schema) => + SameIdentifier(actual: schema, expected: "main"); + + private static bool SamePortableType(PortableType actual, PortableType expected, bool isSqlite) + { + if (Equals(actual, expected)) + { + return true; + } + return isSqlite && IsIntegerType(actual) && IsIntegerType(expected); + } + + private static bool IsIntegerType(PortableType type) => + type is TinyIntType or SmallIntType or IntType or BigIntType; + + private static bool SameSqlSequence( + IReadOnlyList<string> actual, + IReadOnlyList<string> expected + ) => + actual.Count == expected.Count + && actual.Zip(expected).All(pair => SameSql(pair.First, pair.Second)); + + private static bool SameSql(string? actual, string? expected) => + string.Equals(NormalizeSql(actual), NormalizeSql(expected), StringComparison.Ordinal); + + private static string NormalizeSql(string? value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return string.Empty; + } + var builder = new StringBuilder(); + AppendNormalizedSql(value: value, builder: builder); + return builder.ToString().Trim().TrimEnd(';').Trim(); + } + + private static void AppendNormalizedSql(string value, StringBuilder builder) + { + var pendingWhitespace = false; + foreach (var ch in value.Trim()) + { + if (char.IsWhiteSpace(ch)) + { + pendingWhitespace = builder.Length > 0; + continue; + } + AppendPendingWhitespace(builder: builder, pendingWhitespace: pendingWhitespace); + builder.Append(char.ToLowerInvariant(ch)); + pendingWhitespace = false; + } + } + + private static void AppendPendingWhitespace(StringBuilder builder, bool pendingWhitespace) + { + if (pendingWhitespace) + { + builder.Append(' '); + } + } + + private static void AddIf( + bool condition, + string message, + ImmutableArray<string>.Builder mismatches + ) + { + if (condition) + { + mismatches.Add(message); + } + } + + private static string TablePath(TableDefinition table) => $"{table.Schema}.{table.Name}"; + + private static string FunctionPath(PostgresFunctionDefinition function) => + $"{function.Schema}.{function.Name}({Format(function.Arguments.Select(a => a.Type).ToArray())})"; + + private static string Nullability(ColumnDefinition column) => + column.IsNullable ? "NULL" : "NOT NULL"; + + private static string Format(IReadOnlyList<string> values) => string.Join(", ", values); + + private static string UniqueConstraintName( + string tableName, + UniqueConstraintDefinition unique + ) => + string.IsNullOrWhiteSpace(unique.Name) + ? $"UQ_{tableName}_{string.Join("_", unique.Columns)}" + : unique.Name; + + private static string ForeignKeyName(string tableName, ForeignKeyDefinition foreignKey) => + string.IsNullOrWhiteSpace(foreignKey.Name) + ? $"FK_{tableName}_{string.Join("_", foreignKey.Columns)}" + : foreignKey.Name; +} diff --git a/Migration/Nimblesite.DataProvider.Migration.Tests/DataProviderMigrateIntegrityTests.cs b/Migration/Nimblesite.DataProvider.Migration.Tests/DataProviderMigrateIntegrityTests.cs new file mode 100644 index 00000000..0609eb92 --- /dev/null +++ b/Migration/Nimblesite.DataProvider.Migration.Tests/DataProviderMigrateIntegrityTests.cs @@ -0,0 +1,261 @@ +using System.Globalization; + +namespace Nimblesite.DataProvider.Migration.Tests; + +/// <summary> +/// CLI integrity tests for DataProviderMigrate. +/// </summary> +[Collection(PostgresTestSuite.Name)] +[System.Diagnostics.CodeAnalysis.SuppressMessage( + "Usage", + "CA1001:Types that own disposable fields should be disposable", + Justification = "Disposed via IAsyncLifetime.DisposeAsync" +)] +public sealed class DataProviderMigrateIntegrityTests(PostgresContainerFixture fixture) + : IAsyncLifetime +{ + private NpgsqlConnection _connection = null!; + private string _connectionString = string.Empty; + private readonly ILogger _logger = NullLogger.Instance; + + public async Task InitializeAsync() + { + var fixtureConnectionString = await fixture + .CreateDatabaseConnectionStringAsync(namePrefix: "migrate_integrity_test") + .ConfigureAwait(continueOnCapturedContext: false); + _connectionString = new NpgsqlConnectionStringBuilder(fixtureConnectionString) + { + Pooling = false, + }.ConnectionString; + _connection = new NpgsqlConnection(connectionString: _connectionString); + await _connection.OpenAsync().ConfigureAwait(continueOnCapturedContext: false); + } + + public async Task DisposeAsync() + { + await _connection.DisposeAsync().ConfigureAwait(continueOnCapturedContext: false); + } + + [Fact] + public void Migrate_PostApplyIntegrityCheckFailsWhenColumnNullabilityDrifts() + { + var baseline = CreateAgentConfigsSchema(nameIsNullable: true); + ApplySchema(schema: baseline); + + var schemaPath = WriteTempSchemaFile(contents: DesiredAgentConfigsYaml()); + + try + { + var result = RunMigrate(schemaPath: schemaPath); + + Assert.Equal(expected: 1, actual: result.ExitCode); + Assert.True( + condition: result.Output.Contains( + value: "SCHEMA INTEGRITY CHECK FAILED", + comparisonType: StringComparison.Ordinal + ), + userMessage: result.Output + ); + Assert.True( + condition: result.Output.Contains( + value: "public.agent_configs.name: nullability expected NOT NULL but found NULL", + comparisonType: StringComparison.Ordinal + ), + userMessage: result.Output + ); + } + finally + { + File.Delete(path: schemaPath); + } + } + + [Fact] + public void Migrate_AddsDeclaredCompositeUniqueConstraintToExistingPostgresTable() + { + var baseline = CreateAgentConfigsSchema(nameIsNullable: false); + ApplySchema(schema: baseline); + + var schemaPath = WriteTempSchemaFile(contents: DesiredAgentConfigsWithUniqueYaml()); + + try + { + var first = RunMigrate(schemaPath: schemaPath); + + Assert.Equal(expected: 0, actual: first.ExitCode); + Assert.Contains( + expectedSubstring: "AddUniqueConstraintOperation", + actualString: first.Output, + comparisonType: StringComparison.Ordinal + ); + AssertUniqueConstraintExists(); + + var second = RunMigrate(schemaPath: schemaPath); + + Assert.Equal(expected: 0, actual: second.ExitCode); + Assert.Contains( + expectedSubstring: "Schema is up to date", + actualString: second.Output, + comparisonType: StringComparison.Ordinal + ); + } + finally + { + File.Delete(path: schemaPath); + } + } + + private static SchemaDefinition CreateAgentConfigsSchema(bool nameIsNullable) => + Schema + .Define(name: "nap") + .Table( + schema: "public", + name: "agent_configs", + configure: t => + t.Column( + name: "id", + type: PortableTypes.Uuid, + configure: c => + c.PrimaryKey().Default(defaultValue: "gen_random_uuid()") + ) + .Column( + name: "tenant_id", + type: PortableTypes.Uuid, + configure: c => c.NotNull() + ) + .Column( + name: "name", + type: PortableTypes.Text, + configure: c => + ConfigureNameColumn(builder: c, nameIsNullable: nameIsNullable) + ) + ) + .Build(); + + private static string DesiredAgentConfigsYaml() => + """ + name: nap + tables: + - name: agent_configs + schema: public + columns: + - name: id + type: Uuid + isNullable: false + - name: tenant_id + type: Uuid + isNullable: false + - name: name + type: Text + isNullable: false + primaryKey: + columns: + - id + """; + + private static string DesiredAgentConfigsWithUniqueYaml() => + """ + name: nap + tables: + - name: agent_configs + schema: public + columns: + - name: id + type: Uuid + isNullable: false + - name: tenant_id + type: Uuid + isNullable: false + - name: name + type: Text + isNullable: false + primaryKey: + columns: + - id + uniqueConstraints: + - name: uq_agent_configs_tenant_name + columns: + - tenant_id + - name + """; + + private static void ConfigureNameColumn(ColumnBuilder builder, bool nameIsNullable) + { + if (!nameIsNullable) + { + builder.NotNull(); + } + } + + private void ApplySchema(SchemaDefinition schema) + { + var current = ((SchemaResultOk)PostgresSchemaInspector.Inspect(_connection)).Value; + var operations = ( + (OperationsResultOk)SchemaDiff.Calculate(current: current, desired: schema) + ).Value; + var result = MigrationRunner.Apply( + connection: _connection, + operations: operations, + generateDdl: PostgresDdlGenerator.Generate, + options: MigrationOptions.Default, + logger: _logger + ); + + Assert.True( + condition: result is MigrationApplyResultOk, + userMessage: $"Baseline migration failed: {(result as MigrationApplyResultError)?.Value}" + ); + } + + private (int ExitCode, string Output) RunMigrate(string schemaPath) + { + var originalOut = Console.Out; + using var output = new StringWriter(CultureInfo.InvariantCulture); + Console.SetOut(output); + try + { + var exitCode = DataProviderMigrate.Program.Main( + args: + [ + "migrate", + "--schema", + schemaPath, + "--provider", + "postgres", + "--output", + _connectionString, + ] + ); + return (exitCode, output.ToString()); + } + finally + { + Console.SetOut(originalOut); + } + } + + private static string WriteTempSchemaFile(string contents) + { + var schemaPath = Path.Combine( + Path.GetTempPath(), + string.Create( + CultureInfo.InvariantCulture, + $"dataprovider-integrity-{Guid.NewGuid():N}.yaml" + ) + ); + File.WriteAllText(path: schemaPath, contents: contents); + return schemaPath; + } + + private void AssertUniqueConstraintExists() + { + var schema = ((SchemaResultOk)PostgresSchemaInspector.Inspect(_connection)).Value; + var table = Assert.Single(schema.Tables, t => t.Name == "agent_configs"); + var unique = Assert.Single( + table.UniqueConstraints, + uc => uc.Name == "uq_agent_configs_tenant_name" + ); + + Assert.Equal(expected: ["tenant_id", "name"], actual: unique.Columns); + } +} diff --git a/Migration/Nimblesite.DataProvider.Migration.Tests/DataProviderMigrateRlsUnsupportedTests.cs b/Migration/Nimblesite.DataProvider.Migration.Tests/DataProviderMigrateRlsUnsupportedTests.cs new file mode 100644 index 00000000..92e8edcc --- /dev/null +++ b/Migration/Nimblesite.DataProvider.Migration.Tests/DataProviderMigrateRlsUnsupportedTests.cs @@ -0,0 +1,102 @@ +using System.Globalization; + +namespace Nimblesite.DataProvider.Migration.Tests; + +// Tests [RLS-MSSQL] SQL Server package absence guard from docs/specs/rls-spec.md. + +/// <summary> +/// CLI tests for unsupported SQL Server RLS migration attempts. +/// </summary> +public sealed class DataProviderMigrateRlsUnsupportedTests +{ + [Fact] + public void Migrate_SqlServerProviderWithRlsSchema_ReturnsCanonicalUnsupportedError() + { + var schemaPath = WriteTempSchemaFile(contents: RlsSchemaYaml()); + + try + { + var result = RunMigrate(schemaPath: schemaPath); + + Assert.Equal(expected: 1, actual: result.ExitCode); + Assert.Contains( + expectedSubstring: "MIG-E-RLS-MSSQL-UNSUPPORTED", + actualString: result.Output, + comparisonType: StringComparison.Ordinal + ); + Assert.Contains( + expectedSubstring: "Nimblesite.DataProvider.Migration.SqlServer package does not exist", + actualString: result.Output, + comparisonType: StringComparison.Ordinal + ); + } + finally + { + File.Delete(path: schemaPath); + } + } + + private static (int ExitCode, string Output) RunMigrate(string schemaPath) + { + var originalOut = Console.Out; + using var output = new StringWriter(CultureInfo.InvariantCulture); + Console.SetOut(output); + try + { + var exitCode = DataProviderMigrate.Program.Main( + args: + [ + "migrate", + "--schema", + schemaPath, + "--provider", + "sqlserver", + "--output", + "unused", + ] + ); + return (exitCode, output.ToString()); + } + finally + { + Console.SetOut(originalOut); + } + } + + private static string WriteTempSchemaFile(string contents) + { + var schemaPath = Path.Combine( + Path.GetTempPath(), + string.Create( + CultureInfo.InvariantCulture, + $"dataprovider-rls-unsupported-{Guid.NewGuid():N}.yaml" + ) + ); + File.WriteAllText(path: schemaPath, contents: contents); + return schemaPath; + } + + private static string RlsSchemaYaml() => + """ + name: sqlserver_rls_guard + tables: + - name: documents + schema: public + columns: + - name: id + type: Uuid + isNullable: false + - name: owner_id + type: Uuid + isNullable: false + primaryKey: + columns: + - id + rowLevelSecurity: + policies: + - name: owner_isolation + operations: + - Select + using: owner_id = current_user_id() + """; +} diff --git a/Migration/Nimblesite.DataProvider.Migration.Tests/MigrationRunnerHardFailTests.cs b/Migration/Nimblesite.DataProvider.Migration.Tests/MigrationRunnerHardFailTests.cs new file mode 100644 index 00000000..69ff54e7 --- /dev/null +++ b/Migration/Nimblesite.DataProvider.Migration.Tests/MigrationRunnerHardFailTests.cs @@ -0,0 +1,93 @@ +namespace Nimblesite.DataProvider.Migration.Tests; + +/// <summary> +/// Implements [MIG-RUNNER-HARD-FAIL]: when any operation fails, MigrationRunner.Apply +/// must return an Error result, even when ContinueOnError=true. The purpose of +/// ContinueOnError is to keep applying remaining operations after a failure for +/// diagnostic visibility; it must NEVER cause the runner to claim success when +/// operations were missed. Closes the spirit of issues #53 and #55: the migrator +/// must hard-fail when schema migrations are missed. +/// </summary> +public sealed class MigrationRunnerHardFailTests +{ + [Fact] + public void Apply_WithContinueOnError_FailedOperationCausesErrorResult() + { + var dbPath = Path.Combine(Path.GetTempPath(), $"hardfail_{Guid.NewGuid()}.db"); + using var connection = new SqliteConnection($"Data Source={dbPath}"); + connection.Open(); + + try + { + var goodOp = new CreateTableOperation( + new TableDefinition + { + Schema = "main", + Name = "good_table", + Columns = + [ + new ColumnDefinition + { + Name = "id", + Type = PortableTypes.BigInt, + IsNullable = false, + }, + ], + PrimaryKey = new PrimaryKeyDefinition { Columns = ["id"] }, + } + ); + + var brokenOp = new CreateTableOperation( + new TableDefinition + { + Schema = "main", + Name = "broken_table", + Columns = + [ + new ColumnDefinition + { + Name = "id", + Type = PortableTypes.BigInt, + IsNullable = false, + }, + ], + PrimaryKey = new PrimaryKeyDefinition { Columns = ["id"] }, + } + ); + + string GenerateDdl(SchemaOperation op) => + op == brokenOp + ? "INSERT INTO no_such_table (col) VALUES ('forced failure')" + : SqliteDdlGenerator.Generate(op); + + var result = MigrationRunner.Apply( + connection: connection, + operations: [goodOp, brokenOp], + generateDdl: GenerateDdl, + options: new MigrationOptions { ContinueOnError = true, UseTransaction = false } + ); + + Assert.True( + condition: result is MigrationApplyResultError, + userMessage: "Apply must return Error when any operation fails, even with " + + "ContinueOnError=true. The runner reported success while a migration " + + "was missed." + ); + } + finally + { + connection.Close(); + if (File.Exists(dbPath)) + { + try + { + File.Delete(dbPath); + } + catch + { + /* file may be locked */ + } + } + } + } +} diff --git a/Migration/Nimblesite.DataProvider.Migration.Tests/Nimblesite.DataProvider.Migration.Tests.csproj b/Migration/Nimblesite.DataProvider.Migration.Tests/Nimblesite.DataProvider.Migration.Tests.csproj index 5f0576ed..05d5d035 100644 --- a/Migration/Nimblesite.DataProvider.Migration.Tests/Nimblesite.DataProvider.Migration.Tests.csproj +++ b/Migration/Nimblesite.DataProvider.Migration.Tests/Nimblesite.DataProvider.Migration.Tests.csproj @@ -25,9 +25,11 @@ </ItemGroup> <ItemGroup> + <ProjectReference Include="../DataProviderMigrate/DataProviderMigrate.csproj" /> <ProjectReference Include="../Nimblesite.DataProvider.Migration.Core/Nimblesite.DataProvider.Migration.Core.csproj" /> <ProjectReference Include="../Nimblesite.DataProvider.Migration.SQLite/Nimblesite.DataProvider.Migration.SQLite.csproj" /> <ProjectReference Include="../Nimblesite.DataProvider.Migration.Postgres/Nimblesite.DataProvider.Migration.Postgres.csproj" /> + <ProjectReference Include="../DataProviderMigrate/DataProviderMigrate.csproj" /> </ItemGroup> <ItemGroup> diff --git a/Migration/Nimblesite.DataProvider.Migration.Tests/SchemaIntegrityVerifierFunctionDriftTests.cs b/Migration/Nimblesite.DataProvider.Migration.Tests/SchemaIntegrityVerifierFunctionDriftTests.cs new file mode 100644 index 00000000..1751ae24 --- /dev/null +++ b/Migration/Nimblesite.DataProvider.Migration.Tests/SchemaIntegrityVerifierFunctionDriftTests.cs @@ -0,0 +1,89 @@ +using System.Collections.Immutable; +using SchemaIntegrityOk = Outcome.Result< + System.Collections.Immutable.ImmutableArray<string>, + Nimblesite.DataProvider.Migration.Core.MigrationError +>.Ok< + System.Collections.Immutable.ImmutableArray<string>, + Nimblesite.DataProvider.Migration.Core.MigrationError +>; + +namespace Nimblesite.DataProvider.Migration.Tests; + +public sealed class SchemaIntegrityVerifierFunctionDriftTests +{ + [Fact] + public void Verify_WhenFunctionDefinitionDrifts_ReturnsDetailedMismatches() + { + var live = new SchemaDefinition + { + Name = "live", + Functions = + [ + new PostgresFunctionDefinition + { + Schema = "public", + Name = "current_tenant_id", + Arguments = [new PostgresFunctionArgumentDefinition { Type = "text" }], + Returns = "text", + Language = "plpgsql", + Volatility = "volatile", + SecurityDefiner = false, + Body = "select null::text", + }, + ], + }; + var desired = new SchemaDefinition + { + Name = "desired", + Functions = + [ + new PostgresFunctionDefinition + { + Schema = "public", + Name = "current_tenant_id", + Arguments = [new PostgresFunctionArgumentDefinition { Type = "text" }], + Returns = "uuid", + Language = "sql", + Volatility = "stable", + SecurityDefiner = true, + Body = "select current_setting('app.tenant_id')::uuid", + }, + ], + }; + + var mismatches = Verify(live: live, desired: desired); + + Assert.Contains( + "public.current_tenant_id(text): returns expected uuid but found text", + mismatches + ); + Assert.Contains( + "public.current_tenant_id(text): language expected sql but found plpgsql", + mismatches + ); + Assert.Contains( + "public.current_tenant_id(text): volatility expected stable but found volatile", + mismatches + ); + Assert.Contains( + "public.current_tenant_id(text): security definer expected True but found False", + mismatches + ); + Assert.Contains("public.current_tenant_id(text): function body drifted", mismatches); + } + + private static ImmutableArray<string> Verify(SchemaDefinition live, SchemaDefinition desired) + { + var result = SchemaIntegrityVerifier.Verify( + live: live, + desired: desired, + logger: NullLogger.Instance + ); + + Assert.True( + result is SchemaIntegrityOk, + "Expected schema integrity verification to succeed." + ); + return ((SchemaIntegrityOk)result).Value; + } +} diff --git a/Migration/Nimblesite.DataProvider.Migration.Tests/SchemaIntegrityVerifierTests.cs b/Migration/Nimblesite.DataProvider.Migration.Tests/SchemaIntegrityVerifierTests.cs new file mode 100644 index 00000000..d3e56b21 --- /dev/null +++ b/Migration/Nimblesite.DataProvider.Migration.Tests/SchemaIntegrityVerifierTests.cs @@ -0,0 +1,530 @@ +using System.Collections.Immutable; +using SchemaIntegrityError = Outcome.Result< + System.Collections.Immutable.ImmutableArray<string>, + Nimblesite.DataProvider.Migration.Core.MigrationError +>.Error< + System.Collections.Immutable.ImmutableArray<string>, + Nimblesite.DataProvider.Migration.Core.MigrationError +>; +using SchemaIntegrityOk = Outcome.Result< + System.Collections.Immutable.ImmutableArray<string>, + Nimblesite.DataProvider.Migration.Core.MigrationError +>.Ok< + System.Collections.Immutable.ImmutableArray<string>, + Nimblesite.DataProvider.Migration.Core.MigrationError +>; + +namespace Nimblesite.DataProvider.Migration.Tests; + +public sealed class SchemaIntegrityVerifierTests +{ + [Fact] + public void Verify_EquivalentDefaultSchemasAndSqliteIntegerWidths_ReturnsNoMismatches() + { + var live = new SchemaDefinition + { + Name = "live", + Tables = + [ + new TableDefinition + { + Schema = "main", + Name = "items", + Columns = + [ + new ColumnDefinition + { + Name = "id", + Type = PortableTypes.BigInt, + IsNullable = false, + DefaultValue = "CURRENT_TIMESTAMP ;", + }, + ], + PrimaryKey = new PrimaryKeyDefinition { Columns = ["id"] }, + Indexes = + [ + new IndexDefinition + { + Name = "idx_items_id", + Columns = ["id"], + Filter = "id IS NOT NULL", + }, + ], + }, + ], + }; + var desired = new SchemaDefinition + { + Name = "desired", + Tables = + [ + new TableDefinition + { + Schema = "public", + Name = "items", + Columns = + [ + new ColumnDefinition + { + Name = "id", + Type = PortableTypes.Int, + IsNullable = false, + DefaultValue = " current_timestamp", + }, + ], + PrimaryKey = new PrimaryKeyDefinition { Columns = ["ID"] }, + Indexes = + [ + new IndexDefinition + { + Name = "IDX_ITEMS_ID", + Columns = ["ID"], + Filter = "ID is not null;", + }, + ], + }, + ], + }; + + var mismatches = Verify(live: live, desired: desired); + + Assert.Empty(mismatches); + } + + [Fact] + public void Verify_WhenColumnsAndConstraintsDrift_ReturnsDetailedMismatches() + { + var live = new SchemaDefinition + { + Name = "live", + Tables = + [ + new TableDefinition + { + Schema = "public", + Name = "accounts", + Columns = + [ + new ColumnDefinition + { + Name = "id", + Type = PortableTypes.Int, + IsNullable = true, + IsIdentity = false, + }, + new ColumnDefinition { Name = "tenant_id", Type = PortableTypes.Uuid }, + new ColumnDefinition + { + Name = "code", + Type = PortableTypes.VarChar(64), + DefaultValue = "'old'", + }, + new ColumnDefinition { Name = "status", Type = PortableTypes.VarChar(20) }, + ], + PrimaryKey = new PrimaryKeyDefinition { Columns = ["tenant_id"] }, + ForeignKeys = + [ + new ForeignKeyDefinition + { + Name = "fk_accounts_tenant", + Columns = ["id"], + ReferencedTable = "organizations", + ReferencedColumns = ["tenant_key"], + OnDelete = ForeignKeyAction.NoAction, + OnUpdate = ForeignKeyAction.Cascade, + }, + ], + UniqueConstraints = + [ + new UniqueConstraintDefinition + { + Name = "uq_accounts_tenant_code", + Columns = ["code", "tenant_id"], + }, + ], + Indexes = + [ + new IndexDefinition + { + Name = "idx_accounts_code", + Columns = ["tenant_id"], + Expressions = ["lower(old_code)"], + Filter = "code is not null", + }, + ], + CheckConstraints = + [ + new CheckConstraintDefinition + { + Name = "ck_accounts_code", + Expression = "length(code) > 1", + }, + ], + }, + ], + }; + var desired = new SchemaDefinition + { + Name = "desired", + Tables = + [ + new TableDefinition { Schema = "public", Name = "missing_table" }, + new TableDefinition + { + Schema = "public", + Name = "accounts", + Columns = + [ + new ColumnDefinition + { + Name = "id", + Type = PortableTypes.Uuid, + IsNullable = false, + IsIdentity = true, + DefaultLqlExpression = "gen_uuid()", + }, + new ColumnDefinition + { + Name = "code", + Type = PortableTypes.VarChar(64), + DefaultValue = "'new'", + }, + new ColumnDefinition + { + Name = "status", + Type = PortableTypes.VarChar(20), + CheckConstraint = "status in ('active')", + }, + new ColumnDefinition { Name = "missing", Type = PortableTypes.Text }, + ], + PrimaryKey = new PrimaryKeyDefinition { Columns = ["id"] }, + ForeignKeys = + [ + new ForeignKeyDefinition + { + Name = "fk_accounts_tenant", + Columns = ["tenant_id"], + ReferencedTable = "tenants", + ReferencedColumns = ["id"], + OnDelete = ForeignKeyAction.Cascade, + OnUpdate = ForeignKeyAction.NoAction, + }, + new ForeignKeyDefinition + { + Columns = ["missing_fk"], + ReferencedTable = "tenants", + ReferencedColumns = ["id"], + }, + ], + UniqueConstraints = + [ + new UniqueConstraintDefinition + { + Name = "uq_accounts_tenant_code", + Columns = ["tenant_id", "code"], + }, + new UniqueConstraintDefinition { Columns = ["status"] }, + ], + Indexes = + [ + new IndexDefinition + { + Name = "idx_accounts_code", + Columns = ["code"], + Expressions = ["lower(code)"], + IsUnique = true, + Filter = "code <> ''", + }, + new IndexDefinition { Name = "idx_accounts_missing", Columns = ["status"] }, + ], + CheckConstraints = + [ + new CheckConstraintDefinition + { + Name = "ck_accounts_code", + Expression = "length(code) > 3", + }, + new CheckConstraintDefinition + { + Name = "ck_accounts_missing", + Expression = "status is not null", + }, + ], + }, + ], + }; + + var mismatches = Verify(live: live, desired: desired); + + Assert.Contains("public.missing_table: missing table", mismatches); + Assert.Contains( + $"public.accounts.id: type expected {PortableTypes.Uuid} but found {PortableTypes.Int}", + mismatches + ); + Assert.Contains( + "public.accounts.id: nullability expected NOT NULL but found NULL", + mismatches + ); + Assert.Contains("public.accounts.id: identity expected True but found False", mismatches); + Assert.Contains( + "public.accounts.id: default expected gen_uuid() but found <none>", + mismatches + ); + Assert.Contains("public.accounts.code: default expected 'new' but found 'old'", mismatches); + Assert.Contains("public.accounts.missing: missing column", mismatches); + Assert.Contains( + "public.accounts: primary key columns expected (id) but found (tenant_id)", + mismatches + ); + Assert.Contains( + "public.accounts: foreign key fk_accounts_tenant columns drifted", + mismatches + ); + Assert.Contains( + "public.accounts: foreign key fk_accounts_tenant referenced table drifted", + mismatches + ); + Assert.Contains( + "public.accounts: foreign key fk_accounts_tenant referenced columns drifted", + mismatches + ); + Assert.Contains( + "public.accounts: foreign key fk_accounts_tenant on delete expected Cascade but found NoAction", + mismatches + ); + Assert.Contains( + "public.accounts: foreign key fk_accounts_tenant on update expected NoAction but found Cascade", + mismatches + ); + Assert.Contains( + "public.accounts: missing foreign key FK_accounts_missing_fk on (missing_fk)", + mismatches + ); + Assert.Contains( + "public.accounts: unique constraint uq_accounts_tenant_code columns expected (tenant_id, code) but found (code, tenant_id)", + mismatches + ); + Assert.Contains( + "public.accounts: missing unique constraint UQ_accounts_status on (status)", + mismatches + ); + Assert.Contains("public.accounts: index idx_accounts_code uniqueness drifted", mismatches); + Assert.Contains("public.accounts: index idx_accounts_code columns drifted", mismatches); + Assert.Contains("public.accounts: index idx_accounts_code expressions drifted", mismatches); + Assert.Contains("public.accounts: index idx_accounts_code filter drifted", mismatches); + Assert.Contains("public.accounts: missing index idx_accounts_missing", mismatches); + Assert.Contains( + "public.accounts: check constraint ck_accounts_code expression drifted", + mismatches + ); + Assert.Contains( + "public.accounts: missing check constraint ck_accounts_missing", + mismatches + ); + Assert.Contains("public.accounts.status: missing check constraint", mismatches); + } + + [Fact] + public void Verify_WhenPrimaryKeyIsMissing_ReturnsMissingPrimaryKey() + { + var live = Schema + .Define("live") + .Table("public", "accounts", table => table.Column("id", PortableTypes.Uuid)) + .Build(); + var desired = Schema + .Define("desired") + .Table( + "public", + "accounts", + table => table.Column("id", PortableTypes.Uuid, column => column.PrimaryKey()) + ) + .Build(); + + var mismatches = Verify(live: live, desired: desired); + + Assert.Contains("public.accounts: missing primary key", mismatches); + } + + [Fact] + public void Verify_WhenRlsAndSupportObjectsDrift_ReturnsDetailedMismatches() + { + var live = new SchemaDefinition + { + Name = "live", + Tables = + [ + new TableDefinition + { + Schema = "public", + Name = "documents", + RowLevelSecurity = new RlsPolicySetDefinition + { + Enabled = false, + Forced = false, + Policies = [new RlsPolicyDefinition { Name = "tenant_read_old" }], + }, + }, + ], + Roles = + [ + new PostgresRoleDefinition + { + Name = "app_user", + Login = false, + BypassRls = true, + }, + ], + }; + var desired = new SchemaDefinition + { + Name = "desired", + Tables = + [ + new TableDefinition + { + Schema = "public", + Name = "documents", + RowLevelSecurity = new RlsPolicySetDefinition + { + Enabled = true, + Forced = true, + Policies = [new RlsPolicyDefinition { Name = "tenant_read" }], + }, + }, + ], + Roles = + [ + new PostgresRoleDefinition + { + Name = "app_user", + Login = true, + BypassRls = false, + }, + new PostgresRoleDefinition { Name = "missing_role" }, + ], + Functions = + [ + new PostgresFunctionDefinition + { + Schema = "public", + Name = "current_tenant_id", + Arguments = [new PostgresFunctionArgumentDefinition { Type = "text" }], + }, + ], + Grants = + [ + new PostgresGrantDefinition + { + Schema = "public", + Target = PostgresGrantTarget.Table, + ObjectName = "documents", + Privileges = ["select"], + Roles = ["app_user"], + }, + ], + }; + + var mismatches = Verify(live: live, desired: desired); + + Assert.Contains( + "public.documents: row-level security expected True but found False", + mismatches + ); + Assert.Contains( + "public.documents: forced row-level security expected True but found False", + mismatches + ); + Assert.Contains( + "public.documents: missing row-level security policy tenant_read", + mismatches + ); + Assert.Contains("role app_user: login drifted", mismatches); + Assert.Contains("role app_user: bypassRls drifted", mismatches); + Assert.Contains("role missing_role: missing role", mismatches); + Assert.Contains("public.current_tenant_id: missing function", mismatches); + Assert.Contains("grant Table documents: missing grant", mismatches); + } + + [Fact] + public void Verify_WhenRlsAndSupportObjectsAreExcluded_IgnoresThoseMismatches() + { + var live = new SchemaDefinition + { + Name = "live", + Tables = [new TableDefinition { Schema = "public", Name = "documents" }], + }; + var desired = new SchemaDefinition + { + Name = "desired", + Tables = + [ + new TableDefinition + { + Schema = "public", + Name = "documents", + RowLevelSecurity = new RlsPolicySetDefinition + { + Enabled = true, + Forced = true, + Policies = [new RlsPolicyDefinition { Name = "tenant_read" }], + }, + }, + ], + Roles = [new PostgresRoleDefinition { Name = "app_user" }], + Functions = [new PostgresFunctionDefinition { Name = "current_tenant_id" }], + Grants = + [ + new PostgresGrantDefinition + { + Schema = "public", + Target = PostgresGrantTarget.Schema, + Privileges = ["usage"], + Roles = ["app_user"], + }, + ], + }; + + var mismatches = Verify( + live: live, + desired: desired, + includeSupportObjects: false, + includeRls: false + ); + + Assert.Empty(mismatches); + } + + [Fact] + public void Verify_WhenSchemaCannotBeRead_ReturnsError() + { + var result = SchemaIntegrityVerifier.Verify( + live: Schema.Define("live").Build(), + desired: new SchemaDefinition { Tables = null! }, + logger: NullLogger.Instance + ); + + Assert.True(result is SchemaIntegrityError); + } + + private static ImmutableArray<string> Verify( + SchemaDefinition live, + SchemaDefinition desired, + bool includeSupportObjects = true, + bool includeRls = true + ) + { + var result = SchemaIntegrityVerifier.Verify( + live: live, + desired: desired, + includeSupportObjects: includeSupportObjects, + includeRls: includeRls, + logger: NullLogger.Instance + ); + + Assert.True( + result is SchemaIntegrityOk, + "Expected schema integrity verification to succeed." + ); + return ((SchemaIntegrityOk)result).Value; + } +} diff --git a/Migration/README.md b/Migration/README.md index 5fd2f250..5ebb8fc6 100644 --- a/Migration/README.md +++ b/Migration/README.md @@ -202,4 +202,4 @@ Regenerate the database on every build so developers never run migrations manual - [DataProvider](../DataProvider/README.md) — generated extension methods for the tables defined here - [LQL](../Lql/README.md) — write portable queries against the migrated schema -- Migration CLI spec: [docs/specs/migration-cli-spec.md](../docs/specs/migration-cli-spec.md) +- Migration spec: [docs/specs/migration-spec.md](../docs/specs/migration-spec.md#74-dataprovidermigrate-cli-mig-cli) diff --git a/Reporting/spec.md b/Reporting/spec.md index e4efb31b..de563787 100644 --- a/Reporting/spec.md +++ b/Reporting/spec.md @@ -259,12 +259,12 @@ The renderer is a standalone JS bundle. Embed in any page: ## Database Schema -**All database schemas MUST be created using the Migration library with YAML definitions.** Raw SQL for creating database schemas is strictly prohibited. Use `Migration.Cli` with YAML schema files as the single source of truth. +**All database schemas MUST be created using the Migration library with YAML definitions.** Raw SQL for creating database schemas is strictly prohibited. Use `DataProviderMigrate` with YAML schema files as the single source of truth. If the reporting platform requires its own persistence (e.g., for saved reports, scheduled executions), the schema MUST be defined in a `reporting-schema.yaml` file and created via: ```bash -dotnet run --project Migration/Migration.Cli -- --schema reporting-schema.yaml --output reporting.db --provider sqlite +dotnet run --project Migration/DataProviderMigrate/DataProviderMigrate.csproj -- migrate --schema reporting-schema.yaml --output reporting.db --provider sqlite ``` For the MVP, report definitions are loaded from JSON files on disk (no database persistence needed). Future phases will add YAML-migrated schema for saved reports. diff --git a/coverage-thresholds.json b/coverage-thresholds.json index cdf273e2..c602ac07 100644 --- a/coverage-thresholds.json +++ b/coverage-thresholds.json @@ -18,7 +18,7 @@ "include": "[Nimblesite.Lql.TypeProvider.FSharp]*,[Nimblesite.Lql.Core]*,[Nimblesite.Lql.SQLite]*,[Nimblesite.Sql.Model]*" }, "Migration/Nimblesite.DataProvider.Migration.Core": { - "threshold": 84, + "threshold": 86, "include": "[Nimblesite.DataProvider.Migration.Core]*,[Nimblesite.DataProvider.Migration.SQLite]*,[Nimblesite.DataProvider.Migration.Postgres]*" }, "Sync/Nimblesite.Sync.Core": { @@ -50,10 +50,10 @@ "include": "[Nimblesite.Reporting.Engine]*,[Nimblesite.Reporting.Api]*" }, "Lql/lql-lsp-rust": { - "threshold": 83 + "threshold": 89 }, "Lql/LqlExtension": { - "threshold": 38 + "threshold": 44 } } } diff --git a/docs/plans/RELEASE-PLAN.md b/docs/plans/RELEASE-PLAN.md deleted file mode 100644 index 8eb38c17..00000000 --- a/docs/plans/RELEASE-PLAN.md +++ /dev/null @@ -1,138 +0,0 @@ -# DataProvider NuGet Package Release Plan - -## Goal -Release DataProvider packages to NuGet.org so apps can consume them without git submodules. - ---- - -## Decisions -- **Naming**: Libraries depending on DataProvider use `DataProvider.*` prefix; standalone libs keep simple names -- **Publish to**: NuGet.org (public) -- **Release trigger**: Git tags (`v*`) -- **Outcome**: Already on NuGet, just reference it - ---- - -## Packages - -### Libraries - -| Package | Description | Dependencies | -|---------|-------------|--------------| -| `Selecta` | Utility library | None | -| `Migration` | Schema definition/DDL | YamlDotNet | -| `Migration.SQLite` | SQLite DDL generator | Migration | -| `Migration.Postgres` | Postgres DDL generator | Migration, Npgsql | -| `DataProvider` | Core code generation | Selecta, Outcome | -| `DataProvider.SQLite` | SQLite source generator | DataProvider, Selecta, Antlr4 | - -### CLI Tools (dotnet tools) - -| Tool | Command | Install | -|------|---------|---------| -| `DataProviderMigrate` | `DataProviderMigrate` | `dotnet tool install DataProviderMigrate -g` | - -> `DataProvider.Postgres.Cli` and `DataProvider.SQLite.Cli` are **no longer the -> consumer integration point** for code generation. The source-generator path -> defined in [`../specs/codegen-source-generator.md`](../specs/codegen-source-generator.md) -> supersedes them. The CLI projects remain in the tree as standalone -> debugging utilities only and are not published to nuget.org as tools. - ---- - -## Files to Modify - -| File | Action | -|------|--------| -| `Directory.Build.props` | Add shared NuGet metadata | -| `Other/Selecta/Selecta.csproj` | Add PackageId, Description | -| `Migration/Migration/Migration.csproj` | Add PackageId, Description | -| `Migration/Migration.SQLite/Migration.SQLite.csproj` | Add PackageId, Description | -| `Migration/Migration.Postgres/Migration.Postgres.csproj` | Add PackageId, Description | -| `Migration/Migration.Cli/Migration.Cli.csproj` | Add PackAsTool, ToolCommandName | -| `DataProvider/DataProvider/DataProvider.csproj` | Add PackageId, Description | -| `DataProvider/DataProvider.SQLite/DataProvider.SQLite.csproj` | Add PackageId, Description | -| `.github/workflows/release.yml` | **CREATE** - Automated release workflow | - ---- - -## Release Workflow - -**File: `.github/workflows/release.yml`** - -```yaml -name: Release NuGet Packages - -on: - push: - tags: - - 'v*' - -env: - DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true - DOTNET_CLI_TELEMETRY_OPTOUT: true - -jobs: - publish: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - - name: Setup .NET - uses: actions/setup-dotnet@v4 - with: - dotnet-version: '9.0.x' - - - name: Extract version from tag - id: version - run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT - - - name: Restore - run: dotnet restore - - - name: Build - run: dotnet build -c Release --no-restore - - - name: Test - run: dotnet test -c Release --no-build - - - name: Pack libraries - run: | - dotnet pack Other/Selecta/Selecta.csproj -c Release -p:Version=${{ steps.version.outputs.VERSION }} -o ./nupkgs - dotnet pack Migration/Migration/Migration.csproj -c Release -p:Version=${{ steps.version.outputs.VERSION }} -o ./nupkgs - dotnet pack Migration/Migration.SQLite/Migration.SQLite.csproj -c Release -p:Version=${{ steps.version.outputs.VERSION }} -o ./nupkgs - dotnet pack Migration/Migration.Postgres/Migration.Postgres.csproj -c Release -p:Version=${{ steps.version.outputs.VERSION }} -o ./nupkgs - dotnet pack DataProvider/DataProvider/DataProvider.csproj -c Release -p:Version=${{ steps.version.outputs.VERSION }} -o ./nupkgs - dotnet pack DataProvider/DataProvider.SQLite/DataProvider.SQLite.csproj -c Release -p:Version=${{ steps.version.outputs.VERSION }} -o ./nupkgs - - - name: Pack CLI tools - run: | - dotnet pack Migration/Migration.Cli/Migration.Cli.csproj -c Release -p:Version=${{ steps.version.outputs.VERSION }} -o ./nupkgs - - - name: Push to NuGet - run: dotnet nuget push "./nupkgs/*.nupkg" --api-key ${{ secrets.NUGET_API_KEY }} --source https://api.nuget.org/v3/index.json --skip-duplicate - - - name: Create GitHub Release - uses: softprops/action-gh-release@v1 - with: - files: ./nupkgs/*.nupkg - generate_release_notes: true -``` - ---- - -## Release Process - -1. Make all .csproj changes -2. Add `NIMBLESITE_NUGET_KEY` secret to GitHub repo settings -3. Tag and push: `git tag v0.1.0 && git push origin v0.1.0` -4. GitHub Actions runs automatically: build → test → pack → push to NuGet.org - ---- - -## Versioning - -- Semantic Versioning: `MAJOR.MINOR.PATCH` -- Start at `0.1.0` -- All packages share same version (monorepo style) -- Version from git tag: `v0.1.0` → version `0.1.0` diff --git a/docs/plans/RLS-PLAN.md b/docs/plans/RLS-PLAN.md index 1134079f..6b92f2c2 100644 --- a/docs/plans/RLS-PLAN.md +++ b/docs/plans/RLS-PLAN.md @@ -85,14 +85,14 @@ Predicates that query other tables (e.g. group membership) MUST use LQL, transpi - [x] Write failing `RlsPredicateTranspiler` unit tests in new `RlsPredicateTranspilerTests.cs` - [x] Make `RlsPredicateTranspiler` tests pass - [x] Implement RLS operation handling in `PostgresDdlGenerator.cs` (Enable, Create, Drop, Disable) -- [ ] Write failing Postgres RLS E2E tests in `PostgresMigrationTests.cs` +- [x] Write failing Postgres RLS E2E tests in `PostgresRlsE2ETests.cs` - [x] Extend `PostgresSchemaInspector.cs` to read `pg_policies` into `RlsPolicySetDefinition` -- [ ] Make Postgres E2E tests pass +- [x] Make Postgres E2E tests pass - [x] Extend `SchemaDiff.Calculate` in `SchemaDiff.cs` with RLS diff logic - [x] Write failing SQLite RLS E2E tests in `SqliteRlsMigrationTests.cs` - [x] Implement `__rls_context` table, trigger generation, and `_secure` view generation in `SqliteDdlGenerator.cs` - [x] Extend `SqliteSchemaInspector.cs` to reverse-map `rls_*` triggers - [x] Make SQLite E2E tests pass -- [ ] Add `MIG-E-RLS-MSSQL-UNSUPPORTED` error guard for SQL Server -- [ ] Run `make ci` -- all tests pass, coverage thresholds maintained -- [ ] Update `Migration/README.md` with RLS usage examples +- [x] Add `MIG-E-RLS-MSSQL-UNSUPPORTED` error guard for SQL Server +- [x] Run `make ci` -- all tests pass, coverage thresholds maintained +- [x] Update `Migration/README.md` with RLS usage examples diff --git a/docs/plans/codegen-v2.md b/docs/plans/codegen-v2.md new file mode 100644 index 00000000..146a41c9 --- /dev/null +++ b/docs/plans/codegen-v2.md @@ -0,0 +1,59 @@ +# PLAN: Codegen v2 — Schema-Doc Default, Conventions, LSP/Watch, Templates + +> Implements the v2 sections of [`../specs/codegen-cli-tool.md`](../specs/codegen-cli-tool.md): `## SCHEMA-DOC`, `## CONVENTIONS`, `## TEMPLATES`, plus the mode-aware edits to `## CONSTRAINT`, `## TOOL`, `## CONSUMER`, `## DX`, `## TEST`. + +## Goal + +Ship one tool change with four user-visible features: + +1. `--mode schema-doc` is the default. No DB, no env vars, no migrations needed to compile. +2. `DataProvider.json` becomes optional. Convention resolver decides what to emit. +3. `--watch` runs the tool as a hand-rolled LSP, watching every `*.schema.yaml` across the solution and regenerating only the affected project's outputs. +4. Output is template-driven (Scriban). Default templates emit C#; users can drop a `templates/` folder to emit any language. + +Live-db mode behavior is **unchanged**. + +## Files to modify / create + +| Path | Change | +|---|---| +| [`DataProvider/Nimblesite.DataProvider.Core/DataProviderConfig.cs`](../../DataProvider/Nimblesite.DataProvider.Core/DataProviderConfig.cs) | Add `Mode { SchemaDoc, LiveDb }` enum (default `SchemaDoc`). | +| [`DataProvider/DataProvider/Program.cs`](../../DataProvider/DataProvider/Program.cs) | Parse `--mode` first; dispatch to schema-doc pipeline or live-db pipeline. Add `--watch`, `--templates` flags. | +| [`DataProvider/DataProvider/PostgresCli.cs`](../../DataProvider/DataProvider/PostgresCli.cs), `SqliteCli.cs` | Untouched on the live-db path. Connection-string check stays where it is. | +| `DataProvider/Nimblesite.DataProvider.Core/CodeGeneration/SchemaDocPipeline.cs` | NEW. Loads `*.schema.yaml` via `SchemaYamlSerializer.FromYaml`, walks tables, calls existing `SqlAntlrCodeGenerator`. | +| `DataProvider/Nimblesite.DataProvider.Core/CodeGeneration/SchemaDocColumnResolver.cs` | NEW. Replaces `IDatabaseEffects.GetColumnMetadataFromSqlAsync` for schema-doc mode. SELECT-list resolution via base-column lookup against `SchemaDefinition`. ANTLR `IParseTree` only — **no regex** per CLAUDE.md. | +| `DataProvider/Nimblesite.DataProvider.Core/CodeGeneration/NamingConvention.cs` | NEW. Lifts `ToPascalCase`/`ToCamelCase` from [`PostgresCli.cs:2200`](../../DataProvider/DataProvider/PostgresCli.cs) into Core. Adds `snake`/`kebab`/`screaming-snake`. Eliminates per-platform duplication. | +| `DataProvider/Nimblesite.DataProvider.Core/CodeGeneration/ConventionResolver.cs` | NEW. Given `DatabaseSchema` returns per-table CRUD flags per [CONV-CRUD]. `TableConfig` overrides applied on top. | +| `DataProvider/Nimblesite.DataProvider.Core/CodeGeneration/Templating/ScribanRenderer.cs` | NEW. Wraps Scriban; exposes `Func<TemplateName, ImmutableDictionary<string,object>, string>` seam (no interfaces per CLAUDE.md). Sandbox via `TemplateContext.MemberFilter` denying `System.IO`/`System.Reflection`/`System.Diagnostics.Process`/`System.Environment`. | +| `DataProvider/Nimblesite.DataProvider.Core/CodeGeneration/Templating/Templates/Csharp/*.scriban` | NEW. Embedded resources: `Model.scriban`, `Insert.scriban`, `Update.scriban`, `Delete.scriban`, `Select.scriban`, `Extensions.scriban`. Replaces the string-builder bodies in `ModelGenerator.cs` + `DataAccessGenerator.cs`. | +| `DataProvider/Nimblesite.DataProvider.Core/CodeGeneration/Templating/manifest.yaml` (embedded) | NEW. Default C# manifest: `{ id: csharp, language: csharp, templates: [...] }`. | +| [`DataProvider/Nimblesite.DataProvider.Core/CodeGeneration/ModelGenerator.cs`](../../DataProvider/Nimblesite.DataProvider.Core/CodeGeneration/) | Refactor: body becomes "render `Model.scriban`". Logic identical, mechanism switches. | +| [`DataProvider/Nimblesite.DataProvider.Core/CodeGeneration/DataAccessGenerator.cs`](../../DataProvider/Nimblesite.DataProvider.Core/CodeGeneration/) | Same: bodies → templates. | +| `DataProvider/DataProvider.Lsp/` (new project) | Hand-rolled JSON-RPC over stdio. LSP `Content-Length` framing. v1 messages per [WATCH-MSGS]. References Core, no `OmniSharp.Extensions.LanguageServer`, no `StreamJsonRpc`. | +| `DataProvider/Wire/*.td` | NEW. typeDiagram source for every wire DTO. Build step generates C# records under `DataProvider/DataProvider.Lsp/Wire/Generated/` and TypeScript types under `Lql/LqlExtension/src/dataprovider-wire/` (TS output is committed; the future VS Code DataProvider client consumes it). | +| `DataProvider/DataProvider/build/DataProvider.targets` | Add `<DataProviderMode>` default `schema-doc`; `<Error>` on missing connection scoped to `DataProviderMode == 'live-db'`; pass `--mode`, `--templates` to the exe; include `*.schema.yaml` in `AdditionalFiles`. | +| [`Directory.Build.props`](../../Directory.Build.props) | Add `Scriban` PackageReference scoped to `DataProvider.Core`. | +| `DataProvider/DataProvider.Tool.Tests/` | New tests: `SchemaDocModeTests` (no DB), `WatchTests` (LSP boot + edit + assert scoped regen), `TemplatesTests` (sample TS template + sandbox rejection), `ParityTests` (schema-doc vs live-db byte-identical for same schema). | +| [`docs/specs/codegen-cli-tool.md`](../specs/codegen-cli-tool.md) | Already edited to v2 (this plan implements those edits). | + +## Reused, not reinvented + +- [`SchemaYamlSerializer.FromYaml`](../../Migration/Nimblesite.DataProvider.Migration.Core/SchemaYamlSerializer.cs:16-31) — already loads the canonical YAML. +- [`SchemaDefinition`](../../Migration/Nimblesite.DataProvider.Migration.Core/SchemaDefinition.cs) — schema-doc mode adopts this format verbatim. **No new format.** +- [`SqlAntlrCodeGenerator`](../../DataProvider/Nimblesite.DataProvider.Core/CodeGeneration/) — same generator runs in both modes; only the column resolver differs. +- [`PostgresCli.cs:55-68`](../../DataProvider/DataProvider/PostgresCli.cs) `--offline` proof-of-concept — promoted to default. + +## Verification + +- `make ci` passes. +- `TEST-UNIT` parity test: same schema → schema-doc and live-db modes emit byte-identical `.g.cs`. +- `TEST-DIAG` covers `DPSG001–DPSG015`. +- `TEST-WATCH` boots the LSP against a two-project temp solution, edits one schema doc, asserts only that project's outputs rewrite. +- `TEST-TEMPLATES` ships a TypeScript template, runs `tsc --noEmit`, and asserts a `System.IO`-touching template is rejected with `DPSG014`. +- `dotnet build` on a fresh consumer with **only** a `*.schema.yaml` and a `<PackageReference>` succeeds — no env vars, no DB, no `DataProvider.json`. + +## Out of scope + +- VS Code DataProvider client UI (the LSP server is shipped; the VSIX consumer side is a follow-up). +- SqlServer schema-doc support beyond the existing day-one platform tier. +- Generator marketplace / distribution mechanism. diff --git a/docs/specs/codegen-cli-tool.md b/docs/specs/codegen-cli-tool.md index 06d1fea9..6ca32e58 100644 --- a/docs/specs/codegen-cli-tool.md +++ b/docs/specs/codegen-cli-tool.md @@ -10,7 +10,7 @@ Renaming any `DataProvider.*` artifact to add a prefix or suffix is grounds for ## Overview -`DataProvider` is a console exe shipped **inside** the `DataProvider` NuGet package and invoked by an auto-imported `build/DataProvider.targets` file in the same package. At codegen time it opens a real driver connection, introspects the live schema, transpiles `.lql` → platform SQL, and emits typed C# data-access code into the consumer's `obj/` directory for `csc` to compile. +`DataProvider` is a console exe shipped **inside** the `DataProvider` NuGet package and invoked by an auto-imported `build/DataProvider.targets` file in the same package. By default it runs in **`schema-doc` mode**: it loads a YAML schema doc from the project, applies convention-driven CRUD emission, transpiles `.lql` → platform SQL, and emits typed data-access code through pluggable templates (default C#, optional any language) into the consumer's `obj/` directory for compilation. In opt-in **`live-db` mode** (`--mode live-db`) it instead opens a real driver connection and introspects the live schema. With `--watch` the same exe stays resident as a hand-rolled LSP, watches `*.schema.yaml` across the solution, and regenerates only the outputs of the project whose schema changed. ## CONSTRAINT @@ -18,9 +18,11 @@ Renaming any `DataProvider.*` artifact to add a prefix or suffix is grounds for |---|---| | CON-NOTOOLMANIFEST | Zero `.config/dotnet-tools.json`, zero `dotnet tool restore` in any consumer. The tool ships inside the package and runs from the auto-imported `.targets`. | | CON-SIMPLE | One `PackageReference` + one MSBuild property per consumer csproj. No custom `<Target>`, no `<Compile Include="Generated/**">`, no `RemoveDir`/`MakeDir`/`Touch`, no `sed`. | -| CON-DBLIVE | Live driver connection + live schema introspection only. Static YAML/JSON schema parsing forbidden. Unreachable DB → loud diagnostic + build fail. | -| CON-NOFALLBACK | Zero fallbacks. Zero degraded modes. No offline cache. Build passes against a live DB or fails loud. | -| CON-MIG-FIRST | `dataprovider-migrate` runs against the live DB **before** every codegen invocation. Every flow (CI, local, IDE) runs migrations first. The migration tool stays. | +| CON-MODE | Two modes: `schema-doc` (default) and `live-db`. Mode resolved from `--mode {schema-doc\|live-db}`. `schema-doc` mode is specified in `## SCHEMA-DOC` below. `live-db` mode is what the rest of this spec describes — no behavioral change. | +| CON-NOCONFIG | `DataProvider.json` is **not required**. In `schema-doc` mode the YAML doc + conventions provide everything needed. The config file remains as overrides only. | +| CON-DBLIVE | **Live-db mode only.** Live driver connection + live schema introspection. Static YAML/JSON schema parsing forbidden in this mode. Unreachable DB → loud diagnostic + build fail. | +| CON-NOFALLBACK | **Live-db mode only.** Zero fallbacks. Zero degraded modes. No offline cache. Build passes against a live DB or fails loud. | +| CON-MIG-FIRST | **Live-db mode only.** `dataprovider-migrate` runs against the live DB **before** every codegen invocation. Every live-db flow (CI, local, IDE) runs migrations first. The migration tool stays. | | CON-UNIVERSAL | One single mechanism for **every** consumer of **every** platform. No consumer-specific names, paths, or domain models. | | CON-PLATFORM-AGNOSTIC | One single mechanism for **every** database platform. Identical contract. | | CON-PROCESS-ISOLATION | Codegen runs in its own process, separate from `dotnet build` / MSBuild / IDE. Process isolation is the only way to load any TFM driver + any native dep on any host. | @@ -80,13 +82,13 @@ To bump a grammar: |---|---| | TOOL-NAME | `DataProvider` | | TOOL-PROJECT | `DataProvider/DataProvider/`. `<OutputType>Exe</OutputType>`, `<TargetFramework>net9.0</TargetFramework>`, `<AssemblyName>DataProvider</AssemblyName>`. **No** `PackAsTool=true`. | -| TOOL-ARGS | Required: `--connection`, `--config`, `--out`, `--platform`. Optional: `--namespace` (default `<RootNamespace>.DataProvider.Generated`), `--accessibility` (default `public`), `--verbosity`. | +| TOOL-ARGS | Required: `--out`, `--platform`. `--mode {schema-doc\|live-db}` defaults to `schema-doc`. `--connection` required iff `--mode live-db`. `--config` optional in both modes (overrides only — see [CON-NOCONFIG]). `--watch` runs the tool as a long-lived LSP per `## SCHEMA-DOC`. `--templates` points at a custom template directory per `## TEMPLATES`. Optional: `--namespace` (default `<RootNamespace>.DataProvider.Generated`), `--accessibility` (default `public`), `--verbosity`. | | TOOL-DEPS | References `DataProvider.Core`, every `DataProvider.{Platform}`, every `Nimblesite.Lql.{Platform}`, `Nimblesite.Sql.Model`, `Outcome`, plus every database driver. **All TFMs are net9.0**. No netstandard2.0. | | TOOL-PROCESS | Own process per invocation. Process isolation guarantees: any native driver dep loads cleanly; tool crashes do not corrupt MSBuild build-server state; tool memory fully reclaimed at exit; tool dll never locks files in IDE edit-rebuild cycles. | | TOOL-EXIT | `0` on success, `1` on any error. Errors → **stderr** in MSBuild error format `path/to/file(line,col): error DPSGxxx: message` so `<Exec>` parses them into structured IDE errors. Diagnostics → stdout. | -| TOOL-PIPELINE | Per invocation: (1) validate `--connection` → `DPSG001`; (2) parse `--config` JSON → `DPSG006`; (3) open the platform driver per [TOOL-DBOPEN]; (4) introspect only the explicit `(schema, table)` pairs in `DataProvider.json` (never auto-`pg_catalog`/`information_schema`/`sys`); (5) for each `*.lql`, transpile `LqlStatementConverter.ToStatement(content).To{Platform}Sql()`; (6) feed SQL + `DatabaseSchema` into `DataAccessGenerator.Generate(...)`; (7) write `.g.cs` files to `--out`; (8) close + exit. | +| TOOL-PIPELINE | Per invocation: (1) resolve `--mode`. (2a) **schema-doc**: locate `*.schema.yaml` in the project (recursive; one schema doc per project per [DOC-ONE-PER-PROJECT]) → `DPSG010`; load via `SchemaSerializer.FromYaml` → `DPSG011`; skip [TOOL-DBOPEN] entirely. (2b) **live-db**: validate `--connection` → `DPSG001`; open driver per [TOOL-DBOPEN]; introspect schema. (3) parse `--config` JSON if present → `DPSG006` (optional per [CON-NOCONFIG]); apply convention resolver per [CONV-CRUD]. (4) for each `*.lql`, transpile `LqlStatementConverter.ToStatement(content).To{Platform}Sql()`. (5) render templates per `## TEMPLATES`. (6) write generated files to `--out`. (7) if `--watch`, run the LSP loop per `## SCHEMA-DOC` instead of exiting. | | TOOL-DBOPEN | Synchronous open. Append `Pooling=false;Connect Timeout=5;Command Timeout=10` if absent. On any driver-level connection exception: catch once, emit `DPSG002`, exit `1`. Connection-string sanitised through the platform's connection-string builder before any diagnostic — host:port:database equivalent only, never password / auth. | -| TOOL-EMIT | Per-table `DataProvider.{schema}.{table}.g.cs`; per-LQL `DataProvider.lql.{slug}.g.cs`; per-SQL `DataProvider.sql.{slug}.g.cs`; aggregate `DataProvider.Extensions.g.cs`. Slug rules: replace `[/\\.]` → `_`, lowercase, non-ASCII → punycode. Collisions → `DPSG007`. Every generated file carries `[GeneratedCodeAttribute("DataProvider", "{version}")]` with **LF line endings**. | +| TOOL-EMIT | File names + extensions are **template-driven** per `## TEMPLATES`. Default C# manifest: per-table `DataProvider.{schema}.{table}.g.cs`; per-LQL `DataProvider.lql.{slug}.g.cs`; per-SQL `DataProvider.sql.{slug}.g.cs`; aggregate `DataProvider.Extensions.g.cs`. Custom template manifests may emit any extension (`.ts`, `.kt`, `.swift`, etc.). Slug rules: replace `[/\\.]` → `_`, lowercase, non-ASCII → punycode. Collisions → `DPSG007`. Every generated C# file carries `[GeneratedCodeAttribute("DataProvider", "{version}")]` with **LF line endings**. | | TOOL-NAMESPACE | Default `<RootNamespace>.DataProvider.Generated`. Override via `<DataProviderGeneratedNamespace>` → `--namespace`. Default accessibility `public`. Override via `<DataProviderGeneratedAccessibility>` → `--accessibility`. | | TOOL-ADHOC | Same exe is invocable directly: `dotnet path/to/DataProvider.dll --connection "..." --config DataProvider.json --out /tmp/dp --verbosity diagnostic --platform Postgres`. Same binary, same args. No separate debug tool. | @@ -101,6 +103,52 @@ To bump a grammar: | DPSG005 | Error | LQL → platform SQL transpile error | | DPSG006 | Error | `DataProvider.json` parse / schema validation error | | DPSG007 | Error | Generated source emit collision | +| DPSG010 | Error | Schema-doc mode: no `*.schema.yaml` found in project tree | +| DPSG011 | Error | Schema-doc mode: YAML parse / `SchemaDefinition` validation failed | +| DPSG012 | Error | Schema-doc mode: more than one `*.schema.yaml` in a single project (per [DOC-ONE-PER-PROJECT]) | +| DPSG013 | Warning | Convention resolver skipped CRUD for a table (no PK / composite PK / unmapped column type) | +| DPSG014 | Error | Template render failure (path, line, message) | +| DPSG015 | Error | Template manifest invalid | + +## SCHEMA-DOC + +| ID | Spec | +|---|---| +| DOC-DEFAULT | `--mode schema-doc` is the default. Tool never opens a DB in this mode. | +| DOC-FORMAT | Schema doc YAML = the existing `SchemaDefinition` shape used by Migration. Loader = `Nimblesite.DataProvider.Migration.Core.SchemaYamlSerializer.FromYaml`. **No new format.** Anything missing is added to `SchemaDefinition`, not forked. | +| DOC-DISCOVER | Recursive glob `**/*.schema.yaml` rooted at the project directory. Symlinks not followed. | +| DOC-ONE-PER-PROJECT | Exactly one `*.schema.yaml` per project. Zero → `DPSG010`. Two or more → `DPSG012`. | +| DOC-COLINFER | SELECT-list column types resolve via base-column lookup against the loaded `SchemaDefinition`. Expression columns require `AS alias::PortableType` annotation in the SQL or a `<query>.columns.yaml` sidecar. Unresolvable → `DPSG013`. | +| WATCH-MODE | `--watch` runs the same exe as a long-lived process. Hand-rolled JSON-RPC over stdio with LSP `Content-Length` framing. **No `OmniSharp.Extensions.LanguageServer`, no `StreamJsonRpc`.** | +| WATCH-FS | Recursive `FileSystemWatcher` rooted at the solution dir for `*.schema.yaml`. 250 ms trailing-edge debounce. | +| WATCH-MULTI | One schema doc per project, but the watcher tracks **every** project in the solution and which `obj/Generated/` tree belongs to which project. State map: `ImmutableDictionary<projectPath, (schemaPath, ImmutableHashSet<outputFile>)>`. Persisted at `<solution>/.dataprovider/watch-state.json`. | +| WATCH-INCREMENTAL | Per-table FNV-1a hash. Only flipped tables regenerate. One-hop FK closure pulls in dependents. Sibling projects' `.g.cs` keep mtime. | +| WATCH-RESILIENCE | YAML parse failure emits a `dataProvider/regenError` notification + `warn` log. Loop never throws and never exits on a single-file failure. | +| WATCH-WIRE | LSP wire models (notifications, project list, schema state, regen events) defined in `.td` files under `DataProvider/Wire/` using **typeDiagram** ([typediagram.dev](https://typediagram.dev/docs/language-reference.html)). Same `.td` source generates C# records for the server and TypeScript types for the future VS Code client. **No hand-written wire DTOs on either side.** | +| WATCH-MSGS | v1 messages: `initialize`, `initialized`, `shutdown`, `workspace/didChangeWatchedFiles`, `dataProvider/projects` (list known projects + their schema state), `dataProvider/regenerated` (per completed regen), `dataProvider/regenError`. | + +## CONVENTIONS + +| ID | Spec | +|---|---| +| CONV-NO-CONFIG | `DataProvider.json` is **optional** per [CON-NOCONFIG]. With zero config, the convention resolver decides what to emit. | +| CONV-NAME | Model name = `TableDefinition.Name` PascalCased. Column names emit verbatim (preserve casing). Naming helpers (`pascal`/`camel`/`snake`/`kebab`/`screaming-snake`) live in shared `Core/CodeGeneration/NamingConvention.cs` — no per-platform duplication. | +| CONV-CRUD | `Insert` emits iff every column maps to a known `PortableType`. `Update` emits iff `Insert` preconditions hold and at least one non-PK column exists. `Delete` emits iff a single-column PK exists. `BulkInsert`/`BulkUpsert` emit iff `Insert`/`Upsert` preconditions hold and PK is server-generated or supplied. | +| CONV-PK-VIOLATION | No PK or composite PK: model still emits, CRUD methods skipped, `DPSG013` warning logged. | +| CONV-OVERRIDE | `DataProvider.json`'s `tables[].generateInsert/Update/Delete` and `excludeColumns` are pure overrides on top of the convention result. Never required. | + +## TEMPLATES + +| ID | Spec | +|---|---| +| TPL-ENGINE | **Scriban** (single MIT-licensed NuGet, sandboxable, AOT-friendly, language-agnostic output). No Razor, no T4, no Handlebars. Internal seam is `Func<TemplateName, ImmutableDictionary<string,object>, string>` per CLAUDE.md "no interfaces". | +| TPL-DISC | Resolution order: `--templates <dir>` arg → `templates/` next to the schema doc → embedded defaults shipped in the tool. | +| TPL-MANIFEST | Each generator ships `generator.yaml`: `{ id, version, language, templates: [{ input, foreach: schema\|table, output }] }`. `output` is itself a Scriban expression. | +| TPL-CONTEXT | Frozen template context exposes: `schema`, `table`, `columns`, `meta.tool_version`, `meta.target_language`. Built-in helper modules: `naming` (case conversions), `types` (`to_typescript`/`to_kotlin`/`to_swift`/`to_dart`/`to_go`). | +| TPL-SANDBOX | `Scriban.TemplateContext.MemberFilter` denies access to `System.IO`, `System.Reflection`, `System.Diagnostics.Process`, `System.Environment`. Templates are pure data → text. | +| TPL-OUT | Default route `obj/DataProvider/<generator-id>/<rendered-path>`. Output extension comes from the manifest, not hard-coded. C# generation routes to `obj/DataProvider/csharp/`. | +| TPL-DETERMINISM | `meta.generated_at_utc` is **off by default** so byte-identical inputs produce byte-identical outputs. Opt in via `--templates-include-timestamp`. | +| TPL-TESTING | `Nimblesite.DataProvider.Generators.Testing` package shipped for plugin authors: golden-file harness, fake `SchemaDefinition` builder, snapshot diffing. | ## PKG @@ -117,6 +165,7 @@ To bump a grammar: <Project> <ItemGroup> <AdditionalFiles Include="$(MSBuildProjectDirectory)/**/*.lql" /> + <AdditionalFiles Include="$(MSBuildProjectDirectory)/**/*.schema.yaml" /> <AdditionalFiles Include="$(MSBuildProjectDirectory)/DataProvider.json" Condition="Exists('$(MSBuildProjectDirectory)/DataProvider.json')" /> </ItemGroup> @@ -124,10 +173,10 @@ To bump a grammar: BeforeTargets="CoreCompile" Inputs="@(AdditionalFiles);$(MSBuildThisFileDirectory)tool/net9.0/DataProvider.dll" Outputs="$(IntermediateOutputPath)DataProvider/.timestamp"> - <Error Condition="'$(DataProviderConnectionString)' == ''" - Code="DPSG001" Text="DataProviderConnectionString MSBuild property is not set." /> + <Error Condition="'$(DataProviderMode)' == 'live-db' AND '$(DataProviderConnectionString)' == ''" + Code="DPSG001" Text="DataProviderConnectionString MSBuild property is required when DataProviderMode=live-db." /> <MakeDir Directories="$(IntermediateOutputPath)DataProvider" /> - <Exec Command="dotnet "$(MSBuildThisFileDirectory)tool/net9.0/DataProvider.dll" --connection "$(DataProviderConnectionString)" --config "$(MSBuildProjectDirectory)/DataProvider.json" --out "$(IntermediateOutputPath)DataProvider" --platform "$(DataProviderPlatform)" --namespace "$(DataProviderGeneratedNamespace)" --accessibility "$(DataProviderGeneratedAccessibility)"" + <Exec Command="dotnet "$(MSBuildThisFileDirectory)tool/net9.0/DataProvider.dll" --mode "$(DataProviderMode)" --connection "$(DataProviderConnectionString)" --config "$(MSBuildProjectDirectory)/DataProvider.json" --out "$(IntermediateOutputPath)DataProvider" --platform "$(DataProviderPlatform)" --namespace "$(DataProviderGeneratedNamespace)" --accessibility "$(DataProviderGeneratedAccessibility)" --templates "$(DataProviderTemplates)"" IgnoreExitCode="false" ConsoleToMSBuild="true" CustomErrorRegularExpression="^.*\([0-9]+,[0-9]+\):\s*error\s+DPSG[0-9]+:.*$" CustomWarningRegularExpression="^.*\([0-9]+,[0-9]+\):\s*warning\s+DPSG[0-9]+:.*$" /> @@ -147,10 +196,11 @@ Auto-imported by NuGet from `build/DataProvider.targets`. Consumer never sees th | ID | Spec | |---|---| -| CONSUMER-CSPROJ | One `<PackageReference Include="DataProvider">`. One `<DataProviderConnectionString>$(DATAPROVIDER_CONN)</DataProviderConnectionString>`. One `<DataProviderPlatform>` (required, no auto-detection). Optional: `<DataProviderGeneratedNamespace>`, `<DataProviderGeneratedAccessibility>`. Nothing else. | -| CONSUMER-CONNSTR | Resolves through MSBuild from `$DATAPROVIDER_CONN`. Never committed. CI sets it in env. Local sets it via shell rc / `direnv`. IDE sets it via system env + IDE restart. | -| CONSUMER-DELETE | Migrating consumers delete: every `.config/dotnet-tools.json` legacy entry, every `<Target Name="GenerateDataProvider">`, every `Generated/` folder + `.gitignore` line, every `dotnet tool restore` step, every `<Exec>` calling a legacy DataProvider CLI, every `sed` codegen post-step. `dataprovider-migrate` stays per [CON-MIG-FIRST]. | -| CONSUMER-DETERMINISM | Build is deterministic across dev/CI iff both run the same migrations against the same schema first. Per [CON-MIG-FIRST]. The tool itself is deterministic on `(SHA256(schema bytes), SHA256(*.lql), SHA256(*.sql))`. | +| CONSUMER-CSPROJ | One `<PackageReference Include="DataProvider">`. One `<DataProviderPlatform>` (required, no auto-detection). `<DataProviderMode>` defaults to `schema-doc`; set to `live-db` to opt in. `<DataProviderConnectionString>$(DATAPROVIDER_CONN)</DataProviderConnectionString>` required iff mode=`live-db`. Optional: `<DataProviderGeneratedNamespace>`, `<DataProviderGeneratedAccessibility>`, `<DataProviderTemplates>`. Nothing else. | +| CONSUMER-CONNSTR | Live-db mode only. Resolves through MSBuild from `$DATAPROVIDER_CONN`. Never committed. CI sets it in env. Local sets it via shell rc / `direnv`. IDE sets it via system env + IDE restart. | +| CONSUMER-SCHEMA | Schema-doc mode (default). Consumer ships exactly one `*.schema.yaml` somewhere under the project (recursive discovery per [DOC-ONE-PER-PROJECT]). Format is the canonical `DatabaseSchema` YAML loaded by `SchemaSerializer.FromYaml`. Zero connection strings, zero env vars, zero migrations required to compile. | +| CONSUMER-DELETE | Migrating consumers delete: every `.config/dotnet-tools.json` legacy entry, every `<Target Name="GenerateDataProvider">`, every `Generated/` folder + `.gitignore` line, every `dotnet tool restore` step, every `<Exec>` calling a legacy DataProvider CLI, every `sed` codegen post-step. `DataProvider.json` is no longer required per [CON-NOCONFIG]; delete it unless explicit overrides are still needed. `dataprovider-migrate` stays for live-db mode per [CON-MIG-FIRST]. | +| CONSUMER-DETERMINISM | Schema-doc mode: deterministic on `(SHA256(schema.yaml bytes), SHA256(*.lql), SHA256(*.sql), SHA256(template files))`. Live-db mode: deterministic across dev/CI iff both run the same migrations against the same schema first per [CON-MIG-FIRST]. | ## DEPS @@ -165,25 +215,30 @@ Auto-imported by NuGet from `build/DataProvider.targets`. Consumer never sees th | ID | Spec | |---|---| -| TEST-UNIT | `DataProvider.Tool.Tests` (net9.0) drives the tool E2E against a real platform testcontainer. No in-memory DBs. Asserts on literal generated source for fixed schema fixtures. | -| TEST-DIAG | One test per `DPSG001`–`DPSG007`. Each arranges the failure mode and asserts the canonical stderr line + exit code. | -| TEST-MSBUILD | A test project consumes the packed `DataProvider.{version}.nupkg` from a local feed, builds against a testcontainer, asserts `.g.cs` lands in `obj/DataProvider/`, asserts MSBuild surfaces `DPSGxxx` errors structurally in the IDE Error List. | -| TEST-NATIVE | Matrix runs on `win-x64`, `linux-x64`, `linux-arm64`, `osx-x64`, `osx-arm64`. Each combo loads its driver's native asset from `runtimes/` and runs an introspection against a testcontainer. Catches RID-specific packaging mistakes before they reach a consumer. | +| TEST-UNIT | `DataProvider.Tool.Tests` (net9.0) drives the tool E2E in **both modes**. Schema-doc tests run with no DB, asserting literal generated source from a fixture `*.schema.yaml`. Live-db tests run against a real platform testcontainer (no in-memory DBs). For a given schema both modes must produce byte-identical output. | +| TEST-DIAG | One test per `DPSG001`–`DPSG015`. Each arranges the failure mode and asserts the canonical stderr line + exit code. | +| TEST-MSBUILD | A test project consumes the packed `DataProvider.{version}.nupkg` from a local feed, builds in **both modes** (schema-doc default + live-db opt-in), asserts `.g.cs` lands in `obj/DataProvider/`, asserts MSBuild surfaces `DPSGxxx` errors structurally in the IDE Error List. | +| TEST-NATIVE | Live-db only. Matrix runs on `win-x64`, `linux-x64`, `linux-arm64`, `osx-x64`, `osx-arm64`. Each combo loads its driver's native asset from `runtimes/` and runs an introspection against a testcontainer. Schema-doc mode bypasses native drivers entirely. | | TEST-PARSER | Per platform, `Nimblesite.DataProvider.{Platform}.Tests` parses real fixture SQL and asserts on the resulting syntax tree shape (rule contexts, terminal nodes, parameter list, projection list). Mandatory for every platform under [CON-PARSER-ONLY]. | +| TEST-WATCH | Boot the LSP against a temp solution with two projects, edit one column in one project's `*.schema.yaml`, assert only that project's `.g.cs` rewrites; sibling project keeps mtime. Snapshot test on the typeDiagram-generated TS types confirms the wire surface. | +| TEST-TEMPLATES | Sample TypeScript template under `Samples/` emits `*.d.ts` for the `example` schema; `tsc --noEmit` passes. Sandbox test asserts a malicious template that touches `System.IO` is rejected with `DPSG014`. | ## DX | Scenario | Behavior | |---|---| -| Fresh clone | `git clone` → set `DATAPROVIDER_CONN` → start DB → `dataprovider-migrate` → `dotnet build`. | -| DB unreachable | `DPSG002` (sanitised target + driver error). No fallback. No offline cache. | -| `dotnet restore` | Tool does **not** run during restore. No DB needed. | -| `dotnet pack` / `dotnet test` | Both invoke compile, therefore invoke the tool, therefore need a reachable DB. | +| Fresh clone (schema-doc, default) | `git clone` → `dotnet build`. No DB, no env vars, no config file required. | +| Fresh clone (live-db) | `git clone` → set `DATAPROVIDER_CONN` → start DB → `dataprovider-migrate` → `dotnet build`. | +| DB unreachable | Live-db only: `DPSG002` (sanitised target + driver error). Schema-doc mode never opens a DB. | +| Schema doc missing / invalid | Schema-doc only: `DPSG010` / `DPSG011`. | +| `dotnet restore` | Tool does **not** run during restore. | +| `dotnet pack` / `dotnet test` | Both invoke compile, therefore invoke the tool. Schema-doc mode needs only the `.schema.yaml` on disk; live-db needs a reachable DB. | +| `--watch` | Tool stays resident, watches every `*.schema.yaml` in the solution, regenerates only the affected project's outputs per `## SCHEMA-DOC`. | ## RISK | ID | Spec | |---|---| -| RISK-NATIVE-RID | New RID = add native asset to package + republish. Mitigated by [TEST-NATIVE] running the full RID matrix in CI. | -| RISK-DETERMINISM | Live-DB introspection means dev/CI may diverge if schemas diverge. Mitigated by [CON-MIG-FIRST]. | -| RISK-FORK-COST | One `dotnet DataProvider.dll` fork per build. Cold start ≈ 100 ms. < 2 % of a typical build. Negligible. | +| RISK-NATIVE-RID | Live-db only: new RID = add native asset to package + republish. Mitigated by [TEST-NATIVE] running the full RID matrix in CI. Schema-doc mode loads no native driver. | +| RISK-DETERMINISM | Live-db: dev/CI may diverge if schemas diverge. Mitigated by [CON-MIG-FIRST]. Schema-doc: deterministic by file hash per [CONSUMER-DETERMINISM]. | +| RISK-FORK-COST | One `dotnet DataProvider.dll` fork per build. Cold start ≈ 100 ms. < 2 % of a typical build. `--watch` eliminates fork cost entirely for IDE edit cycles. | diff --git a/docs/specs/migration-cli-spec.md b/docs/specs/migration-cli-spec.md deleted file mode 100644 index 2ec05bbe..00000000 --- a/docs/specs/migration-cli-spec.md +++ /dev/null @@ -1,119 +0,0 @@ -# Migration.Cli Specification - -NOTE: leave the JSON serialization/deserialization code as is for now, but deactivate it. The core will eventually offer JSON, but we are focusing on YAML for now. - -## Overview - -`Migration/Migration.Cli/Migration.Cli.csproj` is the **single, canonical CLI tool** for creating databases from schema definitions. All projects that need to spin up a database for code generation MUST use this executable. There is no other way. - -## Architecture - -Migration.Cli contains the DLLs for both SQLite and Postgres migrations. It is database-agnostic at the interface level - callers specify a YAML schema file path, and the CLI handles the rest. - -## Usage - -``` -dotnet run --project Migration/Migration.Cli/Migration.Cli.csproj -- \ - --schema path/to/schema.yaml \ - --output path/to/database.db \ - --provider [sqlite|postgres] -``` - -## Schema Input: YAML Only - -NOTE: leave the JSON serialization/deserialization code as is for now, but deactivate it. The core will eventually offer JSON, but we are focusing on YAML for now. - -The CLI accepts **only YAML schema files**. It does not accept: -- C# code references -- Inline schema definitions -- Project references to schema classes - -If a project defines its schema in C# code (e.g., `ExampleSchema.cs`, `ClinicalSchema.cs`), that schema MUST be serialized to YAML first. The YAML file is then passed to Migration.Cli. - -### Schema-to-YAML Workflow - -1. Schema is defined in a **separate Migrations assembly** (e.g., `MyProject.Migrations/`) with NO dependencies on generated code -2. Build step compiles the Migrations assembly first -3. Migration.Cli `export` subcommand exports C# schema to YAML file -4. Migration.Cli `migrate` subcommand reads YAML and creates database -5. DataProvider code generation runs against the created database -6. Main project (e.g., `MyProject.Api/`) compiles with generated code - -### CRITICAL: Separate Migrations Assemblies - -**Schemas MUST be in separate assemblies to avoid circular build dependencies.** - -**Naming convention: Always use `*.Migrations` suffix, never `*.Schema` or `*BuildDb`.** - -Correct pattern: -``` -MyProject.Migrations/ # Schema definition only, NO generated code deps - └── MyProjectSchema.cs # Defines SchemaDefinition -MyProject.Api/ # References MyProject.Migrations, has generated code - └── Generated/ # DataProvider generated code -``` - -The Migrations assembly: -- Contains ONLY the `SchemaDefinition` class -- References ONLY `Migration` (for schema types) -- Has NO dependencies on generated code -- Can be built BEFORE code generation runs - -The API/main assembly: -- References the Migrations assembly -- Contains generated code from DataProvider -- Is built AFTER code generation - -## Why YAML? - -- **No circular dependencies**: Migration.Cli has zero project references to consumer projects -- **Clean build order**: YAML files are static assets, not compiled code -- **Portable**: Schema can be versioned, diffed, and shared without compilation -- **Single tool**: One CLI handles all schemas for all projects - -## Forbidden Patterns - -- Individual `*BuildDb` projects per consumer (causes circular builds) -- `<Compile Include="../OtherProject/Schema.cs">` in Migration.Cli (circular dependency) -- Multiple CLI tools for database creation -- Hardcoded schema names/switches in the CLI -- CLI tool referencing actual schemas -- **Schema classes in the same project as generated code** (causes circular build deps) -- Migrations assemblies with dependencies on generated code -- Using `*.Schema` or `*BuildDb` naming (must use `*.Migrations`) - -## MSBuild Integration - -Consumer projects call Migration.Cli with `export` then `migrate` subcommands in pre-build targets: - -```xml -<!-- Step 1: Export C# schema to YAML (Migrations assembly must be built first) --> -<Target Name="ExportSchemaToYaml" BeforeTargets="CreateBuildDatabase"> - <Exec Command='dotnet run --project "$(SolutionDir)Migration/Migration.Cli/Migration.Cli.csproj" -- export --assembly "$(SolutionDir)MyProject.Migrations/bin/Debug/net9.0/MyProject.Migrations.dll" --type "MyProject.Migrations.MyProjectSchema" --output "$(MSBuildProjectDirectory)/schema.yaml"' /> -</Target> - -<!-- Step 2: Create database from YAML --> -<Target Name="CreateBuildDatabase" BeforeTargets="GenerateDataProvider"> - <Exec Command='dotnet run --project "$(SolutionDir)Migration/Migration.Cli/Migration.Cli.csproj" -- migrate --schema "$(MSBuildProjectDirectory)/schema.yaml" --output "$(MSBuildProjectDirectory)/build.db" --provider sqlite' /> -</Target> -``` - -No project references. No schema includes. Just paths to assemblies and YAML files. - -## Build Order (CRITICAL) - -To avoid circular dependencies, the build order MUST be: - -``` -1. Migration/Migration.csproj # Core types -2. MyProject.Migrations/ # Schema definition (refs Migration only) -3. Migration.Cli export # Export schema to YAML -4. Migration.Cli migrate # Create DB from YAML -5. DataProvider code generation # Generate C# from DB -6. MyProject.Api/ # Main project with generated code -``` - -The Migrations assembly MUST NOT reference: -- The API/main project -- Any generated code -- Any project that depends on generated code diff --git a/docs/specs/migration-spec.md b/docs/specs/migration-spec.md index 20cbff7c..c88619d7 100644 --- a/docs/specs/migration-spec.md +++ b/docs/specs/migration-spec.md @@ -14,7 +14,8 @@ 10. [Error Handling](#10-error-handling) 11. [Conformance Requirements](#11-conformance-requirements) 12. [E2E Testing Requirements](#12-e2e-testing-requirements) -13. [Appendices](#13-appendices) +13. [Schema Capture and Metadata](#13-schema-capture-and-metadata) +14. [Appendices](#14-appendices) --- @@ -134,7 +135,7 @@ var schema = Schema.Define("MyApp") ### 4.3 YAML Schema Format -Schema files use YAML format. See [migration-cli-spec.md](migration-cli-spec.md) for CLI usage. The YAML format mirrors the C# records: +Schema files use YAML format. The `DataProviderMigrate` CLI contract is defined in [7.4 DataProviderMigrate CLI](#74-dataprovidermigrate-cli-mig-cli). The YAML format mirrors the C# records: ```yaml name: MyApp @@ -625,6 +626,159 @@ idempotency proofs can verify the constraint was materialized. 5. Return result with applied operations ``` +### 7.4 DataProviderMigrate CLI [MIG-CLI] + +`Migration/DataProviderMigrate/DataProviderMigrate.csproj` is the single, canonical CLI tool for creating databases from schema definitions. All projects that need to spin up a database for code generation MUST use this executable or the packaged `DataProviderMigrate` .NET tool. + +The CLI contains the SQLite and PostgreSQL migration providers. It is database-agnostic at the command surface: callers pass a YAML schema file path, an output database path or connection string, and a provider name. + +#### 7.4.1 Commands [MIG-CLI-COMMANDS] + +Installed tool usage: + +```bash +dotnet DataProviderMigrate migrate \ + --schema path/to/schema.yaml \ + --output path/to/database.db \ + --provider sqlite + +dotnet DataProviderMigrate export \ + --assembly path/to/MyProject.Migrations.dll \ + --type MyProject.Migrations.MyProjectSchema \ + --output path/to/schema.yaml +``` + +Repository-local usage: + +```bash +dotnet run --project Migration/DataProviderMigrate/DataProviderMigrate.csproj -- \ + migrate \ + --schema path/to/schema.yaml \ + --output path/to/database.db \ + --provider sqlite +``` + +`migrate` options: + +| Option | Required | Meaning | +|--------|----------|---------| +| `--schema`, `-s` | Yes | Path to a YAML schema definition file | +| `--output`, `-o` | Yes | SQLite database file path or PostgreSQL connection string | +| `--provider`, `-p` | No | `sqlite` or `postgres`; defaults to `sqlite` | +| `--allow-destructive` | No | Permits destructive drift cleanup operations; off by default | +| `--phase` | No | `all`, `structural`, or `rls`; defaults to `all` | + +`export` options: + +| Option | Required | Meaning | +|--------|----------|---------| +| `--assembly`, `-a` | Yes | Compiled assembly containing the schema type | +| `--type`, `-t` | Yes | Fully qualified schema type name | +| `--output`, `-o` | Yes | YAML file path to write | + +Schema export types MUST expose either a static `Definition` property returning `SchemaDefinition` or a static `Build()` method returning `SchemaDefinition`. + +#### 7.4.2 YAML-Only Migration Input [MIG-CLI-YAML-ONLY] + +The `migrate` command accepts only YAML schema files. It does not accept: + +- C# code references +- Inline schema definitions +- Project references to schema classes +- JSON schema files + +JSON serialization/deserialization code may remain dormant in the core library for future support, but the CLI MUST NOT expose JSON schema input until YAML support is complete and the format is explicitly specified. + +If a project defines its schema in C# code, that schema MUST be exported to YAML first. The YAML file is then passed to `DataProviderMigrate migrate`. + +#### 7.4.3 Schema-to-YAML Workflow [MIG-CLI-SCHEMA-YAML-WORKFLOW] + +Consumer build pipelines that start from C# schema definitions MUST use this order: + +1. Define schema in a separate `*.Migrations` assembly with no generated-code dependencies. +2. Build the migrations assembly first. +3. Run `DataProviderMigrate export` to serialize the C# schema to YAML. +4. Run `DataProviderMigrate migrate` to create or update the target database from YAML. +5. Run DataProvider code generation against the created database. +6. Build the main project with generated code included. + +#### 7.4.4 Separate Migrations Assemblies [MIG-CLI-MIGRATIONS-ASSEMBLY] + +Schemas MUST live in separate migrations assemblies to avoid circular build dependencies. + +Naming convention: use the `*.Migrations` suffix. Do not use `*.Schema` or `*BuildDb`. + +Correct pattern: + +```text +MyProject.Migrations/ + MyProjectSchema.cs # Defines SchemaDefinition + +MyProject.Api/ + Generated/ # DataProvider generated code +``` + +The migrations assembly: + +- Contains only schema definition code. +- References only migration schema types and their direct dependencies. +- Has no dependencies on generated code. +- Builds before code generation runs. + +The API or main assembly: + +- References the migrations assembly only when it needs the schema at runtime. +- Contains DataProvider generated code. +- Builds after code generation. + +#### 7.4.5 Build Integration [MIG-CLI-BUILD-INTEGRATION] + +Consumer projects may wire export and migrate into MSBuild pre-build targets: + +```xml +<Target Name="ExportSchemaToYaml" BeforeTargets="CreateBuildDatabase"> + <Exec Command='dotnet DataProviderMigrate export --assembly "$(SolutionDir)MyProject.Migrations/bin/Debug/net10.0/MyProject.Migrations.dll" --type "MyProject.Migrations.MyProjectSchema" --output "$(MSBuildProjectDirectory)/schema.yaml"' /> +</Target> + +<Target Name="CreateBuildDatabase" BeforeTargets="GenerateDataProvider"> + <Exec Command='dotnet DataProviderMigrate migrate --schema "$(MSBuildProjectDirectory)/schema.yaml" --output "$(MSBuildProjectDirectory)/build.db" --provider sqlite' /> +</Target> +``` + +There are no project references from the CLI to consumer schemas, no schema source includes, and no hardcoded schema names or switches in the CLI. + +#### 7.4.6 Required Build Order [MIG-CLI-BUILD-ORDER] + +To avoid circular dependencies, builds that need generated data access code MUST follow this order: + +```text +1. Migration/Nimblesite.DataProvider.Migration.Core +2. MyProject.Migrations +3. DataProviderMigrate export +4. DataProviderMigrate migrate +5. DataProvider code generation +6. MyProject.Api +``` + +The migrations assembly MUST NOT reference: + +- The API or main project. +- Any generated code. +- Any project that depends on generated code. + +#### 7.4.7 Forbidden CLI Patterns [MIG-CLI-FORBIDDEN] + +The following patterns are not conformant: + +- Individual `*BuildDb` projects per consumer. +- `<Compile Include="../OtherProject/Schema.cs">` in the CLI project. +- Multiple CLI tools for database creation. +- Hardcoded schema names or project-specific switches in the CLI. +- CLI references to consumer schema projects. +- Schema classes in the same project as generated code. +- Migrations assemblies with dependencies on generated code. +- `*.Schema` or `*BuildDb` naming for migrations projects. + --- ## 8. Diff Engine diff --git a/opencode.json b/opencode.json index 79756a8a..0c8379c8 100644 --- a/opencode.json +++ b/opencode.json @@ -1,5 +1,5 @@ { - "_agent_pmo": "d75d5c8", + "_agent_pmo": "74cf183", "$schema": "https://opencode.ai/config.json", "instructions": ["CLAUDE.md"] }