From 28fda8bf3a6557bac6032350a0576bbb8dacc816 Mon Sep 17 00:00:00 2001 From: Han Ngo Date: Sat, 18 Jul 2026 20:28:24 +0700 Subject: [PATCH 1/2] feat(sync): one-way insert-only push to a foreign team board Add a `notion-taskboard` app: a one-way, insert-only push of a repo's board rows out to a team-OWNED Notion board (implements ops-toolkit ID-138, design locked 2026-06-16). The board is the source of truth; the sink is never read for merge and the board file is never written. Fields are set only on page-create, so a team member's later edits on the card are never overwritten. The local sync-state map is the identity index, a row is pushed exactly once. Status/Priority/Weight map to the team board's own option names via `.kit.toml [sync]` config, so the sink never mutates the team schema (it does one benign read to resolve the data source, never a PATCH). Dropped rows are skipped; a board state absent from the map uses status_default or fails closed with guidance rather than guessing a column. The two-way mesh and its four live adapters (Reminders/Notion/Hermes/ Multica) are untouched: create-only is a separate planner + apply path, dispatched on a source attribute. The engine's board parser is generalized from ID- to any [A-Z]+-\d+ prefix so it reads every adopted repo's board (e.g. dfoundation DF-NN). Fixtures only, no live team-board writes. 12 new cases incl. negative controls (dropped not pushed, already-pushed row frozen, unmapped status fails closed, skip-tag filter, missing config fails closed). See lib/sync/docs/specs/SPEC-003-oneway-create-push.md. --- docs/CHANGELOG.md | 12 + .../oneway-create-push.md | 50 +++ lib/board/board.sh | 23 ++ lib/config/module-registry.md | 12 +- lib/sync/README.md | 39 ++ lib/sync/backlog_sync.py | 76 +++- lib/sync/docs/proof-of-done.md | 28 ++ .../docs/specs/SPEC-003-oneway-create-push.md | 104 ++++++ lib/sync/sources/notion_taskboard.py | 176 +++++++++ lib/sync/sync_core.py | 37 +- lib/sync/tests/test_notion_taskboard.py | 337 ++++++++++++++++++ 11 files changed, 889 insertions(+), 5 deletions(-) create mode 100644 docs/implementation-notes/oneway-create-push.md create mode 100644 lib/sync/docs/specs/SPEC-003-oneway-create-push.md create mode 100644 lib/sync/sources/notion_taskboard.py create mode 100644 lib/sync/tests/test_notion_taskboard.py diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 05dbab7d..12e92f1b 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -4,6 +4,18 @@ All notable changes to dwarves-kit are documented here. ## [Unreleased] +### Added +- **`sync` one-way insert-only push to a foreign team board** (`notion-taskboard` + app, SPEC-003, implements ops-toolkit ID-138). Pushes a repo's board rows out + to a team-OWNED Notion board (create page, never update, never read for merge, + board file never written); fields are set only on page-create so team edits are + never overwritten, and the local sync-state map is the identity index. Status/ + Priority/Weight map to the team board's own option names via `.kit.toml [sync]` + config, so the team schema is never mutated. The two-way mesh and its four live + adapters are untouched (`create_only` is a separate path). The engine's board + parser was generalized from `ID-` to any `[A-Z]+-\d+` prefix so it reads every + adopted repo's board (e.g. dfoundation `DF-NN`). + ### Changed - **`bridge` module folded into `sync`** (zero live consumers at fold time: no `bridge=on` rows, no snapshot, module off). Legacy `mirror`/`status`/`writeback` diff --git a/docs/implementation-notes/oneway-create-push.md b/docs/implementation-notes/oneway-create-push.md new file mode 100644 index 00000000..4e8146a5 --- /dev/null +++ b/docs/implementation-notes/oneway-create-push.md @@ -0,0 +1,50 @@ +# Implementation notes: one-way insert-only push (SPEC-003 / ID-138) + +Delta from SPEC-003 only. + +## 2026-07-18 create-only is a separate path, not a `plan_sync` flag + +Context: the locked design is insert-only (never update after create). The +two-way `plan_sync` exists to MERGE, so bending it into strict create-only +would need a flag threaded through every branch, with blast radius on the four +live adapters. +Decision: added `plan_create_only` (pure) + `sync_create_only` (I/O) as a +parallel path, dispatched on `getattr(src, "create_only", False)`. The two-way +path is byte-for-byte unchanged. +Why: zero blast radius; the create-only planner is ~15 lines and trivially +proven with a negative control. +Alternatives: a `create_only` flag inside `plan_sync` (rejected: touches shared +merge logic used by 4 live adapters). +Impact: existing adapter tests unchanged; new logic is independently tested. + +## 2026-07-18 the sink is write-only; `read()` returns `[]` + +Context: truly one-way means we should not depend on reading the foreign team +board at all. The identity index is the local sync-state map (locked design). +Decision: `NotionTaskBoardSource.read()` returns `[]`; `plan_create_only` +consults `state["map"]` (not a read of the board) to know which rows are +already pushed. `ensure_binding()` still does ONE benign read to resolve the +data_source_id (required as the page-create parent), never a schema PATCH. +Impact: a team member deleting a card is NOT re-created (its bid stays in the +map); re-creating a deleted card is a v1.1 concern, out of the locked scope. + +## 2026-07-18 status states the design left unmapped + +Context: the locked field map names 5 states (queued/executing/parked/shipped +-> columns, dropped -> skip) but the board has 8 (claimed/speccing/validated +also exist). +Decision: NOT re-designed here. `skip_kw` defaults to `{"dropped"}` only; a +config `status_default` catches any state absent from the status map. If a row +in an unmapped state is reached with no default set, apply raises a guided +SystemExit rather than guessing a column. +Why: the kit ships no tenant policy; the consumer (dfoundation) decides where +claimed/speccing/validated land by setting `status_default` (or extending the +map). Honors "don't re-design" while never silently dropping active work. + +## 2026-07-18 tenant IDs stay in the consumer repo + +Context: this repo (dwarves-kit) is public; the Task Board UUID and Han's +Notion person id are tenant identity. +Decision: the adapter takes db id / maps / owner as config; the dfoundation +`.kit.toml [sync]` carries the real values. No UUID in this repo (asserted by a +grep in the dfoundation PR, not here). diff --git a/lib/board/board.sh b/lib/board/board.sh index 1ba6aae9..80e52df2 100755 --- a/lib/board/board.sh +++ b/lib/board/board.sh @@ -738,6 +738,29 @@ cmd_sync() { v="$(kit_config_get sync.reminders_list "")"; [ -n "$v" ] && args+=(--list "$v") v="$(kit_config_get sync.notion_db "")"; [ -n "$v" ] && args+=(--notion-db "$v") v="$(kit_config_get sync.notion_parent "")"; [ -n "$v" ] && args+=(--notion-parent "$v") + # notion-taskboard: one-way, insert-only push to a foreign team board + # (SPEC-003). Its down-filter uses TOML-friendly underscore keys but the + # engine's --filter app token keeps the hyphenated adapter name. + for fk in only_tags skip_tags intake; do + v="$(kit_config_get "sync.notion_taskboard_${fk}" "")" + [ -n "$v" ] && args+=(--filter "notion-taskboard:${fk}=${v}") + done + v="$(kit_config_get sync.notion_taskboard_db "")" + [ -n "$v" ] && args+=(--notion-taskboard-db "$v") + v="$(kit_config_get sync.notion_taskboard_status_map "")" + [ -n "$v" ] && args+=(--notion-taskboard-status-map "$v") + v="$(kit_config_get sync.notion_taskboard_status_default "")" + [ -n "$v" ] && args+=(--notion-taskboard-status-default "$v") + v="$(kit_config_get sync.notion_taskboard_priority_map "")" + [ -n "$v" ] && args+=(--notion-taskboard-priority-map "$v") + v="$(kit_config_get sync.notion_taskboard_weight_map "")" + [ -n "$v" ] && args+=(--notion-taskboard-weight-map "$v") + v="$(kit_config_get sync.notion_taskboard_owner "")" + [ -n "$v" ] && args+=(--notion-taskboard-owner "$v") + v="$(kit_config_get sync.notion_taskboard_props "")" + [ -n "$v" ] && args+=(--notion-taskboard-props "$v") + v="$(kit_config_get sync.notion_taskboard_types "")" + [ -n "$v" ] && args+=(--notion-taskboard-types "$v") v="$(kit_config_get sync.hermes_target "")"; [ -n "$v" ] && args+=(--hermes-target "$v") v="$(kit_config_get sync.hermes_home "")"; [ -n "$v" ] && args+=(--hermes-home "$v") v="$(kit_config_get sync.multica_url "")"; [ -n "$v" ] && args+=(--multica-url "$v") diff --git a/lib/config/module-registry.md b/lib/config/module-registry.md index 4eaa0b1e..edff0d69 100644 --- a/lib/config/module-registry.md +++ b/lib/config/module-registry.md @@ -153,7 +153,7 @@ single-reader fence). No env vars; per-repo values live in `.kit.toml [sync]`. | Env var | kit.toml key | Default | Status | Module | Doc | |---|---|---|---|---|---| -| - | sync.apps | `""` (sync off) | [impl] | sync | Comma list of apps to sync to (`reminders,notion,hermes,multica`); the plugin mechanism. Legacy aliases `surfaces` and `sources` still read as fallbacks (renamed 2026-07-16 for plain vocabulary; `sources` also collided with SPEC-002's board-side inputs). | +| - | sync.apps | `""` (sync off) | [impl] | sync | Comma list of apps to sync to (`reminders,notion,hermes,multica,notion-taskboard`); the plugin mechanism. Legacy aliases `surfaces` and `sources` still read as fallbacks (renamed 2026-07-16 for plain vocabulary; `sources` also collided with SPEC-002's board-side inputs). `notion-taskboard` is a one-way, insert-only push (SPEC-003), not part of the two-way mesh. | | - | sync.mode | `"manual"` | [impl] | sync | `manual` (default) or `cron`. Read by `lib/sync/deploy/macos/install` (kit ID-289): a repo must set `mode = "cron"` before that installer will render or load a per-repo scheduled-sync LaunchAgent; any other value is a clean refusal, not a silent fallthrough. Not read by `board.sh cmd_sync` itself -- manual `board sync` runs are unaffected by this key. ALSO re-read live by the installed `board-sync-cron` launcher on every scheduled run: flipping `mode` back to `"manual"` after install makes the next scheduled run skip cleanly (exit 0, logged) rather than silently keep syncing against a config that says it shouldn't. | | - | sync.interval_secs | `3600` | [impl] | sync | Cron LaunchAgent `StartInterval` seconds, read by `lib/sync/deploy/macos/install` as the default cadence for `mode = "cron"`; `--interval-secs N` overrides it for one install run. | | - | sync.reminders_list | `"Backlog"` | [impl] | sync | Apple Reminders list name. | @@ -174,6 +174,16 @@ single-reader fence). No env vars; per-repo values live in `.kit.toml [sync]`. | - | sync.multica_only_tags | `""` | [impl] | sync | Down-filter: a row must carry one of these tags to appear on multica. | | - | sync.multica_skip_tags | `""` | [impl] | sync | Down-filter: a row carrying any of these tags never appears on multica. | | - | sync.multica_intake | `""` (all) | [impl] | sync | Up-filter for foreign multica items: `all`, `tagged:`, or `none`. | +| - | sync.notion_taskboard_db | `""` | [impl] | sync | Target Notion database id for the one-way insert-only Task Board push (SPEC-003). Tenant id: lives in the consumer repo's `.kit.toml`, never here. Required when `notion-taskboard` is in `apps`. | +| - | sync.notion_taskboard_status_map | `""` | [impl] | sync | `board-state=OptionName` comma map to the team board's OWN Status options, e.g. `queued=Backlog,executing=In progress,parked=Waiting,shipped=Done`. `dropped` is skipped by default (never pushed). | +| - | sync.notion_taskboard_status_default | `""` | [impl] | sync | Status option for board states absent from the map (e.g. claimed/speccing/validated); without it, an unmapped state is a hard, guided error rather than a guess. | +| - | sync.notion_taskboard_priority_map | `""` | [impl] | sync | `tag=Option` map for Priority, e.g. `u-hi=P0,u-mid=P1,u-lo=P2` (derived from a row's `#u-*` tag). | +| - | sync.notion_taskboard_weight_map | `""` | [impl] | sync | `tag=Value` map for Weight, e.g. `f-hi=2,f-mid=5,f-lo=13` (from a row's `#f-*` tag). Optional. | +| - | sync.notion_taskboard_owner | `""` | [impl] | sync | Owner value set on create (a people-prop user id by default). Tenant id: consumer repo only. | +| - | sync.notion_taskboard_props | `""` | [impl] | sync | JSON overriding the target prop NAMES `{title,status,priority,weight,owner,notes}` (defaults Task/Status/Priority/Weight/Owner/Notes). | +| - | sync.notion_taskboard_types | `""` | [impl] | sync | JSON overriding the target prop TYPES `{status,priority,weight,owner}` (defaults status/select/number/people). | +| - | sync.notion_taskboard_only_tags | `""` | [impl] | sync | Down-filter: a row must carry one of these tags to be pushed to the Task Board. | +| - | sync.notion_taskboard_skip_tags | `""` | [impl] | sync | Down-filter: a row carrying any of these tags is never pushed to the Task Board. | ### session (incl. session-intel / skill-curator, prefix `SKILL_CURATOR_*`) diff --git a/lib/sync/README.md b/lib/sync/README.md index 66660527..e8ba41b9 100644 --- a/lib/sync/README.md +++ b/lib/sync/README.md @@ -125,6 +125,45 @@ Live wiring (2026-07-16): Notion DB `612f123c-7476-4a78-90e2-e1465c0a0df6` workspace `ntn` is logged into; move it if a personal workspace gets connected). Hermes: `mini-tieubao`, `~/hermes-personal/home` (restic-covered). +## One-way push to a foreign team board (`notion-taskboard`, SPEC-003) + +A fifth app, `notion-taskboard`, is NOT part of the two-way mesh: it is a +one-way, **insert-only** push of a repo's board rows to a foreign, team-OWNED +Notion board (implements ops-toolkit ID-138, design locked 2026-06-16). The +board is the source of truth; the sink is never read for merge and the board +file is never written. Fields are set ONLY on page-create, so a team member's +later edits on the card are never overwritten. The local sync-state map is the +identity index (a `bid` already pushed is never re-pushed), because the team +board's own ID column is a read-only auto-increment that cannot hold `DF-NN`. + +Status / Priority / Weight are mapped to the team board's OWN option names via +config, so the sink never mutates the team schema (it does one benign read to +resolve the data source, never a PATCH). `dropped` rows are skipped; a state +absent from the map uses `status_default` or, if unset, is a hard guided error. + +```toml +[sync] +apps = "notion-taskboard" +notion_taskboard_db = "" # tenant id: consumer repo only +notion_taskboard_status_map = "queued=Backlog,executing=In progress,parked=Waiting,shipped=Done" +notion_taskboard_status_default = "Backlog" # claimed/speccing/validated land here +notion_taskboard_priority_map = "u-hi=P0,u-mid=P1,u-lo=P2" +notion_taskboard_weight_map = "f-hi=2,f-mid=5,f-lo=13" # optional +notion_taskboard_owner = "" # optional (people prop) +``` + +Manual full-reconcile (the v1 trigger; a `board set` hook and Discord-on-shipped +are deferred, the latter needs update-detection insert-only v1 does not do): + +``` +_meta/board sync --apps notion-taskboard --dry-run # preview, no writes +_meta/board sync --apps notion-taskboard # push new rows +``` + +Prop NAMES/TYPES are overridable via `notion_taskboard_props` / +`notion_taskboard_types` (JSON); defaults are Task/Status/Priority/Weight/ +Owner/Notes and status/select/number/people. + ## Tests ``` diff --git a/lib/sync/backlog_sync.py b/lib/sync/backlog_sync.py index 74d9ed65..6b764890 100644 --- a/lib/sync/backlog_sync.py +++ b/lib/sync/backlog_sync.py @@ -26,10 +26,13 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) -from sync_core import apply_board, build_state, describe, parse_board, plan_sync # noqa: E402 +from sync_core import (apply_board, build_state, describe, parse_board, # noqa: E402 + plan_create_only, plan_sync) from sources.hermes import HermesSource # noqa: E402 from sources.multica import MulticaSource # noqa: E402 from sources.notion import NotionSource # noqa: E402 +from sources.notion_taskboard import (NotionTaskBoardSource, # noqa: E402 + parse_map) from sources.reminders import RemindersSource # noqa: E402 LEGACY_REMINDERS_STATE = (Path.home() / ".cache" / "backlog-reminders-sync" @@ -60,9 +63,42 @@ def warn_duplicate_ids(text: str) -> None: f"sync) for {', '.join(broken)}; fix the board") +def sync_create_only(src, backlog: Path, state_path: Path, dry_run: bool, + filt: dict | None = None) -> None: + """One-way, insert-only push to a write-only sink (SPEC-003). The board + file is never written; the local state map is the identity index.""" + text = backlog.read_text() + rows = parse_board(text) + state = json.loads(state_path.read_text()) if state_path.exists() else {} + if hasattr(src, "binding") and state.get("binding"): + src.binding = state["binding"] + plan = plan_create_only(rows, state, skip_kw=getattr(src, "skip_kw", None), + filt=filt) + header = (f"{src.name}: {len(rows)} board rows, " + f"{len(plan.src_create)} to create") + if dry_run: + print(f"dry-run {header}") + print(describe(plan)) + return + created = src.apply(plan, {}, rows) + new_map = dict(state.get("map", {})) + for bid, _t, _b, _kw in plan.src_create: + if bid in created: + new_map[bid] = {"rid": created[bid]} + new_state = {"map": new_map} + if getattr(src, "binding", None): + new_state["binding"] = src.binding + state_path.parent.mkdir(parents=True, exist_ok=True) + atomic_write(state_path, json.dumps(new_state, indent=1)) + print(f"synced {header}") + print(describe(plan, created)) + + def sync_source(src, backlog: Path, state_path: Path, dry_run: bool, filt: dict | None = None, cap: int = 20, allow: int = 0) -> None: + if getattr(src, "create_only", False): + return sync_create_only(src, backlog, state_path, dry_run, filt) text = backlog.read_text() rows = parse_board(text) state = json.loads(state_path.read_text()) if state_path.exists() else {} @@ -116,6 +152,24 @@ def build_source(name: str, args): return RemindersSource(args.list_name or "Backlog") if name == "notion": return NotionSource(db=args.notion_db, parent=args.notion_parent) + if name == "notion-taskboard": + if not args.notion_taskboard_db: + sys.exit("notion-taskboard: set notion_taskboard_db in [sync] " + "(.kit.toml) or pass --notion-taskboard-db") + status_map = parse_map(args.notion_taskboard_status_map) + if not status_map and not args.notion_taskboard_status_default: + sys.exit("notion-taskboard: set notion_taskboard_status_map " + "(and/or notion_taskboard_status_default) in [sync]") + props = json.loads(args.notion_taskboard_props) \ + if args.notion_taskboard_props else None + types = json.loads(args.notion_taskboard_types) \ + if args.notion_taskboard_types else None + return NotionTaskBoardSource( + db=args.notion_taskboard_db, status_map=status_map, + status_default=args.notion_taskboard_status_default, + priority_map=parse_map(args.notion_taskboard_priority_map), + weight_map=parse_map(args.notion_taskboard_weight_map), + owner=args.notion_taskboard_owner, props=props, types=types) if name == "hermes": if not args.hermes_home: sys.exit("hermes: set hermes_home in [sync] (.kit.toml) or pass " @@ -153,6 +207,26 @@ def main(argv=None): help="Notion page id to create the board under (bootstrap)") ap.add_argument("--hermes-target") ap.add_argument("--hermes-home") + ap.add_argument("--notion-taskboard-db", + help="target Notion database id for the one-way " + "insert-only Task Board push") + ap.add_argument("--notion-taskboard-status-map", + help="board-state->option map, e.g. " + "'queued=Backlog,executing=In progress'") + ap.add_argument("--notion-taskboard-status-default", + help="Status option for board states not in the map") + ap.add_argument("--notion-taskboard-priority-map", + help="tag->Priority map, e.g. 'u-hi=P0,u-mid=P1,u-lo=P2'") + ap.add_argument("--notion-taskboard-weight-map", + help="tag->Weight map, e.g. 'f-hi=2,f-mid=5,f-lo=13'") + ap.add_argument("--notion-taskboard-owner", + help="Owner value (people-prop user id by default)") + ap.add_argument("--notion-taskboard-props", + help="JSON overriding prop names " + "{title,status,priority,weight,owner,notes}") + ap.add_argument("--notion-taskboard-types", + help="JSON overriding prop types " + "{status,priority,weight,owner}") ap.add_argument("--multica-url", help="Multica server base URL") ap.add_argument("--multica-workspace", help="Multica workspace UUID") ap.add_argument("--multica-project", help="Multica project UUID") diff --git a/lib/sync/docs/proof-of-done.md b/lib/sync/docs/proof-of-done.md index 69bc1df3..4157311e 100644 --- a/lib/sync/docs/proof-of-done.md +++ b/lib/sync/docs/proof-of-done.md @@ -93,6 +93,34 @@ rollback: per spoke, delete the Reminders list / trash the Notion DB / `~/.cache/backlog-sync/.state.json`. The board file is only modified by reverse-flow events. +## SPEC-003, one-way insert-only Task Board push (2026-07-18, ID-138) + +New posture alongside the two-way mesh: `notion-taskboard` app, create-only, +write-only sink. Contract proven by fixtures only, NO live team-board writes +during build (a live smoke against the real Task Board is a deliberate operator +step, per the DAG task's no-live-writes rule). + +| When | Command | Exit | Verdict | +|---|---|---|---| +| 2026-07-18 | `bash tests/test-sync.sh` | 0 | 83 passed (was 76 pre-change + 12 new taskboard cases; two-way suites unchanged, so the create-only path has zero blast radius on Reminders/Notion/Hermes/Multica) | +| 2026-07-18 | `pytest --cov` on changed modules | 0 | `sources/notion_taskboard.py` 87% (uncovered = real-subprocess `_run_ntn` + resolve fallbacks); every changed line in `sync_core`/`backlog_sync` covered | +| 2026-07-18 | CLI e2e via `main()` (fake ntn, temp board) | 0 | `--apps notion-taskboard ...`: DF-1 (queued,#u-hi) → one page Status=Backlog Priority=P0; DF-2 (dropped) skipped; board file byte-identical | + +Negative controls (all in `tests/test_notion_taskboard.py`): +`test_dropped_row_is_never_pushed` (dropped → no create), +`test_already_pushed_row_is_frozen` + the idempotent-rerun assertion in +`test_create_only_path_never_writes_board_and_is_idempotent` (a row in the +state map is never re-pushed, so team edits are never overwritten; the board +file is byte-identical before/after), +`test_unmapped_status_without_default_errors` (unmapped state fails closed, no +silent column guess), `test_skip_tag_down_filter` (skip-tag rows not pushed), +`test_build_source_requires_db` / `_requires_status_map_or_default` (missing +config fails closed before any network call). + +rollback: create-only never touches the board file; to undo a live push, trash +the created pages in Notion and delete +`~/.cache/backlog-sync//notion-taskboard.state.json`. + ## Reproduce ``` diff --git a/lib/sync/docs/specs/SPEC-003-oneway-create-push.md b/lib/sync/docs/specs/SPEC-003-oneway-create-push.md new file mode 100644 index 00000000..611ab96e --- /dev/null +++ b/lib/sync/docs/specs/SPEC-003-oneway-create-push.md @@ -0,0 +1,104 @@ +# SPEC-003, One-way insert-only push to a foreign team board + +Status: DRAFT 2026-07-18. Implements ops-toolkit board row ID-138 (design +locked 2026-06-16). Extends the SPEC-001 engine with a third posture that sits +alongside the two-way mesh, it does not modify it. + +## Problem + +ID-138 asks for the DISTRIBUTE half of the work-radar: push a repo's classified +`BACKLOG.md` rows OUT to a per-group sink so a team can see the work. The v1 +sink is the Dwarves **Notion Task Board**, a foreign, team-OWNED database with +its own schema (`Task` / `Status` / `Priority` / `Weight` / `Owner` / `Notes`) +and team members who edit the cards after they land. + +The existing `NotionSource` (SPEC-001) cannot serve this: + +- it is **two-way** (hub-wins), so it would revert a team member's status edit + on the next run; +- it **mutates the target schema** (`_bind_existing` PATCHes in board-keyword + `Status`/`Notes`/`Tags` props), which is destructive on a team board; +- it maps status to **board keywords** (`queued`, `executing`, ...), not the + team board's own option names (`Backlog`, `In progress`, ...); +- it has no `Priority`/`Weight`/`Owner` mapping. + +## Locked design (2026-06-16, honored verbatim) + +- **One-way**: the board is the source of truth; the sink is never read for + merge and the board file is never written. +- **Insert-only**: fields are set ONLY on page-create; a row already pushed + (recorded in the local sync-state map) is never touched again, so team edits + are never overwritten. +- **Identity map** = a local sync-state file (the Task Board's own ID column is + a read-only auto-increment, it cannot hold `DF-NN`). +- **Field map**: `title -> Task`, `status -> Status` + (`queued->Backlog`, `executing->In progress`, `parked->Waiting`, + `shipped->Done`, `dropped->skip`), `#u-* -> Priority` + (`hi->P0`, `mid->P1`, `lo->P2`), `#f-* -> Weight` + (`hi->2`, `mid->5`, `lo->13`, optional), `notes -> Notes`, `Owner = Han`. +- **Trigger**: a manual full-reconcile command (a hook on `board set` is a + later deploy step, not v1 machinery). +- **Notify**: Discord webhook on `shipped` is deferred to v1.1 (it needs + update-detection, which insert-only v1 deliberately does not do). +- **Out of v1**: family group, Apple Reminders, Hermes board, two-way sync. + +## Design + +A source declares `create_only = True`. The engine then runs a dedicated +planner, `plan_create_only`, instead of the two-way `plan_sync`, and a +dedicated apply path, `sync_create_only`, that never writes the board file. +The two-way path and its four live adapters (Reminders/Notion/Hermes/Multica) +are untouched: `create_only` defaults absent, so their behavior and tests are +unchanged. + +### `plan_create_only(rows, state, skip_kw, filt)` (pure) + +For each board row: skip if its `bid` is already in `state["map"]` (already +pushed), skip if its status is in `skip_kw` (default `{"dropped"}`), skip if it +fails the audience filter (`in_scope`). Everything else becomes one +`src_create`. No `src_set_*`, `board_*`, or `tombstone` is ever produced. + +### `NotionTaskBoardSource` (adapter) + +- `create_only = True`, `sync_fields = False`, `name = "notion-taskboard"`. +- `read()` returns `[]`: the sink is write-only, we never read the team board. +- `ensure_binding()` resolves the target database's `data_source_id` (a benign + read, needed as the page-create parent); it NEVER PATCHes the schema. +- `apply()` handles only `plan.src_create`: it POSTs one page per row with + properties built from the config-driven field map (status/priority/weight + values are the team board's OWN option names, so no schema mutation occurs). +- `skip_kw` defaults to `{"dropped"}`; a `status_default` (config) catches any + board state not in the status map, else a create for an unmapped state is a + hard, guided error (never a silent guess). + +### Config (`.kit.toml [sync]`, resolved in `board.sh cmd_sync`) + +`apps` includes `notion-taskboard`; keys: `notion_taskboard_db` (required), +`notion_taskboard_status_map` (required), `notion_taskboard_status_default`, +`notion_taskboard_priority_map`, `notion_taskboard_weight_map`, +`notion_taskboard_owner`, and per-prop name/type overrides. Tenant IDs (the +Task Board UUID, Han's Notion person id) live ONLY in the consumer repo's +`.kit.toml`, never in this (public) repo. + +## Test plan + +Fake-`ntn` transport, no network, no live writes. + +| # | Case | Assert | +|---|---|---| +| 1 | create pushes mapped fields | Task/Status/Notes/Priority set to the mapped option names; parent is the resolved data_source_id | +| 2 | negative: `dropped` never pushed | a `dropped` row yields no `src_create` | +| 3 | negative: already-pushed row frozen | a `bid` in `state["map"]` yields no `src_create` (idempotent re-run) | +| 4 | `#u-*`/`#f-*` -> Priority/Weight | priority/weight options derived from tags | +| 5 | skip-tag down-filter | a row carrying a skip tag is not pushed | +| 6 | unmapped status + no default | hard SystemExit with guidance | +| 7 | status_default catches unmapped | maps to the default option | +| 8 | board file never written | create-only path performs no board write | +| 9 | missing required config | build_source SystemExit naming the missing key | + +## Verification + +`bash tests/test-sync.sh` (adds the new adapter + planner cases). Live +activation against the real Task Board is an operator step (dfoundation +`.kit.toml` + a dry-run then real `board sync --apps notion-taskboard`), not +run here (no live team-board writes during build). diff --git a/lib/sync/sources/notion_taskboard.py b/lib/sync/sources/notion_taskboard.py new file mode 100644 index 00000000..8c9417d0 --- /dev/null +++ b/lib/sync/sources/notion_taskboard.py @@ -0,0 +1,176 @@ +"""Notion Task Board sink: ONE-WAY, insert-only push of board rows to a +foreign, team-OWNED Notion board. Drives the `ntn` CLI (keychain auth, Han's +Notion rule). See docs/specs/SPEC-003-oneway-create-push.md (ID-138). + +Insert-only by contract: the team owns each card after it lands, so the sink +sets fields on page-create and never touches them again (the local sync-state +map is the identity index; a bid already in the map is never re-pushed). Status +/ Priority / Weight are mapped to the team board's OWN option names via config, +so the sink never mutates the team schema. + +Unlike the two-way NotionSource, this adapter never reads the board for merge +(`read()` returns []) and never PATCHes the target schema. `ensure_binding` +does ONE benign read to resolve the data_source_id (the page-create parent). +""" + +import json +import subprocess + +from sync_core import extract_tags, strip_tags + +RICH_LIMIT = 2000 # Notion rich_text element content cap + + +def _run_ntn(args: list, data: dict | None = None): + cmd = ["ntn", *args] + stdin = None + if data is not None: + cmd += ["-d", "@-"] + stdin = json.dumps(data) + r = subprocess.run(cmd, input=stdin, capture_output=True, text=True, + timeout=120) + if r.returncode != 0: + raise SystemExit( + f"ntn {' '.join(args[:3])} failed: {r.stderr.strip()[:500]}") + return json.loads(r.stdout) if r.stdout.strip() else {} + + +def _rich(text: str) -> list: + chunks = [text[i:i + RICH_LIMIT] for i in range(0, len(text), RICH_LIMIT)] + return [{"type": "text", "text": {"content": c}} for c in chunks[:100]] \ + or [{"type": "text", "text": {"content": ""}}] + + +def parse_map(spec: str | None) -> dict: + """Parse a `k=v,k=v` config string into a dict (empty on blank).""" + out = {} + for pair in (spec or "").split(","): + pair = pair.strip() + if not pair: + continue + k, _, v = pair.partition("=") + if not v: + raise SystemExit(f"notion-taskboard: bad map entry {pair!r} " + "(want key=value)") + out[k.strip()] = v.strip() + return out + + +DEFAULT_PROPS = {"title": "Task", "status": "Status", "priority": "Priority", + "weight": "Weight", "owner": "Owner", "notes": "Notes"} +DEFAULT_TYPES = {"status": "status", "priority": "select", "weight": "number", + "owner": "people"} + + +class NotionTaskBoardSource: + name = "notion-taskboard" + create_only = True # engine runs plan_create_only + the write-only path + sync_fields = False + + def __init__(self, db=None, status_map=None, *, status_default=None, + priority_map=None, weight_map=None, owner=None, + props=None, types=None, skip_statuses=None, + binding=None, runner=_run_ntn): + self.db = db + self.status_map = status_map or {} + self.status_default = status_default + self.priority_map = priority_map or {} + self.weight_map = weight_map or {} + self.owner = owner + self.props = {**DEFAULT_PROPS, **(props or {})} + self.types = {**DEFAULT_TYPES, **(types or {})} + self.skip_kw = set(skip_statuses) if skip_statuses else {"dropped"} + self.binding = binding or {} + self.runner = runner + + # --- binding (resolve the data source; no schema mutation) -------------- + + def ensure_binding(self) -> dict: + if self.binding.get("ds_id"): + return self.binding + if not self.db: + raise SystemExit( + "notion-taskboard: no target. Set notion_taskboard_db in " + "[sync] (.kit.toml) or pass --notion-taskboard-db.") + self.binding = {"db_id": self.db, "ds_id": self._resolve_ds(self.db)} + return self.binding + + def _resolve_ds(self, db_id: str) -> str: + resp = self.runner(["datasources", "resolve", db_id, "--json"]) + if isinstance(resp, list): + return resp[0]["id"] if isinstance(resp[0], dict) else resp[0] + for key in ("data_sources", "results"): + if resp.get(key): + first = resp[key][0] + return first["id"] if isinstance(first, dict) else first + return resp["id"] + + # --- read (write-only sink: nothing to read) --------------------------- + + def read(self) -> list: + return [] + + # --- property mapping -------------------------------------------------- + + def _status_option(self, kw: str) -> str: + name = self.status_map.get(kw, self.status_default) + if name is None: + raise SystemExit( + f"notion-taskboard: no Status mapping for board state {kw!r}; " + "add it to notion_taskboard_status_map or set " + "notion_taskboard_status_default.") + return name + + def _typed_value(self, kind: str, name: str) -> dict: + t = self.types.get(kind) + if t == "status": + return {"status": {"name": name}} + if t == "select": + return {"select": {"name": name}} + if t == "number": + try: + num = float(name) + except ValueError: + raise SystemExit(f"notion-taskboard: {kind} value {name!r} is " + "not a number (types set number).") + return {"number": int(num) if num.is_integer() else num} + if t == "people": + return {"people": [{"id": name}]} + if t == "rich_text": + return {"rich_text": _rich(name)} + raise SystemExit(f"notion-taskboard: unknown prop type {t!r} for {kind}") + + def _tag_value(self, tags: list, mapping: dict) -> str | None: + for tag in tags: + if tag in mapping: + return mapping[tag] + return None + + def _page_props(self, title: str, body: str, kw: str) -> dict: + out = {self.props["title"]: {"title": _rich(title)}, + self.props["status"]: self._typed_value( + "status", self._status_option(kw)), + self.props["notes"]: {"rich_text": _rich(strip_tags(body))}} + tags = extract_tags(body) + prio = self._tag_value(tags, self.priority_map) + if prio is not None: + out[self.props["priority"]] = self._typed_value("priority", prio) + weight = self._tag_value(tags, self.weight_map) + if weight is not None: + out[self.props["weight"]] = self._typed_value("weight", weight) + if self.owner: + out[self.props["owner"]] = self._typed_value("owner", self.owner) + return out + + # --- apply (insert-only) ----------------------------------------------- + + def apply(self, plan, assigned: dict, rows_after: dict) -> dict: + b = self.ensure_binding() + created = {} + for bid, title, body, kw in plan.src_create: + resp = self.runner(["api", "v1/pages", "-X", "POST"], { + "parent": {"type": "data_source_id", + "data_source_id": b["ds_id"]}, + "properties": self._page_props(title, body, kw)}) + created[bid] = resp["id"] + return created diff --git a/lib/sync/sync_core.py b/lib/sync/sync_core.py index 5f7d2866..0d50b5b2 100644 --- a/lib/sync/sync_core.py +++ b/lib/sync/sync_core.py @@ -18,7 +18,13 @@ CLOSED_HEADING = "## Recently closed" TABLE_HEADER = "| ID | Item | Notes & source | Status |" TABLE_RULE = "|---|---|---|---|" -TITLE_RE = re.compile(r"^(ID-\d+)\s*(?:[·:, -]\s*)?(.*)$") +# Any repo-prefixed board id (ID-, DF-, BK-, ...); the documented kit +# convention is `[A-Z]+-[0-9]+`. Generalized from the ID-only original so the +# engine parses every adopted repo's board (SPEC-003 / ID-138: dfoundation +# DF-NN rows). ID- boards still match, so existing behavior is unchanged. +ID_TOKEN = r"[A-Z][A-Z0-9]*-\d+" +ROW_ID_RE = re.compile(r"^\| (" + ID_TOKEN + r") \|") +TITLE_RE = re.compile(r"^(" + ID_TOKEN + r")\s*(?:[·:, -]\s*)?(.*)$") CELL_SPLIT = re.compile(r"(? dict[str, Row]: rows: dict[str, Row] = {} for i, line in enumerate(text.splitlines()): - if not line.startswith("| ID-"): + if not ROW_ID_RE.match(line): continue cells = split_row(line) if not cells: continue rid, item, notes, status = cells - if not re.fullmatch(r"ID-\d+", rid): + if not re.fullmatch(ID_TOKEN, rid): continue kw = status.split()[0].lower() if status.split() else "" if rid in rows: @@ -283,6 +289,31 @@ def plan_sync(rows: dict, items: list, state: dict, return p +def plan_create_only(rows: dict, state: dict, skip_kw: set | None = None, + filt: dict | None = None) -> Plan: + """One-way, insert-only plan for a write-only sink (SPEC-003). + + Emits a `src_create` for every in-scope board row NOT already recorded in + the local sync-state map, skipping rows whose status is in `skip_kw` + (default `{"dropped"}`). Never updates, tombstones, or touches the board: + the map is the identity index and a row is pushed exactly once, so a team + member's later edits on the sink are never overwritten. + """ + p = Plan() + skip = skip_kw if skip_kw is not None else {"dropped"} + known = set(state.get("map", {})) + for bid, row in rows.items(): + if bid in known: + continue + if row.status_kw in skip: + continue + if not in_scope(row, filt): + continue + p.src_create.append((bid, title_for(bid, row.item), row.notes, + row.status_kw)) + return p + + # --- board apply ------------------------------------------------------------- diff --git a/lib/sync/tests/test_notion_taskboard.py b/lib/sync/tests/test_notion_taskboard.py new file mode 100644 index 00000000..ff9cc0ec --- /dev/null +++ b/lib/sync/tests/test_notion_taskboard.py @@ -0,0 +1,337 @@ +"""One-way Task Board sink tests against a fake ntn transport (no network, +no live writes). SPEC-003 test plan.""" + +import json +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from sources.notion_taskboard import NotionTaskBoardSource, parse_map # noqa: E402 +from sync_core import Plan, Row, plan_create_only # noqa: E402 + + +class FakeNtn: + def __init__(self, responses=None): + self.calls = [] + self.responses = responses or {} + self.page_seq = 0 + + def __call__(self, args, data=None): + self.calls.append((tuple(args), data)) + key = " ".join(args[:3]) + for prefix, resp in self.responses.items(): + if key.startswith(prefix): + return resp(data) if callable(resp) else resp + if args[0] == "api" and args[1] == "v1/pages" and "-X" in args: + self.page_seq += 1 + return {"id": f"pg{self.page_seq}"} + return {} + + def page_bodies(self): + return [d for a, d in self.calls + if a[0] == "api" and a[1] == "v1/pages" and d is not None] + + +DF_STATUS = {"queued": "Backlog", "executing": "In progress", + "parked": "Waiting", "shipped": "Done"} +DF_PRIORITY = {"u-hi": "P0", "u-mid": "P1", "u-lo": "P2"} +DF_WEIGHT = {"f-hi": "2", "f-mid": "5", "f-lo": "13"} + + +def binding(): + return {"db_id": "db1", "ds_id": "ds1"} + + +def make_src(**kw): + defaults = dict(db="db1", status_map=dict(DF_STATUS), binding=binding(), + runner=FakeNtn()) + defaults.update(kw) + src = NotionTaskBoardSource(**defaults) + return src, src.runner + + +# --- parse_map ----------------------------------------------------------- + + +def test_parse_map_basic_and_blank(): + assert parse_map("queued=Backlog, executing=In progress") == { + "queued": "Backlog", "executing": "In progress"} + assert parse_map("") == {} and parse_map(None) == {} + + +def test_parse_map_rejects_valueless_entry(): + with pytest.raises(SystemExit, match="bad map entry"): + parse_map("queued") + + +# --- create-only planner (case 2, 3, 5) ---------------------------------- + + +def rows(*specs): + return {bid: Row(bid, item, kw, i, notes) + for i, (bid, item, kw, notes) in enumerate(specs)} + + +def test_dropped_row_is_never_pushed(): + r = rows(("DF-1", "keep", "queued", ""), ("DF-2", "gone", "dropped", "")) + plan = plan_create_only(r, {}) + assert [c[0] for c in plan.src_create] == ["DF-1"] + + +def test_already_pushed_row_is_frozen(): + r = rows(("DF-1", "a", "queued", ""), ("DF-2", "b", "executing", "")) + state = {"map": {"DF-1": {"rid": "pg1"}}} + plan = plan_create_only(r, state) + assert [c[0] for c in plan.src_create] == ["DF-2"] + + +def test_skip_tag_down_filter(): + r = rows(("DF-1", "pub", "queued", "#ops"), + ("DF-2", "sec", "queued", "#family")) + plan = plan_create_only(r, {}, filt={"skip_tags": {"family"}}) + assert [c[0] for c in plan.src_create] == ["DF-1"] + + +def test_custom_skip_kw_extends_dropped(): + r = rows(("DF-1", "a", "shipped", ""), ("DF-2", "b", "queued", "")) + plan = plan_create_only(r, {}, skip_kw={"dropped", "shipped"}) + assert [c[0] for c in plan.src_create] == ["DF-2"] + + +# --- apply: field mapping (case 1, 4) ------------------------------------ + + +def test_create_maps_all_fields(): + src, fake = make_src(priority_map=dict(DF_PRIORITY), + weight_map=dict(DF_WEIGHT), owner="user-han") + plan = Plan(src_create=[("DF-9", "DF-9 · Fix vault", "harden it #u-hi #f-mid", + "executing")]) + created = src.apply(plan, {}, {}) + assert created == {"DF-9": "pg1"} + body = fake.page_bodies()[0] + assert body["parent"] == {"type": "data_source_id", "data_source_id": "ds1"} + props = body["properties"] + assert props["Task"]["title"][0]["text"]["content"] == "DF-9 · Fix vault" + assert props["Status"] == {"status": {"name": "In progress"}} + assert props["Priority"] == {"select": {"name": "P0"}} + assert props["Weight"] == {"number": 5} + assert props["Owner"] == {"people": [{"id": "user-han"}]} + # tags belong to Priority/Weight, not the Notes text + assert "#u-hi" not in props["Notes"]["rich_text"][0]["text"]["content"] + + +def test_priority_and_weight_omitted_when_no_tag(): + src, fake = make_src(priority_map=dict(DF_PRIORITY), + weight_map=dict(DF_WEIGHT)) + plan = Plan(src_create=[("DF-1", "DF-1 · plain", "no tags here", "queued")]) + src.apply(plan, {}, {}) + props = fake.page_bodies()[0]["properties"] + assert "Priority" not in props and "Weight" not in props + assert props["Status"] == {"status": {"name": "Backlog"}} + + +def test_prop_and_type_overrides(): + src, fake = make_src( + props={"status": "Stage", "weight": "Points"}, + types={"status": "select", "weight": "select"}, + weight_map=dict(DF_WEIGHT)) + plan = Plan(src_create=[("DF-1", "t", "x #f-hi", "queued")]) + src.apply(plan, {}, {}) + props = fake.page_bodies()[0]["properties"] + assert props["Stage"] == {"select": {"name": "Backlog"}} + assert props["Points"] == {"select": {"name": "2"}} + + +# --- status default / hard error (case 6, 7) ----------------------------- + + +def test_unmapped_status_without_default_errors(): + src, _ = make_src() # DF_STATUS has no "claimed" + plan = Plan(src_create=[("DF-1", "t", "x", "claimed")]) + with pytest.raises(SystemExit, match="no Status mapping"): + src.apply(plan, {}, {}) + + +def test_status_default_catches_unmapped_state(): + src, fake = make_src(status_default="Backlog") + plan = Plan(src_create=[("DF-1", "t", "x", "speccing")]) + src.apply(plan, {}, {}) + assert fake.page_bodies()[0]["properties"]["Status"] == { + "status": {"name": "Backlog"}} + + +# --- write-only sink (read + binding) ------------------------------------ + + +def test_read_returns_nothing(): + src, fake = make_src() + assert src.read() == [] + assert fake.calls == [] # never touches the board + + +def test_binding_resolves_data_source_read_only(): + fake = FakeNtn({"datasources resolve db1": {"data_sources": [{"id": "ds9"}]}}) + src = NotionTaskBoardSource(db="db1", status_map=dict(DF_STATUS), + runner=fake) + b = src.ensure_binding() + assert b["ds_id"] == "ds9" + # only a resolve read happened; no PATCH, no page write + assert all("PATCH" not in a and a[:2] != ("api", "v1/pages") + for a, _ in fake.calls) + + +def test_unbound_without_db_exits_with_guidance(): + src = NotionTaskBoardSource(status_map=dict(DF_STATUS), runner=FakeNtn()) + with pytest.raises(SystemExit, match="notion_taskboard_db"): + src.ensure_binding() + + +def test_number_weight_rejects_non_numeric(): + src, _ = make_src(weight_map={"f-hi": "heavy"}) + plan = Plan(src_create=[("DF-1", "t", "x #f-hi", "queued")]) + with pytest.raises(SystemExit, match="not a number"): + src.apply(plan, {}, {}) + + +# --- engine path: board never written, state map is the identity (case 8) -- + +BOARD = ( + "| ID | Item | Notes & source | Status |\n" + "|---|---|---|---|\n" + "| DF-1 | Lock the vault | harden #u-hi | queued |\n" + "| DF-2 | Drop this | old idea | dropped |\n" +) + + +def test_create_only_path_never_writes_board_and_is_idempotent(tmp_path): + import backlog_sync + + board = tmp_path / "BACKLOG.md" + board.write_text(BOARD) + state = tmp_path / "notion-taskboard.state.json" + before = board.read_text() + + src, fake = make_src(priority_map=dict(DF_PRIORITY)) + backlog_sync.sync_create_only(src, board, state, dry_run=False) + + # board file byte-identical: strictly one-way + assert board.read_text() == before + # exactly one page created (DF-1); DF-2 dropped is skipped + assert len(fake.page_bodies()) == 1 + saved = json.loads(state.read_text()) + assert list(saved["map"]) == ["DF-1"] + + # second run with the persisted state: nothing new (idempotent) + src2, fake2 = make_src(priority_map=dict(DF_PRIORITY)) + backlog_sync.sync_create_only(src2, board, state, dry_run=False) + assert fake2.page_bodies() == [] + assert board.read_text() == before + + +def test_create_only_dry_run_writes_nothing(tmp_path): + import backlog_sync + + board = tmp_path / "BACKLOG.md" + board.write_text(BOARD) + state = tmp_path / "s.json" + src, fake = make_src() + backlog_sync.sync_create_only(src, board, state, dry_run=True) + assert fake.page_bodies() == [] and not state.exists() + + +# --- build_source wiring (case 9) ---------------------------------------- + + +def args_ns(**over): + from types import SimpleNamespace + base = dict(notion_taskboard_db="db1", + notion_taskboard_status_map="queued=Backlog,executing=In progress", + notion_taskboard_status_default=None, + notion_taskboard_priority_map="u-hi=P0,u-mid=P1", + notion_taskboard_weight_map="f-hi=2,f-mid=5", + notion_taskboard_owner="user-han", + notion_taskboard_props=None, notion_taskboard_types=None) + base.update(over) + return SimpleNamespace(**base) + + +def test_build_source_wires_maps_from_flags(): + import backlog_sync + + src = backlog_sync.build_source("notion-taskboard", args_ns()) + assert isinstance(src, NotionTaskBoardSource) + assert src.status_map == {"queued": "Backlog", "executing": "In progress"} + assert src.priority_map == {"u-hi": "P0", "u-mid": "P1"} + assert src.weight_map == {"f-hi": "2", "f-mid": "5"} + assert src.owner == "user-han" + + +def test_build_source_requires_db(): + import backlog_sync + + with pytest.raises(SystemExit, match="notion_taskboard_db"): + backlog_sync.build_source("notion-taskboard", + args_ns(notion_taskboard_db=None)) + + +def test_build_source_requires_status_map_or_default(): + import backlog_sync + + with pytest.raises(SystemExit, match="notion_taskboard_status_map"): + backlog_sync.build_source( + "notion-taskboard", + args_ns(notion_taskboard_status_map=None, + notion_taskboard_status_default=None)) + + +def test_build_source_status_default_alone_is_enough(): + import backlog_sync + + src = backlog_sync.build_source( + "notion-taskboard", + args_ns(notion_taskboard_status_map=None, + notion_taskboard_status_default="Backlog")) + assert src.status_default == "Backlog" and src.status_map == {} + + +def test_build_source_props_and_types_json(): + import backlog_sync + + src = backlog_sync.build_source( + "notion-taskboard", + args_ns(notion_taskboard_props='{"status": "Stage"}', + notion_taskboard_types='{"status": "select"}')) + assert src.props["status"] == "Stage" and src.types["status"] == "select" + + +def test_cli_end_to_end_pushes_through_main(tmp_path, monkeypatch): + """Full CLI path: argparse -> dispatch -> create-only push, fake ntn.""" + import functools + + import backlog_sync + + board = tmp_path / "BACKLOG.md" + board.write_text(BOARD) + fake = FakeNtn() + monkeypatch.setattr( + backlog_sync, "NotionTaskBoardSource", + functools.partial(NotionTaskBoardSource, runner=fake, + binding=binding())) + backlog_sync.main([ + "--apps", "notion-taskboard", + "--backlog", str(board), + "--state-root", str(tmp_path / "state"), + "--notion-taskboard-db", "db1", + "--notion-taskboard-status-map", "queued=Backlog,executing=In progress", + "--notion-taskboard-priority-map", "u-hi=P0", + ]) + # DF-1 (queued, #u-hi) pushed; DF-2 (dropped) skipped; board untouched + bodies = fake.page_bodies() + assert len(bodies) == 1 + props = bodies[0]["properties"] + assert props["Status"] == {"status": {"name": "Backlog"}} + assert props["Priority"] == {"select": {"name": "P0"}} + assert board.read_text() == BOARD From 90d47801a9f46e93a413cf37dbc485be899ac67a Mon Sep 17 00:00:00 2001 From: Han Ngo Date: Sat, 18 Jul 2026 20:42:11 +0700 Subject: [PATCH 2/2] refactor(sync): apply review findings to the taskboard sink From kit:code-reviewer (architecture) + kit:advisor (critique): - No partial-batch duplicates: apply() checkpoints state after each create via an on_created callback, and a preflight validates the whole batch (maps resolve, options exist) before any POST. A mid-batch ntn failure never leaves a created page unrecorded, so the next run never re-pushes it. Preflight also runs before the dry-run return, so --dry-run surfaces config errors with zero writes. - Never auto-create a team option: preflight reads the board schema (never PATCHes it) and rejects any mapped option name not already on the board; select-type props no longer silently gain options on a typo. Prop types are auto-detected from the schema. - Two-way mesh stays ID-only: reverted TITLE_RE and gave parse_board a strict_id flag (default True). The one-way path reads with strict_id=False (any [A-Z]+-NNN) but never mints ids; warn_duplicate_ids takes the same flag so a DF- board still warns. - ensure_binding discards a cached binding for a different db_id (a repointed target never keeps pushing to the old board). - Dropped the unused skip_statuses param and the dead intake config key. --- docs/CHANGELOG.md | 10 +- .../oneway-create-push.md | 27 ++++ lib/board/board.sh | 7 +- lib/sync/README.md | 10 +- lib/sync/backlog_sync.py | 45 ++++--- lib/sync/docs/proof-of-done.md | 19 ++- .../docs/specs/SPEC-003-oneway-create-push.md | 37 +++++- lib/sync/sources/notion_taskboard.py | 109 ++++++++++++---- lib/sync/sync_core.py | 31 +++-- lib/sync/tests/test_notion_taskboard.py | 123 ++++++++++++++++++ 10 files changed, 348 insertions(+), 70 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 12e92f1b..a44c7700 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -12,9 +12,13 @@ All notable changes to dwarves-kit are documented here. never overwritten, and the local sync-state map is the identity index. Status/ Priority/Weight map to the team board's own option names via `.kit.toml [sync]` config, so the team schema is never mutated. The two-way mesh and its four live - adapters are untouched (`create_only` is a separate path). The engine's board - parser was generalized from `ID-` to any `[A-Z]+-\d+` prefix so it reads every - adopted repo's board (e.g. dfoundation `DF-NN`). + adapters are untouched (`create_only` is a separate path, and the two-way + board parser stays `ID-`-only; only the one-way READ path accepts any + `[A-Z]+-\d+` prefix, e.g. dfoundation `DF-NN`, and it never mints or writes + board ids). The sink validates every mapped option against the target's + schema before any create, so it never auto-creates an option on the team + board; state is checkpointed after each create so a mid-batch failure never + re-pushes a page. ### Changed - **`bridge` module folded into `sync`** (zero live consumers at fold time: no diff --git a/docs/implementation-notes/oneway-create-push.md b/docs/implementation-notes/oneway-create-push.md index 4e8146a5..2d742a5e 100644 --- a/docs/implementation-notes/oneway-create-push.md +++ b/docs/implementation-notes/oneway-create-push.md @@ -41,6 +41,33 @@ Why: the kit ships no tenant policy; the consumer (dfoundation) decides where claimed/speccing/validated land by setting `status_default` (or extending the map). Honors "don't re-design" while never silently dropping active work. +## 2026-07-18 review findings applied (kit:code-reviewer + kit:advisor) + +Two reviewers independently flagged the same landmines; fixes: +- **Partial-batch duplicates**: `apply()` now takes an `on_created(bid, rid)` + callback; `sync_create_only` checkpoints state after EACH create. A mid-batch + `ntn` failure no longer leaves created pages unrecorded (which would re-push + duplicates on a team board). Plus a `preflight(plan)` validates the whole + batch (status/priority/weight map + option existence) BEFORE any POST, so a + config error can't fail a run part-way. Preflight also runs before the + dry-run return, so `--dry-run` surfaces config errors with zero writes. +- **Select auto-create = silent schema mutation**: the "never mutate the team + schema" claim was false for select-type props (Notion auto-creates unknown + select options). Fixed by reading the schema in preflight and rejecting any + mapped option name not already on the board. This also auto-detects prop + TYPES from the schema (dropping the earlier "verify types manually" caveat). +- **Parser generalization was half-applied**: reverted `TITLE_RE` to `ID-`-only + and gave `parse_board` a `strict_id` flag (default True = two-way, ID-only). + The one-way path passes `strict_id=False`; `warn_duplicate_ids` gained the + same flag so a `DF-` board still gets dup warnings. Net: the two-way mesh is + byte-for-byte `ID-`-only again (its id-minting siblings stay consistent), and + only the create-only READ view widened. +- **Stale binding**: `ensure_binding` discards a cached binding whose `db_id` + differs from the configured target (a repointed `notion_taskboard_db` no + longer keeps pushing to the old board). +- Minor: dropped the unused `skip_statuses` ctor param and the dead + `notion_taskboard_intake` config key (a write-only sink has no intake path). + ## 2026-07-18 tenant IDs stay in the consumer repo Context: this repo (dwarves-kit) is public; the Task Board UUID and Han's diff --git a/lib/board/board.sh b/lib/board/board.sh index 80e52df2..f81cc6e6 100755 --- a/lib/board/board.sh +++ b/lib/board/board.sh @@ -739,9 +739,10 @@ cmd_sync() { v="$(kit_config_get sync.notion_db "")"; [ -n "$v" ] && args+=(--notion-db "$v") v="$(kit_config_get sync.notion_parent "")"; [ -n "$v" ] && args+=(--notion-parent "$v") # notion-taskboard: one-way, insert-only push to a foreign team board - # (SPEC-003). Its down-filter uses TOML-friendly underscore keys but the - # engine's --filter app token keeps the hyphenated adapter name. - for fk in only_tags skip_tags intake; do + # (SPEC-003). Down-filter only (a write-only sink has no intake path); the + # keys are TOML-friendly underscores but the engine's --filter app token + # keeps the hyphenated adapter name. + for fk in only_tags skip_tags; do v="$(kit_config_get "sync.notion_taskboard_${fk}" "")" [ -n "$v" ] && args+=(--filter "notion-taskboard:${fk}=${v}") done diff --git a/lib/sync/README.md b/lib/sync/README.md index e8ba41b9..2245f75d 100644 --- a/lib/sync/README.md +++ b/lib/sync/README.md @@ -137,9 +137,13 @@ identity index (a `bid` already pushed is never re-pushed), because the team board's own ID column is a read-only auto-increment that cannot hold `DF-NN`. Status / Priority / Weight are mapped to the team board's OWN option names via -config, so the sink never mutates the team schema (it does one benign read to -resolve the data source, never a PATCH). `dropped` rows are skipped; a state -absent from the map uses `status_default` or, if unset, is a hard guided error. +config, so the sink never mutates the team schema: it reads the schema (never +PATCHes it) and validates every mapped option name exists BEFORE any create, so +a typo'd map value is a hard error instead of a silently auto-created select +option. `dropped` rows are skipped; a state absent from the map uses +`status_default` or, if unset, is a hard guided error. Validation runs at +preflight, so `--dry-run` surfaces config errors with zero writes, and state is +checkpointed after each create so a mid-batch failure never re-pushes a page. ```toml [sync] diff --git a/lib/sync/backlog_sync.py b/lib/sync/backlog_sync.py index 6b764890..004abc60 100644 --- a/lib/sync/backlog_sync.py +++ b/lib/sync/backlog_sync.py @@ -26,8 +26,11 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) -from sync_core import (apply_board, build_state, describe, parse_board, # noqa: E402 - plan_create_only, plan_sync) +from sync_core import (ID_TOKEN, apply_board, build_state, describe, # noqa: E402 + parse_board, plan_create_only, plan_sync) + +# apps that push one-way and read boards with any repo prefix (not just ID-) +CREATE_ONLY_APPS = {"notion-taskboard"} from sources.hermes import HermesSource # noqa: E402 from sources.multica import MulticaSource # noqa: E402 from sources.notion import NotionSource # noqa: E402 @@ -50,13 +53,14 @@ def atomic_write(path: Path, text: str) -> None: raise -def warn_duplicate_ids(text: str) -> None: - ids = re.findall(r"^\| (ID-\d+) \|", text, flags=re.M) +def warn_duplicate_ids(text: str, strict_id: bool = True) -> None: + token = r"ID-\d+" if strict_id else ID_TOKEN + ids = re.findall(r"^\| (" + token + r") \|", text, flags=re.M) dups = sorted({i for i in ids if ids.count(i) > 1}) if dups: print(f"WARNING: duplicate board rows for {', '.join(dups)}; " "first occurrence wins, fix the board") - parsed = set(parse_board(text)) + parsed = set(parse_board(text, strict_id=strict_id)) broken = sorted(set(ids) - parsed - set(dups)) if broken: print(f"WARNING: malformed board rows (not 4 cells, invisible to " @@ -66,30 +70,38 @@ def warn_duplicate_ids(text: str) -> None: def sync_create_only(src, backlog: Path, state_path: Path, dry_run: bool, filt: dict | None = None) -> None: """One-way, insert-only push to a write-only sink (SPEC-003). The board - file is never written; the local state map is the identity index.""" + file is never written; the local state map is the identity index. State is + checkpointed after EACH create, so a mid-batch failure never re-pushes an + already-created page (no duplicate cards on a team-owned board).""" text = backlog.read_text() - rows = parse_board(text) + rows = parse_board(text, strict_id=False) state = json.loads(state_path.read_text()) if state_path.exists() else {} if hasattr(src, "binding") and state.get("binding"): src.binding = state["binding"] plan = plan_create_only(rows, state, skip_kw=getattr(src, "skip_kw", None), filt=filt) + # Validate the whole batch (maps resolve, options exist) BEFORE any write, + # so `--dry-run` surfaces config errors and a live run never dies part-way. + if hasattr(src, "preflight"): + src.preflight(plan) header = (f"{src.name}: {len(rows)} board rows, " f"{len(plan.src_create)} to create") if dry_run: print(f"dry-run {header}") print(describe(plan)) return - created = src.apply(plan, {}, rows) + new_map = dict(state.get("map", {})) - for bid, _t, _b, _kw in plan.src_create: - if bid in created: - new_map[bid] = {"rid": created[bid]} - new_state = {"map": new_map} - if getattr(src, "binding", None): - new_state["binding"] = src.binding state_path.parent.mkdir(parents=True, exist_ok=True) - atomic_write(state_path, json.dumps(new_state, indent=1)) + + def checkpoint(bid: str, rid: str) -> None: + new_map[bid] = {"rid": rid} + new_state = {"map": new_map} + if getattr(src, "binding", None): + new_state["binding"] = src.binding + atomic_write(state_path, json.dumps(new_state, indent=1)) + + created = src.apply(plan, {}, rows, on_created=checkpoint) print(f"synced {header}") print(describe(plan, created)) @@ -264,7 +276,8 @@ def main(argv=None): fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) except BlockingIOError: sys.exit("another backlog-sync run holds the lock; try again") - warn_duplicate_ids(args.backlog.read_text()) + strict_id = not (set(names) & CREATE_ONLY_APPS) + warn_duplicate_ids(args.backlog.read_text(), strict_id=strict_id) # one-time migration from the pre-kit single-source tool's state path rem_state = state_dir / "reminders.state.json" diff --git a/lib/sync/docs/proof-of-done.md b/lib/sync/docs/proof-of-done.md index 4157311e..edb29242 100644 --- a/lib/sync/docs/proof-of-done.md +++ b/lib/sync/docs/proof-of-done.md @@ -102,18 +102,25 @@ step, per the DAG task's no-live-writes rule). | When | Command | Exit | Verdict | |---|---|---|---| -| 2026-07-18 | `bash tests/test-sync.sh` | 0 | 83 passed (was 76 pre-change + 12 new taskboard cases; two-way suites unchanged, so the create-only path has zero blast radius on Reminders/Notion/Hermes/Multica) | -| 2026-07-18 | `pytest --cov` on changed modules | 0 | `sources/notion_taskboard.py` 87% (uncovered = real-subprocess `_run_ntn` + resolve fallbacks); every changed line in `sync_core`/`backlog_sync` covered | +| 2026-07-18 | `bash tests/test-sync.sh` | 0 | 89 passed (76 pre-change + 13 new taskboard cases; two-way suites unchanged, so the create-only path has zero blast radius on Reminders/Notion/Hermes/Multica) | +| 2026-07-18 | `pytest --cov` on changed modules | 0 | `sources/notion_taskboard.py` ~90% (uncovered = real-subprocess `_run_ntn` + resolve list-fallback); every changed line in `sync_core`/`backlog_sync` covered | | 2026-07-18 | CLI e2e via `main()` (fake ntn, temp board) | 0 | `--apps notion-taskboard ...`: DF-1 (queued,#u-hi) → one page Status=Backlog Priority=P0; DF-2 (dropped) skipped; board file byte-identical | +| 2026-07-18 | kit:code-reviewer (architecture) + kit:advisor (critique, fable) then re-run | 0 | 6 findings applied: preflight validation + per-create checkpoint (no partial-batch duplicates), schema-option validation (no silent select auto-create), `strict_id` parser split (two-way stays ID-only), stale-binding db check, dead intake/skip_statuses removed; suite 89 green | Negative controls (all in `tests/test_notion_taskboard.py`): `test_dropped_row_is_never_pushed` (dropped → no create), `test_already_pushed_row_is_frozen` + the idempotent-rerun assertion in `test_create_only_path_never_writes_board_and_is_idempotent` (a row in the -state map is never re-pushed, so team edits are never overwritten; the board -file is byte-identical before/after), -`test_unmapped_status_without_default_errors` (unmapped state fails closed, no -silent column guess), `test_skip_tag_down_filter` (skip-tag rows not pushed), +state map is never re-pushed, so team edits are never overwritten; board file +byte-identical), +`test_partial_batch_failure_checkpoints_and_never_duplicates` (2nd create fails +after the 1st succeeds → 1st is persisted, recovery run pushes only the 2nd, +never a duplicate), +`test_preflight_rejects_unknown_select_option` (a typo'd option is a hard error, +never an auto-created team option), +`test_preflight_surfaces_unmapped_status_on_dry_run` (dry-run fails closed with +zero writes), `test_unmapped_status_without_default_errors`, +`test_skip_tag_down_filter`, `test_stale_binding_for_a_different_db_is_rediscarded`, `test_build_source_requires_db` / `_requires_status_map_or_default` (missing config fails closed before any network call). diff --git a/lib/sync/docs/specs/SPEC-003-oneway-create-push.md b/lib/sync/docs/specs/SPEC-003-oneway-create-push.md index 611ab96e..73b86953 100644 --- a/lib/sync/docs/specs/SPEC-003-oneway-create-push.md +++ b/lib/sync/docs/specs/SPEC-003-oneway-create-push.md @@ -63,13 +63,31 @@ fails the audience filter (`in_scope`). Everything else becomes one - `create_only = True`, `sync_fields = False`, `name = "notion-taskboard"`. - `read()` returns `[]`: the sink is write-only, we never read the team board. - `ensure_binding()` resolves the target database's `data_source_id` (a benign - read, needed as the page-create parent); it NEVER PATCHes the schema. -- `apply()` handles only `plan.src_create`: it POSTs one page per row with - properties built from the config-driven field map (status/priority/weight - values are the team board's OWN option names, so no schema mutation occurs). -- `skip_kw` defaults to `{"dropped"}`; a `status_default` (config) catches any - board state not in the status map, else a create for an unmapped state is a - hard, guided error (never a silent guess). + read, needed as the page-create parent); it NEVER PATCHes the schema. A + cached binding for a different `db_id` is discarded (a repointed target never + keeps pushing to the old board). +- `preflight(plan)` reads the schema ONCE (read, never PATCH) to learn each + prop's type and to validate that every mapped option name (status, priority, + and select-type weight) already EXISTS on the board. An unknown option is a + hard error, not a silent auto-create, so the "never mutate the team schema" + invariant holds even for select-type props (which Notion would otherwise + auto-extend). Preflight runs before the dry-run return too, so `--dry-run` + surfaces every config error with zero writes. +- `apply()` handles only `plan.src_create`: it POSTs one page per row and calls + an `on_created(bid, rid)` callback after each success. The engine checkpoints + state on that callback, so a mid-batch failure (rate limit, network) never + leaves a created page unrecorded, i.e. never re-pushes a duplicate next run. +- `skip_kw` = `{"dropped"}`; a `status_default` (config) catches any board + state not in the status map, else it is a hard, guided error (never a guess). + +### Parser scope (`strict_id`) + +`parse_board(strict_id=True)` (the two-way mesh) recognizes ONLY `ID-NNN`, so +its id-minting siblings (`next_id`, `apply_board`, `warn_duplicate_ids`) stay +consistent and the two-way path is byte-for-byte unchanged. The one-way push +reads with `strict_id=False` (any `[A-Z]+-NNN`, e.g. `DF-NN`) but never mints +or writes ids; `warn_duplicate_ids` is called with the matching prefix so a +`DF-` board still gets its duplicate/malformed-row warnings. ### Config (`.kit.toml [sync]`, resolved in `board.sh cmd_sync`) @@ -95,6 +113,11 @@ Fake-`ntn` transport, no network, no live writes. | 7 | status_default catches unmapped | maps to the default option | | 8 | board file never written | create-only path performs no board write | | 9 | missing required config | build_source SystemExit naming the missing key | +| 10 | unknown mapped option | preflight raises (no auto-created team option) | +| 11 | dry-run validates | unmapped status fails on `--dry-run` before any write | +| 12 | partial-batch failure | first create checkpointed; re-run pushes only the rest, never duplicates | +| 13 | repointed target db | cached binding for a stale db_id is rediscarded | +| 14 | `DF-` duplicate row | `warn_duplicate_ids(strict_id=False)` catches it | ## Verification diff --git a/lib/sync/sources/notion_taskboard.py b/lib/sync/sources/notion_taskboard.py index 8c9417d0..91981c81 100644 --- a/lib/sync/sources/notion_taskboard.py +++ b/lib/sync/sources/notion_taskboard.py @@ -5,12 +5,13 @@ Insert-only by contract: the team owns each card after it lands, so the sink sets fields on page-create and never touches them again (the local sync-state map is the identity index; a bid already in the map is never re-pushed). Status -/ Priority / Weight are mapped to the team board's OWN option names via config, -so the sink never mutates the team schema. +/ Priority / Weight are mapped to the team board's OWN option names via config. -Unlike the two-way NotionSource, this adapter never reads the board for merge -(`read()` returns []) and never PATCHes the target schema. `ensure_binding` -does ONE benign read to resolve the data_source_id (the page-create parent). +Never mutates the team schema: `ensure_binding` resolves the data source +(a benign read), `preflight` reads the schema once to (a) learn each prop's +type and (b) validate that every configured option name already exists on the +board, so a typo'd map value is a hard error instead of a silently +auto-created select option. `read()` returns [] (the sink is write-only). """ import json @@ -58,8 +59,11 @@ def parse_map(spec: str | None) -> dict: DEFAULT_PROPS = {"title": "Task", "status": "Status", "priority": "Priority", "weight": "Weight", "owner": "Owner", "notes": "Notes"} +# Fallback types used only when the schema has not been read (e.g. unit tests +# that inject a binding directly); a real run derives types from the schema. DEFAULT_TYPES = {"status": "status", "priority": "select", "weight": "number", "owner": "people"} +OPTION_TYPES = ("select", "status") class NotionTaskBoardSource: @@ -69,8 +73,7 @@ class NotionTaskBoardSource: def __init__(self, db=None, status_map=None, *, status_default=None, priority_map=None, weight_map=None, owner=None, - props=None, types=None, skip_statuses=None, - binding=None, runner=_run_ntn): + props=None, types=None, binding=None, runner=_run_ntn): self.db = db self.status_map = status_map or {} self.status_default = status_default @@ -78,20 +81,24 @@ def __init__(self, db=None, status_map=None, *, status_default=None, self.weight_map = weight_map or {} self.owner = owner self.props = {**DEFAULT_PROPS, **(props or {})} - self.types = {**DEFAULT_TYPES, **(types or {})} - self.skip_kw = set(skip_statuses) if skip_statuses else {"dropped"} + self.types = {**(types or {})} # explicit overrides only + self.skip_kw = {"dropped"} self.binding = binding or {} self.runner = runner + self._schema_types: dict = {} + self._prop_opts: dict = {} + self._schema_loaded = False - # --- binding (resolve the data source; no schema mutation) -------------- + # --- binding + schema (reads only; never a schema PATCH) ---------------- def ensure_binding(self) -> dict: - if self.binding.get("ds_id"): - return self.binding + if self.binding.get("ds_id") and self.binding.get("db_id") == self.db: + return self.binding # cached (possibly restored from state) if not self.db: raise SystemExit( "notion-taskboard: no target. Set notion_taskboard_db in " "[sync] (.kit.toml) or pass --notion-taskboard-db.") + # a stale cached binding for a different db is discarded self.binding = {"db_id": self.db, "ds_id": self._resolve_ds(self.db)} return self.binding @@ -105,11 +112,61 @@ def _resolve_ds(self, db_id: str) -> str: return first["id"] if isinstance(first, dict) else first return resp["id"] + def _load_schema(self) -> None: + if self._schema_loaded: + return + b = self.ensure_binding() + schema = self.runner(["api", f"v1/data_sources/{b['ds_id']}"]) + props = schema.get("properties", {}) + for kind, pname in self.props.items(): + p = props.get(pname) + if p is None: + continue # optional props (owner/priority/weight) may be absent + ptype = p.get("type") + self._schema_types[kind] = ptype + if ptype in OPTION_TYPES: + self._prop_opts[pname] = { + o["name"] for o in p.get(ptype, {}).get("options", [])} + self._schema_loaded = True + + def _eff_type(self, kind: str) -> str: + return (self.types.get(kind) or self._schema_types.get(kind) + or DEFAULT_TYPES.get(kind)) + # --- read (write-only sink: nothing to read) --------------------------- def read(self) -> list: return [] + # --- validation (preflight, before any write) -------------------------- + + def preflight(self, plan) -> None: + self._load_schema() + for bid, _title, body, kw in plan.src_create: + self._validate_row(bid, body, kw) + + def _validate_row(self, bid: str, body: str, kw: str) -> None: + self._check_option(bid, "status", self._status_option(kw)) + tags = extract_tags(body) + prio = self._tag_value(tags, self.priority_map) + if prio is not None: + self._check_option(bid, "priority", prio) + weight = self._tag_value(tags, self.weight_map) + if weight is not None: + if self._eff_type("weight") == "number": + self._number(weight, "weight") + else: + self._check_option(bid, "weight", weight) + + def _check_option(self, bid: str, kind: str, name: str) -> None: + opts = self._prop_opts.get(self.props[kind]) + if opts is not None and name not in opts: + raise SystemExit( + f"notion-taskboard: {bid}: {kind} value {name!r} is not an " + f"option of the {self.props[kind]!r} prop {sorted(opts)}; fix " + "the map or add the option on the board (a create must never " + "auto-create an option on a team board).") + # --- property mapping -------------------------------------------------- def _status_option(self, kw: str) -> str: @@ -121,19 +178,23 @@ def _status_option(self, kw: str) -> str: "notion_taskboard_status_default.") return name + @staticmethod + def _number(value: str, kind: str): + try: + num = float(value) + except ValueError: + raise SystemExit(f"notion-taskboard: {kind} value {value!r} is not " + "a number (types set number).") + return int(num) if num.is_integer() else num + def _typed_value(self, kind: str, name: str) -> dict: - t = self.types.get(kind) + t = self._eff_type(kind) if t == "status": return {"status": {"name": name}} if t == "select": return {"select": {"name": name}} if t == "number": - try: - num = float(name) - except ValueError: - raise SystemExit(f"notion-taskboard: {kind} value {name!r} is " - "not a number (types set number).") - return {"number": int(num) if num.is_integer() else num} + return {"number": self._number(name, kind)} if t == "people": return {"people": [{"id": name}]} if t == "rich_text": @@ -162,9 +223,10 @@ def _page_props(self, title: str, body: str, kw: str) -> dict: out[self.props["owner"]] = self._typed_value("owner", self.owner) return out - # --- apply (insert-only) ----------------------------------------------- + # --- apply (insert-only; checkpoints each create via on_created) -------- - def apply(self, plan, assigned: dict, rows_after: dict) -> dict: + def apply(self, plan, assigned: dict, rows_after: dict, + on_created=None) -> dict: b = self.ensure_binding() created = {} for bid, title, body, kw in plan.src_create: @@ -172,5 +234,8 @@ def apply(self, plan, assigned: dict, rows_after: dict) -> dict: "parent": {"type": "data_source_id", "data_source_id": b["ds_id"]}, "properties": self._page_props(title, body, kw)}) - created[bid] = resp["id"] + rid = resp["id"] + created[bid] = rid + if on_created is not None: + on_created(bid, rid) # persist state before the next POST return created diff --git a/lib/sync/sync_core.py b/lib/sync/sync_core.py index 0d50b5b2..a7d715a4 100644 --- a/lib/sync/sync_core.py +++ b/lib/sync/sync_core.py @@ -18,15 +18,19 @@ CLOSED_HEADING = "## Recently closed" TABLE_HEADER = "| ID | Item | Notes & source | Status |" TABLE_RULE = "|---|---|---|---|" -# Any repo-prefixed board id (ID-, DF-, BK-, ...); the documented kit -# convention is `[A-Z]+-[0-9]+`. Generalized from the ID-only original so the -# engine parses every adopted repo's board (SPEC-003 / ID-138: dfoundation -# DF-NN rows). ID- boards still match, so existing behavior is unchanged. -ID_TOKEN = r"[A-Z][A-Z0-9]*-\d+" -ROW_ID_RE = re.compile(r"^\| (" + ID_TOKEN + r") \|") -TITLE_RE = re.compile(r"^(" + ID_TOKEN + r")\s*(?:[·:, -]\s*)?(.*)$") +TITLE_RE = re.compile(r"^(ID-\d+)\s*(?:[·:, -]\s*)?(.*)$") CELL_SPLIT = re.compile(r"(? dict[str, Row]: +def parse_board(text: str, strict_id: bool = True) -> dict[str, Row]: + """Parse `| id | item | notes | status |` rows. `strict_id=True` (default, + the two-way mesh) recognizes ONLY `ID-NNN`, keeping the ID-minting path + (next_id/apply_board) consistent. `strict_id=False` (the one-way create + push) accepts any repo prefix `[A-Z]+-NNN` but never mints or writes ids. + """ + row_re = STRICT_ROW_RE if strict_id else GEN_ROW_RE + id_full = r"ID-\d+" if strict_id else ID_TOKEN rows: dict[str, Row] = {} for i, line in enumerate(text.splitlines()): - if not ROW_ID_RE.match(line): + if not row_re.match(line): continue cells = split_row(line) if not cells: continue rid, item, notes, status = cells - if not re.fullmatch(ID_TOKEN, rid): + if not re.fullmatch(id_full, rid): continue kw = status.split()[0].lower() if status.split() else "" if rid in rows: diff --git a/lib/sync/tests/test_notion_taskboard.py b/lib/sync/tests/test_notion_taskboard.py index ff9cc0ec..01365714 100644 --- a/lib/sync/tests/test_notion_taskboard.py +++ b/lib/sync/tests/test_notion_taskboard.py @@ -335,3 +335,126 @@ def test_cli_end_to_end_pushes_through_main(tmp_path, monkeypatch): assert props["Status"] == {"status": {"name": "Backlog"}} assert props["Priority"] == {"select": {"name": "P0"}} assert board.read_text() == BOARD + + +# --- schema-aware validation (never auto-create a team option) ------------ + +SCHEMA = {"properties": { + "Task": {"type": "title", "title": {}}, + "Status": {"type": "status", "status": {"options": [ + {"name": "Backlog"}, {"name": "In progress"}, {"name": "Waiting"}, + {"name": "Done"}]}}, + "Priority": {"type": "select", "select": {"options": [ + {"name": "P0"}, {"name": "P1"}, {"name": "P2"}]}}, + "Notes": {"type": "rich_text", "rich_text": {}}, +}} + + +def schema_src(**kw): + fake = FakeNtn({"api v1/data_sources/ds1": SCHEMA}) + defaults = dict(db="db1", status_map=dict(DF_STATUS), binding=binding(), + runner=fake) + defaults.update(kw) + return NotionTaskBoardSource(**defaults), fake + + +def test_preflight_rejects_unknown_select_option(): + # a typo'd Priority value would otherwise auto-create a team option + src, _ = schema_src(priority_map={"u-hi": "P-zero"}) + plan = Plan(src_create=[("DF-1", "t", "x #u-hi", "queued")]) + with pytest.raises(SystemExit, match="not an option of the 'Priority'"): + src.preflight(plan) + + +def test_preflight_passes_known_options_and_learns_types(): + src, _ = schema_src(priority_map={"u-hi": "P0"}) + plan = Plan(src_create=[("DF-1", "t", "x #u-hi", "queued")]) + src.preflight(plan) # no raise + # types discovered from the schema (not the fallback guesses) + assert src._eff_type("status") == "status" + assert src._eff_type("priority") == "select" + + +def test_preflight_surfaces_unmapped_status_on_dry_run(tmp_path): + import backlog_sync + + board = tmp_path / "BACKLOG.md" + board.write_text( + "| ID | Item | Notes & source | Status |\n|---|---|---|---|\n" + "| DF-1 | mid-flight | x | speccing |\n") # speccing not mapped + src, _ = schema_src() # no status_default + with pytest.raises(SystemExit, match="no Status mapping"): + backlog_sync.sync_create_only(src, board, tmp_path / "s.json", + dry_run=True) + + +def test_stale_binding_for_a_different_db_is_rediscarded(): + fake = FakeNtn({"datasources resolve db-new": {"data_sources": [{"id": "ds-new"}]}}) + src = NotionTaskBoardSource(db="db-new", status_map=dict(DF_STATUS), + binding={"db_id": "db-old", "ds_id": "ds-old"}, + runner=fake) + b = src.ensure_binding() + assert b["db_id"] == "db-new" and b["ds_id"] == "ds-new" + + +# --- partial-batch failure never re-pushes (checkpoint per create) -------- + + +class FailOnNth(FakeNtn): + def __init__(self, nth, responses=None): + super().__init__(responses) + self.nth = nth + + def __call__(self, args, data=None): + if args[0] == "api" and args[1] == "v1/pages" and "-X" in args: + if self.page_seq + 1 == self.nth: + raise SystemExit("ntn api v1/pages failed: 429 rate limited") + return super().__call__(args, data) + + +TWO_ROW_BOARD = ( + "| ID | Item | Notes & source | Status |\n|---|---|---|---|\n" + "| DF-1 | first | x | queued |\n" + "| DF-2 | second | y | queued |\n" +) + + +def test_partial_batch_failure_checkpoints_and_never_duplicates(tmp_path): + import backlog_sync + + board = tmp_path / "BACKLOG.md" + board.write_text(TWO_ROW_BOARD) + state = tmp_path / "s.json" + + # run 1: second create fails after the first succeeded + fake = FailOnNth(2, {"api v1/data_sources/ds1": {"properties": {}}}) + src = NotionTaskBoardSource(db="db1", status_map=dict(DF_STATUS), + binding=binding(), runner=fake) + with pytest.raises(SystemExit, match="429"): + backlog_sync.sync_create_only(src, board, state, dry_run=False) + # DF-1's page was persisted even though the batch aborted + saved = json.loads(state.read_text()) + assert list(saved["map"]) == ["DF-1"] + + # run 2 (recovery): only DF-2 is created; DF-1 is NOT re-pushed + fake2 = FakeNtn({"api v1/data_sources/ds1": {"properties": {}}}) + src2 = NotionTaskBoardSource(db="db1", status_map=dict(DF_STATUS), + binding=binding(), runner=fake2) + backlog_sync.sync_create_only(src2, board, state, dry_run=False) + assert len(fake2.page_bodies()) == 1 + assert sorted(json.loads(state.read_text())["map"]) == ["DF-1", "DF-2"] + + +def test_warn_duplicate_ids_general_prefix(): + import backlog_sync + + text = ("| DF-1 | a | x | queued |\n" + "| DF-1 | dup | y | queued |\n") + # strict (ID-only) sees no DF dup; general prefix catches it + backlog_sync.warn_duplicate_ids(text, strict_id=True) # silent + import io + import contextlib + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + backlog_sync.warn_duplicate_ids(text, strict_id=False) + assert "duplicate board rows for DF-1" in buf.getvalue()