Skip to content

fix(docx-core): make accept all paragraph removal mark-based, not content-based (closes g5)#342

Merged
stevenobiajulu merged 1 commit into
mainfrom
g5-accept-mark-based
Jun 7, 2026
Merged

fix(docx-core): make accept all paragraph removal mark-based, not content-based (closes g5)#342
stevenobiajulu merged 1 commit into
mainfrom
g5-accept-mark-based

Conversation

@stevenobiajulu
Copy link
Copy Markdown
Member

Summary

Makes the TS engine's Accept All paragraph removal purely mark-based — the symmetric accept-side mirror of #337's reject fix (G4). Closes G5, the last characterized gap in the Lean↔TS helper differential.

A paragraph whose only content is inside w:del (or w:moveFrom) but whose paragraph mark is untracked means text deleted from a pre-existing paragraph — which Word, LibreOffice, and the (broadened, #340) Lean accept all keep as an empty <w:p>. The TS engine over-deleted it via a content-based heuristic. Accept now drops a paragraph iff its mark is PPR-DEL (<w:pPr><w:rPr><w:del/>).

Changes

  • acceptAllChanges (trackChangesAcceptorAst.ts) and primitive acceptChanges (accept_changes.ts): removed the content-based all-w:del/w:moveFrom drop heuristic on both accept entry points in lockstep; kept only the mark-based PPR-DEL drop (strict direct-child traversal, so a w:del nested in a w:pPrChange snapshot is ignored). Removed now-dead helpers (paragraphHasOnlyRemovedContent, containsRun, isWElement, the locals sets).
  • [LEAN-HELP-08] (G5) flipped from characterized divergence to agreement; module header records no KNOWN gap remains. With G5 closed, every G-case G1–G5 agrees Lean ↔ engine.
  • New targeted regression exercising BOTH accept paths over four shapes (PPR-DEL → drop; del-only untracked → keep empty; moveFrom-only untracked → keep empty; pPrChange-snapshot → ignore), asserting the two paths agree.
  • OpenSpec: new make-accept-paragraph-collapse-mark-based ([ACCEPT-MARK-01..04]); pending harness + broaden-lean-accept G5 scenarios revised to agreement; verification/ROADMAP.md records G5 closed.

Lean is unchanged (its accept keep-empty, broadened in #340, was already faithful).

Verification

  • Full scoped @usejunior/docx-core suite: 1347 passed / 3 skipped (+4 new tests, zero regressions).
  • Differential 8/8 with the real leanHelperDifferential exe; tsc --noEmit clean; openspec validate --strict ✅; spec-coverage + allure-labels + allure-quality 0 errors.
  • Peer-reviewed (codex + agy, both dynamic/executed): APPROVE. Both ran the suites against the worktree; one low doc-accuracy note (tasks.md evidence wording) addressed.

Closes G5. Deferred follow-up: PR-B (wire the LibreOffice accept/reject oracle voter into the harness).

…tent-based (closes G5)

The TS engine over-deleted a paragraph whose only content was inside w:del
(or w:moveFrom) but whose paragraph MARK was untracked — text deleted from a
pre-existing paragraph, which Word/LibreOffice keep (empty) on accept. This is
the exact accept-side mirror of the reject over-deletion fixed in #337 (G4),
pinned as the differential's G5 gap.

Accept is now purely mark-based on BOTH accept entry points: drop a paragraph
iff its mark is PPR-DEL (<w:pPr><w:rPr><w:del/>). The content-based
all-w:del/w:moveFrom drop heuristic is removed from acceptAllChanges
(trackChangesAcceptorAst.ts) and the primitive acceptChanges
(accept_changes.ts) in lockstep, with their now-dead helpers
(paragraphHasOnlyRemovedContent, containsRun, isWElement, the locals sets).

- Flip [LEAN-HELP-08] (G5) from characterized divergence to agreement; with G5
  closed, every G-case (G1-G5) agrees Lean <-> engine, no KNOWN gap remains.
- Add a targeted regression exercising BOTH accept paths over four shapes
  (PPR-DEL drop, del-only keep, moveFrom-only keep, pPrChange-snapshot ignore),
  asserting the two paths agree.
- OpenSpec: new make-accept-paragraph-collapse-mark-based change
  ([ACCEPT-MARK-01..04]); revise the pending harness + broaden-lean-accept G5
  scenarios to agreement; ROADMAP records G5 closed.

Lean is unchanged (its accept keep-empty, broadened in #340, was already
faithful). Full scoped docx-core suite: 1347 passed / 3 skipped.
@vercel
Copy link
Copy Markdown

vercel Bot commented Jun 7, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
site Ready Ready Preview, Comment Jun 7, 2026 9:27am

Request Review

@github-actions github-actions Bot added the fix label Jun 7, 2026
@stevenobiajulu stevenobiajulu enabled auto-merge (squash) June 7, 2026 09:27
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 7, 2026

LLM-Based Quality Gate

Overall: ✅ PASS (6 pass · 0 warn · 8 skipped · 14 total)

Check Verdict
⏭️ SKIPPED read_file response metadata parity paths not touched by this PR
⏭️ SKIPPED Live DOM namespace-safe OOXML writes paths not touched by this PR
Deleted field markup keeps w:fldChar outside w:del The PR modifies paragraph-drop heuristics on accept to close G5 and does not touch field atomization, validateFieldStructure, hasFldCharInsideDel, w:fldChar, w:instrText, w:delInstrText, or collapsed field comparison logic.
⏭️ SKIPPED Field validation per story, not global paths not touched by this PR
⏭️ SKIPPED Revision IDs seeded from all revision-bearing side parts paths not touched by this PR
Accept/reject sweep side parts and caches The PR updates the underlying acceptChanges primitive in packages/docx-core/src/primitives/accept_changes.ts:125-135, which is called in packages/docx-core/src/primitives/document.ts:489-504 for document.xml and side parts in REVISION_STORY_PART_PATHS. The side-parts sweep in packages/docx-core/src/primitives/document.ts:484-570 correctly processes all revision-bearing story parts, writes back only changed parts while refreshing the footnotesXml cache, and prunes orphaned footnotes in lines 144-155 while preserving reserved separators.
⏭️ SKIPPED DocumentViewNode.heading stays canonical paths not touched by this PR
⏭️ SKIPPED AI-author parity across entry points paths not touched by this PR
⏭️ SKIPPED Property-change wrapper discipline paths not touched by this PR
SUPPORT.md Table A drift vs. implementation The PR modifies accept_changes.ts to adjust revision acceptance behavior, but does not modify OOXML revision emission behavior or touch SUPPORT.md, so the precondition is not met.
⏭️ SKIPPED Table A / Table B boundary on side-part revisions paths not touched by this PR
Canonical-emission surface completeness The PR only modifies packages/docx-core/src/primitives/accept_changes.ts:125, which is a tracked-change consumer rather than a tracked-edit emitting surface, so the precondition for updating canonical emission tests or Table A documentation is not met.
Lean predicate drift against engine semantics (asymmetric) The PR refactors TS atomizer paragraph-deletion in packages/docx-core/src/baselines/atomizer/trackChangesAcceptorAst.ts:356 to be purely mark-based, aligning it closer to Lean's existing model in verification/lean/Tier2/AcceptReject.lean:43 which preserves untracked empty paragraphs. The Lean proof and bridge tests remain fully valid.
Unit-test quality (avoid tautological / change-detector tests) The tests added in trackChangesAcceptorAst.test.ts:858 and updated in lean-differential-helpers.test.ts:655 use concrete assertions constructed from first principles with no mocks, successfully exercising the G5 mark-based paragraph deletion fix.
Full checklist questions
  1. read_file response metadata parity: If this PR touches packages/docx-mcp/src/tools/read_file.ts, budgeted pagination returns, or additive response metadata like warnings / comment_load_error, do every successful return path (default budgeted early return, non-budget fallthrough, explicit limit/node_ids) preserve the same additive diagnostic fields? read_file has multiple success exits; diagnostics have already disappeared on one path before. Reference: fix(docx-core): declare xmlns:w14/w15 on comments root before writing prefixed attributes (#154) #180 surfaced comment_load_error, fix(docx-mcp): warn when read_file budget is exceeded by a single node (closes #184) #186 added an early budget return + warnings, fix(docx-mcp): surface comment_load_error on the default budgeted read path (closes #189) #191 fixed the missing comment_load_error on the default budgeted path.

  2. Live DOM namespace-safe OOXML writes: If this PR touches packages/docx-core/src/primitives/comments.ts or writes prefixed OOXML attributes/elements (w14:*, w15:*, xmlns:*, comments.xml, commentsExtended.xml, people.xml), are prefixed OOXML names written with namespace-aware APIs — root aliases bound with setAttributeNS(XMLNS_NS, ...), prefixed attributes with setAttributeNS(W14_NS/W15_NS, ...), and is there a test that proves the live DOM works before serialization/reparse? String-prefixed attributes can serialize plausibly while the live DOM still throws namespace errors. Reference: fix(docx-core): declare xmlns:w14/w15 on comments root before writing prefixed attributes (#154) #180 (xmlns:w14/w15 declared on comments root before writing prefixed attrs).

  3. Deleted field markup keeps w:fldChar outside w:del: If this PR touches field atomization, validateFieldStructure, hasFldCharInsideDel, w:fldChar, w:instrText, w:delInstrText, or collapsed field comparison logic, does deleted field output stay ECMA-376-conformant — w:fldChar sibling-level (never inside w:del), deleted instructions use w:delInstrText only inside valid delete wrappers, accept/reject safety checks still reject malformed combined output? Word treats deleted field-state markup in the wrong container as document-corrupting. References: fix(docx-core): validate w:delInstrText placement and reject w:fldChar inside <w:del> #211, fix(docx-core): partition field-closure validation by ECMA-376 story (#212) #225, fix(docx-core): fragment w:fldChar outside w:del per ECMA-376 Part 4 #228.

  4. Field validation per story, not global: If this PR touches packages/docx-core/src/baselines/atomizer/pipeline.ts, splitStories, validateFieldStructure, side-part merge logic, or footnote/endnote field handling, is field validation run independently per ECMA story (document.xml, each footnote, each endnote), with sidecars from both original and revised archives considered, and global counter balance not treated as sufficient? A document can be globally balanced but have an invalid field sequence inside one story. References: fix(docx-core): partition field-closure validation by ECMA-376 story (#212) #225, fix(docx-core): fragment w:fldChar outside w:del per ECMA-376 Part 4 #228, feat(docx-core): sweep side-part revisions on accept/reject #218.

  5. Revision IDs seeded from all revision-bearing side parts: If this PR touches packages/docx-mcp/src/session/manager.ts (especially getRevisionContextForSession or FIXED_REVISION_ID_SEED_PARTS), createRevisionContext, revision-ID allocation, or MCP tools that create tracked changes/comments/footnotes, does revision-ID allocation scan all relevant package parts before issuing new IDs — comments, footnotes, endnotes, glossary, headers, footers — ignore non-revision w:id values (comment IDs, bookmarks), and handle malformed optional parts gracefully? Revision IDs are package-wide; document-only seeding collides with existing side-part revisions. Reference: fix(docx-mcp): seed revision ids from side parts #216 (seed revision ids from side parts).

  6. Accept/reject sweep side parts and caches: If this PR touches DocxDocument.acceptChanges, DocxDocument.rejectChanges, REVISION_STORY_PART_PATHS, accept_changes, reject_changes, or side-part revision markup, does accept/reject process every revision-bearing story — updating document.xml + footnotes.xml + endnotes.xml + comments.xml, writing back only changed side parts while refreshing cached XML, and pruning orphan footnotes without deleting reserved separator entries? Accepting only in the main document leaves stale revisions and dangling references in the package. References: feat(docx-core): sweep side-part revisions on accept/reject #218, fix(docx-mcp): seed revision ids from side parts #216, fix(docx-core): partition field-closure validation by ECMA-376 story (#212) #225.

  7. DocumentViewNode.heading stays canonical: If this PR touches packages/docx-core/src/primitives/document_view.ts, HeadingValue, heading heuristics, ListMetadata.header_style, or Google Docs document-view heading normalization, does node.heading remain a structural heading signal — exact Word styles Heading1Heading6 win, heuristic sources suppressed inside table cells while real Word heading styles still pass, ordinary body paragraphs omit the heading key? Consumers use node.heading != null as a structural test; heuristic false positives break downstream navigation. References: fix(docx-core): harden heading detection (#157 Phase 1) #178, fix(docx-core): suppress non-sectional false-positive headings (closes #187) #188, feat(docx-core): add derived heading object to DocumentViewNode (closes #179) #190.

  8. AI-author parity across entry points: If this PR touches packages/docx-mcp/src/server.ts, packages/docx-mcp/src/cli/tool_runner.ts, packages/docx-mcp/src/cli/commands/**, or adds any new new SessionManager(...) call site in docx-mcp, does every entry point that constructs a SessionManager resolve SAFE_DOCX_AI_AUTHOR with the same three-way semantics (set → use it; empty string → opt out to untracked; unset → defaultAiAuthor), or has a new entry path silently bypassed tracked emission? Each entry path looks locally correct while diverging from another; tracked emission has gone dark in one path before anyone noticed. References: feat(docx-mcp): wire configurable AI author through MCP layer (#142) #172 (production MCP wiring would have kept tracked emission dark), fix(docx-mcp): honor SAFE_DOCX_AI_AUTHOR in CLI entry points (#181) #182 (CLI runners constructing bare SessionManager() silently produced untracked edits).

  9. Property-change wrapper discipline: If this PR touches packages/docx-core/src/primitives/layout.ts, packages/docx-core/src/primitives/text.ts, packages/docx-mcp/src/tools/clear_formatting.ts, or packages/docx-core/src/primitives/track-changes-emitter.ts, do tracked formatting/property edits emit exactly one correct *PrChange wrapper (pPrChange / rPrChange / trPrChange / tcPrChange) carrying a snapshot of the prior live properties — not stacking stale wrappers, not stripping valid historical children (cellIns/cellDel/cellMerge), and not omitting the snapshot when the operation is formatting-aware? Emitted OOXML is visually plausible but subtle snapshot mistakes only surface during later accept/reject or in Word's tracked-changes UI. References: feat(docx-core): emit pPrChange/trPrChange/tcPrChange from layout setters (#140) #167 (duplicate pPrChange/trPrChange/tcPrChange stacking + over-broad tcPr exclusion), feat(docx-mcp): emit rPrChange from clear_formatting MCP tool (#141) #170 (clear_formatting failing to strip stale rPrChange), feat(docx-core): emit rPrChange for formatted paragraph replacements #215 (rPrChange for formatted paragraph replacements + filtering nested stale records).

  10. SUPPORT.md Table A drift vs. implementation: If this PR modifies OOXML revision emission behavior (w:ins, w:del, w:rPrChange, etc.) in packages/docx-core/src/primitives/**, or touches packages/docx-core/SUPPORT.md, does the PR symmetrically update Table A in SUPPORT.md when the supported revision-emission surface in primitives changed — added, removed, or weakened — or is the documented contract now lying about what's supported? Reviewers focus on TS AST correctness and golden tests; Markdown contract tables get treated as an afterthought, so the documented surface drifts from the actual surface. Reference: [120.8] Regression suite for canonical revision emission across the surface #143 review caught replaceParagraphTextRange should emit w:rPrChange when run formatting changes #173 (formatting mismatch in Table A) and addCommentReply should emit body revision markup OR SUPPORT.md should be softened #174 (comment body revision omission forcing a Table A softening) late in peer review.

  11. Table A / Table B boundary on side-part revisions: If this PR touches packages/docx-core/src/primitives/comments.ts, packages/docx-core/src/primitives/footnotes.ts, or other side-part primitives, and adds/changes revision markup (w:ins, w:del), does tracked-change revision logic stay scoped to Table A (document-body content inside the side part) without leaking revision markup into Table B (the side-part package bootstrap — comments.xml/footnotes.xml element registration itself)? Body runs and side-part package elements share nearly identical XML namespace schemas; revisions emitted in the wrong table corrupt the package contract while looking plausible. References: [120.3] Emit w:ins/w:del for comment body anchors #138 (comment-body straddle constraints), [120.4] Emit w:ins/w:del for footnote reference and text #139 (footnote-reference straddle constraints).

  12. Canonical-emission surface completeness: If this PR adds or changes a tracked-edit surface in packages/docx-core/src/primitives/** or packages/docx-mcp/src/tools/**, are the paired artifacts updated together — packages/docx-core/src/integration/canonical-emission-regression.test.ts, packages/docx-mcp/src/integration/canonical-emission-mcp.test.ts, and the documented emitter surface (Table A) — or is the rollout only partially wired? The primitive change looks done before the MCP path, regression matrix, and documented contract are wired through; partial rollouts ship undocumented surface that drifts. References: feat(docx-mcp): wire configurable AI author through MCP layer (#142) #172 (RevisionContext threaded through every Table A MCP tool), test(docx-core,docx-mcp): final regression suite for canonical emission (#143) #175 (24-test regression suite + verified write-time emitter rows), feat(docx-core): emit rPrChange for formatted paragraph replacements #215 (re-enabled rPrChange regression + updated support surface for replaceParagraphTextRange).

  13. Lean predicate drift against engine semantics (asymmetric): If this PR changes field-wrapper semantics, the proof boundary, or atomizer behavior — packages/docx-core/src/baselines/atomizer/**, verification/lean/LeanSpike/Spec.lean, verification/lean/Tier2/**, or packages/docx-core/src/integration/lean-spec-bridge.test.ts — and if the TS engine semantics shifted, did the PR also update the Lean residual predicate and bridge tests, or is the proof now pinned to a stale stronger/weaker assumption? Asymmetric: a TS change without a corresponding Lean update is WARN; a Lean-only change without a TS update should not fire. The Lean side can still compile while the abstraction boundary is subtly wrong for the next engine refactor. References: feat(verification): close inv_field_001 with Tier 2 OoxmlDoc subset #208 (closed inv_field_001 using stronger recursivelyWellformed), refactor(verification): weaken inv_field_001 axiom to document-level preservationFriendly (rebased follow-up to #208) #220 (weakened the axiom to document-level preservationFriendly to avoid breakage when field fragmentation lands).

  14. Unit-test quality (avoid tautological / change-detector tests): If this PR adds or modifies any **/*.test.ts (or other test files), are the test assertions independent of the system under test — expected values constructed from first principles rather than re-derived from the function under test, mocks limited to external boundaries (filesystem, network, clocks) rather than mocking the SUT itself, assertions making concrete semantic claims rather than just snapshotting current behavior or asserting non-null, and any test added alongside a bug fix actually exercising the bug? Tests that re-implement the production code as the "expected" value, or mock out the system under test, pass green while providing no regression protection.

Estimated cost (this run): $0.0146 — 44,811 input + 445 output tokens (≈4 chars/token) on gemini-3.5-flash. Char-count estimate, not provider telemetry.

@stevenobiajulu stevenobiajulu merged commit 0d014b8 into main Jun 7, 2026
40 checks passed
@stevenobiajulu stevenobiajulu deleted the g5-accept-mark-based branch June 7, 2026 09:33
@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 7, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@stevenobiajulu
Copy link
Copy Markdown
Member Author

✅ Post-merge smoke passed

Merged: 0d014b8 (squash)
Built from: main @ b4aadf1
Smoke: clean build + lint + full suite + real-DOCX G5 accept/reject

Steps

  • ✅ build (tsc -p tsconfig.build.json, clean dist)
  • ✅ lint (tsc --noEmit)
  • ✅ tests (1347 passed / 3 skipped, 92 files)
  • ✅ real-DOCX G5 accept/reject (11/11)

Real-world fixtures

  • bonterms-mutual-nda (local template at tests/test_documents/open-agreements/bonterms-mutual-nda.docx, 58 paragraphs)
    • Baseline: accept-all / reject-all the unmodified doc keep all 58 untracked paragraphs and strip every w:del/w:ins.
    • G5 fix: injected a del-only untracked-mark paragraph into the real body (58→59); both acceptAllChanges and the primitive acceptChanges keep it as an empty <w:p> (59→59), deleted text gone — pre-G5 the content heuristic dropped it.
    • Control: a PPR-DEL-marked paragraph still drops on accept (59→58); both engine paths agree.

Log: /tmp/automerge-smoke-342-20260607-054855.log (local).

Cleanup

  • ⚠️ remote branch g5-accept-mark-based already deleted — auto-delete on merge is enabled
  • ✅ removed worktree .worktrees/g5-accept-mark
  • ✅ deleted local branch g5-accept-mark-based

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant