feat: SpacyEntityRelationExtractor (LLM-free KG construction)

.plans/progress-spacy-entity-relation-extractor.txt

@@ -0,0 +1,46 @@
+## TASK-005 — 2026-06-24T16:00:00Z
+Unit tests for SpacyEntityRelationExtractor, spacy.load mocked.
+13 tests covering: two-chunk extraction with Entity nodes + confidence property, use_linear_extractor=True/False toggle, create_lexical_graph=True adding Chunk/Document nodes, ValueError for missing/non-installed model, on_error=OnError.IGNORE suppression, custom word_filter exclusion, schema fail-open behaviour (unmapped NER types pass through), known NER type filtering. All 13 tests pass with no model download.
+Files: +tests/unit/experimental/components/test_spacy_entity_relation_extractor.py
+
+## TASK-006 — 2026-06-24T12:00:00Z
+Created `HierarchicalTextSplitter(TextSplitter)` with four header strategies (markdown, capitalization, blank_line, spacy_verbless), recursive overlap splitting for large sections, and optional SpaCy-based verbless-sentence filtering. Ruff, mypy, and existing unit tests all pass.
+Files: +src/neo4j_graphrag/experimental/components/text_splitters/hierarchical_splitter.py
+Decisions: blank_line strategy uses ≤6-word cap to avoid treating body sentences as section headers; chunk index assigned with single-pass `len(chunks)` pattern; negative chunk_overlap validated; assert replaced with RuntimeError guard; docstring warns that default drop_verbless_sentences=True requires SpaCy.
+
+## TASK-003 — 2026-06-24T10:00:00Z
+Implemented `extract_linear_triples(doc)` in a new file (task-002 was in_progress on spacy_utils.py) with 14 unit tests — all green.
+Files: +src/neo4j_graphrag/experimental/components/spacy_linear_extractor.py, +tests/unit/experimental/components/test_spacy_linear_extractor.py
+Decisions: Separate file because task-002 owned spacy_utils.py concurrently; graceful fallback when no sentence segmenter is present (treats whole doc as one sentence); distance measured start-to-start.
+
+## TASK-001 — 2026-06-24T08:00:00Z
+Created `spacy_utils.py` with `normalize()` (lowercases, NFC-normalises Unicode, collapses whitespace) and `WORD_FILTER` frozenset of ~90 common English stopwords; 21 unit tests all green.
+Files: +src/neo4j_graphrag/experimental/components/spacy_utils.py, +tests/unit/experimental/components/test_spacy_utils.py
+Decisions: Added `unicodedata.normalize("NFC", ...)` step per reviewer feedback to ensure visually identical Unicode strings compare equal as graph node keys.
+
+## TASK-002 — 2026-06-24T10:00:00Z
+Added `extract_dependency_triples(doc)` to `spacy_utils.py`: walks the dependency tree per sentence finding nsubj/nsubjpass → ROOT → dobj/pobj patterns, handles passive voice (_BY suffix, conf=0.9), prepositional triples (conf=0.7), and active SVO (conf=1.0). Uses spaCy 3.x retokenizer API (not deprecated `Span.merge()`) to collapse multi-word noun chunks and named entities. Falls back gracefully when sentence boundaries are unavailable. 18 unit tests in a new dedicated file; spaCy-free tests remain ungated.
+Files: ~src/neo4j_graphrag/experimental/components/spacy_utils.py, ~tests/unit/experimental/components/test_spacy_utils.py, +tests/unit/experimental/components/test_spacy_dependency_triples.py
+Decisions: Reviewer flagged `Span.merge()` removed in spaCy 3.x (silently swallowed); fixed to use `doc.retokenize()`. Moved spaCy-gated tests to separate file so normalize/WORD_FILTER tests stay runnable without spaCy installed.
+
+## TASK-007 — 2026-06-24T14:00:00Z
+Added 20 unit tests for `HierarchicalTextSplitter` covering all 4 header strategies (markdown, capitalization, blank_line, spacy_verbless), recursive overlap splitting with exact last-N==first-N invariant verification, drop_verbless_sentences filter (SpaCy fully mocked via patch("spacy.load")), chunk index sequencing across sections, and edge cases (empty/whitespace input, invalid constructor args). No network calls or model downloads required.
+Files: +tests/unit/experimental/components/text_splitters/test_hierarchical_splitter.py
+
+## TASK-004 — 2026-06-24T15:00:00Z
+Implemented `SpacyEntityRelationExtractor(EntityRelationExtractor)` in `src/neo4j_graphrag/experimental/components/spacy_entity_relation_extractor.py`.
+Key design decisions: SpaCy model loaded once at construction time with user-friendly ValueError (covering both missing model and missing package); NER→schema label mapping via `_SPACY_NER_TO_SCHEMA_LABEL` forward-lookup + direct-uppercase Strategy (removed dead reverse map); fail-open NER filtering with `logger.warning` when schema types don't map to SpaCy NER labels; dependency + linear triples unioned and filtered by word filter, relation type, and NER type; lexical graph built via `LexicalGraphBuilder` following the same pattern as `LLMEntityRelationExtractor`; `on_error=IGNORE` logs and skips per-chunk exceptions. All types use lowercase builtins (`set[...]`, `tuple[...]`). ruff, mypy, pytest all clean.
+Files: +src/neo4j_graphrag/experimental/components/spacy_entity_relation_extractor.py
+Review feedback applied: removed `_SCHEMA_LABEL_TO_SPACY_NER` dead code, added warning log for unmapped schema types, broadened `except (OSError, ImportError)` in constructor, switched to lowercase type builtins.
+
+## TASK-009 — 2026-06-24T16:00:00Z
+Exported SpacyEntityRelationExtractor and HierarchicalTextSplitter from __init__ files with ImportError guards.
+Files: ~src/neo4j_graphrag/experimental/components/__init__.py, ~src/neo4j_graphrag/experimental/components/text_splitters/__init__.py
+
+## TASK-010 — 2026-06-24T17:00:00Z
+Example script for end-to-end spacy KG pipeline.
+Files: +examples/customize/build_graph/components/spacy_kg_pipeline.py
+
+## TASK-008 — 2026-06-24T17:00:00Z
+Integration tests for SpacyEntityRelationExtractor and HierarchicalTextSplitter (skip when en_core_web_sm absent).
+Files: +tests/unit/experimental/components/test_spacy_extractor_integration.py, +tests/unit/experimental/components/text_splitters/test_hierarchical_splitter_integration.py

.plans/tasks-spacy-entity-relation-extractor.yml

@@ -0,0 +1,224 @@
+feature: spacy-entity-relation-extractor
+prd: ./prd-spacy-entity-relation-extractor.md
+created_at: 2026-06-24T00:00:00Z
+updated_at: 2026-06-24T00:00:00Z
+
+tasks:
+  - id: task-001
+    title: "Implement normalize() helper and WORD_FILTER constant"
+    description: |
+      Create shared normalisation utilities used by both components.
+      Add a `normalize(text: str) -> str` function that lowercases and
+      collapses whitespace, and a `WORD_FILTER: frozenset[str]` of common
+      stopwords to exclude from entity positions. Place in
+      `src/neo4j_graphrag/experimental/components/spacy_utils.py`.
+    status: completed
+    completed_at: "2026-06-24T08:00:00Z"
+    dependencies: []
+    acceptance_criteria:
+      - "normalize('  Supply Chain  ') returns 'supply chain'"
+      - "normalize('') returns ''"
+      - "WORD_FILTER contains common English stopwords (e.g. 'the', 'a', 'is')"
+      - "Unit tests cover edge cases: empty string, leading/trailing spaces, multiple internal spaces"
+    completed_at: null
+
+  - id: task-002
+    title: "Implement dependency-based triple extractor function"
+    description: |
+      Implement `extract_dependency_triples(doc: spacy.tokens.Doc) -> set[tuple[str, str, str, float]]`
+      in `spacy_utils.py`. Walk the dependency tree for each sentence to find
+      nsubj/nsubjpass → root_verb → dobj/pobj patterns. Apply phrasal merging
+      (`merge_noun_chunks`, `merge_entities`) before extraction. Handle passive
+      constructions by swapping subject/object and appending `_BY` to the
+      relation. Yield `(head, relation_lemma, tail, confidence)` tuples.
+    status: completed
+    completed_at: "2026-06-24T10:00:00Z"
+    dependencies:
+      - task-001
+    acceptance_criteria:
+      - "Active SVO sentence yields (subject, verb_lemma, object, conf) triple"
+      - "Passive sentence yields (object, verb_lemma_BY, subject, conf) triple"
+      - "Prepositional phrase yields secondary (object, prep_label, pobj, conf) triple"
+      - "Multi-token noun phrases are collapsed into single entity strings"
+      - "Unit tests use spacy.blank + manually crafted Doc objects (no model download required)"
+
+  - id: task-003
+    title: "Implement linear triple extractor function"
+    description: |
+      Implement `extract_linear_triples(doc: spacy.tokens.Doc) -> set[tuple[str, str, str, float]]`
+      in `spacy_utils.py`. Scan NER entity pairs within the same sentence that
+      appear within 10 tokens of each other with no other NER entity between
+      them. Emit `(entity_a, "RELATED_TO", entity_b, 0.3)` for each such pair.
+      This catches appositions, list items, and implicit associations not
+      surfaced by the dependency tree.
+    status: completed
+    completed_at: "2026-06-24T10:00:00Z"
+    dependencies:
+      - task-001
+    acceptance_criteria:
+      - "Two NER entities within 10 tokens in same sentence yield a RELATED_TO triple"
+      - "Two NER entities more than 10 tokens apart produce no triple"
+      - "Three NER entities A–B–C produce A-RELATED_TO-B and B-RELATED_TO-C but not A-RELATED_TO-C (intervening entity)"
+      - "Confidence on all linear triples is exactly 0.3"
+      - "Unit tests use synthetic spacy Doc with manually set ents"
+    note: "Implemented in spacy_linear_extractor.py (separate from spacy_utils.py because task-002 was in_progress on that file). task-004 should import from both spacy_utils and spacy_linear_extractor."
+
+  - id: task-004
+    title: "Implement SpacyEntityRelationExtractor component"
+    description: |
+      Create `src/neo4j_graphrag/experimental/components/spacy_entity_relation_extractor.py`
+      with class `SpacyEntityRelationExtractor(EntityRelationExtractor)`. The
+      constructor loads the SpaCy model once (raising ValueError with an install
+      hint if not found) and accepts `word_filter: Optional[list[str]] = None`
+      (uses WORD_FILTER constant if None). The async `run()` method accepts
+      `schema: Optional[GraphSchema] = None` (same as LLMEntityRelationExtractor).
+      When schema is provided, apply two filters: (1) type-level NER filter —
+      map schema node_type labels to SpaCy NER entity types and keep only
+      matching entities; (2) post-filter relations — only keep predicates that
+      match a schema relationship_type name. Without schema, all extracted
+      triples pass through. Unions dependency and linear triples (when enabled),
+      normalises and filters them, maps results to Neo4jNode/Neo4jRelationship
+      objects, and delegates lexical graph construction to LexicalGraphBuilder
+      when `create_lexical_graph=True`. Confidence scores go in relationship
+      properties.
+    status: completed
+    completed_at: "2026-06-24T15:00:00Z"
+    dependencies:
+      - task-002
+      - task-003
+    acceptance_criteria:
+      - "Class is importable from neo4j_graphrag.experimental.components.spacy_entity_relation_extractor"
+      - "Constructor with non-installed model name raises ValueError containing 'python -m spacy download'"
+      - "Constructor accepts word_filter list; custom list overrides WORD_FILTER default"
+      - "run() returns Neo4jGraph with nodes labelled 'Entity' and relationships with 'confidence' property"
+      - "run() with schema filters out entities whose NER type has no matching schema node_type"
+      - "run() with schema filters out relationships whose predicate has no matching schema relationship_type"
+      - "use_linear_extractor=False produces no RELATED_TO relationships"
+      - "create_lexical_graph=True adds Chunk and Document nodes to the output graph"
+      - "on_error=OnError.IGNORE suppresses per-chunk exceptions; on_error=OnError.RAISE propagates them"
+
+  - id: task-005
+    title: "Unit tests for SpacyEntityRelationExtractor"
+    description: |
+      Add `tests/unit/experimental/components/test_spacy_entity_relation_extractor.py`.
+      Mock `spacy.load` and the nlp pipeline so no model download is required.
+      Cover: successful extraction from two-chunk input, linear extractor toggle,
+      lexical graph nodes, invalid model error, and on_error behaviour.
+    status: completed
+    dependencies:
+      - task-004
+    acceptance_criteria:
+      - "All tests pass with pytest without a live Neo4j instance or network calls"
+      - "spacy.load is mocked; no en_core_web_sm download required"
+      - "Test covers use_linear_extractor=True and False"
+      - "Test covers create_lexical_graph=True producing Chunk/Document nodes"
+      - "Test covers ValueError raised for missing model"
+      - "Test covers on_error=OnError.IGNORE swallowing a per-chunk exception"
+    completed_at: "2026-06-24T16:00:00Z"
+
+  - id: task-006
+    title: "Implement HierarchicalTextSplitter component"
+    description: |
+      Create `src/neo4j_graphrag/experimental/components/text_splitters/hierarchical_splitter.py`
+      with class `HierarchicalTextSplitter(TextSplitter)`. Constructor params:
+      `max_chunk_size: int = 2048`, `chunk_overlap: int = 200`,
+      `header_strategy: str = "markdown"` (one of: "markdown" — lines starting
+      with `#`; "capitalization" — short Title Case or ALL_CAPS lines without
+      terminal punctuation; "blank_line" — short lines surrounded by blank
+      lines; "spacy_verbless" — SpaCy-detected verbless sentences that look
+      like headers), `model: str = "en_core_web_sm"` (only loaded when
+      header_strategy="spacy_verbless" or drop_verbless_sentences=True),
+      `drop_verbless_sentences: bool = True`. The "markdown" strategy is the
+      right default for MarkdownLoader output. Use "capitalization" or
+      "blank_line" for plain-text output from loaders like LiteParseLoader
+      (which returns plain text, not markdown). Recursively split sections
+      exceeding max_chunk_size with chunk_overlap. Output TextChunks with
+      sequential index values.
+    status: completed
+    completed_at: "2026-06-24T12:00:00Z"
+    dependencies:
+      - task-001
+    acceptance_criteria:
+      - "Class is importable from neo4j_graphrag.experimental.components.text_splitters.hierarchical_splitter"
+      - "header_strategy='markdown' splits on # header lines"
+      - "header_strategy='capitalization' splits on short ALL_CAPS or Title Case lines without terminal punctuation"
+      - "header_strategy='blank_line' splits on short lines surrounded by blank lines"
+      - "Section longer than max_chunk_size is split into multiple chunks with chunk_overlap character overlap"
+      - "Section shorter than max_chunk_size is emitted as a single chunk without splitting"
+      - "drop_verbless_sentences=True removes sentences containing no verb token (SpaCy mocked in unit tests)"
+      - "Output TextChunks has sequential index 0, 1, 2, ..."
+    completed_at: null
+
+  - id: task-007
+    title: "Unit tests for HierarchicalTextSplitter"
+    description: |
+      Add `tests/unit/experimental/components/text_splitters/test_hierarchical_splitter.py`.
+      Test section splitting, recursive character splitting with overlap, and
+      index sequencing. For drop_verbless_sentences tests, mock the SpaCy
+      pipeline so no model download is needed.
+    status: completed
+    dependencies:
+      - task-006
+    acceptance_criteria:
+      - "All tests pass without network calls or model downloads"
+      - "Test verifies correct chunk_overlap: last N chars of chunk K equal first N chars of chunk K+1"
+      - "Test verifies section boundary respected: two Markdown-header sections produce separate chunks"
+      - "Test verifies verbless sentence dropped when drop_verbless_sentences=True (spacy mocked)"
+      - "Test verifies chunk indices are sequential starting from 0"
+    completed_at: "2026-06-24T14:00:00Z"
+
+  - id: task-008
+    title: "Integration tests for both components against real SpaCy model"
+    description: |
+      Add integration tests in
+      `tests/unit/experimental/components/test_spacy_extractor_integration.py`
+      and `tests/unit/experimental/components/text_splitters/test_hierarchical_splitter_integration.py`.
+      Use pytest fixtures that skip the test if `en_core_web_sm` is not installed.
+      Verify that real SpaCy output produces a populated Neo4jGraph and correct
+      TextChunks respectively.
+    status: completed
+    dependencies:
+      - task-005
+      - task-007
+    acceptance_criteria:
+      - "Tests are skipped (not failed) when en_core_web_sm is not installed"
+      - "With model installed, SpacyEntityRelationExtractor.run() returns Neo4jGraph with at least one node and one relationship for a 2-sentence input"
+      - "With model installed, HierarchicalTextSplitter.run() returns correct chunks for a multi-section document"
+      - "Confidence values on relationships are floats in [0, 1]"
+    completed_at: "2026-06-24T17:00:00Z"
+
+  - id: task-009
+    title: "Export new components from experimental __init__"
+    description: |
+      Add `SpacyEntityRelationExtractor` and `HierarchicalTextSplitter` to the
+      public exports in `src/neo4j_graphrag/experimental/components/__init__.py`
+      and the text_splitters sub-package `__init__.py`. Guard imports with a
+      try/except so the rest of the package still imports cleanly when SpaCy is
+      not installed.
+    status: completed
+    completed_at: "2026-06-24T16:00:00Z"
+    dependencies:
+      - task-004
+      - task-006
+    acceptance_criteria:
+      - "from neo4j_graphrag.experimental.components import SpacyEntityRelationExtractor works when spacy is installed"
+      - "from neo4j_graphrag.experimental.components.text_splitters import HierarchicalTextSplitter works when spacy is installed"
+      - "Importing neo4j_graphrag.experimental.components without spacy installed does not raise ImportError"
+
+  - id: task-010
+    title: "Add example script for end-to-end spacy pipeline"
+    description: |
+      Create `examples/customize/build_graph/components/spacy_kg_pipeline.py`
+      demonstrating a full pipeline:
+      DataLoader → HierarchicalTextSplitter → SpacyEntityRelationExtractor → KGWriter.
+      Include a comment block explaining the nlp extra install step
+      (`pip install 'neo4j-graphrag[nlp]'` + `python -m spacy download en_core_web_sm`).
+    status: completed
+    dependencies:
+      - task-009
+    acceptance_criteria:
+      - "Script is syntactically valid Python"
+      - "Script includes install instructions in a comment block at the top"
+      - "Script constructs all four pipeline components and wires them together"
+      - "Script includes a __main__ guard and can be run standalone"
+    completed_at: "2026-06-24T17:00:00Z"