feat: the raw layer — lossless, addressable, writable OOXML access (v0.4.0)

CHANGELOG.md

@@ -7,6 +7,56 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [0.4.0] — 2026-07-07
+
+### Added — the raw layer (`contextifier.raw`)
+
+Contextifier now offers **two views of every document**: the existing
+AI-friendly extraction pipeline, and `open_raw()` — a lossless,
+addressable, **writable** view of OOXML files (`.xlsx` / `.docx` /
+`.pptx`).
+
+- **`open_raw(source)`** (top-level export) and
+  `DocumentProcessor.open_raw()` — dispatch to `XlsxRawDocument` /
+  `DocxRawDocument` / `PptxRawDocument` by extension or package sniffing.
+- **Byte-preservation contract** (`raw/opc.py`): saving writes untouched
+  parts back byte-identically (BOM, whitespace, extensions and all).
+  Charts, pivot tables, sparklines, custom XML, styles — everything the
+  usual Office libraries destroy on a load→save round-trip — survives.
+  Contract-tested against synthetic and real files, including a contrast
+  test showing openpyxl 3.1.5 dropping sparklines/customXml/chart styles
+  that the raw layer preserves.
+- **XLSX** (`raw/xlsx.py`): surgical `set_cell` (style attribute
+  preserved, strings as `inlineStr`, rows/cells kept sorted),
+  `append_rows`, cached-formula reads + `get_formula`, merged ranges,
+  defined names, `charts`; overwriting a formula sets
+  `calcPr fullCalcOnLoad` so Excel recalculates on open.
+- **DOCX** (`raw/docx.py`): python-docx-compatible paragraph/table
+  addressing (incl. gridSpan/vMerge grid semantics and nested tables),
+  **run-preserving** `set_paragraph_text` and `cell.set_text` (inline
+  images, drawings, bookmarks and hyperlink elements survive; hyperlinks
+  are emptied, never deleted — `strip_empty_hyperlinks()` when you want
+  them gone), `insert_paragraph_after`/`delete_paragraph`,
+  `insert_row`/`delete_row`, headers/footers, `body_order()`.
+- **PPTX** (`raw/pptx.py`): slide/shape inventory (tables, charts,
+  diagrams, groups), format-preserving `set_text`, table cell edits +
+  row insert/delete, `notes_text`, **`replace_content(new_xml,
+  preserve_native=True)`** — swap a slide's XML while carrying over its
+  native charts/tables/pictures (id-renumbered), and `remove_slide()`
+  with transitive orphan-part sweeping (chart XML, embedded workbooks,
+  media) that provably spares shared layouts/masters.
+- **Charts** (`raw/chart.py`): one `ChartModel` for all three formats —
+  reads kind/title/series from classic `c:` charts (incl. scatter/bubble
+  x/y, literals, multi-level caches) and chartEx; **writes** titles and
+  full data (`set_data`) with schema-valid cache/ref rewriting, series
+  grow/shrink keeping styling, and embedded-workbook regeneration —
+  verified by reopening with python-pptx/openpyxl.
+- `lxml` promoted to an explicit dependency.
+
+99 new tests (598 total).
+
+---
+
 ## [0.3.0] — 2025-07-19 (released 2026-07-07)
 
 ### Added (RAG provenance & payload ingestion — 2026-07-07)

README.md

@@ -5,6 +5,7 @@
 ## Key Features
 
 - **Broad Format Support**: PDF, DOCX, DOC, PPTX, PPT, XLSX, XLS, HWP, HWPX, RTF, CSV, TSV, TXT, MD, HTML, images, code files, and 80+ extensions
+- **Two Views of Every Document**: the AI-friendly pipeline (lightweight, normalized text for LLMs) *and* `open_raw()` — a lossless, addressable, **writable** view of OOXML files where saving keeps untouched parts byte-identical
 - **Intelligent Text Extraction**: Preserves document structure (headings, tables, image positions) with automatic metadata extraction
 - **Table Processing**: Converts tables to HTML/Markdown/Text with `rowspan`/`colspan` support for merged cells
 - **OCR Integration**: 5 Vision LLM engines — OpenAI, Anthropic, Google Gemini, AWS Bedrock, vLLM
@@ -35,7 +36,39 @@ text = processor.extract_text("document.pdf")
 print(text)
 ```
 
-### 2. Extract + Chunk in One Step
+### 2. Raw Access — Read *and Write* OOXML Losslessly
+
+The extraction pipeline renders an AI-friendly view and discards the
+rest. `open_raw()` is its lossless twin: the full package stays
+available, edits are surgical, and **untouched parts round-trip
+byte-identically** — charts, pivot tables, sparklines, styles and
+custom XML all survive (unlike a load→save round-trip through the
+usual Office libraries).
+
+```python
+from contextifier import open_raw
+
+raw = open_raw("report.xlsx")              # XlsxRawDocument
+raw.sheets["Sales"].set_cell("B3", 142)    # surgical edit
+raw.charts[0].set_data(                    # real chart-data editing
+    categories=["Q1", "Q2", "Q3"],
+    series=[("Sales", [120, 135, 150])],
+)
+raw.save("report-edited.xlsx")
+
+raw = open_raw("paper.docx")               # DocxRawDocument
+raw.set_paragraph_text(3, "Revised text")  # runs & inline images preserved
+raw.tables[0].insert_row(2)
+
+raw = open_raw("deck.pptx")                # PptxRawDocument
+raw.slides[0].set_text(shape_id=2, new_text="New title")
+raw.save("deck2.pptx")
+```
+
+Supported raw formats: `.xlsx` / `.docx` / `.pptx` (the OOXML trio).
+Every model also exposes `.package` for part-level OPC access.
+
+### 3. Extract + Chunk in One Step
 
 ```python
 from contextifier import DocumentProcessor
@@ -50,7 +83,7 @@ for i, chunk in enumerate(result.chunks, 1):
 result.save_to_md("output/chunks")
 ```
 
-### 3. Custom Configuration
+### 4. Custom Configuration
 
 ```python
 from contextifier import DocumentProcessor
@@ -65,7 +98,7 @@ processor = DocumentProcessor(config=config)
 text = processor.extract_text("report.xlsx")
 ```
 
-### 4. OCR Integration
+### 5. OCR Integration
 
 ```python
 from contextifier import DocumentProcessor

contextifier/__init__.py

@@ -24,11 +24,17 @@
       └── OCR (optional vision-based extraction)
 
 Usage:
-    from contextifier import DocumentProcessor
+    from contextifier import DocumentProcessor, open_raw
 
+    # AI-friendly view (lightweight, normalized)
     processor = DocumentProcessor()
     text = processor.extract_text("document.pdf")
     chunks = processor.extract_chunks("document.pdf", chunk_size=1000)
+
+    # Raw view (lossless, addressable, WRITABLE — xlsx/docx/pptx)
+    raw = open_raw("report.xlsx")
+    raw.sheets["Sales"].set_cell("B3", 142)
+    raw.save("report-edited.xlsx")     # untouched parts stay byte-identical
 """
 
 from importlib.metadata import version, PackageNotFoundError
@@ -43,6 +49,7 @@
 from contextifier.types import ExtractionResult, FileContext, Chunk, ChunkMetadata
 from contextifier.chunking.chunker import TextChunker
 from contextifier.errors import ContextifierError, UnsupportedFormatError
+from contextifier.raw import open_raw
 from contextifier.async_processor import AsyncDocumentProcessor
 from contextifier.cached_processor import CachedDocumentProcessor
 
@@ -63,6 +70,8 @@
     "ChunkMetadata",
     # Chunking
     "TextChunker",
+    # Raw (lossless, writable) access
+    "open_raw",
     # Errors
     "ContextifierError",
     "UnsupportedFormatError",

contextifier/cached_processor.py

@@ -27,7 +27,7 @@
 import json
 import logging
 from pathlib import Path
-from typing import Any, Dict, List, Optional, Protocol, Union
+from typing import Any, Dict, Optional, Protocol, Union
 
 from contextifier.config import ProcessingConfig
 from contextifier.document_processor import DocumentProcessor, ChunkResult
@@ -47,6 +47,7 @@
 
 # ── Cache backend protocol ────────────────────────────────────────────────
 
+
 class CacheBackend(Protocol):
     """Minimal interface for a cache backend."""
 
@@ -61,6 +62,7 @@ def put(self, key: str, value: str) -> None:
 
 # ── Built-in backends ────────────────────────────────────────────────────
 
+
 class MemoryCacheBackend:
     """
     In-memory LRU cache using OrderedDict.
@@ -125,6 +127,7 @@ def put(self, key: str, value: str) -> None:
 
 # ── Cached processor ─────────────────────────────────────────────────────
 
+
 class CachedDocumentProcessor:
     """Document processor with transparent result caching."""
 
@@ -176,7 +179,10 @@ def process(
     ) -> ExtractionResult:
         """Process a file with cache lookup for the full ExtractionResult."""
         cache_key = self._make_key(
-            str(file_path), extract_metadata, ocr_processing, suffix="process",
+            str(file_path),
+            extract_metadata,
+            ocr_processing,
+            suffix="process",
         )
         cached = self._cache.get(cache_key)
         if cached is not None:
@@ -208,9 +214,11 @@ def extract_chunks(
     ) -> ChunkResult:
         """Extract and chunk text with cache lookup."""
         cache_key = self._make_key(
-            str(file_path), extract_metadata, ocr_processing,
+            str(file_path),
+            extract_metadata,
+            ocr_processing,
             suffix=f"chunks|cs={chunk_size}|co={chunk_overlap}"
-                   f"|pt={preserve_tables}|pm={include_position_metadata}",
+            f"|pt={preserve_tables}|pm={include_position_metadata}",
         )
         cached = self._cache.get(cache_key)
         if cached is not None:
@@ -283,6 +291,7 @@ def __repr__(self) -> str:
 
 # ── Serialization helpers ─────────────────────────────────────────────────
 
+
 def _serialize_extraction_result(result: ExtractionResult) -> str:
     """Serialize an ExtractionResult to a JSON string."""
     data: Dict[str, Any] = {
@@ -302,7 +311,9 @@ def _deserialize_extraction_result(s: str) -> ExtractionResult:
     data = json.loads(s)
     return ExtractionResult(
         text=data.get("text", ""),
-        metadata=DocumentMetadata.from_dict(data["metadata"]) if data.get("metadata") else None,
+        metadata=DocumentMetadata.from_dict(data["metadata"])
+        if data.get("metadata")
+        else None,
         images=data.get("images", []),
         page_count=data.get("page_count", 0),
         warnings=data.get("warnings", []),
@@ -325,7 +336,9 @@ def _serialize_chunk_result(result: ChunkResult) -> str:
                     "line_end": c.metadata.line_end,
                     "global_start": c.metadata.global_start,
                     "global_end": c.metadata.global_end,
-                } if c.metadata else None,
+                }
+                if c.metadata
+                else None,
             }
             for c in result.chunks_with_metadata
         ]
@@ -366,17 +379,19 @@ def _deserialize_chunk_result(s: str) -> ChunkResult:
 def _table_to_dict(table: TableData) -> Dict[str, Any]:
     rows = []
     for row in table.rows:
-        rows.append([
-            {
-                "content": c.content,
-                "row_span": c.row_span,
-                "col_span": c.col_span,
-                "is_header": c.is_header,
-                "row_index": c.row_index,
-                "col_index": c.col_index,
-            }
-            for c in row
-        ])
+        rows.append(
+            [
+                {
+                    "content": c.content,
+                    "row_span": c.row_span,
+                    "col_span": c.col_span,
+                    "is_header": c.is_header,
+                    "row_index": c.row_index,
+                    "col_index": c.col_index,
+                }
+                for c in row
+            ]
+        )
     return {
         "rows": rows,
         "num_rows": table.num_rows,
@@ -389,17 +404,19 @@ def _table_to_dict(table: TableData) -> Dict[str, Any]:
 def _dict_to_table(d: Dict[str, Any]) -> TableData:
     rows = []
     for row_data in d.get("rows", []):
-        rows.append([
-            TableCell(
-                content=c["content"],
-                row_span=c.get("row_span", 1),
-                col_span=c.get("col_span", 1),
-                is_header=c.get("is_header", False),
-                row_index=c.get("row_index", 0),
-                col_index=c.get("col_index", 0),
-            )
-            for c in row_data
-        ])
+        rows.append(
+            [
+                TableCell(
+                    content=c["content"],
+                    row_span=c.get("row_span", 1),
+                    col_span=c.get("col_span", 1),
+                    is_header=c.get("is_header", False),
+                    row_index=c.get("row_index", 0),
+                    col_index=c.get("col_index", 0),
+                )
+                for c in row_data
+            ]
+        )
     return TableData(
         rows=rows,
         num_rows=d.get("num_rows", 0),
@@ -414,10 +431,7 @@ def _chart_to_dict(chart: ChartData) -> Dict[str, Any]:
         "chart_type": chart.chart_type,
         "title": chart.title,
         "categories": chart.categories,
-        "series": [
-            {"name": s.name, "values": s.values}
-            for s in chart.series
-        ],
+        "series": [{"name": s.name, "values": s.values} for s in chart.series],
     }
 
 

contextifier/chunking/chunker.py

@@ -29,7 +29,9 @@
 from contextifier.chunking.strategies.base import BaseChunkingStrategy
 from contextifier.chunking.strategies.page_strategy import PageChunkingStrategy
 from contextifier.chunking.strategies.table_strategy import TableChunkingStrategy
-from contextifier.chunking.strategies.protected_strategy import ProtectedChunkingStrategy
+from contextifier.chunking.strategies.protected_strategy import (
+    ProtectedChunkingStrategy,
+)
 from contextifier.chunking.strategies.plain_strategy import PlainChunkingStrategy
 
 
@@ -69,12 +71,14 @@ def __init__(
             self._strategies.extend(custom_strategies)
 
         # Built-in strategies
-        self._strategies.extend([
-            TableChunkingStrategy(),
-            PageChunkingStrategy(),
-            ProtectedChunkingStrategy(),
-            PlainChunkingStrategy(),
-        ])
+        self._strategies.extend(
+            [
+                TableChunkingStrategy(),
+                PageChunkingStrategy(),
+                ProtectedChunkingStrategy(),
+                PlainChunkingStrategy(),
+            ]
+        )
 
         # Sort by priority (lower = higher priority)
         self._strategies.sort(key=lambda s: s.priority)
@@ -152,11 +156,14 @@ def chunk(
                 from contextifier.chunking.metadata_enricher import (
                     enrich_chunk_metadata,
                 )
+
                 result = enrich_chunk_metadata(result)
             return result
         except NotImplementedError:
             # Strategy not yet implemented — try next
-            return self._fallback_chunk(text, effective_config, ext, include_meta, **kwargs)
+            return self._fallback_chunk(
+                text, effective_config, ext, include_meta, **kwargs
+            )
         except Exception as e:
             raise ChunkingError(
                 f"Chunking failed with strategy '{strategy.strategy_name}': {e}",
@@ -209,7 +216,8 @@ def _fallback_chunk(
             try:
                 if strategy.can_handle(text, config, file_extension=ext, **context):
                     return strategy.chunk(
-                        text, config,
+                        text,
+                        config,
                         file_extension=ext,
                         include_position_metadata=include_meta,
                         **context,
@@ -220,7 +228,9 @@ def _fallback_chunk(
                 continue
 
         # Ultimate fallback: return text as single chunk
-        self._logger.warning("All chunking strategies failed, returning text as single chunk")
+        self._logger.warning(
+            "All chunking strategies failed, returning text as single chunk"
+        )
         if include_meta:
             return [Chunk(text=text, metadata=ChunkMetadata(chunk_index=0))]
         return [text]