feat: add repl precompiler opt-in and docs hardening

README.md

@@ -19,8 +19,16 @@ The model performs reasoning and generation while the compiler manages premise a
 ```bash
 pip install context-compiler
 context-compiler
+context-compiler --with-precompiler
 ```
 
+`context-compiler` launches the interactive REPL.
+
+`--with-precompiler` enables the experimental precompiler before each REPL turn
+(heuristic + validation only). Near-miss inputs are not rewritten and are
+passed through to the engine, which continues to return clarify behavior for
+those forms.
+
 Or in code:
 ```python
 from context_compiler import create_engine
@@ -375,8 +383,9 @@ into canonical directives before compilation.
 
 It is designed to be conservative and must be used with validation:
 
-- heuristic-first, with LLM fallback when needed
+- reject-first; directive-adjacent unsafe forms abstain instead of rewriting
 - all outputs must be validated with `parse_precompiler_output(...)`
+- no directive grammar expansion
 - raw outputs must not be passed directly to the compiler
 
 See [LLM preprocessor](docs/llm-preprocessor.md) and

docs/DescriptionAndMilestones.md

@@ -63,6 +63,21 @@ The current authoritative state shape and directive semantics are defined in `Di
 
 After correcting or constraining the assistant once, the behavior remains consistent for the rest of the conversation.
 
+### 0.6.9 — Precompiler Hardening + REPL Opt-In (implemented)
+
+**Goal**
+Harden experimental precompiler behavior while preserving core engine semantics.
+
+**Deliverables:**
+
+- Reject-first precompiler classification behavior (`directive` / `no_directive` / `unknown`)
+- Portable precompiler conformance fixtures for cross-language runners (TS-ready shape)
+- REPL opt-in flag: `context-compiler --with-precompiler`
+- No directive grammar expansion and no engine semantic changes
+
+**User-visible outcome:**
+Safer precompiler behavior and explicit REPL opt-in without changing deterministic engine outcomes.
+
 ### M3 — Cross-Session Recall
 
 **Goal**

examples/integrations/litellm/README.md

@@ -129,19 +129,20 @@ These files are importable integration references for host applications.
 - Basic: passes raw user input to `engine.step(...)`.
 - With preprocessor: runs heuristic precompiler first.
   - If heuristic returns a directive, that directive is passed to `engine.step(...)`.
-  - If heuristic does not resolve to a directive (`no_directive`), LLM fallback prompt conversion runs.
+  - If heuristic does not produce a directive (`no_directive` or `unknown`), LLM fallback prompt conversion runs.
   - If fallback yields nothing usable or errors, behavior safely remains equivalent to basic.
+  - Behavior is reject-first and does not expand directive grammar.
 
 ## Example checks
 
-- Near-miss canonicalization (`with_preprocessor.py`):
-  - `set premise to concise replies` -> precompiler can canonicalize to `set premise concise replies`.
+- Near-miss passthrough (`with_preprocessor.py`):
+  - `set premise to concise replies` is not rewritten by the precompiler and is passed through unchanged.
+  - Engine returns clarify (`Did you mean 'set premise concise replies'?`).
 - Lifecycle enforcement (both):
   - `change premise to formal tone` with no premise -> clarify (`set premise ...` first).
 - Conflict semantics (both):
   - `use docker` then `prohibit docker` -> conflict clarify.
 - Replacement precondition (both):
   - `use podman instead of docker` without prior `use docker` -> replacement clarify.
-- NL upgrade / abstain (`with_preprocessor.py`):
-  - `please use docker` may upgrade to `use docker`.
-  - `I usually use docker` should abstain (`no directive`).
+- Directive-adjacent abstain (`with_preprocessor.py`):
+  - `change premise concise replies` is classified as `unknown`, not rewritten, and handled by engine clarify.

examples/integrations/openwebui/README.md

@@ -99,16 +99,16 @@ Validate clarify short-circuit, passthrough forwarding, update injection with on
 
 **Case 4**
 
-- prompt(s): `clear state` → `set premise to concise replies` → `set premise formal tone`
-- base model: accepts both as conversational style requests
-- basic pipe: `Did you mean 'set premise concise replies'?` then conversational formal-tone rewrite
-- preprocessor pipe: `Premise set: Concise replies.` then `Premise already exists...`
-- why this is a real win: preprocessor canonicalizes near-miss form and preserves premise-slot semantics end-to-end.
+- prompt(s): `clear state` → `set premise to concise replies`
+- base model: accepts conversational style phrasing
+- basic pipe: `Did you mean 'set premise concise replies'?`
+- preprocessor pipe: same clarify (near-miss is not rewritten)
+- why this is a real win: precompiler stays reject-first and preserves engine-owned clarify behavior.
 
 **Case 5**
 
 - prompt(s): `clear state` → `change premise concise replies`
 - base model: generic “please clarify changes” response
 - basic pipe: `Did you mean 'change premise to concise replies'?`
-- preprocessor pipe: `No premise exists yet. Use 'set premise ...' first.`
-- why this is a real win: preprocessor upgrades near-miss form and reaches the correct lifecycle clarify state instead of stopping at syntax clarify.
+- preprocessor pipe: same clarify (near-miss is passed through unchanged)
+- why this is a real win: near-miss inputs are not canonicalized, so directive semantics stay engine-owned.

experimental/preprocessor/README.md

@@ -30,14 +30,31 @@ Public validator entry point:
 All preprocessor outputs (heuristic or LLM) must be validated with
 `parse_precompiler_output(...)` before being applied.
 
+Classification contract:
+
+- `directive`: safe, validated canonical directive (`output` is a directive string)
+- `no_directive`: confident ordinary content (`output` is `null`)
+- `unknown`: unsafe to rewrite (`output` is `null`)
+
+`unknown` is reject/abstain behavior. Malformed, ambiguous, mixed-intent,
+quoted/reported, unsupported, or unsafe outputs must not be rewritten.
+
+Only validated `directive` output may be used as rewritten compiler input.
+`no_directive` and `unknown` must fall back to original user input.
+
 `source_input` is optional at the API level for backward compatibility.
-For integration behavior, it is required for LLM fallback validation calls:
+For integration behavior, it is REQUIRED for LLM fallback validation calls:
 pass `source_input=<original user text>` so source-aware reject rules can
-block unsafe rewrites (for example, engine-owned premise near-miss
-canonicalization).
+block unsafe rewrites.
+
+Engine-owned near-misses are reject cases (for example `set premise to X`,
+`change premise X`) and must remain `unknown` (not rewritten).
 
 Raw preprocessor/LLM outputs must not be passed directly to the compiler.
 
+The precompiler does not expand directive grammar. It may emit only validated
+canonical directives accepted by the compiler.
+
 ## Safe usage pattern
 
 1. Run `precompile_heuristic(message)`.

src/context_compiler/repl.py

@@ -1,18 +1,21 @@
 import sys
 from typing import TextIO
 
+from experimental.preprocessor.output_validation import parse_precompiler_output
+
 from . import __version__, create_engine, get_policy_items, get_premise_value
-from .engine import Decision, DecisionKind, State
+from .engine import Decision, DecisionKind, Engine, State
 
 _EXIT_TOKENS = {"exit", "quit"}
 _HELP_TOKENS = {"help", "?"}
 _MULTI_COMMAND_PROMPT = "Multiple commands detected.\nEnter one command per line."
 _CLI_HELP_TEXT = """Usage:
-  context-compiler [--help] [--version]
+  context-compiler [--help] [--version] [--with-precompiler]
 
 Options:
-  --help      Show this help message and exit.
-  --version   Show the installed context-compiler version and exit.
+  --help               Show this help message and exit.
+  --version            Show the installed context-compiler version and exit.
+  --with-precompiler   Enable precompiler before each REPL turn (heuristic + validation only)
 """
 
 
@@ -94,7 +97,21 @@ def _print_decision_lines(decision: Decision, out_stream: TextIO, *, leading_bla
         print(line, file=out_stream)
 
 
-def run_repl(in_stream: TextIO, out_stream: TextIO) -> None:
+def _has_pending_clarification(engine: Engine) -> bool:
+    checkpoint = engine.export_checkpoint()
+    return checkpoint["pending"] is not None
+
+
+def _compile_input(raw_input: str, engine: Engine, *, use_precompiler: bool) -> str:
+    if not use_precompiler:
+        return raw_input
+    if _has_pending_clarification(engine):
+        return raw_input
+    parsed = parse_precompiler_output(raw_input, source_input=raw_input)
+    return parsed if parsed is not None else raw_input
+
+
+def run_repl(in_stream: TextIO, out_stream: TextIO, *, use_precompiler: bool = False) -> None:
     engine = create_engine()
 
     if _is_interactive(in_stream, out_stream):
@@ -118,7 +135,8 @@ def run_repl(in_stream: TextIO, out_stream: TextIO) -> None:
                 _print_interactive_help(out_stream)
                 continue
 
-            decision = engine.step(user_input)
+            compile_input = _compile_input(user_input, engine, use_precompiler=use_precompiler)
+            decision = engine.step(compile_input)
             _print_decision_lines(decision, out_stream, leading_blank=True)
         return
 
@@ -129,7 +147,8 @@ def run_repl(in_stream: TextIO, out_stream: TextIO) -> None:
         user_input = line.rstrip("\n")
         if user_input.strip().lower() in _EXIT_TOKENS:
             return
-        decision = engine.step(user_input)
+        compile_input = _compile_input(user_input, engine, use_precompiler=use_precompiler)
+        decision = engine.step(compile_input)
         _print_decision_lines(decision, out_stream, leading_blank=False)
 
 
@@ -147,6 +166,10 @@ def main() -> int:  # pragma: no cover
         print(__version__, file=sys.stdout)
         return 0
 
+    if args == ["--with-precompiler"]:
+        run_repl(sys.stdin, sys.stdout, use_precompiler=True)
+        return 0
+
     bad_arg = args[0]
     print(f"error: unknown option '{bad_arg}'", file=sys.stderr)
     print("Try 'context-compiler --help' for usage.", file=sys.stderr)