Native Windows AST

[giveen]

AST Parser Adoption Plan (Unix + Windows)

Goal

Adopt AST-backed command analysis to improve safety classification and reduce brittle string parsing across platforms.

Scope

• Unix shell parsing via mvdan/sh
• Windows PowerShell parsing via System.Management.Automation (Parser::ParseInput)
• Shared intermediate representation (IR) consumed by a single policy engine

Non-Goals (for initial rollout)

• Full semantic evaluation/execution simulation
• Rewriting command execution engine
• Replacing all existing safeguards on day 1 (we will stage and compare)

---

Architecture Overview

1) Parser Interface

Introduce a platform-neutral parser interface:

• Parse(command string, platform Platform) (ParsedIR, error)

2) Shared Parsed IR

Define a compact JSON-safe IR shape with:

• platform (unix | windows)
• commands[] (base commands/cmdlets found)
• operators[] (|, &&, ;, etc.)
• redirects[] (>, >>, <, etc.)
• expansions[] (var, glob, subshell, etc.)
• risk_flags[] (normalized policy signals)
• parse_errors[] (syntax/parse diagnostics)

3) Platform Adapters

• Unix adapter: in-process parser using mvdan/sh
• Windows adapter: out-of-process parser adapter using pwsh + System.Management.Automation.Language.Parser::ParseInput

4) Policy Engine

Policy consumes only ParsedIR:

• approval decisions
• deny/require-confirm decisions
• reason codes
• auto-approval carveouts (e.g., safe new-path create)

---

Implementation Phases

Phase 0: Baseline & Safety Net

• Freeze current behavior with tests (goldens/table tests).
• Capture representative command corpus (safe, unsafe, edge cases).
• Add decision snapshot tests for current policy outputs.

Exit Criteria

• Baseline decisions are reproducible and versioned.

---

Phase 1: Shared IR + Parser Abstraction

• Add ParsedIR types and parser interface.
• Keep existing policy unchanged for now.
• Add parser selection scaffolding by platform.

Exit Criteria

• Build passes with no behavior changes.

---

Phase 2: Unix AST Adapter (mvdan/sh)

• Integrate mvdan/sh parser.
• Map AST nodes to ParsedIR risk signals.
• Add focused tests for operators/redirections/expansions/subshell.

Exit Criteria

• Unix parser adapter stable with deterministic IR outputs.

---

Phase 3: Windows AST Adapter (System.Management.Automation)

• Create PowerShell parser bridge script:
  • parse via Parser::ParseInput
  • emit compact JSON IR
• Invoke bridge from Go (pwsh -NoProfile -NonInteractive).
• Handle parser transport errors and malformed output safely (fail closed).

Exit Criteria

• Windows parser adapter returns stable IR in CI compile/runtime checks.

---

Phase 4: Dual-Path (Shadow Mode)

• Run AST policy path in parallel with legacy string logic.
• Do not enforce AST yet; compare and log decision deltas.
• Collect mismatch telemetry for command corpus and real-world usage.

Exit Criteria

• Mismatch rate is low and understood.
• No critical false-negative safety regressions.

---

Phase 5: Gradual Enforcement

• Enable AST-backed policy for low-risk cohorts first.
• Keep fallback to legacy path behind feature flag.
• Expand enforcement after validation windows.

Exit Criteria

• Stable approval behavior with reduced false prompts.
• No increase in unsafe auto-approvals.

---

Phase 6: Decommission Legacy Parsing

• Remove duplicated string-parsing branches once AST path is stable.
• Keep minimal defensive guards (input sanitation/fail-closed behavior).
• Simplify tests to AST-first contracts + integration.

Exit Criteria

• Reduced code size and complexity.
• Single authoritative decision path.

---

Windows Adapter Invocation Strategy

Recommended (initial)

Out-of-process call:

• Go -> pwsh process -> System.Management.Automation parser -> JSON IR -> Go

Why

• Avoids .NET hosting complexity in Go process.
• Easier to test, isolate, and deploy incrementally.

Optimization path (later)

• Persistent parser worker (long-lived process over stdio/named pipe) to reduce startup cost.

---

Security & Reliability Requirements

1. Fail Closed

• If parser fails or IR invalid -> require confirmation (or block, per policy).

2. Strict Output Contract

• Validate JSON schema/version from adapters.
• Reject unknown/malformed payloads.

3. Deterministic Reason Codes

• Emit normalized reason enums (e.g., operator, redirect, expansion, invoke_expression).

4. No Full AST in Policy

• Policy consumes compact IR only (keeps policy deterministic and small).

---

Testing Plan

Unit

• Unix adapter IR mapping tests
• Windows adapter JSON contract tests
• Policy tests over synthetic IR fixtures

Integration

• End-to-end parse + policy decision tests by platform
• Snapshot tests for known tricky commands

Regression Corpus

• Commands previously causing over/under-prompting
• Include pipelines, redirects, nested expressions, quoting edge cases

CI

• go test ./... (existing matrix)
• Windows path compile/runtime checks for parser adapter availability
• Contract test for parser bridge script output schema

---

Rollout Controls

• Feature flags:
  • ast_policy_enabled
  • ast_shadow_mode
  • ast_windows_enabled
• Metrics:
  • parser failures
  • policy mismatch rates (shadow mode)
  • prompt rate changes
  • unsafe auto-approval incidents (target: zero)

---

Risks & Mitigations

1. Risk: AST mismatch with runtime shell behavior

• Mitigation: shadow mode + curated corpus + conservative fail-closed policy.

2. Risk: Windows adapter process overhead

• Mitigation: start with correctness; optimize with persistent worker later.

3. Risk: Parser bridge drift/schema break

• Mitigation: schema versioning + strict validation + contract tests.

4. Risk: Increased complexity during migration

• Mitigation: phased rollout; delete legacy path only after stable metrics.

---

Deliverables Checklist

• ParsedIR schema + versioning
• Parser interface and platform registry
• Unix adapter (mvdan/sh)
• Windows bridge + adapter (System.Management.Automation)
• Shadow-mode comparison framework
• Feature flags + rollout toggles
• Contract/unit/integration/regression tests
• Migration docs and operational playbook

---

Success Criteria

• Lower prompt noise without weakening safety
• Single policy engine across platforms
• Fewer parser-related edge-case regressions
• Net reduction in duplicated command parsing logic
• Measurable codebase simplification after legacy removal

[giveen]

@mlhher I'll wait till you are done with your AST work before looking more in depth with this.