[Compiler] Decide tokenizer + recursive-descent parser direction (ADR)

[cssbruno]

Problem

The parser is line-oriented: it scans the source line by line with a
bufio.Scanner, matches patterns per trimmed line, and returns on the first
syntax error with no recovery. Spans are line-wide approximations and the
formatter is whitespace-only string manipulation. Most of the work deferred to
#250 — parser error recovery, an AST-backed formatter, exact token spans, and
granular per-construct diagnostic codes — is downstream of this single
foundation. Right now the line-oriented parser is effectively deferred by
omission rather than by an explicit decision.

Relevant code:

• internal/parser/syntax.go: bufio.Scanner line loop, first-error return
• internal/parser/patterns.go: per-line lexLine tokenizer
• internal/parser/braces.go: stateful brace scanner (accepted JS-regex limit)
• internal/lang/format.go: line-oriented formatter

Scope

This is a decision/ADR issue, not an implementation task. Decide whether M2
commits to a real tokenizer + recursive-descent parser with error recovery, or
explicitly keeps the line-oriented parser with documented limits for now.

If committing: outline the token-stream layer, the recovery strategy
(continue-after-error so one syntax error does not hide the rest of the file),
and the span model — which depends on byte offsets in SourcePosition.

If deferring: document the limitation explicitly and link the dependent issues so
it is a conscious choice rather than an omission.

Acceptance Criteria

• An ADR (docs/engineering/decisions/) records the chosen direction and
  rationale.
• If deferred, the line-oriented limitation is documented (not deferred by
  omission) and the dependent deferral ([Language] Track remaining parser/spec hardening follow-ups #250) references the decision.
• If committed, a phased plan exists covering tokenizer, recovery, span model,
  and formatter, with byte-offset spans as a dependency.

Verification

• ADR present under docs/engineering/decisions/.
• docs/compiler/pipeline.md and docs/engineering/architecture.md updated to
  match the decision.

[cssbruno]

Decision recorded: ADR 0010 — Tokenizer and Recursive-Descent Parser Direction (docs/engineering/decisions/0010-tokenizer-recursive-descent-parser.md), status Accepted.

Outcome: commit to a single shared tokenizer + recursive-descent parser with error recovery, producing the existing internal/gwdkast AST, migrated incrementally behind that AST seam.

Rationale in brief:

• The documented target pipeline already names a lex/parse full AST front-end, so this makes the parser-internals decision explicit rather than introducing a new direction.
• A real rune tokenizer already exists in internal/lang.Lex but only tooling uses it; the compiler parser re-lexes per line. The decision unifies on one lexer.
• gwdkast.File is a stable seam — gwdkanalysis/gwdkir/validation/codegen consume it, so the parser can be replaced behind it without disturbing downstream passes.

Sequencing: #294 (byte offsets) is the prerequisite and lands first → shared tokenizer → recursive-descent parser to gwdkast.File with recovery, gated by golden AST-equivalence tests + the conformance corpus (#295) → cut over per declaration kind → remove the line-oriented path. AST-backed formatter and granular per-construct codes (#250) follow; #296 is the interim formatter guard.

Docs updated: docs/engineering/decisions/README.md index and docs/compiler/pipeline.md target-pipeline pointer.

[cssbruno]

Implemented and merged in #300.