Skip to content

[Compiler] Populate byte offsets from the parser front-end #301

@cssbruno

Description

@cssbruno

Follow-up from #294 and PR #300.

Problem

#294 added a byte Offset to source.SourcePosition plus source.PositionAt
and source.OffsetOf conversion helpers, but left the field unpopulated by the
front-end: the line-oriented parser does not scan byte positions, so positions it
produces have Offset == 0 while Line/Column are set. The substrate exists;
nothing fills it yet.

This is the deferred half of #294's acceptance ("the parser populates the offset
for records where it scans byte positions"). It is sequenced behind the parser
work in ADR 0010 (docs/engineering/decisions/0010-tokenizer-recursive-descent-parser.md).

Relevant code:

  • internal/source/source.go: SourcePosition.Offset, PositionAt, OffsetOf
  • internal/lang/lexer.go: the rune scanner that already tracks an index
  • internal/parser: AST span construction
  • internal/compiler/validate_spans.go: span synthesis that could set Offset

Scope

Populate byte offsets at the points that genuinely scan source: the shared
tokenizer (once it exists per ADR 0010) emits offsets on every token, and AST
span builders carry them into gwdkast/gwdkir. Where a buffer is available,
backfill offsets with source.PositionAt/OffsetOf rather than leaving them
zero.

Acceptance Criteria

  • Tokens and AST node spans carry accurate byte offsets, byte-for-byte
    consistent with their line/column via the round-trip helpers.
  • Positions with no scanned offset remain explicitly zero (no guessing).
  • check --json shape stays backward compatible.
  • Tests assert offset/line/column consistency on parsed fixtures.

Verification

go test ./internal/source ./internal/parser ./internal/lang ./internal/compiler

Metadata

Metadata

Assignees

No one assigned

    Labels

    compilerCompiler internals, pipeline, and generated metadataparser.gwdk parser and syntax handling

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions