Follow-up from #294 and PR #300.
Problem
#294 added a byte Offset to source.SourcePosition plus source.PositionAt
and source.OffsetOf conversion helpers, but left the field unpopulated by the
front-end: the line-oriented parser does not scan byte positions, so positions it
produces have Offset == 0 while Line/Column are set. The substrate exists;
nothing fills it yet.
This is the deferred half of #294's acceptance ("the parser populates the offset
for records where it scans byte positions"). It is sequenced behind the parser
work in ADR 0010 (docs/engineering/decisions/0010-tokenizer-recursive-descent-parser.md).
Relevant code:
internal/source/source.go: SourcePosition.Offset, PositionAt, OffsetOf
internal/lang/lexer.go: the rune scanner that already tracks an index
internal/parser: AST span construction
internal/compiler/validate_spans.go: span synthesis that could set Offset
Scope
Populate byte offsets at the points that genuinely scan source: the shared
tokenizer (once it exists per ADR 0010) emits offsets on every token, and AST
span builders carry them into gwdkast/gwdkir. Where a buffer is available,
backfill offsets with source.PositionAt/OffsetOf rather than leaving them
zero.
Acceptance Criteria
- Tokens and AST node spans carry accurate byte offsets, byte-for-byte
consistent with their line/column via the round-trip helpers.
- Positions with no scanned offset remain explicitly zero (no guessing).
check --json shape stays backward compatible.
- Tests assert offset/line/column consistency on parsed fixtures.
Verification
go test ./internal/source ./internal/parser ./internal/lang ./internal/compiler
Follow-up from #294 and PR #300.
Problem
#294 added a byte
Offsettosource.SourcePositionplussource.PositionAtand
source.OffsetOfconversion helpers, but left the field unpopulated by thefront-end: the line-oriented parser does not scan byte positions, so positions it
produces have
Offset == 0whileLine/Columnare set. The substrate exists;nothing fills it yet.
This is the deferred half of #294's acceptance ("the parser populates the offset
for records where it scans byte positions"). It is sequenced behind the parser
work in ADR 0010 (
docs/engineering/decisions/0010-tokenizer-recursive-descent-parser.md).Relevant code:
internal/source/source.go:SourcePosition.Offset,PositionAt,OffsetOfinternal/lang/lexer.go: the rune scanner that already tracks an indexinternal/parser: AST span constructioninternal/compiler/validate_spans.go: span synthesis that could set OffsetScope
Populate byte offsets at the points that genuinely scan source: the shared
tokenizer (once it exists per ADR 0010) emits offsets on every token, and AST
span builders carry them into
gwdkast/gwdkir. Where a buffer is available,backfill offsets with
source.PositionAt/OffsetOfrather than leaving themzero.
Acceptance Criteria
consistent with their line/column via the round-trip helpers.
check --jsonshape stays backward compatible.Verification
go test ./internal/source ./internal/parser ./internal/lang ./internal/compiler