Package documentation for parser and server

AGENTS.md

@@ -7,15 +7,19 @@ For project documentation, read the package documentation written in Go doc form
 - `doc.go`: General overview of the Fastbelt framework
 - `grammar/doc.go`: Reference documentation for the grammar language
 - `lexer/doc.go`: Shared lexer runtime used by Fastbelt-generated languages
+- `parser/doc.go`: Shared parser runtime used by Fastbelt-generated languages
 - `textdoc/doc.go`: Text documents, overlays, and LSP position mapping
 - `linking/doc.go`: Cross-reference resolution, symbol tables, and scopes
 - `workspace/doc.go`: Document lifecycle, loading, edits, and the build pipeline
+- `server/doc.go`: Language Server Protocol (LSP) implementation
+- `test/doc.go`: Utilities for testing Fastbelt language implementations
 
 Utility packages:
 
 - `util/codegen/doc.go`: Building indented multi-line source for code generators
 - `util/collections/doc.go`: Generic collection data structures (e.g. MultiMap)
 - `util/extiter/doc.go`: Utilities for `iter.Seq` sequences
+- `util/parallel/doc.go`: Utilities for parallel iteration over slices
 - `util/service/doc.go`: Typed dependency injection container
 
 ## VS Code extensions

parser/doc.go

@@ -0,0 +1,121 @@
+// Copyright 2026 TypeFox GmbH
+// This program and the accompanying materials are made available under the
+// terms of the MIT License, which is available in the project root.
+
+// Package parser provides the shared parser runtime used by Fastbelt-generated
+// languages. It does not implement parsing for any particular language; each
+// grammar gets its own generated recursive-descent parser that drives the
+// parser state, adaptive prediction, error recovery, and completion machinery
+// of this package.
+//
+// Run [typefox.dev/fastbelt/cmd/fastbelt] generate on a .fb grammar file to
+// emit the files described below. See [typefox.dev/fastbelt] for the overall
+// toolchain and [typefox.dev/fastbelt/grammar] for how parser rules are
+// declared in the grammar language.
+//
+// # Generated parsers
+//
+// Code generation turns the parser rules of a grammar into four cooperating
+// files:
+//
+//   - `parser_gen.go` — a Parser implementing [Parser], with one ParseX method
+//     per parser rule. Each method allocates the rule's AST node, matches
+//     tokens through [ParserState], assigns them to node fields, and records
+//     the node's token segment for source ranges.
+//   - `parser_lookahead_gen.go` — the lookahead service, with one method per
+//     decision point in the grammar: which alternative to take, whether to
+//     enter an optional group, whether to iterate a loop once more. It is an
+//     interface with a default implementation, so individual decisions can be
+//     overridden without touching the parser.
+//   - `atn_gen.go` — the [RuntimeATN], an augmented transition network that
+//     encodes the grammar as a graph of states with token, epsilon, and rule
+//     transitions. It backs adaptive prediction, error recovery, and
+//     completion.
+//   - `completion_parser_gen.go` — a stripped-down twin of the parser that
+//     builds no AST and instead records the parse progress snapshots that
+//     code completion needs.
+//
+// The generated `SetupGeneratedServices` registers the Parser, the lookahead
+// service, and the default [ErrorRecoveryStrategy] and [ErrorMessageProvider]
+// in the service container ([typefox.dev/fastbelt/util/service]), each unless
+// a custom implementation was registered first. During document processing,
+// [typefox.dev/fastbelt/workspace.DefaultDocumentParser] obtains the [Parser],
+// calls Parse on the tokens produced by [typefox.dev/fastbelt/lexer], and
+// stores the resulting root [core.AstNode] and syntax errors ([ParseResult])
+// on the document.
+//
+// # Parsing model
+//
+// A generated parser is a recursive-descent parser: one method per rule,
+// descending into sub-rules and consuming tokens left to right. [ParserState]
+// carries the per-parse state: token slice, cursor, error mode, and rule stack.
+//
+// Wherever the grammar offers a choice — between alternatives, at an optional
+// element, at a loop continuation — the parser asks its lookahead service.
+// Depending on the grammar, the generated decision method uses one of three
+// strategies, cheapest first:
+//
+//  1. A direct check of the next token's type, when a single expected token
+//     decides.
+//  2. An [LL1Lookahead] table resolved by [ParserState.Lookahead] in O(1),
+//     when one token of lookahead decides.
+//  3. Adaptive ALL(*) prediction via [ParserState.AdaptivePredict], when
+//     arbitrary lookahead is required. The predictor simulates the
+//     [RuntimeATN] ahead of the current position and caches its decisions in
+//     per-decision DFAs, so hot decisions approach table-lookup cost over
+//     time. The cache is internally synchronized and shared across
+//     concurrent parses.
+//
+// Adaptive prediction runs in [PredictionModeSLL] by default;
+// [ParserLookahead.SetPredictionMode] enables full-context [PredictionModeLL]
+// for grammars that need it.
+//
+// Cross-reference assignments in the grammar do not resolve anything at parse
+// time: the parser stores an unresolved reference created by the generated
+// references constructor, and the [typefox.dev/fastbelt/linking] phase
+// resolves it later.
+//
+// # Error recovery
+//
+// Parse errors do not stop the parse. [Parser] implementations always return
+// an AST — with nil tokens or unset fields where input was missing — plus the
+// collected [core.ParserError] values, so language tooling keeps working on
+// broken input. While the parser is recovering, follow-up errors are
+// suppressed so one underlying mistake produces one diagnostic. The pluggable
+// [ErrorRecoveryStrategy] controls resynchronization:
+//
+//   - [DefaultErrorRecovery] attempts single-token deletion when Consume hits
+//     an unexpected token, discards tokens before optional/loop guards until
+//     one fits the decision or an enclosing follow set (Sync), and skips to
+//     the follow set after unwinding from a failed rule (Recover). Because
+//     generated AST nodes tolerate nil tokens, missing tokens are reported
+//     but never fabricated.
+//   - [BailErrorRecovery] stops at the first mismatch and unwinds, for
+//     two-stage parsing or when a broken parse result would be discarded
+//     anyway.
+//
+// Message texts come from the pluggable [ErrorMessageProvider]; replace it in
+// the service container to change wording or to localize.
+//
+// # Code completion
+//
+// The package also contains the parser side of code completion, driven by
+// [typefox.dev/fastbelt/server.DefaultCompletionProvider]:
+//
+//  1. The generated CompletionParser reparses the document's tokens up to the
+//     cursor. It mirrors the main parser's control flow but mutates no AST;
+//     via [CompletionParserState] it records an [ATNSnapshot] — token index,
+//     ATN state, rule stack — at every rule entry and sync point.
+//  2. [CompletionParseResult.SimulateAt] picks a suitable snapshot and
+//     advances an NFA-style live set over the ATN ([RuntimeATN.Simulate])
+//     through the remaining tokens. Unlike prediction, the simulation keeps
+//     every alternative alive: the user has not committed to a branch yet.
+//  3. [RuntimeATN.NextCompletionsFromSet] reports what may come next as a
+//     [CompletionInfo]: valid token types, plus [CompletionHint] entries that
+//     mark cross-reference positions so the provider can offer resolvable
+//     symbols instead of a bare identifier token.
+//
+// [LanguageCompletionAdapter] is the contract between the framework's
+// completion provider and these generated artifacts; the code generator emits
+// its implementation together with the parsers.
+package parser

server/definition_provider.go

@@ -25,6 +25,7 @@ type DefaultDefinitionProvider struct {
 	sc *service.Container
 }
 
+// NewDefaultDefinitionProvider creates a new default definition provider.
 func NewDefaultDefinitionProvider(sc *service.Container) DefinitionProvider {
 	return &DefaultDefinitionProvider{sc: sc}
 }

server/diagnostics_publisher.go

@@ -16,12 +16,14 @@ import (
 )
 
 // DiagnosticsPublisher is responsible for publishing diagnostics to the LSP client.
-// The default implementation publishes diagnostics in the following scenarios:
-// Document is opened (reuses diagnostics from build).
-// Document is closed (clears diagnostics).
-// Document is validated by the builder.
+// It publishes diagnostics in the following scenarios:
 //
-// When registered to a service container, it automatically hooks into the document lifecycle events.
+//   - A document is opened (reuses diagnostics from build).
+//   - A document is closed (clears diagnostics).
+//   - A document is validated by the builder.
+//
+// When registered to a service container, it automatically hooks into the
+// document lifecycle events during server initialization.
 type DiagnosticsPublisher struct {
 	sc *service.Container
 }
@@ -31,6 +33,8 @@ func NewDiagnosticsPublisher(sc *service.Container) *DiagnosticsPublisher {
 	return &DiagnosticsPublisher{sc: sc}
 }
 
+// OnServerInitialize implements [InitializeParticipant] by subscribing to the
+// document lifecycle events that trigger publishing diagnostics.
 func (d *DiagnosticsPublisher) OnServerInitialize(_ *lsp.ParamInitialize) {
 	store, err := service.Get[textdoc.Store](d.sc)
 	if err != nil {

server/doc.go

@@ -0,0 +1,84 @@
+// Copyright 2026 TypeFox GmbH
+// This program and the accompanying materials are made available under the
+// terms of the MIT License, which is available in the project root.
+
+// Package server implements the Language Server Protocol (LSP) side of
+// Fastbelt. It exposes the language services built by
+// [typefox.dev/fastbelt/workspace] — parsed, linked, and validated documents —
+// to editors and IDEs: text documents are synchronized through LSP
+// notifications, and language features such as completion, go to definition,
+// find references, hover, and rename are answered from the workspace.
+//
+// # The language server and its method services
+//
+// [DefaultLanguageServer] implements the [typefox.dev/lsp.Server] interface,
+// but contains no feature logic of its own. Each supported LSP method is
+// delegated to a dedicated service interface (for example
+// [CompletionProvider] or [DocumentSyncher]) that is looked up in the
+// [typefox.dev/fastbelt/util/service.Container] when the request arrives.
+// Request-handling methods first acquire the
+// [typefox.dev/fastbelt/workspace.Lock] for reading, so a request never
+// observes a half-built workspace. The capabilities announced to the client
+// in the initialize response are derived from which services are registered,
+// so the set of advertised features always matches the set of available
+// services.
+//
+// [SetupDefaultServices] registers the default implementation of every LSP
+// method service that has not been registered yet. To customize a feature,
+// register your own implementation of the corresponding interface. Services
+// that need to inspect the client's initialize request implement
+// [InitializeParticipant].
+//
+// # Shared name and reference lookup
+//
+// Most position-based LSP features start by answering the same question:
+// which named symbol does the cursor point at? Two services encapsulate this
+// so that all features resolve names consistently:
+//
+//   - [NameFinder] turns the token(s) at a cursor position into a
+//     [FoundName], pairing the unit under the cursor with the unit that
+//     holds the name of the referenced symbol — following cross-references
+//     to their target, or staying in place when the cursor is already on a
+//     declaration's name.
+//   - [ReferencesFinder] yields every reference to a given AST node, drawing
+//     on the reference descriptions indexed per document. [FindReferencesOptions]
+//     controls whether the search is restricted to a single document and
+//     whether the declaration itself is included.
+//
+// The default definition, references, document highlight, hover, and rename
+// providers are thin compositions of these two services plus conversion to
+// LSP result types. Registering a custom [NameFinder] or [ReferencesFinder]
+// therefore adjusts all of these features at once.
+//
+// # Starting a server
+//
+// [StartLanguageServer] is the main entry point. It expects a fully
+// configured service container: the language-specific and workspace
+// services, the server services from [SetupDefaultServices], and a transport
+// registered with [SetupStdioServices] (communication over stdin/stdout) or
+// SetupWasmServices (communication with a JavaScript host, available in
+// js/wasm builds). It dials a JSON-RPC connection using the registered
+// [golang.org/x/exp/jsonrpc2.Dialer] and [golang.org/x/exp/jsonrpc2.Binder]
+// — by default [DefaultBinder], which attaches the [typefox.dev/lsp.Server]
+// from the container as the message handler — and blocks until the
+// connection is closed, which the default server triggers on the LSP exit
+// notification.
+//
+// A typical main function for a generated language looks like this:
+//
+//	func main() {
+//		sc := service.NewContainer()
+//		mylang.SetupServices(sc) // language, textdoc, linking, workspace services
+//		mylang.SetupGeneratedServerServices(sc)
+//		server.SetupDefaultServices(sc)
+//		server.SetupStdioServices(sc)
+//		sc.Seal()
+//		if err := server.StartLanguageServer(context.Background(), sc); err != nil {
+//			log.Fatal(err)
+//		}
+//	}
+//
+// While the server runs, [DiagnosticsPublisher] pushes parser, linker, and
+// validation diagnostics to the client whenever an open document is rebuilt,
+// and [SlogHandler] forwards log output to the client as LSP log messages.
+package server

server/document_highlight_provider.go

@@ -13,14 +13,19 @@ import (
 	"typefox.dev/lsp"
 )
 
+// DocumentHighlightProvider is a service for handling LSP document highlight requests.
 type DocumentHighlightProvider interface {
 	HandleDocumentHighlightRequest(ctx context.Context, params *lsp.DocumentHighlightParams) ([]lsp.DocumentHighlight, error)
 }
 
+// DefaultDocumentHighlightProvider is the default implementation of [DocumentHighlightProvider].
+// It highlights all references to the symbol at the cursor position that are
+// located in the same document, using [NameFinder] and [ReferencesFinder].
 type DefaultDocumentHighlightProvider struct {
 	sc *service.Container
 }
 
+// NewDefaultDocumentHighlightProvider creates a new default document highlight provider.
 func NewDefaultDocumentHighlightProvider(sc *service.Container) DocumentHighlightProvider {
 	return &DefaultDocumentHighlightProvider{sc: sc}
 }

server/document_sync.go

@@ -72,6 +72,7 @@ type DefaultDocumentSyncher struct {
 	mu                        sync.RWMutex
 }
 
+// NewDefaultDocumentSyncher creates a new default document syncher.
 func NewDefaultDocumentSyncher(sc *service.Container) DocumentSyncher {
 	return &DefaultDocumentSyncher{sc: sc}
 }

server/folding_range_provider.go

@@ -60,18 +60,25 @@ func (f *DefaultFoldingRangeFilter) IncludeLastFoldingLineForComment() bool {
 	return true
 }
 
+// DefaultFoldingRangeProvider is the default implementation of [FoldingRangeProvider].
+// It derives folding ranges from multi-line AST nodes and comments, with the
+// details controlled by a [FoldingRangeFilter].
 type DefaultFoldingRangeProvider struct {
 	sc     *service.Container
 	filter FoldingRangeFilter
 }
 
+// NewDefaultFoldingRangeProvider creates a folding range provider with the
+// [DefaultFoldingRangeFilter].
 func NewDefaultFoldingRangeProvider(sc *service.Container) FoldingRangeProvider {
 	return &DefaultFoldingRangeProvider{
 		sc:     sc,
 		filter: &DefaultFoldingRangeFilter{},
 	}
 }
 
+// NewFoldingRangeProviderWithFilter creates a folding range provider with a
+// custom [FoldingRangeFilter].
 func NewFoldingRangeProviderWithFilter(sc *service.Container, filter FoldingRangeFilter) FoldingRangeProvider {
 	return &DefaultFoldingRangeProvider{
 		sc:     sc,

server/hover_provider.go

@@ -23,6 +23,7 @@ type DefaultHoverProvider struct {
 	sc *service.Container
 }
 
+// NewDefaultHoverProvider creates a new default hover provider.
 func NewDefaultHoverProvider(sc *service.Container) HoverProvider {
 	return &DefaultHoverProvider{sc: sc}
 }

server/name_finder.go

@@ -12,6 +12,10 @@ import (
 	"typefox.dev/fastbelt/util/service"
 )
 
+// FoundName is the result of a [NameFinder.Find] call. Either field may be
+// nil if the search could not determine the corresponding unit. Source and
+// Target are the same unit when the search started on a symbol's declaration
+// name rather than on a reference to it.
 type FoundName struct {
 	// The unit that contains the token that was used to start the search.
 	Source core.StringUnit
@@ -31,10 +35,16 @@ type NameFinder interface {
 	Find(ctx context.Context, first, second *core.Token) FoundName
 }
 
+// DefaultNameFinder is the default implementation of [NameFinder].
+// If a given token belongs to a cross-reference, it resolves the reference
+// and returns the name unit of the target symbol; if the token lies within
+// the name of a declaration, it returns that name unit as both source and
+// target.
 type DefaultNameFinder struct {
 	sc *service.Container
 }
 
+// NewDefaultNameFinder creates a new default name finder.
 func NewDefaultNameFinder(sc *service.Container) NameFinder {
 	return &DefaultNameFinder{sc: sc}
 }

server/references_finder.go

@@ -15,6 +15,7 @@ import (
 	"typefox.dev/fastbelt/workspace"
 )
 
+// FindReferencesOptions controls the scope of a [ReferencesFinder.Find] search.
 type FindReferencesOptions struct {
 	// Whether to include the declaration of the symbol as a dedicated [core.ReferenceDescription] in the results.
 	// If true, will contain the declaration as a reference with the same source and target node, and the range of the name as the first element.
@@ -25,14 +26,26 @@ type FindReferencesOptions struct {
 	TargetURI core.URI
 }
 
+// ReferencesFinder finds all references to a given symbol. It is shared by
+// the LSP services that need the full set of references to a symbol, such as
+// the default references, document highlight, and rename providers.
+// Adopters should customize this service if they want to change how
+// references are collected; downstream LSP services will automatically use
+// the new implementation.
 type ReferencesFinder interface {
+	// Find returns the references pointing at the target node,
+	// with the search scope controlled by options.
 	Find(ctx context.Context, target core.AstNode, options FindReferencesOptions) iter.Seq[*core.ReferenceDescription]
 }
 
+// DefaultReferencesFinder is the default implementation of [ReferencesFinder].
+// It draws on the reference descriptions indexed per document, iterating all
+// documents of the workspace unless a target URI is given.
 type DefaultReferencesFinder struct {
 	sc *service.Container
 }
 
+// NewDefaultReferencesFinder creates a new default references finder.
 func NewDefaultReferencesFinder(sc *service.Container) ReferencesFinder {
 	return &DefaultReferencesFinder{sc: sc}
 }

server/references_provider.go

@@ -23,6 +23,7 @@ type DefaultReferencesProvider struct {
 	sc *service.Container
 }
 
+// NewDefaultReferencesProvider creates a new default references provider.
 func NewDefaultReferencesProvider(sc *service.Container) ReferencesProvider {
 	return &DefaultReferencesProvider{sc: sc}
 }