diff --git a/AGENTS.md b/AGENTS.md index ecb1d56..2273391 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,10 +19,25 @@ pakt/ │ ├── unmarshal.go # PAKT text → Go struct │ ├── tags.go # Struct tag parsing (pakt:"name") │ └── *_test.go # Tests for each component +├── dotnet/ # .NET library (net10.0) +│ ├── src/Pakt/ # Core library +│ │ ├── PaktReader.cs # Tier 0: ref struct token reader (state machine) +│ │ ├── PaktMemoryReader.cs # Tier 1: sync statement reader (memory-backed) +│ │ ├── PaktStreamReader.cs # Tier 1: async statement reader (stream-backed) +│ │ ├── PaktFramedSource.cs # Internal: NUL-aware async buffer for stream reader +│ │ ├── PaktSerializer.cs # Tier 2: convenience Deserialize/DeserializeAsync/Serialize +│ │ ├── PaktUnitMaterializer.cs # Whole-unit binding (sync + async) +│ │ ├── PaktReaderExtensions.cs # Callback-based composite navigation helpers +│ │ ├── PaktWriter.cs # Forward-only PAKT output writer +│ │ └── Serialization/ # Runtime: converters, options, type info, deserialization +│ ├── src/Pakt.Generators/ # Source generator (netstandard2.0) +│ ├── tests/ # xUnit tests +│ └── benchmarks/ # BenchmarkDotNet suites (FS, Fin, Small, Wide, Deep, Collections) ├── main.go # CLI entry point (Kong) ├── cli.go # CLI commands: parse, validate, version ├── cli_test.go # CLI integration tests (build binary, run against testdata) ├── spec/pakt-v0.md # Formal PAKT v0 specification +├── spec/benchmarks-v0.md # Cross-platform benchmark specification ├── docs/guide.md # Human-friendly PAKT guide ├── design/ # Architecture documents │ └── state-machine-rewrite.md # Decoder state machine design @@ -69,6 +84,32 @@ The decoder in `encoding/` uses an explicit-stack state machine rather than recu - `reader_type.go` — Type annotation parser (recursive descent, bounded depth — stays separate from state machine) - `design/state-machine-rewrite.md` — Full design doc with state transition narrative, frame payloads, risky areas, and observable behavior contract +## Architecture: .NET Two-Reader Model + +The .NET library uses a layered architecture with two distinct Tier 1 readers: + +### Tier 0: `PaktReader` (ref struct) + +Stack-only, zero-copy tokenizer over `ReadOnlySpan`. Same state-machine design as the Go decoder but adapted for .NET's ref struct constraints. Source-generated deserializers operate at this level for maximum performance. + +### Tier 1: Two readers + +- **`PaktMemoryReader`** — Sync-only, `IDisposable`. For `ReadOnlyMemory` or `IMemoryOwner` input. No artificial async. The memory-backed fast path. +- **`PaktStreamReader`** — Async-only, `IAsyncDisposable`. Real `Stream.ReadAsync` at I/O refill boundaries. Uses `PaktFramedSource` internally for NUL-delimited unit framing with correct leftover handling. No sync wrappers. + +The two readers share no interface or base class. This is intentional — async exists only where the underlying code path is genuinely async. + +### Tier 2: `PaktSerializer` + +Static convenience API: `Deserialize(ReadOnlyMemory)` (sync, uses `PaktMemoryReader`), `DeserializeAsync(Stream)` (async, uses `PaktStreamReader`), `Serialize`. Sugar over Tier 1. + +### Key design rules + +- No fake async adapters. Async only on `PaktStreamReader` where `Stream.ReadAsync` is real. +- `IMemoryOwner` is the canonical ownership transfer mechanism. +- Source-generated code targets `PaktReader` directly for zero-alloc scalar reads. +- `PaktConvertContext` is a `readonly ref struct` — no heap allocation for converter context. + ## Spec Compliance The specification (`spec/pakt-v0.md`) is the authoritative source for PAKT semantics. When working on the parser or event model: diff --git a/dotnet/README.md b/dotnet/README.md index 829629b..79c567b 100644 --- a/dotnet/README.md +++ b/dotnet/README.md @@ -77,11 +77,12 @@ byte[] bytes = PaktSerializer.Serialize(server, AppPaktContext.Default); ### Layered Architecture ``` - PaktSerializer High-level convenience (whole-unit materialization / serialization) + PaktSerializer High-level convenience (Deserialize / DeserializeAsync / Serialize) ↓ - PaktMemoryReader Statement-level iteration (assigns, packs) + PaktMemoryReader Sync statement-level iteration (memory-backed) + PaktStreamReader Async statement-level iteration (stream-backed) ↓ - PaktReader / PaktWriter Low-level token-by-token I/O + PaktReader / PaktWriter Low-level token-by-token I/O ↓ Source Generator Compile-time (de)serialization code via [PaktSerializable] ``` @@ -92,8 +93,9 @@ byte[] bytes = PaktSerializer.Serialize(server, AppPaktContext.Default); |---|---| | `PaktReader` | Forward-only, zero-copy tokenizer over `ReadOnlySpan`. Ref struct — stack-only, pooled allocations. | | `PaktWriter` | Forward-only PAKT output writer to `IBufferWriter`. | -| `PaktMemoryReader` | Statement-level reader. Iterates top-level assigns and packs. Supports `ReadValue()`, `ReadPack()`, and `ReadMapPack()`. | -| `PaktSerializer` | Static convenience API for whole-unit deserialize/serialize over generated metadata. | +| `PaktMemoryReader` | Sync statement-level reader for `ReadOnlyMemory` / `IMemoryOwner`. `ReadValue()`, `ReadPack()`, `ReadMapPack()`. | +| `PaktStreamReader` | Async statement-level reader for `Stream`. Real `Stream.ReadAsync` at I/O boundaries. `ReadValueAsync()`, `ReadPackAsync()` (`IAsyncEnumerable`). | +| `PaktSerializer` | Static convenience API. `Deserialize` (sync memory), `DeserializeAsync` (async stream), `Serialize`. | | `PaktSerializerContext` | Base class for source-generated serialization contexts. Provides `GetTypeInfo()` for type resolution. | | `PaktType` | Immutable PAKT type descriptor (scalars, structs, tuples, lists, maps, atom sets). | | `PaktException` | Parse/validation error with `PaktPosition` and `PaktErrorCode`. | @@ -178,9 +180,10 @@ Consumer projects must reference the generator as an analyzer — not a regular - ✅ Streaming tokenizer (`PaktReader`) — all PAKT scalar and composite types - ✅ Forward-only writer (`PaktWriter`) -- ✅ Statement-level reader (`PaktMemoryReader`) — assigns and packs +- ✅ Statement-level reader (`PaktMemoryReader`) — sync, memory-backed +- ✅ Async stream reader (`PaktStreamReader`) — real `Stream.ReadAsync` at I/O boundaries - ✅ Source-generated (de)serialization — structs, lists, maps, nullable types, nested types -- ✅ Convenience API (`PaktSerializer`) — whole-unit deserialize/serialize +- ✅ Convenience API (`PaktSerializer`) — `Deserialize` (sync) + `DeserializeAsync` (stream) **Deferred:** @@ -188,4 +191,3 @@ Consumer projects must reference the generator as an analyzer — not a regular - Source-generated tuple serialization/deserialization coverage - Spec projection (`.spec.pakt` filtering) - NuGet packaging -- A dedicated stream-native `PaktStreamReader` with real async I/O (planned) diff --git a/site/content/docs/examples.md b/site/content/docs/examples.md index 26207eb..5df60e4 100644 --- a/site/content/docs/examples.md +++ b/site/content/docs/examples.md @@ -292,9 +292,9 @@ rollback-version:(int, int, int)? = nil # not set # Service metadata map meta: = < - 'owner' = 'platform-team' + 'owner' ; 'platform-team' 'region' ; 'us-east-1' - 'tier' = 'critical' + 'tier' ; 'critical' > # Health check configuration diff --git a/site/content/docs/install.md b/site/content/docs/install.md index 5f5b156..14b7b9e 100644 --- a/site/content/docs/install.md +++ b/site/content/docs/install.md @@ -186,6 +186,26 @@ var server = PaktSerializer.Deserialize(paktBytes, AppPaktContext.Defaul byte[] bytes = PaktSerializer.Serialize(server, AppPaktContext.Default); ``` +### Read from a stream (async) + +```csharp +await using var reader = PaktStreamReader.Create(networkStream, AppPaktContext.Default); + +while (await reader.ReadStatementAsync(ct)) +{ + if (reader.IsPack && reader.StatementType.IsList) + { + await foreach (var s in reader.ReadPackAsync(ct)) + Console.WriteLine($"{s.Host}:{s.Port}"); + } + else + { + var s = await reader.ReadValueAsync(ct); + Console.WriteLine($"{s.Host}:{s.Port}"); + } +} +``` + ### Requirements - .NET 10