Fixture-based test harness with 331 suites and type system gap tracking#26
Merged
Fixture-based test harness with 331 suites and type system gap tracking#26
Conversation
Canonical path for multi-file type system testing, execution verification, and benchmarking. Fixtures are directories under testdata/fixtures/ with .lua files and optional manifest.json. Inline expect-error/expect-warning annotations in source files for diagnostic assertions. Includes mini-require runtime for multi-module fixture execution, golden file comparison with FIXTURE_UPDATE=1 support, and recursive discovery with go test -run filtering by category path.
Migrate narrowing_test.go (18 cases) and minimal_narrowing_test.go (3 cases) to testdata/fixtures/narrowing/. Migrate typedef_test.go (22 cases) to testdata/fixtures/types/. Remove original Go test files. Also fixes BenchFixtures -> BenchmarkFixtures naming and replaces PreloadModule approach with installRequire mini-runtime for proper require() support in multi-module fixtures.
Migrate generics_test.go (19 cases), functions_test.go (26 cases), control_flow_test.go (17 cases), and wippy_suite_test.go easy cases (21 cases) to fixture harness. Remove original Go files. Fix inline expect-error to absorb all diagnostics on annotated line, not just the first match. Keeps manifest-based regression tests (TestModuleGenericInstantiation, TestRegistryTypeLoss) in place.
Migrate scenarios_test.go (65 cases) to testdata/fixtures/core/, type_cast_test.go (39 cases) to testdata/fixtures/types/cast-*, and union_narrowing_test.go (12 cases) to testdata/fixtures/narrowing/union-*. Remove original Go test files. Fix harness: expect-error annotation now absorbs all diagnostics on that line, not just the first match (handles multi-error lines).
Add "packages" field to manifest.json to declare system packages (channel, funcs) that the type checker resolves via predefined manifests. Create 5 real-world multi-file fixtures that exercise patterns found in production wippy code: - channel-select-pattern: event/timer select with narrowing (passes) - error-handling-chain: discriminated union result types (passes) - factory-constructor: OOP factory pattern (skipped - syntax gap) - module-with-generics: generic collection type (skipped - parser gap) - table-builder-pattern: fluent method chaining (skipped - return tracking) Skipped fixtures mark type system hardening targets derived from analyzing 687 :: any casts across 93 production Lua files.
Use check.errors count instead of skip for real-world fixtures with known type system limitations. This ensures gaps are actively tracked and tests fail when fixes reduce the error count, preventing regression and making progress visible.
Migrate method_calls_test.go (39 cases) covering string methods, table patterns, pcall/xpcall, goto/break, higher-order functions, closures, and variadic functions to fixtures. Migrate assert_narrowing_test.go (26 cases) covering assert narrowing, custom assert libraries, error-return patterns, assert wrapper propagation, and inferred contracts. Remove optional_narrowing_test.go (duplicate of already-migrated cases).
Six new multi-file fixtures reproducing production patterns from wippy framework code that currently require :: any casts: - metatable-oop: EventEmitter + Counter with setmetatable, callbacks (5 errors) - fluent-prompt-builder: method chaining with typed return self (4 errors) - typed-callback-chain: StreamCallbacks with ToolCall/ErrorInfo payloads (29 errors) - result-type-narrowing: Generic Result<T> with map/and_then combinators (4 errors) - generic-registry: Handler registry with typed register/call/list (12 errors) - context-merge-pipeline: Pipeline with fluent add + Context merging (3 errors) These fixtures establish the hardening targets for the type system. Combined with the 3 existing gap fixtures, we now track 9 real-world patterns with a total of 68 known type errors to eliminate.
New fixtures reproducing production patterns:
- lookup-table-cast: typed map tables {[string]: string} (11 errors)
- service-locator: cross-module service resolution (14 errors)
- trait-registry: union type normalization (string | record) (0 errors - passes!)
- sql-repository: enum-keyed table access (4 errors)
- iterator-pipeline: generic map/filter/reduce (11 errors)
- typed-enum-constants: literal union types as enums (6 errors)
- multi-return-error-chain: chained (value, err) narrowing (0 errors - passes!)
- discriminated-tool-dispatch: tagged union dispatch (0 errors - passes!)
Fix error-handling-chain to export types as runtime values (M.AppError = AppError)
and use :is() for runtime validation. Cross-module :is() returns unknown - tracked
as expect-error, confirming this is a real type system gap.
Total: 331 fixtures, 117 tracked type errors across 14 gap fixtures.
Fix module iteration in check phase to use deterministic file order instead of map iteration. This resolves flaky test results caused by Go map randomization affecting checker fixpoint convergence. Update error-handling-chain to demonstrate idiomatic type export pattern (M.AppError = AppError) with runtime :is() validation. Cross-module :is() returning unknown is tracked as expect-error. Set error counts for multi-return-error-chain (was passing due to lucky map ordering, now deterministically tracked).
Contributor
Author
|
This seems like massive PR but it's a lot of mechanical work around harness and canonization. This is step 1 of new vm design port to wippy. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
testdata/fixtures/Harness features
fixture_harness_test.go(~400 lines) +fixture_test.go(~45 lines)-- expect-error/-- expect-warningannotations for line-precise assertionsmanifest.jsonwith file ordering, system packages, error countsFIXTURE_UPDATE=1regenerationgo test -run TestFixtures/category/namefilteringBenchmarkFixturesfor performance testing from the same dataFixture categories
Real-world stress tests
Derived from analyzing 687
:: anycasts across 93 production Lua files.Test plan
go test -v -run TestFixtures ./...— all 331 suites passgo test -bench BenchmarkFixtures ./...— benchmark suite worksgo test -count=10 -run TestFixtures .— deterministic (0 flakes in 10 runs)go test ./...— full suite passes, no regressions-- expect-error, 14 with error counts) correctly catch type errors