Skip to content

Compiled generator/async-generator fixes: #547, #631, #542, #583#702

Merged
nickna merged 4 commits into
mainfrom
wrk/xenodochial-kilby-d696a2
Jun 16, 2026
Merged

Compiled generator/async-generator fixes: #547, #631, #542, #583#702
nickna merged 4 commits into
mainfrom
wrk/xenodochial-kilby-d696a2

Conversation

@nickna

@nickna nickna commented Jun 16, 2026

Copy link
Copy Markdown
Owner

Fixes four compiled-mode generator / async-generator issues. All are compiled-only — the interpreter uses an independent model and is unaffected. Each issue is a separate commit.

#547 — for-in with yield stops after the first key

The generator MoveNext emitter hoisted the for-of enumerator across yields but kept the for-in key list + index in IL locals, which a MoveNext re-entry wipes. Mirror the for-of treatment: GeneratorStateAnalyzer records for-in loops whose body yields, HoistingManager allocates a key-list field + index field per such loop, and GeneratorMoveNextEmitter.EmitForIn reads/writes both from those fields.

#631 + #542 — truly-async, serialized async-generator next()

The compiled async generator drove its body synchronously (MoveNextAsync().AsTask().GetResult()), so a not-yet-settled await deadlocked the event-loop thread (#631), and concurrent/re-entrant next() recursed into MoveNextAsync (only kept from overflowing by a reject-on-busy guard, never queued — #542).

  • next() no longer blocks: it drives one step and returns the Task<object> built by an emitted AsyncGeneratorBuildResult helper.
  • Requests serialize on a per-generator _pendingTail task chain (ECMA-262 §27.6.3 AsyncGeneratorQueue modeled as a Task chain) — overlapping next() calls run FIFO. A synchronously re-entrant next() still rejects with a TypeError to avoid a stack overflow (the self-reentrant case would deadlock under a real queue anyway).
  • for await…of now suspends on the iterator protocol via the async state machine instead of blocking (AsyncStateAnalyzer reserves a suspension state for each implicit next()/return() await; a try enclosing a for await takes the flag-based path so its resume labels aren't branched into).
  • AsyncGeneratorAwaitContinue no longer short-circuits on the awaited task's fault, so a pending rejection reaches the body's own resume point and its try/catch (also unblocks the pending sub-case of Async generator: a rejected await inside a try is not caught by that try's catch (both modes) #617).

#583 (part 1) — lift capturing nested generator/async functions

Two layers:

  1. Calling a capturing arrow inside a generator threw "Non-static method requires a target" (the sync generator emitter bound a null $TSFunction target and never built the display instance). GeneratorMoveNextEmitter now has the capturing-arrow handling the async-generator emitter already has.
  2. NestedFunctionLifter now lambda-lifts inside-function captures (was module-block only): the captured function-scope bindings become leading parameters of the relocated top-level declaration, forwarded by an in-place arrow. Self-recursive declarations decline cleanly (a compiled arrow snapshots captures by value, and the forwarding arrow's own let is in TDZ at creation). Compiled: capturing nested generator/async/state-machine-nested functions still not lowered (lifting limitation) #583 §2 (relocated identity shared across enclosing-call instances) remains.

Tests

  • New shared (interp+compiled) tests for each fix in GeneratorTests, AsyncGeneratorTests, GeneratorClosureCaptureTests; the emitter-sync guardrail (EmitterSyncTests) updated for the two new GeneratorMoveNextEmitter overrides.
  • Full SharpTS.Tests suite green (12702 passing, 0 failing). Standalone-DLL tests green (the new emitted helpers use only BCL types).

Follow-up gaps filed while here

@nickna nickna force-pushed the wrk/xenodochial-kilby-d696a2 branch from 6ad3a93 to e2e523f Compare June 16, 2026 03:08
@nickna

nickna commented Jun 16, 2026

Copy link
Copy Markdown
Owner Author

Rebased onto current main (now MERGEABLE). Two integration notes after the rebase:

Full SharpTS.Tests suite re-run on the rebased state is in progress; the targeted async/generator/closure + emitter-sync suites (264 tests) and all manual repros are green.

nickna added 4 commits June 15, 2026 20:17
The generator MoveNext emitter hoists the for-of enumerator across yields
but kept the for-in key list and current index in IL locals, which a
MoveNext re-entry wipes — so a yielding for-in restarted from (or stopped
after) the first key. Mirror the for-of treatment: GeneratorStateAnalyzer
records for-in loops whose body yields (ForInLoopsWithYield), Hoisting-
Manager allocates a key-list field + index field per such loop, and
GeneratorMoveNextEmitter.EmitForIn reads/writes both from those fields.
Interpreter was already correct; compiled-mode only.
Compiled async generators drove the body synchronously (next() did
MoveNextAsync().AsTask().GetResult(); for await...of likewise blocked on
GetResult), so a not-yet-settled await deadlocked the event-loop thread
(#631) and concurrent/re-entrant next() recursed into MoveNextAsync,
only kept from overflowing by a reject-on-busy guard, never queued (#542).

- next() is now truly asynchronous: it drives one step and returns the
  Task<object> built by an emitted AsyncGeneratorBuildResult helper.
- Requests serialize on a per-generator _pendingTail task chain (the
  ECMA-262 27.6.3 AsyncGeneratorQueue modeled as a Task chain) so
  overlapping next() calls run FIFO. A synchronously re-entrant next()
  still rejects with a TypeError (the self-reentrant case would deadlock
  under a real queue anyway).
- for await...of suspends on the iterator protocol via the async state
  machine instead of blocking. The for-await lowering is shared in
  StatementEmitterBase (main's #430/#645); AsyncMoveNextEmitter overrides
  EmitForAwaitOf to SUSPEND on each next()/return() await (the shared base
  still blocks for async arrows / async generators -- async-gen tracked by
  #697). AsyncStateAnalyzer reserves a suspension state per implicit await;
  a try enclosing a for-await takes the flag-based path (ContainsAwait
  treats for-await as containing awaits) so resume labels aren't branched
  into.
- AsyncGeneratorAwaitContinue no longer short-circuits on the awaited
  task's fault; it resumes MoveNextAsync so a pending rejection reaches
  the body's resume point and its try/catch (unblocks the pending sub-case
  of #617).

Interpreter async generators use a separate eager-drain model and are
unaffected; compiled-mode only.
NestedFunctionLifter refused to relocate any declaration capturing an
enclosing function scope, so a nested generator/async/plain function
reading an outer local failed with "Yield not supported in this context".
Extend the existing lambda-lift (previously module-block only) to inside-
function captures: the captured function-scope bindings become leading
parameters of the relocated top-level declaration, forwarded by an
in-place arrow that closes over them. Relies on capturing arrows in
generator/state-machine bodies binding their display instance (landed on
main as #674) so the arrow reads its captures live.

Self-recursive nested declarations decline cleanly (left nested): a
compiled arrow snapshots captures by value and its own let binding is in
its TDZ at creation, so a forwarded self would be null. #583 part 2
(relocated identity shared across enclosing-call instances) remains.
…c allowlist

GeneratorMoveNextEmitter.EmitForIn (#547) and AsyncMoveNextEmitter.Emit-
ForAwaitOf (#631, override of the shared base to suspend) are new
overrides; add them to EmitterSyncTests.StateMachineEmitters_OnlyOverride-
AllowedMethods.
@nickna nickna force-pushed the wrk/xenodochial-kilby-d696a2 branch from e2e523f to 2d3b358 Compare June 16, 2026 03:18
@nickna nickna merged commit 5ec5cf3 into main Jun 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant