Compiled generator/async-generator fixes: #547, #631, #542, #583#702
Merged
Conversation
6ad3a93 to
e2e523f
Compare
Owner
Author
The generator MoveNext emitter hoists the for-of enumerator across yields but kept the for-in key list and current index in IL locals, which a MoveNext re-entry wipes — so a yielding for-in restarted from (or stopped after) the first key. Mirror the for-of treatment: GeneratorStateAnalyzer records for-in loops whose body yields (ForInLoopsWithYield), Hoisting- Manager allocates a key-list field + index field per such loop, and GeneratorMoveNextEmitter.EmitForIn reads/writes both from those fields. Interpreter was already correct; compiled-mode only.
Compiled async generators drove the body synchronously (next() did MoveNextAsync().AsTask().GetResult(); for await...of likewise blocked on GetResult), so a not-yet-settled await deadlocked the event-loop thread (#631) and concurrent/re-entrant next() recursed into MoveNextAsync, only kept from overflowing by a reject-on-busy guard, never queued (#542). - next() is now truly asynchronous: it drives one step and returns the Task<object> built by an emitted AsyncGeneratorBuildResult helper. - Requests serialize on a per-generator _pendingTail task chain (the ECMA-262 27.6.3 AsyncGeneratorQueue modeled as a Task chain) so overlapping next() calls run FIFO. A synchronously re-entrant next() still rejects with a TypeError (the self-reentrant case would deadlock under a real queue anyway). - for await...of suspends on the iterator protocol via the async state machine instead of blocking. The for-await lowering is shared in StatementEmitterBase (main's #430/#645); AsyncMoveNextEmitter overrides EmitForAwaitOf to SUSPEND on each next()/return() await (the shared base still blocks for async arrows / async generators -- async-gen tracked by #697). AsyncStateAnalyzer reserves a suspension state per implicit await; a try enclosing a for-await takes the flag-based path (ContainsAwait treats for-await as containing awaits) so resume labels aren't branched into. - AsyncGeneratorAwaitContinue no longer short-circuits on the awaited task's fault; it resumes MoveNextAsync so a pending rejection reaches the body's resume point and its try/catch (unblocks the pending sub-case of #617). Interpreter async generators use a separate eager-drain model and are unaffected; compiled-mode only.
NestedFunctionLifter refused to relocate any declaration capturing an enclosing function scope, so a nested generator/async/plain function reading an outer local failed with "Yield not supported in this context". Extend the existing lambda-lift (previously module-block only) to inside- function captures: the captured function-scope bindings become leading parameters of the relocated top-level declaration, forwarded by an in-place arrow that closes over them. Relies on capturing arrows in generator/state-machine bodies binding their display instance (landed on main as #674) so the arrow reads its captures live. Self-recursive nested declarations decline cleanly (left nested): a compiled arrow snapshots captures by value and its own let binding is in its TDZ at creation, so a forwarded self would be null. #583 part 2 (relocated identity shared across enclosing-call instances) remains.
e2e523f to
2d3b358
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes four compiled-mode generator / async-generator issues. All are compiled-only — the interpreter uses an independent model and is unaffected. Each issue is a separate commit.
#547 — for-in with
yieldstops after the first keyThe generator MoveNext emitter hoisted the for-of enumerator across yields but kept the for-in key list + index in IL locals, which a
MoveNextre-entry wipes. Mirror the for-of treatment:GeneratorStateAnalyzerrecords for-in loops whose body yields,HoistingManagerallocates a key-list field + index field per such loop, andGeneratorMoveNextEmitter.EmitForInreads/writes both from those fields.#631 + #542 — truly-async, serialized async-generator
next()The compiled async generator drove its body synchronously (
MoveNextAsync().AsTask().GetResult()), so a not-yet-settled await deadlocked the event-loop thread (#631), and concurrent/re-entrantnext()recursed intoMoveNextAsync(only kept from overflowing by a reject-on-busy guard, never queued — #542).next()no longer blocks: it drives one step and returns theTask<object>built by an emittedAsyncGeneratorBuildResulthelper._pendingTailtask chain (ECMA-262 §27.6.3 AsyncGeneratorQueue modeled as a Task chain) — overlappingnext()calls run FIFO. A synchronously re-entrantnext()still rejects with a TypeError to avoid a stack overflow (the self-reentrant case would deadlock under a real queue anyway).for await…ofnow suspends on the iterator protocol via the async state machine instead of blocking (AsyncStateAnalyzerreserves a suspension state for each implicitnext()/return()await; atryenclosing afor awaittakes the flag-based path so its resume labels aren't branched into).AsyncGeneratorAwaitContinueno longer short-circuits on the awaited task's fault, so a pending rejection reaches the body's own resume point and itstry/catch(also unblocks the pending sub-case of Async generator: a rejected await inside a try is not caught by that try's catch (both modes) #617).#583 (part 1) — lift capturing nested generator/async functions
Two layers:
$TSFunctiontarget and never built the display instance).GeneratorMoveNextEmitternow has the capturing-arrow handling the async-generator emitter already has.NestedFunctionLifternow lambda-lifts inside-function captures (was module-block only): the captured function-scope bindings become leading parameters of the relocated top-level declaration, forwarded by an in-place arrow. Self-recursive declarations decline cleanly (a compiled arrow snapshots captures by value, and the forwarding arrow's ownletis in TDZ at creation). Compiled: capturing nested generator/async/state-machine-nested functions still not lowered (lifting limitation) #583 §2 (relocated identity shared across enclosing-call instances) remains.Tests
GeneratorTests,AsyncGeneratorTests,GeneratorClosureCaptureTests; the emitter-sync guardrail (EmitterSyncTests) updated for the two newGeneratorMoveNextEmitteroverrides.SharpTS.Testssuite green (12702 passing, 0 failing). Standalone-DLL tests green (the new emitted helpers use only BCL types).Follow-up gaps filed while here
yield*to a genuinely-async iterator still blocks (same synchronous-drive root, async yield* loop).for awaitbody; concurrentnext()).for awaitbody that also awaits/breaks, inside atry, escapes the catch (narrow tradeoff of the Compiled async generator hangs on a not-yet-settled (genuinely-async) await #631 for-await fix).for await…ofinside an async generator still blocks on a genuinely-async source (AsyncGeneratorMoveNextEmitter.EmitForAwaitOf— same root cause as Compiled async generator hangs on a not-yet-settled (genuinely-async) await #631, separate emitter; left as follow-up to keep this rework scoped to Compiled async generator hangs on a not-yet-settled (genuinely-async) await #631's filed repro).