Add async generator support for dynamic deps in build system#84
Open
Add async generator support for dynamic deps in build system#84
Conversation
- Support async functions with @task decorator (run_aio, call_aio) - Add LoadableTask.load_aio with load/load_aio mutual fallback (mirrors run/run_aio) - Add Task.load_aio override using target().load_aio() - Add _open_aio to InMemoryFileSystemTarget for complete async interface - Add _FuncReturnT TypeVar for decorator overload clarity - Guard call() and call_aio() against sync/async mismatch Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Build sequential and concurrent executors now handle async generators from run_aio() (detect via isasyncgenfunction, iterate with async for) - @task decorator rejects generator functions with clear error message pointing to class-based API for dynamic dependencies - Modal _run supports async-only tasks via asyncio.run(run_aio()) - Add AsyncDynamicDiamondTask helper and tests for both build paths Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
andhus
force-pushed
the
async-dynamic-deps-build
branch
from
34874f4 to
5facc34
Compare
- Extract _iter_dynamic_deps() async helper to deduplicate sync/async generator handling in _run_task_sequential_aio - Rewrite Modal _run to handle dynamic deps via idempotent re-execution (mirrors _run_task_in_process): drive generator forward while deps are complete, return TaskStruct of incomplete deps for ModalTaskExecutor Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
run_aio()for dynamic dependencies (class-based Task API)@taskdecorator rejects generator functions with a clear error message pointing users to the class-based API_runfunction now supports async-only tasks viaasyncio.run(run_aio())AsyncDynamicDiamondTasktest helper and async dynamic deps tests for both sequential and concurrent build pathsContext
The build system previously only handled sync generators (
__next__) for dynamic dependencies. When a class-based task'srun_aio()is an async generator (yieldingTaskStructdeps), the build system would fail becauseawaitdoesn't work on async generators. This PR addsinspect.isasyncgenfunction()detection andasync foriteration alongside the existing sync generator handling.The
@taskdecorator API is intentionally kept simple — dynamic dependencies are complex and better served by the class-based API where type annotations and generator patterns are more natural.Changes
build/_sequential.pyrun_aio, iterate withasync forbuild/_concurrent.py_handle_result(now async),_handle_generator_aio,_resume_generator_aio, updated typing_core/decorator.pyTypeErrorintegration/modal/_app.py_rundetects async-only tasks, usesasyncio.run(run_aio())utils/testing/helper_tasks.pyAsyncDynamicDiamondTaskclasstests/test__core/test_decorator.pytests/test_build/test_sequential.pytests/test_build/test_concurrent.pyTest plan
test_decorator.py,test_sequential.py,test_concurrent.py)🤖 Generated with Claude Code