asyncio: Add guest mode for running asyncio inside external event loops

[congzhangzh]

Feature request

Add guest mode to asyncio, enabling it to run cooperatively inside a host event loop such as Tkinter, Qt, GTK, or any other framework that owns the main thread.

Motivation

GUI toolkits (Tk, Qt, GTK, pygame, ...) each have their own event loop that must run on the main thread. Today there is no standard way to use asyncio coroutines inside a GUI application without:

• Polling hacks (periodic root.after() calls)
• Replacing the event loop entirely (fragile, framework-specific)
• Running asyncio in a separate thread and manually marshalling results

Trio solved this problem with trio.lowlevel.start_guest_run() and the approach has proven robust across many GUI frameworks (see trio-guest).

Proposed API

import asyncio

task = asyncio.start_guest_run(
    async_fn, *args,
    run_sync_soon_threadsafe=host.schedule_callback,
    done_callback=host.on_done,
)

The implementation decomposes BaseEventLoop._run_once() into three new public methods — poll_events(), process_events(), and process_ready() — then uses a dual-thread architecture where the host thread processes callbacks and a daemon I/O thread polls for events.

Prior art and discussion

• Trio guest mode
• trio-guest — multi-framework reference implementation
• asyncio-guest — proof-of-concept for asyncio (this proposal is derived from it)
• discuss.python.org: Connecting asyncio and tkinter event loops

A reference implementation with tests (12 tests passing, full asyncio test suite regression-free) is ready to submit as a PR.

Linked PRs

• gh-145342: asyncio: Add guest mode for running inside external event loops #145343

[x42005e1f]

Note that there is already a guest mode inspired by Trio: oremanj/aioguest. There is also my gist, which works without greenlets (and which I am currently refactoring/improving/etc. to support Curio and release as a standalone package). If you want to implement a reliable guest mode, you should also consider related issues such as #85093 (and also handle the wakeup fd, otherwise you will have issues with signal handlers).

[x42005e1f]

The implementation decomposes BaseEventLoop._run_once() into three new public methods — poll_events(), process_events(), and process_ready() — then uses a dual-thread architecture where the host thread processes callbacks and a daemon I/O thread polls for events.

I believe that this is insufficient, especially in terms of performance. Let us suppose we have a set of tasks, and at least one of them is active at any point in time. In this case, the timeout will always be zero. But since in your model, events are always polled by a separate thread, even a zero timeout (even with a single task that simply does asyncio.sleep(0) in an infinite loop) will result in quite expensive ping-ponging between the two threads, and we will also lose determinism between guest tasks and host tasks.

The problem can be solved by providing a method for obtaining an effective timeout and delegating polling to a separate thread only when it is non-zero. The entire poll_events() code before the select() call is responsible for this.

However, these methods are too low-level. Any guest mode implementation will have to repeat almost the entire asyncio.Runner.run() (both thread-local state management and event loop shutdown; I do not see the latter or even loop._run_forever_setup()/loop._run_forever_cleanup() in the linked PR).