Unexpected attached to a different loop errors

[sk-]

We had a hard to debug issue when testing async sqlalchemy.

When running the following test

from sqlalchemy.sql import text
from sqlalchemy.ext.asyncio import create_async_engine
from sqlalchemy.pool import NullPool

engine = create_async_engine(
    "postgresql+asyncpg://user:password@localhost:5432/db_name", future=True#, poolclass=NullPool
)


async def test_echo():
    async with engine.begin() as conn:
        r = await conn.execute(text("SELECT 'hello' as echo"))
        assert r.scalar() == "hello"

async def test_echo2():
    async with engine.begin() as conn:
        r = await conn.execute(text("SELECT 'hello' as echo"))
        assert r.scalar() == "hello"

async def test_echo3():
    async with engine.begin() as conn:
        r = await conn.execute(text("SELECT 'hello' as echo"))
        assert r.scalar() == "hello"

we would get two errors:

attached to a different loop
<class 'asyncpg.exceptions._base.InterfaceError'>: cannot perform operation: another operation is in progress

This is fixed by having a global event loop as suggested in #38 (comment). Should there maybe be a command line option to enable this?

I'm opening this issue, as the creator of sqlalchemy suggested to report this to the underlying asyncio runner. See MagicStack/asyncpg#863 (comment)

[Tinche]

Using a single, global event loop is less than ideal tbh, your unit tests won't be independent of each other (so, won't be unit tests).

In your case here, I'd say make engine a fixture and have it depend on the event_loop fixture, but I realize that might be impossible for a non-toy example.

[seifertm]

This is fixed by having a global event loop as suggested in #38 (comment). Should there maybe be a command line option to enable this?

There has been a proposal to deprecate overrides of the event loop fixture in favour of pytest config options: #311 (comment)

The proposal has been close in lieu of alternative solutions, namely aioloop-proxy. However, I'm a bit wary of integrating aioloop-proxy (see #312) when nobody except Andrew has access to the repository and the pypi packages.

Maybe it's time to re-evaluate the deprecation of event loop fixture overrides? I'd be happy for any feedback.

[nuno-andre]

FWIW. I get the Future <Future pending> attached to a different loop exception when changing the suggested fixture:

@pytest.fixture(scope='session')
def event_loop():
    policy = asyncio.get_event_loop_policy()
    loop = policy.new_event_loop()
    yield loop
    loop.close()

with

@pytest.fixture(scope='session')
def run_in_loop():
    policy = asyncio.get_event_loop_policy()
    loop = policy.new_event_loop()
    yield loop.run_until_complete
    loop.close()

(Yep. This is only for replacing event_loop.run_until_complete with run_in_loop 🦥).

I don't know if this is an expected session-fixture behaviour that I don't understand or could be a bug.

Happy to open a new issue if this is a bug but it's not related with this issue.

[seifertm]

@nuno-andre pytest-asyncio injects the fixture with the name event_loop into every async test case. If you create another fixture named run_in_loop and use it in a test, the corresponding test will evaluate both fixtures. Since both fixtures—event_loop and run_in_loop—create a new event loop, the warning is to be expected.

If you want to have a session-wide event loop, do override the event_loop fixture only. Anything else will lead to trouble.

[naquad]

I've got the same errors and lots of them. It seems that pytest-asyncio somehow and somewhy starts new loops while previous ones are still active and SQLAlchemy 2.0+ goes nuts :( None of the suggested fixes have helped. It looks to be a collision between asyncio, greenlets and the way SQLAlchemy tries to interoperate between both.

[seifertm]

pytest-asyncio closes existing loops when initializing the event_loop fixture. As outlined in this comment, this behavior can make it difficult for the user to spot errors in their code or problems with dependencies trying to manage the event loop in parallel to pytest-asyncio.

Most importantly, though, parts of the low-level asyncio API have been deprecated, so the "close existing loops" functionality in pytest-asyncio will also have to go.

The upcoming release of pytest-asyncio will print a warning when the event_loop fixture goes out of scope and the loop is not properly closed. @naquad Maybe this helps debugging your issue. Other than that, feel free to provide a minimal reproducer.

[seifertm]

The OP experienced issues with changing event loops in their code.

pytest-asyncio warns about event loops that are not properly cleaned up since v0.21. In addition to that, #620 introduces a way to have an event loop with larger scope, which reportedly solved the OP's issue.

I'm closing this issue as resolved, since there's not much more to go on.