Skip to content

test(integration): function-scoped fixtures for order-independent annotation tests#166

Draft
henrycgbaker wants to merge 6 commits intofix/annotation-teardown-clears-all-datasetsfrom
test/annotation-fixture-isolation
Draft

test(integration): function-scoped fixtures for order-independent annotation tests#166
henrycgbaker wants to merge 6 commits intofix/annotation-teardown-clears-all-datasetsfrom
test/annotation-fixture-isolation

Conversation

@henrycgbaker
Copy link
Copy Markdown
Collaborator

@henrycgbaker henrycgbaker commented Apr 25, 2026

Goal

Make the annotation integration tests truly order-independent. They were structured as an implicit scenario chain (scope="module" fixtures, each test depending on what the previous one left in Argilla), which broke whenever you ran pytest -k <one_test> or in any order other than the alphabetical default.

Scope

Both test_annotation_setup.py and test_annotation_import.py:

  • Change autouse cleanup fixtures from scope="module" to function scope.
  • Each test now sets up its own preconditions (e.g. tests that need workspaces call setup_workspaces themselves rather than relying on a previous test).
  • test_annotation_setup.py purges the test user account around every test (production teardown_resources deliberately preserves users, so test isolation needs an explicit purge).
  • Drop redundant per-test cleanup that the autouse fixture now subsumes (test_dataset_auto_creation no longer manually tears down).
  • Drop the unused first teardown_resources(client, _SETTINGS) call in test_annotation_import.py — a full teardown (empty dataset_id) clears every dataset in the workspace anyway, so the scoped teardown beforehand is redundant.

Implementation

Setup fixture before:

@pytest.fixture(autouse=True, scope="module")
def clean_slate(client):
    teardown_resources(client, _DEFAULT_SETTINGS)
    yield
    teardown_resources(client, _DEFAULT_SETTINGS)

After (combines #165's orphan-purge with this PR's user-purge + function scope):

@pytest.fixture(autouse=True)  # function scope (default)
def clean_slate(client):
    for ws_base in _DEFAULT_SETTINGS.workspace_dataset_map:
        purge_workspace_datasets(client, ws_base)
    teardown_resources(client, _DEFAULT_SETTINGS)
    _purge_test_user(client)
    yield
    for ws_base in _DEFAULT_SETTINGS.workspace_dataset_map:
        purge_workspace_datasets(client, ws_base)
    teardown_resources(client, _DEFAULT_SETTINGS)
    _purge_test_user(client)

Each test then explicitly bootstraps its preconditions:

def test_idempotent_rerun(client):
    setup_workspaces(client, _DEFAULT_SETTINGS)  # explicit, not implicit

    result = setup_workspaces(client, _DEFAULT_SETTINGS)

    assert result.created_workspaces == []

Trade-offs

Slower: each test re-creates workspaces (~3-5x slower, but still seconds, not minutes — each Argilla op is HTTP). Worth it for proper isolation. Alternative would be folding the seven setup tests into a single stateful scenario test, but the tests are answering distinct questions about library functions, not narrating a user workflow — function scope is the right idiom.

Testing

  • pytest tests/integration/test_annotation_setup.py tests/integration/test_annotation_import.py -m "integration and annotation" (Docker up): 14 passed, no pre-existing-state required.
  • All 7 setup tests pass individually (pytest -k <name>).
  • All 7 import tests pass when reordered or run individually.
  • Pytest's standard ordering, reverse ordering, and arbitrary subsets all green.

References

@henrycgbaker henrycgbaker force-pushed the fix/annotation-teardown-clears-all-datasets branch from 681db9c to cb9af28 Compare April 26, 2026 11:46
@henrycgbaker henrycgbaker force-pushed the test/annotation-fixture-isolation branch from fc9d4b1 to 451befd Compare April 26, 2026 11:51
@henrycgbaker
Copy link
Copy Markdown
Collaborator Author

NB pending decision in #165 re: global vs scoped ds deletion -> update this PR accordingly

…am (#167)

* refactor(makefile): consolidate docker-up variants behind PROFILE param

* refactor: makefile comments

* Fix Makefile syntax for .PHONY target
@henrycgbaker henrycgbaker force-pushed the fix/annotation-teardown-clears-all-datasets branch from 493f0ab to 8e4e535 Compare April 30, 2026 10:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant