test(integration): function-scoped fixtures for order-independent annotation tests#166
Draft
henrycgbaker wants to merge 6 commits intofix/annotation-teardown-clears-all-datasetsfrom
Draft
Conversation
This was referenced Apr 25, 2026
681db9c to
cb9af28
Compare
fc9d4b1 to
451befd
Compare
Collaborator
Author
|
NB pending decision in #165 re: global vs scoped ds deletion -> update this PR accordingly |
…am (#167) * refactor(makefile): consolidate docker-up variants behind PROFILE param * refactor: makefile comments * Fix Makefile syntax for .PHONY target
493f0ab to
8e4e535
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
Make the annotation integration tests truly order-independent. They were structured as an implicit scenario chain (
scope="module"fixtures, each test depending on what the previous one left in Argilla), which broke whenever you ranpytest -k <one_test>or in any order other than the alphabetical default.Scope
Both
test_annotation_setup.pyandtest_annotation_import.py:scope="module"to function scope.setup_workspacesthemselves rather than relying on a previous test).test_annotation_setup.pypurges the test user account around every test (productionteardown_resourcesdeliberately preserves users, so test isolation needs an explicit purge).test_dataset_auto_creationno longer manually tears down).teardown_resources(client, _SETTINGS)call intest_annotation_import.py— a full teardown (emptydataset_id) clears every dataset in the workspace anyway, so the scoped teardown beforehand is redundant.Implementation
Setup fixture before:
After (combines #165's orphan-purge with this PR's user-purge + function scope):
Each test then explicitly bootstraps its preconditions:
Trade-offs
Slower: each test re-creates workspaces (~3-5x slower, but still seconds, not minutes — each Argilla op is HTTP). Worth it for proper isolation. Alternative would be folding the seven setup tests into a single stateful scenario test, but the tests are answering distinct questions about library functions, not narrating a user workflow — function scope is the right idiom.
Testing
pytest tests/integration/test_annotation_setup.py tests/integration/test_annotation_import.py -m "integration and annotation"(Docker up): 14 passed, no pre-existing-state required.pytest -k <name>).References
purge_workspace_datasetshelper introduced there).