Skip to content

fix(tests): apply integration marker and timeout defaults per directory - #164

Merged
pofallon merged 2 commits into
mainfrom
163-integration-test-timeouts
Jul 25, 2026
Merged

fix(tests): apply integration marker and timeout defaults per directory#164
pofallon merged 2 commits into
mainfrom
163-integration-test-timeouts

Conversation

@pofallon

@pofallon pofallon commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Closes #163.

The flakiness

Three integration tests intermittently timed out under -n auto and passed in isolation. Not a hang — insufficient headroom against the global 30s default in pyproject.toml. Serial runtimes on an idle 8-core box:

Test Serial Headroom
test_scenario_2_rollback_on_gate_failure 19.9s 10.1s
test_scenario_1_clean_retroactive_application 9.2s 20.8s
test_record_commit_stamp_flow 6.2s 23.8s

test_scenario_2 needed only a ~1.5x contention slowdown to breach the budget. These tests shell out to real jj/bd/git, so under seven competing xdist workers that's routine.

Where they bite: local full-suite runs, not CI. GitHub Actions installs jj but not bd, and both modules pytest.skip(allow_module_level=True) without their tooling — so they never run here. make ci on a dev box is the documented pre-push gate (CLAUDE.md), which is where they failed. (I had this wrong in the first draft of this PR and in the conftest docstring; corrected in 3ef9746.)

A second bug found while fixing it

tests/integration/conftest.py applied its marker via a module-level pytestmark — which pytest ignores in a conftest; it only honours pytestmark in test modules. So the integration marker only ever covered the four files that happened to declare it themselves, and -m integration silently under-collected the other thirteen. Anyone running -m integration or -m "not integration" has been getting wrong results.

Both are the same root cause — the directory had no working mechanism for applying defaults — so both are fixed by the same replacement: a pytest_collection_modifyitems hook that applies the marker and a default timeout, filtered to this directory and skipped wherever a test already said otherwise.

-m integration now collects 113 tests.

Sizing

120s, chosen from the measurements above rather than picked round: it absorbs a ~6x contention slowdown over the slowest observed test while still failing a genuinely hung test in bounded time. Explicit per-test overrides already in the tree (60s, 90s, 150s) still win — verified.

Not doing

The issue also suggested adding @pytest.mark.slow to integration tests over ~5s "so make test-fast excludes them consistently". Checking the Makefile, test-fast runs tests/unit/ by path — integration tests are already excluded regardless of markers. That suggestion was unfounded and is skipped.

Verification

make ci-coverage (the exact CI target): 4388 passed, 1 xfailed, coverage 85.45%.

New tests/unit/test_integration_suite_defaults.py covers the conftest itself, since both bugs here were the invisible kind — the suite passed the whole time, just with the wrong marks and too little headroom. It asserts marker coverage, that no integration test falls through to the 30s global, that explicit overrides win, and that nothing leaks into the unit suite.

Two things I checked rather than assumed:

  • The tests genuinely catch the regression — restored the old conftest and watched 3 of them fail.
  • They're environment-independent — first version named specific tests and broke in CI for the collection reason above; rewritten as invariants over whatever collects, and verified by running with bd removed from PATH (7 passed, 1 skipped with a stated reason).

🤖 Generated with Claude Code

https://claude.ai/code/session_012av72R5SPCc84chsUNjhLC

Paul O'Fallon and others added 2 commits July 25, 2026 23:38
Closes #163.

`tests/integration/conftest.py` set a module-level `pytestmark`, which
pytest ignores in a conftest — it only honours `pytestmark` in test
modules. So the `integration` marker covered only the four files that
declared it themselves, and `-m integration` silently missed the other
thirteen (113 tests now collect under that selector; far fewer did).

Replacing it with a `pytest_collection_modifyitems` hook fixes that and
gives the suite a directory-scoped default timeout. These tests shell out
to real jj/bd/git, so under `-n auto` they take several times their
serial runtime; the global 30s default from pyproject.toml — right for
pure-Python unit tests — left too little headroom and produced
intermittent CI failures that passed on re-run and in isolation.

The default is sized from measurement: the slowest offender
(test_scenario_2_rollback_on_gate_failure) runs ~20s serially on an idle
8-core box, so it needed only a ~1.5x contention slowdown to breach 30s.
120s absorbs ~6x while still failing a genuinely hung test in bounded
time. Explicit per-test markers (60/90/150s) still win.

Not doing the `slow`-marker part of the issue: `make test-fast` scopes to
`tests/unit/` by path, so integration tests are already excluded from it
regardless of markers — that suggestion was unfounded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012av72R5SPCc84chsUNjhLC
The first version named specific tests (test_scenario_2..., the ledger
overrides) and asserted they collect. They don't in CI: GitHub Actions
installs jj but not bd, and both modules call
`pytest.skip(allow_module_level=True)` when their tooling is missing,
which prevents collection entirely.

Restated as invariants over whatever collected — every integration test
carries a budget, the default is reachable, nothing leaks into the unit
suite — with the one genuinely test-specific check skipping explicitly
when bd is absent. Verified by running with bd removed from PATH.

Also corrects the conftest rationale: these failures surface in local
full-suite runs, not CI, for the same reason. `make ci` on a dev box is
the documented pre-push gate, which is where they bit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012av72R5SPCc84chsUNjhLC
@pofallon
pofallon merged commit 2e68188 into main Jul 25, 2026
2 checks passed
@pofallon
pofallon deleted the 163-integration-test-timeouts branch July 25, 2026 23:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: three real-jj/bd integration tests time out under parallel load

1 participant