Report test results broken down by project structure - #703
Open
danielfrankcom wants to merge 32 commits into
Open
Report test results broken down by project structure#703danielfrankcom wants to merge 32 commits into
danielfrankcom wants to merge 32 commits into
Conversation
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
The feature tree (from test paths) replaced marker/by_tag grouping. Drop the pytest.ini marker parsing, extract_markers, the by_tag build, and the per-test tags field; ResultAnalyzer no longer needs a pytest.ini path. Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
The analyzer recomputed its own summary dict in parallel with reconciliation, which supersedes it (adds collected/deselected, uses the correct pass-rate denominator). Drop the recomputed summary; JSON output and the CLI exit code now use reconciliation. The exit code also fails on errors, not just failures. Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
A runtime pytest.skip() inside a test body records its reason in the call phase, but only setup was read, so those skips showed a blank reason in the report. Scan for the phase whose outcome is skipped, matching how failures already find their phase. Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
The sidecar is written at collection time, but the report's directory may not exist until pytest-json-report creates it at session end. In CI the write therefore failed and was swallowed, so the feature tree showed every area as 0 unsupported. Create the directory, and warn instead of failing silently. Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Known gaps, skipped, and the feature breakdown were bare collapsibles, so they read as more entries in the needs-attention list rather than sections of their own. Give each a heading carrying its count, leaving the detail behind a Show all toggle. Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
|
🤖 Auto-triaged by documentdb-triage-tool. Applied: Reasoningcomponent from path globs (result-analyzer, docs, test-framework, ci); effort from diff stats (1733+442 LOC, 18 files); LLM: Reworks CI test reporting and result-analyzer logic across multiple concerns (report format, xpass strictness, reason enforcement), touching multiple files in the result_analyzer component and CI integration. If a label is wrong, remove it manually and ping |
documentdb-triage-tool
Bot
added
documentation
pytest records a skip as the repr of a ('<path>', <lineno>, 'Skipped: <reason>')
tuple, and repr switches the reason to double quotes whenever it contains an
apostrophe ("doesn't apply here"). The extraction regex only accepted a
single-quoted reason, so any reason with an apostrophe was silently dropped from
the report — while the new validator simultaneously forces authors to write one.
Accept either closing quote.
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Test IDs, reasons, and tracebacks embed arbitrary test inputs (parametrized IDs and repr'd documents), but the renderer interpolated them raw into three markup contexts where they can be markup themselves: - a | in a table cell splits the row, so escape it - a < in a <summary> can be eaten as an HTML tag, so html-escape the name - a ``` inside a traceback closes the fixed 3-backtick fence early and corrupts everything after it, so size the fence past the longest backtick run in the content No current test data hits these, so this is insurance against silent report corruption when future data does. Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
_merge_json_reports summed only passed/failed/error/skipped/total, so a Phase 2 (no_parallel) xfail or xpass vanished from the merged summary while its entry remained in the merged tests array. The report then contradicted itself: the known-gaps section listed the test but the breakdown count excluded it, and a Phase 2 strict-xpass would not fail the verdict. Sum every outcome bucket. Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
The sidecar was only written when this run deselected something, so a sidecar
left by a previous run survived a run with no deselection and the analyzer
credited phantom unsupported tests into the report. Write it unconditionally
(as {} when nothing was deselected) so it always reflects the current run.
Phase 2's subprocess also writes a sidecar next to its temp report; that one is
redundant (Phase 1 collects everything, so its sidecar covers all requires-based
deselections) and was never cleaned up. Remove it with the temp report.
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Under xdist every worker runs collection and writes the same sidecar path with a plain truncate-and-write, which can tear the file. The analyzer degrades gracefully on invalid JSON by returning no reasons, so the failure mode is a silently reason-less report - the exact quiet incompleteness the write warning was added to prevent. Write to a per-process temp file and os.replace it into place, so readers only ever see a complete sidecar. Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
The exit code checked only failed/error counts, while determine_verdict also fails a run for raw xpasses (strict xfail not applied, results untrustworthy) and for runs where nothing executed. A consumer scripting on the exit code got 0 for a run the report itself declared FAIL. Reuse the verdict so the two can't disagree. Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
feature_path located the taxonomy root with rfind("tests/"), a bare substring
match that also hits directories merely ending in "tests" - e.g.
"documentdb_tests/framework/test_x.py" resolved to ['framework', 'test_x.py']
instead of no path. Harmless today because only compatibility/tests nodeids
reach it, but the trap is invisible at the call site. Anchor the match to a
path-component boundary.
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
The validator matched any attribute call named skip/fail/xfail, so a PyMongo cursor's .skip(n) was reported as a pytest.skip() missing its message. A violation aborts collection for the whole suite, so one idiomatic pagination test would have blocked every run. Signed-off-by: Daniel Frankcom <frankcom@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change reworks the CI test report to provide as much information as possible at a glance.
The main aims were to:
The new report format is easiest to look at in the form of real test scenarios I ran on CI during testing. These are contrived failures for the purpose of generating the specific report types.
xfailtest starts unexpectedly passing, then it appears in the "needs attention" sectionAnalyzer correctness fixes along the way
xfailreasons are now mandatory and enforced by the validator, to provide better info in the reportxpassis now configured as strict, meaningxfailtests will now fail if they unexpectedly start to passFixes #121