Daily smoke benchmark ci#440
Conversation
smoke-test.yml ran the 124-script BASE_SCRIPTS functional suite at -j 1 (fully sequential) across all 5 platform configs on every push/PR. Reuse the same $CPU_COUNT already computed for the build/ctest steps in this job instead. Also drop -DENABLE_BENCH=ON: smoke test built bench_tapyrus but never ran it, and the corresponding bench_results.txt artifact-upload entry was dead (never produced here, silently no-op via if-no-files-found: ignore). Removes wasted build time for a binary nothing uses. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
daily-test.yml triggers on both schedule and pull_request, so every PR was already paying for the full 13-config functional-test matrix plus the benchmark step and its history-cache round-trip. Benchmarks add real time and don't need to run per-PR - gate the benchmark execution, and its two history-cache steps, behind github.event_name != 'pull_request'. The functional-test matrix itself is untouched and still runs on every PR. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
No structured CI reporting existed - test_runner.py only prints to console and dumps logs into an uploaded artifact directory, so triaging a failure meant opening and reading raw log artifacts. Render a per-job markdown table (category, pass/fail/skip, duration) to $GITHUB_STEP_SUMMARY instead, visible directly on the run's Summary page for each of the 13 matrix combinations: unit tests, Qt GUI tests, util tests, functional tests (or debugscripts on Debug builds), and benchmark (or its skip reason). Existing fail-fast behavior (exit 1 on the first broken phase) is preserved - each phase's if/else now also calls record_phase/ record_skipped before deciding whether to exit, so the summary reflects however far the job got even on failure, not just full passes. This is intentionally just the pass/fail/timing table for now; the benchmark_results/data history-cache (already present but unpopulated) and a trend graph per category are follow-up work. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Append each scheduled run's per-benchmark medians to a cached CSV and render a log-scale trend chart (one line per benchmark) into the job summary via a new contrib/devtools/plot_benchmark_history.py. Also pin history collection to one canonical config per platform instead of matching on platform+arch, since several matrix configs share the same platform+arch and would otherwise race on the same cache key/file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Two things I'd like to see folded in before this lands. Both are small; R2 in particular is directly on-topic for this PR's title. R1 — Move the benchmark-history opt-in from three literal
|
…ed, fix bugs in core dump creation steps and enable writing the backtrace to job summary
|
Apart from adding the review recommendations, the last commit also fixes the core dump creation and debugging steps. |
Improvements to CI:
smoke test - run jobs in parallel
CI on PR - do not run benchmark test to save time
Daily CI - run benchmark. plot a graph using the benchmark data using matplotlib python library