Skip to content

ci: upload system-tests results to Test Optimization and add tracer-release nightly#3660

Draft
nccatoni wants to merge 10 commits intomasterfrom
add-nightly-system-tests
Draft

ci: upload system-tests results to Test Optimization and add tracer-release nightly#3660
nccatoni wants to merge 10 commits intomasterfrom
add-nightly-system-tests

Conversation

@nccatoni
Copy link

@nccatoni nccatoni commented Feb 18, 2026

Description

Two changes to the system-tests CI integration:

  1. Upload all system-tests results to Test Optimization — adds an after_script to the .system_tests base job that copies JUnit XML files and uploads them via the existing silent-upload-junit-to-datadog.sh script.

  2. Add tracer-release nightly scenarios — extends configure_system_tests to include the tracer-release scenario group when running on schedule or on master, using GitLab CI rules with variable overrides.

Follows the same pattern used in system-tests-dashboard nightly.

Reviewer checklist

  • This is a CI-only change, no code changes.
  • Appropriate labels assigned.

Run the official system-tests reusable workflow daily at 4 AM GMT
using dev mode against the latest commit on the default branch.

Co-authored-by: Cursor <cursoragent@cursor.com>
@datadog-official
Copy link

datadog-official bot commented Feb 18, 2026

⚠️ Tests

Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 1028 Tests failed

testSearchPhpBinaries from integration.DDTrace\Tests\Integration\PHPInstallerTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integration\PHPInstallerTest::testSearchPhpBinaries
Test code or tested code printed unexpected output: Searching for available php binaries, this operation might take a while.
testSimplePushAndProcess from laravel-58-test.DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 69a0715a000000003a01fa7c6ce5b33a
tid: 69a0715a00000000
hexProcessTraceId: 3a01fa7c6ce5b33a
hexProcessSpanId: f36b5b336bc46f17
processTraceId: 4179897341486412602
processSpanId: 17540213450134023959

phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:106
testSimplePushAndProcess from laravel-8x-test.DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 69a07f8f00000000bcc093c6a776d081
tid: 69a07f8f00000000
hexProcessTraceId: bcc093c6a776d081
hexProcessSpanId: 46254faa2639fc94
processTraceId: 13601033356081287297
processSpanId: 5054533748997618836
View all

ℹ️ Info

❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 50f858b | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

Co-authored-by: Cursor <cursoragent@cursor.com>
@nccatoni nccatoni changed the title Add nightly system-tests workflow ci: add nightly system-tests workflow Feb 18, 2026
@codecov-commenter
Copy link

codecov-commenter commented Feb 18, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.11%. Comparing base (a9caaa0) to head (50f858b).
⚠️ Report is 7 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3660      +/-   ##
==========================================
- Coverage   62.21%   62.11%   -0.11%     
==========================================
  Files         141      141              
  Lines       13387    13387              
  Branches     1753     1753              
==========================================
- Hits         8329     8315      -14     
- Misses       4260     4273      +13     
- Partials      798      799       +1     

see 3 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a9caaa0...50f858b. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nccatoni and others added 6 commits February 18, 2026 15:40
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
- Add after_script to .system_tests base job to upload JUnit results
  to Datadog Test Optimization using the existing upload script
- Add tracer-release scenario group to configure_system_tests on
  schedule and master runs
- Remove separate GitHub Actions nightly workflow

Made-with: Cursor
@nccatoni nccatoni changed the title ci: add nightly system-tests workflow ci: upload system-tests results to Test Optimization and add tracer-release nightly Feb 26, 2026
The silent wrapper's grep returns exit code 1 when no summary lines
are found, causing the after_script to fail. Use the non-silent
version with || true to prevent after_script failures.

Made-with: Cursor
Replace the generic upload-junit-to-datadog.sh with a custom
after_script that:
- Uses --service system-tests (not dd-trace-php-tests)
- References logs directly from system-tests/logs*/
- Uses the datadog-ci standalone binary (no npm needed)
- Uses Python for JSON/zip (guaranteed available in the image)
- Provides explicit error messages instead of silent exit 0

Made-with: Cursor
@pr-commenter
Copy link

pr-commenter bot commented Feb 26, 2026

Benchmarks [ tracer ]

Benchmark execution time: 2026-02-26 17:23:58

Comparing candidate commit 50f858b in PR branch add-nightly-system-tests with baseline commit a9caaa0 in branch master.

Found 2 performance improvements and 4 performance regressions! Performance is the same for 188 metrics, 0 unstable metrics.

scenario:MessagePackSerializationBench/benchMessagePackSerialization

  • 🟩 execution_time [-7.788µs; -7.132µs] or [-7.270%; -6.657%]

scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache

  • 🟩 execution_time [-6.832µs; -5.588µs] or [-6.144%; -5.025%]

scenario:SamplingRuleMatchingBench/benchRegexMatching1

  • 🟥 execution_time [+114.047ns; +153.753ns] or [+9.980%; +13.455%]

scenario:SamplingRuleMatchingBench/benchRegexMatching2

  • 🟥 execution_time [+96.248ns; +160.752ns] or [+8.364%; +13.970%]

scenario:SamplingRuleMatchingBench/benchRegexMatching3

  • 🟥 execution_time [+85.931ns; +140.069ns] or [+7.386%; +12.039%]

scenario:SamplingRuleMatchingBench/benchRegexMatching4

  • 🟥 execution_time [+88.585ns; +146.215ns] or [+7.612%; +12.565%]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants