Improve instructions on how to run tests for AI agents#5592
Open
Improve instructions on how to run tests for AI agents#5592
Conversation
🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: fb870dc | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback! |
vpellan
approved these changes
Apr 15, 2026
Comment on lines
+61
to
+62
| - Contrib/integration tests need Docker: `docker compose run --rm tracer-3.4 /bin/bash`, then run the rake task inside | ||
| - `test:main` can run locally on any Ruby for quick feedback |
Contributor
There was a problem hiding this comment.
In #5588, I added instructions to tell agents to run the tests twice: once with the earliest version we support (2.5) and once with the latest version (4.0) because it often happens that we miss regressions in older rubies.
Also, it often happens after a dependency update that cursor fails to run bundler. I think we can also add an instruction like If Bundler fails in the container, run bundle install and retry once
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR aims to improve the identification of a correct test task for AI agents.
Motivation:
AI agents (Claude Code, Cursor, etc.) currently try running isolated spec files with
bundle exec rspec ..., which fails for contrib/integration tests that require specific appraisal Gemfiles. This change teaches the agents thelib/→spec/→rake test:TASK_KEYmapping, including the product-level pattern (namespaced tasks for appsec/profiling/di/ai_guard, flat tasks for tracing contribs), and how to self-serve edge cases by checking theRakefileandMatrixfile.Change log entry
None.
Additional Notes:
I used AI to write these changes, used AI to test if these instructions actually improve anything.
How to test the change?
Try to do some simple change using your favorite coding agent, see if it can correctly identify the test tasks it needs to run.