Skip to content

Add tests for new feature API Endpoints: Business logic events for LLM-based SDKs #6339

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
estringana wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

Add tests for new feature API Endpoints: Business logic events for LLM-based SDKs #6339

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
6772469
Add tests
estringana Feb 18, 2026
cf24a2f
Remove irrelevant tags not used anymore
estringana Feb 19, 2026
db84958
Mark irrelevant nodejs tests
estringana Feb 19, 2026
fc2fb08
Refactor tests
estringana Feb 19, 2026
857c6b5
Inject rules
estringana Feb 19, 2026
7cb22da
Add fake responses to openai calls
estringana Feb 19, 2026
349ab13
Fix mock responses
estringana Feb 20, 2026
cc5dc1a
Explains mock openai responses
estringana Feb 20, 2026
888f9c3
[PHP] Enable llm tests php (#6354)
estringana Feb 23, 2026
4a7f6a5
Mark php tests as missing
estringana Feb 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions docs/weblog/end-to-end_weblog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1263,6 +1263,33 @@ It takes a raw (unparsed) request body, and a signature located in header `Strip
The endpoint must return as JSON in the response body, the sub-object `event.data.object` returned by the `constructEvent()` Stripe SDK method.
If an error happens, the endpoint must respond with a 403 error code.

### GET /llm

This endpoint is implemented by Python, Node.js, and PHP (using openai-php/client).

This endpoint collects interactions with LLMs. The request will have the following query parameters
- `model`: Identifies the LLM model invoked. Examples are: `gpt-4.1`, `gpt-4o-mini`, `text-davinci-003`.
- `operation`: Instead of having one each point for each function wrapped, this parameter will be used to decide what method to trigger. The following table maps all operation values to wrapped method:
| Value | Python mapped method | Node.js mapped method | PHP mapped method |
| --- | --- | --- | --- |
| `openai-latest-responses.create` | `OpenAI().responses.create(...)` | `client.responses.create` | `$client->responses()->create(...)` |
| `openai-latest-chat.completions.create` | `OpenAI().chat.completions.create(...)` | `client.chat.completions.create` | `$client->chat()->create(...)` |
| `openai-latest-completions.create` | `OpenAI().completions.create(...)` | `client.completions.create` | `$client->completions()->create(...)` |
| `openai-legacy-chat.completions.create` | `openai.ChatCompletion.create` | `openai.createChatCompletion` | `$client->chat()->create(...)` |
| `openai-legacy-completions.create` | `openai.Completion.create` | `openai.createCompletion` | `$client->completions()->create(...)` |
| `openai-async-responses.create` | `AsyncOpenAI().responses.create(...)` | not implemented | `$client->responses()->create(...)` |
| `openai-async-chat.completions.create` | `AsyncOpenAI().chat.completions.create(...)` | not implemented | `$client->chat()->create(...)` |
| `openai-async-completions.create` | `AsyncOpenAI().completions.create(...)` | not implemented | `$client->completions()->create(...)` |

For example a call to `/llm?model=gpt-4.1&operation=openai-latest-responses.create` (URL-encoded) will require that Python does the following call
```
OpenAI().responses.create(model="gpt-4.1", ...)
```

This approach makes the endpoint ready to be expanded in the future.

In scenarios that use this endpoint, `OPENAI_BASE_URL` is set to the system-tests internal server, which mocks the OpenAI API responses.

## Weblog specification

There are several rules shared between all the existing end-to-end weblogs.
Expand Down
9 changes: 9 additions & 0 deletions manifests/cpp.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,15 @@
# use `>1.0.0` to indicate that requirement.
# NOTE: only parametric tests are run for cpp, there is no need to mark other tests as "missing_feature" here.
manifest:
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_root_has_no_llm_tags: irrelevant (language not implementing this feature)
tests/parametric/test_128_bit_traceids.py::Test_128_Bit_Traceids: ">1.0.0"
tests/parametric/test_128_bit_traceids.py::Test_128_Bit_Traceids::test_b3single_128_bit_generation_disabled: missing_feature (propagation style not supported)
tests/parametric/test_128_bit_traceids.py::Test_128_Bit_Traceids::test_b3single_128_bit_generation_enabled: missing_feature (propagation style not supported)
Expand Down
9 changes: 9 additions & 0 deletions manifests/cpp_httpd.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,15 @@ manifest:
tests/ai_guard/test_ai_guard_sdk.py::Test_SDK_Disabled: missing_feature
tests/apm_tracing_e2e/: missing_feature (missing /e2e_otel_span endpoint on weblog)
tests/appsec/: irrelevant (ASM is not implemented in C++)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_root_has_no_llm_tags: irrelevant (language not implementing this feature)
tests/debugger/: irrelevant
tests/ffe/test_dynamic_evaluation.py: missing_feature
tests/ffe/test_exposures.py: missing_feature
Expand Down
9 changes: 9 additions & 0 deletions manifests/cpp_nginx.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,15 @@ manifest:
tests/appsec/api_security/test_custom_data_classification.py::Test_API_Security_Custom_Data_Classification_Scanner: v1.10.0
tests/appsec/api_security/test_endpoint_discovery.py: irrelevant (not applicable to proxies)
tests/appsec/api_security/test_endpoint_fallback.py: irrelevant (not applicable to proxies)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_root_has_no_llm_tags: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_schemas.py::Test_Scanners: v1.8.0
tests/appsec/api_security/test_schemas.py::Test_Schema_Request_Cookies: v1.8.0
tests/appsec/api_security/test_schemas.py::Test_Schema_Request_FormUrlEncoded_Body: v1.8.0
Expand Down
9 changes: 9 additions & 0 deletions manifests/dotnet.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,15 @@ manifest:
tests/appsec/api_security/test_custom_data_classification.py::Test_API_Security_Custom_Data_Classification_Scanner: missing_feature
tests/appsec/api_security/test_endpoint_discovery.py::Test_Endpoint_Discovery: v3.24.0
tests/appsec/api_security/test_endpoint_fallback.py: missing_feature
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_root_has_no_llm_tags: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_schemas.py::Test_Scanners: v2.46.0
tests/appsec/api_security/test_schemas.py::Test_Schema_Request_Cookies: v2.46.0
tests/appsec/api_security/test_schemas.py::Test_Schema_Request_FormUrlEncoded_Body: v2.46.0
Expand Down
9 changes: 9 additions & 0 deletions manifests/envoy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/DataDog/system-tests/refs/heads/main/utils/manifest/schema.json
---
manifest:
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_root_has_no_llm_tags: irrelevant (language not implementing this feature)
tests/appsec/test_alpha.py: v1.72.0
tests/appsec/test_blocking_addresses.py: v1.72.0
tests/appsec/test_blocking_addresses.py::Test_BlockingGraphqlResolvers: irrelevant
Expand Down
9 changes: 9 additions & 0 deletions manifests/golang.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,15 @@ manifest:
tests/appsec/api_security/test_custom_data_classification.py::Test_API_Security_Custom_Data_Classification_Scanner: v2.4.0
tests/appsec/api_security/test_endpoint_discovery.py::Test_Endpoint_Discovery: missing_feature
tests/appsec/api_security/test_endpoint_fallback.py: missing_feature
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_root_has_no_llm_tags: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_schemas.py::Test_Scanners: v2.0.0
tests/appsec/api_security/test_schemas.py::Test_Schema_Request_Cookies: v1.60.0
tests/appsec/api_security/test_schemas.py::Test_Schema_Request_FormUrlEncoded_Body: v1.60.0
Expand Down
9 changes: 9 additions & 0 deletions manifests/haproxy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/DataDog/system-tests/refs/heads/main/utils/manifest/schema.json
---
manifest:
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_root_has_no_llm_tags: irrelevant (language not implementing this feature)
tests/appsec/test_alpha.py: v2.4.0
tests/appsec/test_blocking_addresses.py: v2.4.0
tests/appsec/test_blocking_addresses.py::Test_BlockingGraphqlResolvers: irrelevant
Expand Down
9 changes: 9 additions & 0 deletions manifests/java.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,15 @@ manifest:
uds-spring-boot: irrelevant (Not applicable to weblog variant)
spring-boot-jetty: irrelevant (Not applicable to weblog variant)
tests/appsec/api_security/test_endpoint_fallback.py: missing_feature
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_root_has_no_llm_tags: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_schemas.py::Test_Scanners:
- weblog_declaration:
"*": v1.31.0
Expand Down
9 changes: 9 additions & 0 deletions manifests/java_otel.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/DataDog/system-tests/refs/heads/main/utils/manifest/schema.json
---
manifest:
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_async_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_latest_responses_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_chat_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_openai_legacy_completions_create: irrelevant (language not implementing this feature)
tests/appsec/api_security/test_endpoints.py::Test_LLM_Endpoint::test_root_has_no_llm_tags: irrelevant (language not implementing this feature)
tests/integrations/test_open_telemetry.py::Test_MsSql::test_obfuscate_query: bug (OTEL-2778)
tests/integrations/test_open_telemetry.py::Test_MySql::test_obfuscate_query: bug (OTEL-2778)
tests/integrations/test_open_telemetry.py::Test_MySql::test_properties: bug (OTEL-2778)
Expand Down
Loading
Loading