Skip to content

fix(openfeature): support custom agentless endpoints#9468

Draft
leoromanovsky wants to merge 3 commits into
masterfrom
leo.romanovsky/ffl-2697-allow-custom-agentless-http
Draft

fix(openfeature): support custom agentless endpoints#9468
leoromanovsky wants to merge 3 commits into
masterfrom
leo.romanovsky/ffl-2697-allow-custom-agentless-http

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Motivation

DataDog/system-tests#7355 enables the Node.js Feature Flagging configuration-source contract starting with 6.5.0. That contract exposed three concrete behaviors missing from Node.js.

The earlier local system-test pass was real; the later failure came from a change to the dd-trace-js branch between the two runs:

  • At 3:31 PM MDT on July 21, the local Docker run used system-tests 51756483a265 and dd-trace-js 94a16dc3017b. Node.js was manually ungated in the manifest, and the FFE suite completed with 53 passed and 18 existing XPASS cases; all 29 configuration-source tests passed.
  • 94a16dc explicitly treated host.docker.internal as a controlled local target. The system-test mock backend was therefore reachable over HTTP from the library container and received the API key.
  • After that green run, the remote dd-trace-js branch advanced to 30338367a818. Its dcdc37f52275 commit replaced the controlled-host exception with a loopback-only HTTPS guard. The same Docker fixture is addressed as http://host.docker.internal:<port>, which is not loopback inside the container, so Node.js threw Feature Flagging agentless URL must use HTTPS unless it targets loopback before making the request.

Docker makes the distinction observable because the host-side mock server cannot be addressed as container loopback. It was not an intermittent Docker failure and the earlier test was not skipped; the regression was introduced by the loopback-only guard after the green SHA.

  • Default agentless delivery points the tracer at a controlled http://host.docker.internal:<port> UFC endpoint, requires an authenticated request to the canonical config path, evaluates the delivered flag, and verifies that Remote Configuration was not selected. Node.js rejected that explicit non-loopback HTTP URL before making the request. This also blocked local laptop development and the dogfooding app when it points at a local proxy.
  • Invalid and reserved source selection accesses the provider with invalid or reserved offline, then requires zero agentless requests and no FFE Remote Configuration capability or product. Node.js allowed invalid input to fall back to the default agentless path instead of throwing before any delivery path was selected.
  • Missing agentless authentication requires the custom endpoint to receive a request without an authentication header, return 401/403, and leave the provider not ready. Node.js rejected the missing API key locally, so the configured backend could not enforce or expose its own authentication behavior.
Local Docker reproduction on the current fixed head

The current PR head was tested against the current system-tests PR head through the real PARAMETRIC Docker topology:

dd-trace-js       6ee9987831336ca15627be1a1e60cc678a5cc9eb
system-tests      a5dc9b8433ee3fd2892ea2968b90acb26274313c
Docker image      nodejs-test-client:latest
image digest      sha256:c2d8906f95547191b994db19f5eca91a82fd97f6ef7fd44d3083ccdf54497fab
Docker server     29.5.2, Ubuntu 24.04.4 LTS, arm64, 8 CPUs
container runtime Linux arm64, Node v18.10.0
dd-trace          7.0.0-pre
FFE provider      @datadog/openfeature-node-server 2.0.1

The image was freshly built with the standard system-tests command:

TEST_LIBRARY=nodejs ./run.sh PARAMETRIC -vv -s \
  tests/parametric/test_ffe/test_configuration_sources.py

The same freshly built image was then run sequentially to remove unrelated local high-worker test-agent port-routing noise:

PYTEST_XDIST_AUTO_NUM_WORKERS=1 TEST_LIBRARY=nodejs \
  ./run.sh PARAMETRIC --skip-parametric-build -vv -s \
  tests/parametric/test_ffe/test_configuration_sources.py

The final JUnit and JSON reports agree:

29 passed in 361.13s
tests=29 failures=0 errors=0 skipped=0
{"passed":29,"total":29,"collected":29}

Selected JUnit cases proving the missing Node.js behaviors now pass:

test_default_agentless_positive[source-absent]                 passed  4.608s
test_invalid_configuration_source_fails_closed                 passed  3.019s
test_reserved_offline_configuration_source_fails_closed        passed  3.028s
test_missing_auth_cold                                         passed 29.636s

Those cases assert the behavior, not merely process liveness:

  • Default agentless performed a request, received HTTP 200, sent authentication to the canonical config path, evaluated the delivered flag, and did not activate FFE Remote Configuration.
  • Invalid and reserved offline sources produced zero agentless requests and no FFE Remote Configuration capability or product after provider access.
  • Missing authentication reached the configured endpoint without an auth header, received 401/403 on the canonical path, and left the provider not ready.

The container logs include the expected runtime and failure-path signals:

DATADOG TRACER CONFIGURATION ... "os_name":"Linux" ... "architecture":"arm64" ...
"version":"7.0.0-pre" ... "lang_version":"18.10.0"
FlaggingProvider created with timeout: 30000ms
FFE Writer enabled - agent has EVP proxy support
Invalid value: 'invalid' for DD_FEATURE_FLAGS_CONFIGURATION_SOURCE (source: env_var)
Feature Flagging agentless endpoint returned HTTP 401; verify DD_API_KEY is configured and valid

The code mounted into the library container was byte-identical to the current PR checkout:

cc089e7f...  package.json                                      host = container
7e2bc546...  packages/dd-trace/src/config/defaults.js          host = container
c3389e23...  packages/dd-trace/src/openfeature/configuration_source.js
                                                               host = container
a656edc1...  system-tests parametric server.js                  host = image

This reproduces the system-tests Dockerfile, library container, test-agent topology, and host.docker.internal boundary locally. The local Docker engine is arm64; GitHub's Linux runner is amd64.

These are delivery-safety requirements, not only test-harness accommodations: an explicit custom endpoint is needed for local development and proxies, while invalid source configuration must never silently select a billed network path.

@datadog/openfeature-node-server 2.0.1 also contains the latest evaluator correction and needs to be consumed before the Node.js tracer release.

Changes

  • Allow an explicitly configured Feature Flagging agentless URL to use HTTP with any hostname.
  • Preserve the API key for that explicit FFE endpoint while retaining the shared transport cleartext protection for unrelated callers.
  • Keep the configuration registry restricted to agentless, remote_config, and reserved offline.
  • Throw when an enabled provider accesses offline or an arbitrary invalid source, before selecting any delivery path.
  • Start the source without DD_API_KEY, omit the header, and let the endpoint report an authentication failure.
  • Update @datadog/openfeature-node-server from 2.0.0 to 2.0.1.
  • Add endpoint, transport, source-lifecycle, configuration-precedence, and integration-matrix regression tests.

Decisions

  • Treat an explicit agentless URL override as an operator-owned trust decision.
  • Keep the default Datadog UFC endpoint on HTTPS.
  • Keep blank and whitespace-only source values semantically absent.
  • Keep the stable Feature Flagging kill switch authoritative; when disabled, provider access remains disabled regardless of the configured source.
  • Fail closed with an exception for explicit unsupported sources instead of silently selecting agentless or Remote Configuration.
  • Apply legacy Remote Configuration compatibility only when the configuration source is absent.

Split scratchpad

This draft preserves the combined investigation and proof while the implementation is split into three separately scoped draft PRs. The configuration-grandfathering PR is stacked on the Docker/custom-endpoint PR; the provider bump is outside that stack and on hold. Do not use this combined branch as the merge path.

Combined state captured here

  • Published combined head: 6ee9987831336ca15627be1a1e60cc678a5cc9eb.
  • Original base: 2003c40d2989f7cd9ed19cfef0f2d3df76ccaf48.
  • System-tests proof head: a5dc9b8433ee3fd2892ea2968b90acb26274313c.
  • Provider package observed inside Docker: @datadog/openfeature-node-server@2.0.1.
  • Current local follow-up changes are intentionally not pushed to this combined branch; they are source material for the split PRs.

Original commit partition

  1. 8a11b63794c6 — custom agentless endpoint and Docker/proxy transport support.
  2. d58eb1dd35ff — explicit invalid/reserved source handling and configuration grandfathering.
  3. 6ee998783133 — provider dependency update from 2.0.0 to 2.0.1.

The first two commits overlap in configuration parsing because they were developed serially. The split branches must reconstruct their final scopes from latest master, not mechanically preserve that accidental stack.

Split PRs

PR Branch Base Scope
Provider bump leo.romanovsky/ffl-2697-openfeature-node-server-2.0.1 latest master package.json, yarn.lock, and generated @datadog/js-core license attribution
Docker/proxy endpoints leo.romanovsky/ffl-2697-custom-agentless-endpoints latest master explicit HTTP custom endpoints, request transport opt-in, custom-endpoint API-key behavior, endpoint/unit tests
Configuration grandfathering leo.romanovsky/ffl-2697-configuration-source-grandfathering custom-endpoint branch (#9481) blank/legacy precedence, invalid and reserved offline fail-closed behavior, registry and integration tests

The configuration change remains separately scoped from endpoint transport behavior, but #9482 is intentionally stacked on #9481 so CI validates the fail-closed contract with the custom host.docker.internal Docker delivery surface.

CI failure captured before splitting

Two root checks failed on the combined head; all-green was only their aggregate:

  • Project / lint: Missing 3rd-party license for @datadog/js-core.
  • Update 3rd-party licenses / check-licenses: generated exactly one missing row:
"@datadog/js-core","https://github.com/DataDog/browser-sdk","['Apache-2.0']","['DataDog']"

That row belongs only in the provider-bump PR. With it applied locally, the full npm run lint command passed, including the license checker, ESLint, CODEOWNERS audit, and exercised-test verification.

Pending review feedback captured before splitting

  • API-key scope: require DD_API_KEY for the default Datadog UFC API while allowing a custom API to start without one. The focused unit tests passed after applying this condition locally.
  • Configuration stability: remove Experimental: from all four new Feature Flagging configuration descriptions.

These threads remain open on this scratch PR. Their code changes belong in the endpoint and configuration split PRs respectively.

Docker proof after both review changes

The combined local tree was mounted into the already-built PARAMETRIC Node.js image and rerun sequentially:

PYTEST_XDIST_AUTO_NUM_WORKERS=1 TEST_LIBRARY=nodejs \
  ./run.sh PARAMETRIC --skip-parametric-build -vv -s \
  tests/parametric/test_ffe/test_configuration_sources.py

Result:

29 passed in 361.35s

This rerun included default custom-endpoint delivery, invalid source, reserved offline, missing-auth cold and warm paths, recovery, ETag handling, timeout behavior, and poller concurrency. The detailed image/runtime/hash proof remains in the collapsible Motivation block above.

Exact earlier-pass/later-failure provenance

  • Earlier green Docker run: system-tests 51756483a265, dd-trace-js 94a16dc3017b, 53 passed plus 18 existing XPASS; configuration sources were 29/29.
  • Later failing dd-trace-js head: 30338367a818.
  • Culprit in that transition: dcdc37f52275, which replaced the explicit host.docker.internal allowance with a loopback-only HTTPS guard.
  • Docker addresses the host-side mock backend as http://host.docker.internal:<port>; that hostname is not container loopback.

Validation allocation for the split

  • Provider bump: dependency resolution, license checks, OpenFeature unit suite.
  • Endpoint support: endpoint/request unit tests plus positive and missing-auth Docker cases.
  • Configuration grandfathering: config registry/unit/integration tests; full Docker configuration-source proof is cross-PR evidence shared with endpoint support.

Split PRs now live

  • #9480 — provider dependency bump, head c60c2700e1abd3b2e603c86663eadb4314230994. This draft remains blocked: 2.0.1 pulls @datadog/js-core from DataDog/browser-sdk, so it must not merge in its current form.
  • #9481 — custom Docker/local-proxy endpoint behavior, head 51cae1bdd2378ecf0f3c77147e2934e33ca2ceb4. This branch is independent on master.
  • #9482 — configuration-source grandfathering and fail-closed behavior, head 3395eccc707ff0b06ea504ae668a80017e6787b4. This PR is stacked on fix(openfeature): allow custom agentless endpoints #9481 so CI evaluates the configuration contract with the Docker/custom-endpoint parent.
  • openfeature-js-client#342 — removes the browser SDK dependency from the Node.js package graph and adds a recursive install-time purity gate to PR CI, head 459352867b471ae9c332fd3ee697321fa2dcdbd5.

The provider-bump draft will be updated with a new commit after a clean upstream patch release exists. The generated @datadog/js-core license row in #9480 is evidence of the bad 2.0.1 graph, not an acceptable long-term Node.js dependency.

@dd-octo-sts

dd-octo-sts Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 7.51 MB
Deduped: 8.17 MB
No deduping: 8.17 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.2 | 124.41 kB | 440.65 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@datadog-official

datadog-official Bot commented Jul 21, 2026

Copy link
Copy Markdown

Pipelines  Tests

⚠️ Warnings

🚦 3 Pipeline jobs failed

All Green | all-green   View in Datadog   GitHub Actions

Project | lint   View in Datadog   GitHub Actions

Update 3rd-party licenses | check-licenses   View in Datadog   GitHub Actions

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 1 passed on retry View in Datadog

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 93.19% (-5.25%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 6ee9987 | Docs | Datadog PR Page | Give us feedback!

@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.40%. Comparing base (2003c40) to head (6ee9987).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9468      +/-   ##
==========================================
- Coverage   96.36%   91.40%   -4.96%     
==========================================
  Files         938      915      -23     
  Lines      126936   123727    -3209     
  Branches    11022     8004    -3018     
==========================================
- Hits       122316   113088    -9228     
- Misses       4620    10639    +6019     
Flag Coverage Δ
aiguard 57.86% <75.00%> (+<0.01%) ⬆️
aiguard-integration 56.23% <41.66%> (+<0.01%) ⬆️
apm-bucket-0 57.72% <75.00%> (+<0.01%) ⬆️
apm-bucket-1 64.02% <75.00%> (-0.01%) ⬇️
apm-bucket-2 62.78% <75.00%> (-0.01%) ⬇️
apm-bucket-3 60.14% <75.00%> (-0.01%) ⬇️
apm-capabilities-tracing 64.47% <100.00%> (-0.02%) ⬇️
apm-integrations-aerospike 56.73% <75.00%> (+<0.01%) ⬆️
apm-integrations-confluentinc-kafka-javascript 61.66% <75.00%> (-0.01%) ⬇️
apm-integrations-couchbase 57.15% <75.00%> (-0.01%) ⬇️
apm-integrations-http 62.81% <75.00%> (-0.01%) ⬇️
apm-integrations-kafkajs ?
apm-integrations-next 59.26% <75.00%> (+<0.01%) ⬆️
apm-integrations-prisma ?
appsec ?
appsec-_fastify_graphql 74.68% <75.00%> (?)
appsec-express_fastify_graphql ?
appsec-integration ?
appsec-kafka_ldapjs_lodash 64.01% <75.00%> (-0.01%) ⬇️
appsec-mongodb-core_mongoose_mysql ?
appsec-mongodb-core_mysql_node-serialize 67.54% <75.00%> (?)
appsec-next 57.58% <75.00%> (+<0.01%) ⬆️
appsec-node-serialize_passport_postgres ?
appsec-passport_sourcing_stripe 64.44% <75.00%> (?)
appsec-sourcing_stripe_template ?
appsec-template 61.17% <75.00%> (?)
debugger ?
instrumentations-bucket-0 51.69% <75.00%> (+<0.01%) ⬆️
instrumentations-bucket-1 60.27% <75.00%> (-0.01%) ⬇️
instrumentations-bucket-10 62.12% <75.00%> (+0.28%) ⬆️
instrumentations-bucket-11 51.73% <75.00%> (-5.02%) ⬇️
instrumentations-bucket-12 52.30% <75.00%> (+0.24%) ⬆️
instrumentations-bucket-13 51.85% <75.00%> (-0.23%) ⬇️
instrumentations-bucket-2 53.38% <75.00%> (+<0.01%) ⬆️
instrumentations-bucket-3 53.67% <75.00%> (+<0.01%) ⬆️
instrumentations-bucket-4 59.27% <75.00%> (+<0.01%) ⬆️
instrumentations-bucket-5 50.09% <75.00%> (+<0.01%) ⬆️
instrumentations-bucket-6 60.98% <75.00%> (-0.01%) ⬇️
instrumentations-bucket-7 58.50% <75.00%> (+0.08%) ⬆️
instrumentations-bucket-8 59.66% <75.00%> (+0.15%) ⬆️
instrumentations-bucket-9 61.65% <75.00%> (+9.58%) ⬆️
instrumentations-instrumentation-couchbase 51.13% <75.00%> (+<0.01%) ⬆️
instrumentations-instrumentation-zlib ?
instrumentations-integration-esbuild 34.11% <8.33%> (-0.01%) ⬇️
llmobs-ai_anthropic_bedrock 62.61% <75.00%> (-0.01%) ⬇️
llmobs-bucket-1 62.72% <75.00%> (+1.14%) ⬆️
llmobs-openai ?
llmobs-sdk 66.24% <75.00%> (-0.01%) ⬇️
master-coverage 91.40% <100.00%> (?)
openfeature 55.94% <100.00%> (-0.27%) ⬇️
openfeature-unit 53.48% <86.95%> (-0.02%) ⬇️
platform-core_esbuild_instrumentations-misc 40.42% <75.00%> (?)
platform-integration ?
platform-shimmer_unit-guardrails_webpack 39.12% <75.00%> (?)
plugins-bucket-1 54.45% <41.66%> (+0.01%) ⬆️
plugins-bucket-10 61.88% <75.00%> (?)
plugins-bucket-15 61.95% <75.00%> (?)
plugins-bucket-18 64.35% <75.00%> (?)
plugins-bucket-4 58.65% <75.00%> (?)
plugins-bullmq_cassandra_cookie 61.84% <75.00%> (-0.01%) ⬇️
plugins-cookie-parser_crypto_dd-trace-api 56.83% <75.00%> (+<0.01%) ⬆️
plugins-fetch_fs_generic-pool 58.86% <75.00%> (+<0.01%) ⬆️
plugins-google-cloud-pubsub_grpc_handlebars ?
plugins-grpc_handlebars_hapi 61.77% <75.00%> (?)
plugins-hapi_hono_ioredis ?
plugins-hono_langgraph_ldapjs 58.09% <75.00%> (?)
plugins-jest_langgraph_ldapjs ?
plugins-light-my-request_limitd-client_lodash 58.70% <75.00%> (-0.02%) ⬇️
plugins-mariadb_memcached_mercurius ?
plugins-mongodb-core_multer_mysql 58.97% <75.00%> (?)
plugins-mongodb_mongodb-core_mongoose ?
plugins-multer_mysql_mysql2 ?
plugins-mysql2_nats_node-serialize 60.80% <75.00%> (?)
plugins-nats_node-serialize_opensearch ?
plugins-opensearch_passport-http_postgres 60.30% <75.00%> (?)
plugins-passport-http_pino_postgres ?
plugins-process_pug_redis 57.76% <75.00%> (+0.01%) ⬆️
plugins-undici_url_valkey 58.63% <75.00%> (+<0.01%) ⬆️
plugins-vm_winston_ws ?
profiling 62.06% <100.00%> (-0.19%) ⬇️
serverless-aws-sdk-aws-sdk 55.15% <75.00%> (+<0.01%) ⬆️
serverless-aws-sdk-bedrockruntime ?
serverless-aws-sdk-client ?
serverless-aws-sdk-dynamodb ?
serverless-aws-sdk-eventbridge 49.46% <75.00%> (+<0.01%) ⬆️
serverless-aws-sdk-kinesis ?
serverless-aws-sdk-lambda 57.48% <75.00%> (+<0.01%) ⬆️
serverless-aws-sdk-s3 55.69% <75.00%> (?)
serverless-aws-sdk-sns 60.23% <75.00%> (-0.01%) ⬇️
serverless-aws-sdk-sqs ?
serverless-aws-sdk-stepfunctions ?
serverless-aws-sdk-util 51.58% <75.00%> (+<0.01%) ⬆️
serverless-azure-durable-functions_lambda 58.73% <100.00%> (?)
serverless-bucket-0 58.20% <75.00%> (+3.87%) ⬆️
serverless-bucket-1 62.09% <75.00%> (+2.74%) ⬆️
test-bucket-0 64.69% <41.66%> (?)
test-optimization-cucumber ?
test-optimization-cypress 61.46% <41.66%> (-4.28%) ⬇️
test-optimization-jest ?
test-optimization-mocha ?
test-optimization-playwright-playwright-atr 59.98% <41.66%> (-0.44%) ⬇️
test-optimization-playwright-playwright-efd ?
test-optimization-playwright-playwright-final-status 60.07% <41.66%> (-0.51%) ⬇️
test-optimization-playwright-playwright-impacted-tests ?
test-optimization-playwright-playwright-reporting ?
test-optimization-playwright-playwright-test-management ?
test-optimization-playwright-playwright-test-span 60.22% <41.66%> (-0.37%) ⬇️
test-optimization-selenium 60.03% <41.66%> (-0.11%) ⬇️
test-optimization-testopt ?
test-optimization-vitest 65.73% <41.66%> (-4.56%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pr-commenter

pr-commenter Bot commented Jul 21, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-21 23:45:45

Comparing candidate commit 6ee9987 in PR branch leo.romanovsky/ffl-2697-allow-custom-agentless-http with baseline commit 2003c40 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2313 metrics, 45 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-24

  • unstable execution_time [-211.390ms; +207.036ms] or [-7.982%; +7.818%]

scenario:appsec-appsec-enabled-26

  • unstable execution_time [-240.963ms; +237.871ms] or [-9.446%; +9.325%]

scenario:appsec-appsec-enabled-with-attacks-24

  • unstable execution_time [-166.227ms; +158.017ms] or [-5.300%; +5.039%]

scenario:appsec-appsec-enabled-with-attacks-26

  • unstable execution_time [-188968.979µs; +187421.612µs] or [-6.485%; +6.432%]

scenario:appsec-control-20

  • unstable execution_time [-116.681ms; +126.522ms] or [-7.120%; +7.721%]

scenario:appsec-control-24

  • unstable execution_time [-113.287ms; +115.798ms] or [-9.125%; +9.327%]

scenario:appsec-control-26

  • unstable execution_time [-126.547ms; +128.788ms] or [-10.185%; +10.365%]

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-20

  • unstable execution_time [-25.974ms; +30.448ms] or [-4.742%; +5.559%]

scenario:debugger-line-probe-with-snapshot-default-26

  • unstable cpu_user_time [-2249.190ms; +755.294ms] or [-23.497%; +7.891%]
  • unstable execution_time [-2273.336ms; +747.897ms] or [-22.081%; +7.264%]
  • unstable instructions [-20.3G instructions; +6.7G instructions] or [-25.427%; +8.382%]
  • unstable throughput [-150.424op/s; +446.468op/s] or [-4.676%; +13.877%]

scenario:debugger-line-probe-with-snapshot-minimal-24

  • unstable cpu_user_time [-2705.891ms; +2727.910ms] or [-30.541%; +30.790%]
  • unstable execution_time [-2752.919ms; +2783.652ms] or [-28.809%; +29.130%]
  • unstable instructions [-23741.2M instructions; +23939.5M instructions] or [-32.635%; +32.908%]
  • unstable max_rss_usage [-11766.050KB; +11590.050KB] or [-7.375%; +7.265%]
  • unstable throughput [-751.454op/s; +730.123op/s] or [-21.365%; +20.759%]

scenario:debugger-line-probe-with-snapshot-minimal-26

  • unstable cpu_user_time [-2322.825ms; +774.416ms] or [-24.334%; +8.113%]
  • unstable execution_time [-2426.159ms; +838.074ms] or [-23.609%; +8.155%]
  • unstable instructions [-20.5G instructions; +6.5G instructions] or [-25.737%; +8.113%]
  • unstable throughput [-168.404op/s; +460.155op/s] or [-5.202%; +14.213%]

scenario:debugger-line-probe-without-snapshot-26

  • unstable cpu_user_time [-2593.884ms; +4120.830ms] or [-27.110%; +43.069%]
  • unstable execution_time [-2560.832ms; +4132.401ms] or [-24.876%; +40.143%]
  • unstable instructions [-23.3G instructions; +37.0G instructions] or [-29.191%; +46.378%]
  • unstable max_rss_usage [-8.673MB; +13.583MB] or [-5.436%; +8.514%]
  • unstable throughput [-813.229op/s; +499.760op/s] or [-25.279%; +15.535%]

scenario:dogstatsd-with-tags-20

  • unstable cpu_user_time [-380.060ms; +247.459ms] or [-8.184%; +5.329%]
  • unstable execution_time [-382.891ms; +243.711ms] or [-8.109%; +5.161%]
  • unstable throughput [-83918.304op/s; +136853.649op/s] or [-4.727%; +7.709%]

scenario:plugin-claude-agent-sdk-compact-stream-scan-24

  • unstable cpu_usage_percentage [-5.510%; +4.515%]
  • unstable cpu_user_time [-3836.098µs; +2421.757µs] or [-6.261%; +3.953%]

scenario:plugin-claude-agent-sdk-compact-stream-scan-26

  • unstable cpu_usage_percentage [-4.965%; +5.527%]

scenario:plugin-couchbase-upsert-24

  • unstable cpu_user_time [-234.376ms; +372.849ms] or [-5.145%; +8.184%]
  • unstable execution_time [-232.256ms; +372.743ms] or [-5.085%; +8.162%]

scenario:plugin-graphql-long-with-depth-and-collapse-off-20

  • unstable max_rss_usage [-33.367MB; +44.594MB] or [-8.342%; +11.149%]

scenario:plugin-graphql-long-with-depth-off-20

  • unstable max_rss_usage [-7.307MB; +10.173MB] or [-5.632%; +7.842%]

scenario:plugin-graphql-long-with-depth-on-max-20

  • unstable cpu_user_time [-598.040ms; +583.895ms] or [-5.178%; +5.055%]
  • unstable execution_time [-612713.824µs; +611909.224µs] or [-5.190%; +5.183%]
  • unstable throughput [-3.559op/s; +3.564op/s] or [-5.220%; +5.227%]

scenario:plugin-mongodb-core-plain-find-26

  • unstable execution_time [-102.499ms; +140.966ms] or [-5.027%; +6.914%]

scenario:plugin-pg-service-26

  • unstable cpu_usage_percentage [-6.232%; +5.218%]
  • unstable execution_time [-66.314ms; +88.183ms] or [-7.239%; +9.626%]
  • unstable throughput [-509244.291op/s; +371507.502op/s] or [-7.650%; +5.581%]

scenario:test-optimization-wide-tags-24

  • unstable cpu_user_time [-208.543ms; +258.624ms] or [-5.037%; +6.246%]
  • unstable execution_time [-208.539ms; +258.086ms] or [-4.994%; +6.181%]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant