Add agentless Feature Flagging configuration source

[leoromanovsky]

Motivation

Java Feature Flagging needs a tracer-side configuration-source split so SDKs can fetch UFC from an agentless HTTP backend while preserving the existing Agent Remote Configuration path.

Changes

• Adds DD_FEATURE_FLAGS_CONFIGURATION_SOURCE with agentless, remote_config, and reserved offline; default is agentless.
• Adds agentless endpoint selection through DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL.
• Adds agentless polling controls: DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDS and DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS.
• Adds AgentlessConfigurationSource, an HTTP UFC poller used for both the default Datadog-managed endpoint and a configured custom HTTP endpoint.
• Keeps remote_config on the existing RemoteConfigServiceImpl / Agent RC path.
• Handles DD-API-KEY, accepted-200-only ETags/304, malformed UFC rejection, last-known-good preservation, and no overlapping polls.
• Retries timeout/429/5xx failures twice per poll cycle with jittered delays derived from the poll period: clamp(P/6, 2s, 10s) then clamp(P/3, 5s, 30s).
• Cancels in-flight HTTP and pending retry work during shutdown, suppresses post-shutdown retries, and makes subsystem startup idempotent.
• Rejects nonpositive poll intervals and request timeouts by warning and using the documented defaults.
• Removes the proposed DD_FEATURE_FLAGS_ENABLED and extra-header env from this PR.

Decisions

No new provider kill switch. This keeps the existing provider bootstrap behavior through DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED; changing enablement is out of scope for this PR.

Custom HTTP delivery remains under agentless. The source mode describes direct HTTP UFC delivery without the Datadog Agent. With no base URL, the SDK derives the first-party Datadog endpoint as https://api.<site>/api/v2/feature-flagging/config/server-distribution?dd_env=<env>. Setting DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL chooses a system-test, dogfood, or operator-managed HTTP endpoint without introducing a fourth source mode. A bare host uses the standard server-distribution path; a URL with a path is used as the exact UFC endpoint.

Remote Configuration remains a distinct mode. It is Agent-mediated and uses the RC protocol's security, signing, targeting, capability, and subscription lifecycle. Those semantics are materially different from direct HTTP polling, so remote_config remains a peer source mode rather than another agentless endpoint choice.

No offline factory yet. offline remains a reserved configuration-source value. This PR does not expose a startup-bytes API or offline factory because that API still needs design work.

Source selection stays SDK-local. DD_FEATURE_FLAGS_CONFIGURATION_SOURCE decides which local source implementation starts. It is not sent to the backend.

Initialization Modes

Solid connectors are implemented in this PR. The dashed offline connector previews startup-provided UFC bytes.

flowchart TD
    Gate{DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED}
    Disabled[Feature Flagging subsystem not started]
    System[FeatureFlaggingSystem.start]
    Select{DD_FEATURE_FLAGS_CONFIGURATION_SOURCE}
    Exposure[ExposureWriter starts for every enabled mode]

    subgraph AgentlessMode[agentless mode - current default]
        AgentlessSource[AgentlessConfigurationSource]
        OperationalDefaults[SDK-owned operation<br/>30s poll default; 2s request timeout default<br/>ETag; retry; last-known-good; no overlap]
        EndpointChoice{agentless base URL set?}
        DatadogManaged[Datadog-managed agentless<br/>first-party; CDN-backed]
        CustomHttp[Custom HTTP endpoint<br/>system tests; dogfood; operator backend]

        AgentlessSource --- OperationalDefaults
        AgentlessSource -- polls --> EndpointChoice
        EndpointChoice -- no --> DatadogManaged
        EndpointChoice -- yes --> CustomHttp
    end

    RemoteConfig[remote_config<br/>explicit opt-in; Agent-mediated]
    RemoteConfigSource[RemoteConfigServiceImpl]
    AgentRc[Datadog Agent Remote Configuration<br/>RC security; signing; targeting; subscriptions]

    OfflineReserved[offline today<br/>reserved; no configuration service]
    OfflineSource[Later: OfflineConfigurationSource<br/>customer UFC JSON bytes at startup; no network]

    Ufc[Shared UFC deserialize and evaluate pipeline]
    Gateway[FeatureFlaggingGateway]
    Provider[OpenFeature provider]

    Gate -- false --> Disabled
    Gate -- true --> System
    System --> Select
    System --> Exposure

    Select -- unset or agentless --> AgentlessSource
    DatadogManaged --> Ufc
    CustomHttp --> Ufc

    Select -- remote_config --> RemoteConfig
    RemoteConfig --> RemoteConfigSource
    RemoteConfigSource -- subscribes --> AgentRc
    RemoteConfigSource --> Ufc

    Select -- offline today --> OfflineReserved
    OfflineReserved -. later .-> OfflineSource
    OfflineSource -. startup UFC bytes .-> Ufc

    Ufc --> Gateway
    Gateway --> Provider

Loading

Customer Usage Example

Default Datadog-managed agentless delivery:

export DD_API_KEY=<datadog-api-key>
export DD_SITE=datadoghq.com
export DD_ENV=prod

# Optional because agentless is the default configuration source.
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=agentless
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDS=30
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS=2

java -javaagent:/path/to/dd-java-agent.jar -jar app.jar

Custom HTTP delivery keeps agentless mode and changes only the endpoint:

export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=agentless
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL=https://flags.example.com/ufc

java -javaagent:/path/to/dd-java-agent.jar -jar app.jar

Explicit Agent Remote Configuration delivery still uses the existing Agent path:

export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=remote_config
export DD_REMOTE_CONFIGURATION_ENABLED=true

java -javaagent:/path/to/dd-java-agent.jar -jar app.jar

Verification

./gradlew :products:feature-flagging:feature-flagging-agent:test :products:feature-flagging:feature-flagging-lib:test --no-daemon
result: BUILD SUCCESSFUL

./gradlew :products:feature-flagging:feature-flagging-agent:jacocoTestCoverageVerification :products:feature-flagging:feature-flagging-lib:jacocoTestCoverageVerification :products:feature-flagging:feature-flagging-agent:spotlessCheck :products:feature-flagging:feature-flagging-lib:spotlessCheck --no-daemon
result: BUILD SUCCESSFUL

./gradlew :dd-smoke-tests:openfeature:cleanTest :dd-smoke-tests:openfeature:test --tests datadog.smoketest.springboot.OpenFeatureProviderSmokeTest --no-daemon
result: BUILD SUCCESSFUL

git diff --check
result: clean

System Test Evidence

I built dd-java-agent, dd-trace-api, and dd-openfeature from the rewritten
head f9e70e67c4, rebuilt the Java Spring Boot weblog image with those local
artifacts, and verified the JAR inside the image matched the local agent JAR.
I then ran the Java manifest-enabled agentless scenario from the companion
system-tests draft:

TEST_LIBRARY=java ./run.sh +v \
  FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS \
  tests/ffe/test_agentless_configuration.py

Scenario: FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS
Library: java@1.65.0-SNAPSHOT+f9e70e67c4
Weblog variant: spring-boot
collected 1 item
tests/ffe/test_agentless_configuration.py .                        [100%]
1 passed

This exercises the Java manifest row proposed by the companion draft against
the exact code currently published in this PR.

Next Steps

• Review the draft Java manifest activation:
  DataDog/system-tests#7300.
• Keep that PR draft until this Java behavior is available to system-tests CI.
• After this PR lands, rerun the manifest activation in Java development CI and
  merge the activation only when that gate is green.

[datadog-datadog-prod-us1-2]

🎯 Code Coverage (details)
• Patch Coverage: 80.12%
• Overall Coverage: 57.67% (+0.08%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 6df48d1 | Docs | Datadog PR Page | Give us feedback!}

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.01 s	14.00 s	[-0.7%; +0.8%] (no difference)
startup:insecure-bank:tracing:Agent	12.86 s	12.95 s	[-1.8%; +0.4%] (no difference)
startup:petclinic:appsec:Agent	16.88 s	16.66 s	[+0.1%; +2.6%] (maybe worse)
startup:petclinic:iast:Agent	16.87 s	16.50 s	[-2.3%; +6.8%] (no difference)
startup:petclinic:profiling:Agent	16.49 s	16.17 s	[-2.5%; +6.5%] (no difference)
startup:petclinic:sca:Agent	16.80 s	16.46 s	[-2.1%; +6.2%] (no difference)
startup:petclinic:tracing:Agent	16.18 s	16.09 s	[-0.2%; +1.4%] (no difference)

Commit: 488b4770 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[dd-octo-sts]

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

• Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

[datadog-datadog-prod-us1-2]

All retry, ETag, shutdown, and idempotency paths look correct. One real gap: apply() silently returns false for 401/403 with no log above DEBUG — users with a wrong or missing DD_API_KEY have no way to diagnose why feature flags never load. Fixed by separating the auth-failure branch in apply() with a LOGGER.warn(...) call.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit f9e70e6 · What is Autotest? · Any feedback? Reach out in #autotest}

[pavlokhrebto]

Advisory review of the agentless configuration source, cross-checked against the Agentless-mode RFC. The core state machine (cold vs. warm handling, retry classification, readiness, atomic snapshot swap, shutdown / last-known-good) matches the RFC precisely — nice work. Findings below are precision-first and mostly non-blocking; inline comments are attached to the relevant lines.

Descoped config note (C8): The RFC lists DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_EXTRA_HEADERS (default empty object); this PR intentionally removes it. Fine if deliberate, but noting it since the mock-CDN test flow may want custom headers — worth tracking as a follow-up.

[leoromanovsky]

Contract update from the staging CDN response: the managed agentless endpoint does not return raw UFC at the top level. The review follow-up changes the managed response contract to:

GET https://ufc-server.ff-cdn.<site>/api/v2/feature-flagging/config/rules-based/server?dd_env=<environment>
DD-API-KEY: <api key>

{"data":{"id":"1","type":"universal-flag-configuration","attributes":{"createdAt":"...","environment":{"name":"Staging"},"flags":{}}}}

The Java implementation requires data.type == universal-flag-configuration, unwraps data.attributes, and feeds those attributes through the existing UFC parser. Missing or mismatched envelopes are rejected without replacing last-known-good state. Explicitly configured compatibility endpoints retain the raw-UFC fallback.

The matching system-tests fixture change is DataDog/system-tests#7315. It wraps the existing raw fixture only at the agentless HTTP boundary, so RC fixture consumers are unchanged.

Local system-tests proof:

./run.sh TEST_THE_TEST tests/test_the_test/test_mock_ffe_agentless_backend.py
3 passed in 2.32s

The self-tests verify the canonical route, exact DD-API-KEY auth behavior, JSON:API resource type and UFC attributes, metadata-only status, and host-gateway mapping.

[sarahchen6]

Two final comments - I think both are worth addressing, but definitely the internal-api dependency refinement. Otherwise, looks good! Thanks for all the iterations. Pre-approving in the meantime!

[sarahchen6]

IIUC if a typo / invalid value is set as config source, the Feature Flagging subsystem would not start and just log an IllegalArgumentException: https://github.com/DataDog/dd-trace-java/pull/11892/changes#diff-a2718be99f97d918aebd130483af618925dd3121c802f00eed3fe9c3af712dfeR122. Instead we should probably catch invalid values here immediately and fall-back to the default agentless setup with a warning printed in logs. We could possibly leverage ConfigProvider.getEnum to do this. WDYT?

[sarahchen6]

instead of re-implementing an HTTP retry policy, could we re-use the existing OkHttpUtils.sendWithRetries/HttpRetryPolicy from the :communication module? the implementation differs slightly, for example HttpURLConnection.HTTP_CLIENT_TIMEOUT errors are not retried, but ideally we can leverage existing tools in the repo

[sarahchen6]

For context - ideally products avoid developing their own product capabilities that can be shared across a platform, such as with HTTP retries and HTTP interactions in general. It'd be a pain to maintain these implementations individually. If there's something lacking with the current :communication module HTTP implementation, this would be a good callout so that we (LP team) can update the tool for your use.

Since you mentioned this PR is waiting on a few env var changes anyway, this would be a great opportunity to either use the existing :communication module retry as-is or I can update the tool to better fit if needed!

[leoromanovsky]

Review update

This update applies the decisions from this review and from the cross-SDK review.

Changes

• The agent does not start Agentless polling during tracer startup. It waits until application code initializes the OpenFeature provider. The first poll cycle completes during provider activation. Later polls use a fixed delay.
• An invalid or unsupported configuration source does not cause an exception. The agent disables the provider and writes a warning.
• The request timeout default is now 5 seconds.
• Agentless requests use the shared OkHttpUtils.sendWithRetries and HttpRetryPolicy implementation.
• The client makes a maximum of three attempts. It uses the same retry ranges and jitter as the Node.js implementation.
• A custom endpoint can use HTTP for local development. The client does not send the Datadog API key to a custom endpoint.
• The client sends the Datadog API key only to the default Datadog HTTPS endpoint.
• HTTP 401 and 403 responses write an authentication warning.

System-test evidence

I built these artifacts from Java commit 6df48d1cfd:

• dd-java-agent-1.65.0-SNAPSHOT.jar
• dd-trace-api-1.65.0-SNAPSHOT.jar
• dd-openfeature-1.65.0-SNAPSHOT.jar

I used system-tests main at 03f0c053c. I removed the Java missing_feature waiver locally. I rebuilt the parametric image with the three local artifacts. I then ran the complete configuration-source test file.

PYTEST_XDIST_AUTO_NUM_WORKERS=1 TEST_LIBRARY=java \
  ./run.sh PARAMETRIC --skip-parametric-build \
  tests/parametric/test_ffe/test_configuration_sources.py

29 passed in 290.19s

The affected Java tests and formatting checks also passed:

BUILD SUCCESSFUL in 29s
298 actionable tasks: 7 executed, 291 up-to-date

Node.js alignment

• Unsupported sources disable the provider in Java and Node.js.
• Agentless polling starts when application code initializes the provider in Java and Node.js.
• Both clients use three attempts, the same retry ranges, and 20 percent jitter. See Java and Node.js.
• Both clients allow custom HTTP endpoints and omit the API key for those endpoints. See Java and Node.js.
• Both default endpoints use HTTPS. See Java and Node.js.
• Both clients use a 5-second request timeout by default. See Java and Node.js.

The Node.js links use the current head of #9482. That PR includes the endpoint behavior from merged #9481.