Skip to content

test(openfeature): add null targeting key test coverage#5523

Open
leoromanovsky wants to merge 6 commits intomasterfrom
leo.romanovsky/fix-optional-targeting-key
Open

test(openfeature): add null targeting key test coverage#5523
leoromanovsky wants to merge 6 commits intomasterfrom
leo.romanovsky/fix-optional-targeting-key

Conversation

@leoromanovsky
Copy link
Copy Markdown
Contributor

@leoromanovsky leoromanovsky commented Mar 27, 2026

What does this PR do?

Adds test coverage for OpenFeature spec OF.2 (optional targeting key) to prove libdatadog handles nil targeting key correctly via the Ruby FFI layer.

Motivation:

The OpenFeature spec (Requirement 3.1.1, OF.2) defines targeting key as optional. The correct behavior (matching libdatadog's reference implementation) is:

Static (no rules, no shards) Split (shards) Targeted (rules)
"" empty string Returns value Hashes "", returns value Evaluates rules with id="", returns value
null/missing Returns value TARGETING_KEY_MISSING error, returns default Evaluates rules (no id fallback), returns value or falls through

Ruby's evaluator delegates to libdatadog which already handles this correctly — the C extension passes NULL when targeting_key is absent from the context hash. This PR adds explicit test coverage proving the behavior.

Change log entry

No customer-visible change. Test-only — adds coverage for optional targeting key behavior that already works correctly.

Additional Notes:

  • Tests placed in spec/datadog/open_feature/ so @DataDog/feature-flagging-and-experimentation-sdk is auto-requested via CODEOWNERS
  • Tests call Configuration#get_assignment directly to exercise the Rust FFI layer without mocks
  • Used partial shard range (0-4999 of 10000) instead of full range to avoid libdatadog's insignificant shard optimization
  • No logic changes to Ruby code

Question for reviewers: The existing evaluation_engine_spec.rb mocks NativeEvaluator, so there's no integration spec testing the full OpenFeature provider → native evaluator → libdatadog path with real flag configs. Should we add one?

How to test the change?

bundle exec rspec spec/datadog/open_feature/null_targeting_key_spec.rb

3 scenarios:

  1. Static flag + nil targeting key → returns value with STATIC reason
  2. Sharded flag + nil targeting key → returns TARGETING_KEY_MISSING error
  3. Rule-match flag + nil targeting key → returns value with TARGETING_MATCH reason

Next Steps

  • Add test-case-null-targeting-key.json to ffe-system-test-data shared fixtures (separate fixture centralization project)
  • Wire the fixture loop spec to cover null targeting key via JSON fixtures
  • System test coverage for null targeting key across all tracers

…ags_spec

- Add 3 new test cases: static flag (returns value), sharded flag (returns TARGETING_KEY_MISSING), rule-match flag (returns value)
- All 3 scenarios pass empty context hash to simulate nil targeting key
- Confirms Ruby FFI correctly delegates null targeting key handling to libdatadog
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 27, 2026

Thank you for updating Change log entry section 👏

Visited at: 2026-03-27 21:29:50 UTC

@github-actions github-actions bot added the dev/testing Involves testing processes (e.g. RSpec) label Mar 27, 2026
@datadog-datadog-prod-us1
Copy link
Copy Markdown
Contributor

datadog-datadog-prod-us1 bot commented Mar 27, 2026

⚠️ Tests

Fix all issues with BitsAI or with Cursor

⚠️ Other Violations

❄️ 3 New flaky tests detected

    OpenFeature OF.2: Optional Targeting Key with nil targeting key evaluates rule-match flag when rule matches on non-id attribute from rspec   View in Datadog   (Fix with Cursor)

    OpenFeature OF.2: Optional Targeting Key with nil targeting key evaluates static flag successfully from rspec   View in Datadog   (Fix with Cursor)

    OpenFeature OF.2: Optional Targeting Key with nil targeting key returns TARGETING_KEY_MISSING for sharded flag from rspec   View in Datadog   (Fix with Cursor)

ℹ️ Info

No other issues found (see more)

🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 95.43% (+0.08%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b4fc266 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

…lable

Configuration.new(json) is defined by the C extension as a singleton
method. Without the extension, Ruby's default Class#new calls initialize
with 0 args, causing ArgumentError.
@leoromanovsky leoromanovsky marked this pull request as ready for review March 30, 2026 18:34
@leoromanovsky leoromanovsky requested a review from a team as a code owner March 30, 2026 18:34
@pr-commenter
Copy link
Copy Markdown

pr-commenter bot commented Mar 31, 2026

Benchmarks

Benchmark execution time: 2026-04-13 17:24:38

Comparing candidate commit b4fc266 in PR branch leo.romanovsky/fix-optional-targeting-key with baseline commit 5f68ddc in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 1 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Comment thread spec/datadog/open_feature/null_targeting_key_spec.rb Outdated
Copy link
Copy Markdown
Member

@p-datadog p-datadog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question below.

RSpec.describe 'Datadog Provider OF.2: Optional Targeting Key' do
before do
skip 'Requires native libdatadog extension' unless Datadog::Core::FeatureFlags::Configuration.method(:new).arity == 1
end
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question (non-blocking): These tests call the native C extension directly (Configuration.newget_assignment), but spec:open_feature doesn't have a .enhance([:compile]) dependency in the Rakefile — unlike spec:core_with_libdatadog_api (Rakefile:260) which does.

Right now the extension is available because the batch seed happens to place a compile-triggering task (like core_with_libdatadog_api) in the same CI job, and it runs first. A different seed could put open_feature in a batch without one, and the skip guard here would silently skip all three tests.

Would it make sense to add something like this to the Rakefile?

Rake::Task['spec:open_feature'].enhance(["compile:libdatadog_api.#{RUBY_VERSION[/\d+.\d+/]}_#{RUBY_PLATFORM}"])

@p-datadog
Copy link
Copy Markdown
Member

@codex review

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b4fc266283

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


context 'with nil targeting key' do
it 'evaluates static flag successfully' do
result = config.get_assignment('static-flag', :string, {})
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 Badge Cover explicit nil targeting_key in OF.2 tests

These examples are labeled as “nil targeting key” coverage, but they pass contexts without targeting_key ({} / {'email' => ...}), which only exercises the missing key path. This can give false confidence for OF.2 null handling: if the Ruby→FFI context conversion ever diverges between missing and explicit nil, this spec would still pass while the null case regresses. Add at least one call with {'targeting_key' => nil, ...} to assert the advertised behavior.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dev/testing Involves testing processes (e.g. RSpec)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants