[ruby] Enable FFE flag evaluation metrics tests [ruby@sameerank/FFL-1945/add-flag-eval-metrics]

[sameerank]

Motivation

Enable Ruby FFE (Feature Flagging and Experimentation) flag evaluation metrics tests for rails72 weblog, testing against dd-trace-rb PR DataDog/dd-trace-rb#5599.

Changes

• Add rails72 declaration for test_flag_eval_metrics.py in manifests/ruby.yml
• Add OTel metrics SDK dependencies to Gemfile (opentelemetry-metrics-sdk, opentelemetry-exporter-otlp-metrics)
• Upgrade openfeature-sdk to ~> 0.5 for provider hooks support
• Add OTel metrics flush to /flush endpoint for proper test timing

Notes

• Prod CI failures are expected until dd-trace-rb PR 5599 is merged and released. The released gem doesn't have FFE metrics support yet.
• Once dd-trace-rb is released with FFE metrics, remove [ruby@sameerank/FFL-1945/add-flag-eval-metrics] from the PR title.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on your PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed? -> No, existing scenario

[github-actions]

CODEOWNERS have been resolved as:

manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
utils/build/docker/ruby/rails72/Gemfile                                 @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails72/app/controllers/internal_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core

[datadog-prod-us1-6]

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 17 Tests failed

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing.test_ffe_eval_config_exists_flag_missing[rails72] from system_tests_suite     (Fix with Cursor)

AssertionError: Expected metric for non-existent flag 'non-existent-eval-metric-flag', found none. All: []
assert 0 > 0
 +  where 0 = len([])

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing object at 0x7ff58dff5460>

    def test_ffe_eval_config_exists_flag_missing(self):
        """Test that missing flag (with config loaded) produces error.type=flag_not_found."""
        assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}"
    
...

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Lowercase_Consistency.test_ffe_lowercase_error_type[rails72] from system_tests_suite     (Fix with Cursor)

AssertionError: Expected metric for flag 'lowercase-error-flag', found none.
assert 0 > 0
 +  where 0 = len([])

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Lowercase_Consistency object at 0x7ff58dff5be0>

    def test_ffe_lowercase_error_type(self):
        """Test that error.type values are lowercase."""
        assert self.r_error.status_code == 200, f"Flag evaluation request failed: {self.r_error.text}"
    
...

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Lowercase_Consistency.test_ffe_lowercase_reason[rails72] from system_tests_suite     (Fix with Cursor)

AssertionError: Expected metric for flag 'lowercase-test-flag', found none.
assert 0 > 0
 +  where 0 = len([])

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Lowercase_Consistency object at 0x7ff58dff5b20>

    def test_ffe_lowercase_reason(self):
        """Test that reason values are lowercase."""
        assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}"
    
...

View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 10bb760 | Docs | Datadog PR Page | Give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 10bb7608c8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Constrain openfeature-sdk to 0.5.1-compatible versions

openfeature-sdk is declared as ~> 0.5, but this range also allows 0.6.x (and newer <1.0) releases. Since this change explicitly depends on provider hooks support from 0.5.1+, allowing 0.6.x can pull in a different API line during image builds and break the Datadog OpenFeature integration used by the FFE metrics tests. Pin this to the intended compatibility window (for example ~> 0.5.1 or >= 0.5.1, < 0.6).

Useful? React with 👍 / 👎.

[sameerank]

This matches the upper bound that was previously in place (up to < 1.0), and the provided justification for limiting the minor version to < 0.6 contradicts our passing tests which use the latest available openfeature-sdk 0.6.5 version

https://github.com/DataDog/system-tests/actions/runs/24794277678/job/72560160287

Look for Using openfeature-sdk 0.6.5 in the "Build weblog" step during the bundle install output.

[Strech]

I would suggest to surface exception as it will be better to see, this is testing app and suppose to highlight issues.

[Strech]

What is the goal of this check?