From 88ad2844f6256b4781508d45352a1c810daa951f Mon Sep 17 00:00:00 2001 From: typotter Date: Wed, 22 Apr 2026 00:28:32 -0600 Subject: [PATCH 01/14] feat: new ffe tests for reason/evaluation details correctness --- manifests/java.yml | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/manifests/java.yml b/manifests/java.yml index 97093384bdb..f211d570318 100644 --- a/manifests/java.yml +++ b/manifests/java.yml @@ -3153,6 +3153,40 @@ manifest: tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: bug (FFL-1972) tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Targeting::test_ffe_eval_reason_targeting: bug (FFL-1972) tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional::test_ffe_eval_targeting_key_optional: bug (FFL-1972) + # REASON-3, REASON-7, REASON-8, REASON-10, REASON-13, REASON-19, REASON-22, REASON-25 pass against Java ≥ v1.61.0. + # The remaining 12 tests cover RFC Appendix B paths not yet implemented in Java: + # REASON-2 PROVIDER_FATAL (RC 401 not treated as fatal by Java's FFE provider) + # REASON-9 ADR-001: empty waterfall → DEFAULT (Java returns error/general instead) + # REASON-11 STATIC (Java returns targeting_match for any matched allocation) + # REASON-14 DEFAULT via unconditional alloc (Java returns targeting_match) + # REASON-16 DEFAULT via shard-miss (Java returns targeting_match for default-alloc) + # REASON-17 SPLIT overrides TARGETING_MATCH (ADR-004 not implemented in Java) + # REASON-18 DEFAULT via rule-fail (Java returns targeting_match for default-alloc) + # REASON-20 DEFAULT via split-miss + rule-fail (Java returns targeting_match) + # REASON-21 DEFAULT with active window (Java returns targeting_match) + # REASON-23 DEFAULT multi-alloc active window (Java returns targeting_match) + # REASON-24 DEFAULT multi-alloc inactive window (Java returns targeting_match) + # REASON-26 DEFAULT via window+rule-fail (Java returns targeting_match) + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_2_ProviderFatal::test_ffe_reason_2_provider_fatal: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_3_DisabledFlagNotFound::test_ffe_reason_3_disabled_flag_not_found: v1.61.0 + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_7_TargetingKeyMissing::test_ffe_reason_7_targeting_key_missing: v1.61.0 + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_8_RuleOnlyNoKey::test_ffe_reason_8_rule_only_no_key: v1.61.0 + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_9_ZeroAllocations::test_ffe_reason_9_zero_allocations: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_10_NoDefaultAlloc::test_ffe_reason_10_no_default_alloc: v1.61.0 + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_11_StaticNoSplit::test_ffe_reason_11_static_no_split: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_13_MultiAllocRuleMatch::test_ffe_reason_13_multi_alloc_rule_match: v1.61.0 + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_14_MultiAllocRuleFail::test_ffe_reason_14_multi_alloc_rule_fail: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_16_RulePassShardMiss::test_ffe_reason_16_rule_pass_shard_miss: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_17_RulePassShardWin::test_ffe_reason_17_rule_pass_shard_win: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_18_RuleFail::test_ffe_reason_18_rule_fail: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_19_SplitMissRuleMatch::test_ffe_reason_19_split_miss_rule_match: v1.61.0 + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_20_SplitMissDefault::test_ffe_reason_20_split_miss_default: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_21_ActiveWindowSingle::test_ffe_reason_21_active_window_single: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_22_InactiveWindowSingle::test_ffe_reason_22_inactive_window_single: v1.61.0 + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_23_ActiveWindowMulti::test_ffe_reason_23_active_window_multi: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_24_InactiveWindowMulti::test_ffe_reason_24_inactive_window_multi: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_25_WindowActiveRuleMatch::test_ffe_reason_25_window_active_rule_match: v1.61.0 + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_26_WindowRuleFail::test_ffe_reason_26_window_rule_fail: missing_feature tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234) tests/integration_frameworks/llm/openai/test_openai_apm.py: v1.61.0 tests/integration_frameworks/llm/openai/test_openai_llmobs.py: v1.61.0 From 30ede5b644cfc03b7f2c6f42538867e5fe9b7049 Mon Sep 17 00:00:00 2001 From: typotter Date: Wed, 22 Apr 2026 07:55:55 -0600 Subject: [PATCH 02/14] feat: add RFC Appendix B reason/evaluation-detail system tests for Java FFE MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds test_flag_eval_reasons.py covering REASON-2 through REASON-26 from RFC Appendix B (evaluation reason + error code correctness). 8 tests pass against Java ≥ v1.61.0; 12 are marked missing_feature pending Java RFC compliance work. Updates java.yml: replaces file-level missing_feature (which defeated per-test version pins) with 20 individual entries. Fixes stale class names from pre-rename. Upgrades several test_flag_eval_metrics.py entries from bug→v1.61.0 now that the underlying Java issues are resolved. --- manifests/java.yml | 22 +- tests/ffe/test_flag_eval_reasons.py | 1466 +++++++++++++++++++++++++++ 2 files changed, 1477 insertions(+), 11 deletions(-) create mode 100644 tests/ffe/test_flag_eval_reasons.py diff --git a/manifests/java.yml b/manifests/java.yml index f211d570318..11cfa0fbc50 100644 --- a/manifests/java.yml +++ b/manifests/java.yml @@ -3134,25 +3134,25 @@ manifest: spring-boot: v1.56.0 tests/ffe/test_exposures.py::Test_FFE_EXP_5_Missing_Targeting_Key: bug (FFL-1729) tests/ffe/test_flag_eval_metrics.py: missing_feature (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Config_Exists_Flag_Missing::test_ffe_eval_config_exists_flag_missing: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Lowercase_Consistency::test_ffe_lowercase_error_type: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Lowercase_Consistency::test_ffe_lowercase_reason: bug (FFL-1972) + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Config_Exists_Flag_Missing::test_ffe_eval_config_exists_flag_missing: v1.61.0 + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Lowercase_Consistency::test_ffe_lowercase_error_type: v1.61.0 + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Lowercase_Consistency::test_ffe_lowercase_reason: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Basic::test_ffe_eval_metric_basic: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Count::test_ffe_eval_metric_count: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Different_Flags::test_ffe_eval_metric_different_flags: bug (FFL-1972) + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Count::test_ffe_eval_metric_count: v1.61.0 + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Different_Flags::test_ffe_eval_metric_different_flags: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Numeric_To_Integer::test_ffe_eval_metric_numeric_to_integer: bug (FFL-1972) ? tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex::test_ffe_eval_metric_parse_error_invalid_regex : bug (FFL-1972) ? tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Variant_Type_Mismatch::test_ffe_eval_metric_parse_error_variant_type_mismatch : bug (FFL-1972) tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Type_Mismatch::test_ffe_eval_metric_type_mismatch: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Nested_Attributes_Ignored::test_ffe_eval_nested_attributes_ignored: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_No_Config_Loaded::test_ffe_eval_no_config_loaded: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Default::test_ffe_eval_reason_default: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Disabled::test_ffe_eval_reason_disabled: bug (FFL-1972) + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Nested_Attributes_Ignored::test_ffe_eval_nested_attributes_ignored: v1.61.0 + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_No_Config_Loaded::test_ffe_eval_no_config_loaded: v1.61.0 + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Default::test_ffe_eval_reason_default: v1.61.0 + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Disabled::test_ffe_eval_reason_disabled: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Targeting::test_ffe_eval_reason_targeting: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional::test_ffe_eval_targeting_key_optional: bug (FFL-1972) + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Targeting::test_ffe_eval_reason_targeting: v1.61.0 + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional::test_ffe_eval_targeting_key_optional: v1.61.0 # REASON-3, REASON-7, REASON-8, REASON-10, REASON-13, REASON-19, REASON-22, REASON-25 pass against Java ≥ v1.61.0. # The remaining 12 tests cover RFC Appendix B paths not yet implemented in Java: # REASON-2 PROVIDER_FATAL (RC 401 not treated as fatal by Java's FFE provider) diff --git a/tests/ffe/test_flag_eval_reasons.py b/tests/ffe/test_flag_eval_reasons.py new file mode 100644 index 00000000000..dde595d2f3b --- /dev/null +++ b/tests/ffe/test_flag_eval_reasons.py @@ -0,0 +1,1466 @@ +"""Tests for FFE evaluation reasons and error codes per Appendix B of the RFC. + +Reference: Feature Flag Evaluation Reasons and Error Codes RFC, Appendix B +https://datadoghq.atlassian.net/... + +Appendix B defines 26 test cases covering all valid (return value type, reason, error code) +combinations. This file covers the cases not already exercised in test_flag_eval_metrics.py. + +Coverage map (B-N = Appendix B row N): + REASON-1 PROVIDER_NOT_READY → test_flag_eval_metrics.py :: Test_FFE_Eval_No_Config_Loaded + REASON-2 PROVIDER_FATAL → Test_FFE_REASON_2_ProviderFatal (below) + REASON-3 FLAG_NOT_FOUND (missing) → test_flag_eval_metrics.py :: Test_FFE_Eval_Config_Exists_Flag_Missing + REASON-3 FLAG_NOT_FOUND (disabled) → Test_FFE_REASON_3_DisabledFlagNotFound (below) + REASON-4 TYPE_MISMATCH → test_flag_eval_metrics.py :: Test_FFE_Eval_Metric_Type_Mismatch + REASON-5 PARSE_ERROR → not testable via RC mock (see notes at bottom) + REASON-6 GENERAL → not testable (see notes at bottom) + REASON-7 TARGETING_KEY_MISSING → Test_FFE_REASON_7_TargetingKeyMissing + REASON-8 TARGETING_MATCH (no key) → Test_FFE_REASON_8_RuleOnlyNoKey + REASON-9 zero allocations → DEFAULT → Test_FFE_REASON_9_ZeroAllocations + REASON-10 no-default-alloc → DEFAULT → Test_FFE_REASON_10_NoDefaultAlloc + REASON-11 STATIC (no split/rules) → Test_FFE_REASON_11_StaticNoSplit (uses vacuous split in UFC, + structurally same as REASON-12; see fixture docstring) + REASON-12 STATIC (vacuous split) → test_flag_eval_metrics.py :: Test_FFE_Eval_Metric_Basic + REASON-13 TARGETING_MATCH multi → Test_FFE_REASON_13_MultiAllocRuleMatch + REASON-14 DEFAULT multi (rule fails) → Test_FFE_REASON_14_MultiAllocRuleFail + REASON-15 SPLIT (non-vacuous) → test_flag_eval_metrics.py :: Test_FFE_Eval_Reason_Split + REASON-16 DEFAULT (rule+shard, miss) → Test_FFE_REASON_16_RulePassShardMiss + REASON-17 SPLIT (rule+shard, both) → Test_FFE_REASON_17_RulePassShardWin + REASON-18 DEFAULT (rule+shard, rule fails) → Test_FFE_REASON_18_RuleFail + REASON-19 TARGETING_MATCH after miss → Test_FFE_REASON_19_SplitMissRuleMatch + REASON-20 DEFAULT after split miss → Test_FFE_REASON_20_SplitMissDefault + REASON-21 DEFAULT (active window, single) → Test_FFE_REASON_21_ActiveWindowSingle + REASON-22 DEFAULT (inactive window) → Test_FFE_REASON_22_InactiveWindowSingle + REASON-23 DEFAULT (active window, multi) → Test_FFE_REASON_23_ActiveWindowMulti + REASON-24 DEFAULT (inactive window, multi) → Test_FFE_REASON_24_InactiveWindowMulti + REASON-25 TARGETING_MATCH (window+rules) → Test_FFE_REASON_25_WindowActiveRuleMatch + REASON-26 DEFAULT (window+rules, fail) → Test_FFE_REASON_26_WindowRuleFail +""" + +from typing import Any + +from utils import ( + weblog, + interfaces, + scenarios, + features, + remote_config as rc, +) +from utils.proxy.mocked_response import StaticJsonMockedTracerResponse + + +RC_PRODUCT = "FFE_FLAGS" +RC_PATH = f"datadog/2/{RC_PRODUCT}" + +# --------------------------------------------------------------------------- +# Shard helpers +# --------------------------------------------------------------------------- +# 100% shard: always buckets the subject. A non-empty shard spec always triggers +# hash computation, so a targeting key is required to use this constant. +SHARD_ALWAYS_HIT = [{"salt": "rfc-salt", "totalShards": 10000, "ranges": [{"start": 0, "end": 10000}]}] +# 0% shard: guaranteed miss — the shard entry is present (salt + totalShards), so the +# SDK still computes the hash (targeting key required), but the empty ranges list means +# no subject falls in any bucket. Distinct from shards:[] (vacuous split, no hash at all). +SHARD_ALWAYS_MISS = [{"salt": "rfc-salt", "totalShards": 10000, "ranges": []}] + +# Date windows +WINDOW_ACTIVE_START = "2020-01-01T00:00:00Z" +WINDOW_ACTIVE_END = "2099-01-01T00:00:00Z" +WINDOW_INACTIVE_START = "2020-01-01T00:00:00Z" +WINDOW_INACTIVE_END = "2020-12-31T00:00:00Z" + +# --------------------------------------------------------------------------- +# Metric helpers +# --------------------------------------------------------------------------- +# These are intentionally duplicated from test_flag_eval_metrics.py rather than +# imported. Cross-module imports between test files are fragile: selective test +# runs and different collection roots can cause ImportError at setup time, which +# per the project guidelines must never happen in setup methods. The functions +# are small enough that duplication is the lesser risk. + + +def find_eval_metrics(flag_key: str | None = None) -> list[dict[str, Any]]: + results = [] + for _, point in interfaces.agent.get_metrics(): + if point.get("metric") != "feature_flag.evaluations": + continue + tags = point.get("tags", []) + if flag_key is not None: + if not any(t == f"feature_flag.key:{flag_key}" for t in tags): + continue + results.append(point) + return results + + +def get_tag_value(tags: list[str], key: str) -> str | None: + prefix = f"{key}:" + for tag in tags: + if tag.startswith(prefix): + return tag[len(prefix) :] + return None + + +# --------------------------------------------------------------------------- +# UFC fixture builders +# --------------------------------------------------------------------------- + + +def _base_flag(flag_key: str, variation_type: str = "STRING", *, enabled: bool = True) -> dict: + """Minimal flag definition with no allocations yet.""" + values: dict[str, dict[str, str | bool | float | int]] = { + "STRING": {"on": "on-value", "off": "off-value"}, + "BOOLEAN": {"on": True, "off": False}, + "NUMERIC": {"on": 1.5, "off": 0.0}, + "INTEGER": {"on": 42, "off": 0}, + } + v = values[variation_type] + return { + "key": flag_key, + "enabled": enabled, + "variationType": variation_type, + "variations": { + "on": {"key": "on", "value": v["on"]}, + "off": {"key": "off", "value": v["off"]}, + }, + "allocations": [], + } + + +def _wrap_fixture(flag_key: str, flag_def: dict) -> dict: + return { + "createdAt": "2024-04-17T19:40:53.716Z", + "format": "SERVER", + "environment": {"name": "Test"}, + "flags": {flag_key: flag_def}, + } + + +def make_zero_alloc_fixture(flag_key: str) -> dict: + """REASON-9: Flag with zero allocations.""" + fd = _base_flag(flag_key) + fd["allocations"] = [] + return _wrap_fixture(flag_key, fd) + + +def make_no_default_alloc_fixture(flag_key: str, attribute: str, match_value: str) -> dict: + """REASON-10: Non-empty waterfall but no default allocation and rule won't match.""" + fd = _base_flag(flag_key) + fd["allocations"] = [ + { + "key": "rule-only-alloc", + "rules": [{"conditions": [{"operator": "ONE_OF", "attribute": attribute, "value": [match_value]}]}], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": True, + } + ] + return _wrap_fixture(flag_key, fd) + + +def make_pure_static_fixture(flag_key: str) -> dict: + """REASON-11: Single allocation, no rules, no date window → STATIC. + + Uses shards:[] (vacuous split) which, like an absent shards key, means no + hash computation is performed. Both forms produce STATIC per ADR-003. We use + the vacuous-split form (shards:[]) because it is the established pattern across + the codebase and is guaranteed parseable by all SDK implementations. A split + entry with a missing shards field is not tested in any other fixture and may + trigger a parse error in strict-deserialisation SDKs. + """ + fd = _base_flag(flag_key) + fd["allocations"] = [ + { + "key": "static-alloc", + "rules": [], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": True, + } + ] + return _wrap_fixture(flag_key, fd) + + +def make_multi_alloc_rule_then_default(flag_key: str, attribute: str, match_value: str) -> dict: + """REASON-13/REASON-14: Rule-based alloc + unconditional default alloc.""" + fd = _base_flag(flag_key) + fd["allocations"] = [ + { + "key": "rule-alloc", + "rules": [{"conditions": [{"operator": "ONE_OF", "attribute": attribute, "value": [match_value]}]}], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": True, + }, + { + "key": "default-alloc", + "rules": [], + "splits": [{"variationKey": "off", "shards": []}], + "doLog": True, + }, + ] + return _wrap_fixture(flag_key, fd) + + +def make_real_shard_fixture(flag_key: str) -> dict: + """REASON-7: Non-trivial shard; requires targeting key for hash computation.""" + fd = _base_flag(flag_key) + fd["allocations"] = [ + { + "key": "shard-alloc", + "rules": [], + "splits": [{"variationKey": "on", "shards": SHARD_ALWAYS_HIT}], + "doLog": True, + } + ] + return _wrap_fixture(flag_key, fd) + + +def make_rule_only_no_shard_fixture(flag_key: str, attribute: str, match_value: str) -> dict: + """REASON-8: Targeting rule + vacuous split (no hash computed). Matching without key.""" + fd = _base_flag(flag_key) + fd["allocations"] = [ + { + "key": "rule-alloc", + "rules": [{"conditions": [{"operator": "ONE_OF", "attribute": attribute, "value": [match_value]}]}], + "splits": [{"variationKey": "on", "shards": []}], # Vacuous split → no hash + "doLog": True, + } + ] + return _wrap_fixture(flag_key, fd) + + +def make_rule_plus_shard_with_default(flag_key: str, attribute: str, match_value: str, shard_spec: list) -> dict: + """REASON-16/REASON-17/REASON-18: Same alloc has rule + non-trivial shard; default alloc follows.""" + fd = _base_flag(flag_key) + fd["allocations"] = [ + { + "key": "rule-shard-alloc", + "rules": [{"conditions": [{"operator": "ONE_OF", "attribute": attribute, "value": [match_value]}]}], + "splits": [{"variationKey": "on", "shards": shard_spec}], + "doLog": True, + }, + { + "key": "default-alloc", + "rules": [], + "splits": [{"variationKey": "off", "shards": []}], + "doLog": True, + }, + ] + return _wrap_fixture(flag_key, fd) + + +def make_split_first_then_rule_then_default(flag_key: str, attribute: str, match_value: str) -> dict: + """REASON-19/REASON-20: Shard alloc (guaranteed miss) → rule alloc → default alloc.""" + fd = _base_flag(flag_key) + fd["allocations"] = [ + { + "key": "split-alloc", + "rules": [], + "splits": [{"variationKey": "on", "shards": SHARD_ALWAYS_MISS}], + "doLog": True, + }, + { + "key": "rule-alloc", + "rules": [{"conditions": [{"operator": "ONE_OF", "attribute": attribute, "value": [match_value]}]}], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": True, + }, + { + "key": "default-alloc", + "rules": [], + "splits": [{"variationKey": "off", "shards": []}], + "doLog": True, + }, + ] + return _wrap_fixture(flag_key, fd) + + +def make_active_window_single_alloc(flag_key: str) -> dict: + """REASON-21: Single alloc with active startAt/endAt, no rules, no split.""" + fd = _base_flag(flag_key) + fd["allocations"] = [ + { + "key": "window-alloc", + "startAt": WINDOW_ACTIVE_START, + "endAt": WINDOW_ACTIVE_END, + "rules": [], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": True, + } + ] + return _wrap_fixture(flag_key, fd) + + +def make_inactive_window_single_alloc(flag_key: str) -> dict: + """REASON-22: Single alloc with expired window → no allocation matches → coded default.""" + fd = _base_flag(flag_key) + fd["allocations"] = [ + { + "key": "window-alloc", + "startAt": WINDOW_INACTIVE_START, + "endAt": WINDOW_INACTIVE_END, + "rules": [], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": True, + } + ] + return _wrap_fixture(flag_key, fd) + + +def make_active_window_multi_alloc(flag_key: str) -> dict: + """REASON-23: First alloc has active window (no rules/split); default alloc follows.""" + fd = _base_flag(flag_key) + fd["allocations"] = [ + { + "key": "window-alloc", + "startAt": WINDOW_ACTIVE_START, + "endAt": WINDOW_ACTIVE_END, + "rules": [], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": True, + }, + { + "key": "default-alloc", + "rules": [], + "splits": [{"variationKey": "off", "shards": []}], + "doLog": True, + }, + ] + return _wrap_fixture(flag_key, fd) + + +def make_inactive_window_multi_alloc(flag_key: str) -> dict: + """REASON-24: First alloc has expired window; default alloc catches.""" + fd = _base_flag(flag_key) + fd["allocations"] = [ + { + "key": "window-alloc", + "startAt": WINDOW_INACTIVE_START, + "endAt": WINDOW_INACTIVE_END, + "rules": [], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": True, + }, + { + "key": "default-alloc", + "rules": [], + "splits": [{"variationKey": "off", "shards": []}], + "doLog": True, + }, + ] + return _wrap_fixture(flag_key, fd) + + +def make_window_plus_rules_multi_alloc(flag_key: str, attribute: str, match_value: str) -> dict: + """REASON-25/REASON-26: First alloc has active window + targeting rules; default alloc follows.""" + fd = _base_flag(flag_key) + fd["allocations"] = [ + { + "key": "window-rule-alloc", + "startAt": WINDOW_ACTIVE_START, + "endAt": WINDOW_ACTIVE_END, + "rules": [{"conditions": [{"operator": "ONE_OF", "attribute": attribute, "value": [match_value]}]}], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": True, + }, + { + "key": "default-alloc", + "rules": [], + "splits": [{"variationKey": "off", "shards": []}], + "doLog": True, + }, + ] + return _wrap_fixture(flag_key, fd) + + +# --------------------------------------------------------------------------- +# REASON-2: PROVIDER_FATAL +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_2_ProviderFatal: + """REASON-2: Provider received non-retryable 4XX → ERROR / PROVIDER_FATAL. + + Simulates a 401 Unauthorized from the RC endpoint. Per ADR-009, a non-retryable + 4XX (400, 401, 403, 404) from the assignments endpoint puts the provider into an + unrecoverable PROVIDER_FATAL state. + + Testability note: this depends on the SDK treating an RC-level 401 as a fatal + provider error. SDKs that only treat 4XX from a separate assignments endpoint + may not enter PROVIDER_FATAL via this path; the test is marked accordingly. + """ + + def setup_ffe_reason_2_provider_fatal(self): + self.config_request_data = None + + def wait_for_config_401(data: dict) -> bool: + if data["path"] == "/v0.7/config" and data["response"]["status_code"] == 401: + self.config_request_data = data + return True + return False + + # Return 401 before any valid config has been received + StaticJsonMockedTracerResponse( + path="/v0.7/config", mocked_json={"error": "Unauthorized"}, status_code=401 + ).send() + + # wait_for is the framework's sequencing mechanism, not time.sleep(). It + # blocks until the tracer has observed the 401 response, which ensures the + # provider has had the opportunity to enter PROVIDER_FATAL state before the + # evaluation request is made. Without this gate the evaluation would race + # the tracer's RC polling cycle and could arrive before the 401 is processed. + # The same pattern is used in test_dynamic_evaluation.py (setup_ffe_rc_*). + interfaces.library.wait_for(wait_for_config_401, timeout=60) + + self.flag_key = "b2-provider-fatal-flag" + self.default_value = "coded-default" + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": self.default_value, + "targetingKey": "user-1", + "attributes": {}, + }, + ) + + # Restore normal RC behavior + StaticJsonMockedTracerResponse(path="/v0.7/config", mocked_json={}).send() + + def test_ffe_reason_2_provider_fatal(self): + """REASON-2: 401 from config endpoint → ERROR / PROVIDER_FATAL; coded default returned.""" + assert self.config_request_data is not None, "No 401 response was captured from /v0.7/config" + + assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "error", ( + f"REASON-2: Expected reason=error, got tags: {tags}" + ) + assert get_tag_value(tags, "error.type") == "provider_fatal", ( + f"REASON-2: Expected error.type=provider_fatal, got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-3: Disabled flag → FLAG_NOT_FOUND (ADR-005) +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_3_DisabledFlagNotFound: + """REASON-3: Disabled flag is absent from the dataset → ERROR / FLAG_NOT_FOUND. + + Per ADR-005, DISABLED is not produced by this system. A disabled flag is + excluded from the precomputed dataset, making it operationally absent. + The SDK must return FLAG_NOT_FOUND, not DISABLED. + """ + + def setup_ffe_reason_3_disabled_flag_not_found(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b3-disabled" + self.flag_key = "b3-disabled-flag" + + # Disabled flags are absent from the precomputed dataset the SDK receives. + # Model this by loading a config that contains a *different* flag key — our + # flag is simply not present, which is operationally identical to disabled. + other_flag = _base_flag("some-other-flag") + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", _wrap_fixture("some-other-flag", other_flag) + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "default", + "targetingKey": "user-1", + "attributes": {}, + }, + ) + + def test_ffe_reason_3_disabled_flag_not_found(self): + """REASON-3: Flag absent from config (as disabled flags are) → ERROR / FLAG_NOT_FOUND.""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "error", ( + f"REASON-3: Expected reason=error, got tags: {tags}" + ) + assert get_tag_value(tags, "error.type") == "flag_not_found", ( + f"REASON-3: Expected error.type=flag_not_found (not 'disabled'), got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-7: TARGETING_KEY_MISSING — shard computation reached without targeting key +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_7_TargetingKeyMissing: + """REASON-7: Evaluation path reaches a non-trivial shard; no targeting key → ERROR / TARGETING_KEY_MISSING. + + Per ADR-008, TARGETING_KEY_MISSING fires only when a shard hash computation + is required and no targeting key is available. Evaluations that never reach a + shard proceed normally without a key. + """ + + def setup_ffe_reason_7_targeting_key_missing(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b7-tkm" + self.flag_key = "b7-targeting-key-missing-flag" + rc.tracer_rc_state.set_config(f"{RC_PATH}/{config_id}/config", make_real_shard_fixture(self.flag_key)).apply() + + # Evaluate with no targeting key — shard computation requires one + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "default", + "targetingKey": "", # Missing key + "attributes": {}, + }, + ) + + def test_ffe_reason_7_targeting_key_missing(self): + """REASON-7: Shard computation without targeting key → ERROR / TARGETING_KEY_MISSING.""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "error", ( + f"REASON-7: Expected reason=error, got tags: {tags}" + ) + assert get_tag_value(tags, "error.type") == "targeting_key_missing", ( + f"REASON-7: Expected error.type=targeting_key_missing, got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-8: TARGETING_MATCH — rule-only, no shard, no targeting key required +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_8_RuleOnlyNoKey: + """REASON-8: Rule matches via attribute; vacuous split (no hash); no targeting key needed → TARGETING_MATCH. + + Per ADR-008, a targeting key is not required unless a shard computation is + reached. A flag with targeting rules but only a vacuous split can match + without a targeting key. + """ + + def setup_ffe_reason_8_rule_only_no_key(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b8-rule-no-key" + self.flag_key = "b8-rule-only-no-key-flag" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", + make_rule_only_no_shard_fixture(self.flag_key, "tier", "gold"), + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "default", + "targetingKey": "", # No targeting key + "attributes": {"tier": "gold"}, # Attribute match + }, + ) + + def test_ffe_reason_8_rule_only_no_key(self): + """REASON-8: Rule matches without targeting key (no shard) → TARGETING_MATCH.""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "targeting_match", ( + f"REASON-8: Expected reason=targeting_match, got tags: {tags}" + ) + assert get_tag_value(tags, "error.type") is None, ( + f"REASON-8: Expected no error.type for successful eval, got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-9: Zero allocations → DEFAULT (coded default, ADR-001) +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_9_ZeroAllocations: + """REASON-9: Flag has zero allocations → coded default / DEFAULT (no error code, ADR-001). + + Per ADR-001, an empty waterfall returns the coded default with reason DEFAULT + and no error code. This is not an SDK error — it represents a data invariant + violation on the platform side. + """ + + def setup_ffe_reason_9_zero_allocations(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b9-zero-alloc" + self.flag_key = "b9-zero-alloc-flag" + self.default_value = "coded-default" + rc.tracer_rc_state.set_config(f"{RC_PATH}/{config_id}/config", make_zero_alloc_fixture(self.flag_key)).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": self.default_value, + "targetingKey": "user-1", + "attributes": {}, + }, + ) + + def test_ffe_reason_9_zero_allocations(self): + """REASON-9: Zero allocations → DEFAULT (coded default); no error code.""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( + f"REASON-9: Expected reason=default, got tags: {tags}" + ) + assert get_tag_value(tags, "error.type") is None, ( + f"REASON-9: Expected no error.type (ADR-001: not an SDK error), got tags: {tags}" + ) + # Empty waterfall → no allocation matched → no platform variation selected. + # SDKs may emit the tag as absent (None) or use a sentinel like "n/a"; both are valid. + assert get_tag_value(tags, "feature_flag.result.variant") in (None, "n/a"), ( + f"REASON-9: Expected no platform variation (coded default returned, no allocation matched), got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-10: Non-empty waterfall, no default allocation, no rule matches → DEFAULT +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_10_NoDefaultAlloc: + """REASON-10: Waterfall has allocations; none matched; no default allocation → coded default / DEFAULT. + + Per ADR-001, this is the same treatment as an empty waterfall. The coded + default is returned with reason DEFAULT and no error code. + """ + + def setup_ffe_reason_10_no_default_alloc(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b10-no-default" + self.flag_key = "b10-no-default-alloc-flag" + self.default_value = "coded-default" + # Rule requires tier=platinum; we'll send tier=bronze → no match, no default alloc + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", + make_no_default_alloc_fixture(self.flag_key, "tier", "platinum"), + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": self.default_value, + "targetingKey": "user-1", + "attributes": {"tier": "bronze"}, # Does not match "platinum" + }, + ) + + def test_ffe_reason_10_no_default_alloc(self): + """REASON-10: Waterfall exhausted, no default alloc → DEFAULT (coded default); no error code.""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( + f"REASON-10: Expected reason=default, got tags: {tags}" + ) + assert get_tag_value(tags, "error.type") is None, ( + f"REASON-10: Expected no error.type (ADR-001: not an SDK error), got tags: {tags}" + ) + # Waterfall exhausted, no allocation matched → no platform variation selected. + # SDKs may emit the tag as absent (None) or use a sentinel like "n/a"; both are valid. + assert get_tag_value(tags, "feature_flag.result.variant") in (None, "n/a"), ( + f"REASON-10: Expected no platform variation (coded default returned, no allocation matched), got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-11: STATIC — single allocation, no rules, no date window +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_11_StaticNoSplit: + """REASON-11: Single allocation; no targeting rules, no date window → STATIC. + + The RFC distinguishes REASON-11 ("no split") from REASON-12 ("vacuous split, shards:[]"), + but in the UFC format used by this framework both are represented as a split entry + with an empty shards array — omitting the shards key entirely is not tested across + SDKs and may trigger strict-deserialisation errors. The fixture therefore uses the + same vacuous-split form as REASON-12. Both forms produce STATIC per ADR-003 (no hash + computation, no targeting rules, no date window). This test exercises the STATIC + path via a distinct config ID and flag key, providing independent signal from the + REASON-12 coverage in test_flag_eval_metrics.py :: Test_FFE_Eval_Metric_Basic. + """ + + def setup_ffe_reason_11_static_no_split(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b11-static" + self.flag_key = "b11-static-no-split-flag" + rc.tracer_rc_state.set_config(f"{RC_PATH}/{config_id}/config", make_pure_static_fixture(self.flag_key)).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "default", + "targetingKey": "user-1", + "attributes": {}, + }, + ) + + def test_ffe_reason_11_static_no_split(self): + """REASON-11: Single alloc, no rules, no split → STATIC (platform value).""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "static", ( + f"REASON-11: Expected reason=static, got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-13: TARGETING_MATCH — multi-alloc, rule matches, no split +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_13_MultiAllocRuleMatch: + """REASON-13: Multi-allocation waterfall; rule-based alloc matches; no split in matching alloc → TARGETING_MATCH.""" + + def setup_ffe_reason_13_multi_alloc_rule_match(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b13-multi-rule" + self.flag_key = "b13-multi-alloc-rule-match-flag" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", + make_multi_alloc_rule_then_default(self.flag_key, "plan", "enterprise"), + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "default", + "targetingKey": "user-1", + "attributes": {"plan": "enterprise"}, # Matches rule + }, + ) + + def test_ffe_reason_13_multi_alloc_rule_match(self): + """REASON-13: Multi-alloc, rule matches → TARGETING_MATCH (platform value).""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "targeting_match", ( + f"REASON-13: Expected reason=targeting_match, got tags: {tags}" + ) + assert get_tag_value(tags, "feature_flag.result.variant") == "on", ( + f"REASON-13: Expected variant=on (from rule alloc), got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-14: DEFAULT — multi-alloc, rule-based allocs fail, default alloc catches +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_14_MultiAllocRuleFail: + """REASON-14: Multi-alloc; rule-based allocations fail; default allocation catches → DEFAULT (platform value).""" + + def setup_ffe_reason_14_multi_alloc_rule_fail(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b14-multi-default" + self.flag_key = "b14-multi-alloc-rule-fail-flag" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", + make_multi_alloc_rule_then_default(self.flag_key, "plan", "enterprise"), + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "coded-default", + "targetingKey": "user-1", + "attributes": {"plan": "starter"}, # Does NOT match "enterprise" + }, + ) + + def test_ffe_reason_14_multi_alloc_rule_fail(self): + """REASON-14: Rule fails, default alloc catches → DEFAULT (platform value, not coded default).""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( + f"REASON-14: Expected reason=default, got tags: {tags}" + ) + assert get_tag_value(tags, "feature_flag.result.variant") == "off", ( + f"REASON-14: Expected variant=off (from default alloc), got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-16: DEFAULT — same alloc: rule passes, shard misses, default catches +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_16_RulePassShardMiss: + """REASON-16: Same allocation has targeting rule + split; rule passes; shard misses (0%); default catches → DEFAULT.""" + + def setup_ffe_reason_16_rule_pass_shard_miss(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b16-rule-shard-miss" + self.flag_key = "b16-rule-pass-shard-miss-flag" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", + make_rule_plus_shard_with_default(self.flag_key, "group", "beta", SHARD_ALWAYS_MISS), + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "coded-default", + "targetingKey": "user-1", + "attributes": {"group": "beta"}, # Rule matches; shard guaranteed miss + }, + ) + + def test_ffe_reason_16_rule_pass_shard_miss(self): + """REASON-16: Rule passes, shard misses → allocation skipped; default alloc → DEFAULT.""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( + f"REASON-16: Expected reason=default, got tags: {tags}" + ) + assert get_tag_value(tags, "feature_flag.result.variant") == "off", ( + f"REASON-16: Expected variant=off (from default-alloc, not rule-shard-alloc), got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-17: SPLIT — same alloc: rule passes, shard wins (SPLIT overrides TARGETING_MATCH) +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_17_RulePassShardWin: + """REASON-17: Same allocation: targeting rule passes AND shard wins → SPLIT (ADR-004: SPLIT overrides TARGETING_MATCH).""" + + def setup_ffe_reason_17_rule_pass_shard_win(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b17-rule-shard-win" + self.flag_key = "b17-rule-pass-shard-win-flag" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", + make_rule_plus_shard_with_default(self.flag_key, "group", "alpha", SHARD_ALWAYS_HIT), + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "coded-default", + "targetingKey": "user-1", + "attributes": {"group": "alpha"}, # Rule matches; shard always hits + }, + ) + + def test_ffe_reason_17_rule_pass_shard_win(self): + """REASON-17: Rule passes and shard wins → SPLIT (ADR-004: SPLIT takes precedence).""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "split", ( + f"REASON-17: Expected reason=split (SPLIT overrides TARGETING_MATCH per ADR-004), got tags: {tags}" + ) + assert get_tag_value(tags, "feature_flag.result.variant") == "on", ( + f"REASON-17: Expected variant=on (rule-shard-alloc won), got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-18: DEFAULT — same alloc: rule fails, default catches +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_18_RuleFail: + """REASON-18: Same allocation: rule fails; default allocation catches → DEFAULT.""" + + def setup_ffe_reason_18_rule_fail(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b18-rule-fail" + self.flag_key = "b18-rule-fail-flag" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", + make_rule_plus_shard_with_default(self.flag_key, "group", "alpha", SHARD_ALWAYS_HIT), + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "coded-default", + "targetingKey": "user-1", + "attributes": {"group": "beta"}, # Rule requires "alpha" → fails + }, + ) + + def test_ffe_reason_18_rule_fail(self): + """REASON-18: Rule fails; default alloc catches → DEFAULT.""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( + f"REASON-18: Expected reason=default, got tags: {tags}" + ) + assert get_tag_value(tags, "feature_flag.result.variant") == "off", ( + f"REASON-18: Expected variant=off (default-alloc caught; rule-shard-alloc rule failed), got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-19: TARGETING_MATCH — split alloc skipped (miss), rule-based alloc 2 matches +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_19_SplitMissRuleMatch: + """REASON-19: Split alloc skipped (shard miss); rule-based alloc 2 matches; no split → TARGETING_MATCH.""" + + def setup_ffe_reason_19_split_miss_rule_match(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b19-split-miss-rule" + self.flag_key = "b19-split-miss-rule-match-flag" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", + make_split_first_then_rule_then_default(self.flag_key, "region", "us-east"), + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "coded-default", + "targetingKey": "user-1", + "attributes": {"region": "us-east"}, # Matches rule alloc + }, + ) + + def test_ffe_reason_19_split_miss_rule_match(self): + """REASON-19: Split alloc skipped (guaranteed miss); rule alloc 2 matches → TARGETING_MATCH.""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "targeting_match", ( + f"REASON-19: Expected reason=targeting_match, got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-20: DEFAULT — split alloc skipped (miss), default catches +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_20_SplitMissDefault: + """REASON-20: Split alloc skipped (shard miss); default alloc catches → DEFAULT.""" + + def setup_ffe_reason_20_split_miss_default(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b20-split-miss-default" + self.flag_key = "b20-split-miss-default-flag" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", + make_split_first_then_rule_then_default(self.flag_key, "region", "us-east"), + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "coded-default", + "targetingKey": "user-1", + "attributes": {"region": "eu-west"}, # Does NOT match rule; split guaranteed miss + }, + ) + + def test_ffe_reason_20_split_miss_default(self): + """REASON-20: Split miss, rule fails, default alloc catches → DEFAULT.""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( + f"REASON-20: Expected reason=default, got tags: {tags}" + ) + assert get_tag_value(tags, "feature_flag.result.variant") == "off", ( + f"REASON-20: Expected variant=off (default-alloc caught; split-alloc missed, rule-alloc failed), got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-21: DEFAULT — single alloc with active date window, no rules, no split +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_21_ActiveWindowSingle: + """REASON-21: Single allocation; startAt/endAt present; window active; no rules, no split → DEFAULT. + + Per ADR-003, startAt/endAt are scheduling metadata, not targeting rules. + A single alloc with an active date window does NOT produce STATIC because + the result could change when the window expires. Reason is DEFAULT. + """ + + def setup_ffe_reason_21_active_window_single(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b21-active-window-single" + self.flag_key = "b21-active-window-single-flag" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", make_active_window_single_alloc(self.flag_key) + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "coded-default", + "targetingKey": "user-1", + "attributes": {}, + }, + ) + + def test_ffe_reason_21_active_window_single(self): + """REASON-21: Active date window, single alloc, no rules → DEFAULT (platform value).""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( + f"REASON-21: Expected reason=default (not static — date window precludes STATIC per ADR-003), got tags: {tags}" + ) + # Variant assertion confirms the window-alloc fired (variant "on"), not a fallback. + # Without this, a coded-default return of "coded-default" would also satisfy reason=default. + assert get_tag_value(tags, "feature_flag.result.variant") == "on", ( + f"REASON-21: Expected variant=on (window-alloc fired), got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-22: DEFAULT — single alloc with inactive (expired) date window +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_22_InactiveWindowSingle: + """REASON-22: Single allocation; startAt/endAt present; window inactive (expired) → coded default / DEFAULT. + + The allocation's window has closed. The flag is present in config (not absent), + but its only allocation is outside its time window. Per ADR-001, this is the + non-empty-waterfall / no-match / no-default-allocation case → coded default, DEFAULT. + This is NOT FLAG_NOT_FOUND (the flag exists in config). + """ + + def setup_ffe_reason_22_inactive_window_single(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b22-inactive-window" + self.flag_key = "b22-inactive-window-single-flag" + self.default_value = "coded-default" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", make_inactive_window_single_alloc(self.flag_key) + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": self.default_value, + "targetingKey": "user-1", + "attributes": {}, + }, + ) + + def test_ffe_reason_22_inactive_window_single(self): + """REASON-22: Expired window → no active allocation; coded default / DEFAULT; no error code.""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( + f"REASON-22: Expected reason=default (ADR-001 data-invariant case, not FLAG_NOT_FOUND), got tags: {tags}" + ) + assert get_tag_value(tags, "error.type") is None, f"REASON-22: Expected no error.type (ADR-001), got tags: {tags}" + # No allocation matched (window expired), so no platform variation was selected. + # SDKs may emit the tag as absent (None) or use a sentinel like "n/a"; both are valid. + # This distinguishes REASON-22 from REASON-24 where the default-alloc fires and produces variant=off. + assert get_tag_value(tags, "feature_flag.result.variant") in (None, "n/a"), ( + f"REASON-22: Expected no platform variation (coded default, window expired — no alloc matched), got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-23: DEFAULT — multi-alloc, first alloc active window, no rules/split +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_23_ActiveWindowMulti: + """REASON-23: Multi-alloc; first alloc has active startAt/endAt; no rules, no split → DEFAULT.""" + + def setup_ffe_reason_23_active_window_multi(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b23-active-window-multi" + self.flag_key = "b23-active-window-multi-flag" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", make_active_window_multi_alloc(self.flag_key) + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "coded-default", + "targetingKey": "user-1", + "attributes": {}, + }, + ) + + def test_ffe_reason_23_active_window_multi(self): + """REASON-23: Multi-alloc, first has active window (no rules/split) → DEFAULT.""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( + f"REASON-23: Expected reason=default, got tags: {tags}" + ) + # Variant "on" confirms the window-alloc (first alloc) fired, distinguishing this from + # the default-alloc catching (variant "off") or a coded-default return. + assert get_tag_value(tags, "feature_flag.result.variant") == "on", ( + f"REASON-23: Expected variant=on (window-alloc fired, not default-alloc), got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-24: DEFAULT — multi-alloc, first alloc window inactive, default catches +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_24_InactiveWindowMulti: + """REASON-24: Multi-alloc; first alloc has expired window; default alloc catches → DEFAULT (platform value). + + Per the row-24 note in Appendix B: this is the ADR-001 data-invariant case + (non-empty waterfall, no match, default alloc provides platform value). + The flag is present in config; an inactive window gate is not FLAG_NOT_FOUND. + """ + + def setup_ffe_reason_24_inactive_window_multi(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b24-inactive-window-multi" + self.flag_key = "b24-inactive-window-multi-flag" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", make_inactive_window_multi_alloc(self.flag_key) + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "coded-default", + "targetingKey": "user-1", + "attributes": {}, + }, + ) + + def test_ffe_reason_24_inactive_window_multi(self): + """REASON-24: First alloc window expired; default alloc catches → DEFAULT (platform value).""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( + f"REASON-24: Expected reason=default, got tags: {tags}" + ) + assert get_tag_value(tags, "feature_flag.result.variant") == "off", ( + f"REASON-24: Expected variant=off (from default alloc, platform value), got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-25: TARGETING_MATCH — window active + rule matches +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_25_WindowActiveRuleMatch: + """REASON-25: Multi-alloc; alloc has active startAt/endAt + targeting rules; rule matches; no split → TARGETING_MATCH.""" + + def setup_ffe_reason_25_window_active_rule_match(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b25-window-rule-match" + self.flag_key = "b25-window-active-rule-match-flag" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", + make_window_plus_rules_multi_alloc(self.flag_key, "segment", "vip"), + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "coded-default", + "targetingKey": "user-1", + "attributes": {"segment": "vip"}, # Matches rule + }, + ) + + def test_ffe_reason_25_window_active_rule_match(self): + """REASON-25: Active window + rule matches → TARGETING_MATCH (window alone doesn't imply TARGETING_MATCH).""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "targeting_match", ( + f"REASON-25: Expected reason=targeting_match, got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# REASON-26: DEFAULT — window + rules, rule fails, default catches +# --------------------------------------------------------------------------- + + +@scenarios.feature_flagging_and_experimentation +@features.feature_flags_eval_metrics +class Test_FFE_REASON_26_WindowRuleFail: + """REASON-26: Multi-alloc; alloc has startAt/endAt + targeting rules; rule fails; default catches → DEFAULT.""" + + def setup_ffe_reason_26_window_rule_fail(self): + rc.tracer_rc_state.reset().apply() + + config_id = "ffe-b26-window-rule-fail" + self.flag_key = "b26-window-rule-fail-flag" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", + make_window_plus_rules_multi_alloc(self.flag_key, "segment", "vip"), + ).apply() + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": "coded-default", + "targetingKey": "user-1", + "attributes": {"segment": "standard"}, # Does NOT match "vip" + }, + ) + + def test_ffe_reason_26_window_rule_fail(self): + """REASON-26: Active window + rule fails; default alloc catches → DEFAULT.""" + assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" + + metrics = find_eval_metrics(self.flag_key) + assert len(metrics) > 0, f"Expected metric for '{self.flag_key}', found none. All: {find_eval_metrics()}" + + point = metrics[0] + tags = point.get("tags", []) + + assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( + f"REASON-26: Expected reason=default, got tags: {tags}" + ) + assert get_tag_value(tags, "feature_flag.result.variant") == "off", ( + f"REASON-26: Expected variant=off (default-alloc caught; window-rule-alloc rule failed), got tags: {tags}" + ) + + +# --------------------------------------------------------------------------- +# Untestable / deferred cases (noted for completeness) +# --------------------------------------------------------------------------- +# +# REASON-5 PARSE_ERROR (entire payload): StaticJsonMockedTracerResponse always +# serialises a Python dict through json.dumps and sends valid JSON. There +# is no current mechanism in the mock framework to inject a raw non-JSON +# body. Even sending a valid-JSON body with unexpected structure (e.g., +# {"corrupted": true}) will not trigger PARSE_ERROR — SDKs will treat it as +# an empty/unknown config and produce FLAG_NOT_FOUND or DEFAULT, not +# PARSE_ERROR. This case requires either raw-body mock support or a +# lower-level fault injection mechanism not currently available. +# +# REASON-6 GENERAL: No reliable way to trigger an unclassified SDK error via the +# weblog/RC interface. GENERAL is a catch-all for internal SDK failures +# that don't fit a more specific code. Requires SDK internals access or +# fault injection at a layer below what system tests can reach. +# +# REASON-2 PROVIDER_FATAL: Test_FFE_REASON_2_ProviderFatal above sends a 401 via +# the RC mock. Whether this triggers PROVIDER_FATAL depends on whether the +# SDK treats RC-level 4XX responses as fatal provider errors vs. transient +# config fetch failures. SDKs where the FFE provider has a separate +# assignments endpoint (not RC) will not enter PROVIDER_FATAL via this +# path. The test is a best-effort probe; SDK teams must verify their +# specific PROVIDER_FATAL transition path independently. +# +# REASON-13 INVALID_CONTEXT: Per the RFC, this error code is reserved and likely to +# be deprecated pending OF.3 dot-flattening resolution. Not tested here. +# The OF.3 behavior (nested attributes ignored vs. error) is covered by +# Test_FFE_Eval_Nested_Attributes_Ignored in test_flag_eval_metrics.py. From d5ceadf5f3f1bce37a9076fce2875eacf16636d4 Mon Sep 17 00:00:00 2001 From: typotter Date: Wed, 22 Apr 2026 14:47:53 -0600 Subject: [PATCH 03/14] chore(ffe): activate 6 eval-metrics tests for Java 1.62.0-SNAPSHOT Fixes in dd-trace-java master (PRs #11036, #11037, #11071) resolve the FFL-1972 bugs. Use v1.62.0-SNAPSHOT so these tests run as active (not xfail) in CI against Java master. Also removes the file-level missing_feature entry and fixes two YAML explicit-key entries for the parse-error tests. --- manifests/java.yml | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/manifests/java.yml b/manifests/java.yml index 11cfa0fbc50..e4f4ed17b47 100644 --- a/manifests/java.yml +++ b/manifests/java.yml @@ -3133,24 +3133,21 @@ manifest: "*": irrelevant spring-boot: v1.56.0 tests/ffe/test_exposures.py::Test_FFE_EXP_5_Missing_Targeting_Key: bug (FFL-1729) - tests/ffe/test_flag_eval_metrics.py: missing_feature (FFL-1972) tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Config_Exists_Flag_Missing::test_ffe_eval_config_exists_flag_missing: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Lowercase_Consistency::test_ffe_lowercase_error_type: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Lowercase_Consistency::test_ffe_lowercase_reason: v1.61.0 - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Basic::test_ffe_eval_metric_basic: bug (FFL-1972) + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Basic::test_ffe_eval_metric_basic: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Count::test_ffe_eval_metric_count: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Different_Flags::test_ffe_eval_metric_different_flags: v1.61.0 - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Numeric_To_Integer::test_ffe_eval_metric_numeric_to_integer: bug (FFL-1972) - ? tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex::test_ffe_eval_metric_parse_error_invalid_regex - : bug (FFL-1972) - ? tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Variant_Type_Mismatch::test_ffe_eval_metric_parse_error_variant_type_mismatch - : bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Type_Mismatch::test_ffe_eval_metric_type_mismatch: bug (FFL-1972) + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Numeric_To_Integer::test_ffe_eval_metric_numeric_to_integer: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex::test_ffe_eval_metric_parse_error_invalid_regex: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Variant_Type_Mismatch::test_ffe_eval_metric_parse_error_variant_type_mismatch: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Type_Mismatch::test_ffe_eval_metric_type_mismatch: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Nested_Attributes_Ignored::test_ffe_eval_nested_attributes_ignored: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_No_Config_Loaded::test_ffe_eval_no_config_loaded: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Default::test_ffe_eval_reason_default: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Disabled::test_ffe_eval_reason_disabled: v1.61.0 - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: bug (FFL-1972) + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Targeting::test_ffe_eval_reason_targeting: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional::test_ffe_eval_targeting_key_optional: v1.61.0 # REASON-3, REASON-7, REASON-8, REASON-10, REASON-13, REASON-19, REASON-22, REASON-25 pass against Java ≥ v1.61.0. From 011c1746967d093cb9ed47d91d427f34e9d91104 Mon Sep 17 00:00:00 2001 From: typotter Date: Wed, 22 Apr 2026 16:04:26 -0600 Subject: [PATCH 04/14] fix(ffe): sort REASON manifest entries in alphabetical key order Manifest validator requires string-sorted keys. REASON_10 < REASON_2_ etc. because '0' (48) < '_' (95) in ASCII. Reorders all 20 REASON entries. --- manifests/java.yml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/manifests/java.yml b/manifests/java.yml index e4f4ed17b47..b2a9356a429 100644 --- a/manifests/java.yml +++ b/manifests/java.yml @@ -3164,11 +3164,6 @@ manifest: # REASON-23 DEFAULT multi-alloc active window (Java returns targeting_match) # REASON-24 DEFAULT multi-alloc inactive window (Java returns targeting_match) # REASON-26 DEFAULT via window+rule-fail (Java returns targeting_match) - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_2_ProviderFatal::test_ffe_reason_2_provider_fatal: missing_feature - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_3_DisabledFlagNotFound::test_ffe_reason_3_disabled_flag_not_found: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_7_TargetingKeyMissing::test_ffe_reason_7_targeting_key_missing: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_8_RuleOnlyNoKey::test_ffe_reason_8_rule_only_no_key: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_9_ZeroAllocations::test_ffe_reason_9_zero_allocations: missing_feature tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_10_NoDefaultAlloc::test_ffe_reason_10_no_default_alloc: v1.61.0 tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_11_StaticNoSplit::test_ffe_reason_11_static_no_split: missing_feature tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_13_MultiAllocRuleMatch::test_ffe_reason_13_multi_alloc_rule_match: v1.61.0 @@ -3184,6 +3179,11 @@ manifest: tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_24_InactiveWindowMulti::test_ffe_reason_24_inactive_window_multi: missing_feature tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_25_WindowActiveRuleMatch::test_ffe_reason_25_window_active_rule_match: v1.61.0 tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_26_WindowRuleFail::test_ffe_reason_26_window_rule_fail: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_2_ProviderFatal::test_ffe_reason_2_provider_fatal: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_3_DisabledFlagNotFound::test_ffe_reason_3_disabled_flag_not_found: v1.61.0 + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_7_TargetingKeyMissing::test_ffe_reason_7_targeting_key_missing: v1.61.0 + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_8_RuleOnlyNoKey::test_ffe_reason_8_rule_only_no_key: v1.61.0 + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_9_ZeroAllocations::test_ffe_reason_9_zero_allocations: missing_feature tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234) tests/integration_frameworks/llm/openai/test_openai_apm.py: v1.61.0 tests/integration_frameworks/llm/openai/test_openai_llmobs.py: v1.61.0 From 36916820c49a3ec25678de90f40754fe33a8c5be Mon Sep 17 00:00:00 2001 From: typotter Date: Thu, 23 Apr 2026 13:27:06 -0600 Subject: [PATCH 05/14] test(ffe): add feature_flag.key assertions and fix REASON-2 setup - Add feature_flag.key tag assertion to all 20 test methods (H-1) - Add variant=on assertion to REASON-11 test (H-3) - Move rc.tracer_rc_state.reset().apply() inside try block in REASON-2 setup so it is covered by the finally cleanup (H-4) - Wrap REASON-2 setup body in try/finally for exception-safe mock restoration (M-1); wrap finally cleanup in try/except so cleanup failures do not mask the primary exception (M-2) - Fix ruff formatting on long assert in REASON-22 --- tests/ffe/test_flag_eval_reasons.py | 184 +++++++++++++++++++++++----- 1 file changed, 153 insertions(+), 31 deletions(-) diff --git a/tests/ffe/test_flag_eval_reasons.py b/tests/ffe/test_flag_eval_reasons.py index dde595d2f3b..3196badbf4b 100644 --- a/tests/ffe/test_flag_eval_reasons.py +++ b/tests/ffe/test_flag_eval_reasons.py @@ -57,6 +57,12 @@ # --------------------------------------------------------------------------- # 100% shard: always buckets the subject. A non-empty shard spec always triggers # hash computation, so a targeting key is required to use this constant. +# +# README note: "A single variation covering 100% (0-10000) returns STATIC, not SPLIT" +# applies only to allocations with no targeting rules. Every use of SHARD_ALWAYS_HIT +# here is in an allocation that also has a targeting rule (REASON-7, REASON-17, REASON-18), +# so the evaluation path performs a real hash computation; the correct expected reason +# for REASON-17 is SPLIT (ADR-004), not STATIC. SHARD_ALWAYS_HIT = [{"salt": "rfc-salt", "totalShards": 10000, "ranges": [{"start": 0, "end": 10000}]}] # 0% shard: guaranteed miss — the shard entry is present (salt + totalShards), so the # SDK still computes the hash (targeting key required), but the empty ranges list means @@ -391,6 +397,11 @@ class Test_FFE_REASON_2_ProviderFatal: def setup_ffe_reason_2_provider_fatal(self): self.config_request_data = None + # Assign flag_key and default_value before the mock is sent so that the + # closure and subsequent assertions can reference them without depending on + # execution order after wait_for returns. + self.flag_key = "b2-provider-fatal-flag" + self.default_value = "coded-default" def wait_for_config_401(data: dict) -> bool: if data["path"] == "/v0.7/config" and data["response"]["status_code"] == 401: @@ -398,35 +409,45 @@ def wait_for_config_401(data: dict) -> bool: return True return False - # Return 401 before any valid config has been received - StaticJsonMockedTracerResponse( - path="/v0.7/config", mocked_json={"error": "Unauthorized"}, status_code=401 - ).send() - - # wait_for is the framework's sequencing mechanism, not time.sleep(). It - # blocks until the tracer has observed the 401 response, which ensures the - # provider has had the opportunity to enter PROVIDER_FATAL state before the - # evaluation request is made. Without this gate the evaluation would race - # the tracer's RC polling cycle and could arrive before the 401 is processed. - # The same pattern is used in test_dynamic_evaluation.py (setup_ffe_rc_*). - interfaces.library.wait_for(wait_for_config_401, timeout=60) - - self.flag_key = "b2-provider-fatal-flag" - self.default_value = "coded-default" - - self.r = weblog.post( - "/ffe", - json={ - "flag": self.flag_key, - "variationType": "STRING", - "defaultValue": self.default_value, - "targetingKey": "user-1", - "attributes": {}, - }, - ) - - # Restore normal RC behavior - StaticJsonMockedTracerResponse(path="/v0.7/config", mocked_json={}).send() + try: + # reset() inside try so that if it raises, the finally cleanup still runs + # and self.r is assigned on the normal path (avoiding AttributeError in the + # test method on reset failure). + rc.tracer_rc_state.reset().apply() + + # Return 401 before any valid config has been received + StaticJsonMockedTracerResponse( + path="/v0.7/config", mocked_json={"error": "Unauthorized"}, status_code=401 + ).send() + + # wait_for is the framework's sequencing mechanism, not time.sleep(). It + # blocks until the tracer has observed the 401 response, which ensures the + # provider has had the opportunity to enter PROVIDER_FATAL state before the + # evaluation request is made. Without this gate the evaluation would race + # the tracer's RC polling cycle and could arrive before the 401 is processed. + # The same pattern is used in test_dynamic_evaluation.py (setup_ffe_rc_*). + interfaces.library.wait_for(wait_for_config_401, timeout=60) + + self.r = weblog.post( + "/ffe", + json={ + "flag": self.flag_key, + "variationType": "STRING", + "defaultValue": self.default_value, + "targetingKey": "user-1", + "attributes": {}, + }, + ) + finally: + # Restore normal RC behavior. send() is fire-and-forget; no wait_for is needed + # after restoration because each subsequent test class calls + # rc.tracer_rc_state.reset().apply() in its own setup, providing a clean-state + # guarantee independently of whether this send() has been processed yet. + # Wrapped in try/except so a cleanup failure does not mask the primary exception. + try: + StaticJsonMockedTracerResponse(path="/v0.7/config", mocked_json={}).send() + except Exception: # noqa: BLE001 + pass def test_ffe_reason_2_provider_fatal(self): """REASON-2: 401 from config endpoint → ERROR / PROVIDER_FATAL; coded default returned.""" @@ -440,12 +461,20 @@ def test_ffe_reason_2_provider_fatal(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-2: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "error", ( f"REASON-2: Expected reason=error, got tags: {tags}" ) assert get_tag_value(tags, "error.type") == "provider_fatal", ( f"REASON-2: Expected error.type=provider_fatal, got tags: {tags}" ) + # On PROVIDER_FATAL, no platform variation was selected; the SDK returns the + # coded default. The variant tag should be absent or a sentinel (not a flag variation). + assert get_tag_value(tags, "feature_flag.result.variant") in (None, "n/a"), ( + f"REASON-2: Expected no platform variant on PROVIDER_FATAL (coded default returned), got tags: {tags}" + ) # --------------------------------------------------------------------------- @@ -498,6 +527,9 @@ def test_ffe_reason_3_disabled_flag_not_found(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-3: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "error", ( f"REASON-3: Expected reason=error, got tags: {tags}" ) @@ -550,6 +582,9 @@ def test_ffe_reason_7_targeting_key_missing(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-7: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "error", ( f"REASON-7: Expected reason=error, got tags: {tags}" ) @@ -604,9 +639,18 @@ def test_ffe_reason_8_rule_only_no_key(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-8: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "targeting_match", ( f"REASON-8: Expected reason=targeting_match, got tags: {tags}" ) + assert get_tag_value(tags, "feature_flag.result.variant") == "on", ( + f"REASON-8: Expected variant=on (rule-alloc fired), got tags: {tags}" + ) + assert get_tag_value(tags, "feature_flag.result.allocation_key") == "rule-alloc", ( + f"REASON-8: Expected allocation_key=rule-alloc, got tags: {tags}" + ) assert get_tag_value(tags, "error.type") is None, ( f"REASON-8: Expected no error.type for successful eval, got tags: {tags}" ) @@ -656,6 +700,9 @@ def test_ffe_reason_9_zero_allocations(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-9: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( f"REASON-9: Expected reason=default, got tags: {tags}" ) @@ -716,6 +763,9 @@ def test_ffe_reason_10_no_default_alloc(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-10: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( f"REASON-10: Expected reason=default, got tags: {tags}" ) @@ -777,9 +827,18 @@ def test_ffe_reason_11_static_no_split(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-11: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "static", ( f"REASON-11: Expected reason=static, got tags: {tags}" ) + assert get_tag_value(tags, "feature_flag.result.variant") == "on", ( + f"REASON-11: Expected variant=on (static-alloc fired), got tags: {tags}" + ) + assert get_tag_value(tags, "feature_flag.result.allocation_key") == "static-alloc", ( + f"REASON-11: Expected allocation_key=static-alloc, got tags: {tags}" + ) # --------------------------------------------------------------------------- @@ -823,12 +882,18 @@ def test_ffe_reason_13_multi_alloc_rule_match(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-13: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "targeting_match", ( f"REASON-13: Expected reason=targeting_match, got tags: {tags}" ) assert get_tag_value(tags, "feature_flag.result.variant") == "on", ( f"REASON-13: Expected variant=on (from rule alloc), got tags: {tags}" ) + assert get_tag_value(tags, "feature_flag.result.allocation_key") == "rule-alloc", ( + f"REASON-13: Expected allocation_key=rule-alloc, got tags: {tags}" + ) # --------------------------------------------------------------------------- @@ -872,6 +937,9 @@ def test_ffe_reason_14_multi_alloc_rule_fail(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-14: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( f"REASON-14: Expected reason=default, got tags: {tags}" ) @@ -921,6 +989,9 @@ def test_ffe_reason_16_rule_pass_shard_miss(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-16: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( f"REASON-16: Expected reason=default, got tags: {tags}" ) @@ -970,6 +1041,9 @@ def test_ffe_reason_17_rule_pass_shard_win(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-17: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "split", ( f"REASON-17: Expected reason=split (SPLIT overrides TARGETING_MATCH per ADR-004), got tags: {tags}" ) @@ -1019,6 +1093,9 @@ def test_ffe_reason_18_rule_fail(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-18: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( f"REASON-18: Expected reason=default, got tags: {tags}" ) @@ -1068,9 +1145,18 @@ def test_ffe_reason_19_split_miss_rule_match(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-19: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "targeting_match", ( f"REASON-19: Expected reason=targeting_match, got tags: {tags}" ) + assert get_tag_value(tags, "feature_flag.result.variant") == "on", ( + f"REASON-19: Expected variant=on (rule-alloc fired after split-alloc missed), got tags: {tags}" + ) + assert get_tag_value(tags, "feature_flag.result.allocation_key") == "rule-alloc", ( + f"REASON-19: Expected allocation_key=rule-alloc, got tags: {tags}" + ) # --------------------------------------------------------------------------- @@ -1114,6 +1200,9 @@ def test_ffe_reason_20_split_miss_default(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-20: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( f"REASON-20: Expected reason=default, got tags: {tags}" ) @@ -1167,6 +1256,9 @@ def test_ffe_reason_21_active_window_single(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-21: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( f"REASON-21: Expected reason=default (not static — date window precludes STATIC per ADR-003), got tags: {tags}" ) @@ -1224,10 +1316,15 @@ def test_ffe_reason_22_inactive_window_single(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-22: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( f"REASON-22: Expected reason=default (ADR-001 data-invariant case, not FLAG_NOT_FOUND), got tags: {tags}" ) - assert get_tag_value(tags, "error.type") is None, f"REASON-22: Expected no error.type (ADR-001), got tags: {tags}" + assert get_tag_value(tags, "error.type") is None, ( + f"REASON-22: Expected no error.type (ADR-001), got tags: {tags}" + ) # No allocation matched (window expired), so no platform variation was selected. # SDKs may emit the tag as absent (None) or use a sentinel like "n/a"; both are valid. # This distinguishes REASON-22 from REASON-24 where the default-alloc fires and produces variant=off. @@ -1244,7 +1341,14 @@ def test_ffe_reason_22_inactive_window_single(self): @scenarios.feature_flagging_and_experimentation @features.feature_flags_eval_metrics class Test_FFE_REASON_23_ActiveWindowMulti: - """REASON-23: Multi-alloc; first alloc has active startAt/endAt; no rules, no split → DEFAULT.""" + """REASON-23: Multi-alloc; first alloc has active startAt/endAt; no rules, no split → DEFAULT. + + Per ADR-003: the presence of startAt/endAt scheduling metadata on an allocation + precludes STATIC, because the evaluation result is time-dependent and could change + when the window expires. This applies regardless of whether the split is vacuous + (shards:[]). The reason is DEFAULT, not STATIC. The variant assertion (variant=on) + confirms the window-alloc fired rather than the default-alloc (variant=off). + """ def setup_ffe_reason_23_active_window_multi(self): rc.tracer_rc_state.reset().apply() @@ -1276,6 +1380,9 @@ def test_ffe_reason_23_active_window_multi(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-23: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( f"REASON-23: Expected reason=default, got tags: {tags}" ) @@ -1331,6 +1438,9 @@ def test_ffe_reason_24_inactive_window_multi(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-24: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( f"REASON-24: Expected reason=default, got tags: {tags}" ) @@ -1380,9 +1490,18 @@ def test_ffe_reason_25_window_active_rule_match(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-25: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "targeting_match", ( f"REASON-25: Expected reason=targeting_match, got tags: {tags}" ) + assert get_tag_value(tags, "feature_flag.result.variant") == "on", ( + f"REASON-25: Expected variant=on (window-rule-alloc fired), got tags: {tags}" + ) + assert get_tag_value(tags, "feature_flag.result.allocation_key") == "window-rule-alloc", ( + f"REASON-25: Expected allocation_key=window-rule-alloc, got tags: {tags}" + ) # --------------------------------------------------------------------------- @@ -1426,6 +1545,9 @@ def test_ffe_reason_26_window_rule_fail(self): point = metrics[0] tags = point.get("tags", []) + assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( + f"REASON-26: Expected feature_flag.key={self.flag_key}, got tags: {tags}" + ) assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( f"REASON-26: Expected reason=default, got tags: {tags}" ) From e6e46abcc8eae8e123c667b7dae7f18ad785afbf Mon Sep 17 00:00:00 2001 From: typotter Date: Thu, 23 Apr 2026 13:35:40 -0600 Subject: [PATCH 06/14] fix(lint): remove unused noqa directive (BLE001 not enabled in ruff config) --- tests/ffe/test_flag_eval_reasons.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/ffe/test_flag_eval_reasons.py b/tests/ffe/test_flag_eval_reasons.py index 3196badbf4b..3adc73aa859 100644 --- a/tests/ffe/test_flag_eval_reasons.py +++ b/tests/ffe/test_flag_eval_reasons.py @@ -446,7 +446,7 @@ def wait_for_config_401(data: dict) -> bool: # Wrapped in try/except so a cleanup failure does not mask the primary exception. try: StaticJsonMockedTracerResponse(path="/v0.7/config", mocked_json={}).send() - except Exception: # noqa: BLE001 + except Exception: pass def test_ffe_reason_2_provider_fatal(self): From 5409ee610ab7c23d4f65e398de21d7e929cb0b22 Mon Sep 17 00:00:00 2001 From: typotter Date: Thu, 23 Apr 2026 13:46:55 -0600 Subject: [PATCH 07/14] fix(lint): use contextlib.suppress for cleanup; activate all reason tests - Replace try/except/pass with contextlib.suppress (SIM105, S110) - Activate all 12 remaining missing_feature reason tests at v1.62.0-SNAPSHOT --- manifests/java.yml | 38 +++++++++-------------------- tests/ffe/test_flag_eval_reasons.py | 7 +++--- 2 files changed, 15 insertions(+), 30 deletions(-) diff --git a/manifests/java.yml b/manifests/java.yml index b2a9356a429..4ca35680aff 100644 --- a/manifests/java.yml +++ b/manifests/java.yml @@ -3150,40 +3150,26 @@ manifest: tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Targeting::test_ffe_eval_reason_targeting: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional::test_ffe_eval_targeting_key_optional: v1.61.0 - # REASON-3, REASON-7, REASON-8, REASON-10, REASON-13, REASON-19, REASON-22, REASON-25 pass against Java ≥ v1.61.0. - # The remaining 12 tests cover RFC Appendix B paths not yet implemented in Java: - # REASON-2 PROVIDER_FATAL (RC 401 not treated as fatal by Java's FFE provider) - # REASON-9 ADR-001: empty waterfall → DEFAULT (Java returns error/general instead) - # REASON-11 STATIC (Java returns targeting_match for any matched allocation) - # REASON-14 DEFAULT via unconditional alloc (Java returns targeting_match) - # REASON-16 DEFAULT via shard-miss (Java returns targeting_match for default-alloc) - # REASON-17 SPLIT overrides TARGETING_MATCH (ADR-004 not implemented in Java) - # REASON-18 DEFAULT via rule-fail (Java returns targeting_match for default-alloc) - # REASON-20 DEFAULT via split-miss + rule-fail (Java returns targeting_match) - # REASON-21 DEFAULT with active window (Java returns targeting_match) - # REASON-23 DEFAULT multi-alloc active window (Java returns targeting_match) - # REASON-24 DEFAULT multi-alloc inactive window (Java returns targeting_match) - # REASON-26 DEFAULT via window+rule-fail (Java returns targeting_match) tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_10_NoDefaultAlloc::test_ffe_reason_10_no_default_alloc: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_11_StaticNoSplit::test_ffe_reason_11_static_no_split: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_11_StaticNoSplit::test_ffe_reason_11_static_no_split: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_13_MultiAllocRuleMatch::test_ffe_reason_13_multi_alloc_rule_match: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_14_MultiAllocRuleFail::test_ffe_reason_14_multi_alloc_rule_fail: missing_feature - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_16_RulePassShardMiss::test_ffe_reason_16_rule_pass_shard_miss: missing_feature - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_17_RulePassShardWin::test_ffe_reason_17_rule_pass_shard_win: missing_feature - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_18_RuleFail::test_ffe_reason_18_rule_fail: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_14_MultiAllocRuleFail::test_ffe_reason_14_multi_alloc_rule_fail: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_16_RulePassShardMiss::test_ffe_reason_16_rule_pass_shard_miss: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_17_RulePassShardWin::test_ffe_reason_17_rule_pass_shard_win: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_18_RuleFail::test_ffe_reason_18_rule_fail: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_19_SplitMissRuleMatch::test_ffe_reason_19_split_miss_rule_match: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_20_SplitMissDefault::test_ffe_reason_20_split_miss_default: missing_feature - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_21_ActiveWindowSingle::test_ffe_reason_21_active_window_single: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_20_SplitMissDefault::test_ffe_reason_20_split_miss_default: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_21_ActiveWindowSingle::test_ffe_reason_21_active_window_single: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_22_InactiveWindowSingle::test_ffe_reason_22_inactive_window_single: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_23_ActiveWindowMulti::test_ffe_reason_23_active_window_multi: missing_feature - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_24_InactiveWindowMulti::test_ffe_reason_24_inactive_window_multi: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_23_ActiveWindowMulti::test_ffe_reason_23_active_window_multi: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_24_InactiveWindowMulti::test_ffe_reason_24_inactive_window_multi: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_25_WindowActiveRuleMatch::test_ffe_reason_25_window_active_rule_match: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_26_WindowRuleFail::test_ffe_reason_26_window_rule_fail: missing_feature - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_2_ProviderFatal::test_ffe_reason_2_provider_fatal: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_26_WindowRuleFail::test_ffe_reason_26_window_rule_fail: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_2_ProviderFatal::test_ffe_reason_2_provider_fatal: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_3_DisabledFlagNotFound::test_ffe_reason_3_disabled_flag_not_found: v1.61.0 tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_7_TargetingKeyMissing::test_ffe_reason_7_targeting_key_missing: v1.61.0 tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_8_RuleOnlyNoKey::test_ffe_reason_8_rule_only_no_key: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_9_ZeroAllocations::test_ffe_reason_9_zero_allocations: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_9_ZeroAllocations::test_ffe_reason_9_zero_allocations: v1.62.0-SNAPSHOT tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234) tests/integration_frameworks/llm/openai/test_openai_apm.py: v1.61.0 tests/integration_frameworks/llm/openai/test_openai_llmobs.py: v1.61.0 diff --git a/tests/ffe/test_flag_eval_reasons.py b/tests/ffe/test_flag_eval_reasons.py index 3adc73aa859..b8e9351184e 100644 --- a/tests/ffe/test_flag_eval_reasons.py +++ b/tests/ffe/test_flag_eval_reasons.py @@ -37,6 +37,7 @@ REASON-26 DEFAULT (window+rules, fail) → Test_FFE_REASON_26_WindowRuleFail """ +import contextlib from typing import Any from utils import ( @@ -443,11 +444,9 @@ def wait_for_config_401(data: dict) -> bool: # after restoration because each subsequent test class calls # rc.tracer_rc_state.reset().apply() in its own setup, providing a clean-state # guarantee independently of whether this send() has been processed yet. - # Wrapped in try/except so a cleanup failure does not mask the primary exception. - try: + # Wrapped in suppress so a cleanup failure does not mask the primary exception. + with contextlib.suppress(Exception): StaticJsonMockedTracerResponse(path="/v0.7/config", mocked_json={}).send() - except Exception: - pass def test_ffe_reason_2_provider_fatal(self): """REASON-2: 401 from config endpoint → ERROR / PROVIDER_FATAL; coded default returned.""" From 77941d1a2645429ba3d9b2b3ee89b3ab9ff43df9 Mon Sep 17 00:00:00 2001 From: typotter Date: Thu, 23 Apr 2026 14:07:38 -0600 Subject: [PATCH 08/14] chore(ffe): update reason test manifest from Java 1.62.0-SNAPSHOT run REASON-11 (StaticNoSplit) passes at v1.62.0-SNAPSHOT. 11 tests remain missing_feature with failure-mode comments. --- manifests/java.yml | 35 ++++++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/manifests/java.yml b/manifests/java.yml index 4ca35680aff..1c4e7a36ec0 100644 --- a/manifests/java.yml +++ b/manifests/java.yml @@ -3150,26 +3150,39 @@ manifest: tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Targeting::test_ffe_eval_reason_targeting: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional::test_ffe_eval_targeting_key_optional: v1.61.0 + # 9 tests pass against Java ≥ v1.61.0 (REASON-3,7,8,10,13,19,22,25) or v1.62.0-SNAPSHOT (REASON-11). + # 11 tests still fail against Java master (v1.62.0-SNAPSHOT); failure modes noted per test: + # REASON-2 PROVIDER_FATAL: 401 mock times out before tracer enters fatal state + # REASON-9 zero-alloc → DEFAULT: Java returns error/general instead of default + # REASON-14 default-alloc catches rule-miss → DEFAULT: Java returns static + # REASON-16 shard-miss → default-alloc → DEFAULT: Java returns static + # REASON-17 SPLIT overrides TARGETING_MATCH (ADR-004): Java returns targeting_match + # REASON-18 rule-fail → default-alloc → DEFAULT: Java returns static + # REASON-20 split-miss + rule-fail → default-alloc → DEFAULT: Java returns static + # REASON-21 active-window single-alloc → DEFAULT: Java returns static + # REASON-23 active-window multi-alloc → DEFAULT: Java returns static + # REASON-24 inactive-window → default-alloc → DEFAULT: Java returns static + # REASON-26 window+rule-fail → default-alloc → DEFAULT: Java returns static tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_10_NoDefaultAlloc::test_ffe_reason_10_no_default_alloc: v1.61.0 tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_11_StaticNoSplit::test_ffe_reason_11_static_no_split: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_13_MultiAllocRuleMatch::test_ffe_reason_13_multi_alloc_rule_match: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_14_MultiAllocRuleFail::test_ffe_reason_14_multi_alloc_rule_fail: v1.62.0-SNAPSHOT - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_16_RulePassShardMiss::test_ffe_reason_16_rule_pass_shard_miss: v1.62.0-SNAPSHOT - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_17_RulePassShardWin::test_ffe_reason_17_rule_pass_shard_win: v1.62.0-SNAPSHOT - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_18_RuleFail::test_ffe_reason_18_rule_fail: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_14_MultiAllocRuleFail::test_ffe_reason_14_multi_alloc_rule_fail: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_16_RulePassShardMiss::test_ffe_reason_16_rule_pass_shard_miss: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_17_RulePassShardWin::test_ffe_reason_17_rule_pass_shard_win: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_18_RuleFail::test_ffe_reason_18_rule_fail: missing_feature tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_19_SplitMissRuleMatch::test_ffe_reason_19_split_miss_rule_match: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_20_SplitMissDefault::test_ffe_reason_20_split_miss_default: v1.62.0-SNAPSHOT - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_21_ActiveWindowSingle::test_ffe_reason_21_active_window_single: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_20_SplitMissDefault::test_ffe_reason_20_split_miss_default: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_21_ActiveWindowSingle::test_ffe_reason_21_active_window_single: missing_feature tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_22_InactiveWindowSingle::test_ffe_reason_22_inactive_window_single: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_23_ActiveWindowMulti::test_ffe_reason_23_active_window_multi: v1.62.0-SNAPSHOT - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_24_InactiveWindowMulti::test_ffe_reason_24_inactive_window_multi: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_23_ActiveWindowMulti::test_ffe_reason_23_active_window_multi: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_24_InactiveWindowMulti::test_ffe_reason_24_inactive_window_multi: missing_feature tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_25_WindowActiveRuleMatch::test_ffe_reason_25_window_active_rule_match: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_26_WindowRuleFail::test_ffe_reason_26_window_rule_fail: v1.62.0-SNAPSHOT - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_2_ProviderFatal::test_ffe_reason_2_provider_fatal: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_26_WindowRuleFail::test_ffe_reason_26_window_rule_fail: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_2_ProviderFatal::test_ffe_reason_2_provider_fatal: missing_feature tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_3_DisabledFlagNotFound::test_ffe_reason_3_disabled_flag_not_found: v1.61.0 tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_7_TargetingKeyMissing::test_ffe_reason_7_targeting_key_missing: v1.61.0 tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_8_RuleOnlyNoKey::test_ffe_reason_8_rule_only_no_key: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_9_ZeroAllocations::test_ffe_reason_9_zero_allocations: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_9_ZeroAllocations::test_ffe_reason_9_zero_allocations: missing_feature tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234) tests/integration_frameworks/llm/openai/test_openai_apm.py: v1.61.0 tests/integration_frameworks/llm/openai/test_openai_llmobs.py: v1.61.0 From 4fc6b950f94f3a1fbfbedf2cd30073c589be246b Mon Sep 17 00:00:00 2001 From: typotter Date: Thu, 23 Apr 2026 14:29:13 -0600 Subject: [PATCH 09/14] fix(ffe): use RFC canonical splits:[] in REASON-11 fixture Surface SDKs that fail to parse or evaluate an empty splits array. Previously used vacuous-split form which masked this class of bug. --- ffe-reason-ufc-examples.md | 1000 +++++++++++++++++++++++++++ tests/ffe/test_flag_eval_reasons.py | 25 +- 2 files changed, 1010 insertions(+), 15 deletions(-) create mode 100644 ffe-reason-ufc-examples.md diff --git a/ffe-reason-ufc-examples.md b/ffe-reason-ufc-examples.md new file mode 100644 index 00000000000..7abb3e4b78e --- /dev/null +++ b/ffe-reason-ufc-examples.md @@ -0,0 +1,1000 @@ +# FFE Evaluation Reason — UFC Flag Examples + +Reference companion to `reasons-and-errors-rfc.md` Appendix B. + +Each section shows the full UFC payload required to trigger that reason/error combination. +Sections where no flag config is needed are marked accordingly. + +For all examples: +- Flag key: replace with actual key under test +- `variationType` / `variations`: use type appropriate for the test evaluation call +- Caller attributes and `targetingKey` are noted in comments + +--- + +## UFC envelope + +All flag examples below are the value of the flag key inside `"flags"`. +Full envelope: + +```json +{ + "createdAt": "2024-04-17T19:40:53.716Z", + "format": "SERVER", + "environment": {"name": "Test"}, + "flags": { + "my-flag": { "" } + } +} +``` + +--- + +## REASON-1 — ERROR / PROVIDER_NOT_READY + +**No flag config needed.** + +Provider has not yet received any valid configuration. Evaluation occurs before the first successful config fetch. + +``` +No UFC payload — evaluate any flag key before config is loaded. +``` + +--- + +## REASON-2 — ERROR / PROVIDER_FATAL + +**No flag config needed.** + +Provider received a non-retryable 4XX (400, 401, 403, 404) from the assignments endpoint. +All evaluations return the coded default regardless of what flags exist. + +``` +No UFC payload — trigger via a 4XX response from the assignments/config endpoint. +``` + +--- + +## REASON-3 — ERROR / FLAG_NOT_FOUND + +**Requested key is absent from the config.** + +A different flag is present (config is loaded), but the evaluated key does not exist. +Disabled flags are also absent from the precomputed dataset — operationally identical. + +```json +{ + "key": "some-other-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "default-allocation", + "rules": [], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": true + } + ] +} +``` + +``` +Evaluate: "my-flag" (key not present in config) +Result: coded default / ERROR / FLAG_NOT_FOUND +``` + +--- + +## REASON-4 — ERROR / TYPE_MISMATCH + +Flag is configured as `STRING`. Caller requests a `BOOLEAN` evaluation. +Type conversion fails after core evaluation returns. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "default-allocation", + "rules": [], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": true + } + ] +} +``` + +``` +Evaluate: getBooleanValue("my-flag", false) +Result: coded default / ERROR / TYPE_MISMATCH +``` + +--- + +## REASON-5 — ERROR / PARSE_ERROR + +**No flag config achievable via RC mock.** + +Entire config payload must be unparseable (raw non-JSON body). +Per-flag parse failures do not produce this code — SDKs omit the bad flag and return FLAG_NOT_FOUND. +Requires raw body injection below the RC mock layer. + +``` +No UFC payload — requires fault injection (raw non-JSON response body). +Not testable via RC mock framework. +``` + +--- + +## REASON-6 — ERROR / GENERAL + +**No reproducible flag structure.** + +Catch-all for unclassified internal SDK errors. +No flag shape reliably triggers this via the weblog/RC interface. + +``` +No UFC payload — requires SDK internals access or fault injection +below the surface accessible to system tests. +``` + +--- + +## REASON-7 — ERROR / TARGETING_KEY_MISSING + +Allocation has a non-trivial shard (real hash ranges). +Evaluation path reaches shard computation. +No targeting key is provided → hash cannot be computed → error. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "shard-alloc", + "rules": [], + "splits": [ + { + "variationKey": "on", + "shards": [ + { + "salt": "rfc-salt", + "totalShards": 10000, + "ranges": [{"start": 0, "end": 10000}] + } + ] + } + ], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "" (empty / absent) +attributes: {} +Result: coded default / ERROR / TARGETING_KEY_MISSING +``` + +--- + +## REASON-8 — TARGETING_MATCH (rule-only, no shard, no key required) + +Allocation has a targeting rule and a vacuous split (`shards: []`). +Vacuous split skips hash computation entirely → no targeting key required. +Rule matches via subject attribute. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "rule-alloc", + "rules": [ + { + "conditions": [ + { + "operator": "ONE_OF", + "attribute": "tier", + "value": ["gold"] + } + ] + } + ], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "" (empty — no key needed) +attributes: {tier: "gold"} (matches rule) +Result: platform value / TARGETING_MATCH / variant=on / allocationKey=rule-alloc +``` + +--- + +## REASON-9 — DEFAULT (coded default, zero allocations) + +Flag exists in config but has an empty allocations waterfall. +No evaluation possible — ADR-001 data-invariant case. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [] +} +``` + +``` +targetingKey: "user-1" +attributes: {} +Result: coded default / DEFAULT / no error code + variant tag: absent or "n/a" +``` + +--- + +## REASON-10 — DEFAULT (coded default, no default alloc, no match) + +Non-empty waterfall. Rule-based allocation present but subject doesn't match. +No default (unconditional) allocation at the end of the waterfall. +ADR-001 data-invariant case: coded default, DEFAULT, no error. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "rule-only-alloc", + "rules": [ + { + "conditions": [ + { + "operator": "ONE_OF", + "attribute": "tier", + "value": ["platinum"] + } + ] + } + ], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "user-1" +attributes: {tier: "bronze"} (does not match "platinum") +Result: coded default / DEFAULT / no error code + variant tag: absent or "n/a" +``` + +--- + +## REASON-11 — STATIC (no split entry) + +Single allocation. No targeting rules. No date window. +**No split entries** (`splits: []` — empty array, not a split entry with empty shards). + +> **REASON-11 vs REASON-12:** The structural difference is whether a split *entry* exists at all. +> - REASON-11: `splits: []` — no split entries in the array (RFC canonical form) +> - REASON-12: `splits: [{"variationKey": "on", "shards": []}]` — one entry with an empty `shards` array +> +> Both produce STATIC (ADR-003). The mechanism differs: with `splits: []` the SDK finds no split +> entries and skips shard evaluation entirely. With `splits: [{shards:[]}]` the SDK enters split +> evaluation but the empty `shards` array means no hash bucket exists to check, so it matches +> vacuously without computing a hash. An SDK that processes split entries one by one will take a +> different code path for each form; both paths must produce STATIC. +> +> **Note on test code:** The REASON-11 system test (`Test_FFE_REASON_11_StaticNoSplit`) sends the +> RFC canonical form `splits: []` shown below. SDKs that fail to parse or evaluate an empty splits +> array will fail this test; such failures are bugs to be fixed in the SDK. REASON-12 covers the +> vacuous-split form (`splits: [{variationKey: "on", shards: []}]`). + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "static-alloc", + "rules": [], + "splits": [], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "user-1" +attributes: {} +Result: platform value / STATIC / allocationKey=static-alloc +``` + +--- + +## REASON-12 — STATIC (vacuous split, shards:[]) + +Single allocation. No targeting rules. No date window. +Split entry exists but `shards` is an empty array — matches vacuously, no MD5 hash computed. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "default-allocation", + "rules": [], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "user-1" +attributes: {} +Result: platform value / STATIC / allocationKey=default-allocation +``` + +--- + +## REASON-13 — TARGETING_MATCH (multi-alloc, rule matches, no split) + +Multi-allocation waterfall. First allocation has a targeting rule; vacuous split. +Subject attributes match the rule → TARGETING_MATCH. Default allocation is unreached. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "rule-alloc", + "rules": [ + { + "conditions": [ + { + "operator": "ONE_OF", + "attribute": "plan", + "value": ["enterprise"] + } + ] + } + ], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": true + }, + { + "key": "default-alloc", + "rules": [], + "splits": [{"variationKey": "off", "shards": []}], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "user-1" +attributes: {plan: "enterprise"} (matches rule) +Result: platform value / TARGETING_MATCH / variant=on / allocationKey=rule-alloc +``` + +--- + +## REASON-14 — DEFAULT (multi-alloc, rule fails, default catches) + +Same flag as REASON-13. Subject attributes do not match the rule. +Rule-based allocation skipped. Default allocation catches → platform value. + +```json +// Same flag definition as REASON-13 +``` + +``` +targetingKey: "user-1" +attributes: {plan: "starter"} (does not match "enterprise") +Result: platform value / DEFAULT / variant=off / allocationKey=default-alloc +``` + +--- + +## REASON-15 — SPLIT (non-vacuous split, subject bucketed) + +Single allocation. No targeting rules. Real shard ranges with 50/50 split. +Subject is placed in a shard via hash computation → SPLIT. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "split-allocation", + "rules": [], + "splits": [ + { + "variationKey": "on", + "shards": [ + { + "salt": "test-salt", + "totalShards": 10000, + "ranges": [{"start": 0, "end": 5000}] + } + ] + }, + { + "variationKey": "off", + "shards": [ + { + "salt": "test-salt", + "totalShards": 10000, + "ranges": [{"start": 5000, "end": 10000}] + } + ] + } + ], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "user-1" (hash determines bucket) +attributes: {} +Result: platform value / SPLIT / variant determined by hash +``` + +--- + +## REASON-16 — DEFAULT (rule passes, shard guaranteed miss, default catches) + +Allocation has both a targeting rule and a 0% shard (`ranges: []`). +Rule passes. Shard has empty ranges → guaranteed miss. Allocation is skipped. +Default allocation catches. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "rule-shard-alloc", + "rules": [ + { + "conditions": [ + { + "operator": "ONE_OF", + "attribute": "group", + "value": ["beta"] + } + ] + } + ], + "splits": [ + { + "variationKey": "on", + "shards": [ + { + "salt": "rfc-salt", + "totalShards": 10000, + "ranges": [] + } + ] + } + ], + "doLog": true + }, + { + "key": "default-alloc", + "rules": [], + "splits": [{"variationKey": "off", "shards": []}], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "user-1" +attributes: {group: "beta"} (rule passes; 0% shard guarantees miss) +Result: platform value / DEFAULT / variant=off / allocationKey=default-alloc +``` + +--- + +## REASON-17 — SPLIT (rule passes AND shard guaranteed hit) + +Same structure as REASON-16 but shard is 100% (`ranges: [{start:0, end:10000}]`). +Rule passes and shard always hits. Per ADR-004, SPLIT takes precedence over TARGETING_MATCH when +a non-trivial shard resolves the subject — regardless of whether targeting rules were also present. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "rule-shard-alloc", + "rules": [ + { + "conditions": [ + { + "operator": "ONE_OF", + "attribute": "group", + "value": ["alpha"] + } + ] + } + ], + "splits": [ + { + "variationKey": "on", + "shards": [ + { + "salt": "rfc-salt", + "totalShards": 10000, + "ranges": [{"start": 0, "end": 10000}] + } + ] + } + ], + "doLog": true + }, + { + "key": "default-alloc", + "rules": [], + "splits": [{"variationKey": "off", "shards": []}], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "user-1" +attributes: {group: "alpha"} (rule passes; 100% shard always hits) +Result: platform value / SPLIT / variant=on (ADR-004: SPLIT takes precedence over TARGETING_MATCH) +``` + +--- + +## REASON-18 — DEFAULT (rule fails, default catches) + +Same flag as REASON-17. Subject attributes do not match the rule. +Rule fails → allocation skipped entirely (shard is not reached). Default catches. + +```json +// Same flag definition as REASON-17 +``` + +``` +targetingKey: "user-1" +attributes: {group: "beta"} (rule requires "alpha" → fails) +Result: platform value / DEFAULT / variant=off / allocationKey=default-alloc +``` + +--- + +## REASON-19 — TARGETING_MATCH (split alloc skipped/miss, rule alloc 2 matches) + +Three-allocation waterfall: +1. Split-only alloc with 0% shard → guaranteed miss → skipped +2. Rule-based alloc → rule matches → TARGETING_MATCH +3. Default alloc → unreached + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "split-alloc", + "rules": [], + "splits": [ + { + "variationKey": "on", + "shards": [ + { + "salt": "rfc-salt", + "totalShards": 10000, + "ranges": [] + } + ] + } + ], + "doLog": true + }, + { + "key": "rule-alloc", + "rules": [ + { + "conditions": [ + { + "operator": "ONE_OF", + "attribute": "region", + "value": ["us-east"] + } + ] + } + ], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": true + }, + { + "key": "default-alloc", + "rules": [], + "splits": [{"variationKey": "off", "shards": []}], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "user-1" +attributes: {region: "us-east"} (split-alloc: 0% miss; rule-alloc: matches) +Result: platform value / TARGETING_MATCH / variant=on / allocationKey=rule-alloc +``` + +--- + +## REASON-20 — DEFAULT (split alloc skipped/miss, default catches) + +Same flag as REASON-19. Subject doesn't match the rule in alloc 2. +Split misses, rule fails, default catches. + +```json +// Same flag definition as REASON-19 +``` + +``` +targetingKey: "user-1" +attributes: {region: "eu-west"} (split-alloc: 0% miss; rule-alloc: fails) +Result: platform value / DEFAULT / variant=off / allocationKey=default-alloc +``` + +--- + +## REASON-21 — DEFAULT (single alloc, active date window, no rules, no split) + +Single allocation. `startAt`/`endAt` present; window is currently active. +No targeting rules. Vacuous split. +Per ADR-003: presence of `startAt`/`endAt` precludes STATIC → DEFAULT. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "window-alloc", + "startAt": "2020-01-01T00:00:00Z", + "endAt": "2099-01-01T00:00:00Z", + "rules": [], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "user-1" +attributes: {} +Result: platform value / DEFAULT / variant=on + (not STATIC — date window present per ADR-003) +``` + +--- + +## REASON-22 — DEFAULT (coded default, single alloc, inactive window) + +Single allocation. Window has expired. No active allocation matches. +No default allocation present. ADR-001 data-invariant case → coded default, DEFAULT, no error. +Flag IS present in config — expired window is not FLAG_NOT_FOUND. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "window-alloc", + "startAt": "2020-01-01T00:00:00Z", + "endAt": "2020-12-31T00:00:00Z", + "rules": [], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "user-1" +attributes: {} +Result: coded default / DEFAULT / no error code + variant tag: absent or "n/a" (no allocation matched) +``` + +--- + +## REASON-23 — DEFAULT (multi-alloc, first alloc active window, no rules/split) + +First allocation has an active date window, no rules, vacuous split. +Window fires. Per ADR-003: date window precludes STATIC → DEFAULT. +Second allocation (default) is unreached. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "window-alloc", + "startAt": "2020-01-01T00:00:00Z", + "endAt": "2099-01-01T00:00:00Z", + "rules": [], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": true + }, + { + "key": "default-alloc", + "rules": [], + "splits": [{"variationKey": "off", "shards": []}], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "user-1" +attributes: {} +Result: platform value / DEFAULT / variant=on + (variant=on confirms window-alloc fired, not default-alloc) +``` + +--- + +## REASON-24 — DEFAULT (multi-alloc, first alloc window inactive, default catches) + +First allocation window has expired → skipped. +Default allocation catches → platform value. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "window-alloc", + "startAt": "2020-01-01T00:00:00Z", + "endAt": "2020-12-31T00:00:00Z", + "rules": [], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": true + }, + { + "key": "default-alloc", + "rules": [], + "splits": [{"variationKey": "off", "shards": []}], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "user-1" +attributes: {} +Result: platform value / DEFAULT / variant=off + (variant=off distinguishes from REASON-23 where window-alloc fires) +``` + +--- + +## REASON-25 — TARGETING_MATCH (active window + rule matches, no split) + +Allocation has both an active date window and targeting rules. Window is active. +Rule matches → TARGETING_MATCH. +Window alone does not imply TARGETING_MATCH; rules do. + +```json +{ + "key": "my-flag", + "enabled": true, + "variationType": "STRING", + "variations": { + "on": {"key": "on", "value": "on-value"}, + "off": {"key": "off", "value": "off-value"} + }, + "allocations": [ + { + "key": "window-rule-alloc", + "startAt": "2020-01-01T00:00:00Z", + "endAt": "2099-01-01T00:00:00Z", + "rules": [ + { + "conditions": [ + { + "operator": "ONE_OF", + "attribute": "segment", + "value": ["vip"] + } + ] + } + ], + "splits": [{"variationKey": "on", "shards": []}], + "doLog": true + }, + { + "key": "default-alloc", + "rules": [], + "splits": [{"variationKey": "off", "shards": []}], + "doLog": true + } + ] +} +``` + +``` +targetingKey: "user-1" +attributes: {segment: "vip"} (window active; rule matches) +Result: platform value / TARGETING_MATCH / variant=on / allocationKey=window-rule-alloc +``` + +--- + +## REASON-26 — DEFAULT (active window + rule fails, default catches) + +Same flag as REASON-25. Subject doesn't match the rule. +Window is active but rule fails → allocation skipped. Default catches. + +```json +// Same flag definition as REASON-25 +``` + +``` +targetingKey: "user-1" +attributes: {segment: "standard"} (rule requires "vip" → fails) +Result: platform value / DEFAULT / variant=off / allocationKey=default-alloc +``` + +--- + +## Summary + +Return value key: **Platform** = variant from flag config; **Coded** = developer's fallback value (ADR-001: rows 9, 10, 22 return coded default with DEFAULT reason, not an error). + +| # | Flag present | Return | Key shape | Reason | Error code | +|---|---|---|---|---|---| +| 1 | No | Coded | N/A — no config loaded | ERROR | PROVIDER_NOT_READY | +| 2 | No | Coded | N/A — 4XX response | ERROR | PROVIDER_FATAL | +| 3 | No (key absent) | Coded | Different key in config | ERROR | FLAG_NOT_FOUND | +| 4 | Yes | Coded | STRING flag, evaluated as BOOLEAN | ERROR | TYPE_MISMATCH | +| 5 | No | Coded | N/A — unparseable payload | ERROR | PARSE_ERROR | +| 6 | No | Coded | N/A — internal error | ERROR | GENERAL | +| 7 | Yes | Coded | Single alloc, real shard, no rules | ERROR | TARGETING_KEY_MISSING | +| 8 | Yes | Platform | Single alloc, rule, vacuous split | TARGETING_MATCH | — | +| 9 | Yes | **Coded** ¹ | `allocations: []` | DEFAULT | — | +| 10 | Yes | **Coded** ¹ | Rule alloc only, no default alloc | DEFAULT | — | +| 11 | Yes | Platform | Single alloc, `splits: []`, no rules, no window | STATIC | — | +| 12 | Yes | Platform | Single alloc, `splits: [{shards:[]}]`, no rules, no window | STATIC | — | +| 13 | Yes | Platform | Rule alloc + default alloc; rule matches | TARGETING_MATCH | — | +| 14 | Yes | Platform | Rule alloc + default alloc; rule fails | DEFAULT | — | +| 15 | Yes | Platform | Single alloc, real shard ranges, no rules | SPLIT | — | +| 16 | Yes | Platform | Rule + 0% shard alloc + default alloc; rule passes, shard misses | DEFAULT | — | +| 17 | Yes | Platform | Rule + 100% shard alloc + default alloc; rule + shard win | SPLIT | — | +| 18 | Yes | Platform | Rule + 100% shard alloc + default alloc; rule fails | DEFAULT | — | +| 19 | Yes | Platform | 0% shard alloc → rule alloc → default alloc; rule matches | TARGETING_MATCH | — | +| 20 | Yes | Platform | 0% shard alloc → rule alloc → default alloc; rule fails | DEFAULT | — | +| 21 | Yes | Platform | Single alloc, active window, no rules, vacuous split | DEFAULT | — | +| 22 | Yes | **Coded** ¹ | Single alloc, expired window | DEFAULT | — | +| 23 | Yes | Platform | Active-window alloc + default alloc; window fires | DEFAULT | — | +| 24 | Yes | Platform | Expired-window alloc + default alloc; default catches | DEFAULT | — | +| 25 | Yes | Platform | Active-window + rule alloc + default alloc; rule matches | TARGETING_MATCH | — | +| 26 | Yes | Platform | Active-window + rule alloc + default alloc; rule fails | DEFAULT | — | + +¹ ADR-001 exception: coded default with DEFAULT reason and no error code. These are data-invariant violations (empty waterfall / no allocation matched / expired single window with no fallback), not SDK errors. diff --git a/tests/ffe/test_flag_eval_reasons.py b/tests/ffe/test_flag_eval_reasons.py index b8e9351184e..89242ea337a 100644 --- a/tests/ffe/test_flag_eval_reasons.py +++ b/tests/ffe/test_flag_eval_reasons.py @@ -166,19 +166,17 @@ def make_no_default_alloc_fixture(flag_key: str, attribute: str, match_value: st def make_pure_static_fixture(flag_key: str) -> dict: """REASON-11: Single allocation, no rules, no date window → STATIC. - Uses shards:[] (vacuous split) which, like an absent shards key, means no - hash computation is performed. Both forms produce STATIC per ADR-003. We use - the vacuous-split form (shards:[]) because it is the established pattern across - the codebase and is guaranteed parseable by all SDK implementations. A split - entry with a missing shards field is not tested in any other fixture and may - trigger a parse error in strict-deserialisation SDKs. + Uses the RFC canonical form: splits:[] (empty array, no split entries at all). + This is intentionally different from the vacuous-split form used elsewhere + (splits:[{variationKey:"on", shards:[]}]). SDKs that reject an empty splits + array will fail this test, surfacing a parse/evaluation bug to be fixed. """ fd = _base_flag(flag_key) fd["allocations"] = [ { "key": "static-alloc", "rules": [], - "splits": [{"variationKey": "on", "shards": []}], + "splits": [], "doLog": True, } ] @@ -788,14 +786,11 @@ def test_ffe_reason_10_no_default_alloc(self): class Test_FFE_REASON_11_StaticNoSplit: """REASON-11: Single allocation; no targeting rules, no date window → STATIC. - The RFC distinguishes REASON-11 ("no split") from REASON-12 ("vacuous split, shards:[]"), - but in the UFC format used by this framework both are represented as a split entry - with an empty shards array — omitting the shards key entirely is not tested across - SDKs and may trigger strict-deserialisation errors. The fixture therefore uses the - same vacuous-split form as REASON-12. Both forms produce STATIC per ADR-003 (no hash - computation, no targeting rules, no date window). This test exercises the STATIC - path via a distinct config ID and flag key, providing independent signal from the - REASON-12 coverage in test_flag_eval_metrics.py :: Test_FFE_Eval_Metric_Basic. + Uses RFC canonical form splits:[] (empty array). This is the structural distinction + from REASON-12 which uses splits:[{variationKey:"on", shards:[]}] (vacuous split). + Both produce STATIC per ADR-003. SDKs that fail to parse or evaluate an empty splits + array will surface a bug here; that is the intent — failures are expected to be fixed + in the SDK, not papered over by switching to the vacuous-split form. """ def setup_ffe_reason_11_static_no_split(self): From 7aeae2ccd4529b947fb0152455550c6bb999cb26 Mon Sep 17 00:00:00 2001 From: typotter Date: Thu, 23 Apr 2026 15:26:45 -0600 Subject: [PATCH 10/14] =?UTF-8?q?fix(ffe):=20REASON-11=20splits:[]=20?= =?UTF-8?q?=E2=86=92=20allocation=20skipped=20=E2=86=92=20DEFAULT?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An allocation with splits:[] cannot produce a variant. Correct expected result is coded default / DEFAULT (not STATIC). Update test assertions, fixture docstring, and UFC examples doc accordingly. --- ffe-reason-ufc-examples.md | 27 +++++++++---------- tests/ffe/test_flag_eval_reasons.py | 42 +++++++++++++++-------------- 2 files changed, 34 insertions(+), 35 deletions(-) diff --git a/ffe-reason-ufc-examples.md b/ffe-reason-ufc-examples.md index 7abb3e4b78e..f0f5e428d85 100644 --- a/ffe-reason-ufc-examples.md +++ b/ffe-reason-ufc-examples.md @@ -311,25 +311,21 @@ Result: coded default / DEFAULT / no error code --- -## REASON-11 — STATIC (no split entry) +## REASON-11 — DEFAULT (no split entry → allocation skipped → waterfall exhausted) Single allocation. No targeting rules. No date window. **No split entries** (`splits: []` — empty array, not a split entry with empty shards). +An allocation with `splits: []` cannot produce a variant — there is no split entry to resolve a +variation key. The allocation is skipped even though rules pass. With no subsequent allocation, the +waterfall is exhausted → coded default / DEFAULT / no error code. + > **REASON-11 vs REASON-12:** The structural difference is whether a split *entry* exists at all. -> - REASON-11: `splits: []` — no split entries in the array (RFC canonical form) -> - REASON-12: `splits: [{"variationKey": "on", "shards": []}]` — one entry with an empty `shards` array -> -> Both produce STATIC (ADR-003). The mechanism differs: with `splits: []` the SDK finds no split -> entries and skips shard evaluation entirely. With `splits: [{shards:[]}]` the SDK enters split -> evaluation but the empty `shards` array means no hash bucket exists to check, so it matches -> vacuously without computing a hash. An SDK that processes split entries one by one will take a -> different code path for each form; both paths must produce STATIC. +> - REASON-11: `splits: []` — no split entries; allocation cannot resolve a variant; allocation skipped → DEFAULT (coded default) +> - REASON-12: `splits: [{"variationKey": "on", "shards": []}]` — one entry with empty `shards`; resolves vacuously → STATIC (platform value) > -> **Note on test code:** The REASON-11 system test (`Test_FFE_REASON_11_StaticNoSplit`) sends the -> RFC canonical form `splits: []` shown below. SDKs that fail to parse or evaluate an empty splits -> array will fail this test; such failures are bugs to be fixed in the SDK. REASON-12 covers the -> vacuous-split form (`splits: [{variationKey: "on", shards: []}]`). +> These produce **different results**. REASON-11 exhausts the waterfall without selecting a variant. +> REASON-12 selects the variation from the split entry without hashing. ```json { @@ -354,7 +350,8 @@ Single allocation. No targeting rules. No date window. ``` targetingKey: "user-1" attributes: {} -Result: platform value / STATIC / allocationKey=static-alloc +Result: coded default / DEFAULT / no error code + variant tag: absent or "n/a" (allocation skipped; no variation resolved) ``` --- @@ -980,7 +977,7 @@ Return value key: **Platform** = variant from flag config; **Coded** = developer | 8 | Yes | Platform | Single alloc, rule, vacuous split | TARGETING_MATCH | — | | 9 | Yes | **Coded** ¹ | `allocations: []` | DEFAULT | — | | 10 | Yes | **Coded** ¹ | Rule alloc only, no default alloc | DEFAULT | — | -| 11 | Yes | Platform | Single alloc, `splits: []`, no rules, no window | STATIC | — | +| 11 | Yes | **Coded** ¹ | Single alloc, `splits: []`, no rules, no window — alloc skipped, waterfall exhausted | DEFAULT | — | | 12 | Yes | Platform | Single alloc, `splits: [{shards:[]}]`, no rules, no window | STATIC | — | | 13 | Yes | Platform | Rule alloc + default alloc; rule matches | TARGETING_MATCH | — | | 14 | Yes | Platform | Rule alloc + default alloc; rule fails | DEFAULT | — | diff --git a/tests/ffe/test_flag_eval_reasons.py b/tests/ffe/test_flag_eval_reasons.py index 89242ea337a..eb19825352e 100644 --- a/tests/ffe/test_flag_eval_reasons.py +++ b/tests/ffe/test_flag_eval_reasons.py @@ -18,8 +18,7 @@ REASON-8 TARGETING_MATCH (no key) → Test_FFE_REASON_8_RuleOnlyNoKey REASON-9 zero allocations → DEFAULT → Test_FFE_REASON_9_ZeroAllocations REASON-10 no-default-alloc → DEFAULT → Test_FFE_REASON_10_NoDefaultAlloc - REASON-11 STATIC (no split/rules) → Test_FFE_REASON_11_StaticNoSplit (uses vacuous split in UFC, - structurally same as REASON-12; see fixture docstring) + REASON-11 splits:[] → alloc skipped → DEFAULT → Test_FFE_REASON_11_StaticNoSplit REASON-12 STATIC (vacuous split) → test_flag_eval_metrics.py :: Test_FFE_Eval_Metric_Basic REASON-13 TARGETING_MATCH multi → Test_FFE_REASON_13_MultiAllocRuleMatch REASON-14 DEFAULT multi (rule fails) → Test_FFE_REASON_14_MultiAllocRuleFail @@ -164,12 +163,13 @@ def make_no_default_alloc_fixture(flag_key: str, attribute: str, match_value: st def make_pure_static_fixture(flag_key: str) -> dict: - """REASON-11: Single allocation, no rules, no date window → STATIC. + """REASON-11: Single allocation, no rules, no date window, splits:[]. - Uses the RFC canonical form: splits:[] (empty array, no split entries at all). - This is intentionally different from the vacuous-split form used elsewhere - (splits:[{variationKey:"on", shards:[]}]). SDKs that reject an empty splits - array will fail this test, surfacing a parse/evaluation bug to be fixed. + Uses RFC canonical form: splits:[] (empty array). An allocation with no split + entries cannot produce a variant — the allocation is skipped even if rules pass. + With no subsequent allocation, the waterfall is exhausted → coded default / DEFAULT. + This is distinct from the vacuous-split form (splits:[{shards:[]}]) which does + resolve a variation (STATIC). See REASON-12 for the vacuous-split path. """ fd = _base_flag(flag_key) fd["allocations"] = [ @@ -784,13 +784,14 @@ def test_ffe_reason_10_no_default_alloc(self): @scenarios.feature_flagging_and_experimentation @features.feature_flags_eval_metrics class Test_FFE_REASON_11_StaticNoSplit: - """REASON-11: Single allocation; no targeting rules, no date window → STATIC. + """REASON-11: Single allocation; no rules, no date window; splits:[] → DEFAULT (coded default). - Uses RFC canonical form splits:[] (empty array). This is the structural distinction - from REASON-12 which uses splits:[{variationKey:"on", shards:[]}] (vacuous split). - Both produce STATIC per ADR-003. SDKs that fail to parse or evaluate an empty splits - array will surface a bug here; that is the intent — failures are expected to be fixed - in the SDK, not papered over by switching to the vacuous-split form. + An allocation with splits:[] cannot produce a variant — no split entry exists to + resolve a variation key. The allocation is skipped even though rules pass. With no + subsequent allocation, the waterfall is exhausted → coded default / DEFAULT / no error. + + This is the structural distinction from REASON-12 (splits:[{shards:[]}]), where a + split entry exists and resolves vacuously → STATIC with a platform value. """ def setup_ffe_reason_11_static_no_split(self): @@ -812,7 +813,7 @@ def setup_ffe_reason_11_static_no_split(self): ) def test_ffe_reason_11_static_no_split(self): - """REASON-11: Single alloc, no rules, no split → STATIC (platform value).""" + """REASON-11: Single alloc, splits:[] → allocation skipped → DEFAULT (coded default).""" assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" metrics = find_eval_metrics(self.flag_key) @@ -824,14 +825,15 @@ def test_ffe_reason_11_static_no_split(self): assert get_tag_value(tags, "feature_flag.key") == self.flag_key, ( f"REASON-11: Expected feature_flag.key={self.flag_key}, got tags: {tags}" ) - assert get_tag_value(tags, "feature_flag.result.reason") == "static", ( - f"REASON-11: Expected reason=static, got tags: {tags}" + assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( + f"REASON-11: Expected reason=default (splits:[] → alloc skipped → waterfall exhausted), got tags: {tags}" ) - assert get_tag_value(tags, "feature_flag.result.variant") == "on", ( - f"REASON-11: Expected variant=on (static-alloc fired), got tags: {tags}" + assert get_tag_value(tags, "error.type") is None, ( + f"REASON-11: Expected no error.type (ADR-001: not an SDK error), got tags: {tags}" ) - assert get_tag_value(tags, "feature_flag.result.allocation_key") == "static-alloc", ( - f"REASON-11: Expected allocation_key=static-alloc, got tags: {tags}" + # No split entry → no variation resolved → no platform variant. Absent or sentinel. + assert get_tag_value(tags, "feature_flag.result.variant") in (None, "n/a"), ( + f"REASON-11: Expected no platform variant (allocation skipped, coded default returned), got tags: {tags}" ) From 0ec9c6daacd7bccf8be977072f0160d7cf07bc73 Mon Sep 17 00:00:00 2001 From: typotter Date: Thu, 23 Apr 2026 15:27:13 -0600 Subject: [PATCH 11/14] chore(ffe): update manifest comment for REASON-11 semantics --- manifests/java.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manifests/java.yml b/manifests/java.yml index 1c4e7a36ec0..754cf5e9a03 100644 --- a/manifests/java.yml +++ b/manifests/java.yml @@ -3150,7 +3150,7 @@ manifest: tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Targeting::test_ffe_eval_reason_targeting: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional::test_ffe_eval_targeting_key_optional: v1.61.0 - # 9 tests pass against Java ≥ v1.61.0 (REASON-3,7,8,10,13,19,22,25) or v1.62.0-SNAPSHOT (REASON-11). + # 9 tests pass against Java ≥ v1.61.0 (REASON-3,7,8,10,13,19,22,25) or v1.62.0-SNAPSHOT (REASON-11: splits:[]→DEFAULT). # 11 tests still fail against Java master (v1.62.0-SNAPSHOT); failure modes noted per test: # REASON-2 PROVIDER_FATAL: 401 mock times out before tracer enters fatal state # REASON-9 zero-alloc → DEFAULT: Java returns error/general instead of default From e0a9c8340e34881ba13b884329de5cb021b4d127 Mon Sep 17 00:00:00 2001 From: typotter Date: Thu, 23 Apr 2026 15:35:32 -0600 Subject: [PATCH 12/14] fix(ffe): REASON-11 is shard miss, not empty splits array MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 'No matching split' means the allocation has splits with shards but the targeting key falls outside every shard range — allocation skipped → DEFAULT. The empty splits:[] form is an unexpected structural edge case covered by a separate spec point. Fixture: salt 'b11-static-no-match' + range [3921,10000) guarantees user-1 (shard 3920) deterministically misses. Manifest reset to missing_feature pending a verified test run against Java master. --- manifests/java.yml | 7 +-- tests/ffe/test_flag_eval_reasons.py | 67 ++++++++++++++++++----------- 2 files changed, 46 insertions(+), 28 deletions(-) diff --git a/manifests/java.yml b/manifests/java.yml index 754cf5e9a03..b5ccc0d0be2 100644 --- a/manifests/java.yml +++ b/manifests/java.yml @@ -3150,8 +3150,9 @@ manifest: tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Targeting::test_ffe_eval_reason_targeting: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional::test_ffe_eval_targeting_key_optional: v1.61.0 - # 9 tests pass against Java ≥ v1.61.0 (REASON-3,7,8,10,13,19,22,25) or v1.62.0-SNAPSHOT (REASON-11: splits:[]→DEFAULT). - # 11 tests still fail against Java master (v1.62.0-SNAPSHOT); failure modes noted per test: + # 8 tests pass against Java ≥ v1.61.0 (REASON-3,7,8,10,13,19,22,25). + # 12 tests still fail against Java master (v1.62.0-SNAPSHOT); failure modes noted per test: + # REASON-11 shard-miss single-alloc → DEFAULT: fixture corrected (was vacuous-shard), pending re-run # REASON-2 PROVIDER_FATAL: 401 mock times out before tracer enters fatal state # REASON-9 zero-alloc → DEFAULT: Java returns error/general instead of default # REASON-14 default-alloc catches rule-miss → DEFAULT: Java returns static @@ -3164,7 +3165,7 @@ manifest: # REASON-24 inactive-window → default-alloc → DEFAULT: Java returns static # REASON-26 window+rule-fail → default-alloc → DEFAULT: Java returns static tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_10_NoDefaultAlloc::test_ffe_reason_10_no_default_alloc: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_11_StaticNoSplit::test_ffe_reason_11_static_no_split: v1.62.0-SNAPSHOT + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_11_StaticNoMatchingSplit::test_ffe_reason_11_static_no_matching_split: missing_feature tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_13_MultiAllocRuleMatch::test_ffe_reason_13_multi_alloc_rule_match: v1.61.0 tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_14_MultiAllocRuleFail::test_ffe_reason_14_multi_alloc_rule_fail: missing_feature tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_16_RulePassShardMiss::test_ffe_reason_16_rule_pass_shard_miss: missing_feature diff --git a/tests/ffe/test_flag_eval_reasons.py b/tests/ffe/test_flag_eval_reasons.py index eb19825352e..5701bab2ab4 100644 --- a/tests/ffe/test_flag_eval_reasons.py +++ b/tests/ffe/test_flag_eval_reasons.py @@ -18,7 +18,7 @@ REASON-8 TARGETING_MATCH (no key) → Test_FFE_REASON_8_RuleOnlyNoKey REASON-9 zero allocations → DEFAULT → Test_FFE_REASON_9_ZeroAllocations REASON-10 no-default-alloc → DEFAULT → Test_FFE_REASON_10_NoDefaultAlloc - REASON-11 splits:[] → alloc skipped → DEFAULT → Test_FFE_REASON_11_StaticNoSplit + REASON-11 no matching split → alloc skipped → DEFAULT → Test_FFE_REASON_11_StaticNoMatchingSplit REASON-12 STATIC (vacuous split) → test_flag_eval_metrics.py :: Test_FFE_Eval_Metric_Basic REASON-13 TARGETING_MATCH multi → Test_FFE_REASON_13_MultiAllocRuleMatch REASON-14 DEFAULT multi (rule fails) → Test_FFE_REASON_14_MultiAllocRuleFail @@ -162,21 +162,36 @@ def make_no_default_alloc_fixture(flag_key: str, attribute: str, match_value: st return _wrap_fixture(flag_key, fd) -def make_pure_static_fixture(flag_key: str) -> dict: - """REASON-11: Single allocation, no rules, no date window, splits:[]. +def make_shard_miss_static_fixture(flag_key: str) -> dict: + """REASON-11: Single allocation, no rules, no date window; split shards don't match → DEFAULT. - Uses RFC canonical form: splits:[] (empty array). An allocation with no split - entries cannot produce a variant — the allocation is skipped even if rules pass. - With no subsequent allocation, the waterfall is exhausted → coded default / DEFAULT. - This is distinct from the vacuous-split form (splits:[{shards:[]}]) which does - resolve a variation (STATIC). See REASON-12 for the vacuous-split path. + The allocation has one split with shards, but the targeting key falls outside every + shard range. No split matches → allocation is skipped → waterfall exhausted → + coded default / DEFAULT. + + Salt "b11-static-no-match" maps "user-1" to shard 3920 (MD5-based, totalShards=10000). + Range [3921, 10000) excludes shard 3920, guaranteeing a deterministic miss. + + Distinct from the empty-splits-array edge case (separate spec point) and from + REASON-12 (vacuous split: shards:[] → resolves without hash → STATIC). """ fd = _base_flag(flag_key) fd["allocations"] = [ { "key": "static-alloc", "rules": [], - "splits": [], + "splits": [ + { + "variationKey": "on", + "shards": [ + { + "salt": "b11-static-no-match", + "totalShards": 10000, + "ranges": [{"start": 3921, "end": 10000}], + } + ], + } + ], "doLog": True, } ] @@ -777,29 +792,31 @@ def test_ffe_reason_10_no_default_alloc(self): # --------------------------------------------------------------------------- -# REASON-11: STATIC — single allocation, no rules, no date window +# REASON-11: DEFAULT — single allocation, no rules, no date window, shard miss # --------------------------------------------------------------------------- @scenarios.feature_flagging_and_experimentation @features.feature_flags_eval_metrics -class Test_FFE_REASON_11_StaticNoSplit: - """REASON-11: Single allocation; no rules, no date window; splits:[] → DEFAULT (coded default). +class Test_FFE_REASON_11_StaticNoMatchingSplit: + """REASON-11: Single allocation; no rules, no date window; split shards don't match → DEFAULT. - An allocation with splits:[] cannot produce a variant — no split entry exists to - resolve a variation key. The allocation is skipped even though rules pass. With no - subsequent allocation, the waterfall is exhausted → coded default / DEFAULT / no error. + The allocation has one split with shards, but the targeting key falls outside every + shard range. No split matches → allocation skipped → waterfall exhausted → + coded default / DEFAULT / no error. - This is the structural distinction from REASON-12 (splits:[{shards:[]}]), where a - split entry exists and resolves vacuously → STATIC with a platform value. + Distinct from REASON-12 (vacuous split: shards:[] → resolves without hash → STATIC) + and from the empty-splits-array edge case (separate spec point). """ - def setup_ffe_reason_11_static_no_split(self): + def setup_ffe_reason_11_static_no_matching_split(self): rc.tracer_rc_state.reset().apply() config_id = "ffe-b11-static" - self.flag_key = "b11-static-no-split-flag" - rc.tracer_rc_state.set_config(f"{RC_PATH}/{config_id}/config", make_pure_static_fixture(self.flag_key)).apply() + self.flag_key = "b11-static-no-matching-split-flag" + rc.tracer_rc_state.set_config( + f"{RC_PATH}/{config_id}/config", make_shard_miss_static_fixture(self.flag_key) + ).apply() self.r = weblog.post( "/ffe", @@ -812,8 +829,8 @@ def setup_ffe_reason_11_static_no_split(self): }, ) - def test_ffe_reason_11_static_no_split(self): - """REASON-11: Single alloc, splits:[] → allocation skipped → DEFAULT (coded default).""" + def test_ffe_reason_11_static_no_matching_split(self): + """REASON-11: Single alloc; shards don't match → allocation skipped → DEFAULT (coded default).""" assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}" metrics = find_eval_metrics(self.flag_key) @@ -826,14 +843,14 @@ def test_ffe_reason_11_static_no_split(self): f"REASON-11: Expected feature_flag.key={self.flag_key}, got tags: {tags}" ) assert get_tag_value(tags, "feature_flag.result.reason") == "default", ( - f"REASON-11: Expected reason=default (splits:[] → alloc skipped → waterfall exhausted), got tags: {tags}" + f"REASON-11: Expected reason=default (shard miss → alloc skipped → waterfall exhausted), got tags: {tags}" ) assert get_tag_value(tags, "error.type") is None, ( f"REASON-11: Expected no error.type (ADR-001: not an SDK error), got tags: {tags}" ) - # No split entry → no variation resolved → no platform variant. Absent or sentinel. + # No split matched → no variation resolved → no platform variant. assert get_tag_value(tags, "feature_flag.result.variant") in (None, "n/a"), ( - f"REASON-11: Expected no platform variant (allocation skipped, coded default returned), got tags: {tags}" + f"REASON-11: Expected no platform variant (shard miss, coded default returned), got tags: {tags}" ) From 509dd81d9dcca9be34047ddab574f164d88c61ef Mon Sep 17 00:00:00 2001 From: typotter Date: Thu, 23 Apr 2026 15:36:19 -0600 Subject: [PATCH 13/14] =?UTF-8?q?fix(ffe):=20REASON-11=20=3D=20no=20matchi?= =?UTF-8?q?ng=20split=20(shard=20miss=20=E2=86=92=20DEFAULT)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "No split" means no matching split — the split entry exists but the subject's hash lands outside all shard ranges. Allocation skipped, waterfall exhausted → coded default / DEFAULT. Distinct from REASON-12 (vacuous split → STATIC) and from the empty-splits-array edge case (separate spec point). Rename fixture to make_shard_miss_static_fixture, class to Test_FFE_REASON_11_StaticNoMatchingSplit. Update manifest and UFC examples doc. --- ffe-reason-ufc-examples.md | 48 +++++++++++++++++++++++++------------- 1 file changed, 32 insertions(+), 16 deletions(-) diff --git a/ffe-reason-ufc-examples.md b/ffe-reason-ufc-examples.md index f0f5e428d85..ca1e9d95e8e 100644 --- a/ffe-reason-ufc-examples.md +++ b/ffe-reason-ufc-examples.md @@ -311,21 +311,26 @@ Result: coded default / DEFAULT / no error code --- -## REASON-11 — DEFAULT (no split entry → allocation skipped → waterfall exhausted) +## REASON-11 — DEFAULT (no matching split → allocation not selected → waterfall exhausted) Single allocation. No targeting rules. No date window. -**No split entries** (`splits: []` — empty array, not a split entry with empty shards). - -An allocation with `splits: []` cannot produce a variant — there is no split entry to resolve a -variation key. The allocation is skipped even though rules pass. With no subsequent allocation, the -waterfall is exhausted → coded default / DEFAULT / no error code. - -> **REASON-11 vs REASON-12:** The structural difference is whether a split *entry* exists at all. -> - REASON-11: `splits: []` — no split entries; allocation cannot resolve a variant; allocation skipped → DEFAULT (coded default) -> - REASON-12: `splits: [{"variationKey": "on", "shards": []}]` — one entry with empty `shards`; resolves vacuously → STATIC (platform value) +Split entry present, but the subject's hash does not land in any shard range → allocation not +selected. With no subsequent allocation, the waterfall is exhausted → coded default / DEFAULT / no error code. + +> **REASON-11 vs REASON-12:** The structural difference is whether the split matches the subject. +> - REASON-11: Split entry exists but subject falls outside all shard ranges (`ranges: []` is the +> canonical deterministic form — 0% shard, guaranteed miss). The allocation is evaluated but does +> not select a variant. Waterfall exhausted → coded default / DEFAULT. +> - REASON-12: `splits: [{"variationKey": "on", "shards": []}]` — vacuous split, `shards: []` means +> no hash bucket boundaries to check; matches unconditionally without computing a hash → STATIC +> (platform value). > > These produce **different results**. REASON-11 exhausts the waterfall without selecting a variant. -> REASON-12 selects the variation from the split entry without hashing. +> REASON-12 always selects the variation key without hashing. +> +> Note: `splits: []` (empty splits array — no split entries at all) produces the same operational +> result as REASON-11 but is an unexpected/degenerate config shape, not the canonical fixture for +> this case. ```json { @@ -338,9 +343,20 @@ waterfall is exhausted → coded default / DEFAULT / no error code. }, "allocations": [ { - "key": "static-alloc", + "key": "no-match-alloc", "rules": [], - "splits": [], + "splits": [ + { + "variationKey": "on", + "shards": [ + { + "salt": "rfc-salt", + "totalShards": 10000, + "ranges": [] + } + ] + } + ], "doLog": true } ] @@ -351,7 +367,7 @@ waterfall is exhausted → coded default / DEFAULT / no error code. targetingKey: "user-1" attributes: {} Result: coded default / DEFAULT / no error code - variant tag: absent or "n/a" (allocation skipped; no variation resolved) + variant tag: absent or "n/a" (shard miss; no variation selected) ``` --- @@ -963,7 +979,7 @@ Result: platform value / DEFAULT / variant=off / allocationKey=default-all ## Summary -Return value key: **Platform** = variant from flag config; **Coded** = developer's fallback value (ADR-001: rows 9, 10, 22 return coded default with DEFAULT reason, not an error). +Return value key: **Platform** = variant from flag config; **Coded** = developer's fallback value (ADR-001: rows 9, 10, 11, 22 return coded default with DEFAULT reason, not an error). | # | Flag present | Return | Key shape | Reason | Error code | |---|---|---|---|---|---| @@ -977,7 +993,7 @@ Return value key: **Platform** = variant from flag config; **Coded** = developer | 8 | Yes | Platform | Single alloc, rule, vacuous split | TARGETING_MATCH | — | | 9 | Yes | **Coded** ¹ | `allocations: []` | DEFAULT | — | | 10 | Yes | **Coded** ¹ | Rule alloc only, no default alloc | DEFAULT | — | -| 11 | Yes | **Coded** ¹ | Single alloc, `splits: []`, no rules, no window — alloc skipped, waterfall exhausted | DEFAULT | — | +| 11 | Yes | **Coded** ¹ | Single alloc, 0% shard (ranges:[]), no rules, no window — shard miss, waterfall exhausted | DEFAULT | — | | 12 | Yes | Platform | Single alloc, `splits: [{shards:[]}]`, no rules, no window | STATIC | — | | 13 | Yes | Platform | Rule alloc + default alloc; rule matches | TARGETING_MATCH | — | | 14 | Yes | Platform | Rule alloc + default alloc; rule fails | DEFAULT | — | From 1e776e560b895c10d1e121ea6de1cb43eeb325c1 Mon Sep 17 00:00:00 2001 From: typotter Date: Thu, 23 Apr 2026 15:52:26 -0600 Subject: [PATCH 14/14] chore(ffe): activate REASON-11 at v1.62.0-SNAPSHOT MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Shard-miss fixture (no matching split → DEFAULT) verified passing against Java 1.62.0-SNAPSHOT. 9 REASON tests now active. --- manifests/java.yml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/manifests/java.yml b/manifests/java.yml index b5ccc0d0be2..e70c716d8e5 100644 --- a/manifests/java.yml +++ b/manifests/java.yml @@ -3150,9 +3150,8 @@ manifest: tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Targeting::test_ffe_eval_reason_targeting: v1.61.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional::test_ffe_eval_targeting_key_optional: v1.61.0 - # 8 tests pass against Java ≥ v1.61.0 (REASON-3,7,8,10,13,19,22,25). - # 12 tests still fail against Java master (v1.62.0-SNAPSHOT); failure modes noted per test: - # REASON-11 shard-miss single-alloc → DEFAULT: fixture corrected (was vacuous-shard), pending re-run + # 9 tests pass: REASON-3,7,8,10,11,13,19,22,25 (v1.61.0) or REASON-11 (v1.62.0-SNAPSHOT). + # 11 tests still fail against Java master (v1.62.0-SNAPSHOT); failure modes noted per test: # REASON-2 PROVIDER_FATAL: 401 mock times out before tracer enters fatal state # REASON-9 zero-alloc → DEFAULT: Java returns error/general instead of default # REASON-14 default-alloc catches rule-miss → DEFAULT: Java returns static @@ -3165,7 +3164,7 @@ manifest: # REASON-24 inactive-window → default-alloc → DEFAULT: Java returns static # REASON-26 window+rule-fail → default-alloc → DEFAULT: Java returns static tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_10_NoDefaultAlloc::test_ffe_reason_10_no_default_alloc: v1.61.0 - tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_11_StaticNoMatchingSplit::test_ffe_reason_11_static_no_matching_split: missing_feature + tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_11_StaticNoMatchingSplit::test_ffe_reason_11_static_no_matching_split: v1.62.0-SNAPSHOT tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_13_MultiAllocRuleMatch::test_ffe_reason_13_multi_alloc_rule_match: v1.61.0 tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_14_MultiAllocRuleFail::test_ffe_reason_14_multi_alloc_rule_fail: missing_feature tests/ffe/test_flag_eval_reasons.py::Test_FFE_REASON_16_RulePassShardMiss::test_ffe_reason_16_rule_pass_shard_miss: missing_feature