Context
The fractional-nested-var test scenarios in evaluator/gherkin/fractional.feature test the case where var("color") resolves to a value (e.g., "yellow" or "") that is not a valid variant name. The current test only asserts the resolved value equals the fallback:
| jon@company.com | yellow | fallback |
| jon@company.com | | fallback |
Problem
Every evaluator implementation returns a GENERAL error when the resolved variant doesn't exist in the flag's variants map. The test step definitions in each SDK testkit work around this by catching the error and substituting the fallback value (mimicking SDK client behavior):
Python (evaluation_steps.py):
except OpenFeatureError as e:
return FlagResolutionDetails(default_value, error_code=e.error_code, reason=Reason.ERROR)Java (EvaluationSteps.java):
catch (OpenFeatureError e) {
state.evaluation = ProviderEvaluation.builder()
.value(state.defaultValue)
.errorCode(e.getErrorCode())
.build();
}Go/flagd (gherkin_test.go):
if errCode != model.FlagNotFoundErrorCode && errCode != model.TypeMismatchErrorCode {
tc.resultValue = parseTypedValue(tc.flagType, tc.defaultVal)
tc.resultReason = model.DefaultReason
tc.resultError = nil
}All three implementations converge on the same workaround: catch the error, return the fallback value, and effectively hide the error from the assertion.
Proposal
The Gherkin tests should make the expected behavior explicit. Options:
- Assert the error code: Add
And the error-code should be "GENERAL" to make it clear an error is expected
- Assert the reason: Add
And the reason should be "ERROR"
- Define evaluator-level fallback behavior: If the evaluator should return the fallback value (not an error), document this and update evaluator implementations accordingly
This would eliminate the need for SDK-level workarounds in all testkits and make the expected behavior unambiguous.
Affected implementations
Context
The
fractional-nested-vartest scenarios inevaluator/gherkin/fractional.featuretest the case wherevar("color")resolves to a value (e.g.,"yellow"or"") that is not a valid variant name. The current test only asserts the resolved value equals the fallback:Problem
Every evaluator implementation returns a GENERAL error when the resolved variant doesn't exist in the flag's variants map. The test step definitions in each SDK testkit work around this by catching the error and substituting the fallback value (mimicking SDK client behavior):
Python (
evaluation_steps.py):Java (
EvaluationSteps.java):Go/flagd (
gherkin_test.go):All three implementations converge on the same workaround: catch the error, return the fallback value, and effectively hide the error from the assertion.
Proposal
The Gherkin tests should make the expected behavior explicit. Options:
And the error-code should be "GENERAL"to make it clear an error is expectedAnd the reason should be "ERROR"This would eliminate the need for SDK-level workarounds in all testkits and make the expected behavior unambiguous.
Affected implementations
tools/flagd-api-testkitin java-sdk-contrib (on main)