[FR] [DaC] Add fine-grained bypass env var for ES|QL keep and metadata validation

[eric-forte-elastic]

Pull Request

Issue link(s): Resolves #5868

Related to: elastic/DaC-Reference#53

Summary - What I changed

Introduces a fine-grained environment variable to skip only the local ES|QL keep validation in ESQLRuleData.validates_esql_data (presence of | keep and _id / _version / _index in keep for non-aggregate queries). Other checks in that method (for example FROM … METADATA for non-aggregate queries) also now can be bypassed.

The variable is included in set_all_validation_bypass(), so bypass_optional_elastic_validation: true in _config.yaml also enables this bypass.

How To Test

Run the Unit Tests or test with a single rule.

Example:

source env/detection-rules-build/bin/activate
python -m pytest tests/test_schemas.py::TestESQLValidation -v --tb=short

Or, a single rule test:

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a feature to the code.

Documentation and Context

• Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• Include additional context or screenshots.
• Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Ensure that the code is modular and reusable where applicable.

Testing

• New unit tests have been added to cover the enhancement.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• Validate that any rules affected by the enhancement are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.
• Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Checks

• Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• Confirm that the proper version label is applied to the PR patch, minor, major.

[Mikaayenson]

lgtm w/unit test and ci passing.

q: any thoughts on adding a param to the config?

[eric-forte-elastic]

q: any thoughts on adding a param to the config?

I think this is a good idea, will add shortly. I expect many users will want to bypass this. Will also add the other bypass env vars to the config as well, so that they can be used in a more fine grain way.

[eric-forte-elastic]

Testing new config vars: