Fix: Add comprehensive unit tests for non-ecs-schema.json and clean up data (#2322)

[chidoziemanagwu]

Pull Request

Issue link(s): Resolves #2322

Summary - What I changed

This PR resolves Issue #2322 by constructing a robust constraint unit-test specifically for the non-ecs-schema.json mapping dictionary. Historically, this file lacked isolated structure validation—leaving the main rule engine vulnerable to schema bugs. Any contributor could accidentally introduce duplicate fields, assign mismatched datatypes across indices, or define fields that actually belong in canonical ECS, causing potential cross-index mapping explosions.

To prevent this permanently, I built a new 6-part unit test suite in tests/test_non_ecs_schema.py.

⚡️ Key Test Snippet - Finding canonical ECS overlaps:

    def test_fields_not_in_ecs(self):
        """Verify that fields in non-ecs-schema.json are not already present in the ECS flat schema."""
        ecs_schema = get_schema()
        overlapping = []

        for index_pattern, fields in self.non_ecs_schema.items():
            flattened = flatten(fields)
            overlapping.extend(
                f"  {index_pattern} -> {field_name}" for field_name in flattened if field_name in ecs_schema
            )

        if overlapping:
            self.fail(
                "The following fields in non-ecs-schema.json are already present in the ECS schema "
                "and should be removed to prevent redundancy:\n" + "\n".join(overlapping)
            )

🔧 Data Integrity Cleanup:

While building these tests, the suite correctly identified 8 fields inside non-ecs-schema.json that had secretly been added upstream to the canonical ECS system in recent versions (like http.request.method and file.size). Instead of ignoring these test failures, I directly purged these overlapping variables to clear up technical debt, ensuring the new suite passes 100%.

⚡️ Schema Fix Example (detection_rules/etc/non-ecs-schema.json):

  "logs-network_traffic.http*": {
-   "data_stream.dataset": "keyword",
-   "url.path": "keyword",
-   "http.request.referrer": "keyword",
    "http.request.headers.content-type": "keyword",
-   "network.direction": "keyword",
-   "http.request.method": "keyword",
    "request": "keyword",
-   "http.request.body.bytes": "long",
-   "http.request.body.content": "keyword",
    "http.response.headers.server": "keyword"
  },

How To Test

You can manually verify the infrastructure changes locally using the built-in testing commands.

1. Run the targeted new test suite:

python -m pytest tests/test_non_ecs_schema.py -v

Output Details:

tests/test_non_ecs_schema.py::TestNonEcsSchema::test_fields_not_in_ecs PASSED
tests/test_non_ecs_schema.py::TestNonEcsSchema::test_no_conflicting_field_types_across_indices PASSED
tests/test_non_ecs_schema.py::TestNonEcsSchema::test_no_duplicate_fields_within_index PASSED
tests/test_non_ecs_schema.py::TestNonEcsSchema::test_no_empty_index_patterns PASSED
tests/test_non_ecs_schema.py::TestNonEcsSchema::test_valid_field_types PASSED
tests/test_non_ecs_schema.py::TestNonEcsSchema::test_valid_json_structure PASSED

2. Verify 0 regressions against the entire repository:
Because we removed fields from non-ecs-schema.json, test the entire platform to ensure zero rule regressions:

python -m detection_rules test

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[cla-checker-service]

💚 CLA has been signed