feat(openaev): compatibility with new JSON:API format (#317)

[guzmud]

Proposed changes

• Add compatibility with JSON:API format for payloads to the openaev collector

Testing Instructions

1. Try to import with JSON:API formatted payload available in the source repository

Related issues

• Closes feat(payload): support e2e workflow in the OpenAEV payload collector #317

Checklist

• I consider the submitted work as finished
• I tested the code for its functionality
• I wrote test cases for the relevant uses case
• I added/update the relevant documentation (either on github or on notion)
• Where necessary I refactored code to improve the overall quality
• For bug fix -> I implemented a test that covers the bug

[codecov]

Codecov Report

❌ Patch coverage is 89.36170% with 25 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
openaev/openaev/processors/common_processor.py	83.80%	17 Missing ⚠️
openaev/openaev/openaev_openaev.py	82.50%	7 Missing ⚠️
openaev/openaev/processors/json_api_processor.py	96.77%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #358       +/-   ##
===========================================
- Coverage   78.59%   46.11%   -32.49%     
===========================================
  Files         270      276        +6     
  Lines       12507    12682      +175     
===========================================
- Hits         9830     5848     -3982     
- Misses       2677     6834     +4157     

Files with missing lines	Coverage Δ
openaev/openaev/constants.py	100.00% <100.00%> (ø)
openaev/openaev/github_crawler.py	100.00% <100.00%> (ø)
openaev/openaev/processors/__init__.py	100.00% <100.00%> (ø)
openaev/openaev/processors/json_flat_processor.py	100.00% <100.00%> (ø)
openaev/openaev/processors/json_api_processor.py	96.77% <96.77%> (ø)
openaev/openaev/openaev_openaev.py	84.31% <82.50%> (+65.31%)	⬆️
openaev/openaev/processors/common_processor.py	83.80% <83.80%> (ø)

... and 90 files with indirect coverage changes

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[guzmud]

Adding do not merge since the product orientation of the overall feature in unclear in its expectations regarding implementation (same goes for OpenAEV-Platform/payloads#8). PR in standby in order to have groundwork already done.

[guzmud]

Note: changing the tag from bug to feature due to the redefinition of the issue / redefinition of the target (more details in the attached issue #317 )

[Copilot]

Pull request overview

Adds JSON:API (new OpenAEV export) ingestion support to the openaev collector while keeping backward compatibility with the legacy “flat” payload format, enabling payloads exported from the UI to be re-imported into the community payloads repository.

Changes:

• Introduces a format-detecting payload processing flow that routes payloads to either a JSON:API or legacy-flat processor.
• Adds a GitHub crawler to enumerate and fetch payload JSON files directly from a repo/ref.
• Adds unit + functional tests and fixtures to validate JSON:API vs legacy processing parity.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
openaev/openaev/openaev_openaev.py	Refactors collector loop to fetch payloads from GitHub and process JSON:API vs legacy automatically.
openaev/openaev/github_crawler.py	New helper for listing/fetching payload JSON files and generating raw download URLs.
openaev/openaev/processors/common_processor.py	Shared processing logic for tags, attack patterns, and document handling.
openaev/openaev/processors/json_api_processor.py	JSON:API-specific payload normalization into legacy-like shape.
openaev/openaev/processors/json_flat_processor.py	Legacy flat payload processing extracted into a dedicated processor.
openaev/openaev/processors/init.py	Exposes processor classes at the package level.
openaev/openaev/constants.py	Moves default expected security platforms mapping into a dedicated module.
openaev/pyproject.toml	Updates dependencies (adds JSON:API deserializer, test/dev deps) and project metadata.
openaev/tests/unit/test_openaev_openaev.py	Unit tests for format detection and per-payload processing routing.
openaev/tests/unit/test_github_crawler.py	Unit tests for repo/ref parsing, listing JSON paths, and JSON retrieval.
openaev/tests/unit/processors/test_common_processor.py	Unit tests for shared processor behaviors (tags, attack patterns, documents).
openaev/tests/unit/processors/test_json_api_processor.py	Unit tests for JSON:API deserialization + normalization behavior.
openaev/tests/unit/processors/test_json_flat_processor.py	Unit tests for legacy flat processing behavior.
openaev/tests/test_default_expected_security_platforms.py	Updates tests to call the refactored defaulting logic on CommonProcessor.
openaev/tests/functional/test_payload_processing.py	Functional comparison test to ensure JSON:API and legacy formats produce equivalent outputs.
openaev/tests/functional/old_format.json	Legacy payload fixture.
openaev/tests/functional/new_format.json	JSON:API payload fixture.
openaev/tests/unit/processors/init.py	Test package initializer for processor unit tests.