feat(ransomwarelive): enrich groups with aliases, leak sites and TTPs (#6299)

[Lyra-Fox]

Proposed changes

• Add group altname as aliases on IntrusionSet and ThreatActor.
• Add the ransomware.live group profile URL as an external reference on IntrusionSet and ThreatActor.
• Ingest leak-site FQDNs as Domain-Name observables linked to the IntrusionSet via related-to, and add leak-site URLs as external references (gated by CONNECTOR_CREATE_LEAK_SITE_DOMAINS).
• Create uses relationships from IntrusionSet to ATT&CK AttackPattern objects for each group TTP. Correlation resolves against AttackPatterns already present in OpenCTI (e.g. via the MITRE connector); techniques with no match are silently skipped.
• Add CONNECTOR_CREATE_LEAK_POST_REFS to toggle inclusion of the direct leak-post URL on victim Reports.
• Deduplicate external references within a group entry, and skip re-enriching a group already processed in the same run.

Both new toggles default to false to preserve backwards compatibility and to make it easier to comply with differing local regulations around handling links to leaked data.

Related issues

Closes #6299

Maintainer review & reconcile (@SamuelHassine)

• Reconciled the feature onto master after a large connector refactor that landed since this PR was opened (Campaign support, ConverterToStix(marking_value), a helper-based RansomwareAPIClient with 429 retry/backoff and a 30s timeout, and a restructured create_bundle_list). The PR's standalone 30s timeout is now covered by master's client, so that change was dropped in favour of master's version.
• Re-applied the group enrichment on top of master's structure: aliases + external references on IntrusionSet/ThreatActor, leak-site Domain-Name observables and IntrusionSet -> AttackPattern uses relationships wired into create_bundle_list (gated on create_intrusion_set), and the create_leak_post_refs toggle gating the leak-post URL (post_url) on report external references.
• Regenerated __metadata__/connector_config_schema.json and __metadata__/CONNECTOR_CONFIG_DOC.md from the config model; documented the two toggles in README.md and docker-compose.yml.
• Bumped pinned pycti to match connectors-sdk @ master (the old pin conflicted at install time). Master moved requirements.txt to src/requirements.txt; the test include now points there.
• Reworked the unit tests to the new signatures (ConverterToStix(marking_value), post_url).

Review-and-fix pass (de2419c)

• Fixed a per-run dedup bug: processed_groups was created once in __init__ and never reset. Since schedule_iso reuses the connector instance across scheduled runs, a group enriched on an earlier run stayed in the set forever and its newly disclosed leak sites / TTPs were skipped until the process restarted. The set is now reset at the start of each collection sweep (collect_intelligence / collect_historic_intelligence).
• Corrected the process_external_references docstring to state it returns a list of ExternalReference objects (empty list when none apply), not a single object.
• Added unit tests for the per-run reset of both collectors and tidied a duplicate comment header in the test module.
• Kept the structured-logging call in attack_pattern_fetcher consistent with the connector's other 8 error logs (positional meta dict); pycti's AppLogger.error(message, meta=None) binds the second positional argument to meta and already passes exc_info=1, so the change is cosmetic and a one-off would only introduce inconsistency. Noted as a possible repo-wide logging-cleanup follow-up.

Review-and-fix pass (6ea348d)

• Made the two compliance-sensitive enrichment helpers fail closed: ConverterToStix(create_leak_site_domains=...) and process_external_references(create_leak_post_refs=...) now default to False, matching the CONNECTOR_CREATE_LEAK_SITE_DOMAINS / CONNECTOR_CREATE_LEAK_POST_REFS config defaults. The runtime already passes the configured values explicitly, so there is no behaviour change for the connector; the fix prevents an ad-hoc or future call site that forgets the flag from silently emitting leak-site domains/URLs or direct leak-post URLs (behaviour that exists specifically to help comply with local rules on handling leaked-data links).
• Updated the two tests that exercise leak-site slug references to construct the converter with create_leak_site_domains=True explicitly, and added regression tests asserting the default converter omits leak-site slug URLs and the default process_external_references omits the leak-post URL.
• pycti stays pinned to 7.260604.0 to match the exact pin in connectors-sdk @ master (this connector installs the SDK from master); bumping it further would conflict at pip install.
• Full suite passes (44 tests); black / isort / flake8 clean. All GitHub Actions checks (including the Lint & Format and STIX ID Linter workflows) and Codecov patch/project are green; 0 unresolved review threads.

Checklist

• I consider the submitted work as finished
• I have signed my commits using GPG key.
• I tested the code for its functionality using different use cases
• I added/update the relevant documentation (either on github or on notion)
• Where necessary I refactored code to improve the overall quality

[filigran-cla-bot]

Contributor License Agreement

✅ CLA signed 💚

Thank you @Lyra-Fox for signing the Contributor License Agreement! Your pull request can now be reviewed and merged.

We appreciate your contribution to Filigran's open source projects! ❤️

_{This is an automated message from the Filigran CLA Bot.}

[Copilot]

Pull request overview

This PR enhances the external-import/ransomwarelive connector to ingest additional group- and victim-level enrichment from the ransomware.live /v2/groups API, with new safety toggles for ingesting leak-site related domains/URLs, plus adds unit tests and documentation updates.

Changes:

• Enrich IntrusionSet/ThreatActor with group aliases and external references (group profile + optional leak-site URLs), and enrich Reports with additional external references (including an optional direct leak-post URL).
• Optionally ingest leak-site FQDNs as DomainName observables related to the IntrusionSet, and create uses relationships from IntrusionSet to existing ATT&CK AttackPattern objects.
• Add per-run group deduplication and a 30s HTTP timeout; introduce a new unit test suite and update connector docs/config schema.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
external-import/ransomwarelive/tests/test-requirements.txt	Adds test dependencies for the connector’s new unit test suite.
external-import/ransomwarelive/tests/test_group_enrichment.py	Adds unit tests for group alias/ext-ref extraction, leak site domain creation, TTP relationships, and per-run deduplication.
external-import/ransomwarelive/tests/conftest.py	Ensures src/ is importable during tests.
external-import/ransomwarelive/src/ransomwarelive/utils.py	Adds get_group_entry helper for group enrichment lookups.
external-import/ransomwarelive/src/ransomwarelive/ransom_conn.py	Implements per-run group enrichment dedup + ATT&CK lookup + toggled leak enrichment; includes marking in bundles.
external-import/ransomwarelive/src/ransomwarelive/converter_to_stix.py	Adds group alias/ext-ref extraction, leak-site domain processing, and toggled leak-post report refs.
external-import/ransomwarelive/src/ransomwarelive/api_client.py	Adds a 30s request timeout and refines typing.
external-import/ransomwarelive/src/models/configs/connector_configs.py	Adds create_leak_site_domains and create_leak_post_refs configuration toggles (default false).
external-import/ransomwarelive/README.md	Updates entity mapping and documents the new leak-link toggles and their compliance implications.
external-import/ransomwarelive/docker-compose.yml	Documents new env vars in the sample compose configuration.
external-import/ransomwarelive/metadata/connector_config_schema.json	Adds the new toggle keys to the generated config schema.
external-import/ransomwarelive/metadata/CONNECTOR_CONFIG_DOC.md	Adds the new toggle keys to the generated config documentation.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

[codecov]

Codecov Report

❌ Patch coverage is 88.65979% with 11 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...omwarelive/src/ransomwarelive/converter_to_stix.py	88.00%	6 Missing ⚠️
...t/ransomwarelive/src/ransomwarelive/ransom_conn.py	87.50%	5 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (9fb2a96) and HEAD (6ea348d). Click for more details.

HEAD has 64 uploads less than BASE

Flag	BASE (9fb2a96)	HEAD (6ea348d)
connectors	66	2

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #6300       +/-   ##
===========================================
- Coverage   11.82%    0.44%   -11.39%     
===========================================
  Files        1911     1885       -26     
  Lines      120103   119264      -839     
===========================================
- Hits        14204      527    -13677     
- Misses     105899   118737    +12838     

Files with missing lines	Coverage Δ
...omwarelive/src/models/configs/connector_configs.py	96.55% <100.00%> (+96.55%)	⬆️
...-import/ransomwarelive/src/ransomwarelive/utils.py	50.00% <100.00%> (+50.00%)	⬆️
...t/ransomwarelive/src/ransomwarelive/ransom_conn.py	17.63% <87.50%> (+17.63%)	⬆️
...omwarelive/src/ransomwarelive/converter_to_stix.py	55.08% <88.00%> (+55.08%)	⬆️

... and 514 files with indirect coverage changes

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[SamuelHassine]

Approving. I read the full changed files (not just the diff) and reconciled the group-enrichment feature onto master's refactored connector. All checks are green (Tests Connectors, codecov patch/project, signed commits, PR conventions), the unit suite passes (40 tests), and all three Copilot threads are addressed and resolved.

[SamuelHassine]

Review & reconcile summary

This PR was 215 commits behind master and conflicted with a large refactor of the connector that landed after it was opened. I merged master and reconciled the feature on top of the new structure:

• Reconciled onto master: Campaign support, ConverterToStix(marking_value), helper-based RansomwareAPIClient (429 retry/backoff + 30s timeout), and the restructured create_bundle_list. The PR's standalone 30s timeout is now covered by master's client, so that change was dropped in favour of master's version.
• Re-applied the feature: aliases + external references on IntrusionSet/ThreatActor; leak-site Domain-Name observables and IntrusionSet→AttackPattern uses relationships wired into create_bundle_list (gated on create_intrusion_set); create_leak_post_refs gating the leak-post URL (post_url) on report external references. Both new toggles default to false.
• Metadata/docs: regenerated __metadata__/connector_config_schema.json and CONNECTOR_CONFIG_DOC.md from the config model; documented both toggles in README.md and docker-compose.yml.
• Dependencies: bumped pinned pycti to 7.260604.0 to match connectors-sdk @ master (the old 7.260423.0 pin conflicted at install). Master moved requirements.txt to src/requirements.txt; the test include now points there.
• Tests: reworked the suite to the new signatures (ConverterToStix(marking_value), post_url); full suite passes (40 tests). All three Copilot review threads addressed and resolved.

CI is fully green (Tests Connectors, codecov patch/project, signed commits, PR conventions, do-not-merge, CLA). mergeable: MERGEABLE — no outstanding non-CI blockers.

Waiting for the team

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

[SamuelHassine]

Reviewed the full changed files independently (not just the diff) and addressed every outstanding review thread.

• Fixed the per-run processed_groups dedup reset - it was suppressing group enrichment across scheduled runs because the connector instance is reused by schedule_iso.
• Corrected the process_external_references docstring return type.
• Added unit tests for the per-run reset on both collectors (42 tests pass); black / isort / flake8 clean.

All five previously-open Copilot threads are resolved. CI is fully green and the PR is mergeable. LGTM.

[SamuelHassine]

Review & fix pass complete

Independent senior review of the full changed files plus resolution of every open thread:

• Bug fix: processed_groups is now reset at the start of each collection sweep (collect_intelligence / collect_historic_intelligence). It was only initialised in __init__, and since schedule_iso reuses the connector instance, a group enriched on one run was permanently skipped - so newly disclosed leak sites / TTPs were never picked up until the process restarted.
• Docs: corrected the process_external_references docstring to state it returns a list of ExternalReference objects (empty list when none apply).
• Tests / lint: added TestProcessedGroupsResetPerRun covering both collectors and tidied a duplicate comment header; full suite passes (42 tests), black / isort / flake8 clean.
• Logging threads: kept consistent with the connector's other 8 error logs (positional meta dict). pycti's AppLogger.error(message, meta=None) binds the second positional argument to meta and already passes exc_info=1, so the suggested change is cosmetic; changing one of eight identical sites would only add inconsistency. Flagged a repo-wide logging cleanup as an optional follow-up.

All five previously-open Copilot threads are resolved. CI is fully green (Tests Connectors, codecov patch/project, PR conventions, do-not-merge, CLA). mergeable: MERGEABLE - no outstanding non-CI blockers.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

[SamuelHassine]

Re-approving after an independent full-file re-review of the connector and resolving the two remaining Copilot threads in signed commit 6ea348d6da.

Both threads were the same class of issue, and both are legitimate: the compliance-sensitive enrichment helpers (ConverterToStix(create_leak_site_domains=...) and process_external_references(create_leak_post_refs=...)) defaulted their parameters to True while the corresponding config toggles default to False. The runtime always passes the configured value explicitly, so the connector's behaviour was correct, but the method-level defaults failed open - any future/ad-hoc call site that forgot the flag would silently emit leak-site domains/URLs and direct leak-post URLs. They now default to False (fail closed), mirroring the config and the connector's stated compliance posture.

Tests updated accordingly: the two cases that exercise leak-site slug references opt in explicitly, plus new regression tests assert the defaults omit leak-site slugs and the leak-post URL (44 passing; black / isort / flake8 clean).

I also confirmed pycti must stay pinned at 7.260604.0 to match the exact pin in connectors-sdk @ master (this connector installs the SDK from master) - bumping it would conflict at install. All GitHub Actions checks - including the Lint & Format and STIX ID Linter workflows - plus Codecov patch/project are green; 0 unresolved review threads; mergeable: MERGEABLE. LGTM.

[SamuelHassine]

Review & fix summary (round 3)

Independent full-file re-review of the connector plus the two remaining Copilot threads from the latest review, fixed in signed commit 6ea348d6da.

• Fail-closed enrichment defaults: ConverterToStix(create_leak_site_domains=...) and process_external_references(create_leak_post_refs=...) now default to False, matching the CONNECTOR_CREATE_LEAK_SITE_DOMAINS / CONNECTOR_CREATE_LEAK_POST_REFS config defaults. The runtime always passes the configured values explicitly, so there is no behaviour change for the connector; the fix prevents an ad-hoc/future call site that forgets the flag from silently emitting leak-site domains/URLs or direct leak-post URLs (behaviour kept off by default for compliance with local leaked-data rules).
• Tests: the two cases exercising leak-site slug references now opt in with create_leak_site_domains=True; added regression tests asserting the default converter omits leak-site slugs and the default process_external_references omits the leak-post URL (44 passing; black / isort / flake8 clean).
• Dependencies: confirmed pycti must stay pinned at 7.260604.0 to match the exact pin in connectors-sdk @ master (the SDK is installed from master here) - bumping further would conflict at pip install.

Status

• All checks green: Base Linter (flake8), Ensure Formatting, STIX ID Linter, Tests Connectors, codecov/patch, codecov/project, signed commits, PR-conventions, CLA.
• 0 unresolved review threads. mergeable: MERGEABLE, re-approved.

Non-blocking follow-up (future iteration)

• Optional repo-wide logging-cleanup pass to migrate the connector's positional meta error-log calls to the meta= keyword form (cosmetic; pycti's AppLogger.error(message, meta=None) already binds the positional dict to meta and captures exc_info).