feat(microsoft-defender-incidents): migrate connector to manager-supported mode (#5260)

[jabesq]

Proposed changes

• Add ConnectorSettings Pydantic class — replaces all legacy get_config_variable calls with a typed, validated settings model; this is the prerequisite for manager-supported mode and makes misconfiguration fail fast at startup.
• Refactor connector.py to ConnectorBase pattern — rewrites the connector class to extend ConnectorBase and consume ConnectorSettings, decoupling lifecycle management from business logic and enabling the platform manager to orchestrate runs.
• Add main.py entry point — provides the standard manager-supported entry point (python main.py) and removes the need for a separate entrypoint.sh shell wrapper.
• Set manager_supported: true in connector_manifest.json — signals to the OpenCTI platform that this connector can be managed directly by the connector manager.
• Make import_start_date optional and fix error handling — prevents a hard crash when the field is absent and aligns error surfacing with the ConnectorBase pattern.
• Add unit tests for settings and main entry point — covers ConnectorSettings validation (required fields, defaults, type coercion) and smoke-tests the main.py bootstrap path.
• Auto-generate CONNECTOR_CONFIG_DOC.md and connector_config_schema.json — replaces the hand-maintained README config table with schema-driven documentation.
• Add .env.sample — documents every connector env var with required/optional sections.
• Migrate Dockerfile — switches to WORKDIR-based layout and a direct ENTRYPOINT ["python", "main.py"], removing entrypoint.sh.
• Expand .dockerignore — covers all Python cache artefacts and build outputs.
• Update f-strings in client_api.py and converter_to_stix.py — consistency refactor; no behaviour change.
• Simplify README — removes the manual configuration table and points readers to the auto-generated CONNECTOR_CONFIG_DOC.md instead.

Related issues

• Closes refactor(microsoft-defender-incidents): migrate connector to be connector manager supported #5260

Checklist

• I consider the submitted work as finished
• I have signed my commits using GPG key.
• I tested the code for its functionality using different use cases
• I added/update the relevant documentation (either on github or on notion)
• Where necessary I refactored code to improve the overall quality

Further comments

This PR applies the manager-supported migration pattern:

1. ConnectorSettings (Pydantic BaseSettings) — all connector-specific configuration is declared as a typed Pydantic model. Environment variables are read and validated automatically, replacing the ad-hoc get_config_variable helper. Required fields raise a ValidationError on startup rather than failing silently at runtime.

2. ConnectorBase subclass — the connector class extends ConnectorBase (from connectors-sdk) and receives a ConnectorSettings instance via dependency injection. The base class owns the OpenCTI client lifecycle, the run loop, and error recovery. Setting manager_supported = True in the manifest tells the platform it may call into these lifecycle hooks directly.

[codecov]

Codecov Report

❌ Patch coverage is 90.90909% with 5 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...soft-defender-incidents/src/connector/connector.py	81.25%	3 Missing ⚠️
...oft-defender-incidents/src/connector/client_api.py	80.00%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (c691e18) and HEAD (5b12c2b). Click for more details.

HEAD has 114 uploads less than BASE

Flag	BASE (c691e18)	HEAD (5b12c2b)
connectors	118	4

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #6684       +/-   ##
===========================================
- Coverage   32.26%    0.56%   -31.70%     
===========================================
  Files        1986     1894       -92     
  Lines      122165   119611     -2554     
===========================================
- Hits        39413      680    -38733     
- Misses      82752   118931    +36179     

Files with missing lines	Coverage Δ
...osoft-defender-incidents/src/connector/__init__.py	100.00% <100.00%> (ø)
...ender-incidents/src/connector/converter_to_stix.py	100.00% <100.00%> (ø)
...osoft-defender-incidents/src/connector/settings.py	100.00% <100.00%> (ø)
...icrosoft-defender-incidents/src/connector/utils.py	100.00% <ø> (ø)
...al-import/microsoft-defender-incidents/src/main.py	100.00% <100.00%> (+100.00%)	⬆️
...oft-defender-incidents/src/connector/client_api.py	97.29% <80.00%> (ø)
...soft-defender-incidents/src/connector/connector.py	95.97% <81.25%> (ø)

... and 1120 files with indirect coverage changes

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR migrates the Microsoft Defender Incidents connector toward a manager-supported layout by introducing typed Pydantic-based settings, updating the connector bootstrap entry point, and adding generated configuration schema/docs plus a new test suite.

Changes:

• Introduces ConnectorSettings (connectors-sdk) and refactors runtime config access to use typed settings.
• Updates runtime/bootstrap and packaging (new connector/ module layout, updated Docker/compose, manifest set to manager_supported: true).
• Adds extensive unit tests plus generated config documentation (connector_config_schema.json, CONNECTOR_CONFIG_DOC.md) and .env.sample.

Reviewed changes

Copilot reviewed 28 out of 29 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
external-import/microsoft-defender-incidents/tests/tests_connector/test_utils.py	Adds tests for small utility helpers (IP detection, date formatting, matching files, priority mapping).
external-import/microsoft-defender-incidents/tests/tests_connector/test_settings.py	Adds tests validating ConnectorSettings happy-path and validation errors.
external-import/microsoft-defender-incidents/tests/tests_connector/test_converter_to_stix.py	Adds tests for STIX conversion helpers and error handling decorator.
external-import/microsoft-defender-incidents/tests/tests_connector/test_connector.py	Adds tests for connector state handling, extraction flow, and run loop integration points.
external-import/microsoft-defender-incidents/tests/tests_connector/test_client_api.py	Adds tests for OAuth token retrieval, query building, pagination, and incident fetching behavior.
external-import/microsoft-defender-incidents/tests/tests_connector/conftest.py	Provides shared fixtures (stub settings, helper/config mocks, converter/client instances).
external-import/microsoft-defender-incidents/tests/test-requirements.txt	Defines connector test dependencies (includes connector requirements + pytest).
external-import/microsoft-defender-incidents/tests/test_main.py	Adds tests for main.py bootstrap path and error exit behavior.
external-import/microsoft-defender-incidents/tests/conftest.py	Ensures src/ is importable during tests by adjusting sys.path.
external-import/microsoft-defender-incidents/src/requirements.txt	Adds connectors-sdk + Pydantic dependency needed for settings/schema generation.
external-import/microsoft-defender-incidents/src/microsoft_defender_incidents_connector/config_variables.py	Removes legacy get_config_variable-based configuration loader.
external-import/microsoft-defender-incidents/src/microsoft_defender_incidents_connector/init.py	Removes legacy package export path.
external-import/microsoft-defender-incidents/src/main.py	Updates entry point to instantiate settings + helper + connector explicitly.
external-import/microsoft-defender-incidents/src/connector/utils.py	Adds new shared utility helpers used by connector and converter.
external-import/microsoft-defender-incidents/src/connector/settings.py	Adds typed ConnectorSettings and connector-specific config models.
external-import/microsoft-defender-incidents/src/connector/converter_to_stix.py	Adjusts imports and refactors some string building to f-strings.
external-import/microsoft-defender-incidents/src/connector/connector.py	Refactors connector initialization to accept injected settings/helper; updates scheduling and error handling.
external-import/microsoft-defender-incidents/src/connector/client_api.py	Updates client to use nested settings model and improves query-builder typing.
external-import/microsoft-defender-incidents/src/connector/init.py	Exposes connector + settings as the public module surface.
external-import/microsoft-defender-incidents/README.md	Simplifies configuration documentation and links to generated config docs.
external-import/microsoft-defender-incidents/entrypoint.sh	Removes shell entrypoint in favor of direct Python entrypoint.
external-import/microsoft-defender-incidents/Dockerfile	Migrates Docker build/entrypoint structure for manager-supported mode.
external-import/microsoft-defender-incidents/docker-compose.yml	Updates compose examples/comments for the new settings model defaults and options.
external-import/microsoft-defender-incidents/config.yml.sample	Updates sample config defaults and boolean formatting.
external-import/microsoft-defender-incidents/.env.sample	Adds a complete env var sample aligned with the settings model.
external-import/microsoft-defender-incidents/.dockerignore	Expands ignore rules for Python caches/build outputs and local artifacts.
external-import/microsoft-defender-incidents/metadata/connector_manifest.json	Sets manager_supported: true and updates verification date.
external-import/microsoft-defender-incidents/metadata/connector_config_schema.json	Adds generated JSON schema for connector configuration.
external-import/microsoft-defender-incidents/metadata/CONNECTOR_CONFIG_DOC.md	Adds generated config documentation table derived from schema.

Comments suppressed due to low confidence (4)

external-import/microsoft-defender-incidents/src/connector/connector.py:405

• In the (KeyboardInterrupt, SystemExit) handler, work_id may be unbound if the connector is interrupted before initiate_work() runs (e.g., during auth or incident fetch). Accessing if work_id: would then raise UnboundLocalError, masking the original shutdown.
  external-import/microsoft-defender-incidents/src/connector/connector.py:329
• When _get_last_incident_date() returns None (no state + no import_start_date), process_message() still calls get_incidents(None). That triggers an exception path in ConnectorClient.get_incidents() and the connector then logs a “successfully run, no new data” message, effectively skipping the initial import silently.
  external-import/microsoft-defender-incidents/src/connector/client_api.py:66
• query_builder() now returns a requests.PreparedRequest (.prepare()), but the docstring still describes returning a URL string. This mismatch makes the API harder to use correctly and can confuse type-aware tooling.
  external-import/microsoft-defender-incidents/src/connector/connector.py:12
• PR description says the connector was refactored to the ConnectorBase pattern to enable manager-supported orchestration, but MicrosoftDefenderIncidentsConnector is still a standalone class (no ConnectorBase inheritance / lifecycle hooks). If manager-supported mode requires the base class, this looks incomplete or the PR description should be updated.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

external-import/microsoft-defender-incidents/src/connector/connector.py:71

• _get_last_incident_date() has unreachable code (return None) and will also raise if import_start_date is missing/None (it calls .timestamp() unconditionally). This contradicts the intent of making the start date optional and can crash the connector on first run.
  external-import/microsoft-defender-incidents/src/connector/connector.py:328
• work_id can be referenced in the (KeyboardInterrupt, SystemExit) handler before being assigned (e.g., if the interruption happens before initiate_work()), which will raise UnboundLocalError and prevent graceful shutdown handling.