Skip to content

feat(microsoft-defender-o365): mvp1/chk1 - project architecture setup (#471) - #491

Draft
Megafredo wants to merge 13 commits into
feat/470-microsoft-defender-o365-mvp1-email-focused-collect-and-match-defender-for-o365-alertsfrom
feat/471-chk1-project-architecture-setup
Draft

feat(microsoft-defender-o365): mvp1/chk1 - project architecture setup (#471)#491
Megafredo wants to merge 13 commits into
feat/470-microsoft-defender-o365-mvp1-email-focused-collect-and-match-defender-for-o365-alertsfrom
feat/471-chk1-project-architecture-setup

Conversation

@Megafredo

Copy link
Copy Markdown
Member

CHK.1 Architecture - Template usage - Blank Collector - Catalog Integration

Goal

Launch a working skeleton collector that appears in the OpenAEV catalog with no business logic implemented.
This establishes the project structure, Docker setup, and catalog registration, the foundation for all subsequent chunks.

Contracts

Input Contracts

None, this is the root chunk, no upstream data dependency.

Output Contracts

None, this chunk produces only a structural scaffold, no Pydantic model.

Acceptance Criteria

@lifecycle @container @startup

Feature: Containerized service starts without errors
  As a developer
  I want my containerized service to start cleanly in daemon mode
  So that I can verify the deployment scaffold is functional before adding business logic

  Background:
    Given a <PLATFORM> instance is running
    And a docker-compose.yml is configured with the service container

  Scenario Outline: Service process remains alive with no unhandled exceptions
    Given a minimal <SERVICE_NAME> instantiated from <BASE_CLASS> with stub Source wired
    When the service process is started via docker-compose
    Then the process remains alive in daemon mode
    And no unhandled exception appears in the service logs within <startup_window_seconds> seconds of startup

    Examples:
      | PLATFORM | SERVICE_NAME | BASE_CLASS    | startup_window_seconds |
      | OpenAEV  | Collector    | BaseCollector | 10                     |
@lifecycle @catalog @registration

Feature: Service registers in a platform catalog on startup
  As a platform operator
  I want deployed services to self-register in the platform catalog
  So that they are discoverable and their status is tracked centrally

  Background:
    Given the platform catalog is accessible
    And the service is configured with a valid registration

  Scenario Outline: Service appears in the catalog after startup
    Given the service is running with its catalog registration configuration
    When the platform catalog is queried for registered services
    Then "<service_name>" appears in the service list
    And its status is "<registration_status>"

    Examples:
      | service_name     | registration_status |
      | Collector   | Deployed          |
@ci @quality @automation

Feature: CI pipeline runs lint and tests on push
  As a developer
  I want every push to trigger automated lint and test jobs
  So that regressions are caught before code review

  Background:
    Given a CI pipeline is configured (GitHub Actions, GitLab CI, or equivalent)
    And lint and test jobs are defined in the pipeline configuration

  Scenario Outline: Push triggers lint and test jobs that both pass
    Given a commit is pushed to the repository
    When the CI pipeline runs
    Then the lint job completes with exit code <lint_exit_code>
    And the test job completes with exit code <test_exit_code>

    Examples:
      | lint_exit_code | test_exit_code |
      | 0              | 0              |

Done Checklist

  • Clone the OpenAEV collector template into a new repository
  • Create defender_o365_collector.py with main() wiring stub Source (placeholder DefenderO365DataFetcher, DefenderO365SourceData, empty SUPPORTED_SIGNATURES) into BaseCollector
  • Create __main__.py calling main() for python -m execution
  • Add pyproject.toml with pyoaev (from the template codebase) pinned as a dependency
  • Add Dockerfile and docker-compose.yml
  • Verify collector starts without errors: docker compose up , check logs for no exceptions
  • Confirm that the Collector appears in OpenAEV UI
  • Verify that the collector is visible through the Catalog
  • Verify that the collector run through the Catalog work with no exceptions
  • Write README.md with project description and placeholder configuration table
  • Add GitHub Actions CI skeleton: lint job + test job
  • Create a dedicated PR linked to the Umbrella issue

Closes #471

@Megafredo Megafredo added the filigran team Item from the Filigran team. label Jul 16, 2026
@Megafredo Megafredo changed the title feat(microsoft-defender-o365): chk1 - project architecture setup (#471) feat(microsoft-defender-o365): mvp1/chk1 - project architecture setup (#471) Jul 16, 2026
Megafredo and others added 4 commits July 17, 2026 09:45
…branch (#471)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ature files (#471)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…hk1 features (#471)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…o pass BDD tests (#471)

- Clone template/src collector framework verbatim (generic engine, models, protocols, types, internals, settings)
- Rename Template-specific bits: collector id/name defaults, icon path, source config aliases (SOURCE_*), source classes (MicrosoftDefenderO365DataFetcher/SourceData), empty SUPPORTED_SIGNATURES per #471
- Wire microsoft_defender_o365_collector.py main() using the same import names patched by the red tests
- Adapt test_ci_pipeline.py to check the repository's existing CircleCI + shared GitHub Actions pipelines instead of a dedicated per-collector workflow file
- All 3 BDD tests (container startup, catalog registration, CI pipeline) now pass; isort/black/flake8 clean

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Megafredo
Megafredo force-pushed the feat/471-chk1-project-architecture-setup branch from 4a386b6 to 1d831ac Compare July 17, 2026 13:31
…oot (#471)

isort was previously run from within the collector directory, which
resolves first-party import grouping differently than CI's
'isort --profile black --check .' invoked from the repo root, causing
CI failures. Re-run isort from the repo root to match CI's behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
285 1 284 0
View the full list of 1 ❄️ flaky test(s)
::tests.test_openaev_microsoft_defender

Flake rate in main: 100.00% (Passed 0 times, Failed 8 times)

Stack Traces | 0s run time
../../../../......./pypoetry/virtualenvs/openaev-microsoft-defender-collector-ysPYSAA0-py3.13/lib/python3.13.../site-packages/_pytest/python.py:508: in importtestmodule
    mod = import_path(
../../../../......./pypoetry/virtualenvs/openaev-microsoft-defender-collector-ysPYSAA0-py3.13/lib/python3.13.../site-packages/_pytest/pathlib.py:596: in import_path
    importlib.import_module(module_name)
.../hostedtoolcache/Python/3.13.14.../x64/lib/python3.13/importlib/__init__.py:88: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<frozen importlib._bootstrap>:1395: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:935: in _load_unlocked
    ???
../../../../......./pypoetry/virtualenvs/openaev-microsoft-defender-collector-ysPYSAA0-py3.13/lib/python3.13.../_pytest/assertion/rewrite.py:188: in exec_module
    exec(co, module.__dict__)
tests/test_openaev_microsoft_defender.py:77: in <module>
    from microsoft_defender.openaev_microsoft_defender import (
microsoft_defender/openaev_microsoft_defender.py:12: in <module>
    from msgraph import GraphServiceClient
../../../../......./pypoetry/virtualenvs/openaev-microsoft-defender-collector-ysPYSAA0-py3.13/lib/python3.13.../site-packages/msgraph/__init__.py:5: in <module>
    from .graph_request_adapter import GraphRequestAdapter
../../../../......./pypoetry/virtualenvs/openaev-microsoft-defender-collector-ysPYSAA0-py3.13/lib/python3.13.../site-packages/msgraph/graph_request_adapter.py:3: in <module>
    from kiota_abstractions.authentication import AuthenticationProvider
../../../../......./pypoetry/virtualenvs/openaev-microsoft-defender-collector-ysPYSAA0-py3.13/lib/python3.13.../kiota_abstractions/authentication/__init__.py:3: in <module>
    from .anonymous_authentication_provider import AnonymousAuthenticationProvider
../../../../......./pypoetry/virtualenvs/openaev-microsoft-defender-collector-ysPYSAA0-py3.13/lib/python3.13.../kiota_abstractions/authentication/anonymous_authentication_provider.py:9: in <module>
    from ..request_information import RequestInformation
../../../../......./pypoetry/virtualenvs/openaev-microsoft-defender-collector-ysPYSAA0-py3.13/lib/python3.13.../site-packages/kiota_abstractions/request_information.py:17: in <module>
    from opentelemetry import trace
E     File "/home/runner/......./pypoetry/virtualenvs/openaev-microsoft-defender-collector-ysPYSAA0-py3.13/lib/python3.13.../opentelemetry/trace/__init__.py", line 669
E       ]
E       ^
E   SyntaxError: unmatched ']'

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@Megafredo Megafredo self-assigned this Jul 22, 2026

@mariot mariot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! Just one question and a nitpick

Comment thread microsoft-defender-o365/src/collector/collector.py
"Source": patch.object(collector, "Source"),
"BaseCollector": patch.object(collector, "BaseCollector"),
}
return {name: patcher.start() for name, patcher in patchers.items()}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not important and with zero priority:

helpers call patcher.start() for multiple patchers and never stop them. That can leak mocks across tests and create order-dependent behavior/flaky runs.

We could wrap them in context managers (or register patcher.stop() cleanup) so mocks don’t leak between tests?

Megafredo and others added 8 commits July 29, 2026 09:36
…branch (#471)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ature files (#471)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…hk1 features (#471)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…o pass BDD tests (#471)

- Clone template/src collector framework verbatim (generic engine, models, protocols, types, internals, settings)
- Rename Template-specific bits: collector id/name defaults, icon path, source config aliases (SOURCE_*), source classes (MicrosoftDefenderO365DataFetcher/SourceData), empty SUPPORTED_SIGNATURES per #471
- Wire microsoft_defender_o365_collector.py main() using the same import names patched by the red tests
- Adapt test_ci_pipeline.py to check the repository's existing CircleCI + shared GitHub Actions pipelines instead of a dedicated per-collector workflow file
- All 3 BDD tests (container startup, catalog registration, CI pipeline) now pass; isort/black/flake8 clean

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…oot (#471)

isort was previously run from within the collector directory, which
resolves first-party import grouping differently than CI's
'isort --profile black --check .' invoked from the repo root, causing
CI failures. Re-run isort from the repo root to match CI's behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

filigran team Item from the Filigran team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(microsoft-defender-o365): mvp1/chk1 - project architecture setup

3 participants