From 38274158550c0d8ceae02eb72ddfa0f36e126607 Mon Sep 17 00:00:00 2001 From: Sebastian Wagner Date: Fri, 6 Mar 2026 08:20:43 +0100 Subject: [PATCH 1/9] docs: improve help page remove duplicate link to github issues in the intro mention announcements on users list also briefly mention dev list make a separate section for the support in closed communities, add the Shadowserver Alliance chat and FIRST --- docs/help.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/help.md b/docs/help.md index 6ea8f77340..e10ff36a54 100644 --- a/docs/help.md +++ b/docs/help.md @@ -13,7 +13,7 @@ In case you are lost, you need assistance or something is not discussed in this - Logs of bots or terminal output - Any other useful messages, screenshots -Please report any errors and suggest improvements via [issues](https://github.com/certtools/intelmq/issues). Thank you! +Please report to us any errors and suggest improvements. Thank you! ## GitHub @@ -25,15 +25,24 @@ To participate on GitHub, you first need to create an account on the platform. ## Mailing list -The most traditional way is to ask your question, make a proposal or discuss a topic on the -mailing [IntelMQ Users mailing list](https://lists.cert.at/mailman3/hyperkitty/list/intelmq-users@lists.cert.at/). You need to subscribe to the mailing list before posting, but the archive is publicly available: [IntelMQ Users Archive](https://lists.cert.at/mailman3/hyperkitty/list/intelmq-users@lists.cert.at/latest). +The most popular way to ask questions, make proposals or discuss a topic is the users +mailing [IntelMQ Users mailing list](https://lists.cert.at/mailman3/hyperkitty/list/intelmq-users@lists.cert.at/). You need to subscribe to the mailing list before posting. The archive is publicly available: [IntelMQ Users Archive](https://lists.cert.at/mailman3/hyperkitty/list/intelmq-users@lists.cert.at/latest). +All announcements such as new IntelMQ releases are also published to this mailing list. + +There is also a mailing list for all development-related questions: The [IntelMQ Dev mailing list](https://lists.cert.at/mailman3/hyperkitty/list/intelmq-users@lists.cert.at/) ## Assistance -If your organisation is a member of the [CSIRTs Network](https://csirtsnetwork.eu/), you are eligible for support in the [MeliCERTes project](https://melicertes.github.io/docs/). You can also ask on for individual support, some members offer support, including, but not limited to: +You can always ask for individual support to some community members, including, but not limited to: - [Aaron Kaplan](https://github.com/aaronkaplan/) (founder of IntelMQ) - [Institute for Common Good Technology](https://commongoodtechnology.org/) (chairmen Sebastian Wager is an IntelMQ maintainer and developer) - [Intevation GmbH](https://intevation.de/) (Develops and maintains several IntelMQ components) +## Closed communities + +If you or your organization are member of some of these groups, you can ask for help in there communities too. +* Shadowserver Alliance: In the [Shadowserver Alliance Chat](https://chat.shadowserver.org/), in the channel *InfoAPI-Integration-Community* +* [CSIRTs Network](https://csirtsnetwork.eu/): Members are eligible for support in the [MeliCERTes project](https://melicertes.github.io/docs/). +* FIRST: Ask for help in the *FIRST Automation SIG* From a4bad4e263ede7bfe4ee6630e7a7b5393deaf45a Mon Sep 17 00:00:00 2001 From: Sebastian Wagner Date: Fri, 6 Mar 2026 08:21:06 +0100 Subject: [PATCH 2/9] docs: fix to type in event docs the types are documented in a separate document, fixes the link --- scripts/generate-event-docs.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/scripts/generate-event-docs.py b/scripts/generate-event-docs.py index 8570460ad8..20c4e4e35f 100755 --- a/scripts/generate-event-docs.py +++ b/scripts/generate-event-docs.py @@ -19,7 +19,7 @@ @@ -160,7 +160,7 @@ - If an event describes IP address where a command and control server is running, the event's `classification.type` is `c2server`. The `malware.name` can have the full name, eg. `zeus_p2p`. - + ## Additional Information Information that do not fit into any of the event fields should be placed in the `extra` namespace.Therefore the keys must be prefixed `extra.` string. There are no other rules on key names and values for additional information. @@ -188,7 +188,7 @@ def main(): # f"[{value['type']}](#{value['type'].lower()})", # value['description']) output += f"""### `{key}`
\n\n""" - output += f"**Type:** [{value['type']}](#{value['type'].lower()})\n\n" + output += f"**Type:** [{value['type']}](../dev/data-format.md#{value['type'].lower()})\n\n" output += value['description'] output += "\n\n" From c81dbb09667463f1fc834209a4c95e6627885de6 Mon Sep 17 00:00:00 2001 From: Sebastian Wagner Date: Fri, 6 Mar 2026 08:31:03 +0100 Subject: [PATCH 3/9] docs: add news to built docs, fix links --- CHANGELOG.md | 2 +- docs/NEWS.md | 1 + mkdocs.yml | 1 + 3 files changed, 3 insertions(+), 1 deletion(-) create mode 120000 docs/NEWS.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 77cba765b0..099d23faf1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1501,7 +1501,7 @@ IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu 16.04), the Dropped support for Python 3.4. ### Core -- `__init__`: Changes to the path-handling, see [User Guide, section _/opt and LSB paths_](docs/User-Guide.md#opt-and-lsb-paths) for more information +- `__init__`: Changes to the path-handling, see [User Guide, section _/opt and LSB paths_](admin/configuration/intelmq.md#directories) for more information - The environment variable `INTELMQ_ROOT_DIR` can be used to set custom root directories instead of `/opt/intelmq/` (#805) in case of non LSB-path installations. - The environment variable `ROOT_DIR` can be used to set custom root directories instead of `/` (#805) in case of LSB-path installations. - `intelmq.lib.exceptions`: Added `MissingDependencyError` for show error messages about a missing library and how to install it (#1471). diff --git a/docs/NEWS.md b/docs/NEWS.md new file mode 120000 index 0000000000..9868b02659 --- /dev/null +++ b/docs/NEWS.md @@ -0,0 +1 @@ +../NEWS.md \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 5c4be87a79..2735e6d2e2 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -150,6 +150,7 @@ nav: - Release: 'dev/release.md' - Changelog: 'changelog.md' + - News: 'NEWS.md' - Security: 'security.md' - Community: 'community.md' - Help: 'help.md' From cd09baeb2994b805bc80cc8b677043231df3dc11 Mon Sep 17 00:00:00 2001 From: Sebastian Wagner Date: Sun, 12 Apr 2026 22:26:03 +0200 Subject: [PATCH 4/9] doc: various small bot documentation fixes mostly formatting --- docs/user/bots.md | 38 +++++++++++++++++++++++----------- scripts/generate-event-docs.py | 2 +- 2 files changed, 27 insertions(+), 13 deletions(-) diff --git a/docs/user/bots.md b/docs/user/bots.md index 44e90971dd..6eeaf7be6f 100644 --- a/docs/user/bots.md +++ b/docs/user/bots.md @@ -212,7 +212,7 @@ pipeline. Requires the [pika](https://pypi.org/project/pika/) library, minimum v **`connection_vhost`** -(optional, string) Virtual host to connect, on an HTTP(S) connection would be . +(optional, string) Virtual host to connect, on an HTTP(S) connection would be ``. **`expect_intelmq_message`** @@ -240,8 +240,9 @@ Defaults to false. ### API
-This bot collects data from HTTP or Socket REST API. The API is available at `/intelmq/push` when the HTTP interface is -used. Requires the [tornado](https://pypi.org/project/tornado/) library. +This bot collects data from HTTP or Socket REST API. +The API is available at `/intelmq/push` when the HTTP interface is used. +Requires the [tornado](https://pypi.org/project/tornado/) library. **Module:** `intelmq.bots.collectors.api.collector` @@ -1945,6 +1946,7 @@ If the input data did not contain the field `classification.type`, it is set to Supports multiple different modes: #### Input data is one event + Example: ```json { INTELMQ data... } @@ -1961,6 +1963,7 @@ Configuration: * `multiple_events`: False #### Input data is in JSON stream format + Example: ```json { INTELMQ data... } @@ -1973,6 +1976,7 @@ Configuration: * `multiple_events`: False #### Input data is a list of events + Example: ```json [ @@ -2883,7 +2887,7 @@ Both parameters accept string values describing absolute or relative time: (optional, string) Taken from the event will be compared to this value to decide the filter behavior. - relative -- accepted string formatted like this " ", where epoch could be any of following strings (could +- accepted string formatted like this ` `, where epoch could be any of following strings (could optionally end with trailing 's'): hour, day, week, month, year - time.source taken from the event will be compared to the value (now - relative) to decide the filter behavior @@ -3822,7 +3826,7 @@ if extra.hostnames :subsetof ['ns1.example.com', 'ns2.example.com'] { ... } if extra.tags :supersetof ['iot', 'vulnerable'] { ... } ``` -* `:before` tests if the date value occurred before given time ago. The time might be absolute (basically anything parseable by pendulum parser, eg. “2015-09-12T06:22:11+00:00”) or relative (accepted string formatted like this “ ”, where epoch could be any of following strings (could optionally end with trailing ‘s’): hour, day, week, month, year) +* `:before` tests if the date value occurred before given time ago. The time might be absolute (basically anything parseable by pendulum parser, eg. “2015-09-12T06:22:11+00:00”) or relative (accepted string formatted like this ` `, where epoch could be any of following strings (could optionally end with trailing ‘s’): hour, day, week, month, year) ``` if time.observation :before '1 week' { ... } @@ -4408,6 +4412,17 @@ Requires the [pika python library](https://pypi.org/project/pika/). (optional, string) Type of the exchange, e.g. `topic`, `fanout` etc. +**`format_routing_key`** + +(optional, boolean) Whether the routing key should be formatted with the event data. Defaults to false. + +Uses Python formatted strings. See: + +In the routing key, the event is accessible with the name `ev`. +Example: The routing key `{event[source.geolocation.cc]}` will be (for example) `BR`. + +If the field used in the format string is not defined, `None` will be used as fallback. + **`keep_raw_field`** (optional, boolean) Whether to keep the `raw` field or not. Defaults to false. @@ -5185,7 +5200,7 @@ REST API is the bot responsible to send events to a REST API listener through PO **Module:** `intelmq.bots.outputs.restapi.output` -**Parameters:** +**Parameters (also accepts [HTTP parameters](#http-parameters)):** **`host`** @@ -5193,23 +5208,23 @@ REST API is the bot responsible to send events to a REST API listener through PO **`auth_type`** -(required, string) Allowed values: `http_basic_auth` or `http_header`. Type of authentication to use. +(optional, string) Allowed values: `http_basic_auth` or `http_header` or empty/null. Type of authentication to use. **`auth_token`** -(required, string) Username or HTTP header key. +(optional, string) Password or HTTP header value. Required if `auth_type` is set. **`auth_token_name`** -(required, string) Password or HTTP header value. +(optional, string) Username or HTTP header key. Required if `auth_type` is set. **`hierarchical_output`** -(optional, boolean) Whether the resulting dictionary should be hierarchical (field names split by a dot). Defaults to false. +(optional, boolean) Whether the resulting dictionary should be hierarchical (field names split by a dot). Defaults to false. Set to `true` to keep the IntelMQ data format. **`use_json`** -(optional, boolean) Whether to use JSON. Defaults to true. +(optional, boolean) Whether to use JSON as HTTP Content Type (`application/json)`. Defaults to true. --- @@ -5577,7 +5592,6 @@ The parameters marked with 'PostgreSQL' will be sent to libpq via psycopg2. Chec (optional, boolean) Whether an error should cause the bot to fail (raise an exception) or otherwise rollback. If false, the bot eventually waits and re-try (e.g. re-connect) etc. to solve the issue. If true, the bot raises an exception and - depending on the IntelMQ error handling configuration - stops. Defaults to false. - ### STOMP This bot pushes data to any STOMP stream. STOMP stands for Streaming Text Oriented Messaging Protocol. See: diff --git a/scripts/generate-event-docs.py b/scripts/generate-event-docs.py index 20c4e4e35f..1bbf2a9db0 100755 --- a/scripts/generate-event-docs.py +++ b/scripts/generate-event-docs.py @@ -15,7 +15,7 @@ yaml = YAML(typ="safe", pure=True) -HEADER = """\ +HEADER = r"""\ + +# Connecting IntelMQ instances + +The same methods can be used to connect to link up with other systems via + +## Use cases + +### Connecting multiple IntelMQ instances in the same organization + +* Separate IntelMQ instances for different purposes +* IntelMQ instances running in different departments +* Workload management, load balancing and redundancy: Especially effective with AMQP + +### Data feeds + +One organization generates a data feed, processes it with IntelMQ and another organization wants to receive it with IntelMQ. + +### Connections between trusted organizations + +If two organizations both use IntelMQ, they can route messages to each other. + +### Connections between arbitrary organizations + +FIXME + +Injection of + +## Architecture + +FIXME + +### From IntelMQ outwards + +FIXME + +### Message broker + +FIXME + +It does not matter whether the message broker is technically or organizationally located in the sending or receiving side. + +### Into IntelMQ + +FIXME + +## Security Considerations + +FIXME + +## Data exchange methods + +FIXME + +### AMQP + +FIXME: Diagram + +This is the most sophisticated option and gives the best security, data flow control, stability. +The downside is, that it is more complex to setup and to operate. + +#### Security considerations + +FIXME + +#### Prerequisites + +Install the Python Library `pika` for AMQP on both the sender and the receiver: +``` bash +# Debian/Ubuntu +sudo apt install python3-pika +``` + +#### On the receiver + +##### Collector + +FIXME + +Module: `intelmq.bots.collectors.amqp.collector_amqp` + +###### Parameters + +These configuration parameters are required: + +- `connection_host`: Hostname of the AMQP server. +- `connection_port` Port of the AMQP server. Defaults to 5672. +- `connection_vhost`: Virtual host to connect, on an HTTP(S) connection would be ``. +- `expect_intelmq_message`: `true`. FIXME This parameter denotes whether the the data is from IntelMQ or not. If true, then the data can be any Report or Event and will be passed to the next bot as is. Otherwise a new Report is created with the raw data. Defaults to false. +* `queue_name`: (optional, string) The name of the queue to fetch the data from. +* `username`: Optional username for authentication to the AMQP server. +* `password`: Optional Password +* `use_ssl`: Use of TLS for the connection. Make sure to also set the correct port. Defaults to false. + +A full description and all parameters of the bot are in the [AMQP collector's documentation](../../user/bots.md#intelmq.bots.collectors.amqp.collector_amqp). + +##### Parser + +No parser is needed. + +When configuring this via the IntelMQ Manager, it will notify you that the parser is missing. In this case, you can ignore this message. + +#### On the sender + +##### Output + +Module: `intelmq.bots.outputs.amqptopic.output` + +###### Parameters + +AMQP Connection: + +- `connection_host`: Hostname of the AMQP server +- `connection_port`: Port of the AMQP server. Defaults to 5672. +- `connection_vhost`: (optional, string) Virtual host to connect, on an http(s) connection would be `http://IP/`. FIXME test +- `use_ssl`: Set to `true` to use SSL/TLS. Also adapt the connection port +- `delivery_mode`: `2` for persistent delivery +- `require_confirmation`: `true` to guarantee delivery + +AMQP Routing: + +- `exchange_durable`: `true` to make the exchange survive server restarts +- `exchange_name`: Optional name of the exchange +- `exchange_type`: `topic` +- `routing_key`: The routing key for the AMQP Topic + +If authentication is in place: + +- `username`: Optional username for authentication +- `password`: Optional password for authentication + +Message settings: + +- `keep_raw_field`: `true` to keep the data original +* `message_hierarchical_output`: `false` (the default value) +* `message_with_type`: `true` +* `message_jsondict_as_string`: `true` + +A full description and all parameters of the bot are in the [AMQP output's documentation](../../user/bots.md#intelmq.bots.outputs.amqptopic.output). + +### ValKey / Redis + +FIXME: Diagram + +Open Redis ports + +### API + +FIXME: Diagram + +#### Security considerations + +FIXME + +#### Prerequisites + +Install the HTTP server library `tornado` (4.5.3 or later) on the receiver: +``` bash +# Debian/Ubuntu +sudo apt install python3-tornado +``` + +#### On the receiver + +##### Collector + +This bot starts a minimalist HTTP server and processes all messages that are received under the `/intelmq/push` path. + +Module: `intelmq.bots.outputs.api.collector` + +###### Parameters + +- `port`: The port to listen on. Default: 5000 +- All parameters related to UNIX sockets are not used for inter-machine communication + +A full description and all parameters of the bot are in the [API collector's documentation](../../user/bots.md#intelmq.bots.collectors.api.collector). + +##### Parser + +No parser is needed. + +When configuring this via the IntelMQ Manager, it will notify you that the parser is missing. In this case, you can ignore this message. + +#### On the sender + +##### Output + +Module: `intelmq.bots.outputs.restapi.output` + +###### Parameters + +- `host`: The full URL, e.g. `https://intelmq.example.com:4433/intelmq/push` +- `auth_type`, `auth_token` and `auth_token_name`: Optional if there is an additional reverse proxy with authentication in between +- `hierarchical_output`: `false` (the default value) +- `use_json`: `true` (the default value) + +A full description and all parameters of the bot are in the [API output's documentation](../../user/bots.md#intelmq.bots.outputs.restapi.output). diff --git a/intelmq/bots/collectors/amqp/REQUIREMENTS.txt b/intelmq/bots/collectors/amqp/REQUIREMENTS.txt index 70a5360fe4..b62ffbf3ba 100644 --- a/intelmq/bots/collectors/amqp/REQUIREMENTS.txt +++ b/intelmq/bots/collectors/amqp/REQUIREMENTS.txt @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: 2020 Sebastian Wagner +# SPDX-FileCopyrightText: 2020 nic.at GmbH # SPDX-License-Identifier: AGPL-3.0-or-later pika diff --git a/intelmq/bots/collectors/amqp/collector_amqp.py b/intelmq/bots/collectors/amqp/collector_amqp.py index 291aa1ecfc..7d789f8bb7 100644 --- a/intelmq/bots/collectors/amqp/collector_amqp.py +++ b/intelmq/bots/collectors/amqp/collector_amqp.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: 2019 Sebastian Wagner +# SPDX-FileCopyrightText: 2019 nic.at GmbH # # SPDX-License-Identifier: AGPL-3.0-or-later diff --git a/intelmq/bots/outputs/amqptopic/output.py b/intelmq/bots/outputs/amqptopic/output.py index 03c0faba5e..eaf60f0f26 100644 --- a/intelmq/bots/outputs/amqptopic/output.py +++ b/intelmq/bots/outputs/amqptopic/output.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: 2016 Pedro Reis +# SPDX-FileCopyrightText: 2016 Pedro Reis, 2016-2021 nic.at GmbH # # SPDX-License-Identifier: AGPL-3.0-or-later diff --git a/mkdocs.yml b/mkdocs.yml index 2735e6d2e2..e95e1388c3 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -115,6 +115,7 @@ nav: - IntelMQ API: 'admin/configuration/intelmq-api.md' - IntelMQ Manager: 'admin/configuration/intelmq-manager.md' - Redis: 'admin/configuration/redis.md' + - Connecting Instances: 'admin/configuration/connecting-instances.md' - Management: - IntelMQ: 'admin/management/intelmq.md' - IntelMQ API: 'admin/management/intelmq-api.md' From 267abd2e05e16e21d3323c6ada54648888e1c2b9 Mon Sep 17 00:00:00 2001 From: Sebastian Wagner Date: Sun, 12 Apr 2026 22:31:02 +0200 Subject: [PATCH 6/9] doc: guide to setup all components in one a combined installation instruction for all tools --- CHANGELOG.md | 1 + docs/admin/installation/combined.md | 375 ++++++++++++++++++++++++++++ mkdocs.yml | 1 + 3 files changed, 377 insertions(+) create mode 100644 docs/admin/installation/combined.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 099d23faf1..997f4a3377 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,7 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o ### Documentation - Updates to Contrib and Overview pages (PR#2672 by Sebastian Wagner). +- Add a combined setup documentation for all IntelMQ tools including the CERTBund workflow (PR#2698). ### Packaging diff --git a/docs/admin/installation/combined.md b/docs/admin/installation/combined.md new file mode 100644 index 0000000000..31dd932745 --- /dev/null +++ b/docs/admin/installation/combined.md @@ -0,0 +1,375 @@ + + +This guide covers installing the full IntelMQ tool suite on Ubuntu 24.04: + +- **IntelMQ**: the core pipeline that processes the data +- **IntelMQ Manager**: web UI for managing and monitoring the pipeline +- **Fody**: web UI for managing contacts and viewing notifications (tickets) +- **IntelMQ CERTBund Contact**: enriches events with contact data from a contact database and includes the RIPE import +- **IntelMQ Mailgen**: notification tool to send emails to constituents + +In the future it will also include: + +- **IntelMQ Webinput**: web UI to interactively process uploaded data files and send it to constituents + +The setup can be deployed on a **single server** (application and database co-located) or split across **two servers** (dedicated database host). Both variants are described below. + +!!! warning + Never expose any interface of IntelMQ services to untrusted networks. IntelMQ is meant to be operated only in internal networks by internal operators. + +!!! note + HTTPS is not configured automatically. See [mod_md](https://httpd.apache.org/docs/2.4/mod/mod_md.html) for a straightforward way to provision TLS certificates via Apache. + +### Single-server setup + +```mermaid +graph TD + subgraph intelmq_server["IntelMQ and Database Server"] + subgraph pipeline + intelmq["IntelMQ (bots)"] + certbund["CERTBundContact-Expert CERTBundRule-Expert"] + output["SQL output bot"] + end + manager["IntelMQ Manager & API"] + fody_be["Fody Backend"] + mailgen["intelmq-mailgen"] + subgraph database + events_db[("intelmq-events (PostgreSQL)")] + contact_db[("contactdb (PostgreSQL)")] + end + end + + intelmq --> certbund + output -->|"write events"| events_db + certbund -->|"read contacts"| contact_db + fody_be -->|"read events"| events_db + fody_be -->|"read/write contacts"| contact_db + mailgen -->|"send emails"| smtp["SMTP"] + mailgen -->|"read directives"| events_db + manager -->|"manage"| intelmq + certbund --> output +``` + +### Two-server setup + +```mermaid +graph TD + subgraph intelmq_server["IntelMQ Server"] + subgraph pipeline + intelmq["IntelMQ (bots)"] + certbund["CERTBundContact-Expert CERTBundRule-Expert"] + output["SQL output bot"] + end + manager["IntelMQ Manager & API (Apache, port 80/443)"] + fody_be["Fody Backend (Apache, port 8000)"] + mailgen["intelmq-mailgen"] + end + + subgraph db_server["Database Server (separate host)"] + events_db[("intelmq-events (PostgreSQL)")] + contact_db[("contactdb (PostgreSQL)")] + end + + intelmq -->|"user: intelmq"| certbund + output -->|"write events (user: intelmq)"| events_db + certbund -->|"read contacts (user: intelmq)"| contact_db + fody_be -->|"read events (user: mailgen)"| events_db + fody_be -->|"read/write contacts (user: fody)"| contact_db + mailgen -->|"send emails"| smtp["SMTP"] + mailgen -->|"read directives (user: mailgen)"| events_db + manager -->|"manage"| intelmq + certbund --> output +``` + +### Ansible + +For automated orchestration, an ansible role is available: [`ansible-galaxy role install sebix.intelmq`](https://galaxy.ansible.com/ui/standalone/roles/sebix/intelmq/) (Source: [github.com/sebix/ansible-intelmq](https://github.com/sebix/ansible-intelmq)). +It covers all components listed here. + +### Network + +The webinterfaces will be reachable to you on these ports: + +- `http://yourip:80/intelmq-manager`: IntelMQ Manager & API +- `http://yourip:8000/`: IntelMQ Fody + +In the firewall settings, both ports need to be opened to the operator's network. + +In case of the two-server layout, the PostgreSQL port (5432/tcp) on the database server needs to be reachable from the IntelMQ server. + +## Application installation + +1. For IntelMQ follow https://docs.intelmq.org/develop/admin/installation/linux-packages/#ubuntu-2004-2204-and-2404 +2. Install some dependencies for IntelMQ bots: `sudo apt install python3-psycopg2 python3-pyasn python3-textx` +3. On the production systems: `sudo apt install intelmq-autostart moreutils` +4. On the test systems: `sudo apt install dsmtpd-intelmq mutt` + +### Fody + +On the IntelMQ server: + +1. Run `sudo dpkg-reconfigure locales` and select `en_US.UTF-8` +2. Then reboot the server +3. `sudo apt install intelmq-certbund-contact intelmq-mailgen intelmq-fody intelmq-fody-backend` +4. Install the PostgreSQL client library: `sudo apt install postgresql-client-16` +5. To use the same authentication in IntelMQ Fody as IntelMQ Manager: + 1. From `/etc/intelmq/api-config.json` copy the value of the field `session_store` (likely `/var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi`) + 1. Programmatically: `jq -r .session_store /etc/intelmq/api-config.json` + 2. In `/etc/intelmq/fody-session.conf` set this value at `session_store`. + 3. `sudo systemctl restart apache2` + +## Database installation and setup + +These instructions are partially based on and compatible with https://docs.intelmq.org/latest/admin/database/postgresql/#Setup + +1. On the database server install PostgreSQL and configure it for connections from the application server: + 1. `sudo apt install postgresql-16` Or, alternatively, use the APT repository by the PostgreSQL team: https://www.postgresql.org/download/linux/ubuntu/ + 2. If the database server is separate from the IntelMQ-server: + 1. In `/etc/postgresql/16/main/postgresql.conf` adapt the `listen_address` and set the public interface of the server + 2. In `/etc/postgresql/16/main/pg_hba.conf` add these lines replacing `$intelmqserver` with the IP address of the IntelMQ server accessing the Database server + 1. `host intelmq-events intelmq $intelmqserver/32 scram-sha-256` + 2. `host intelmq-events mailgen $intelmqserver/32 scram-sha-256` + 3. `host contactdb intelmq $intelmqserver/32 scram-sha-256` + 4. `host contactdb fody $intelmqserver/32 scram-sha-256` + 3. `sudo systemctl restart postgresql` +2. Create PostgreSQL roles and database: + 1. `sudo -u postgres createuser --no-superuser --no-createrole --no-createdb --encrypted --pwprompt intelmq` + 1. Set and remember the password + 2. `sudo -u postgres createuser --no-superuser --no-createrole --no-createdb --encrypted --pwprompt fody` + 1. Set and remember the password + 3. `sudo -u postgres createuser --no-superuser --no-createrole --no-createdb --encrypted --pwprompt mailgen` + 1. Set and remember the password + 4. `sudo -u postgres createuser --no-superuser --no-createrole --no-createdb contactdb_owner` + 5. `sudo -u postgres createdb --encoding='utf-8' --owner=intelmq intelmq-events --template template0` + 6. `sudo -u postgres createdb --encoding=UTF8 --template=template0 --owner=contactdb_owner contactdb` +3. To initialize the event database: + 1. Either: + 1. Run `intelmq_psql_initdb` on the IntelMQ server + 1. It creates an SQL-file, normally called `/tmp/initdb.sql` + 2. Copy the file to the database server with `scp` either on the server or via the client, depending on the ssh/network setup to a location where postgres can read it, e.g. `/tmp/` + 2. Or, if copying the file from server to server is not doable: + 1. Download the file on the database server from GitHub: + 2. `wget https://raw.githubusercontent.com/certtools/intelmq/refs/heads/3.3.1/intelmq/tests/bin/initdb.sql` + 3. Create the table `events` with: `sudo -u postgres psql intelmq-events < /tmp/initdb.sql` + 4. Remove the file: `sudo rm /tmp/initdb.sql` +4. To initialize the contact database: + 1. Either + 1. Copy `/usr/share/intelmq-certbund-contact/sql/initdb.sql` from the IntelMQ server to the database server to a location where postgres can read it, e.g. `/tmp/` + 2. Or + 1. download it on the database server from GitHub: + 2. `wget https://raw.githubusercontent.com/Intevation/intelmq-certbund-contact/refs/tags/1.1.0/sql/initdb.sql -O /tmp/initdb.sql` + 3. Create the tables with `sudo -u postgres psql -f /tmp/initdb.sql contactdb` + 4. Remove the file: `rm /tmp/initdb.sql` + 5. Create the roles and grant permissions: + 1. `sudo -u postgres psql contactdb` + 2. `CREATE ROLE contactdb_ro NOLOGIN NOSUPERUSER NOINHERIT NOCREATEDB CREATEROLE;` + 3. `CREATE ROLE contactdb_rw NOLOGIN NOSUPERUSER NOINHERIT NOCREATEDB CREATEROLE;` + 4. `GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO contactdb_rw;` + 5. `GRANT USAGE ON ALL SEQUENCES IN SCHEMA public TO contactdb_rw;` + 6. `GRANT SELECT ON ALL TABLES IN SCHEMA public TO contactdb_ro;` + 7. `GRANT contactdb_ro TO intelmq;` + 8. `GRANT contactdb_rw TO fody;` +5. To setup notifications tables: + 1. 1. Either + 1. Copy `/usr/share/intelmq-mailgen/sql/notifications.sql` from the IntelMQ server to the database server to a location where postgres can read it, e.g. `/tmp/` + 2. Or + 1. download it on the database server from GitHub: + 2. `wget https://raw.githubusercontent.com/Intevation/intelmq-mailgen/refs/tags/1.3.7/sql/notifications.sql -O /tmp/notifications.sql` + 3. Create the tables with `sudo -u postgres psql -f /tmp/notifications.sql intelmq-events` + 4. Remove the file: `rm /tmp/notifications.sql` + 5. Grant permissions: + 1. `sudo -u postgres psql -c "GRANT eventdb_insert TO intelmq;" intelmq-events` + 2. `sudo -u postgres psql -c "GRANT eventdb_send_notifications TO mailgen;" intelmq-events` + +## Fody configuration + +1. Optionally create email tags for Fody: https://github.com/Intevation/intelmq-certbund-contact/?tab=readme-ov-file#adding-default-email-tags +2. Backend configuration: + 1. In `/etc/intelmq/contactdb-serve.conf` set the database server host, user `fody` and its password + 2. In `/etc/intelmq/eventdb-serve.conf` set the database server host, user `mailgen` and its password + 3. In `/etc/intelmq/tickets-serve.conf` set the database server host, user `mailgen` and its password +3. Optionally add predefined tags (for network objects and organisations) in `/etc/intelmq/contactdb-serve.conf`: + ``` + { + "common_tags": [ "Whitelist:Malware", + "Whitelist:DNS-Open-Resolver", + "Whitelist:Shadowserver", + "Whitelist:All", + ], + ... + } + ``` +4. `sudo systemctl restart apache2.service` +5. The login should work now + 1. If not, have a look at `/var/log/apache2/fody-backend-error.log` +6. Add the CERTBund bots to the runtime configuration: + 1. Open the webinterface IntelMQ Manager `http://intelmq.example/intelmq-manager` + 2. Log in + 3. Go to page *Configuration* + 4. Insert these two bots right before your Output Bot: + 1. CERTBundContact-Expert + 1. In its configuration, set the database server, database user `intelmq` and its password + 2. CERTBundRule-Expert + 5. Save the new configuration + +### RIPE Data + +1. Create the configuration file + 1. `/etc/intelmq/contactdb-import.conf` and insert the country code you are interested in (here, as example `AT`): + ```json + { + "restrict_to_country": ["AT"] + } + ``` +2. Import RIPE data the first time + 1. https://github.com/Intevation/intelmq-certbund-contact/blob/master/README-ripe-import.md + 2. `ripe_download` + 3. change to the created directory (named the current date) + 4. `ripe_import` +3. Configure the automatic update + 1. Add a cronjob, e.g. every Monday evening in `/etc/crontab`: + 1. `40 17 * * * intelmq /usr/local/bin/ripe_update.sh` + 2. If the mail setup of the server is correct, an email with the differences will be sent to the administrator + +## Mailgen + +### Configuration + +1. `mv /etc/intelmq/intelmq-mailgen.conf.example /etc/intelmq/intelmq-mailgen.conf` +2. In `/etc/intelmq/intelmq-mailgen.conf` set the database host, user `mailgen` and its password + +### Test-System: Capture E-Mails + +- On the test server: `sudo apt install dsmtpd-intelmq mutt` +- To read the e-mails use this command: `sudo mutt -Rf /var/lib/dsmtpd/Maildir` +- on the test server, in `/etc/intelmq/intelmq-mailgen.conf`, set the SMTP server to `localhost:1025` like this: + ``` + "smtp": { + "host": "127.0.0.1", + "port": 1025 + } + ``` + +### Templates and Scripts + +- The documentation is at https://intevation.github.io/intelmq-mailgen/ +- Configure your notification rule scripts in `/var/lib/intelmq/bots/notification_rules/` on the server + - The example rule scripts are located in `/usr/share/doc/intelmq-certbund-contact/example-rules/` + - A good starting minimal point are these rule scripts + - `05_meta.py` + - `06_whitelist.py` + - `08_remove_invalid.py` + - `10_prioritize_contacts.py` + - `15_demo.py` + - Delete any unneeded files in that directory +- Eventually delete old unsent directives if there are any from previous runs, so the current scripts can process all unprocessed directives: + - `psql intelmq-events intelmq_mailgen` (or use the respective user instead of `intelmq_mailgen`) + - `DELETE FROM directives WHERE sent_id IS NULL;` +- then restart the rules expert: `intelmqctl restart contact-rule-expert +- Configure your mailgen format scripts in `/etc/intelmq/mailgen/format` on the server + - A good starting point are these format scripts: + - `00_add_variables_to_context.py` + - `20_demo.py` + - Delete any unneeded files in that directory +- Configure the mailgen templates from in `/etc/intelmq/mailgen/templates`on the sever: + - Examples templates are located in `/usr/share/doc/intelmq-mailgen/templates/` + - The template `demo` is required for the demo rules and formats previously configured. +- The mailgen configuration is located in `/etc/intelmq/intelmq-mailgen.conf` + - In a two-server scenario, add the database server hostname or IP address. +- Make simulation runs: `intelmqcbmail -n` + - To actually send emails, without `-n` + - Get more informative output (verbose mode) with `-v` + - To view the emails, use `sudo mutt -Rf /var/lib/dsmtpd/Maildir` + +## Permission overview + +### UNIX users + +- **intelmq**: unprivileged user that runs all IntelMQ Bots, + the IntelMQ API and the command line tools including `intelmqcbmail` + usually only exists on the server running IntelMQ, not a dedicated database server (if separated), unless `intelmqcbmail` is run on the database server + +### IntelMQ Webinterface users + +Both the IntelMQ Manager (actually: the IntelMQ API) and Fody have a basic access control with username/password. +The backends use an SQLite file-database for user- and session management. +The applications can use different databases (default) or the same one. + +The Manager's database is defined in `/etc/intelmq/api-config.json`, and for Fody it is in `/etc/intelmq/fody-session.conf`. +The key `session_store` defines the path to the database file. +They can be set to the same value, then the user database is equal for both applications. + +#### Create users + +``` +sudo fody-adduser --user username +# enter password on stdin +``` + +#### Delete users + +``` +sudo sqlite3 $(jq -r .session_store /etc/intelmq/fody-session.conf) +.headers on +# list users: +SELECT * FROM user; +# delete a user +DELETE FROM user WHERE username = 'username_to_delete'; +``` + +### PostgreSQL + +The PostgreSQL users are independent of UNIX users[^1] + +#### Change users + +``` +sudo -u postgres psql +# Create new user: +CREATE USER username NOSUPERUSER NOCREATEROLE NOCREATEDB ENCRYPTED PASSWORD 'password'; +# Show all users: +\dg +# Change password of user +ALTER USER username WITH PASSWORD 'new_password'; +# Rename a user +ALTER USER old_username RENAME TO new_username; +# Delete a user +DROP USER username; +# If the user "cannot be dropped because some objects depend on it" if required, per database where relevant: +REVOKE ALL ON ALL TABLES IN SCHEMA public FROM username; +REVOKE ALL ON ALL SEQUENCES IN SCHEMA public FROM username; +``` + +#### Database `intelmq-events` + +Roles (Users/groups in PostgreSQL-speak): +- `intelmq`: Used by the PostgreSQL Output Bot to insert data into the `events` table. Is a member of `eventdb_insert`. +- `mailgen`: Used by `intelmqcbmail` command line tool to send e-mail notifications. Member of `eventdb_send_notifications` + +##### Permissions + +| Table ↓ \ Role → | `eventdb_insert` (`intelmq`) | `eventdb_send_notifications` (`mailgen`) | `eventdb_owner` | +| ---------------- | ---------------------------- | ---------------------------------------- | --------------- | +| events | INSERT | SELECT | ALL | +| sent | - | SELECT, UPDATE[^2] | ALL | +| directives | - | SELECT, UPDATE | ALL | +| ticket_day | - | SELECT, UPDATE | ALL | + +#### Database `contactdb` + +Roles: +- `intelmq`: read-only user used by IntelMQ contact expert to query information from the contact database. Member of `contactdb_ro` +- `fody`: user used by Fody with write-access to the database tables, member of `contactdb_rw` + +##### Permissions + +| Table ↓ \ Role → | `contactdb_ro` (`intelmq`) | `contactdb_rw` (`fody`) | `contactdb_owner` | +| ---------------- | -------------------------- | ----------------------- | ----------------- | +| \* | SELECT | | ALL | +| \* | | SELECT, INSERT, UPDATE | ALL | + +[^1]: except for peer authentication, which IntelMQ does not use +[^2]: For simplicity, `UPDATE` means any write-access including `INSERT`, `DELETE`, `TRUNCATE` diff --git a/mkdocs.yml b/mkdocs.yml index e95e1388c3..e809476bde 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -108,6 +108,7 @@ nav: - Linux Package: 'admin/installation/linux-packages.md' - PyPI: 'admin/installation/pypi.md' - DockerHub: 'admin/installation/dockerhub.md' + - Combined All in One Setup: 'admin/installation/combined.md' - Upgrade: 'admin/upgrade.md' - Hardware Requirements: 'admin/hardware-requirements.md' - Configuration: From f2a1225cc2f50b52822a56c1118e4db15654d319 Mon Sep 17 00:00:00 2001 From: Sebastian Wagner Date: Wed, 22 Apr 2026 23:09:03 +0200 Subject: [PATCH 7/9] doc: connection intelmq instances --- CHANGELOG.md | 4 +- .../configuration/connecting-instances.md | 479 ++++++++++++++++-- mkdocs.yml | 9 +- 3 files changed, 451 insertions(+), 41 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 997f4a3377..c3d82a0151 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,7 +38,9 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o ### Documentation - Updates to Contrib and Overview pages (PR#2672 by Sebastian Wagner). -- Add a combined setup documentation for all IntelMQ tools including the CERTBund workflow (PR#2698). +- Add new documentation pages (PR#2698): + - A combined setup documentation for all IntelMQ tools including the CERTBund workflow. + - How to connect multiple IntelMQ instances. ### Packaging diff --git a/docs/admin/configuration/connecting-instances.md b/docs/admin/configuration/connecting-instances.md index 59abe18ae8..a44b39ddd1 100644 --- a/docs/admin/configuration/connecting-instances.md +++ b/docs/admin/configuration/connecting-instances.md @@ -5,7 +5,8 @@ # Connecting IntelMQ instances -The same methods can be used to connect to link up with other systems via +IntelMQ instances can be connected to each other by different methods to forward data across offices, departments, or organisations. +The same methods can also be used to connect IntelMQ with other systems. ## Use cases @@ -17,68 +18,279 @@ The same methods can be used to connect to link up with other systems via ### Data feeds -One organization generates a data feed, processes it with IntelMQ and another organization wants to receive it with IntelMQ. +One organisation collects and processes threat intelligence with IntelMQ, providing a data feed for others to consume. +The receiving organisation ingests that feed into their own IntelMQ instance for further processing and local use. + +This model is common for open data sharing initiatives or threat intelligence providers. + +```mermaid +flowchart LR + subgraph Producer["Feed Producer"] + direction TB + Collect["Collect & Process"] --> Publish["Publish Feed"] + end + subgraph Consumer["Feed Consumer"] + direction TB + Ingest["Ingest Feed"] --> Process["Process & Use"] + end + Publish -- "data feed" --> Ingest + + style Publish fill:#E1BEE7 + style Ingest fill:#E1BEE7 +``` ### Connections between trusted organizations -If two organizations both use IntelMQ, they can route messages to each other. +Two organizations that both use IntelMQ can route threat intelligence directly to each other without an intermediary. +Each side controls what data it sends and receives. +Both sides must agree to share data with each other. + -### Connections between arbitrary organizations +Examples: -FIXME +- International corporations forwarding indicators to national branches +- National CSIRTs distributing threat data down to federal state-level, sectoral CSIRTs or network operators +- Peer organizations in an industry sharing sector-specific threat intelligence + +```mermaid +flowchart LR + Org1["Organisation 1"] -- "sends to" --> Org2["Organisation 2"] + Org2 -- "sends to" --> Org1 +``` -Injection of +### Connections between arbitrary organizations using a central instance + +A central data broker ("data hub") that is trusted by the network participants routes the information between them. + +```mermaid +flowchart LR + Org1["Organisation 1"] --> Hub["Data Hub"] + Org2["Organisation 2"] --> Hub + Org3["Organisation 3"] --> Hub + Hub --> Org1 + Hub --> Org2 + Hub --> Org3 + + style Hub fill:#E1BEE7 +``` ## Architecture -FIXME +In all connection methods, data flows from one IntelMQ instance to another via some form of transport: either a direct connection or through an intermediary broker. +A direct connection is simpler to set up but offers no buffering: if the receiver is unavailable, messages are lost. +A message broker decouples the sender and receiver, adding reliability, queuing, and often better security controls at the cost of additional infrastructure. -### From IntelMQ outwards -FIXME -### Message broker +```mermaid +--- +config: +layout: elk +--- +flowchart LR + Org1["IntelMQ Instance 1"] -->|safe connection| Broker["Transport means"] + Broker -->|safe connection| Org2["IntelMQ Instance 2"] + + style Org1 fill:transparent + style Broker fill:#E1BEE7 + style Org2 fill:transparent +``` + +### Getting data into IntelMQ -FIXME +Normally, IntelMQ requires two components to process incoming data: +- A collector bot, responsible for receiving the raw data, and +- A parser bot, responsible for parsing the data and converting it into the IntelMQ Data Format -It does not matter whether the message broker is technically or organizationally located in the sending or receiving side. +When connecting IntelMQ instances, the parsing is not required as the received data is already in the internal format. -### Into IntelMQ +### Message broker -FIXME +A message broker is an intermediary that accepts messages from the sender and holds them until the receiver consumes them. +It does not matter whether the broker is technically or organizationally located on the sending or receiving side. ## Security Considerations -FIXME +Each data exchange method has its own security implications. +Carefully read the security considerations in each method's section for specifics. +The following points apply regardless of the method used: + +- **Treat external data as untrusted**: Data arriving from other organisations may contain unexpected or malignant content. At least set the `feed.provider` and `feed.name` fields to identify its origin, so events can be traced back to the source. +- **Authentication**: Every external connection should require authentication. Anonymous access should not be permitted for inter-organisational connections. +- **Transit encryption**: Use TLS or VPNs for all connections outside of trusted internal networks. +- **Limit what each connection can do**: A recipient/sender should only be able to read/write from/to the queues or endpoints that are necessary (least-privilege principle). ## Data exchange methods -FIXME +IntelMQ supports four methods for connecting instances. +The [recommendation matrix](#recommendation-matrix) below summarises when each is appropriate. + +- AMQP (RabbitMQ) as broker +- Redis/Valkey as broker or message queue with mixed usage +- HTTP API (IntelMQ bots) direct connection +- TCP (IntelMQ bots) basic direct connection + +### Recommendation matrix + +#### By usage + +| | Between organisations | Internal connections
between instances | Between related organisations
and departments | For data feeds | For data hubs | +| ---------------- | --------------------- | ----------------------------------------- | ------------------------------------------------ | ------------------------ | ------------------------ | +| **AMQP** | ⭐⭐⭐ | ⭐⭐⭐
If already in use | ⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐ | +| **Redis/Valkey** | ⭐
With ACLs | ⭐⭐⭐ | ⭐⭐
With ACLs | ❌️ | ❌️ | +| **API** | ⭐⭐⭐ | ⭐⭐ | ⭐⭐⭐ | ⭐⭐
With reverse-proxy | ⭐⭐
With reverse-proxy | +| **TCP** | ❌️ | ⭐⭐⭐ | ⭐ | ❌️ | ❌️ | + +#### By features + +| | Ease of setup | Security | Authentication | Authorization | Message persistence | Buffering if receiver down | TLS support | Monitoring | +| ---------------- | ---------------- | ----------------------- | ------------------------------- | --------------------------------------------- | ------------------------------------------ | ------------------------------ | ------------------------ | ------------------------------------ | +| **AMQP** | ⭐
Complex | ⭐⭐⭐⭐ | ⭐⭐⭐⭐
Per-user with passwords | ⭐⭐⭐⭐
Per-vhost, per-queue | ⭐⭐⭐⭐
Durable queues | ⭐⭐⭐⭐
Queues buffer messages | ⭐⭐⭐⭐
Built-in | ⭐⭐⭐⭐
Management UI + API | +| **Redis/Valkey** | ⭐⭐
Medium | ⭐⭐⭐ | ⭐⭐⭐
`requirepass` or ACLs | ⭐⭐⭐
ACLs (on key and command level) | ⭐⭐
Optional (RDB/AOF) | ⭐⭐⭐⭐
Lists buffer messages | ⭐⭐⭐⭐
Built-in | ⭐⭐
`INFO` command, external tools | +| **API** | ⭐⭐⭐
Low | ⭐⭐
Via reverse proxy | ⭐⭐
Token via reverse proxy | ⭐⭐
IP allowlist via reverse proxy/firewall | ⭐⭐
Reception: no
Sending: by IntelMQ | ⭐
Fails immediately | ⭐⭐⭐
Via reverse proxy | ⭐⭐
Depends on reverse proxy | +| **TCP** | ⭐⭐⭐⭐
Very low | —
None
| —
None | —
None | —
Reception: no
Sending: by IntelMQ | ⭐
Fails immediately | —
None | —
None | ### AMQP -FIXME: Diagram +This is the most sophisticated option and gives the best security, data flow control, and stability. +The downside is that it is a bit more complex to set up. -This is the most sophisticated option and gives the best security, data flow control, stability. -The downside is, that it is more complex to setup and to operate. +The protocol is called AMQP (Advanced Message Queuing Protocol) and is designed for reliable, asynchronous message passing between systems. +It supports features such as message acknowledgement, encryption and flexible routing via exchanges and routing keys. + +The broker software used here is [**RabbitMQ**](https://www.rabbitmq.com/). +It acts as a central hub that receives messages from senders and delivers them to receivers. +RabbitMQ handles queuing, routing, and persistence, so messages are not lost when the receiving side is temporarily unavailable. + +There are two common deployment patterns: + +**Multiple organisations, each with their own broker**: each organisation runs its own RabbitMQ instance. +A sender bot on one side publishes to the remote broker of the receiving organisation, which an AMQP collector bot on that side consumes from. + +```mermaid +graph LR + subgraph Org1["Organisation 1"] + B1["Bots"] --> MQ1["RabbitMQ"] + end + subgraph Org2["Organisation 2"] + MQ2["RabbitMQ"] --> B2["Bots"] + end + MQ1 -- "AMQP output → collector" --> MQ2 + + style MQ1 fill:#E1BEE7 + style MQ2 fill:#E1BEE7 +``` + +**One organisation, multiple instances, one shared broker**: all IntelMQ instances connect to a single central RabbitMQ broker. +Bots on any instance can publish to or consume from shared queues without needing direct instance-to-instance connectivity. + +**Two organisations – pull method:** + +```mermaid +--- +config: +layout: elk +--- +flowchart LR + subgraph Org1["Organisation 1"] + direction TB + C1["Collectors"] + P1["Parsers"] + E1["Experts"] + O1["Output Bot"] + B1["RabbitMQ"] + end + subgraph Org2["Organisation 2"] + direction TB + C2["Collectors"] + P2["Parsers"] + E2["Experts"] + O2["Output Bot"] + end + + C1 --> P1 + P1 --> E1 + E1 --> O1 + O1 --> B1 + B1 -- pull --> C2 + C2 --> P2 + P2 --> E2 + E2 --> O2 + + style B1 fill:#E1BEE7 +``` + +**Two organisations – push method:** + +```mermaid +--- +config: +layout: elk +--- +flowchart LR + subgraph Org1["Organisation 1"] + direction TB + C1["Collectors"] + P1["Parsers"] + E1["Experts"] + O1["Output Bot"] + end + subgraph Org2["Organisation 2"] + direction TB + B1["RabbitMQ"] + C2["Collectors"] + P2["Parsers"] + E2["Experts"] + O2["Output Bot"] + end + + C1 --> P1 + P1 --> E1 + E1 --> O1 + O1 -- push --> B1 + B1 --> C2 + C2 --> P2 + P2 --> E2 + E2 --> O2 + + style B1 fill:#E1BEE7 +``` + +#### Differentiation to using AMQP as message broker + +AMQP can be used in two distinct roles within IntelMQ. + +Internally, AMQP it can replace Redis as the message broker between bots within a single IntelMQ instance. +See [Using the AMQP broker](../beta-features.md#using-amqp-message-broker) for that use case. + +This section focuses on using AMQP to connect separate IntelMQ instances to each other, either within the same organisation or across organisational boundaries. + +It is not necessary to use AMQP as internal message broker when using AMQP for external connections. +A mix of of Redis and AMQP is perfectly possible and useful. #### Security considerations -FIXME +- **Use TLS**: Always encrypt data in transit between organisations and in insecure networks. Set `use_ssl: true` on either side and configure RabbitMQ to listen on the TLS port. +- **Authentication**: always configure a dedicated RabbitMQ user with a strong password for each remote connection. Never use the default `guest` account. +- **Virtual hosts**: use separate RabbitMQ virtual hosts (`connection_vhost`) to isolate traffic between different organisations or pipelines. +- **Least-privilege permissions**: Grant each broker user only the permissions needed: read access on the queue for collectors, write access on the exchange for output bots. +- **Network exposure**: avoid exposing RabbitMQ directly to the internet. Prefer placing it behind a firewall and only opening the AMQP(S) port to known IP ranges. A VPN or dedicated interconnect between organisations is strongly recommended. If RabbitMQ needs to be exposed to the Internet, always configure strong transport encryption and authentication. +- **Resource exhaustion**: Without limits, a stalled consumer can exhaust the broker's memory and disk space. Configure queue length limits and message TTLs to prevent a slow or unresponsive receiver from causing unbounded queue growth. +- **Monitoring**: Integrate RabbitMQ into your existing monitoring systems to detect any problems or warning signs. #### Prerequisites -Install the Python Library `pika` for AMQP on both the sender and the receiver: +Install the Python Library `pika` for AMQP on the servers that access the RabbitMQ instance: ``` bash # Debian/Ubuntu sudo apt install python3-pika ``` -#### On the receiver +#### On the receiver (pull-method) -##### Collector +If the broker is in the sender's organisation, then the receiver configures an AMQP collector, fetching the data from the data provider. -FIXME +##### Collector Module: `intelmq.bots.collectors.amqp.collector_amqp` @@ -89,7 +301,7 @@ These configuration parameters are required: - `connection_host`: Hostname of the AMQP server. - `connection_port` Port of the AMQP server. Defaults to 5672. - `connection_vhost`: Virtual host to connect, on an HTTP(S) connection would be ``. -- `expect_intelmq_message`: `true`. FIXME This parameter denotes whether the the data is from IntelMQ or not. If true, then the data can be any Report or Event and will be passed to the next bot as is. Otherwise a new Report is created with the raw data. Defaults to false. +- `expect_intelmq_message`: `true`. This parameter denotes whether the data is from IntelMQ or not. If true, then the data can be any Report or Event and will be passed to the next bot as is. Otherwise a new Report is created with the raw data. Defaults to false. * `queue_name`: (optional, string) The name of the queue to fetch the data from. * `username`: Optional username for authentication to the AMQP server. * `password`: Optional Password @@ -101,9 +313,12 @@ A full description and all parameters of the bot are in the [AMQP collector's do No parser is needed. -When configuring this via the IntelMQ Manager, it will notify you that the parser is missing. In this case, you can ignore this message. +When configuring this via the IntelMQ Manager, it will notify you that the parser is missing. +In this case, you can ignore this message. -#### On the sender +#### On the sender (push method) + +If the broker is in the recipient's organisation, then the sender configures an AMQP output, pushing the data to the data recipient. ##### Output @@ -115,7 +330,7 @@ AMQP Connection: - `connection_host`: Hostname of the AMQP server - `connection_port`: Port of the AMQP server. Defaults to 5672. -- `connection_vhost`: (optional, string) Virtual host to connect, on an http(s) connection would be `http://IP/`. FIXME test +- `connection_vhost`: (optional, string) Virtual host to connect, on an http(s) connection would be `http://IP/` - `use_ssl`: Set to `true` to use SSL/TLS. Also adapt the connection port - `delivery_mode`: `2` for persistent delivery - `require_confirmation`: `true` to guarantee delivery @@ -127,7 +342,7 @@ AMQP Routing: - `exchange_type`: `topic` - `routing_key`: The routing key for the AMQP Topic -If authentication is in place: +Authentication: - `username`: Optional username for authentication - `password`: Optional password for authentication @@ -141,21 +356,145 @@ Message settings: A full description and all parameters of the bot are in the [AMQP output's documentation](../../user/bots.md#intelmq.bots.outputs.amqptopic.output). -### ValKey / Redis +### Valkey / Redis + +Valkey/Redis are used as default messaging broker internally by IntelMQ for the communication between the bots. +It's fast and trivial to setup for internal usage, and can also be used as a very simple connection between internal, highly trusted instances. + +Redis is the original open-source in-memory data store that IntelMQ has historically used as its internal message broker. +Valkey is a community-maintained fork of Redis created in 2024 after Redis changed its license. +Valkey is a drop-in replacement: IntelMQ supports both, and the configuration is identical. +Valkey is the recommended choice for new deployments on distributions that package it (e.g. Debian 13 trixie and newer). + +#### Multiple instances using the same messaging broker + +To expose a Redis/Valkey instance to a another IntelMQ instance, the simplest approach is to bind it to a non-loopback interface and open the port in the firewall. +The receiving side then points its `source_queues` directly at the remote host. + +!!! warning + Sharing the internal message broker between instances removes isolation between them. + Use this only for highly trusted instances on the same private network. + For any cross-organisational connection, use a separate dedicated broker instead. -FIXME: Diagram -Open Redis ports +```mermaid +flowchart LR + subgraph Org1["Organisation 1"] + MQ["Redis/Valkey"] + subgraph Inst1["Instance 1"] + Bots1["Bots"] + end + subgraph Inst2["Instance 2"] + Bots2["Bots"] + end + Bots1 <-->|"message broker"| MQ + Bots2 <-->|"message broker"| MQ + end -### API + style MQ fill:#E1BEE7 +``` -FIXME: Diagram +#### Separate Redis/Valkey instance + +Rather than exposing the internal message broker, a dedicated Redis/Valkey instance can be set up solely for inter-instance communication. +This keeps the internal queue isolated from external access and limits the capabilities if the shared broker is compromised or overloaded. + +The sender side writes to a queue on the shared broker, the receiver side reads from it, mirroring the pull method used with AMQP. + +```mermaid +flowchart TB + subgraph Org1["Organisation 1"] + Bots1["Bots"] + MQ1["Redis/Valkey"] + end + subgraph Org2["Organisation 2"] + Bots2["Bots"] + MQ["Dedicated Redis/Valkey"] + MQ2["Redis/Valkey"] + end + MQ --> Bots2 + Bots1 -- push --> MQ + Bots1 <-->|"message broker"| MQ1 + Bots2 <-->|"message broker"| MQ2 + + style MQ fill:#E1BEE7 +``` #### Security considerations -FIXME +- **Authentication**: Use at least the simple `requirepass` directive or, preferably, ACL-based authentication (see below) to prevent unauthenticated access. Never expose a password-free instance outside localhost. See [Redis authentication documentation](https://redis.io/docs/latest/operate/oss_and_stack/management/security/#authentication). +- **ACLs**: Redis and Valkey both support Access Control Lists, allowing fine-grained per-user permissions. Create a dedicated user for each remote connection and restrict it to only the commands and key patterns it needs. In `valkey.conf` or `redis.conf`: -#### Prerequisites + ``` + # Sender side: allow only writing to a specific queue for a specific user with a given password + ACL SETUSER intelmq-sender on >yourpassword ~intelmq-queue -@all +LPUSH + + # Receiver side: allow only reading + ACL SETUSER intelmq-receiver on >yourpassword ~intelmq-queue -@all +BRPOP + ``` + + Alternatively, manage ACLs in a separate file via the `aclfile` directive and apply changes at runtime with `ACL LOAD`. See the [Valkey ACL documentation](https://valkey.io/topics/acl/) and the [Redis ACL documentation](https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/). +- **Network exposure**: Bind Redis/Valkey to a specific interface rather than the default `0.0.0.0`. Restrict access to the port via firewall rules, allowing only the IP addresses of known IntelMQ instances. +- **TLS**: Both Redis and Valkey support TLS to encrypt data in transit. Enable it via the `tls-port`, `tls-cert-file`, `tls-key-file`, and `tls-ca-cert-file` directives in the server configuration. See the [Valkey TLS documentation](https://valkey.io/topics/tls/) and the [Redis TLS documentation](https://redis.io/docs/latest/operate/oss_and_stack/management/security/encryption/) for details. +- **Protected mode**: Redis and Valkey ship with protected mode enabled by default, which blocks external connections unless authentication is configured. Do not disable protected mode without putting other controls in place. + +#### On the sender + +##### Output + +Module: `intelmq.bots.outputs.redis.output` + +###### Parameters + +- `redis_server_ip`: Hostname of the remote Redis server +- `redis_server_port`: Port of the Redis server. Defaults to 6379. +- `redis_db`: Redis database number. Defaults to 2. +- `redis_password`: Redis server password. Defaults to null. +- `redis_queue`: Redis queue name, such as `remote-server-queue` +- `redis_timeout`: Connection timeout, in milliseconds. Defaults to 5000. +- `hierarchical_output`: Set to `false`, the default value. Split field names by a dot. +- `with_type`: Set to `true`, the default value. Whether to include `__type` field. + +### API bots + +IntelMQ provides HTTP-based bots for sending and receiving events over a simple REST-like interface. +The receiver side runs a lightweight HTTP server (the API collector bot) and the sender side posts events to it using the REST API output bot. + +This method works well across network boundaries where AMQP or Redis ports are not permitted. +It is a push-only, unidirectional connection. + +A reverse proxy (e.g. nginx or Apache) in front of the API collector is recommended for production deployments, to handle TLS, rate limiting, and authentication. + +```mermaid +flowchart LR + subgraph Org1["Sender"] + Bots1["Bots"] --> Out["REST API Output"] + Out["REST API Output"] + end + subgraph Org2["Receiver"] + Proxy["Reverse Proxy\n(TLS, auth)"] --> In["API Collector"] + In --> Bots2["Bots"] + end + Out -- "HTTPS" --> Proxy + + style Out fill:#E1BEE7 + style In fill:#E1BEE7 + style Proxy fill:#FFF9C4 +``` + +A popular example for the API method in a data feed distribution is the Feed [Have I Been Pwned](../..//user/feeds.md#have-i-been-pwned). +The feed documentation includes an **example nginx configuration** for reverse proxying including TLS offloading and authentication. + +#### Security considerations + +- **TLS**: Always serve the API collector behind HTTPS. +- **Authentication**: The REST API output bot supports token-based authentication via `auth_type` and `auth_token`. Configure the reverse proxy to require this token, so unauthenticated requests are rejected before reaching the collector. +- **Network exposure**: Expose only the HTTPS port. Do not expose the collector's internal port (default: 5000) directly to the internet +- **Network whitelisting**: Restrict access by IP allowlist at the reverse proxy or firewall. + +#### On the receiver + +##### Prerequisites Install the HTTP server library `tornado` (4.5.3 or later) on the receiver: ``` bash @@ -163,8 +502,6 @@ Install the HTTP server library `tornado` (4.5.3 or later) on the receiver: sudo apt install python3-tornado ``` -#### On the receiver - ##### Collector This bot starts a minimalist HTTP server and processes all messages that are received under the `/intelmq/push` path. @@ -182,7 +519,8 @@ A full description and all parameters of the bot are in the [API collector's doc No parser is needed. -When configuring this via the IntelMQ Manager, it will notify you that the parser is missing. In this case, you can ignore this message. +When configuring this via the IntelMQ Manager, it will notify you that the parser is missing. +In this case, you can ignore this message. #### On the sender @@ -198,3 +536,66 @@ Module: `intelmq.bots.outputs.restapi.output` - `use_json`: `true` (the default value) A full description and all parameters of the bot are in the [API output's documentation](../../user/bots.md#intelmq.bots.outputs.restapi.output). + +### TCP bots + +TCP is the simplest connection method: the sender opens a direct TCP connection to the receiver and pushes events one by one, waiting for a plain "Ok" acknowledgement after each message. + +There is no broker, no queuing and no built-in security. + +Use this method only for simple, trusted internal connections where simplicity more than reliability or security. + +```mermaid +flowchart LR + subgraph Org1["Sender"] + Bots1["Bots"] --> Out["TCP Output"] + end + subgraph Org2["Receiver"] + In["TCP Collector"] --> Bots2["Bots"] + end + Out -- "TCP (direct)" --> In + + style Out fill:#E1BEE7 + style In fill:#E1BEE7 +``` + +#### Security considerations + +- **No authentication and authorization**: There is no support for user authentication and authorization. No permission management. +- **No encryption**: Data is transmitted in plain text. Use and VPNs to protect the traffic. +- **Network exposure**: Never expose the TCP collector port to the internet. Restrict it to known sender IP addresses via firewall rules. + +#### On the receiver + +##### Collector + +Module: `intelmq.bots.collectors.tcp.collector` + +###### Parameters + +- `ip`: IP address to listen on. +- `port`: Port to listen on. + +A full description and all parameters of the bot are in the [TCP collector's documentation](../../user/bots.md#intelmq.bots.collectors.tcp.collector). + +##### Parser + +No parser is needed. + +When configuring this via the IntelMQ Manager, it will notify you that the parser is missing. +In this case, you can ignore this message. + +#### On the sender + +##### Output + +Module: `intelmq.bots.outputs.tcp.output` + +###### Parameters + +- `ip`: Hostname of the TCP collector. +- `port`: Port of the TCP collector. +- `counterpart_is_intelmq`: Set to `true` (the default) when connecting to an IntelMQ TCP collector. The output bot will then wait for the "Ok" acknowledgement after each message. +- `hierarchical_output`: `false` (the default value). + +A full description and all parameters of the bot are in the [TCP output's documentation](../../user/bots.md#intelmq.bots.outputs.tcp.output). diff --git a/mkdocs.yml b/mkdocs.yml index e809476bde..09e150eb54 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -75,6 +75,9 @@ extra: - icon: fontawesome/brands/docker link: https://hub.docker.com/u/certat +extra_javascript: + - https://unpkg.com/mermaid/dist/mermaid.min.js + markdown_extensions: - pymdownx.highlight: anchor_linenums: true @@ -85,7 +88,11 @@ markdown_extensions: - pymdownx.snippets - admonition - pymdownx.details - - pymdownx.superfences + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format - pymdownx.keys # keyboard keys - tables - footnotes From fd9618450caf341d5cbc7b782969b279a507c354 Mon Sep 17 00:00:00 2001 From: Sebastian Wagner Date: Thu, 30 Apr 2026 13:56:40 +0200 Subject: [PATCH 8/9] doc: abuse contacts: add models, CRM tools, graphs --- CHANGELOG.md | 3 +- docs/user/abuse-contacts.md | 205 +++++++++++++++++++++++++++++++----- 2 files changed, 182 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c3d82a0151..4b2271a90b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,9 +38,10 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o ### Documentation - Updates to Contrib and Overview pages (PR#2672 by Sebastian Wagner). -- Add new documentation pages (PR#2698): +- Add new documentation (PR#2698): - A combined setup documentation for all IntelMQ tools including the CERTBund workflow. - How to connect multiple IntelMQ instances. + - Abuse contacts: document models and tooling suites. ### Packaging diff --git a/docs/user/abuse-contacts.md b/docs/user/abuse-contacts.md index daa9dc4742..58185d28ec 100644 --- a/docs/user/abuse-contacts.md +++ b/docs/user/abuse-contacts.md @@ -1,43 +1,196 @@ - # Abuse-contact look-ups -The right decision whom to contact about a specific incident is vital to get the incident resolved as quick as possible. Different types of events may required different abuse-contact to be selected. For example, issues about a device, e.g. a vulnerability in the operating system or an application, is better sent to the hoster which can inform the server administrator. For website-related issues, like defacements or phishing, the domain owner (maintaining the content of the website) could be the better and more direct contact. Additionally, different CERT's have different approaches and different contact databases. Multiple information sources have different information, and some sources are more accurate than others. IntelMQ can query multiple sources of abuse-contacts and combine them. Internal databases, like a Constituency Portal provide high-quality and first-hand contact information. The RIPE document [Sources of Abuse Contact Information for Abuse Handlers](https://www.ripe.net/publications/docs/ripe-658) contains a good summary of the complex of themes. +Choosing the right notification contact for a security incident or vulnerability is essential to getting it resolved quickly. +The correct contact depends on the incident type: a device vulnerability is best reported to the hosting provider (who can reach the server administrator), while a defacement or phishing site is better directed to the domain owner responsible for the content. -## Sources for abuse-contacts +Different organisations also have different contact databases and workflows. +A SOC typically manages contacts for its own organisation and subsidiaries, while a national CSIRT covers all networks within a country. +IntelMQ supports both scenarios: it offers multiple tool suits and can query multiple abuse-contact sources and combine the results. + +Management suites for constituencies, contacts and network objects provide the highest-quality contact data because network owners maintain it themselves. +Public sources like RIPE are broader and less precise, but provide a very good base and are also suitable for finding related infrastructure in the first place. + +The RIPE document [Sources of Abuse Contact Information for Abuse Handlers](https://www.ripe.net/publications/docs/ripe-658) gives a thorough overview of the available sources and their trade-offs. -All these bots add the queried contacts to the IntelMQ events in the field `source.abuse_contact` if not state otherwise in the documentation. +## Suites for contact management and CRM + +There are two established integrated solutions for managing contacts and networks. +Both of them are made for and integrated in IntelMQ. +They share also parts of their database models and the RIPE import tooling. + +### CERTBund + +The CERTBund suite was originally created for [CERTBund](https://www.cert-bund.de/) and provides a dual level contact management for automatic and manual contacts. +The processing is highly configurable by user-defined rules and formatting scripts. + +- [Documentation](https://intevation.github.io/intelmq-mailgen/) +- [Installation](../admin/installation/combined.md) +- The [combined installation documentation](../admin/installation/combined.md) includes the installation of this suite + +It consists of these components: + +- [`intelmq-certbund-contact`](https://github.com/Intevation/intelmq-certbund-contact/): two IntelMQ expert bots (Contact-Expert and Rule-Expert) that enrich events with contact information from the contact database. + Based on user-defined rules they determine the recipient, email template, and attachment format. + - RIPE import scripts keep contact data up to date automatically. +- [`intelmq-fody`](https://github.com/Intevation/intelmq-fody) & [`intelmq-fody-backend`](https://github.com/Intevation/intelmq-fody-backend): web interface for reading and editing contacts, querying sent notification tickets, and browsing the IntelMQ event database. +- [`intelmq-mailgen`](https://github.com/Intevation/intelmq-mailgen): tool that reads the notification directives written by the CERTBund experts and sends grouped event emails to the determined contacts. + Supports OpenPGP encryption, formatting is configured by scripts and templates. +- `contactdb`: The PostgreSQL database that covers Autonomous Systems, network ranges (CIDR), single IP addresses, and domains. + + +```mermaid +--- +config: + layout: dagre +--- +graph TD + subgraph intelmq["IntelMQ Core"] + collectors["Collectors"] --> parsers["Parsers"] + parsers --> cb_contact["CERTBundContact-Expert"] + cb_contact --> cb_rules["CERTBundRules-Expert"] + cb_rules --> output["SQL Output"] + end + subgraph certbundcontact["IntelMQ CERTBund Contact"] + rules[("Notification Rules")] --> cb_rules + ripe[("RIPE")] -- ripe_import --> contactdb[("ContactDB")] + manual[("manual contacts")] --> contactdb + contactdb --> cb_contact + fody["Fody (web UI)"] --> contactdb + end + + subgraph mailgeng["IntelMQ Mailgen"] + output -- write events & directives --> eventdb[("EventDB (intelmq-events)")] + + templates[("Templates")] --> mailgen["intelmq-mailgen"] + formats[("Notification Formats")] --> mailgen + eventdb -- read directives --> mailgen + mailgen -- send grouped emails --> recipients["Constituents"] + end + +``` + + +**Data model:** + +```mermaid +erDiagram + organisation ||--o{ contact : has + organisation ||--o{ ASNs : owns + organisation ||--o{ CIDRs : owns + organisation ||--o{ FQDNs : owns + contact }o--o{ Tags : "tagged with" + organisation }o--o{ Tags : "tagged with" + ASNs }o--o{ Tags : "tagged with" + FQDNs }o--o{ Tags : "tagged with" + CIDRs }o--o{ Tags : "tagged with" + + events ||--o{ directives : "triggers (on INSERT)" + directives }o--o| sent : "fulfilled by" +``` + +Contrary to tuency, there is no hierarchy of organisations. + +### Tuency + +[Tuency](https://gitlab.com/intevation/tuency/tuency) is a constituency portal that lets organisations self-administer their network objects and notification preferences. +It is maintained by [CERT.at](https://cert.at). + +- [Source Code](https://gitlab.com/intevation/tuency/tuency) +- [Documentation](https://tuency.cert.at/docs/) + +**Components** + +- **Tuency portal**: Web application where constituency members manage their organisations, network objects, and notification rules. Organisations can claim network objects (ASes, IP ranges, domains) themselves, all changes require approval by a tenant admin to prevent hostile takeovers and incorrect data. +- [**Keycloak**](https://www.keycloak.org/): Handles authentication for the Tuency portal and enables single sign-on with other tools in the stack. +- **[Request Tracker for Incident Response](https://bestpractical.com/rtir/) (RTIR)**: Ticketing system used to manage incidents and send notification emails to constituents. The `intelmq.bots.outputs.rt.output` bot creates an Incident ticket in RTIR for each event, and optionally a linked Investigation ticket containing the event details that RTIR then delivers to the contact in `source.abuse_contact`. +- [**intelmq extensions**](https://github.com/certat/intelmq-extensions/tree/main/intelmq_extensions/bots): + - Custom harmonization fields in the IntelMQ data format definition that are required by the workflow + - Tools to create tickets RTIR and deliver notifications to constituents + - Tuency Expert: IntelMQ expert bot that queries the Tuency API to retrieve the appropriate abuse contact and notification interval. + + +```mermaid +--- +config: + layout: dagre +--- +graph TD + subgraph intelmq["IntelMQ"] + collectors["Collectors"] --> parsers["Parsers"] + parsers --> expert["Tuency-Expert"] + expert -->|"enriched event"| output["RT Output"] + end + rtir -- "sends notification emails" --> constituents["Constituents"] + consti["Constituents"] -- "self-service" --> tuencyport + + tuencyport --> keycloak["Keycloak"] + tuencyport --> contactdb + expert -->|"queries"| tuencyport[("Tuency portal")] + output -- "creates tickets" --> rtir["RTIR"] +``` + +**Data model:** + +- **Tenant** (constituency): a sector CSIRT responsible for a set of organisations. Tuency supports multiple tenants. +- **Organisation**: belongs to a tenant and owns network assets, can have sub-organisations to reflect internal hierarchies. +- **Contacts**: assigned roles such as CISO or 24/7 support, tied to a single organisation. +- **Netobjects** (Network objects): Autonomous Systems, IP address ranges (CIDR), and domains owned and administered by an organisation. All claims go through an approval process. + +```mermaid +erDiagram + tenant ||--o{ organisation : "responsible for" + organisation ||--o{ organisation : "sub-organisation of" + organisation ||--o{ contact : has + organisation ||--o{ netobject : "owns and administers" + netobject { + string type "AS / IP range / domain" + int asn "e.g. 1234" + cidr network "e.g. 10.0.0.0/8" + string domain "e.g. example.com" + } + contact { + string role "CISO / 24-7 / …" + string email "e.g. abuse@example.com" + string firstname "e.g. Jane" + string lastname "e.g. Doe" + string tel "e.g. +43 1 234 5678" + } +``` + +The hierarchical model allows three privilege tiers to control access and organise permissions: +- Orgadmin (organisation admins) that manage their own organisation and all of their sub-organisations +- Tenantadmin manage all organisations in their tenant (constituency) +- Portaladmins manage all organisations in all tenants + +## Sources for abuse-contacts -## Sources for domain-based abuse-contacts +The following bots can be combined to build custom contact lookup workflows. +A common pattern is to use high-quality data sources as CERTBund and Tuency first, and fall back to generic public sources for any network resources that are not covered by the internal data sources. -These bots are suitable for domain-based abuse-contact look-ups. +Unless noted otherwise, each bot writes its result to `source.abuse_contact`. -- `intelmq.bots.experts.rdap.expert` expert queries private and public RDAP servers for `source.fqdn` and add the contact information to the event as `source.abuse_contact`. -- `intelmq.bots.experts.trusted_introducer_lookup.expert` expert queries a locally cached [Trusted Introducer team directory](https://www.trusted-introducer.org/directory/teams.json) for the TLD or domain (first match) of `source.fqdn`. +### Domain-based -## Sources for IP address-based abuse-contacts +- `intelmq.bots.experts.rdap.expert` queries private and public RDAP servers for `source.fqdn`. +- `intelmq.bots.experts.trusted_introducer_lookup.expert` looks up the TLD or domain (first match) of `source.fqdn` in a locally cached [Trusted Introducer team directory](https://www.trusted-introducer.org/directory/teams.json). -These bots are suitable for IP address and ASN based abuse-contact look-ups. +### IP address / ASN-based -- `intelmq.bots.experts.abusix.expert` expert queries the online Abusix service. -- `intelmq.bots.experts.do_portal.expert` expert queries an instance of the do-portal software (deprecated). -- `intelmq.bots.experts.tuency.expert` expert queries an instance of the **tuency** Constituency Portal for the IP address. The Portal also takes into account any notification rules, which are saved - additionally in the event. -- `intelmq.bots.experts.ripe.expert` expert queries the online RIPE database for IP-Address and AS contacts. -- `intelmq.bots.experts.trusted_introducer_lookup.expert` expert queries a locally - cached [Trusted Introducer team directory](https://www.trusted-introducer.org/directory/teams.json) - for the Autonomous system `source.asn`. +- `intelmq.bots.experts.abusix.expert` queries the online [Abusix](https://abusix.com/) service. +- `intelmq.bots.experts.do_portal.expert` queries an instance of the do-portal software (deprecated). +- `intelmq.bots.experts.tuency.expert` queries a [Tuency](#tuency) constituency portal. Also retrieves notification rules, which are stored in `extra.ttl`. +- `intelmq.bots.experts.ripe.expert` queries the online RIPE database for IP address and ASN contacts. +- `intelmq.bots.experts.trusted_introducer_lookup.expert` looks up `source.asn` in a locally cached [Trusted Introducer team directory](https://www.trusted-introducer.org/directory/teams.json). -## Generic sources for abuse-contacts +### Generic and local sources -- `intelmq.bots.experts.generic_db_lookup.expert` expert for local data sources, like - database tables mapping ASNs to abuse-contact or Country Codes to abuse-contact. -- `intelmq.bots.experts.uwhoisd.expert` expert for fetching whois-data, not extracting - abuse-contact information +- `intelmq.bots.experts.generic_db_lookup.expert` queries a local database table, e.g. a mapping from ASNs or country codes to abuse contacts. +- `intelmq.bots.experts.uwhoisd.expert` fetches raw WHOIS data, does not extract an abuse contact itself. ## Helpful other bots for pre-processing @@ -58,7 +211,9 @@ These bots are suitable for IP address and ASN based abuse-contact look-ups. ## Combining the lookup approaches -In order to get the best contact, it may be necessary to combine multiple abuse-contact sources. IntelMQ's modularity provides methods to arrange and configure the bots as needed. Among others, the following bots can help in getting the best result: +In order to get the best contact, it may be necessary to combine multiple abuse-contact sources. +IntelMQ's modularity provides methods to arrange and configure the bots as needed. +Among others, the following bots can help in getting the best result: - `intelmq.bots.experts.filter.expert` Your lookup process may be different for different types of data. E.g. website-related issues may be better addressed at the domain owner and device-related issues may be better addressed to the hosting provider. - `intelmq.bots.experts.modify.expert` Allows you to set values based on filter and also format values based on the value of other fields. From 05ccfbfbb13a4b582176fa8c94055ab5927a8457 Mon Sep 17 00:00:00 2001 From: Sebastian Wagner Date: Sun, 3 May 2026 07:41:25 +0200 Subject: [PATCH 9/9] docs: in feeds docs fix syntax for dictionaries for example the csv parser config resulted in defaults_fields: {'classification.type': 'c2-server'} which is not valid JSON because of the single quotes. therefore the config needed adaption before copy-pasting it into the manager or runtime config --- scripts/generate-feeds-docs.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/generate-feeds-docs.py b/scripts/generate-feeds-docs.py index 74f8eb279b..26b1eec598 100755 --- a/scripts/generate-feeds-docs.py +++ b/scripts/generate-feeds-docs.py @@ -18,7 +18,7 @@ @@ -83,7 +83,7 @@ def main(): # format non-empty lists with double-quotes # single quotes are not conform JSON and not correctly detected/transformed by the manager - if isinstance(value, (list, tuple)) and value: + if isinstance(value, (list, tuple, dict)) and value: value = json.dumps(value) output += f" {key}: {value}\n"