Add MySQL DBI config translator and pipeline orchestrator by Klarsen04 · Pull Request #2176 · aws/amazon-cloudwatch-agent

Klarsen04 · 2026-06-24T16:30:31Z

Description of the issue

Add component translators and pipeline orchestrator for MySQL database monitoring via the OpenTelemetry pipeline, enabling the opentelemetry.collect.database_insights.mysql[] config section.

Description of changes

Component translators:
MySQL receiver translator with functional options (endpoint, username, passfile, TLS, localhost detection)
Connectors: count/dbi_dbload_mysql (active session metrics) and signaltometrics/dbi_topsql_mysql (top query metrics from events_statements_summary_by_digest)
Processor: transform/dbi_fix_start_time_mysql (timestamp correction)
Registers mysqlreceiver and adds JSON schema validation for mysql config

Pipeline orchestrator:
Generates per-instance metrics, log-to-metrics, raw events, and server logs pipelines from JSON config
MySQL-specific resource attributes, log group paths, and monitor user exclusion
Golden test config files for end-to-end validation

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Unit tests for receiver translator (all option functions, TLS enforcement)
Unit tests for connector/processor MySQL component creation
Unit tests covering pipeline ID generation, component wiring, multi-instance creation, and conditional server logs pipeline
Golden integration test validating full JSON → YAML translation
go build ./... passes

Create the core MySQL receiver translator with functional options pattern: - translator.go with endpoint, username, passfile, TLS, localhost detection - translator_test.go with unit tests for all option functions - Register mysqlreceiver.NewFactory() in defaultcomponents - Add MySQLKey and MySQL DBI component name constants to common.go - Add go.mod dependency on mysqlreceiver

Add YAML configs and translator wiring for MySQL DBI pipelines: - dbi_dbload_mysql.yaml: count connector for active sessions by wait/user/db/sql/host - dbi_topsql_mysql.yaml: signaltometrics connector for top query metrics - transform_dbi_fix_start_time_mysql.yaml: transform processor for MySQL timestamps - Update count, signaltometrics, and transform translators to handle MySQL component names - Add unit tests for MySQL connector/processor creation

Add mysql[] array schema to database_insights in schema.json: - endpoint, instance_name, username, password_file - tls.ca_file for TLS configuration - logs.file_path for server log ingestion

Extend the DBI pipeline orchestrator to support MySQL instances: - Add translator_mysql.go with dbiMysqlTranslator implementing 4 pipeline types: - metrics/dbi_mysql_N: counter metrics + DBLoad/TopSQL from connectors - logs/dbi_mysql_N: log-to-metrics via count + signaltometrics connectors - logs/dbi_mysql_rawevents_N: raw query sample/top query events - logs/dbi_mysql_serverlogs_N: MySQL server log file ingestion - Add parseDbiMysqlInstances() to parse mysql[] config array - MySQL-specific resource attributes (db.system.name = "mysql") - MySQL-specific log group paths and monitor user exclusion filter - Unit tests for pipeline wiring with single/multiple instances

Add end-to-end config translation test for MySQL DBI: - dbi_mysql_config_linux.conf: input agent config fixture - dbi_mysql_config_linux.json: parsed JSON intermediate fixture - dbi_mysql_config_linux.yaml: golden YAML output (full OTel pipeline) - .mysql_credentials: test passfile for credential loading - Register TestMySQLDbiConfig in tocwconfig_unix_test.go

- Unify the DBI pipeline translator: merge translator_mysql.go into a single dbiTranslator that branches by engine (carried on the instance config), instead of maintaining a separate MySQL translator. - Make component naming consistent across engines using an engine suffix: count/dbi_dbload_<engine>, signaltometrics/dbi_topsql_<engine>, transform/dbi_fix_start_time_<engine>, and per-instance dbi_resource_<engine>_N, dbi_scope_<engine>_N, dbi_logs_<engine>_<dest>_N, dbi_exclude_monitor_<engine>_N. PostgreSQL components now carry the _postgresql suffix for consistency. - Rename engine-specific connector/processor YAMLs for symmetry (dbi_dbload_postgresql.yaml, dbi_topsql_postgresql.yaml, transform_dbi_fix_start_time_postgresql.yaml); the connector and transform translators select the config by engine. - Add the dbi_scope transform processor (cloudwatch.source/solution) to all MySQL pipelines, matching the PostgreSQL implementation. - Use the shared named resourcedetection/opentelemetry processor. - Add the validateOttlSafe check to the unified Translate() so username and instance_name are validated for both engines. - Fix the host metrics process scraper to match the mysqld process (mysqld.*) when MySQL DBI is configured. - Remove TLS/ca_file from the MySQL schema and receiver; MySQL DBI is localhost-only and always connects with TLS disabled. - Fix the MySQL passfile test fixture to the pgpass-style format (host:port:database:username:password) and assert all seven DB Load metrics. - Regenerate golden YAMLs (mysql, postgres, combined configs).

…upport The CloudWatch Console (Cinch) creates separate SSM parameters for each workload configuration (e.g. one for database-insights-postgres and one for database-insights-mysql). When the agent loads multiple config files from the .d/ directory, the config merger needs to deep-merge the database_insights section so that both postgresql and mysql arrays coexist. Without this rule, the agent treats database_insights from two separate config sources as conflicting values and drops one, causing missing log groups in CloudWatch. Register database_insights as a SectionMergeRule under the collect rule, matching the existing pattern for opentelemetry and collect.

- Move the complete 14-metric Top SQL connector config (dbi_topsql_mysql.yaml) into this PR. The full metric set was previously committed in the MySQL workload-detection PR by mistake, which left this PR's embedded config with only 2 metrics while the golden YAML had 14 (mismatch that failed the translation unit tests). All 14 metrics match what the mysqlreceiver publishes on db.server.top_query events. - Remove the user.name attribute from all Top SQL metrics: top queries come from events_statements_summary_by_digest, aggregated by digest and schema, so the receiver does not publish user.name on top_query events (validated against the forked mysqlreceiver RecordDbServerTopQueryEvent). - Remove the duplicate dbiMysqlKey variable in favor of the shared common.DatabaseInsightsMysqlKey. - Regenerate the MySQL golden YAML.

Update the test credential file to use the MySQL-native INI format (Option C) with required host, port, user, and password fields.

Points to amazon-contributing/opentelemetry-collector-contrib PR #577 which replaces pgpass format with .my.cnf INI format for MySQL DBI credential files.

Klarsen04 added 3 commits June 24, 2026 15:58

Add MySQL config schema definition

137120b

Add mysql[] array schema to database_insights in schema.json: - endpoint, instance_name, username, password_file - tls.ca_file for TLS configuration - logs.file_path for server log ingestion

Klarsen04 requested a review from a team as a code owner June 24, 2026 16:30

Klarsen04 force-pushed the larkirs/mysql-pipeline-translator branch 2 times, most recently from 0440510 to a0d627b Compare June 24, 2026 17:41

Klarsen04 added the ready for testing Indicates this PR is ready for integration tests to run label Jun 24, 2026