Add MySQL workload detection#2177
Open
Klarsen04 wants to merge 9 commits into
Open
Conversation
Create the core MySQL receiver translator with functional options pattern: - translator.go with endpoint, username, passfile, TLS, localhost detection - translator_test.go with unit tests for all option functions - Register mysqlreceiver.NewFactory() in defaultcomponents - Add MySQLKey and MySQL DBI component name constants to common.go - Add go.mod dependency on mysqlreceiver
Add YAML configs and translator wiring for MySQL DBI pipelines: - dbi_dbload_mysql.yaml: count connector for active sessions by wait/user/db/sql/host - dbi_topsql_mysql.yaml: signaltometrics connector for top query metrics - transform_dbi_fix_start_time_mysql.yaml: transform processor for MySQL timestamps - Update count, signaltometrics, and transform translators to handle MySQL component names - Add unit tests for MySQL connector/processor creation
Add mysql[] array schema to database_insights in schema.json: - endpoint, instance_name, username, password_file - tls.ca_file for TLS configuration - logs.file_path for server log ingestion
Klarsen04
added
the
ready for testing
Extend the DBI pipeline orchestrator to support MySQL instances: - Add translator_mysql.go with dbiMysqlTranslator implementing 4 pipeline types: - metrics/dbi_mysql_N: counter metrics + DBLoad/TopSQL from connectors - logs/dbi_mysql_N: log-to-metrics via count + signaltometrics connectors - logs/dbi_mysql_rawevents_N: raw query sample/top query events - logs/dbi_mysql_serverlogs_N: MySQL server log file ingestion - Add parseDbiMysqlInstances() to parse mysql[] config array - MySQL-specific resource attributes (db.system.name = "mysql") - MySQL-specific log group paths and monitor user exclusion filter - Unit tests for pipeline wiring with single/multiple instances
Add end-to-end config translation test for MySQL DBI: - dbi_mysql_config_linux.conf: input agent config fixture - dbi_mysql_config_linux.json: parsed JSON intermediate fixture - dbi_mysql_config_linux.yaml: golden YAML output (full OTel pipeline) - .mysql_credentials: test passfile for credential loading - Register TestMySQLDbiConfig in tocwconfig_unix_test.go
Klarsen04
force-pushed
the
larkirs/mysql-workload-detection
branch
2 times, most recently
from
615b5d5 to
7352bdb
Compare
- Unify the DBI pipeline translator: merge translator_mysql.go into a single dbiTranslator that branches by engine (carried on the instance config), instead of maintaining a separate MySQL translator. - Make component naming consistent across engines using an engine suffix: count/dbi_dbload_<engine>, signaltometrics/dbi_topsql_<engine>, transform/dbi_fix_start_time_<engine>, and per-instance dbi_resource_<engine>_N, dbi_scope_<engine>_N, dbi_logs_<engine>_<dest>_N, dbi_exclude_monitor_<engine>_N. PostgreSQL components now carry the _postgresql suffix for consistency. - Rename engine-specific connector/processor YAMLs for symmetry (dbi_dbload_postgresql.yaml, dbi_topsql_postgresql.yaml, transform_dbi_fix_start_time_postgresql.yaml); the connector and transform translators select the config by engine. - Add the dbi_scope transform processor (cloudwatch.source/solution) to all MySQL pipelines, matching the PostgreSQL implementation. - Use the shared named resourcedetection/opentelemetry processor. - Add the validateOttlSafe check to the unified Translate() so username and instance_name are validated for both engines. - Fix the host metrics process scraper to match the mysqld process (mysqld.*) when MySQL DBI is configured. - Remove TLS/ca_file from the MySQL schema and receiver; MySQL DBI is localhost-only and always connects with TLS disabled. - Fix the MySQL passfile test fixture to the pgpass-style format (host:port:database:username:password) and assert all seven DB Load metrics. - Regenerate golden YAMLs (mysql, postgres, combined configs).
Klarsen04
force-pushed
the
larkirs/mysql-workload-detection
branch
from
7352bdb to
3a7fea5
Compare
…upport The CloudWatch Console (Cinch) creates separate SSM parameters for each workload configuration (e.g. one for database-insights-postgres and one for database-insights-mysql). When the agent loads multiple config files from the .d/ directory, the config merger needs to deep-merge the database_insights section so that both postgresql and mysql arrays coexist. Without this rule, the agent treats database_insights from two separate config sources as conflicting values and drops one, causing missing log groups in CloudWatch. Register database_insights as a SectionMergeRule under the collect rule, matching the existing pattern for opentelemetry and collect.
Klarsen04
force-pushed
the
larkirs/mysql-workload-detection
branch
from
3a7fea5 to
97bfc15
Compare
- Move the complete 14-metric Top SQL connector config (dbi_topsql_mysql.yaml) into this PR. The full metric set was previously committed in the MySQL workload-detection PR by mistake, which left this PR's embedded config with only 2 metrics while the golden YAML had 14 (mismatch that failed the translation unit tests). All 14 metrics match what the mysqlreceiver publishes on db.server.top_query events. - Remove the user.name attribute from all Top SQL metrics: top queries come from events_statements_summary_by_digest, aggregated by digest and schema, so the receiver does not publish user.name on top_query events (validated against the forked mysqlreceiver RecordDbServerTopQueryEvent). - Remove the duplicate dbiMysqlKey variable in favor of the shared common.DatabaseInsightsMysqlKey. - Regenerate the MySQL golden YAML.
Klarsen04
force-pushed
the
larkirs/mysql-workload-detection
branch
from
97bfc15 to
6814222
Compare
Implement detection of mysqld processes for the workload discovery framework: - Add mysql.NewDetector that matches mysqld process by executable name - Add port extraction from --port/-P flags, MYSQL_TCP_PORT env var, or default 3306 - Add CategoryMySQL constant to metadata categories - Register MySQL detector in workload-discovery main - Unit tests for detection logic and port extraction (all flag/env variants)
Klarsen04
force-pushed
the
larkirs/mysql-workload-detection
branch
from
6814222 to
c266a98
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the issue
Adds workload detection support for MySQL database servers as part of the workload-discovery feature.
Description of changes
Implements a MySQL process detector that:
New files:
Unit tests for both components
License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
Added unit tests for MySQL detector (mysql_test.go)
Added unit tests for port extractor (port_test.go)
make fmt and make lint pass
Test on AL2:
Requirements
Run make fmt and make fmt-sh
Run make lint