From a59d25fb9eb0f0b9b7ef8183ad813c214c2a755c Mon Sep 17 00:00:00 2001 From: Gabriel Ciciliani Date: Sat, 16 May 2026 10:27:14 -0300 Subject: [PATCH 1/4] Documentation to enable Automatic schema changes --- docs/releem-agent/automatic-schema-changes.md | 108 ++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 docs/releem-agent/automatic-schema-changes.md diff --git a/docs/releem-agent/automatic-schema-changes.md b/docs/releem-agent/automatic-schema-changes.md new file mode 100644 index 0000000..a40a330 --- /dev/null +++ b/docs/releem-agent/automatic-schema-changes.md @@ -0,0 +1,108 @@ +--- +id: automatic-schema-changes +title: Automatic Schema Changes +--- + +# Automatic schema changes in the Releem Agent + +If the **Releem Agent** is already installed and running, you can allow it to execute approved schema changes on the server. Automatic schema changes also include the option of running a pre-change backup, in case a rollback is required. + +Both automatic schema changes and backups were implemented with availability in mind, so they will only run if: +* There is enough disk space to perform both, the backup and the schema change +* The backup won't block the affected tables +* Point-in-time restore is possible on the server +* The schema change won't block the affected tables + +The following steps explain how to configure the agent and the database user to handle this new functionality. + +--- + +## 1. Locate the configuration file + +To enable automatic schema changes, we need to include a few new parameters in the agent configuration file. Below is the default location for Linux servers. Open the file with your favorite editor to add the new parameters. + +| Platform | Default path | +|----------|----------------| +| Linux | `/opt/releem/releem.conf` | + +--- +## 2. Enable automatic schema (DDL) execution + +By default the agent **does not** run schema changes from Releem, even when you approve them in the product. For schema changes to be executed on your database server, activate this feature explicitly by setting `enable_exec_ddl` to `true`. + +Before running the schema change against the real table, the agent will perform a dry-run of the change against an empty table with the same structure. This is to guarantee that the operation can run successfully with the intenteded strategy. + +There are some schema changes that the database server can't execute on its own, without blocking the table. An alternative it to use an external tool called [pt-online-schema-change](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html). This tool creates a copy of the table with the intended changes, copies all data to this new table, and swaps it with the existing one, with minimum impact. + +[pt-online-schema-change](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html) needs to be available on the server and the location of the tool can be specified in the configuration. + +| Setting | Values | What it does | +|---------|--------|----------------| +| `enable_exec_ddl` | `false` (default) or `true` | When `true`, the agent may execute **schema changes** that Releem sends after analysis. When `false`, those changes are not run; the agent reports that execution is disabled. | +| `ptosc_path` | `pt-online-schema-change` | Percona Toolkit is not on `PATH` or you use a non-standard binary location. | +| `online_ddl_test_schema` | `releem_online_ddl_test` (default) or any valid database/schema name | **Optional:** Database/schema name where the agent will test the schema change before executing it against the real table| + + +--- +## 3. Configure your backup settings + +When a pre-change backup is requested, the agent needs tools and extra disk space available on the **same host that runs the agent**. As mentioned before, the Releem agent will look for the best alternative to backup the affected tables before the schema change is executed. + +* If the server and the table supports it, the agent will create a physical backup of the table using `xtrabackup` or `mariabackup` +* If online physical backup is not an option, the agent will use mysqldump to create a logical backup of the data (a `.sql` file with necessary statements to re-create the table and the data) + +Releem only proceeds with the backup when **point-in-time recovery** is available for the instance as Releem detects it. If not, the change that required the backup will not run. + + +| Setting | Values | What it does | +|---------|--------|----------------| +| `backup_dir` | `/tmp/backups` (default) | Directory for backup output. Must exist or be creatable and have enough free space. | +| `mysqldump_path` | `mysqldump` (default) | Full path or name on `PATH` for `mysqldump` (logical backup). | +| `xtrabackup_path` | `xtrabackup` (default) | Full path or name on `PATH` for `xtrabackup` (physical backup when Releem selects that method). | +| `backup_space_buffer` | `20.0` (default) | Extra free space (as a percentage) the agent requires above its estimated backup size before starting a backup. | + + +--- +## 4. Extend database user permissions + +The same **MySQL user** the agent already uses for monitoring must have permission to run the approved ALTER statements. Connect to the target database server and run the he GRANT statements below: + +```sql +-- To allow table ALTERs and New indexes on **any** database +GRANT CREATE, REFERENCES, INDEX, ALTER ON *.* TO `releem`@`127.0.0.1` +``` + +```sql +-- Alternative: grant ALTER permissions *only* on a specific database +GRANT CREATE, REFERENCES, INDEX, ALTER ON `airportdb`.* TO `releem`@`127.0.0.1` +``` + +```sql +-- Needed for schema changes dry-runs (note this only affects the test database) +GRANT CREATE, DROP, INDEX, ALTER ON `releem_online_ddl_test`.* TO `releem`@`127.0.0.1` +``` + +#### Optional - To use pt-online-schema-change as an alternative method when the operation can't be executed online by the server +```sql +GRANT SELECT, INSERT, DROP, RELOAD, SUPER, SHOW VIEW, TRIGGER ON *.* TO `releem`@`127.0.0.1` +``` + +--- + + + +## 5. Restart the agent + + +After editing, **restart the Releem Agent** so changes take effect. + +--- + +## External tools + +Install **mysqldump**, **XtraBackup**, **mariabackup** and **pt-online-schema-change**as appropriate for your Database server and OS flavor. For more information about how to install these tools, please refer to: + +* [pt-online-schema-change](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html) +* [xtrabackup](https://docs.percona.com/percona-xtrabackup/2.4/index.html) +* [mariabackup](https://mariadb.com/docs/server/server-usage/backup-and-restore/mariadb-backup/mariadb-backup-overview#installing-mariadb-backup) +* [mysqldump](https://dev.mysql.com/doc/refman/9.7/en/mysqldump.html) From d40a25d36ea737eade3b8384a71edda328c9dc05 Mon Sep 17 00:00:00 2001 From: Gabriel Ciciliani Date: Sat, 16 May 2026 11:26:31 -0300 Subject: [PATCH 2/4] Automatic schema change troubleshooting guide --- .../schema-change-troubleshooting.md | 141 ++++++++++++++++++ 1 file changed, 141 insertions(+) create mode 100644 docs/releem-agent/schema-change-troubleshooting.md diff --git a/docs/releem-agent/schema-change-troubleshooting.md b/docs/releem-agent/schema-change-troubleshooting.md new file mode 100644 index 0000000..2945afa --- /dev/null +++ b/docs/releem-agent/schema-change-troubleshooting.md @@ -0,0 +1,141 @@ +# Schema change troubleshooting guide + +This guide covers failure scenarios for **automatic schema changes** executed by the Releem Agent. When a change fails, check the task output in the Releem portal and match the **exit code** and message to the table below. + +Exit codes are reported as `task_exit_code` on the task status sent back to Releem. A task with **status 4** failed; **status 1** with exit code **0** succeeded. + +For configuration prerequisites, see [user-guide-task-automation.md](./user-guide-task-automation.md). + +--- + +## Exit codes set before execution starts + + +| Scenario | Exit code | Troubleshooting steps | +| --------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Schema change execution disabled | **10** | Set `enable_exec_ddl = true` in `/opt/releem/releem.conf` (or your config path), restart the agent, and retry the change from Releem. | +| Invalid or malformed task payload | **2** | This is not fixable on the server alone—the task JSON from Releem is invalid or missing required fields (`schema_name`, `ddl_statement`, `analysis_results.schema_name`, `analysis_results.table_name`). Contact Releem support with the task id; retry after the platform resends a valid payload. | +| Empty schema change list | **3** | The task contained no statements to run. Retry from Releem or contact support if the change should have been scheduled. | + + +--- + +## Exit codes set during validation (per statement) + +These stop the task before any DDL or backup runs on the server. + + +| Scenario | Exit code | Troubleshooting steps | +| ----------------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| DDL failed syntax validation | **4** | Fix the SQL in Releem (or cancel and recreate the change). The task output includes `syntax validation failed` and any `syntax_error` detail from analysis. Do not retry the same statement until the DDL is corrected. | +| No safe execution method | **5** | Releem analysis marked the change as neither online DDL nor `pt-online-schema-change` safe (`ok_online_ddl` and `ok_pt_osc` both false). Revise the change (smaller scope, different operation), use a maintenance window with manual DDL, or ask Releem why the statement was classified as blocking-only. | +| Pre-change backup required but PITR unavailable | **6** | The change requested a backup before DDL, but point-in-time recovery is not available on this instance (binlog/archiving, managed-service PITR, etc.). Enable PITR on the server or disable the pre-change backup requirement for this change in Releem if policy allows. | + + +--- + +## Exit code 7 — execution or backup failed + +All rows below use exit code **7**. The task output includes `Statement N failed:` followed by the underlying error. Enable `debug = true` in `releem.conf` and restart the agent for detailed command logs (passwords are masked). + +### Disk space and filesystem capacity + + +| Scenario | Exit code | Troubleshooting steps | +| -------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Insufficient space on MySQL datadir | **7** | Message contains `insufficient datadir free space` (must stay **above 10%** free) or `insufficient datadir capacity` (projected use after change must stay **at or below 90%**). Free space on the datadir filesystem, archive or drop unused data, or shrink large tables before retrying. Only for emergencies: set `disable_space_checks = true` in `releem.conf` if your team accepts skipping these checks. | +| Insufficient space in backup directory | **7** | Message contains `insufficient disk space: required` under `backup_dir`. Free space on the volume that holds `backup_dir` (default `/tmp/backups`), point `backup_dir` to a larger filesystem, or lower `backup_space_buffer` only if you accept less safety margin. | +| Cannot read datadir or table size | **7** | Messages such as `failed to resolve datadir`, `datadir is empty`, `failed to get table size`, or `failed to check datadir filesystem capacity`. Verify the agent MySQL user can run `SHOW VARIABLES LIKE 'datadir'` and query `information_schema.TABLES` for the target schema and table. | +| Cannot check backup directory | **7** | `failed to check disk space` or `failed to create backup directory`. Ensure `backup_dir` exists, is writable by the agent process, and is on a filesystem the host can stat. | + + +### Pre-change backup + + +| Scenario | Exit code | Troubleshooting steps | +| ----------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| mysqldump backup failed | **7** | Message contains `backup failed` and `mysqldump failed`. Install `mysqldump`, set `mysqldump_path` if needed, confirm `mysql_host` / `mysql_user` / `mysql_password` in `releem.conf`, and ensure the user can dump the target table. Run the same mysqldump manually as the agent user to reproduce. | +| XtraBackup backup or prepare failed | **7** | Message contains `xtrabackup backup failed` or `xtrabackup prepare failed`. Install a compatible **xtrabackup** (or **mariabackup** if your deployment maps it via `xtrabackup_path`), fix `xtrabackup_path`, and verify backup user privileges. Review tool output in agent logs with `debug = true`. | +| Backup configuration missing | **7** | `mysql_host is required for backup` or `config is required for backup`. Set MySQL connection settings in `releem.conf` the same way as for normal agent monitoring. | +| Backup size estimate failed | **7** | `failed to estimate backup size`. Check that the target table exists and the agent user can read `information_schema.TABLES`. | + + +### Online DDL (including dry-run on test table) + + +| Scenario | Exit code | Troubleshooting steps | +| --------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Online DDL preflight (dry-run) failed | **7** | Message contains `online DDL preflight failed on test table`. The agent clones the table into `online_ddl_test_schema` (default `releem_online_ddl_test`) and runs the DDL there first. Grant `CREATE`, `DROP`, `INDEX`, `ALTER` on that schema; confirm the DDL is valid for an empty copy (same engine/structure). Fix incompatible DDL or use a change Releem routes to `pt-online-schema-change`. | +| Online DDL failed on production table | **7** | Message contains `schema change execution failed` after preflight succeeded. Often metadata locks, unsupported `ALGORITHM`/`LOCK`, or replication restrictions. Check MySQL error in agent logs; retry in a low-traffic window; resolve blocking sessions. The agent does **not** fall back to pt-osc after online DDL fails. | +| Test schema cannot be created | **7** | `test schema is required`, `failed to create test schema`, or `failed to create test table`. Set `online_ddl_test_schema` if the default name conflicts; grant DDL on that schema; ensure disk space for the empty clone. | +| DDL shape not supported for online path | **7** | `unsupported DDL for online clauses` or `could not locate target table in DDL statement`. Use `ALTER TABLE` or supported `CREATE INDEX` forms; ensure the statement references the analyzed `schema.table`. | +| Lock wait timeout | **7** | Online DDL sets `lock_wait_timeout = 20`. If errors mention lock wait or metadata locks, clear blocking transactions and retry, or use a maintenance window / pt-osc path if Releem allows it. | + + +### pt-online-schema-change + + +| Scenario | Exit code | Troubleshooting steps | +| ---------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| pt-osc dry-run failed | **7** | Message contains `pt-online-schema-change dry-run failed`. Install [Percona Toolkit](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html), set `ptosc_path`, grant privileges in the user guide (`SELECT`, `INSERT`, `DROP`, `RELOAD`, `SUPER`, `SHOW VIEW`, `TRIGGER` on `*.`* when required). Run `pt-online-schema-change --dry-run` manually with the same connection settings. | +| pt-osc execute failed | **7** | Dry-run passed but `pt-online-schema-change failed` on execute. Check pt-osc output in logs (triggers, replicas, disk, permissions). Resolve replica lag or tool errors before retrying. | +| pt-osc configuration missing | **7** | `mysql_host is required for pt-online-schema-change` or `config is required for pt-online-schema-change`. Complete MySQL settings in `releem.conf`. | + + +### Other execution errors (exit code 7) + + +| Scenario | Exit code | Troubleshooting steps | +| -------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Missing table name | **7** | `table name is required for schema change execution`. Internal/task payload issue—contact Releem support with the task id. | +| Failed to parse table name | **7** | `failed to parse table name`. Ensure `analysis_results.schema_name` and `analysis_results.table_name` match the real object and use a valid `schema.table` form. | + + +--- + +## Exit code 8 — no statements executed + + +| Scenario | Exit code | Troubleshooting steps | +| -------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| No schema changes executed | **8** | Task output includes `No schema changes were executed.` This is returned when the loop finishes without applying any statement (unusual if earlier validation passed). Review full task output and agent logs; retry from Releem or contact support with the task id. | + + +--- + +## Success and non-failure notes + + +| Scenario | Exit code | Troubleshooting steps | +| ------------------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Success | **0** | Task status **1**. Output lists `Statement N successful:` for each applied DDL. No action required. | +| Non-InnoDB storage engine | *(none — warning only)* | Output may include `warning: storage engine is ...` without failing the task. Prefer InnoDB for online DDL and backups; plan manual change if you rely on MyISAM or other engines. | + + +--- + +## Quick reference: exit code summary + + +| Exit code | Meaning | +| --------- | ----------------------------------------------------- | +| **0** | Success | +| **2** | Invalid task payload | +| **3** | Empty change list | +| **4** | Syntax validation failed | +| **5** | No online DDL or pt-osc path | +| **6** | Pre-change backup blocked (no PITR) | +| **7** | Backup or DDL execution failed (see sub-tables above) | +| **8** | No statements executed | +| **10** | `enable_exec_ddl` is false | + + +--- + +## Where to look next + +1. Task output in the Releem portal for the exact `Statement N failed:` line. +2. Agent logs with `debug = true` (commands, preflight SQL, tool stderr). +3. MySQL server error log for the time of the failure. +4. [task-type-6-schema-changes.md](./task-type-6-schema-changes.md) for technical behavior and safety limits. + From 33349dcfa1ef5a562d97a793950e552c00962d4f Mon Sep 17 00:00:00 2001 From: Dmitry Kochetov Date: Sat, 23 May 2026 20:04:09 +0400 Subject: [PATCH 3/4] fixed --- docs/getting-started/schema-optimization.md | 2 + .../schema-change-troubleshooting.md | 43 ++++++++++++------- sidebars.js | 1 + 3 files changed, 30 insertions(+), 16 deletions(-) diff --git a/docs/getting-started/schema-optimization.md b/docs/getting-started/schema-optimization.md index 91ef689..29bc6f8 100644 --- a/docs/getting-started/schema-optimization.md +++ b/docs/getting-started/schema-optimization.md @@ -26,6 +26,8 @@ Schema optimization helps you detect and fix: 4. Test changes in a development environment first 5. Execute the SQL on your production database during low-traffic periods +If an automatic schema change fails in Releem, use the [Schema Change Troubleshooting](/releem-agent/schema-change-troubleshooting) guide to match the error to the next action. + For detailed information about each type of schema check and comprehensive best practices, see the [MySQL Database Schema Checks](https://releem.com/blog/mysql-database-schema-checks) article. Schema optimization is essential for maintaining long-term database health and performance as your application grows. diff --git a/docs/releem-agent/schema-change-troubleshooting.md b/docs/releem-agent/schema-change-troubleshooting.md index 2945afa..341e813 100644 --- a/docs/releem-agent/schema-change-troubleshooting.md +++ b/docs/releem-agent/schema-change-troubleshooting.md @@ -1,16 +1,31 @@ -# Schema change troubleshooting guide +--- +id: schema-change-troubleshooting +title: Schema Change Troubleshooting +--- + +# Schema Change Troubleshooting + +This guide helps you troubleshoot failed **automatic schema changes** executed by the Releem Agent. Use it when Releem cannot apply an index or table change automatically and the Releem Dashboard shows a failed task. -This guide covers failure scenarios for **automatic schema changes** executed by the Releem Agent. When a change fails, check the task output in the Releem portal and match the **exit code** and message to the table below. +When a change fails, open the failed task in the Releem Dashboard and check: -Exit codes are reported as `task_exit_code` on the task status sent back to Releem. A task with **status 4** failed; **status 1** with exit code **0** succeeded. +- **Apply Index Error** - the detailed message, usually including `Statement N failed: ...`. +- **Agent logs** - useful when the dashboard message is not enough. See [How to Check Releem Agent Logs](/releem-agent/how-to-check-logs). -For configuration prerequisites, see [user-guide-task-automation.md](./user-guide-task-automation.md). +## Before you retry + +1. Read the exact output in the Releem Dashboard. +2. Match the message to the table below. +3. Fix the server-side issue first. Retrying without changing anything usually fails again. +4. If the error says the payload is invalid or empty, contact Releem support with the task id. + +Automatic schema changes are intended for environments where the Releem Agent is allowed to make DDL changes. The Agent must have enough MySQL privileges, access to the configured backup tools, and `enable_exec_ddl = true` in `/opt/releem/releem.conf` when automatic DDL execution is enabled. +For configuration prerequisites, see [Automatic Schema Changes](releem-agent/automatic-schema-changes). --- ## Exit codes set before execution starts - | Scenario | Exit code | Troubleshooting steps | | --------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Schema change execution disabled | **10** | Set `enable_exec_ddl = true` in `/opt/releem/releem.conf` (or your config path), restart the agent, and retry the change from Releem. | @@ -24,7 +39,6 @@ For configuration prerequisites, see [user-guide-task-automation.md](./user-guid These stop the task before any DDL or backup runs on the server. - | Scenario | Exit code | Troubleshooting steps | | ----------------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | DDL failed syntax validation | **4** | Fix the SQL in Releem (or cancel and recreate the change). The task output includes `syntax validation failed` and any `syntax_error` detail from analysis. Do not retry the same statement until the DDL is corrected. | @@ -38,12 +52,13 @@ These stop the task before any DDL or backup runs on the server. All rows below use exit code **7**. The task output includes `Statement N failed:` followed by the underlying error. Enable `debug = true` in `releem.conf` and restart the agent for detailed command logs (passwords are masked). -### Disk space and filesystem capacity +Because exit code **7** has several possible causes, use the message text to choose the correct row below. For example, `online DDL preflight failed on test table` and `pt-online-schema-change dry-run failed` are different problems even though both return exit code **7**. +### Disk space and filesystem capacity | Scenario | Exit code | Troubleshooting steps | | -------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Insufficient space on MySQL datadir | **7** | Message contains `insufficient datadir free space` (must stay **above 10%** free) or `insufficient datadir capacity` (projected use after change must stay **at or below 90%**). Free space on the datadir filesystem, archive or drop unused data, or shrink large tables before retrying. Only for emergencies: set `disable_space_checks = true` in `releem.conf` if your team accepts skipping these checks. | +| Insufficient space on MySQL datadir | **7** | Message contains `insufficient datadir free space` (must stay **above 10%** free) or `insufficient datadir capacity` (projected use after change must stay **at or below 90%**). Free space on the datadir filesystem, archive or drop unused data, or shrink large tables before retrying. Only for emergencies, set `disable_space_checks = true` in `releem.conf` if your team accepts skipping these checks. | | Insufficient space in backup directory | **7** | Message contains `insufficient disk space: required` under `backup_dir`. Free space on the volume that holds `backup_dir` (default `/tmp/backups`), point `backup_dir` to a larger filesystem, or lower `backup_space_buffer` only if you accept less safety margin. | | Cannot read datadir or table size | **7** | Messages such as `failed to resolve datadir`, `datadir is empty`, `failed to get table size`, or `failed to check datadir filesystem capacity`. Verify the agent MySQL user can run `SHOW VARIABLES LIKE 'datadir'` and query `information_schema.TABLES` for the target schema and table. | | Cannot check backup directory | **7** | `failed to check disk space` or `failed to create backup directory`. Ensure `backup_dir` exists, is writable by the agent process, and is on a filesystem the host can stat. | @@ -51,7 +66,6 @@ All rows below use exit code **7**. The task output includes `Statement N failed ### Pre-change backup - | Scenario | Exit code | Troubleshooting steps | | ----------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | mysqldump backup failed | **7** | Message contains `backup failed` and `mysqldump failed`. Install `mysqldump`, set `mysqldump_path` if needed, confirm `mysql_host` / `mysql_user` / `mysql_password` in `releem.conf`, and ensure the user can dump the target table. Run the same mysqldump manually as the agent user to reproduce. | @@ -62,7 +76,6 @@ All rows below use exit code **7**. The task output includes `Statement N failed ### Online DDL (including dry-run on test table) - | Scenario | Exit code | Troubleshooting steps | | --------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Online DDL preflight (dry-run) failed | **7** | Message contains `online DDL preflight failed on test table`. The agent clones the table into `online_ddl_test_schema` (default `releem_online_ddl_test`) and runs the DDL there first. Grant `CREATE`, `DROP`, `INDEX`, `ALTER` on that schema; confirm the DDL is valid for an empty copy (same engine/structure). Fix incompatible DDL or use a change Releem routes to `pt-online-schema-change`. | @@ -74,10 +87,9 @@ All rows below use exit code **7**. The task output includes `Statement N failed ### pt-online-schema-change - | Scenario | Exit code | Troubleshooting steps | | ---------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| pt-osc dry-run failed | **7** | Message contains `pt-online-schema-change dry-run failed`. Install [Percona Toolkit](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html), set `ptosc_path`, grant privileges in the user guide (`SELECT`, `INSERT`, `DROP`, `RELOAD`, `SUPER`, `SHOW VIEW`, `TRIGGER` on `*.`* when required). Run `pt-online-schema-change --dry-run` manually with the same connection settings. | +| pt-osc dry-run failed | **7** | Message contains `pt-online-schema-change dry-run failed`. Install [Percona Toolkit](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html), set `ptosc_path`, and grant the required permissions for the target table. Depending on your MySQL version and topology, pt-osc may require privileges such as `SELECT`, `INSERT`, `DROP`, `RELOAD`, `SUPER`, `SHOW VIEW`, `TRIGGER`. Run `pt-online-schema-change --dry-run` manually with the same connection settings. | | pt-osc execute failed | **7** | Dry-run passed but `pt-online-schema-change failed` on execute. Check pt-osc output in logs (triggers, replicas, disk, permissions). Resolve replica lag or tool errors before retrying. | | pt-osc configuration missing | **7** | `mysql_host is required for pt-online-schema-change` or `config is required for pt-online-schema-change`. Complete MySQL settings in `releem.conf`. | @@ -134,8 +146,7 @@ All rows below use exit code **7**. The task output includes `Statement N failed ## Where to look next -1. Task output in the Releem portal for the exact `Statement N failed:` line. -2. Agent logs with `debug = true` (commands, preflight SQL, tool stderr). +1. The output in the Releem portal for the exact `Statement N failed:` line. +2. Agent logs with `debug = true` in `/opt/releem/releem.conf` (commands, preflight SQL, tool stderr). 3. MySQL server error log for the time of the failure. -4. [task-type-6-schema-changes.md](./task-type-6-schema-changes.md) for technical behavior and safety limits. - +4. Releem support, if the payload is invalid, the message does not match any row above, or the same task keeps failing after the server-side issue is fixed. diff --git a/sidebars.js b/sidebars.js index 62d0c72..33b15e8 100644 --- a/sidebars.js +++ b/sidebars.js @@ -93,6 +93,7 @@ const sidebars = { 'query-optimization/enable-sql-query-optimization', 'query-optimization/disable-sql-query-optimization', 'query-optimization/prepared-statements-issue', + 'query-optimization/schema-change-troubleshooting', ], }, { From e603a79b31a52b522e6226f5cddd57966c28440b Mon Sep 17 00:00:00 2001 From: Dmitry Kochetov Date: Sat, 23 May 2026 20:07:23 +0400 Subject: [PATCH 4/4] fixed --- .../automatic-schema-changes.md | 0 .../schema-change-troubleshooting.md | 2 +- sidebars.js | 3 ++- 3 files changed, 3 insertions(+), 2 deletions(-) rename docs/{releem-agent => query-optimization}/automatic-schema-changes.md (100%) rename docs/{releem-agent => query-optimization}/schema-change-troubleshooting.md (99%) diff --git a/docs/releem-agent/automatic-schema-changes.md b/docs/query-optimization/automatic-schema-changes.md similarity index 100% rename from docs/releem-agent/automatic-schema-changes.md rename to docs/query-optimization/automatic-schema-changes.md diff --git a/docs/releem-agent/schema-change-troubleshooting.md b/docs/query-optimization/schema-change-troubleshooting.md similarity index 99% rename from docs/releem-agent/schema-change-troubleshooting.md rename to docs/query-optimization/schema-change-troubleshooting.md index 341e813..906211b 100644 --- a/docs/releem-agent/schema-change-troubleshooting.md +++ b/docs/query-optimization/schema-change-troubleshooting.md @@ -20,7 +20,7 @@ When a change fails, open the failed task in the Releem Dashboard and check: 4. If the error says the payload is invalid or empty, contact Releem support with the task id. Automatic schema changes are intended for environments where the Releem Agent is allowed to make DDL changes. The Agent must have enough MySQL privileges, access to the configured backup tools, and `enable_exec_ddl = true` in `/opt/releem/releem.conf` when automatic DDL execution is enabled. -For configuration prerequisites, see [Automatic Schema Changes](releem-agent/automatic-schema-changes). +For configuration prerequisites, see [Automatic Schema Changes](query-optimization/automatic-schema-changes). --- diff --git a/sidebars.js b/sidebars.js index 33b15e8..d77b0c5 100644 --- a/sidebars.js +++ b/sidebars.js @@ -93,7 +93,8 @@ const sidebars = { 'query-optimization/enable-sql-query-optimization', 'query-optimization/disable-sql-query-optimization', 'query-optimization/prepared-statements-issue', - 'query-optimization/schema-change-troubleshooting', + 'query-optimization/automatic-schema-changes', + 'query-optimization/schema-change-troubleshooting', ], }, {