diff --git a/docs/cli/kubectl/plan.md b/docs/cli/kubectl/plan.md index 54e27645..9ff8e9f0 100644 --- a/docs/cli/kubectl/plan.md +++ b/docs/cli/kubectl/plan.md @@ -3,10 +3,10 @@ title: plan description: kubectl schemahero plan --- -Calculate a plan to generate a migration statement (DDL) when given a table manifest (YAML) and a running database. +Calculate a plan to generate migration statements (DDL) when given a manifest (YAML) and a running database. ### Usage -kubectl schemahero generate +kubectl schemahero plan Flag | Type | Description -----|------|------------ @@ -14,7 +14,38 @@ Flag | Type | Description `--out` | string | The filename to write DDL statements to. If not present output will be written to stdout. `--uri` | string | The database URI to connect to. Must be accessible from the workstation running the command `--overwrite` | bool | When set, will overwrite the file in `out`. This defaults to `true` -`-spec-file` | string | The filenamme or directory name containing the manifest(s) to apply -`-h`, `--help` | | help for install +`--spec-file` | string | The filename or directory name containing the manifest(s) to apply +`--spec-type` | string | The type of spec to plan. Options: `table` (default), `view`, `function`, `extension`, `datamigration` +`-h`, `--help` | | help for plan + +### Examples + +#### Planning Schema Changes +```bash +# Plan table schema changes +kubectl schemahero plan \ + --driver=postgres \ + --uri="postgres://user:pass@localhost:5432/db" \ + --spec-file=table.yaml \ + --spec-type=table + +# Plan view changes +kubectl schemahero plan \ + --driver=postgres \ + --uri="postgres://user:pass@localhost:5432/db" \ + --spec-file=view.yaml \ + --spec-type=view +``` + +#### Planning Data Migrations +```bash +# Plan data migration +kubectl schemahero plan \ + --driver=postgres \ + --uri="postgres://user:pass@localhost:5432/db" \ + --spec-file=data-migration.yaml \ + --spec-type=datamigration \ + --out=migration.sql +``` In addition to the flags documented above, this command accepts all [shared flags](../kubectl/shared-flags). diff --git a/docs/databases/postgresql/connecting.md b/docs/databases/postgresql/connecting.md index ff729c03..656bd3f1 100644 --- a/docs/databases/postgresql/connecting.md +++ b/docs/databases/postgresql/connecting.md @@ -53,3 +53,49 @@ spec: name: postgresql-secret key: uri ``` + +## PostgreSQL Features + +PostgreSQL is SchemaHero's most fully supported database, including: + +### Schema Management +- Tables, views, functions, and extensions +- All PostgreSQL column types and constraints +- Indexes, foreign keys, and primary keys +- User-defined types and domains + +### Data Migrations +PostgreSQL supports all SchemaHero data migration operations: + +- **Static Updates**: Update columns with static values +- **Calculated Updates**: Update columns using PostgreSQL expressions +- **Data Transformations**: + - Timezone conversions using `AT TIME ZONE` + - Type casting with PostgreSQL cast operators + - String transformations (UPPER, LOWER, REPLACE, SUBSTRING) +- **Custom SQL**: Execute arbitrary PostgreSQL statements with validation + +Example data migration: + +```yaml +apiVersion: schemas.schemahero.io/v1alpha4 +kind: DataMigration +metadata: + name: user-data-cleanup +spec: + database: my-database + name: user-data-cleanup + schema: + postgres: + - staticUpdate: + table: users + set: + status: "active" + updated_at: "CURRENT_TIMESTAMP" + where: "status IS NULL" + - calculatedUpdate: + table: users + calculations: + - column: full_name + expression: "CONCAT(first_name, ' ', last_name)" +``` diff --git a/docs/learn/data-migrations.md b/docs/learn/data-migrations.md index e4b62178..489cb5fe 100644 --- a/docs/learn/data-migrations.md +++ b/docs/learn/data-migrations.md @@ -1,29 +1,160 @@ --- title: Data Migrations -description: Comparing Data and Schema Migrations +description: Understanding and using data migrations with SchemaHero --- -There are two types of migrations that have to be managed and deployed: +SchemaHero supports both schema migrations and data migrations, allowing you to manage complete database changes in a unified workflow. -1. Schema Migrations -2. Data Migrations +## Schema vs Data Migrations -## Schema Migrations +### Schema Migrations +Schema migrations alter the structure of the database - creating tables, changing columns, adding indexes, etc. These are the core focus of SchemaHero and are always expressed declaratively. -A Schema migration can be expressed in SQL syntax, and alters the structure of the database. -These often are new tables, changing columns, altering indexed data and more. -These are commonly written and can always be expressed in an idempotent syntax. Different database engines enforce various rules on how these can be applied. -For example, MySQL will not allow a schema migration to be executed in a transaction, while PostgreSQL will. -Schema management is often unique to the database. -SchemaHero is focused on handling schema migrations. +### Data Migrations +Data migrations modify the actual data in your database - updating column values, calculating derived fields, transforming data formats, etc. SchemaHero now supports imperative data migrations alongside schema changes. -## Data Migrations +## Data Migration Types -Less frequently, a developer must migrate some data to a new format in a database. -This can involve calculating a new column and writing it, or creating new values in code and inserting them. -Many traditional database management tools blend the tasks of schema migrations and data migrations into one tool. +SchemaHero supports four types of data migration operations: -*SchemaHero is currently focused on schema migrations, with plans to support data migrations in the future.* +### 1. Static Updates +Update columns with static values: +```yaml +apiVersion: schemas.schemahero.io/v1alpha4 +kind: DataMigration +metadata: + name: set-defaults +spec: + database: mydb + name: set-defaults + schema: + postgres: + - staticUpdate: + table: users + set: + status: "active" + region: "us-east-1" + where: "status IS NULL" +``` -When looking at adding a data migration to a project, there is often a way to achieve the same result by implementing the update differently. +### 2. Calculated Updates +Update columns using calculated expressions from other columns: + +```yaml +- calculatedUpdate: + table: users + calculations: + - column: full_name + expression: "CONCAT(first_name, ' ', last_name)" + - column: display_email + expression: "LOWER(email)" + where: "full_name IS NULL" +``` + +### 3. Data Transformations +Perform common data transformations like timezone conversions, type casting, and string operations: + +```yaml +- transformUpdate: + table: events + transformations: + - column: created_at + transformType: timezone_convert + fromValue: UTC + toValue: America/New_York + - column: email + transformType: format_change + toValue: lowercase + where: "created_at IS NOT NULL" +``` + +### 4. Custom SQL +Execute arbitrary SQL for complex business logic: + +```yaml +- customSQL: + sql: | + UPDATE orders + SET total_with_tax = subtotal * 1.08, + status = 'processed' + WHERE status = 'pending' + validate: true +``` + +## Execution Control + +### Execution Order +Control when data migrations run relative to schema changes: + +```yaml +spec: + executionOrder: after_schema # or "before_schema" +``` + +### Idempotency +Mark migrations as safe to re-run: + +```yaml +spec: + idempotent: true # Can be executed multiple times safely +``` + +### Dependencies +Specify migration dependencies: + +```yaml +spec: + requires: + - table-schema-migration + - initial-data-setup +``` + +## CLI Usage + +Plan and apply data migrations using the SchemaHero CLI: + +```bash +# Plan data migration +kubectl-schemahero plan \ + --driver=postgres \ + --uri="postgres://user:pass@localhost:5432/db" \ + --spec-file=data-migration.yaml \ + --spec-type=datamigration + +# Apply the generated SQL +kubectl-schemahero apply \ + --driver=postgres \ + --uri="postgres://user:pass@localhost:5432/db" \ + --ddl=migration.sql +``` + +## Database Support + +- ✅ **PostgreSQL** - Full support for all data migration types +- ✅ **CockroachDB** - Uses PostgreSQL implementation (full support) +- ❌ **MySQL** - Not yet implemented +- ❌ **SQLite** - Not yet implemented +- ❌ **TimescaleDB** - Not yet implemented +- ❌ **Cassandra** - Not yet implemented + +*Note: While SchemaHero supports schema migrations for all these databases, data migrations are currently only implemented for PostgreSQL and CockroachDB.* + +## Best Practices + +1. **Test First** - Always test data migrations on a copy of production data +2. **Use Transactions** - Wrap complex migrations in transactions when possible +3. **Idempotent Design** - Design migrations to be safely re-runnable +4. **Backup Data** - Always backup your data before running migrations +5. **Incremental Approach** - Break complex migrations into smaller steps + +## Common Use Cases + +- Setting default values for new columns +- Calculating derived fields from existing data +- Normalizing data formats (email case, phone numbers) +- Converting between data types or units +- Migrating data between columns or tables +- Applying business logic updates to existing records + +Data migrations complement SchemaHero's schema management capabilities, providing a complete solution for database evolution. diff --git a/docs/reference/v1alpha4/datamigration.md b/docs/reference/v1alpha4/datamigration.md new file mode 100644 index 00000000..c2981c05 --- /dev/null +++ b/docs/reference/v1alpha4/datamigration.md @@ -0,0 +1,236 @@ +--- +title: DataMigration +description: Reference for schemas.schemahero.io/v1alpha4, Kind=DataMigration +--- + +```yaml +apiVersion: schemas.schemahero.io/v1alpha4 # version +kind: DataMigration # always DataMigration +metadata: + name: # Kubernetes object name + namespace: # Kubernetes namespace +spec: + database: # name of the database object + name: # actual name of the migration + executionOrder: # when to run: "before_schema" | "after_schema" (default) + idempotent: # boolean, can migration be run multiple times safely + requires: # array of migration dependencies + - migration-name # name of required migration + schema: # database-specific migration operations + postgres: # postgres operations array + - staticUpdate: # static value updates + table: # table name + set: # map of column: value pairs + column_name: "static_value" # column to update with static value + where: # optional WHERE clause + - calculatedUpdate: # calculated value updates + table: # table name + calculations: # array of calculations + - column: # column name to update + expression: # SQL expression for calculation + where: # optional WHERE clause + - transformUpdate: # data transformation operations + table: # table name + transformations: # array of transformations + - column: # column name to transform + transformType: # type: "timezone_convert" | "type_cast" | "format_change" | "string_transform" + fromValue: # source value (for timezone_convert) + toValue: # target value (for timezone_convert, type_cast, format_change) + parameters: # additional parameters for string_transform + type: # "replace" | "substring" + old: # old value for replace + new: # new value for replace + start: # start position for substring + length: # length for substring + where: # optional WHERE clause + - customSQL: # custom SQL execution + sql: # SQL statement(s) to execute + validate: # boolean, perform basic SQL validation +``` + +## Static Updates + +Update columns with static values: + +```yaml +- staticUpdate: + table: users + set: + status: "active" + region: "us-east-1" + updated_at: "CURRENT_TIMESTAMP" + where: "status IS NULL" +``` + +## Calculated Updates + +Update columns using expressions calculated from other columns: + +```yaml +- calculatedUpdate: + table: users + calculations: + - column: full_name + expression: "CONCAT(first_name, ' ', last_name)" + - column: display_email + expression: "LOWER(email)" + where: "full_name IS NULL OR display_email IS NULL" +``` + +## Transform Updates + +### Timezone Conversion +```yaml +- transformUpdate: + table: events + transformations: + - column: created_at + transformType: timezone_convert + fromValue: UTC + toValue: America/New_York + where: "timezone_converted = false" +``` + +### Type Casting +```yaml +- transformUpdate: + table: products + transformations: + - column: price + transformType: type_cast + toValue: "decimal(10,2)" +``` + +### Format Changes +```yaml +- transformUpdate: + table: users + transformations: + - column: email + transformType: format_change + toValue: lowercase + - column: username + transformType: format_change + toValue: uppercase +``` + +### String Transformations +```yaml +- transformUpdate: + table: products + transformations: + - column: sku + transformType: string_transform + parameters: + type: replace + old: "OLD-" + new: "NEW-" + - column: description + transformType: string_transform + parameters: + type: substring + start: "1" + length: "100" +``` + +## Custom SQL + +Execute arbitrary SQL statements: + +```yaml +- customSQL: + sql: | + UPDATE orders + SET total_with_tax = subtotal * 1.08, + loyalty_points = FLOOR(subtotal * 0.01) + WHERE status = 'completed' + AND total_with_tax IS NULL + validate: true +``` + +## Complete Example + +```yaml +apiVersion: schemas.schemahero.io/v1alpha4 +kind: DataMigration +metadata: + name: user-data-cleanup + namespace: production +spec: + database: userdb + name: user-data-cleanup + executionOrder: after_schema + idempotent: true + requires: + - user-table-schema + schema: + postgres: + # Set default values + - staticUpdate: + table: users + set: + status: "active" + notification_enabled: "true" + where: "status IS NULL" + + # Calculate derived fields + - calculatedUpdate: + table: users + calculations: + - column: full_name + expression: "TRIM(CONCAT(first_name, ' ', last_name))" + - column: display_email + expression: "LOWER(email)" + where: "full_name IS NULL" + + # Transform data formats + - transformUpdate: + table: users + transformations: + - column: created_at + transformType: timezone_convert + fromValue: UTC + toValue: America/New_York + - column: phone + transformType: string_transform + parameters: + type: replace + old: "-" + new: "" + where: "timezone_converted = false" + + # Apply business logic + - customSQL: + sql: | + UPDATE users + SET subscription_tier = CASE + WHEN total_purchases > 1000 THEN 'premium' + WHEN total_purchases > 100 THEN 'standard' + ELSE 'basic' + END + WHERE subscription_tier IS NULL + validate: true +``` + +## Execution Control + +### Execution Order +- `before_schema`: Execute before schema migrations +- `after_schema`: Execute after schema migrations (default) + +### Idempotency +- `true`: Migration can be safely re-executed +- `false`: Migration should only run once + +### Dependencies +List other migrations that must complete before this one runs. + +## Database Support + +Currently supported databases: +- **PostgreSQL**: Full support for all operation types +- **CockroachDB**: Uses PostgreSQL implementation (full support) +- **MySQL**: Not yet implemented +- **SQLite**: Not yet implemented +- **TimescaleDB**: Not yet implemented +- **Cassandra**: Not yet implemented diff --git a/mkdocs.yml b/mkdocs.yml index 648ea9c4..6df40314 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -110,6 +110,7 @@ nav: - v1alpha4: - 'Database': 'reference/v1alpha4/database.md' - 'Table': 'reference/v1alpha4/table.md' + - 'DataMigration': 'reference/v1alpha4/datamigration.md' - 'Migration': 'reference/v1alpha4/migration.md' - CLI Reference: - 'install': 'cli/kubectl/install.md'