Automate Database Migrations (updated practice)

practices/automate-database-migrations.md

@@ -1,113 +1,114 @@
 # Automate Database Migrations
 
-Automate Database Migrations is a practice for managing and evolving a database schema over time through version-controlled scripts. These scripts enable incremental changes, such as adding features or fixing issues, while maintaining data integrity. This approach includes rollback capabilities, allowing any modifications to be safely reversed if necessary. It aligns with [Evolutionary Database Design](/practices/automate-database-migrations.md#evolutionary-database-design), promoting adaptability and risk mitigation by integrating changes smoothly into development workflows. This method is key for supporting continuous integration and deployment and minimizing disruption.
+The practice of automating database migrations helps teams manage and evolve a database schema over time. This is key for supporting continuous integration and deployment, and for minimizing disruption. Version-controlled scripts enable incremental changes, such as adding features or fixing issues, while maintaining data integrity. This approach includes rollback capabilities, allowing any modifications to be safely reversed if necessary. Integrating database migrations smoothly into development workflows aligns with [Evolutionary Database Design](/practices/automate-database-migrations.md#evolutionary-database-design), promoting adaptability and risk mitigation. 
 
-## Nuance
+## When to Experiment
+- You're a ...
 
-### Version Control Integration
-Integrating automatic database (db) migrations with version control systems is crucial for tracking and coordinating changes among team members. This ensures that every modification to the database schema is documented, reviewed, and synchronized with the corresponding application code changes, enhancing collaboration and minimizing conflicts. It is also preferred to have the migration scripts live in the same repository as the application code that operates on that database (assuming there is only one application that serves this role for that particular db/table/whatever). Doing this ensures the assumptions that the application makes of the db schema are correct and up to date.
+- You're a ...
 
-### Data Transformation and Migration
-In addition to schema changes, migrations might involve transforming existing data or migrating data between tables or formats. These operations can be complex and risk data integrity, requiring thorough testing and validation to ensure accuracy and completeness. Generally speaking, it is best practice to avoid breaking schema changes by adding new tables/columns as they're needed and deleting old ones only after ensuring they're no longer necessary. It can be tempting to simply update a column in place, which can introduce timing issues during deployments and/or integrity issues during rollbacks.
+## How to Gain Traction
 
-### Rollback Strategies
-While the ability to rollback changes is a fundamental aspect of automatic database migrations, designing effective rollback strategies is nuanced. It involves not just reverting schema changes but also considering the impact on data that has been modified or added since the migration was applied.
+### Create a Simple Migration Script
+Learn the basics of writing a migration script by creating a simple script that adds a new table to your database. Choose a database migration tool compatible with your database system (e.g., Flyway, Liquibase). Write a migration script to create a new table with at least three columns: an ID, a name, and a timestamp. Run the migration script against your local development database. Verify the table was created successfully by querying the database.
 
-### Migrations Execution Sequence
-One should build their migrations to avoid sequential dependencies. However, sometimes there's no avoiding that situation and errors can occur if migrations are not executed in the right order. In these rare situations, frequent communication between team members helps to reduce these types of errors.
+### Implement Rollbacks
+Understand the rollback process by writing a script that undoes a migration. Using the simple migration script you wrote to add a new table, write a rollback script that removes the table. Apply the rollback script to your development database. Confirm that the table has been successfully removed.
 
-### Testing and Validation
-Every application's testing needs are going to vary based on the architectural constraints, team norms, tech stack, etc. Generally speaking, it's a good idea to have some way to automate verification that the database is structured as expected. This can be accomplished in many ways: end to end tests, regression tests, production monitoring / alerting, etc. Select a verification strategy that gives developers the confidence they need without introducing too much complexity or reducing the reliability of the build process.
+### Ease In To Complex Migrations
+Gain experience with complex migrations involving data transformations. Write a migration script that alters an existing table by adding a new column. Populate the new column with data transformed from existing columns (e.g., concatenating two columns into a new one). Apply the migration and verify the data transformation was successful.
 
-### Performance Considerations
-Large-scale migrations can significantly impact database performance, especially in production environments. Planning for off-peak execution times, batch processing, and incremental migrations are strategies to mitigate performance degradation during the migration process.
+### Start a Book Club
+[Evolutionary Database Design](https://martinfowler.com/articles/evodb.html)
+This foundational article by Martin Fowler discusses the principles and practices behind evolutionary database design, which is central to the concept of automatic database migrations. It provides a thorough exploration of how to manage database schema changes in a way that supports agile development practices.
 
-### NoSQL Databases
-When considering migrations for NoSQL databases, it's essential to embrace schema flexibility by supporting multiple schema versions concurrently and allowing application logic to handle different data structures dynamically. Instead of enforcing strict schema updates through upgrade scripts, use a schema version identifier and implement incremental updates or "lazy" migrations as data is accessed. This approach maintains application compatibility across schema changes while reducing downtime and ensuring consistent data accessibility.
+[Migrations Done Well](https://newsletter.pragmaticengineer.com/p/migrations)
+This is an extensive guide on the process of migration. You can use it as a blueprint when preparing and executing migrations.
 
-## How to Improve
+### Host a Roundtable Discussion
+You can use the following conversation prompts:
 
-### Lead Workshops
+#### Alignment with Development Workflow
 
-#### Create a Simple Migration Script
+- How well do our automatic database migration processes integrate with our current development, testing, and deployment workflows?
+- Are there points of friction, and how can we address them?
 
-Learn the basics of writing a migration script by creating a simple script that adds a new table to your database. Choose a database migration tool compatible with your database system (e.g., Flyway, Liquibase, etc). Write a migration script to create a new table with at least three columns: an ID, a name, and a timestamp. Run the migration script against your local development database. Verify the table was created successfully by querying the database.
+#### Version Control Harmony
 
-#### Implementing Rollbacks
+- Are our migration scripts and database schema changes properly version controlled alongside our application code?
+- How can we improve traceability and coordination between code and database changes?
 
-Understand the rollback process by writing a script that undoes a migration. Using the migration script from Exercise 1, write a rollback script that removes the table. Apply the rollback script to your development database. Confirm that the table has been successfully removed.
+#### Sequential Dependency Handling
 
-#### Handling Complex Migrations
+- How effectively do we manage sequential dependencies within our migration scripts?
+- Are there tools or practices we could adopt to better handle complex migrations and dependency chains?
 
-Gain experience with complex migrations involving data transformations. Write a migration script that alters an existing table by adding a new column. Populate the new column with data transformed from existing columns (e.g., concatenating two columns into a new one). Apply the migration and verify the data transformation was successful.
+#### Data Integrity Safeguards
 
-### Start A Book Club
+- What measures do we have in place to ensure data integrity during migrations?
+- How do we test and validate data before and after migrations to prevent loss or corruption?
 
-### [Evolutionary Database Design](https://martinfowler.com/articles/evodb.html)
+#### Rollback Readiness
 
-This foundational article by Martin Fowler discusses the principles and practices behind evolutionary database design, which is central to the concept of automatic database migrations. It provides a thorough exploration of how to manage database schema changes in a way that supports agile development practices.
+- How confident are we in our ability to roll back a migration in case of failure?
+- Do we have a tested and reliable process for reverting changes without affecting data integrity?
 
-### [Migrations Done Well](https://newsletter.pragmaticengineer.com/p/migrations)
+#### Performance Impact Assessment
 
-This issue is an extensive guide on the process of migration. You can use it as a blueprint when preparing and executing migrations.
+- Have we experienced performance impacts during migrations?
+- How can we minimize downtime and performance degradation, especially in production environments?
 
-### Host A Roundtable Discussion
+#### Monitoring and Alerting
 
-## Introspective Questions
+- Do we have adequate monitoring and alerting in place to detect and respond to issues during the migration process?
+- How can we improve real-time visibility into migrations?
 
-### Alignment with Development Workflow
+#### Continuous Improvement
 
-How well do our automatic database migration processes integrate with our current development, testing, and deployment workflows?
-Are there points of friction, and how can we address them?
+- What lessons have we learned from past migrations, and how are we applying these insights to improve future processes?
+- Are there emerging tools or practices that could enhance our migration strategy?
 
-### Version Control Harmony
 
-Are our migration scripts and database schema changes properly version-controlled alongside our application code?
-How can we improve traceability and coordination between code and database changes?
+## Lessons From The Field
 
-### Sequential Dependency Handling
+- *Version Control Integration Enhances Collaboration and Minimizes Conflicts*. Integrating automatic database migrations with version control systems is crucial for tracking and coordinating changes among team members. This ensures that every modification to the database schema is documented, reviewed, and synchronized with the corresponding application code changes. It is also recommended to have the migration scripts live in the same repository as the application code that operates on that database (assuming there is only one application that serves this role for that particular db/table/whatever). Doing this ensures the assumptions that the application makes of the database schema are correct and up to date.
 
-How effectively do we manage sequential dependencies within our migration scripts?
-Are there tools or practices we could adopt to better handle complex migrations and dependency chains?
+- *Take Care With Data Transformation and Migration*. In addition to schema changes, migrations might involve transforming existing data or migrating data between tables or formats. These operations can be complex and risk data integrity, requiring thorough testing and validation to ensure accuracy and completeness. Generally speaking, the best practice is to avoid breaking schema changes by adding new tables/columns as they're needed and deleting old ones only after ensuring they're no longer necessary. It can be tempting to simply update a column in place, which can introduce timing issues during deployments and/or integrity issues during rollbacks.
 
-### Data Integrity Safeguards
+- *Rollback Strategies are Nuanced*. The ability to roll back changes is a fundamental aspect of automatic database migrations. But designing effective rollback strategies involves not just reverting schema changes but also considering the impact on data that has been modified or added since the migration was applied.
 
-What measures do we have in place to ensure data integrity during migrations?
-How do we test and validate data before and after migrations to prevent loss or corruption?
+- *Build Migrations to Avoid Sequential Dependencies*. Sometimes there's no avoiding sequential dependencies and errors can occur if migrations are not executed in the right order. In these rare situations, frequent communication between team members helps to reduce these types of errors.
 
-### Rollback Readiness
+- *Testing and Validation Will Vary But Should Contain Some Automation*. Every application's testing needs are going to vary based on the architectural constraints, team norms, tech stack, etc. Generally speaking, it's a good idea to have some way to automate verification that the database is structured as expected. This can be accomplished in many ways: end-to-end tests, regression tests, or production monitoring/alerting. Select a verification strategy that gives developers the confidence they need without introducing too much complexity or reducing the reliability of the build process.
 
-How confident are we in our ability to rollback a migration in case of failure?
-Do we have a tested and reliable process for reverting changes without affecting data integrity?
+- *Plan Around Performance Disruptions*. Large-scale migrations can significantly impact database performance, especially in production environments. Planning for off-peak execution times, batch processing, and incremental migrations are strategies to mitigate performance degradation during the migration process.
 
-### Performance Impact Assessment
+- *Migrating NoSQL Databases is Unique*. When considering migrations for NoSQL databases, it's essential to embrace schema flexibility. Do this by supporting multiple schema versions concurrently and allowing application logic to handle different data structures dynamically. Instead of enforcing strict schema updates through upgrade scripts, use a schema version identifier and implement incremental updates, or "lazy" migrations, as data is accessed. This approach maintains application compatibility across schema changes while reducing downtime and ensuring consistent data accessibility.
 
-Have we experienced performance impacts during migrations?
-How can we minimize downtime and performance degradation, especially in production environments?
+## Deciding to Pitch or Polish
+After experimenting with this practice for [**timeframe**], bring the team together to determine whether the following metrics and/or signals have changed in a positive direction.
 
-### Monitoring and Alerting
+### Fast & Intangible
 
-Do we have adequate monitoring and alerting in place to detect and respond to issues during the migration process?
-How can we improve real-time visibility into migrations?
+**Title of benefit**. 2-4 sentences about the benefit.
 
-### Continuous Improvement
+### Slow & Measurable
 
-What lessons have we learned from past migrations, and how are we applying these insights to improve future processes?
-Are there emerging tools or practices that could enhance our migration strategy?
+**Title of benefit**. 2-4 sentences about the benefit.
 
 ## Supporting Capabilities
 
-### [Database Change Management](https://dora.dev/devops-capabilities/technical/database-change-management)
-Automatic Database Migrations are integral to the DORA capability of Database Change Management. This practice ensures that database changes are systematically managed and aligned with application updates, reducing the risk of errors and downtime.
+### [Database Change Management](capabilities/database-change-management.md)
+Automatic database migrations ensure that database changes are systematically managed and aligned with application updates, reducing the risk of errors and downtime.
 
 ### [Version Control](/capabilities/version-control.md)
-Version control is foundational for Automatic Database Migrations, enabling teams to track, review, and manage changes to database schemas alongside application code, thus enhancing collaboration and reducing conflicts.
+Version control is foundational for automatic database migrations. It allows teams to track, review, and manage changes to database schemas alongside application code, thus enhancing collaboration and reducing conflicts.
 
-### [Continuous Integration](https://dora.dev/devops-capabilities/technical/continuous-integration)
-Continuous Integration is enhanced by Automatic Database Migrations, facilitating the easy integration of database changes into the development lifecycle. This supports agile practices by allowing for frequent and reliable updates to the database schema.
+### [Continuous Integration](capabilities/continuous-integration.md)
+Automatic database migrations facilitate the easy integration of database changes into the development lifecycle. This supports agile practices by allowing frequent and reliable updates to the database schema.
 
-### [Deployment Automation](https://dora.dev/devops-capabilities/technical/deployment-automation)
-Deployment Automation necessitates Automatic Database Migrations to automate the process of applying database schema changes. This capability reduces manual intervention and improves the reliability of deployments.
+### [Deployment Automation](capabilities/deployment-automation.md)
+The Deployment Automation capability requires automatic database migrations in order to automate the process of applying database schema changes. This reduces manual intervention and improves the reliability of deployments.
 
-### [Monitoring and Observability](https://dora.dev/devops-capabilities/technical/monitoring-and-observability)
-Monitoring and Observability are improved with Automatic Database Migrations by providing insights into the impact of database changes on application performance. This enables quick detection and resolution of issues.
+### [Monitoring and Observability](capabilities/monitoring-and-observability.md)
+Monitoring and observability are improved with automatic database migrations. By understanding the impact of database changes on application performance, issues can be quickly detected and resolved.