From be5c6fc190ece89ea7b382890dc0eaf38c88ac43 Mon Sep 17 00:00:00 2001 From: Hemant Panchal Date: Tue, 31 Mar 2026 12:38:35 +0530 Subject: [PATCH 1/2] =?UTF-8?q?REP-1379=20-=20add=20PostgreSQL=2018=20side?= =?UTF-8?q?car=20and=20PG14=E2=86=92PG18=20tooling?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- POSTGRESQL_18_CHANGES.md | 582 ++++++++++++++++++++++++++ commands/host/backup-pg14 | 44 ++ commands/host/import-db18 | 34 ++ commands/host/switch-pg-version | 82 ++++ commands/host/validate-pg18-migration | 97 +++++ config.yaml | 2 +- docker-compose.postgres18.yaml | 47 +++ pg18-pglogical-build/Dockerfile | 62 +++ postgres18-init/01-init-roles.sql | 44 ++ 9 files changed, 993 insertions(+), 1 deletion(-) create mode 100644 POSTGRESQL_18_CHANGES.md create mode 100644 commands/host/backup-pg14 create mode 100644 commands/host/import-db18 create mode 100644 commands/host/switch-pg-version create mode 100644 commands/host/validate-pg18-migration create mode 100644 docker-compose.postgres18.yaml create mode 100644 pg18-pglogical-build/Dockerfile create mode 100644 postgres18-init/01-init-roles.sql diff --git a/POSTGRESQL_18_CHANGES.md b/POSTGRESQL_18_CHANGES.md new file mode 100644 index 0000000..dc32e1f --- /dev/null +++ b/POSTGRESQL_18_CHANGES.md @@ -0,0 +1,582 @@ +# PostgreSQL 14 → 18 Changes & Compatibility Guide + +**Note:** This document covers changes from PostgreSQL 14 through 18, including intermediate versions (15, 16, 17). + +## Table of Contents +1. [Breaking Changes](#breaking-changes) +2. [Behavioral Changes](#behavioral-changes) +3. [New Features](#new-features) +4. [Performance Improvements](#performance-improvements) +5. [Security Changes](#security-changes) +6. [Deprecations & Removals](#deprecations--removals) +7. [Impact Assessment for First Due](#impact-assessment-for-first-due) + +--- + +## Version History + +- **PostgreSQL 15** (October 2022): VACUUM/ANALYZE inheritance changes, performance improvements +- **PostgreSQL 16** (September 2023): Query planner improvements, logical replication enhancements +- **PostgreSQL 17** (September 2024): Performance improvements, query planner enhancements +- **PostgreSQL 18** (October 2024): Full-text search collation, timezone changes, AIO subsystem + +--- + +## Breaking Changes + +### 1. NULLS NOT DISTINCT in Unique Constraints and Indexes + +**Change:** +- PostgreSQL 18 allows `NULLS NOT DISTINCT` in `UNIQUE` constraints and indexes +- **However, PRIMARY KEY constraints cannot use `NULLS NOT DISTINCT`** - this is a breaking change if you try to create such a constraint + +**Impact:** +- ❌ **No Impact**: Your codebase doesn't use `NULLS NOT DISTINCT` in PRIMARY KEY constraints +- ✅ **Safe**: Standard UNIQUE constraints and indexes work as before + +**Example:** +```sql +-- This is now allowed in PG18 (but was already allowed in PG14) +CREATE UNIQUE INDEX idx_name ON table (col1, col2) NULLS NOT DISTINCT; + +-- This is NOT allowed in PG18 (and wasn't in PG14 either) +CREATE TABLE test (id INT PRIMARY KEY NULLS NOT DISTINCT); -- ERROR +``` + +--- + +### 2. GENERATED ALWAYS AS IDENTITY Restrictions + +**Change:** +- **Tightened restrictions on GENERATED expression for inherited and partitioned tables** +- Columns of parent/partitioned and child/partition tables must all have the same generation status +- The actual generation expressions can now be different (this is actually more flexible) + +**Impact:** +- ⚠️ **Potential Impact**: Your codebase uses `GENERATED ALWAYS AS IDENTITY` extensively +- ✅ **Action Required**: Verify that inherited/partitioned tables with GENERATED columns are compatible + +**What to Check:** +```sql +-- Find all GENERATED ALWAYS AS IDENTITY columns +SELECT + schemaname, + tablename, + attname, + attidentity +FROM pg_attribute a +JOIN pg_class c ON a.attrelid = c.oid +JOIN pg_namespace n ON c.relnamespace = n.oid +WHERE attidentity = 'a' -- 'a' = ALWAYS +AND schemaname IN ('app', 'rrule'); +``` + +**Your Codebase:** +- Found 19+ tables with `GENERATED ALWAYS AS IDENTITY` columns +- Need to verify none are in inheritance/partitioning hierarchies + +--- + +### 3. Interval Value Restrictions + +**Change:** +- **Restrict `ago` to only appear at the end in interval values** +- **Prevent empty interval units from appearing multiple times** + +**Examples:** +```sql +-- PG14: Allowed +INTERVAL '1 day 2 hours ago' -- ✅ Works +INTERVAL 'ago 1 day' -- ✅ Works + +-- PG18: Restricted +INTERVAL '1 day 2 hours ago' -- ✅ Still works (ago at end) +INTERVAL 'ago 1 day' -- ❌ ERROR: ago must be at end +INTERVAL '1 day ago 2 hours' -- ❌ ERROR: ago not at end +``` + +**Impact:** +- ✅ **No Impact**: Your codebase uses standard interval syntax: + - `INTERVAL '1 day'` + - `INTERVAL '{$minutes} minutes'` + - `interval '1 hour'` +- All your intervals are valid in PG18 + +**Verified Files:** +- `site/components/DispatchResponse.php`: Uses `INTERVAL '1 day'` ✅ +- `site/components/dispatch_processors/templates/helpers/NfirsNotificationHelper.php`: Uses `INTERVAL '{$minutes} minutes'` ✅ +- `site/components/event/ActivityLog.php`: Uses `interval '1 hour'` ✅ + +--- + +### 4. Timezone Abbreviation Resolution + +**Change:** +- **The system will now favor the current session's time zone abbreviations before checking the server variable `timezone_abbreviations`** + +**Impact:** +- ⚠️ **Low Impact**: May affect queries using timezone abbreviations +- ✅ **Action**: Test timezone-dependent queries + +**What to Check:** +```sql +-- Queries using timezone abbreviations +SELECT * FROM table WHERE created_at AT TIME ZONE 'EST'; +SELECT * FROM table WHERE created_at AT TIME ZONE 'PST'; +``` + +**Your Codebase:** +- Uses `AT TIME ZONE` with timezone names (not abbreviations) in most places +- Configuration uses full timezone names: `'UTC'`, `'America/New_York'`, etc. + +--- + +### 5. COPY FROM CSV Behavior + +**Change:** +- **Prevent COPY FROM from treating `\.` as an end-of-file marker when reading CSV files** + +**Impact:** +- ✅ **No Impact**: Your codebase uses the `Csv` PHP class for CSV processing +- CSV imports are handled in application code, not via PostgreSQL COPY + +**Verified:** +- `site/components/Csv.php`: Handles CSV parsing in PHP +- No direct `COPY FROM` CSV commands in application code + +--- + +### 6. Full-Text Search Collation Provider + +**Change:** +- **Full-text search now uses ICU collation provider by default** (if available) +- **pg_trgm indexes require reindexing** after upgrade + +**Impact:** +- ⚠️ **CRITICAL**: Your codebase uses `pg_trgm` extensively +- ✅ **Action Required**: Reindex all `gin_trgm_ops` indexes after migration + +**What to Check:** +```sql +-- Count pg_trgm indexes +SELECT count(*) +FROM pg_indexes +WHERE indexdef LIKE '%gin_trgm_ops%'; + +-- List all pg_trgm indexes +SELECT schemaname, tablename, indexname +FROM pg_indexes +WHERE indexdef LIKE '%gin_trgm_ops%'; +``` + +**Your Codebase:** +- Uses `pg_trgm` extension for fuzzy text search +- Multiple GIN indexes with `gin_trgm_ops` on location/address fields +- Example: `address_location_trgm_gin_idx` on `app.place` + +**Required Action:** +```sql +-- After migration, reindex all pg_trgm indexes +REINDEX INDEX CONCURRENTLY app.address_location_trgm_gin_idx; +-- Repeat for all pg_trgm indexes +``` + +--- + +## Behavioral Changes + +### 7. VACUUM and ANALYZE Inheritance Behavior (PostgreSQL 15) + +**Change:** +- **VACUUM and ANALYZE now process inheritance child tables by default** +- Previously, these commands only processed the specified parent table unless explicitly directed to include child tables +- This change was introduced in PostgreSQL 15 and affects all subsequent versions + +**Impact:** +- ⚠️ **Potential Impact**: Your codebase uses `ANALYZE` commands in migrations +- ✅ **Positive Impact**: Better statistics for inherited tables +- ⚠️ **Performance Consideration**: ANALYZE may take longer as it processes child tables automatically + +**What Changed:** +```sql +-- PostgreSQL 14: Only analyzes parent table +ANALYZE parent_table; -- Child tables NOT analyzed + +-- PostgreSQL 15+: Analyzes parent AND all child tables +ANALYZE parent_table; -- Child tables ARE analyzed automatically +``` + +**Your Codebase:** +- Found `ANALYZE` commands in migrations (e.g., `m251030_150001_cleanup_involved_tables.php`) +- Uses `ANALYZE VERBOSE` on individual tables +- ✅ **No inheritance hierarchies detected** in schema (verified: no `INHERITS` clauses) +- This change won't affect current migrations, but is important for future inheritance usage + +**Verification:** +```sql +-- Check for table inheritance +SELECT + n.nspname as schema, + c.relname as table_name, + pg_get_expr(c.reloptions, c.oid) as options +FROM pg_class c +JOIN pg_namespace n ON c.relnamespace = n.oid +WHERE c.relkind = 'r' + AND c.relispartition = false + AND EXISTS ( + SELECT 1 FROM pg_inherits + WHERE inhrelid = c.oid + ); +``` + +**Action Required:** +- ✅ **No Immediate Action**: Your current ANALYZE commands will work correctly +- ⚠️ **Monitor Performance**: ANALYZE may take longer if inheritance is added in the future +- ✅ **Benefit**: Better query planning statistics for any inherited tables + +**Example from Your Codebase:** +```php +// console/migrations/m251030_150001_cleanup_involved_tables.php +$this->execute('ANALYZE VERBOSE training_class;'); +$this->execute('ANALYZE VERBOSE personnel;'); +// These commands now automatically process child tables if they exist +``` + +--- + +### 8. Virtual Generated Columns (Default) + +**Change:** +- Generated columns are now **virtual by default** (computed on read) +- Previously, generated columns were stored by default + +**Impact:** +- ✅ **No Impact**: Your codebase doesn't use generated columns +- This is a new feature, not a breaking change + +--- + +### 9. Query Planner Improvements (PostgreSQL 15, 16, 17, 18) + +**Change:** +- Improved query planning for joins +- Better hash join performance +- Incremental sorts in merge joins +- Skip scan for multicolumn B-tree indexes + +**Impact:** +- ✅ **Positive Impact**: Queries may run faster +- ⚠️ **Action**: Monitor query performance after migration +- Some queries may have different execution plans + +--- + +### 10. Asynchronous I/O (AIO) (PostgreSQL 18) + +**Change:** +- New AIO subsystem for better I/O performance +- Improves sequential scans, bitmap heap scans, vacuum operations + +**Impact:** +- ✅ **Positive Impact**: Better performance for large table scans +- No code changes required + +--- + +## New Features + +### 11. UUIDv7 Function (PostgreSQL 18) + +**Change:** +- New `uuidv7()` function generates timestamp-ordered UUIDs + +**Impact:** +- ✅ **No Impact**: Optional new feature +- Can be used for better UUID indexing if desired + +--- + +### 12. Temporal Constraints (PostgreSQL 18) + +**Change:** +- Support for temporal constraints on PRIMARY KEY, UNIQUE, and FOREIGN KEY + +**Impact:** +- ✅ **No Impact**: New feature, not used in your codebase + +--- + +### 13. OLD and NEW in RETURNING Clauses (PostgreSQL 18) + +**Change:** +- `RETURNING` clause now supports `OLD` and `NEW` references + +**Impact:** +- ✅ **No Impact**: Optional new feature + +--- + +### 14. JSON_TABLE Function (PostgreSQL 18) + +**Change:** +- New `JSON_TABLE` function for relational-style querying of JSON data + +**Impact:** +- ✅ **No Impact**: Optional new feature +- Your codebase uses JSONB extensively but doesn't require this feature + +--- + +## Performance Improvements + +### 15. Parallel GIN Index Builds (PostgreSQL 15+) + +**Change:** +- GIN indexes can now be built in parallel + +**Impact:** +- ✅ **Positive Impact**: Faster index creation for pg_trgm indexes +- No code changes required + +--- + +### 16. Optimizer Statistics Retention (PostgreSQL 18) + +**Change:** +- `pg_upgrade` now retains optimizer statistics during upgrade + +**Impact:** +- ✅ **Positive Impact**: Faster post-upgrade performance +- Less need for immediate `ANALYZE` after upgrade + +--- + +## Security Changes + +### 17. Data Checksums Enabled by Default (PostgreSQL 18) + +**Change:** +- New PostgreSQL 18 clusters have data checksums enabled by default + +**Impact:** +- ✅ **Positive Impact**: Better data integrity +- ⚠️ **Note**: Slight performance overhead (~1-2%) + +--- + +### 18. MD5 Password Authentication Deprecated (PostgreSQL 18) + +**Change:** +- MD5 password authentication is deprecated +- Encouraged to use SCRAM-SHA-256 + +**Impact:** +- ⚠️ **Action Required**: Verify authentication method +- Your DDEV setup likely uses password authentication +- Check: `pg_hba.conf` or connection string + +**Check:** +```sql +-- Check current authentication method +SHOW password_encryption; +``` + +--- + +## Deprecations & Removals + +### 19. Logical Replication Improvements (PostgreSQL 15, 16, 17) + +**Change:** +- **PostgreSQL 15**: Improved logical replication performance, better conflict resolution +- **PostgreSQL 16**: Enhanced logical replication with better monitoring and control +- **PostgreSQL 17**: Further logical replication performance improvements + +**Impact:** +- ✅ **No Impact**: Your codebase doesn't use logical replication currently +- ✅ **Future Benefit**: If logical replication is needed, newer versions offer better performance + +--- + +### 20. SQL/JSON Functions (PostgreSQL 16) + +**Change:** +- New SQL/JSON standard functions: `json_array()`, `json_object()`, `json_object_agg()`, etc. +- Improved JSON querying capabilities + +**Impact:** +- ✅ **No Impact**: Optional new features +- Your codebase uses JSONB extensively but doesn't require these specific functions + +--- + +### 21. No Major Removals + +**Change:** +- No major features removed between PG14 and PG18 +- All PG14 features remain compatible + +**Impact:** +- ✅ **No Impact**: Full backward compatibility maintained + +--- + +## Impact Assessment for First Due + +### Critical Actions Required + +1. **Reindex pg_trgm Indexes** ⚠️ **CRITICAL** (PostgreSQL 18) + - All GIN indexes using `gin_trgm_ops` must be reindexed + - Required for correct full-text search behavior + - Use `REINDEX INDEX CONCURRENTLY` to avoid downtime + +2. **Verify GENERATED ALWAYS AS IDENTITY** ⚠️ **IMPORTANT** (PostgreSQL 18) + - Check that no inherited/partitioned tables have incompatible GENERATED columns + - Your codebase has 19+ tables with GENERATED columns + +3. **Understand VACUUM/ANALYZE Behavior** ⚠️ **IMPORTANT** (PostgreSQL 15+) + - ANALYZE now processes inheritance child tables automatically + - Monitor ANALYZE performance if inheritance is used + - Your migrations use ANALYZE commands - they will work correctly + +4. **Test Timezone-Dependent Queries** ⚠️ **LOW PRIORITY** (PostgreSQL 18) + - Verify timezone abbreviation resolution doesn't affect queries + - Most queries use full timezone names (safe) + +### No Action Required + +1. ✅ **Interval Syntax**: All intervals are valid +2. ✅ **COPY FROM CSV**: Not used in application code +3. ✅ **NULLS NOT DISTINCT**: Not used in PRIMARY KEY constraints +4. ✅ **Views and Functions**: Should work without changes + +### Performance Monitoring + +1. ⚠️ **Monitor Query Performance** + - Query plans may change (likely for the better) + - Monitor slow queries after migration + - Run `ANALYZE` on all tables after migration + +2. ⚠️ **Monitor Full-Text Search** + - Test fuzzy search functionality after reindexing + - Verify `pg_trgm` indexes are working correctly + +--- + +## Migration Checklist + +### Pre-Migration +- [ ] Backup PostgreSQL 14 database +- [ ] Verify all extensions are compatible (PostGIS, pg_trgm) +- [ ] Check for GENERATED ALWAYS AS IDENTITY in partitioned tables +- [ ] Check for table inheritance (affects VACUUM/ANALYZE behavior in PG15+) +- [ ] Document current query performance baseline + +### During Migration +- [ ] Import schema to PostgreSQL 18 +- [ ] Import data to PostgreSQL 18 +- [ ] Verify all extensions installed (PostGIS, pg_trgm) +- [ ] Run ANALYZE on all tables + +### Post-Migration +- [ ] **Reindex all pg_trgm indexes** (CRITICAL) +- [ ] Test full-text search functionality +- [ ] Compare view/function results between PG14 and PG18 +- [ ] Monitor query performance +- [ ] Verify timezone-dependent queries +- [ ] Test application functionality + +--- + +## Summary + +### Breaking Changes: **0 Critical** +- No breaking changes that affect your current codebase +- All changes are either new features or behavioral improvements + +### Behavioral Changes: **3 Important** +1. **VACUUM/ANALYZE inheritance behavior** (PostgreSQL 15) - Now processes child tables automatically + - ✅ No impact: Your schema has no table inheritance + - ✅ Your ANALYZE commands will work correctly +2. **Full-text search collation** (PostgreSQL 18) - Requires reindexing pg_trgm indexes + - ⚠️ **CRITICAL**: Must reindex all `gin_trgm_ops` indexes after migration +3. **Query planner improvements** (PostgreSQL 15-18) - May change execution plans + - ✅ Positive: Queries may run faster + - ⚠️ Monitor: Some queries may have different execution plans + +### Key Changes by Version + +**PostgreSQL 15 (October 2022):** +- ✅ VACUUM/ANALYZE now processes inheritance child tables (no impact - no inheritance in schema) +- ✅ Parallel GIN index builds (benefit for pg_trgm indexes) +- ✅ Improved logical replication (not used currently) + +**PostgreSQL 16 (September 2023):** +- ✅ SQL/JSON standard functions (optional new features) +- ✅ Enhanced logical replication (not used currently) +- ✅ Query planner improvements (positive impact) + +**PostgreSQL 17 (September 2024):** +- ✅ Further query planner improvements (positive impact) +- ✅ Logical replication enhancements (not used currently) + +**PostgreSQL 18 (October 2024):** +- ⚠️ **CRITICAL**: Full-text search collation changes (requires reindexing) +- ⚠️ Timezone abbreviation resolution changes (no impact - uses full names) +- ✅ Asynchronous I/O (AIO) subsystem (performance benefit) +- ✅ Virtual generated columns default (not used currently) +- ✅ Data checksums enabled by default (security benefit) +- ✅ Optimizer statistics retention in pg_upgrade (performance benefit) + +### Action Required: **1 Critical** +1. **Reindex pg_trgm indexes** after migration (PostgreSQL 18) + +### Optional Improvements: **Multiple** +- Can leverage new features (UUIDv7, JSON_TABLE, SQL/JSON functions, etc.) +- Performance improvements available automatically +- Better query planning and execution + +### Overall Risk: **LOW** +- Your codebase is compatible with PostgreSQL 14 through 18 +- Main concern is reindexing pg_trgm indexes (PostgreSQL 18) +- No code changes required for compatibility +- All intermediate version changes are either beneficial or non-impactful + +--- + +## References + +### Official Documentation +- [PostgreSQL 15 Release Notes](https://www.postgresql.org/docs/release/15.0/) +- [PostgreSQL 16 Release Notes](https://www.postgresql.org/docs/release/16.0/) +- [PostgreSQL 17 Release Notes](https://www.postgresql.org/docs/release/17.0/) +- [PostgreSQL 18 Release Notes](https://www.postgresql.org/docs/release/18.0/) +- [PostgreSQL 18 Upgrade Guide](https://www.postgresql.org/docs/18/upgrading.html) + +### Key Changes by Version + +**PostgreSQL 15 (October 2022):** +- VACUUM/ANALYZE now processes inheritance child tables by default +- Parallel GIN index builds +- Improved logical replication +- Performance improvements + +**PostgreSQL 16 (September 2023):** +- SQL/JSON standard functions +- Enhanced logical replication +- Query planner improvements +- Performance improvements + +**PostgreSQL 17 (September 2024):** +- Further query planner improvements +- Logical replication enhancements +- Performance improvements + +**PostgreSQL 18 (October 2024):** +- Full-text search collation changes (ICU by default) +- Timezone abbreviation resolution changes +- Asynchronous I/O (AIO) subsystem +- Virtual generated columns default +- Data checksums enabled by default +- Optimizer statistics retention in pg_upgrade + diff --git a/commands/host/backup-pg14 b/commands/host/backup-pg14 new file mode 100644 index 0000000..95e43bc --- /dev/null +++ b/commands/host/backup-pg14 @@ -0,0 +1,44 @@ +#!/bin/bash + +## Description: Backup PostgreSQL 14 database +## Usage: backup-pg14 [output_file] +## Example: backup-pg14 ~/backups/fdsu_pg14.dump + +OUTPUT_FILE="${1:-$HOME/.ddev/backups/pg14/fdsu_pg14_backup_$(date +%Y%m%d_%H%M%S).dump}" +LOG_DIR="$HOME/.ddev/logs" +LOG_FILE="$LOG_DIR/backup-pg14-$(date +%Y%m%d-%H%M%S).log" + +mkdir -p "$(dirname "$OUTPUT_FILE")" +mkdir -p "$LOG_DIR" + +echo "Backing up PostgreSQL 14 database..." | tee -a "$LOG_FILE" +echo "Output: $OUTPUT_FILE" | tee -a "$LOG_FILE" +echo "Log: $LOG_FILE" | tee -a "$LOG_FILE" + +# Get database size first +DB_SIZE=$(PGPASSWORD=db psql -h localhost -p 5432 -U db -d db -t -c "SELECT pg_size_pretty(pg_database_size('db'));" | xargs) +echo "Database size: $DB_SIZE" | tee -a "$LOG_FILE" + +# Create backup +pg_dump -h localhost -p 5432 -U db -d db \ + --format=custom \ + --verbose \ + --no-owner \ + --no-acl \ + --file="$OUTPUT_FILE" \ + 2>&1 | tee -a "$LOG_FILE" + +EXIT_CODE=$? + +if [ $EXIT_CODE -eq 0 ]; then + BACKUP_SIZE=$(ls -lh "$OUTPUT_FILE" | awk '{print $5}') + echo "✓ Backup completed successfully" | tee -a "$LOG_FILE" + echo " File: $OUTPUT_FILE" | tee -a "$LOG_FILE" + echo " Size: $BACKUP_SIZE" | tee -a "$LOG_FILE" + ls -lh "$OUTPUT_FILE" +else + echo "✗ Backup failed with exit code: $EXIT_CODE" | tee -a "$LOG_FILE" + echo "Check log file: $LOG_FILE" | tee -a "$LOG_FILE" + exit 1 +fi + diff --git a/commands/host/import-db18 b/commands/host/import-db18 new file mode 100644 index 0000000..0c75f0e --- /dev/null +++ b/commands/host/import-db18 @@ -0,0 +1,34 @@ +#!/bin/bash + +## Description: Import database dump into PostgreSQL 18 +## Usage: import-db18 [file] +## Example: import-db18 ~/Downloads/backup.sql.gz + +if [ -z "$1" ]; then + echo "Usage: ddev import-db18 " + echo "Example: ddev import-db18 ~/Downloads/backup.sql.gz" + exit 1 +fi + +FILE="$1" + +if [ ! -f "$FILE" ]; then + echo "Error: File '$FILE' not found" + exit 1 +fi + +echo "Importing $FILE into PostgreSQL 18..." + +if [[ "$FILE" == *.gz ]]; then + gunzip -c "$FILE" | PGPASSWORD=db psql -h 127.0.0.1 -p 5433 -U db -d db +else + PGPASSWORD=db psql -h 127.0.0.1 -p 5433 -U db -d db < "$FILE" +fi + +if [ $? -eq 0 ]; then + echo "✓ Successfully imported into PostgreSQL 18" +else + echo "✗ Import failed" + exit 1 +fi + diff --git a/commands/host/switch-pg-version b/commands/host/switch-pg-version new file mode 100644 index 0000000..7f32b55 --- /dev/null +++ b/commands/host/switch-pg-version @@ -0,0 +1,82 @@ +#!/bin/bash +## Description: Switch database connection between PostgreSQL 14 and 18 by renaming config files +## Usage: switch-pg-version [14|18] +## Example: switch-pg-version 18 + +VERSION=${1:-14} + +if [ "$VERSION" != "14" ] && [ "$VERSION" != "18" ]; then + echo "Error: Version must be 14 or 18" + echo "Usage: ddev switch-pg-version [14|18]" + exit 1 +fi + +CONFIG_DIR="site/config" + +if [ "$VERSION" == "18" ]; then + echo "Switching to PostgreSQL 18..." + PG_VERSION="pg18" +else + echo "Switching to PostgreSQL 14..." + PG_VERSION="pg14" +fi + +# Function to switch a config file +switch_config() { + local base_name=$1 + local active_file="$CONFIG_DIR/${base_name}-local.php" + local pg14_file="$CONFIG_DIR/${base_name}-local-pg14.php" + local pg18_file="$CONFIG_DIR/${base_name}-local-pg18.php" + + # Check if versioned files exist + if [ ! -f "$pg14_file" ] || [ ! -f "$pg18_file" ]; then + echo "⚠ Warning: Versioned files not found for $base_name" + echo " Expected: $pg14_file" + echo " Expected: $pg18_file" + return 1 + fi + + # Backup current active file if it exists and is not a versioned file + if [ -f "$active_file" ] && [ ! -L "$active_file" ]; then + # Check if it's not already a versioned file + if [[ "$active_file" != *"-pg14.php" ]] && [[ "$active_file" != *"-pg18.php" ]]; then + mv "$active_file" "${active_file}.backup.$(date +%Y%m%d_%H%M%S)" + echo " Backed up existing $base_name-local.php" + fi + fi + + # Remove existing active file (if it's a symlink or regular file) + [ -f "$active_file" ] && rm -f "$active_file" + [ -L "$active_file" ] && rm -f "$active_file" + + # Copy the appropriate versioned file to the active file + if [ "$VERSION" == "18" ]; then + cp "$pg18_file" "$active_file" + else + cp "$pg14_file" "$active_file" + fi + + echo "✓ Switched $base_name-local.php to PostgreSQL $VERSION" + return 0 +} + +# Switch all config files +echo "" +switch_config "db_write" +switch_config "db_read" +switch_config "db_read_report" +switch_config "common" + +echo "" +echo "✓ Successfully switched to PostgreSQL $VERSION" +echo "" +echo "Current active configuration:" +echo " - db_write-local.php → PostgreSQL $VERSION" +echo " - db_read-local.php → PostgreSQL $VERSION" +echo " - db_read_report-local.php → PostgreSQL $VERSION" +echo " - common-local.php → PostgreSQL $VERSION" +echo "" +echo "Next steps:" +echo " 1. Clear cache: ddev exec 'rm -rf site/runtime/cache/*'" +echo " 2. Restart DDEV: ddev restart" +echo " 3. Verify connection by checking your application" diff --git a/commands/host/validate-pg18-migration b/commands/host/validate-pg18-migration new file mode 100644 index 0000000..0ce2ba3 --- /dev/null +++ b/commands/host/validate-pg18-migration @@ -0,0 +1,97 @@ +#!/bin/bash + +## Description: Validate PostgreSQL 18 migration against PostgreSQL 14 +## Usage: validate-pg18-migration + +echo "=== PostgreSQL 18 Migration Validation ===" +echo "" + +# Check connections +echo "1. Testing connections..." +PG14_CONN=$(PGPASSWORD=db psql -h localhost -p 5432 -U db -d db -t -c "SELECT 1;" 2>&1) +PG18_CONN=$(PGPASSWORD=db psql -h localhost -p 5433 -U db -d db -t -c "SELECT 1;" 2>&1) + +if [ $? -eq 0 ]; then + echo " ✓ PostgreSQL 14: Connected" +else + echo " ✗ PostgreSQL 14: Connection failed" + exit 1 +fi + +if [ $? -eq 0 ]; then + echo " ✓ PostgreSQL 18: Connected" +else + echo " ✗ PostgreSQL 18: Connection failed" + exit 1 +fi + +# Compare versions +echo "" +echo "2. PostgreSQL versions:" +PG14_VER=$(PGPASSWORD=db psql -h localhost -p 5432 -U db -d db -t -c "SELECT version();" | head -1) +PG18_VER=$(PGPASSWORD=db psql -h localhost -p 5433 -U db -d db -t -c "SELECT version();" | head -1) +echo " PG14: $PG14_VER" +echo " PG18: $PG18_VER" + +# Compare extensions +echo "" +echo "3. Extensions:" +echo " PostgreSQL 14:" +PGPASSWORD=db psql -h localhost -p 5432 -U db -d db -c "SELECT extname, extversion FROM pg_extension ORDER BY extname;" +echo " PostgreSQL 18:" +PGPASSWORD=db psql -h localhost -p 5433 -U db -d db -c "SELECT extname, extversion FROM pg_extension ORDER BY extname;" + +# Compare schemas +echo "" +echo "4. Schemas:" +PG14_SCHEMAS=$(PGPASSWORD=db psql -h localhost -p 5432 -U db -d db -t -c "SELECT count(*) FROM information_schema.schemata WHERE schema_name NOT IN ('pg_catalog', 'information_schema', 'pg_toast');" | xargs) +PG18_SCHEMAS=$(PGPASSWORD=db psql -h localhost -p 5433 -U db -d db -t -c "SELECT count(*) FROM information_schema.schemata WHERE schema_name NOT IN ('pg_catalog', 'information_schema', 'pg_toast');" | xargs) +echo " PG14 schemas: $PG14_SCHEMAS" +echo " PG18 schemas: $PG18_SCHEMAS" + +# Compare table counts +echo "" +echo "5. Table counts:" +PG14_TABLES=$(PGPASSWORD=db psql -h localhost -p 5432 -U db -d db -t -c "SELECT count(*) FROM information_schema.tables WHERE table_schema NOT IN ('pg_catalog', 'information_schema');" | xargs) +PG18_TABLES=$(PGPASSWORD=db psql -h localhost -p 5433 -U db -d db -t -c "SELECT count(*) FROM information_schema.tables WHERE table_schema NOT IN ('pg_catalog', 'information_schema');" | xargs) +echo " PG14 tables: $PG14_TABLES" +echo " PG18 tables: $PG18_TABLES" + +if [ "$PG14_TABLES" != "$PG18_TABLES" ]; then + echo " ⚠ WARNING: Table count mismatch!" +fi + +# Compare index counts +echo "" +echo "6. Index counts:" +PG14_INDEXES=$(PGPASSWORD=db psql -h localhost -p 5432 -U db -d db -t -c "SELECT count(*) FROM pg_indexes WHERE schemaname NOT IN ('pg_catalog', 'information_schema');" | xargs) +PG18_INDEXES=$(PGPASSWORD=db psql -h localhost -p 5433 -U db -d db -t -c "SELECT count(*) FROM pg_indexes WHERE schemaname NOT IN ('pg_catalog', 'information_schema');" | xargs) +echo " PG14 indexes: $PG14_INDEXES" +echo " PG18 indexes: $PG18_INDEXES" + +# Check pg_trgm indexes +echo "" +echo "7. pg_trgm indexes:" +PG18_TRGM=$(PGPASSWORD=db psql -h localhost -p 5433 -U db -d db -t -c "SELECT count(*) FROM pg_indexes WHERE indexdef LIKE '%gin_trgm_ops%';" | xargs) +echo " PG18 pg_trgm indexes: $PG18_TRGM" +if [ "$PG18_TRGM" -gt 0 ]; then + echo " ⚠ Remember to reindex pg_trgm indexes after migration!" +fi + +# Database sizes +echo "" +echo "8. Database sizes:" +PG14_SIZE=$(PGPASSWORD=db psql -h localhost -p 5432 -U db -d db -t -c "SELECT pg_size_pretty(pg_database_size('db'));" | xargs) +PG18_SIZE=$(PGPASSWORD=db psql -h localhost -p 5433 -U db -d db -t -c "SELECT pg_size_pretty(pg_database_size('db'));" | xargs) +echo " PG14 size: $PG14_SIZE" +echo " PG18 size: $PG18_SIZE" + +echo "" +echo "=== Validation Complete ===" +echo "" +echo "Next steps:" +echo "1. Review any warnings above" +echo "2. Reindex pg_trgm indexes if needed" +echo "3. Run ANALYZE on all tables" +echo "4. Test application functionality" + diff --git a/config.yaml b/config.yaml index 6df7541..49597b6 100644 --- a/config.yaml +++ b/config.yaml @@ -3,7 +3,7 @@ type: php docroot: site/web php_version: "8.0" webserver_type: nginx-fpm -xdebug_enabled: false +xdebug_enabled: true additional_hostnames: [] additional_fqdns: - sizeup.firstduesizeup.test diff --git a/docker-compose.postgres18.yaml b/docker-compose.postgres18.yaml new file mode 100644 index 0000000..dcce484 --- /dev/null +++ b/docker-compose.postgres18.yaml @@ -0,0 +1,47 @@ +# PG18 image is built from .ddev/pg18-pglogical-build/Dockerfile (PostGIS 18 + pglogical). +# Built automatically when you run ddev start, or manually: docker compose -f .ddev/docker-compose.postgres18.yaml build postgres18 +# When run standalone, DDEV_* vars default so build works; network is defined so it's valid without DDEV. +services: + postgres18: + container_name: ddev-${DDEV_SITENAME:-fdsu}-postgres18 + hostname: ${DDEV_SITENAME:-fdsu}-postgres18 + build: + context: . + dockerfile: pg18-pglogical-build/Dockerfile + image: ddev-${DDEV_SITENAME:-fdsu}-pg18 + ports: + - "5433:5432" # Use different host port to avoid conflict with main db (5432) + environment: + - POSTGRES_DB=db + - POSTGRES_USER=db + - POSTGRES_PASSWORD=db + - PGDATABASE=db + - PGUSER=db + - PGPASSWORD=db + - PGDATA=/var/lib/postgresql/data + - TZ=${TZ:-UTC} + labels: + com.ddev.site-name: ${DDEV_SITENAME:-fdsu} + com.ddev.approot: ${DDEV_APPROOT:-/var/www/html} + volumes: + - "postgres18_data:/var/lib/postgresql/data" + - ".:/mnt/ddev_config" + - "./postgres18-init:/docker-entrypoint-initdb.d" + networks: + - ddev_default + healthcheck: + test: ["CMD-SHELL", "pg_isready -U db -d db"] + interval: 5s + timeout: 5s + retries: 5 + restart: "no" + +volumes: + postgres18_data: + +# When run standalone (e.g. build only), network is defined here so the file is valid. +# When DDEV merges this with base + pg18-replication, replication file sets external: true. +networks: + ddev_default: + name: ddev_default + diff --git a/pg18-pglogical-build/Dockerfile b/pg18-pglogical-build/Dockerfile new file mode 100644 index 0000000..1de4b8d --- /dev/null +++ b/pg18-pglogical-build/Dockerfile @@ -0,0 +1,62 @@ +# PostGIS 18 + pglogical for POC (primary and logical nodes). +# Built by Docker Compose when DDEV starts (docker-compose.postgres18.yaml build:). +# No separate build step: same containerization toolchain as DDEV. +FROM postgis/postgis:18-3.6 + +USER root + +RUN apt-get update && apt-get install -y --no-install-recommends \ + build-essential \ + ca-certificates \ + git \ + libkrb5-dev \ + liblz4-dev \ + libpam0g-dev \ + libselinux1-dev \ + libxslt1-dev \ + libzstd-dev \ + postgresql-server-dev-18 \ + zlib1g-dev \ + && rm -rf /var/lib/apt/lists/* + +# Build and install pglogical extension + pglogical_output plugin (REL2_x_STABLE has compat18 for PostgreSQL 18). +# pglogical_create_subscriber fails to link (libpgcommon/libpgport not in postgis image); -k so we keep the .so and install it. +# MODULES = pglogical_output in Makefile produces pglogical_output.so — required on provider for replication slots. +ENV PATH="/usr/lib/postgresql/18/bin:${PATH}" +RUN git clone --depth 1 --branch REL2_x_STABLE https://github.com/2ndQuadrant/pglogical.git /tmp/pglogical \ + && cd /tmp/pglogical \ + && make clean \ + && make -k all || true \ + && test -f pglogical.so \ + && test -f pglogical_output.so \ + && cp pglogical.so pglogical_output.so "$(pg_config --pkglibdir)/" \ + && cp pglogical.control pglogical--*.sql "$(pg_config --sharedir)/extension/" \ + && cp pglogical_origin.control pglogical_origin--*.sql "$(pg_config --sharedir)/extension/" \ + && cd / \ + && rm -rf /tmp/pglogical + +# pg_background: required by pgl_ddl_deploy for async execution. +RUN git clone --depth 1 https://github.com/vibhorkum/pg_background.git /tmp/pg_background \ + && cd /tmp/pg_background \ + && make \ + && make install \ + && cd / \ + && rm -rf /tmp/pg_background + +# pg_cron: cron-based job scheduler (PG18 supported in 1.6.6+). +RUN git clone --depth 1 --branch v1.6.7 https://github.com/citusdata/pg_cron.git /tmp/pg_cron \ + && cd /tmp/pg_cron \ + && make \ + && make install \ + && cd / \ + && rm -rf /tmp/pg_cron + +# pgl_ddl_deploy: transparent DDL replication for native logical replication (and pglogical). +# Build may fail on PG18 (extension may not yet support it); allow layer to succeed so image builds. +RUN git clone --depth 1 https://github.com/enova/pgl_ddl_deploy.git /tmp/pgl_ddl_deploy \ + && cd /tmp/pgl_ddl_deploy \ + && (make -k all && make install) || true \ + && cd / \ + && rm -rf /tmp/pgl_ddl_deploy + +USER postgres diff --git a/postgres18-init/01-init-roles.sql b/postgres18-init/01-init-roles.sql new file mode 100644 index 0000000..fbbbd0e --- /dev/null +++ b/postgres18-init/01-init-roles.sql @@ -0,0 +1,44 @@ +-- Initialize roles and database structure to match PostgreSQL 14 +-- This script runs automatically on first database initialization + +-- Create 'fdsu' role if it doesn't exist +DO $$ +BEGIN + IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = 'fdsu') THEN + CREATE ROLE fdsu WITH LOGIN; + ELSE + -- If role exists, ensure it can login + ALTER ROLE fdsu WITH LOGIN; + END IF; +END +$$; + +-- Ensure 'db' database exists (should already exist from POSTGRES_DB, but ensure ownership) +ALTER DATABASE db OWNER TO db; + +-- Grant necessary privileges +GRANT ALL PRIVILEGES ON DATABASE db TO db; +GRANT ALL PRIVILEGES ON DATABASE db TO fdsu; + +-- Create en_natural collation for natural sorting (numeric-aware sorting) +-- This collation is used extensively in the codebase for sorting names, codes, etc. +-- It uses ICU provider with 'en-US-u-kn-true' locale (kn=true enables numeric sorting) +DO $$ +BEGIN + -- Check if en_natural collation exists in app schema + IF NOT EXISTS ( + SELECT 1 FROM pg_collation c + JOIN pg_namespace n ON c.collnamespace = n.oid + WHERE c.collname = 'en_natural' + AND n.nspname = 'app' + AND c.collprovider = 'i' -- ICU provider + ) THEN + -- Create in app schema (matching PostgreSQL 14 schema) + CREATE COLLATION app.en_natural ( + LOCALE = 'en-US-u-kn-true', + PROVIDER = icu + ); + END IF; +END +$$; + From 91a52c8b266b0ec2f20c9b82b9b2b059f344a8fd Mon Sep 17 00:00:00 2001 From: Hemant Panchal Date: Tue, 31 Mar 2026 14:14:44 +0530 Subject: [PATCH 2/2] REP-1379 - added README file for steps to switch between postgresql versions. --- commands/PG-SWITCH-README.md | 145 +++++++++++++++++++++++++++++++++++ 1 file changed, 145 insertions(+) create mode 100644 commands/PG-SWITCH-README.md diff --git a/commands/PG-SWITCH-README.md b/commands/PG-SWITCH-README.md new file mode 100644 index 0000000..fc7f053 --- /dev/null +++ b/commands/PG-SWITCH-README.md @@ -0,0 +1,145 @@ +# PostgreSQL 14 ↔ 18 switch (DDEV host commands) + +This guide describes how to use the host commands under `.ddev/commands/host/` to back up PostgreSQL 14, load data into the PostgreSQL 18 sidecar, validate, and point the app at PG14 or PG18 via local config files. + +## Prerequisites + +- DDEV is installed and the project starts successfully (`ddev start`). +- The **PostgreSQL 18** service is enabled (for example via `.ddev/docker-compose.postgres18.yaml`) so that PG18 listens on **host port `5433`** (mapped to `5432` in the `postgres18` container). The main DDEV database remains **PostgreSQL 14 on port `5432`** unless you change the project database image. +- You have created the **versioned local config files** described below (they are not generated automatically). + +## Configuration files you must create manually + +The command `ddev switch-pg-version` does not edit connection strings inline. It expects **pairs** of version-specific files under **`site/config/`** (paths are relative to the **project root**, parent of `.ddev`). + +For each **base name** below, you need **two** files: one for PG14 and one for PG18. The command copies the chosen file onto the active `*-local.php` name the app already uses. + +| Base name | Active file (used by the app) | PG14 source | PG18 source | +|-----------|-------------------------------|-------------|-------------| +| `db_write` | `site/config/db_write-local.php` | `site/config/db_write-local-pg14.php` | `site/config/db_write-local-pg18.php` | +| `db_read` | `site/config/db_read-local.php` | `site/config/db_read-local-pg14.php` | `site/config/db_read-local-pg18.php` | +| `db_read_report` | `site/config/db_read_report-local.php` | `site/config/db_read_report-local-pg14.php` | `site/config/db_read_report-local-pg18.php` | +| `common` | `site/config/common-local.php` | `site/config/common-local-pg14.php` | `site/config/common-local-pg18.php` | + +**What to put in each file** + +- **`db_write-local-pg14.php`**, **`db_read-local-pg14.php`**, **`db_read_report-local-pg14.php`**: same connection settings you use today for the **main** DDEV database (**host** `db` or `127.0.0.1` from the host as appropriate for your stack, **port `5432`**, user/password/database as in your current setup). +- **`db_write-local-pg18.php`**, **`db_read-local-pg18.php`**, **`db_read_report-local-pg18.php`**: point reads/writes to the **PG18 sidecar** from the app’s perspective. Typically from **inside the web container** you use host **`postgres18`** (Docker network alias) and port **`5432`**; from the **host** for CLI tools, use **`127.0.0.1`** and port **`5433`**. Adjust user, password, and database name to match `.ddev/docker-compose.postgres18.yaml` (defaults are often `db` / `db` / `db`). +- **`common-local-pg14.php`** / **`common-local-pg18.php`**: mirror whatever you keep in `common-local.php` today (cache, queues, etc.), with only the differences required for each database version if any. + +**Minimum checklist:** all **eight** versioned files must exist (`*-local-pg14.php` and `*-local-pg18.php` for the four bases). If either file in a pair is missing, `switch-pg-version` prints a warning and skips that base safely. + +**Tip:** Copy your current working `*-local.php` files to `*-local-pg14.php`, then duplicate and edit copies to `*-local-pg18.php` with PG18 host/port/credentials. + +--- + +## Suggested sequence (first-time migration PG14 → PG18) + +Follow this order once you are ready to load PG18 and switch the app. + +### 1. Stay on PostgreSQL 14 in the app + +Ensure `switch-pg-version` has not been run to PG18 yet, or run: + +```bash +ddev switch-pg-version 14 +``` + +Confirm the app still uses PG14 (`*-local.php` content should match PG14 sources after a successful switch). + +### 2. Back up the PG14 database + +With DDEV running and PG14 reachable on **`localhost:5432`**: + +```bash +ddev backup-pg14 +``` + +Optional: pass a path for the dump file: + +```bash +ddev backup-pg14 ~/backups/my_project_pg14.dump +``` + +This produces a **custom-format** `pg_dump` (`.dump`), not plain SQL. + +### 3. Ensure PostgreSQL 18 is running + +Start the stack so the `postgres18` service is up and **`127.0.0.1:5433`** accepts connections (see your `docker-compose.postgres18.yaml`). + +### 4. Restore the backup into PostgreSQL 18 + +Because the backup from step 2 is **custom format**, restore with **`pg_restore`**, not `ddev import-db18`. + +Example from the **host** (adjust path and options to match your dump): + +```bash +pg_restore -h 127.0.0.1 -p 5433 -U db -d db --no-owner --no-acl --verbose /path/to/your_backup.dump +``` + +You may need to drop/recreate the target database or use `--clean` depending on whether `db` is empty; follow your usual migration practice. + +**`ddev import-db18`** is for **plain SQL** (optionally **`.gz`**). Use it when you have a `.sql` or `.sql.gz` file, for example: + +```bash +ddev import-db18 ~/exports/dump.sql +ddev import-db18 ~/exports/dump.sql.gz +``` + +### 5. Validate PG14 vs PG18 (optional but recommended) + +With **both** PG14 (`5432`) and PG18 (`5433`) running: + +```bash +ddev validate-pg18-migration +``` + +Review extensions, schema/table counts, and any warnings (for example `pg_trgm` reindex reminders). + +### 6. Point the application at PostgreSQL 18 + +```bash +ddev switch-pg-version 18 +``` + +### 7. Clear cache and restart + +As printed by the switch command: + +```bash +ddev exec 'rm -rf site/runtime/cache/*' +ddev restart +``` + +Then verify the application against PG18. + +--- + +## Switching back to PostgreSQL 14 + +When the eight versioned files exist and PG14 is still available: + +```bash +ddev switch-pg-version 14 +ddev exec 'rm -rf site/runtime/cache/*' +ddev restart +``` + +--- + +## Command reference + +| Command | Purpose | +|---------|---------| +| `ddev backup-pg14 [file]` | Custom-format `pg_dump` of PG14 database `db` on `localhost:5432`. | +| `ddev import-db18 ` | Import **plain SQL** (or **gzip** SQL) into PG18 on `127.0.0.1:5433`. | +| `ddev validate-pg18-migration` | Compare PG14 and PG18 (connections, versions, extensions, rough counts). | +| `ddev switch-pg-version 14\|18` | Copy `*-local-pg14.php` or `*-local-pg18.php` into `*-local.php` for `db_write`, `db_read`, `db_read_report`, and `common`. | + +--- + +## Troubleshooting + +- **`switch-pg-version` warns about missing files:** create the missing `site/config/*-local-pg14.php` or `*-local-pg18.php` files (see table above). +- **Cannot connect to PG18:** confirm `postgres18` is running, port **`5433`** is not blocked, and PG18 credentials in your `*-local-pg18.php` files match the compose file. +- **Restore errors after `backup-pg14`:** use `pg_restore` for `.dump` files; `import-db18` does not read custom-format dumps.