Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
582 changes: 582 additions & 0 deletions POSTGRESQL_18_CHANGES.md

Large diffs are not rendered by default.

145 changes: 145 additions & 0 deletions commands/PG-SWITCH-README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
# PostgreSQL 14 ↔ 18 switch (DDEV host commands)

This guide describes how to use the host commands under `.ddev/commands/host/` to back up PostgreSQL 14, load data into the PostgreSQL 18 sidecar, validate, and point the app at PG14 or PG18 via local config files.

## Prerequisites

- DDEV is installed and the project starts successfully (`ddev start`).
- The **PostgreSQL 18** service is enabled (for example via `.ddev/docker-compose.postgres18.yaml`) so that PG18 listens on **host port `5433`** (mapped to `5432` in the `postgres18` container). The main DDEV database remains **PostgreSQL 14 on port `5432`** unless you change the project database image.
- You have created the **versioned local config files** described below (they are not generated automatically).

## Configuration files you must create manually

The command `ddev switch-pg-version` does not edit connection strings inline. It expects **pairs** of version-specific files under **`site/config/`** (paths are relative to the **project root**, parent of `.ddev`).

For each **base name** below, you need **two** files: one for PG14 and one for PG18. The command copies the chosen file onto the active `*-local.php` name the app already uses.

| Base name | Active file (used by the app) | PG14 source | PG18 source |
|-----------|-------------------------------|-------------|-------------|
| `db_write` | `site/config/db_write-local.php` | `site/config/db_write-local-pg14.php` | `site/config/db_write-local-pg18.php` |
| `db_read` | `site/config/db_read-local.php` | `site/config/db_read-local-pg14.php` | `site/config/db_read-local-pg18.php` |
| `db_read_report` | `site/config/db_read_report-local.php` | `site/config/db_read_report-local-pg14.php` | `site/config/db_read_report-local-pg18.php` |
| `common` | `site/config/common-local.php` | `site/config/common-local-pg14.php` | `site/config/common-local-pg18.php` |

**What to put in each file**

- **`db_write-local-pg14.php`**, **`db_read-local-pg14.php`**, **`db_read_report-local-pg14.php`**: same connection settings you use today for the **main** DDEV database (**host** `db` or `127.0.0.1` from the host as appropriate for your stack, **port `5432`**, user/password/database as in your current setup).
- **`db_write-local-pg18.php`**, **`db_read-local-pg18.php`**, **`db_read_report-local-pg18.php`**: point reads/writes to the **PG18 sidecar** from the app’s perspective. Typically from **inside the web container** you use host **`postgres18`** (Docker network alias) and port **`5432`**; from the **host** for CLI tools, use **`127.0.0.1`** and port **`5433`**. Adjust user, password, and database name to match `.ddev/docker-compose.postgres18.yaml` (defaults are often `db` / `db` / `db`).
- **`common-local-pg14.php`** / **`common-local-pg18.php`**: mirror whatever you keep in `common-local.php` today (cache, queues, etc.), with only the differences required for each database version if any.

**Minimum checklist:** all **eight** versioned files must exist (`*-local-pg14.php` and `*-local-pg18.php` for the four bases). If either file in a pair is missing, `switch-pg-version` prints a warning and skips that base safely.

**Tip:** Copy your current working `*-local.php` files to `*-local-pg14.php`, then duplicate and edit copies to `*-local-pg18.php` with PG18 host/port/credentials.

---

## Suggested sequence (first-time migration PG14 → PG18)

Follow this order once you are ready to load PG18 and switch the app.

### 1. Stay on PostgreSQL 14 in the app

Ensure `switch-pg-version` has not been run to PG18 yet, or run:

```bash
ddev switch-pg-version 14
```

Confirm the app still uses PG14 (`*-local.php` content should match PG14 sources after a successful switch).

### 2. Back up the PG14 database

With DDEV running and PG14 reachable on **`localhost:5432`**:

```bash
ddev backup-pg14
```

Optional: pass a path for the dump file:

```bash
ddev backup-pg14 ~/backups/my_project_pg14.dump
```

This produces a **custom-format** `pg_dump` (`.dump`), not plain SQL.

### 3. Ensure PostgreSQL 18 is running

Start the stack so the `postgres18` service is up and **`127.0.0.1:5433`** accepts connections (see your `docker-compose.postgres18.yaml`).

### 4. Restore the backup into PostgreSQL 18

Because the backup from step 2 is **custom format**, restore with **`pg_restore`**, not `ddev import-db18`.

Example from the **host** (adjust path and options to match your dump):

```bash
pg_restore -h 127.0.0.1 -p 5433 -U db -d db --no-owner --no-acl --verbose /path/to/your_backup.dump
```

You may need to drop/recreate the target database or use `--clean` depending on whether `db` is empty; follow your usual migration practice.

**`ddev import-db18`** is for **plain SQL** (optionally **`.gz`**). Use it when you have a `.sql` or `.sql.gz` file, for example:

```bash
ddev import-db18 ~/exports/dump.sql
ddev import-db18 ~/exports/dump.sql.gz
```

### 5. Validate PG14 vs PG18 (optional but recommended)

With **both** PG14 (`5432`) and PG18 (`5433`) running:

```bash
ddev validate-pg18-migration
```

Review extensions, schema/table counts, and any warnings (for example `pg_trgm` reindex reminders).

### 6. Point the application at PostgreSQL 18

```bash
ddev switch-pg-version 18
```

### 7. Clear cache and restart

As printed by the switch command:

```bash
ddev exec 'rm -rf site/runtime/cache/*'
ddev restart
```

Then verify the application against PG18.

---

## Switching back to PostgreSQL 14

When the eight versioned files exist and PG14 is still available:

```bash
ddev switch-pg-version 14
ddev exec 'rm -rf site/runtime/cache/*'
ddev restart
```

---

## Command reference

| Command | Purpose |
|---------|---------|
| `ddev backup-pg14 [file]` | Custom-format `pg_dump` of PG14 database `db` on `localhost:5432`. |
| `ddev import-db18 <file>` | Import **plain SQL** (or **gzip** SQL) into PG18 on `127.0.0.1:5433`. |
| `ddev validate-pg18-migration` | Compare PG14 and PG18 (connections, versions, extensions, rough counts). |
| `ddev switch-pg-version 14\|18` | Copy `*-local-pg14.php` or `*-local-pg18.php` into `*-local.php` for `db_write`, `db_read`, `db_read_report`, and `common`. |

---

## Troubleshooting

- **`switch-pg-version` warns about missing files:** create the missing `site/config/*-local-pg14.php` or `*-local-pg18.php` files (see table above).
- **Cannot connect to PG18:** confirm `postgres18` is running, port **`5433`** is not blocked, and PG18 credentials in your `*-local-pg18.php` files match the compose file.
- **Restore errors after `backup-pg14`:** use `pg_restore` for `.dump` files; `import-db18` does not read custom-format dumps.
44 changes: 44 additions & 0 deletions commands/host/backup-pg14
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
#!/bin/bash

## Description: Backup PostgreSQL 14 database
## Usage: backup-pg14 [output_file]
## Example: backup-pg14 ~/backups/fdsu_pg14.dump

OUTPUT_FILE="${1:-$HOME/.ddev/backups/pg14/fdsu_pg14_backup_$(date +%Y%m%d_%H%M%S).dump}"
LOG_DIR="$HOME/.ddev/logs"
LOG_FILE="$LOG_DIR/backup-pg14-$(date +%Y%m%d-%H%M%S).log"

mkdir -p "$(dirname "$OUTPUT_FILE")"
mkdir -p "$LOG_DIR"

echo "Backing up PostgreSQL 14 database..." | tee -a "$LOG_FILE"
echo "Output: $OUTPUT_FILE" | tee -a "$LOG_FILE"
echo "Log: $LOG_FILE" | tee -a "$LOG_FILE"

# Get database size first
DB_SIZE=$(PGPASSWORD=db psql -h localhost -p 5432 -U db -d db -t -c "SELECT pg_size_pretty(pg_database_size('db'));" | xargs)
echo "Database size: $DB_SIZE" | tee -a "$LOG_FILE"

# Create backup
pg_dump -h localhost -p 5432 -U db -d db \
--format=custom \
--verbose \
--no-owner \
--no-acl \
--file="$OUTPUT_FILE" \
2>&1 | tee -a "$LOG_FILE"

EXIT_CODE=$?
Comment on lines +23 to +31

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Quick static check for pipeline exit-code handling in this script.
rg -n -C2 'pg_dump|PIPESTATUS|set -o pipefail|EXIT_CODE=\$\?' commands/host/backup-pg14

Repository: sandeepfd/fdsu_ddev

Length of output: 268


🏁 Script executed:

cat -n commands/host/backup-pg14

Repository: sandeepfd/fdsu_ddev

Length of output: 1754


Fix pipeline exit-code capture for pg_dump command.

The current code captures tee's exit code (via EXIT_CODE=$?), not pg_dump's. A failed backup can be silently misreported as successful while the script exits with code 0.

Proposed fix
 #!/bin/bash
+set -o pipefail
@@
 pg_dump -h localhost -p 5432 -U db -d db \
@@
     --file="$OUTPUT_FILE" \
     2>&1 | tee -a "$LOG_FILE"
 
-EXIT_CODE=$?
+EXIT_CODE=${PIPESTATUS[0]}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@commands/host/backup-pg14` around lines 23 - 31, The pipeline currently
assigns EXIT_CODE=$? which captures tee's exit status, not pg_dump's; after
running the pg_dump ... 2>&1 | tee -a "$LOG_FILE" pipeline, set EXIT_CODE from
the bash PIPESTATUS array (e.g. EXIT_CODE=${PIPESTATUS[0]}) so you record
pg_dump's exit code (ensure the script runs under bash where PIPESTATUS exists);
reference the pg_dump invocation and the variables OUTPUT_FILE, LOG_FILE and
EXIT_CODE when making this change.


if [ $EXIT_CODE -eq 0 ]; then
BACKUP_SIZE=$(ls -lh "$OUTPUT_FILE" | awk '{print $5}')
echo "✓ Backup completed successfully" | tee -a "$LOG_FILE"
echo " File: $OUTPUT_FILE" | tee -a "$LOG_FILE"
echo " Size: $BACKUP_SIZE" | tee -a "$LOG_FILE"
ls -lh "$OUTPUT_FILE"
else
echo "✗ Backup failed with exit code: $EXIT_CODE" | tee -a "$LOG_FILE"
echo "Check log file: $LOG_FILE" | tee -a "$LOG_FILE"
exit 1
fi

34 changes: 34 additions & 0 deletions commands/host/import-db18
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/bin/bash

## Description: Import database dump into PostgreSQL 18
## Usage: import-db18 [file]
## Example: import-db18 ~/Downloads/backup.sql.gz

if [ -z "$1" ]; then
echo "Usage: ddev import-db18 <file>"
echo "Example: ddev import-db18 ~/Downloads/backup.sql.gz"
exit 1
fi

FILE="$1"

if [ ! -f "$FILE" ]; then
echo "Error: File '$FILE' not found"
exit 1
fi

echo "Importing $FILE into PostgreSQL 18..."

if [[ "$FILE" == *.gz ]]; then
gunzip -c "$FILE" | PGPASSWORD=db psql -h 127.0.0.1 -p 5433 -U db -d db
else
PGPASSWORD=db psql -h 127.0.0.1 -p 5433 -U db -d db < "$FILE"
fi

if [ $? -eq 0 ]; then
echo "✓ Successfully imported into PostgreSQL 18"
else
echo "✗ Import failed"
exit 1
fi
Comment on lines +22 to +33

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Static verification of import error-handling flags/status capture.
rg -n -C2 'ON_ERROR_STOP|pipefail|PIPESTATUS|if \[ \$\? -eq 0 \]' commands/host/import-db18

Repository: sandeepfd/fdsu_ddev

Length of output: 162


🏁 Script executed:

cat -n commands/host/import-db18

Repository: sandeepfd/fdsu_ddev

Length of output: 1034


Import can report success on partial/failed SQL execution.

The piped gunzip command lacks pipefail error propagation, and both psql invocations lack ON_ERROR_STOP=1, allowing SQL errors and decompression failures to be silently ignored. Line 28 uses $? which only captures the exit code of psql in the pipe, not gunzip.

Proposed fix
 #!/bin/bash
+set -o pipefail
 
 ## Description: Import database dump into PostgreSQL 18
 ## Usage: import-db18 [file]
@@ -20,14 +21,18 @@
 echo "Importing $FILE into PostgreSQL 18..."
 
 if [[ "$FILE" == *.gz ]]; then
-    gunzip -c "$FILE" | PGPASSWORD=db psql -h 127.0.0.1 -p 5433 -U db -d db
+    gunzip -c "$FILE" | PGPASSWORD=db psql -v ON_ERROR_STOP=1 -h 127.0.0.1 -p 5433 -U db -d db
+    IMPORT_EXIT=${PIPESTATUS[1]}
 else
-    PGPASSWORD=db psql -h 127.0.0.1 -p 5433 -U db -d db < "$FILE"
+    PGPASSWORD=db psql -v ON_ERROR_STOP=1 -h 127.0.0.1 -p 5433 -U db -d db < "$FILE"
+    IMPORT_EXIT=$?
 fi
 
-if [ $? -eq 0 ]; then
+if [ $IMPORT_EXIT -eq 0 ]; then
     echo "✓ Successfully imported into PostgreSQL 18"
 else
     echo "✗ Import failed"
     exit 1
 fi
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@commands/host/import-db18` around lines 22 - 33, The script can report
success despite decompression or SQL errors because piped commands don't
propagate failures and psql isn't set to stop on SQL errors; enable bash's
pipefail (set -o pipefail) and export or pass ON_ERROR_STOP=1 to psql
invocations so any SQL error causes non-zero exit, and replace the current "$?"
check with a direct test of the full pipeline/command exit status (e.g. check
"$?" immediately after the command or use if ...; then) for both the gunzip |
psql branch and the direct psql branch to ensure failures in gunzip or psql are
detected; update references to FILE, gunzip, and psql invocation to include
ON_ERROR_STOP=1 and rely on pipefail for the piped case.


82 changes: 82 additions & 0 deletions commands/host/switch-pg-version
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
#!/bin/bash
## Description: Switch database connection between PostgreSQL 14 and 18 by renaming config files
## Usage: switch-pg-version [14|18]
## Example: switch-pg-version 18

VERSION=${1:-14}

if [ "$VERSION" != "14" ] && [ "$VERSION" != "18" ]; then
echo "Error: Version must be 14 or 18"
echo "Usage: ddev switch-pg-version [14|18]"
exit 1
fi

CONFIG_DIR="site/config"

if [ "$VERSION" == "18" ]; then
echo "Switching to PostgreSQL 18..."
PG_VERSION="pg18"
else
echo "Switching to PostgreSQL 14..."
PG_VERSION="pg14"
fi

# Function to switch a config file
switch_config() {
local base_name=$1
local active_file="$CONFIG_DIR/${base_name}-local.php"
local pg14_file="$CONFIG_DIR/${base_name}-local-pg14.php"
local pg18_file="$CONFIG_DIR/${base_name}-local-pg18.php"

# Check if versioned files exist
if [ ! -f "$pg14_file" ] || [ ! -f "$pg18_file" ]; then
echo "⚠ Warning: Versioned files not found for $base_name"
echo " Expected: $pg14_file"
echo " Expected: $pg18_file"
return 1
fi

# Backup current active file if it exists and is not a versioned file
if [ -f "$active_file" ] && [ ! -L "$active_file" ]; then
# Check if it's not already a versioned file
if [[ "$active_file" != *"-pg14.php" ]] && [[ "$active_file" != *"-pg18.php" ]]; then
mv "$active_file" "${active_file}.backup.$(date +%Y%m%d_%H%M%S)"
echo " Backed up existing $base_name-local.php"
fi
fi

# Remove existing active file (if it's a symlink or regular file)
[ -f "$active_file" ] && rm -f "$active_file"
[ -L "$active_file" ] && rm -f "$active_file"

# Copy the appropriate versioned file to the active file
if [ "$VERSION" == "18" ]; then
cp "$pg18_file" "$active_file"
else
cp "$pg14_file" "$active_file"
fi
Comment on lines +43 to +57

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Non-atomic replacement can leave configs missing on copy failure.

On Line 43 and Lines 49-57, the current flow removes the active file before confirming the new file is safely staged. If cp fails, the app can be left without an active config.

Suggested safer replacement flow
-            mv "$active_file" "${active_file}.backup.$(date +%Y%m%d_%H%M%S)"
+            cp "$active_file" "${active_file}.backup.$(date +%Y%m%d_%H%M%S)"
             echo "  Backed up existing $base_name-local.php"
         fi
     fi
-    
-    # Remove existing active file (if it's a symlink or regular file)
-    [ -f "$active_file" ] && rm -f "$active_file"
-    [ -L "$active_file" ] && rm -f "$active_file"
-    
-    # Copy the appropriate versioned file to the active file
-    if [ "$VERSION" == "18" ]; then
-        cp "$pg18_file" "$active_file"
-    else
-        cp "$pg14_file" "$active_file"
-    fi
+
+    local source_file tmp_file
+    if [ "$VERSION" == "18" ]; then
+        source_file="$pg18_file"
+    else
+        source_file="$pg14_file"
+    fi
+    tmp_file="${active_file}.tmp.$$"
+
+    if ! cp "$source_file" "$tmp_file"; then
+        echo "✗ Failed to stage $source_file"
+        rm -f "$tmp_file"
+        return 1
+    fi
+    if ! mv -f "$tmp_file" "$active_file"; then
+        echo "✗ Failed to activate $active_file"
+        rm -f "$tmp_file"
+        return 1
+    fi
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@commands/host/switch-pg-version` around lines 43 - 57, The replacement flow
currently deletes $active_file before confirming the new config was staged,
risking a missing active config if cp fails; instead, stage the selected source
($pg18_file or $pg14_file based on $VERSION) into a temporary file (e.g., using
mktemp), verify the copy succeeded, then atomically move (mv) the temp into
place to replace $active_file and remove the old symlink/file only as part of
the atomic rename; ensure the backup step for $active_file still runs before
staging, and update the logic around [ -f "$active_file" ] / [ -L "$active_file"
] so they are not removed until after the temp-to-active mv succeeds.


echo "✓ Switched $base_name-local.php to PostgreSQL $VERSION"
return 0
}

# Switch all config files
echo ""
switch_config "db_write"
switch_config "db_read"
switch_config "db_read_report"
switch_config "common"

echo ""
echo "✓ Successfully switched to PostgreSQL $VERSION"
echo ""
echo "Current active configuration:"
echo " - db_write-local.php → PostgreSQL $VERSION"
echo " - db_read-local.php → PostgreSQL $VERSION"
echo " - db_read_report-local.php → PostgreSQL $VERSION"
echo " - common-local.php → PostgreSQL $VERSION"
Comment on lines +65 to +77

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Final success summary is printed even when one or more switches fail.

switch_config return codes are ignored on Lines 65-68, then Line 71 reports success unconditionally. This can mislead operators during partial failures.

Track failures and exit non-zero on partial/total failure
-# Switch all config files
-echo ""
-switch_config "db_write"
-switch_config "db_read"
-switch_config "db_read_report"
-switch_config "common"
+echo ""
+failures=0
+for base in db_write db_read db_read_report common; do
+    if ! switch_config "$base"; then
+        failures=$((failures + 1))
+    fi
+done
+
+if [ "$failures" -gt 0 ]; then
+    echo ""
+    echo "✗ Switch completed with $failures failure(s)"
+    exit 1
+fi
 
 echo ""
 echo "✓ Successfully switched to PostgreSQL $VERSION"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@commands/host/switch-pg-version` around lines 65 - 77, The success summary is
printed unconditionally even if some switch_config calls fail; update the block
that calls switch_config "db_write", "db_read", "db_read_report", and "common"
to check each call's return code, accumulate a failure flag (or count), and only
print the "✓ Successfully switched to PostgreSQL $VERSION" and the success list
when all calls succeed; if any fail, print a clear error/partial-failure summary
and exit with a non-zero status (e.g., exit 1) instead of the current
unconditional echo block.

echo ""
echo "Next steps:"
echo " 1. Clear cache: ddev exec 'rm -rf site/runtime/cache/*'"
echo " 2. Restart DDEV: ddev restart"
echo " 3. Verify connection by checking your application"
Loading