Non-transactional migrations, callbacks, and superseded repeatables

README.md

@@ -124,6 +124,28 @@ Every command is a method on the loaded `*goway.Flyway` value and takes a
 | `Repair`   | Removes failed entries and realigns recorded checksums.                 |
 | `Clean`    | Drops every object in the managed schemas. Disabled by default.         |
 
+### Callbacks
+
+Register lifecycle callbacks either as SQL scripts placed in the configured
+locations (`beforeMigrate.sql`, `afterMigrate.sql`, `beforeEachMigrate.sql`,
+`afterEachMigrate__description.sql`) or programmatically through
+`Configure().Callbacks(...)` with a value implementing `Callback`, or the
+`CallbackFunc` adapter.
+
+### Non-transactional migrations
+
+A migration that cannot run inside a transaction, such as one using PostgreSQL's
+`CREATE INDEX CONCURRENTLY` or SQLite's `VACUUM`, opts out of the per-migration
+transaction with a directive on the first lines of the script:
+
+```sql
+-- goway:noTransaction
+CREATE INDEX CONCURRENTLY idx_users_email ON users (email);
+```
+
+The statements then run directly on a dedicated connection and the history row
+is still recorded.
+
 ## Command line tool
 
 A command line front end lives in the `cmd/goway` module.
@@ -171,12 +193,12 @@ go -C integration test ./... -count=1
 
 Implemented: versioned and repeatable SQL migrations, the schema history table,
 migrate, info, validate, baseline, repair and clean, placeholder replacement,
-multiple locations, embedded file systems, schema creation, and a command line
-tool.
+multiple locations, embedded file systems, schema creation, lifecycle callbacks
+(SQL scripts and programmatic), per-script non-transactional execution, the
+superseded state for repeatable migrations, and a command line tool.
 
-Not yet implemented: Java style code based migrations, lifecycle callbacks, undo
-migrations, and listing every historical run of a repeatable migration (only the
-latest run is reported).
+Not yet implemented: Go code based migrations, undo migrations, and the grouped
+and mixed transaction modes.
 
 ## Acknowledgements and License
 

callback.go

@@ -0,0 +1,83 @@
+package goway
+
+import (
+	"context"
+	"database/sql"
+	"strings"
+)
+
+// CallbackEvent identifies a point in the migrate lifecycle at which callbacks
+// are invoked. The values match the corresponding Flyway event names.
+type CallbackEvent string
+
+const (
+	// EventBeforeMigrate fires once before any migration is applied.
+	EventBeforeMigrate CallbackEvent = "beforeMigrate"
+
+	// EventAfterMigrate fires once after all migrations have been applied.
+	EventAfterMigrate CallbackEvent = "afterMigrate"
+
+	// EventBeforeEachMigrate fires before each individual migration, inside the
+	// migration's transaction when one is used.
+	EventBeforeEachMigrate CallbackEvent = "beforeEachMigrate"
+
+	// EventAfterEachMigrate fires after each individual migration, inside the
+	// migration's transaction when one is used.
+	EventAfterEachMigrate CallbackEvent = "afterEachMigrate"
+)
+
+// Execer is the minimal interface needed to run a statement. It is satisfied by
+// *sql.DB, *sql.Tx and *sql.Conn, so a callback can execute SQL on whichever
+// handle is active for the current event.
+type Execer interface {
+	ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error)
+}
+
+// Callback receives lifecycle events during a migrate run. For the per-migration
+// events the migration argument describes the migration being processed; it is
+// nil for the run level events.
+type Callback interface {
+	Handle(ctx context.Context, event CallbackEvent, exec Execer, migration *MigrationInfo) error
+}
+
+// CallbackFunc adapts an ordinary function to the Callback interface.
+type CallbackFunc func(ctx context.Context, event CallbackEvent, exec Execer, migration *MigrationInfo) error
+
+// Handle calls the underlying function.
+func (f CallbackFunc) Handle(ctx context.Context, event CallbackEvent, exec Execer, migration *MigrationInfo) error {
+	return f(ctx, event, exec, migration)
+}
+
+// callbackEventsByName maps the lower cased event name to its canonical value.
+var callbackEventsByName = map[string]CallbackEvent{
+	"beforemigrate":     EventBeforeMigrate,
+	"aftermigrate":      EventAfterMigrate,
+	"beforeeachmigrate": EventBeforeEachMigrate,
+	"aftereachmigrate":  EventAfterEachMigrate,
+}
+
+// sqlCallback is a callback backed by a SQL script discovered in the configured
+// locations.
+type sqlCallback struct {
+	event  CallbackEvent
+	script string
+	read   func() ([]byte, error)
+}
+
+// parseCallbackName reports whether a script file name denotes a callback and,
+// if so, which event it handles. The name without its suffix must equal an event
+// name, optionally followed by the separator and a description, for example
+// "afterEachMigrate__seed.sql". Matching is case insensitive.
+func parseCallbackName(fileName, separator string, suffixes []string) (CallbackEvent, bool) {
+	suffix := matchSuffix(fileName, suffixes)
+	if suffix == "" {
+		return "", false
+	}
+	stem := fileName[:len(fileName)-len(suffix)]
+	name := stem
+	if index := strings.Index(stem, separator); index >= 0 {
+		name = stem[:index]
+	}
+	event, ok := callbackEventsByName[strings.ToLower(name)]
+	return event, ok
+}

callback_test.go

@@ -0,0 +1,52 @@
+package goway
+
+import (
+	"context"
+	"testing"
+)
+
+func TestParseCallbackName(t *testing.T) {
+	suffixes := []string{".sql"}
+	recognized := map[string]CallbackEvent{
+		"beforeMigrate.sql":          EventBeforeMigrate,
+		"afterMigrate.sql":           EventAfterMigrate,
+		"beforeEachMigrate.sql":      EventBeforeEachMigrate,
+		"afterEachMigrate__seed.sql": EventAfterEachMigrate,
+		"AFTERMIGRATE.sql":           EventAfterMigrate,
+	}
+	for name, want := range recognized {
+		got, ok := parseCallbackName(name, "__", suffixes)
+		if !ok || got != want {
+			t.Errorf("parseCallbackName(%q) = (%q, %v), want (%q, true)", name, got, ok, want)
+		}
+	}
+
+	rejected := []string{
+		"V1__create.sql",      // versioned migration
+		"R__view.sql",         // repeatable migration
+		"random.sql",          // unknown stem
+		"beforeMigrate.txt",   // wrong suffix
+		"afterMigrateNow.sql", // stem is not exactly an event name
+	}
+	for _, name := range rejected {
+		if _, ok := parseCallbackName(name, "__", suffixes); ok {
+			t.Errorf("parseCallbackName(%q) was recognized, want rejected", name)
+		}
+	}
+}
+
+// TestCallbackFuncImplementsCallback verifies the function adapter satisfies the
+// Callback interface and forwards its arguments.
+func TestCallbackFuncImplementsCallback(t *testing.T) {
+	var seen CallbackEvent
+	var callback Callback = CallbackFunc(func(_ context.Context, event CallbackEvent, _ Execer, _ *MigrationInfo) error {
+		seen = event
+		return nil
+	})
+	if err := callback.Handle(context.Background(), EventBeforeMigrate, nil, nil); err != nil {
+		t.Fatalf("Handle returned error: %v", err)
+	}
+	if seen != EventBeforeMigrate {
+		t.Errorf("callback saw event %q, want %q", seen, EventBeforeMigrate)
+	}
+}

config.go

@@ -49,6 +49,8 @@ type Configuration struct {
 	target            *Version
 	installedBy       string
 
+	callbacks []Callback
+
 	configErr error
 }
 
@@ -228,6 +230,13 @@ func (c *Configuration) Target(version string) *Configuration {
 	return c
 }
 
+// Callbacks registers programmatic callbacks invoked during a migrate run, in
+// addition to any SQL callback scripts found in the configured locations.
+func (c *Configuration) Callbacks(callbacks ...Callback) *Configuration {
+	c.callbacks = append(c.callbacks, callbacks...)
+	return c
+}
+
 // InstalledBy overrides the user recorded for applied migrations.
 func (c *Configuration) InstalledBy(user string) *Configuration {
 	c.installedBy = user
@@ -275,7 +284,7 @@ func (c *Configuration) LoadContext(ctx context.Context) (*Migrator, error) {
 		dialect = detected
 	}
 
-	resolved, err := resolveMigrations(c)
+	resolved, callbacks, err := resolveMigrations(c)
 	if err != nil {
 		return nil, err
 	}
@@ -284,5 +293,6 @@ func (c *Configuration) LoadContext(ctx context.Context) (*Migrator, error) {
 		configuration: c,
 		dialect:       dialect,
 		resolved:      resolved,
+		sqlCallbacks:  callbacks,
 	}, nil
 }

dialect.go

@@ -79,6 +79,12 @@ type Dialect interface {
 	// string when the dialect has no such concept.
 	setSearchPathSQL(schema string) string
 
+	// sessionSearchPathSQL returns a statement that makes the given schema the
+	// default for the whole session rather than a single transaction, for use by
+	// migrations that run without a transaction. It returns an empty string when
+	// the dialect has no such concept.
+	sessionSearchPathSQL(schema string) string
+
 	// cleanStatements returns the statements that drop every object in the given
 	// schema, querying the database when the set of objects must be discovered
 	// dynamically.

dialect_postgres.go

@@ -95,6 +95,13 @@ func (d postgresDialect) setSearchPathSQL(schema string) string {
 	return "SET LOCAL search_path TO " + d.quoteIdentifier(schema)
 }
 
+func (d postgresDialect) sessionSearchPathSQL(schema string) string {
+	if schema == "" {
+		return ""
+	}
+	return "SET search_path TO " + d.quoteIdentifier(schema)
+}
+
 // cleanStatements drops the schema and recreates it, which removes every object
 // it contains. This is simpler and more robust than enumerating each object.
 func (d postgresDialect) cleanStatements(_ context.Context, _ querier, schema string) ([]string, error) {

dialect_sqlite.go

@@ -78,6 +78,8 @@ func (sqliteDialect) splitStatements(sql string) ([]string, error) {
 
 func (sqliteDialect) setSearchPathSQL(string) string { return "" }
 
+func (sqliteDialect) sessionSearchPathSQL(string) string { return "" }
+
 // cleanStatements enumerates the user defined objects from the SQLite catalog
 // and returns statements to drop each of them. Internal objects whose names
 // begin with the reserved prefix are skipped.

docs/design.md

@@ -118,14 +118,28 @@ splitter also recognizes backtick and square bracket identifiers and tracks
 `BEGIN`, `CASE` and `END` so the inner statements of a trigger body are kept
 together.
 
+## Callbacks and non-transactional migrations
+
+Lifecycle callbacks fire around the migrate run (`beforeMigrate`,
+`afterMigrate`) and around each applied migration (`beforeEachMigrate`,
+`afterEachMigrate`). They are discovered as SQL scripts in the configured
+locations by their file name, or registered programmatically through the
+`Callback` interface. The per-migration callbacks run on the same executor as
+the migration, so within its transaction when one is used.
+
+A migration whose first comment lines carry a `-- goway:noTransaction` directive
+runs outside the per-migration transaction, on a dedicated connection, for
+statements such as PostgreSQL's `CREATE INDEX CONCURRENTLY` or SQLite's
+`VACUUM`. Because there is no transaction to roll back, a failure is recorded as
+a failed history row that the repair command can clear.
+
 ## Known divergences
 
-- Only the latest run of a repeatable migration is reported by `Info`; Flyway
-  lists every historical run and marks the superseded ones.
 - Placeholder checksums are always computed on the raw content; Flyway computes
   them on the replaced content for repeatable migrations.
-- Code based migrations, lifecycle callbacks, and undo migrations are not
-  implemented.
+- Go code based migrations and undo migrations are not implemented.
+- The grouped and mixed transaction modes are not implemented; each migration
+  runs in its own transaction unless it opts out.
 
 ## Naming and trademark