Skip to content

Conversation

@pyramation
Copy link
Contributor

@pyramation pyramation commented Jan 8, 2026

feat(pgpm): add slice command for modularizing migration plans

Summary

Adds a new pgpm slice command that partitions a large monolithic migration plan into multiple modular packages. This addresses the maintainability challenge of projects with thousands of migrations in a single package.

The implementation supports two grouping strategies:

  1. Folder-based strategy (default): Groups changes by folder depth in the path (e.g., schemas/auth/tables/usersauth package)
  2. Pattern-based strategy: Groups changes using glob patterns defined in a JSON file (e.g., schemas/*_auth_*/**auth package)

Cross-package dependencies are expressed using package:change syntax, and the algorithm computes a valid deployment order via topological sort.

Key components:

  • Core slicing logic (pgpm/core/src/slice/): Dependency graph building, DAG validation, package assignment, cycle detection, pattern matching via minimatch
  • CLI command (pgpm slice): Supports --strategy, --patterns, --dry-run, configurable depth/prefix, and minimum package size
  • Integration test (pgpm/core/__tests__/slice/slice-deploy-integration.test.ts): End-to-end test that slices plans and deploys packages to a real database
  • Design spec (docs/spec/SLICING.md): Comprehensive documentation of the algorithm and edge cases

Updates since last revision

  • Moved spec file from pgpm/MIGRATION_SLICING_SPEC.md to docs/spec/SLICING.md to follow repo conventions
  • All CI checks passing (50/50)

Review & Testing Checklist for Human

  • Test with a real plan file: Run pgpm slice --dry-run against an actual large plan file to verify grouping and dependency detection work correctly - unit tests may not catch real-world edge cases
  • Verify cross-package dependency syntax: The code generates package:change format for cross-package deps - confirm this matches PGPM's expected resolution format
  • Test pattern-based slicing: Create a JSON pattern file and run pgpm slice --strategy pattern --patterns ./slices.json --dry-run to verify glob matching works correctly
  • Validate generated plan files: Ensure output plan files can be parsed back by parsePlanFile() without errors
  • Review cycle handling: The algorithm warns but proceeds when package cycles are detected - verify this is acceptable behavior

Recommended Test Plan

  1. Test folder-based strategy:
    pgpm slice --dry-run --plan <path>
  2. Test pattern-based strategy with a pattern file:
    {
      "slices": [
        { "packageName": "auth", "patterns": ["schemas/*_auth_*/**"] },
        { "packageName": "users", "patterns": ["schemas/*_users_*/**"] }
      ]
    }
    pgpm slice --strategy pattern --patterns ./slices.json --dry-run
  3. Run without --dry-run and inspect the generated packages directory
  4. Verify each generated pgpm.plan file is valid and deployable

Notes

  • Unit tests and integration tests pass covering core slicing functions including pattern matching and actual deployment
  • The algorithm warns but proceeds when package cycles are detected (may want stricter handling)
  • extensions/* and migrate/* paths are automatically assigned to core package
  • Pattern matching uses first-match semantics (order of slices in JSON matters)

Devin run: https://app.devin.ai/sessions/84604c65fadd4983a065f855e6c82533
Previous session: https://app.devin.ai/sessions/e11be896d4dd45aaa746d1e2c42c186a
Requested by: @pyramation

- Add core slicing logic in pgpm/core/src/slice/ with path-based grouping
- Implement folder-based strategy to extract package names from change paths
- Add dependency graph building, validation, and topological sorting
- Generate cross-package dependencies using package:change syntax
- Add slice command to pgpm CLI with dry-run and configuration options
- Include comprehensive test suite with 18 passing tests
- Add detailed design specification document
@devin-ai-integration
Copy link
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

- Add PatternStrategy and PatternSlice types to slice/types.ts
- Implement findMatchingPattern function using minimatch for glob matching
- Update assignChangesToPackages to handle 'pattern' strategy type
- Update CLI to accept --strategy flag (folder or pattern)
- Add --patterns flag for JSON pattern file path
- Add conditional prompting (only ask for depth/prefix with folder strategy)
- Add tests for pattern-based slicing functionality
- Add minimatch dependency to pgpm/core

This allows users to choose between:
1. Folder-based strategy: depth-based extraction from path hierarchy
2. Pattern-based strategy: glob pattern matching for flexible grouping
This test verifies that sliced packages can actually be deployed to a database:
1. Creates a plan file with changes across multiple schemas
2. Slices it into packages using folder-based and pattern-based strategies
3. Writes the packages to disk
4. Deploys each package in the computed deploy order
5. Verifies the database state (schemas, tables, cross-package deps)

This ensures the slice output is valid and deployable, not just syntactically correct.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants