Add deal templates migration#522
Closed
Sankara-Jefferson wants to merge 27 commits intofeature/auto-prep-deal-schedulingfrom
Closed
Add deal templates migration#522Sankara-Jefferson wants to merge 27 commits intofeature/auto-prep-deal-schedulingfrom
Sankara-Jefferson wants to merge 27 commits intofeature/auto-prep-deal-schedulingfrom
Conversation
fixes for linter env and lint errors --------- Co-authored-by: Arkadiy Kukarkin <arkadiy@archive.org>
feat: allow configurable min-piece-size to reduce excessive padding for small segments This change introduces support for specifying a `--min-piece-size` when preparing data, improving behavior for small DAGs, remainder CARs, and small preparations that would otherwise be padded to the full target piece size (e.g. 32GiB). Such excessive padding leads to inefficiencies and causes sectors to be rejected by Storage Providers or fail verified deal requirements. ### Key changes: - Add support for `--min-piece-size` (default: 256B, subject to adjustment) - Pass both `min` and `target` piece sizes to `GetCommp`, enabling finer control over padding - Retain power-of-2 padding via `target size`, but allow flexibility by setting it to `0` This helps avoid generating 90%+ padding pieces and reduces transfer times in many cases. ### Notes: - Default behavior remains unchanged if `--min-piece-size` is not set - Full support for non-padded pieces now depends on both chunker accuracy and downstream deal acceptance - `pieceType` is now tracked in metadata (e.g., data vs. DAG) ### Out of scope: - No cross-preparation aggregation; that responsibility remains with SPs - Edge cases like aggregating under-1MiB pieces are not yet solved Closes #473 Co-authored-by: Arkadiy Kukarkin <arkadiy@archive.org>
Core auto-deal functionality including: - Onboard command for single-step data onboarding - Auto-deal trigger service - Unified service for managed workers - Workflow orchestrator for automatic job progression - Notification system for observability - Validation handlers for wallets and storage providers Fixes #495 This is a reopened version of #494 which was closed due to history rewrite issues.
- Add auto-deal configuration fields to Preparation model - Add Notification model for system observability - Integrate onboard command in main app - Update database migration tables
- Extend dataprep create handler with auto-deal parameters and validation - Update pack job handler to trigger auto-deals on completion - Add notification and validation systems integration
- Add workflow progression triggering in worker threads - Integrate with auto-deal trigger service - Add lotus client configuration for deal creation
- Add DEMO_AUTO_PREP_DEALS.md with complete demo script - Add auto-deal system documentation - Update CLI reference documentation - Update Swagger API documentation - Add comprehensive usage examples and troubleshooting
- Add detailed feature overview and quick start guide - Document the onboard command and auto-deal workflow - Include architecture diagrams and usage examples - Add troubleshooting and migration guides - Provide comprehensive configuration options
- Add auto-create-deals flag and all deal configuration options - Add wallet and provider validation flags - Add workflow automation flags (auto-start, auto-progress) - Implement automatic scanning and workflow orchestration
- Update AutoDealService to use schedule.Handler interface correctly - Fix method calls to match the actual schedule handler implementation - Ensure proper integration with existing schedule creation system
The version.json file is required by the Go embed directive in singularity.go but was not committed to the repository, causing CI Docker builds to fail with "pattern version.json: no matching files found".
Restores version.json after git history rewrite. Confirmed working locally. [skip ci] -Release-check workflow fails to locate go-version: 1.23.6.x -Safe to ignore for now
Updates version.json to reflect the new release tag v0.6.0-RC3.
Add Wallet Create Handler + Migrate to Numeric Wallet ID. Merging as all local test passed. The go version error will be addressed on a separate issue.
This commit includes automatically generated updates from go generate that reflect: - New DealConfig struct encapsulating deal parameters - Added --json flag to onboard command for automation support - Updated API documentation and client models Generated by: - swag init (Swagger documentation) - go-swagger generate client (API client models) - CLI documentation generator
- Fix unused import in dealconfig_test.go - Update autodeal trigger tests to use DealConfig struct - Remove unused strings import in downloadserver_test.go - Rename duplicate test function in testutil - Fix ApplyOverrides test logic for boolean fields
Collaborator
|
handles this migration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds a new GORM migration to create the
deal_templatestable with support forDealConfig-embedded fields. This supports the expanded preparation workflow introduced in PR-512.Why it's needed
PR-512 introduces support for referencing deal templates from preparations. This PR adds the migration to create the required table, so it’s split cleanly from PR-512 and easier to review/test independently.
Notes
pr-512as base to isolate schema-related changes.sqlite3DB bootstrap (./singularity admin migrate up).