Skip to content

Implement Batch Processing for Sync Operations #9

@caleb-vanlue

Description

@caleb-vanlue

The sync services process releases sequentially in for loops, causing sync operations to take 10-100x longer than necessary. In DiscogsSyncService.syncUserCollection(), syncUserWantlist(), and syncUserSuggestions(), each release is fetched from Discogs and saved to the database one by one. This creates poor user experience during sync operations and unnecessarily long database connection times.

Implement batch processing using Promise.all() for groups of releases with bulk database operations. Process releases in chunks of 10-20 items to balance memory usage and performance. Use TypeORM's save() method with arrays for efficient bulk inserts. Add progress tracking to provide feedback during long sync operations.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions