📋 CheckList
🐛 Bug Description
I discovered a bug in the RunSnapshotItemJob method within item_snapshot.go. When a dataset snapshot is processed in multiple batches (e.g., 100 batches), the snapshot_progress field in the dataset_version table is only updated after the first batch.
For all subsequent batches (from batch 2 onwards), the snapshot_progress in the database remains stuck at the value of the first batch.
This behavior breaks the "resume from breakpoint" mechanism. The initProgressCtx method relies on snapshot_progress to determine where to resume a job. Since the database holds an outdated progress value (from the first batch), if a job fails mid-way (e.g., at batch 50) and restarts, it will incorrectly resume from the position recorded after batch 1, effectively rendering the checkpoint logic useless.
🔄 Steps to Reproduce
Prepare Data: Create or use a dataset with enough items to trigger the snapshot job to run in multiple batches (ensure the data volume exceeds the single-batch limit so that at least 2 batches are generated).
Trigger Action: Publish the dataset version to start the RunSnapshotItemJob.
Observation: Monitor the dataset_version table in the database while the job is running.
Result: You will consistently see that snapshot_progress is updated only after the first batch completes. It fails to update for all subsequent batches, regardless of how many batches follow.
✅ Expected Behavior
The snapshot_progress field in the dataset_version table should be updated after every batch is successfully processed, ensuring the checkpoint is always up-to-date.
❌ Actual Behavior
When a dataset snapshot is processed in multiple batches (e.g., 100 batches), the snapshot_progress field in the dataset_version table is only updated after the first batch.
🚨 Severity
Low - Cosmetic issue or minor inconvenience
🔧 Component
Other
💻 Environment
No response
🔧 Go Environment
No response
📋 Logs
📝 Additional Context
I have already located the issue and plan to submit a PR to fix it. Please assign this to me.
📋 CheckList
🐛 Bug Description
I discovered a bug in the RunSnapshotItemJob method within item_snapshot.go. When a dataset snapshot is processed in multiple batches (e.g., 100 batches), the snapshot_progress field in the dataset_version table is only updated after the first batch.
For all subsequent batches (from batch 2 onwards), the snapshot_progress in the database remains stuck at the value of the first batch.
This behavior breaks the "resume from breakpoint" mechanism. The initProgressCtx method relies on snapshot_progress to determine where to resume a job. Since the database holds an outdated progress value (from the first batch), if a job fails mid-way (e.g., at batch 50) and restarts, it will incorrectly resume from the position recorded after batch 1, effectively rendering the checkpoint logic useless.
🔄 Steps to Reproduce
Prepare Data: Create or use a dataset with enough items to trigger the snapshot job to run in multiple batches (ensure the data volume exceeds the single-batch limit so that at least 2 batches are generated).
Trigger Action: Publish the dataset version to start the RunSnapshotItemJob.
Observation: Monitor the dataset_version table in the database while the job is running.
Result: You will consistently see that snapshot_progress is updated only after the first batch completes. It fails to update for all subsequent batches, regardless of how many batches follow.
✅ Expected Behavior
The snapshot_progress field in the dataset_version table should be updated after every batch is successfully processed, ensuring the checkpoint is always up-to-date.
❌ Actual Behavior
When a dataset snapshot is processed in multiple batches (e.g., 100 batches), the snapshot_progress field in the dataset_version table is only updated after the first batch.
🚨 Severity
Low - Cosmetic issue or minor inconvenience
🔧 Component
Other
💻 Environment
No response
🔧 Go Environment
No response
📋 Logs
📝 Additional Context
I have already located the issue and plan to submit a PR to fix it. Please assign this to me.