feat: Async.chunk source-agnostic chunked processing#51
Open
Mateusz7410 wants to merge 1 commit into
Open
Conversation
Process large data sets in tuned pages across chained Queueables with per-page tracking and retry, without reaching for a Batchable. The source is pluggable via ChunkSource (in-memory records, id set, SOQL cursor, or a query string), so callers stay SOQL-lib free and can supply their own. Pages run as chain members: on settle the framework appends the next page and prunes the recorded one, keeping the serialized chain state flat at any volume. QueueableJob is unchanged; the paging seam lives in QueueableChain. Closes #47
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🧪 Apex Test Results✅ All Tests Passed📦 Download Full Test Results & Logs 📊 Stats: 188 total | ✅ 188 passed | ❌ 0 failed |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #51 +/- ##
==========================================
- Coverage 99.10% 98.96% -0.14%
==========================================
Files 16 20 +4
Lines 1002 1157 +155
==========================================
+ Hits 993 1145 +152
- Misses 9 12 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #47
What
Adds
Async.chunk(...), a Queueable-based way to process a large data set intuned pages across chained Queueables, with the same per-page tracking and
retry/backoff as a normal
QueueableJob, without reaching for aDatabase.Batchable.Public API (all additive, no existing
globalsignature changed)Async.chunk(ChunkJob job, ChunkSource source)entry.ChunkSource(abstract, extensible) with factoriesof(List<SObject>),ofIds(Set<Id>),cursor(Database.Cursor),query(String soql).ChunkJob(overridework(List<SObject> chunk)) andIdChunkJob(override
work(Set<Id> ids); the framework plucks ids).ChunkBuilder:chunkSize,stopRemainingChunksOnFailure,priority,delay,delayBetweenChunks,retry,backoff,retryOn,mockId,keepChunkPages,enqueue.Design notes
QueueableChainappends thenext page and (by default) prunes the recorded one, so the serialized chain
state stays flat regardless of run size. One enqueue per hop.
QueueableJobis untouched. The paging seam lives entirely inQueueableChain+ChunkJob; shared retry helpers moved into internalQueueableManager.ChunkSourceis the polymorphic seam. The pager only callssize()/fetch(offset, count), so it is source-agnostic and consumers canextend
ChunkSourcefor custom sources.Database.Cursorrides platform Queueableserialization across every hop (soft-clone shares it, enqueue serializes it).
deepCloneis rejected for aChunkJobat build time, sinceJSON.serializecannot round-trip a cursor and pages do not need it.
recorded and the run continues);
.stopRemainingChunksOnFailure()halts.Per-page
AsyncResult__chonors the existingCreateResultCMDT gate, nochunk-specific override.
Packaging
New globals (
ChunkSource,ChunkJob,IdChunkJob,ChunkBuilder) added tothe
public->globalrewrite list inscripts/create-unlocked-package-version.sh; chunk-run wiring(
prepareRun,nextPageOrNull,shouldKeepPages) is reverted back topublicso it stays internal.
Verification
verify.sh full(clean no-namespace scratch): 188 tests, 100% pass, chunkclasses 100% covered, org-wide 97%.
verify.sh ns(btcdev namespace, consumer-as-package-install): 19/19.public->globalrewrite deployed to a no-namespacescratch, all chunk tests pass on the globalized code.
with member state carried across every page, 4 completed queueables.
Docs
Adds
website/api/chunk.md(API + cursor governor caps + FLS/SOQL-ownershipnote) and the sidebar entry.
installation.mduntouched (belongs to/release).