Skip to content

feat: Async.chunk source-agnostic chunked processing#51

Open
Mateusz7410 wants to merge 1 commit into
mainfrom
feature/async-0009
Open

feat: Async.chunk source-agnostic chunked processing#51
Mateusz7410 wants to merge 1 commit into
mainfrom
feature/async-0009

Conversation

@Mateusz7410

Copy link
Copy Markdown
Collaborator

Closes #47

What

Adds Async.chunk(...), a Queueable-based way to process a large data set in
tuned pages across chained Queueables, with the same per-page tracking and
retry/backoff as a normal QueueableJob, without reaching for a
Database.Batchable.

// in-memory records or ids
Async.chunk(new AccountRecalcJob(), ChunkSource.of(records))
    .chunkSize(50)
    .retry(2)
    .enqueue();

// large sets via a SOQL cursor, never held in memory at once
Async.chunk(new AccountRecalcJob(), ChunkSource.query('SELECT Id FROM Account WHERE ...'))
    .chunkSize(200)
    .enqueue();

Public API (all additive, no existing global signature changed)

  • Async.chunk(ChunkJob job, ChunkSource source) entry.
  • ChunkSource (abstract, extensible) with factories of(List<SObject>),
    ofIds(Set<Id>), cursor(Database.Cursor), query(String soql).
  • ChunkJob (override work(List<SObject> chunk)) and IdChunkJob
    (override work(Set<Id> ids); the framework plucks ids).
  • ChunkBuilder: chunkSize, stopRemainingChunksOnFailure, priority,
    delay, delayBetweenChunks, retry, backoff, retryOn, mockId,
    keepChunkPages, enqueue.

Design notes

  • Pages are real chain members. On settle, QueueableChain appends the
    next page and (by default) prunes the recorded one, so the serialized chain
    state stays flat regardless of run size. One enqueue per hop.
  • QueueableJob is untouched. The paging seam lives entirely in
    QueueableChain + ChunkJob; shared retry helpers moved into internal
    QueueableManager.
  • ChunkSource is the polymorphic seam. The pager only calls
    size()/fetch(offset, count), so it is source-agnostic and consumers can
    extend ChunkSource for custom sources.
  • Cursor survival. A Database.Cursor rides platform Queueable
    serialization across every hop (soft-clone shares it, enqueue serializes it).
    deepClone is rejected for a ChunkJob at build time, since JSON.serialize
    cannot round-trip a cursor and pages do not need it.
  • Failure model. Best-effort by default (a chunk that exhausts retries is
    recorded and the run continues); .stopRemainingChunksOnFailure() halts.
    Per-page AsyncResult__c honors the existing CreateResult CMDT gate, no
    chunk-specific override.

Packaging

New globals (ChunkSource, ChunkJob, IdChunkJob, ChunkBuilder) added to
the public -> global rewrite list in
scripts/create-unlocked-package-version.sh; chunk-run wiring
(prepareRun, nextPageOrNull, shouldKeepPages) is reverted back to public
so it stays internal.

Verification

  • verify.sh full (clean no-namespace scratch): 188 tests, 100% pass, chunk
    classes 100% covered, org-wide 97%.
  • verify.sh ns (btcdev namespace, consumer-as-package-install): 19/19.
  • Globalized-build proxy: public -> global rewrite deployed to a no-namespace
    scratch, all chunk tests pass on the globalized code.
  • Live org run: 8 records via a cursor at chunkSize 2 (4 pages), all 8 processed
    with member state carried across every page, 4 completed queueables.

Docs

Adds website/api/chunk.md (API + cursor governor caps + FLS/SOQL-ownership
note) and the sidebar entry. installation.md untouched (belongs to /release).

Process large data sets in tuned pages across chained Queueables with
per-page tracking and retry, without reaching for a Batchable. The source
is pluggable via ChunkSource (in-memory records, id set, SOQL cursor, or a
query string), so callers stay SOQL-lib free and can supply their own.

Pages run as chain members: on settle the framework appends the next page
and prunes the recorded one, keeping the serialized chain state flat at any
volume. QueueableJob is unchanged; the paging seam lives in QueueableChain.

Closes #47
@vercel

vercel Bot commented Jul 15, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
async-lib Ready Ready Preview, Comment Jul 15, 2026 12:20pm

Request Review

@github-actions

Copy link
Copy Markdown

🧪 Apex Test Results

✅ All Tests Passed

==========================================
     APEX TEST EXECUTION SUMMARY
==========================================

📊 Total Tests: 188
✅ Passed: 188
❌ Failed: 0
⏭️  Skipped: 0


🎉 All tests passed successfully!

📦 Download Full Test Results & Logs


📊 Stats: 188 total | ✅ 188 passed | ❌ 0 failed
🤖 Automated comment by Salesforce CI

@codecov

codecov Bot commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.18182% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 98.96%. Comparing base (991d0b4) to head (16c859c).

Files with missing lines Patch % Lines
...ce-app/main/default/classes/queue/ChunkBuilder.cls 95.00% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #51      +/-   ##
==========================================
- Coverage   99.10%   98.96%   -0.14%     
==========================================
  Files          16       20       +4     
  Lines        1002     1157     +155     
==========================================
+ Hits          993     1145     +152     
- Misses          9       12       +3     
Flag Coverage Δ
Apex 98.96% <98.18%> (-0.14%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chunked processing across chained Queueables (source-agnostic)

1 participant