Skip to content

Stream multipart UploadPart on Workers (avoid buffering parts in WASM memory) #89

Description

@alukach

Problem

Multipart UploadPart flows through the NeedsBodysend_raw path, which buffers the entire part body in WASM linear memory on the Cloudflare Workers runtime:

  1. collect_js_body copies the request stream JS→WASM (array_buffer() + to_vec())
  2. execute_multipart computes SHA256 over the whole part for the outbound signature (hash_payload)
  3. send_raw copies WASM→JS (js_sys::Uint8Array::from) to hand to fetch

So each part incurs two full body copies across the JS/WASM boundary + an O(N) SHA256, and is held in the worker's 128 MB memory heap. Single-object PutObject does not have this problem — it uses the zero-copy forward() path (the ReadableStream is handed straight to fetch, signed with UNSIGNED-PAYLOAD so no hashing).

For typical part sizes (~8 MiB) this is a few ms of CPU and fine. But the 128 MB memory limit caps the maximum multipart_chunksize a client can use before the worker OOMs.

Proposed fix

Route UploadPart through the streaming forward() path instead of send_raw:

  • Generate a presigned URL for PUT /{key}?partNumber=N&uploadId=X. object_store's Signer::signed_url only takes a Path (no query params), so presign manually via S3RequestSigner — it already builds SigV4 canonical query strings (incl. value-less params, fixed in feat: data edit operations — batch delete, copy rejection, wider write headers #88).
  • Return HandlerAction::Forward so the ReadableStream streams straight through fetch: no WASM buffering, no payload hash.

CompleteMultipartUpload stays on send_raw (its small XML body must be signed/forwarded), but that body is tiny. CreateMultipartUpload/AbortMultipartUpload are body-less and could also move to the forward path opportunistically.

Interaction with Cloudflare's request-body size limit

Even with streaming, every request through a Worker is bounded by the plan's request-body limit (100 MB Free/Pro, 200 MB Business, 500 MB Enterprise default). Streaming changes which limit binds:

  • Today (buffered): a part is capped by the smaller of the 128 MB memory limit and the plan body limit.
  • After streaming: a part is capped only by the plan body limit (no WASM buffering).

So streaming lets clients use the full plan body limit for part sizes. It does not lift the body limit itself — that's a hard platform constraint (see separate doc note). Large objects must still be uploaded as multipart with part sizes ≤ the plan limit.

Impact

  • Removes the 128 MB part-size ceiling on Workers (raises it to the plan body limit).
  • Eliminates 2 body copies + 1 SHA256 per part → lower CPU.
  • Brings UploadPart to parity with the zero-copy PutObject path.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions