Summary
Add optional range request support to the get, getStream, and getBytes methods across all three drivers (S3, GCS, FS), allowing callers to retrieve a subset of a file's bytes rather than the full contents.
Motivation
Range reads are a common requirement when working with large files. For example, reading audio/video segments, implementing resumable downloads, or inspecting file headers without fetching entire objects. All three underlying storage backends support this capability, but it is currently not exposed through the FlyDrive driver interface.
Proposed API
A new ReadOptions type would be added to src/types.ts:
export type ReadOptions = {
range?: {
start?: number
end?: number
}
}While this would currently just support the range key, this approach seems sensible to retain future extensibility.
The three affected methods on DriverContract would be updated to accept this as an optional second argument:
get(key: string, options?: ReadOptions): Promise<string>
getStream(key: string, options?: ReadOptions): Promise<Readable>
getBytes(key: string, options?: ReadOptions): Promise<Uint8Array>
- Both
start and end are optional but if specified represent inclusive byte positions. If provided they must be positive integers - relative or suffix ranges (e.g. last N bytes via bytes=-500) are not supported and will be rejected by the client-side validation.
- When
start and end are omitted, the method behaves exactly as today.
Validation
A consistent error (e.g. RangeNotSatisfiableException) will be thrown by all drivers when the requested range exceeds the object size or is otherwise invalid (e.g. start > end, negative values). Client-side sanity checks (negative values, start > end) will be applied before any I/O across all drivers.
From initial research, it appears the three backends behave differently with out-of-bounds ranges, so validation is handled per driver but always results in a consistent RangeNotSatisfiableException:
- FS and GCS silently return empty or clamped data with no error, so pre-validation is required. Both will fetch object size before the read and throw if the range exceeds the object size. Note that this would incur an additional round-trip for GCS.
- S3 raises a backend error for invalid ranges, so no preflight is needed. The backend error will instead be caught and rethrown as RangeNotSatisfiableException.
Driver implementation
S3 — pass a Range header string (e.g. bytes=0-1023) to GetObjectCommandInput, which natively supports it across all three get methods. Add error handling to deal with invalid range requests to return the consistent RangeInvalidException
GCS — createReadStream({ start, end }) natively supports ranges. get and getBytes do not have a range-capable download method, so both will be reimplemented internally to drive a range-bounded createReadStream and collect the result into a buffer. A shared private helper will be introduced for this. Pre-validate range request against content length using file.getMetadata().
FS — createReadStream(location, { start, end }) natively supports ranges. get and getBytes both delegate to the private #read helper, which will be updated to accept an optional range and use createReadStream when one is provided, preserving the existing fsp.readFile path for non-ranged reads. The #retrier wrapper is retained for all paths. Pre-validate range request against content length using fsp.stat().
Summary
Add optional range request support to the get, getStream, and getBytes methods across all three drivers (S3, GCS, FS), allowing callers to retrieve a subset of a file's bytes rather than the full contents.
Motivation
Range reads are a common requirement when working with large files. For example, reading audio/video segments, implementing resumable downloads, or inspecting file headers without fetching entire objects. All three underlying storage backends support this capability, but it is currently not exposed through the FlyDrive driver interface.
Proposed API
A new ReadOptions type would be added to src/types.ts:
While this would currently just support the
rangekey, this approach seems sensible to retain future extensibility.The three affected methods on DriverContract would be updated to accept this as an optional second argument:
startandendare optional but if specified represent inclusive byte positions. If provided they must be positive integers - relative or suffix ranges (e.g. last N bytes via bytes=-500) are not supported and will be rejected by the client-side validation.startandendare omitted, the method behaves exactly as today.Validation
A consistent error (e.g. RangeNotSatisfiableException) will be thrown by all drivers when the requested range exceeds the object size or is otherwise invalid (e.g. start > end, negative values). Client-side sanity checks (negative values, start > end) will be applied before any I/O across all drivers.
From initial research, it appears the three backends behave differently with out-of-bounds ranges, so validation is handled per driver but always results in a consistent RangeNotSatisfiableException:
Driver implementation
S3 — pass a Range header string (e.g. bytes=0-1023) to GetObjectCommandInput, which natively supports it across all three get methods. Add error handling to deal with invalid range requests to return the consistent RangeInvalidException
GCS — createReadStream({ start, end }) natively supports ranges. get and getBytes do not have a range-capable download method, so both will be reimplemented internally to drive a range-bounded createReadStream and collect the result into a buffer. A shared private helper will be introduced for this. Pre-validate range request against content length using file.getMetadata().
FS — createReadStream(location, { start, end }) natively supports ranges. get and getBytes both delegate to the private #read helper, which will be updated to accept an optional range and use createReadStream when one is provided, preserving the existing fsp.readFile path for non-ranged reads. The #retrier wrapper is retained for all paths. Pre-validate range request against content length using fsp.stat().