Problem
S3Dump.flush() returns a single aggregate FlushResult — it sums every sealed batch across all configured targets into one struct. Consumers that record flush metrics (uploaded/failed objects, records, bytes, backlog) therefore cannot break them down by target.
We want a target label on the s3-dump flush metrics in edge, but there's no truthful way to produce it from the current API:
flush() aggregates across targets; there's no per-target flush call.
- The per-target upload happens inside
uploadOne(... snapshots[sealed_batch.target] ...), so the consumer can't intercept it to attribute a target name.
Suggested change
flush() already iterates sealed batches that each carry a .target index (src/extensions/s3_dump.zig, the upload loop) — it just sums them. Bucket the counters by target instead and return per-target results, e.g.:
pub const TargetFlushResult = struct {
name: []const u8,
objects_uploaded: u32 = 0,
objects_failed: u32 = 0,
objects_requeued: u32 = 0,
records_uploaded: u64 = 0,
records_failed: u64 = 0,
records_dropped: u64 = 0,
bytes_uploaded: u64 = 0,
bytes_failed: u64 = 0,
backlog_bytes: usize = 0,
};
Add targets: []const TargetFlushResult to FlushResult (keep the existing aggregate fields for backward compatibility). The slice can be allocated in the flush arena; consumers copy what they need before it's freed, or it's owned like the rest of the result.
records_dropped is currently a single S3Dump-wide atomic drained per flush — deciding whether/how to attribute drops per target (delivery-time drops know their slot; backlog-cap drops may not) is part of this issue.
Consumer context
edge wires s3-dump for Datadog logs and records these aggregate metrics today:
edge_s3_dump_{flushes,objects_uploaded,objects_failed,records_uploaded,records_dropped,bytes_uploaded}_total + edge_s3_dump_backlog_bytes. Once per-target results land we'll add a target label sourced from TargetFlushResult.name.
Problem
S3Dump.flush()returns a single aggregateFlushResult— it sums every sealed batch across all configured targets into one struct. Consumers that record flush metrics (uploaded/failed objects, records, bytes, backlog) therefore cannot break them down by target.We want a
targetlabel on the s3-dump flush metrics in edge, but there's no truthful way to produce it from the current API:flush()aggregates across targets; there's no per-target flush call.uploadOne(... snapshots[sealed_batch.target] ...), so the consumer can't intercept it to attribute a target name.Suggested change
flush()already iterates sealed batches that each carry a.targetindex (src/extensions/s3_dump.zig, the upload loop) — it just sums them. Bucket the counters by target instead and return per-target results, e.g.:Add
targets: []const TargetFlushResulttoFlushResult(keep the existing aggregate fields for backward compatibility). The slice can be allocated in the flush arena; consumers copy what they need before it's freed, or it's owned like the rest of the result.records_droppedis currently a single S3Dump-wide atomic drained per flush — deciding whether/how to attribute drops per target (delivery-time drops know their slot; backlog-cap drops may not) is part of this issue.Consumer context
edge wires s3-dump for Datadog logs and records these aggregate metrics today:
edge_s3_dump_{flushes,objects_uploaded,objects_failed,records_uploaded,records_dropped,bytes_uploaded}_total+edge_s3_dump_backlog_bytes. Once per-target results land we'll add atargetlabel sourced fromTargetFlushResult.name.