Skip to content

s3-dump: expose per-target flush results so consumers can label metrics by target #80

Description

@jaronoff97

Problem

S3Dump.flush() returns a single aggregate FlushResult — it sums every sealed batch across all configured targets into one struct. Consumers that record flush metrics (uploaded/failed objects, records, bytes, backlog) therefore cannot break them down by target.

We want a target label on the s3-dump flush metrics in edge, but there's no truthful way to produce it from the current API:

  • flush() aggregates across targets; there's no per-target flush call.
  • The per-target upload happens inside uploadOne(... snapshots[sealed_batch.target] ...), so the consumer can't intercept it to attribute a target name.

Suggested change

flush() already iterates sealed batches that each carry a .target index (src/extensions/s3_dump.zig, the upload loop) — it just sums them. Bucket the counters by target instead and return per-target results, e.g.:

pub const TargetFlushResult = struct {
    name: []const u8,
    objects_uploaded: u32 = 0,
    objects_failed: u32 = 0,
    objects_requeued: u32 = 0,
    records_uploaded: u64 = 0,
    records_failed: u64 = 0,
    records_dropped: u64 = 0,
    bytes_uploaded: u64 = 0,
    bytes_failed: u64 = 0,
    backlog_bytes: usize = 0,
};

Add targets: []const TargetFlushResult to FlushResult (keep the existing aggregate fields for backward compatibility). The slice can be allocated in the flush arena; consumers copy what they need before it's freed, or it's owned like the rest of the result.

records_dropped is currently a single S3Dump-wide atomic drained per flush — deciding whether/how to attribute drops per target (delivery-time drops know their slot; backlog-cap drops may not) is part of this issue.

Consumer context

edge wires s3-dump for Datadog logs and records these aggregate metrics today:
edge_s3_dump_{flushes,objects_uploaded,objects_failed,records_uploaded,records_dropped,bytes_uploaded}_total + edge_s3_dump_backlog_bytes. Once per-target results land we'll add a target label sourced from TargetFlushResult.name.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions