Skip to content

bug(api): NDJSON ingest silently drops every object after the first on a line #398

Description

@taitelee

Summary

The NDJSON line reader decodes one object per line with a single dec.Decode and no trailing-token check, so {"a":1}{"a":2} on one line ingests the first object and silently discards the rest.

Detail

internal/api/record_reader.go:148-161 (lineReader.Next): per non-blank line it does one dec.Decode(&m) and returns, never calling dec.More() / a second decode. A client bug that drops a newline between two records produces one line with two objects; the batch response reports succeeded:1 and the second record is lost with a 200. The admin query handler (internal/api/query.go:181) explicitly guards this exact class ("a real risk if a buggy client double-encodes") — the ingest path doesn't.

Fix direction

After decoding, reject a line with trailing non-whitespace tokens (dec.More() or a follow-up decode expecting EOF) as a per-record syntax error, so the loss surfaces instead of passing silently.

Found in a repo-wide audit; verified by code trace.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/apiHTTP handlers, routing, middlewarearea/ingestIngest pipeline (Bento, batching, DLQ)bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions