Skip to content

Writing lots of possibly-bad data vs. data loss #15

@smurfix

Description

@smurfix

Akumuli doesn't tell me which item I just sent caused an error. This is a problem because I receive lots of random UDP packets per second from collectd and want to feed them to Akumuli.

Some of these will get duplicated, or reorderd. Both trigger an error, and there no way of knowing which item the error is referring to, thus no way to log the culprit.

To fix that, at minimum I'd modify the error message to always include a sequence number of the bad data item. I can think of a few other mitigations but they all require some protocol modification.

Also, I'd add a nicer serialization protocol. RESP is very minimal and doesn't support versioning, or positive replies (they would probably confuse older clients), or structured error messages, or updating the dictionary, or …
I'd look into MsgPack, there are great C++ bindings for it and it doesn't require parsing ASCII floats and whatnot. It even has a separate timestamp data type. ;-)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions