feat(dogstatsd): decouple reader from processor

[duncanista]

What does this PR do?

Splits the single-threaded spin() loop into two cooperating tasks connected by a bounded channel:

• Reader task — drains the UDP socket as fast as possible and sends raw packets into the channel
• Processor task — receives packets from the channel, parses metrics, and forwards them to the aggregator

Also adds a configurable queue_size option to DogStatsDConfig (matching the Go agent's DD_DOGSTATSD_QUEUE_SIZE), defaulting to 1024.

Motivation

The previous spin() loop read one packet, parsed every metric in it, inserted them into the aggregator, and only then read the next packet. While the processor was busy, incoming packets accumulated in the kernel's SO_RCVBUF — which on Lambda is capped at ~416 KiB (~7
packets at 64 KB). Under burst traffic (e.g. 300k metrics), the kernel silently drops packets that overflow the buffer.

By decoupling the reader from the processor, buffering moves from kernel space (~416 KiB, not configurable on Lambda) to user space (queue_size × buffer_size ≈ 8 MB at defaults). The reader never blocks on processing, so the kernel buffer stays drained. This is where
the bulk of the metric recovery happens.

SVLS-8551

Notes

• Public API is additive only: DogStatsDConfig gains queue_size: Option. Callers (e.g. bottlecap) need to add queue_size: None to their config construction.
  • consume_statsd() is now #[cfg(test)] only — it exists solely for the MirrorTest unit test path.
  • insert_metrics, prepend_namespace, and process_packet were extracted from impl DogStatsD to free functions since they don't need &self.
  • Windows named pipe transport is unchanged and continues to work through BufferReader::read_into.
    https://github.com/DataDog/datadog-agent/blob/85939a62b5580b2a15549f6936f257e61c5aa153/pkg/config/config_template.yaml#L2184-L2190

Describe how to test/QA your changes

Sending 300k metrics over a 3 second span, we get an improvement of receiving 145k to around 210k metrics

[Copilot]

Pull request overview

This PR refactors the DogStatsD server to improve throughput under burst traffic by decoupling packet reading from metric processing. Previously, the single-threaded spin() loop read one packet, parsed it, and inserted metrics into the aggregator before reading the next packet. This caused packets to accumulate in the kernel's SO_RCVBUF buffer, which is capped at ~416 KiB on Lambda, leading to packet drops.

The new architecture spawns a dedicated reader task that continuously drains the UDP socket into a bounded in-memory channel, while the main task processes packets from that channel. This moves buffering from kernel space to user space (configurable via queue_size, defaulting to ~8 MB), significantly reducing packet loss during bursts.

Changes:

• Introduced a bounded channel (queue_size: 1024 packets) between socket reading and metric processing
• Refactored spin() to spawn a reader task and process packets from a channel
• Extracted insert_metrics, prepend_namespace, and process_packet as standalone functions
• Added queue_size configuration option to DogStatsDConfig

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
crates/dogstatsd/src/dogstatsd.rs	Core refactoring: split single loop into read_loop (spawned task) and process_loop, added queue_size config, refactored helper functions to work without self reference
crates/dogstatsd/tests/integration_test.rs	Added queue_size: None to all DogStatsDConfig test instances
crates/datadog-serverless-compat/src/main.rs	Added queue_size: None to DogStatsDConfig initialization

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[duncanpharvey]

Looks good! Added a few comments for consideration.