feat(inputs.nats_consumer): add durable consumer support for JetStream#18720
Open
aleksclark wants to merge 2 commits intoinfluxdata:masterfrom
Open
feat(inputs.nats_consumer): add durable consumer support for JetStream#18720aleksclark wants to merge 2 commits intoinfluxdata:masterfrom
aleksclark wants to merge 2 commits intoinfluxdata:masterfrom
Conversation
Add support for durable JetStream consumers that survive Telegraf restarts, preventing message loss or replay. New config options: - jetstream_durable_name: bind to a named durable consumer - jetstream_deliver_policy: control where in the stream to start (all, last, new, by_start_sequence, by_start_time) - jetstream_start_sequence / jetstream_start_time: for sequence/time-based delivery - jetstream_ack_wait: server-side ack timeout before redelivery - jetstream_max_deliver: max redelivery attempts per message - jetstream_filter_subjects: filter subjects within a stream - jetstream_consumer_name: explicit consumer name for pre-created consumers When durable_name is set, subscriptions use Drain() instead of Unsubscribe() on shutdown to preserve the consumer on the server. The plugin also now uses js.Subscribe() when no queue group is set, instead of always using QueueSubscribe(). 🐨 Generated with Crush Assisted-by: AWS Claude Opus 4.6 via Crush <crush@charm.land>
Contributor
|
Thanks so much for the pull request! |
telegraf-tiger
bot
added
feat
Author
|
!signed-cla |
🐘 Generated with Crush Assisted-by: AWS Claude Opus 4.6 via Crush <crush@charm.land>
Contributor
|
Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip. 📦 Click here to get additional PR build artifactsArtifact URLs |
Member
|
@aleksclark can you please restore the PR description template, especially the AI section, as we cannot review your PR otherwise! |
Author
|
@srebhan sure thing, sorry about that! |
telegraf-tiger
bot
removed
the
waiting for response
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add durable consumer support to the NATS JetStream input plugin, allowing consumers to survive Telegraf restarts without message loss or replay.
The existing JetStream support uses ephemeral push consumers. When Telegraf restarts, the consumer is lost and messages may be replayed from the beginning of the stream or missed entirely. Durable consumers persist on the NATS server and resume from where they left off.
New Configuration Options
jetstream_durable_name""jetstream_deliver_policy"all"jetstream_start_sequence0jetstream_start_time""jetstream_ack_wait30sjetstream_max_deliver-1jetstream_filter_subjects[]jetstream_consumer_name""Behavior Changes
durable_nameis set, usesDrain()instead ofUnsubscribe()on shutdown to preserve the consumerjs.Subscribe()when no queue group is set, instead of always usingQueueSubscribe()Tests
natsDocker imageFiles Changed
plugins/inputs/nats_consumer/nats_consumer.go— New config fields + subscription optionsplugins/inputs/nats_consumer/nats_consumer_test.go— 6 new integration tests (580 lines)plugins/inputs/nats_consumer/sample.conf— New config options documentationplugins/inputs/nats_consumer/README.md— Updated documentationChecklist
Related issues
resolves #18734