Skip to content

fix(file source): don't warn about empty files too small to fingerprint#25895

Draft
gaurav0107 wants to merge 1 commit into
vectordotdev:masterfrom
gaurav0107:fix/1065-only-warn-about-small-files-when-they-ar
Draft

fix(file source): don't warn about empty files too small to fingerprint#25895
gaurav0107 wants to merge 1 commit into
vectordotdev:masterfrom
gaurav0107:fix/1065-only-warn-about-small-files-when-they-ar

Conversation

@gaurav0107

Copy link
Copy Markdown

Summary

When the file source uses the checksum fingerprinting strategy, files that are too small to be fingerprinted trigger a Currently ignoring file too small to fingerprint. warning. In practice this fires constantly for empty (0-byte) files, which are common and transient in production — for example freshly-created log files or short-lived Kubernetes pods that start and stop quickly. The result is log noise that isn't actionable, which several users have reported on #1065.

This change suppresses the warning for empty files while keeping it for non-empty files that are still too small to fingerprint (that case can genuinely surprise a user wondering why a file isn't being tailed). The retry/cleanup bookkeeping in known_small_files is unchanged, so empty files are still tracked and picked up once they grow.

The fix is confined to Fingerprinter::fingerprint_or_emit in lib/file-source-common/src/fingerprinter.rs: the file's emptiness is read from the metadata that is already fetched, and emit_file_checksum_failed is skipped when the file is empty.

AI assistance disclosure

Per Vector's AI policy: this change was drafted with the assistance of an AI coding agent (model: Claude Opus 4.8). I have read, understood, and locally tested the change, and I own the contribution.

How did you test this PR?

  • Added a unit test fingerprint_or_emit_only_warns_on_non_empty_small_files in fingerprinter.rs using a recording emitter that counts emit_file_checksum_failed calls. It asserts that an empty file emits no warning and returns None, while a non-empty-but-too-small file does emit exactly one warning.
  • cargo test -p file-source-common — 22 passed, 0 failed.
  • cargo fmt -p file-source-common -- --check — clean.
  • cargo clippy -p file-source-common --all-targets — clean.

Change Type

  • Bug fix
  • New feature
  • Dependencies
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

The checksum_errors_total metric still increments for non-empty files that are too small; only the log warning for empty files changes.

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

A changelog fragment is included: changelog.d/1065_file_source_empty_file_warning.fix.md.

References


🤖 Opened with Superhuman, an open-source contribution agent.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Before we can merge this PR, please sign our Contributor License Agreement.

To sign, copy and post the phrase below as a new comment on this PR.

Note: If the bot says your username was not found, the email used in your git commit may not be linked to your GitHub account. Fix this at github.com/settings/emails, then comment recheck to retry.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

@datadog-vectordotdev

datadog-vectordotdev Bot commented Jul 19, 2026

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 1 Pipeline job failed

CLA Assistant | CLAAssistant   View in Datadog   GitHub Actions

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 053b0ec | Docs | Give us feedback!

When the `file` source uses the `checksum` fingerprinting strategy, an
empty file is too small to fingerprint and previously emitted a
"Currently ignoring file too small to fingerprint" warning. Empty files
are common and transient (freshly-created log files, short-lived
Kubernetes pods), so this flooded operator logs with non-actionable
warnings.

Suppress the warning for empty (0-byte) files while still emitting it for
non-empty files that are too small to fingerprint, since those can
genuinely surprise a user wondering why a file isn't being tailed. The
retry/cleanup bookkeeping in `known_small_files` is unchanged.

Closes: vectordotdev#1065
@gaurav0107
gaurav0107 force-pushed the fix/1065-only-warn-about-small-files-when-they-ar branch from 9378144 to 053b0ec Compare July 19, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Only warn about small files when they are not empty

1 participant