-
Notifications
You must be signed in to change notification settings - Fork 552
docs: outputs: file: add log rotation support and configuration examples #2572
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,7 +11,11 @@ The _File_ output plugin lets you write the data received through the input plug | |
| | Key | Description | Default | | ||
| | :--- | :--- | :--- | | ||
| | `file` | Set filename to store the records. If not set, the filename will be the `tag` associated with the records. | _none_ | | ||
| | `files_rotation` | Enable size-based [log rotation](#log-rotation). When enabled, files that exceed `max_size` are rotated and optionally compressed. | `false` | | ||
| | `format` | The [format](#format) of the file content. | _none_ | | ||
| | `gzip` | Compress rotated files using gzip. Only applies when `files_rotation` is enabled. | `true` | | ||
| | `max_files` | Maximum number of rotated files to retain per output file. Oldest files are deleted first. Must be `1` or greater. Only applies when `files_rotation` is enabled. | `7` | | ||
| | `max_size` | Maximum size of the active output file before rotation is triggered. Supports size suffixes: `k` (kilobytes), `m` (megabytes), `g` (gigabytes). Only applies when `files_rotation` is enabled. | `100m` | | ||
|
Comment on lines
+17
to
+18
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What happens if you do not enable rotation? Does it fail to start, report a warning, etc.? Similarly what if you use invalid values here? Does it fail with a config error, use the default, etc?
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @SagiROosto can you answer this for @patrick-stephens ? |
||
| | `mkdir` | Recursively create output directory if it doesn't exist. Permissions set to `0755`. | `false` | | ||
| | `path` | Directory path to store files. If not set, Fluent Bit will write the files in its own working directory. | _none_ | | ||
| | `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `1` | | ||
|
|
@@ -112,6 +116,64 @@ You will get the following output: | |
| 1564462620.000254 used=1045448 free=31760160 total=32805608 | ||
| ``` | ||
|
|
||
| ## Log rotation | ||
|
|
||
| The File output plugin supports size-based log rotation. | ||
|
|
||
| When `files_rotation` is enabled, the plugin monitors the size of each output file. Once a file exceeds `max_size`, the plugin rotates the file by renaming it with a timestamp suffix in the format `<filename>.<YYYYMMDD_HHMMSS_XXXXXXXX>`. The `YYYYMMDD_HHMMSS` is timestamp of the time when the rotation occurred, and `XXXXXXXX` is a random identifier to guarantee unique filenames if multiple rotations happen within the same second. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Where does the timestamp come from? Is it UTC or local timezone based? Be nice to have a controllable format for the future too.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @SagiROosto can you answer this for @patrick-stephens ? |
||
|
|
||
| If `gzip` is enabled (the default), rotated files are compressed with gzip and stored with an additional `.gz` extension (for example, `cpu.log.20260512_134500_a1b2c3d4.gz`). | ||
|
|
||
| The plugin retains up to `max_files` rotated files per output file. When the limit is reached, the oldest rotated files are deleted automatically. | ||
|
|
||
| Log rotation works with all supported output [formats](#format): `plain`, `CSV`, `LTSV`, `template`, and `msgpack`. File operations are thread-safe, so rotation can be used alongside multiple [workers](../../administration/multithreading.md#outputs). | ||
|
|
||
| ### Log rotation example | ||
|
|
||
| The following configuration writes CPU metrics to file with rotation enabled. Files are rotated at 50 MB and the five most recent rotated files are retained with gzip compression: | ||
|
|
||
| {% tabs %} | ||
| {% tab title="fluent-bit.yaml" %} | ||
|
|
||
| ```yaml | ||
| pipeline: | ||
| inputs: | ||
| - name: cpu | ||
| tag: cpu | ||
|
|
||
| outputs: | ||
| - name: file | ||
| match: '*' | ||
| path: /var/log/fluent-bit | ||
| file: cpu.log | ||
| files_rotation: true | ||
| max_size: 50m | ||
| max_files: 5 | ||
| gzip: true | ||
| ``` | ||
|
|
||
| {% endtab %} | ||
| {% tab title="fluent-bit.conf" %} | ||
|
|
||
| ```text | ||
| [INPUT] | ||
| Name cpu | ||
| Tag cpu | ||
|
|
||
| [OUTPUT] | ||
| Name file | ||
| Match * | ||
| Path /var/log/fluent-bit | ||
| File cpu.log | ||
| Files_Rotation true | ||
| Max_Size 50m | ||
| Max_Files 5 | ||
| Gzip true | ||
| ``` | ||
|
|
||
| {% endtab %} | ||
| {% endtabs %} | ||
|
|
||
| ## Get started | ||
|
|
||
| You can run the plugin from the command line or through the configuration file. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a query on how this is done, is it as soon as they reach that size or a periodic check or something else?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@SagiROosto can you answer this for @patrick-stephens ?