Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions pipeline/outputs/file.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,11 @@ The _File_ output plugin lets you write the data received through the input plug
| Key | Description | Default |
| :--- | :--- | :--- |
| `file` | Set filename to store the records. If not set, the filename will be the `tag` associated with the records. | _none_ |
| `files_rotation` | Enable size-based [log rotation](#log-rotation). When enabled, files that exceed `max_size` are rotated and optionally compressed. | `false` |
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a query on how this is done, is it as soon as they reach that size or a periodic check or something else?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SagiROosto can you answer this for @patrick-stephens ?

| `format` | The [format](#format) of the file content. | _none_ |
| `gzip` | Compress rotated files using gzip. Only applies when `files_rotation` is enabled. | `true` |
| `max_files` | Maximum number of rotated files to retain per output file. Oldest files are deleted first. Must be `1` or greater. Only applies when `files_rotation` is enabled. | `7` |
| `max_size` | Maximum size of the active output file before rotation is triggered. Supports size suffixes: `k` (kilobytes), `m` (megabytes), `g` (gigabytes). Only applies when `files_rotation` is enabled. | `100m` |
Comment on lines +17 to +18
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if you do not enable rotation? Does it fail to start, report a warning, etc.?

Similarly what if you use invalid values here? Does it fail with a config error, use the default, etc?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SagiROosto can you answer this for @patrick-stephens ?

| `mkdir` | Recursively create output directory if it doesn't exist. Permissions set to `0755`. | `false` |
| `path` | Directory path to store files. If not set, Fluent Bit will write the files in its own working directory. | _none_ |
| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `1` |
Expand Down Expand Up @@ -112,6 +116,64 @@ You will get the following output:
1564462620.000254 used=1045448 free=31760160 total=32805608
```

## Log rotation

The File output plugin supports size-based log rotation.

When `files_rotation` is enabled, the plugin monitors the size of each output file. Once a file exceeds `max_size`, the plugin rotates the file by renaming it with a timestamp suffix in the format `<filename>.<YYYYMMDD_HHMMSS_XXXXXXXX>`. The `YYYYMMDD_HHMMSS` is timestamp of the time when the rotation occurred, and `XXXXXXXX` is a random identifier to guarantee unique filenames if multiple rotations happen within the same second.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does the timestamp come from? Is it UTC or local timezone based?

Be nice to have a controllable format for the future too.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SagiROosto can you answer this for @patrick-stephens ?


If `gzip` is enabled (the default), rotated files are compressed with gzip and stored with an additional `.gz` extension (for example, `cpu.log.20260512_134500_a1b2c3d4.gz`).

The plugin retains up to `max_files` rotated files per output file. When the limit is reached, the oldest rotated files are deleted automatically.

Log rotation works with all supported output [formats](#format): `plain`, `CSV`, `LTSV`, `template`, and `msgpack`. File operations are thread-safe, so rotation can be used alongside multiple [workers](../../administration/multithreading.md#outputs).

### Log rotation example

The following configuration writes CPU metrics to file with rotation enabled. Files are rotated at 50 MB and the five most recent rotated files are retained with gzip compression:

{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:
inputs:
- name: cpu
tag: cpu

outputs:
- name: file
match: '*'
path: /var/log/fluent-bit
file: cpu.log
files_rotation: true
max_size: 50m
max_files: 5
gzip: true
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```text
[INPUT]
Name cpu
Tag cpu

[OUTPUT]
Name file
Match *
Path /var/log/fluent-bit
File cpu.log
Files_Rotation true
Max_Size 50m
Max_Files 5
Gzip true
```

{% endtab %}
{% endtabs %}

## Get started

You can run the plugin from the command line or through the configuration file.
Expand Down
Loading