Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions pipeline/filters/checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,23 +48,23 @@ pipeline:

```text
[INPUT]
name tail
tag test1
path test1.log
read_from_head true
parser json
Name tail
Tag test1
Path test1.log
Read_From_Head true
Parser json

[FILTER]
name checklist
match test1
file ip_list.txt
lookup_key $remote_addr
record ioc abc
record badurl null
Name checklist
Match test1
File ip_list.txt
Lookup_Key $remote_addr
Record ioc abc
Record badurl null

[OUTPUT]
name stdout
match test1
Name stdout
Match test1
```

{% endtab %}
Expand Down
26 changes: 13 additions & 13 deletions pipeline/filters/ecs-metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,16 @@ The following template variables can be used for values with the `add` option. S
| Variable | Description | Supported with `cluster_metadata_only` on |
| :--- | :--- | :--- |
| `$ClusterName` | The ECS cluster name. Fluent Bit is running on EC2 instances that are part of this cluster. | `Yes` |
| `$ContainerID` | The ID of the container from which the log originated. This is the full 64-character-long container ID. | `No` |
| `$ContainerInstanceArn` | The full ARN of the ECS EC2 Container Instance. This is the instance that Fluent Bit is running on. | `Yes` |
| `$ContainerInstanceID` | The ID of the ECS EC2 Container Instance. | `Yes` |
| `$DockerContainerName` | The name of the container from which the log originated. This is the name obtained from Docker and is the name shown if you run `docker ps` on the instance. | `No` |
| `$ECSAgentVersion` | The version string of the ECS Agent running on the container instance. | `Yes` |
| `$ECSContainerName` | The name of the container from which the log originated. This is the name in your ECS Task Definition. | `No` |
| `$DockerContainerName` | The name of the container from which the log originated. This is the name obtained from Docker and is the name shown if you run `docker ps` on the instance. | `No` |
| `$ContainerID` | The ID of the container from which the log originated. This is the full 64-character-long container ID. | `No` |
| `$TaskARN` | The full ARN of the ECS Task from which the log originated. | `No` |
| `$TaskDefinitionFamily` | The family name of the task definition for the task from which the log originated. | `No` |
| `$TaskDefinitionVersion` | The version or revision of the task definition for the task from which the log originated. | `No` |
| `$TaskID` | The ID of the ECS Task from which the log originated. | `No` |
| `$TaskARN` | The full ARN of the ECS Task from which the log originated. | `No` |

### Configuration file

Expand Down Expand Up @@ -94,15 +94,15 @@ pipeline:
Skip_Long_Lines On
Refresh_Interval 10
Rotate_Wait 30
storage.type filesystem
Storage.Type filesystem
Read_From_Head Off

[FILTER]
Name ecs
Match *
ecs_tag_prefix ecs.var.lib.docker.containers.
add ecs_task_id $TaskID
add cluster $ClusterName
Ecs_Tag_Prefix ecs.var.lib.docker.containers.
Add ecs_task_id $TaskID
Add cluster $ClusterName

[OUTPUT]
Name stdout
Expand Down Expand Up @@ -176,14 +176,14 @@ pipeline:
Skip_Long_Lines On
Refresh_Interval 10
Rotate_Wait 30
storage.type filesystem
Storage.Type filesystem
Read_From_Head Off

[FILTER]
Name ecs
Match *
ecs_tag_prefix ecs.var.lib.docker.containers.
add resource $ClusterName.$TaskDefinitionFamily.$TaskID.$ECSContainerName
Ecs_Tag_Prefix ecs.var.lib.docker.containers.
Add resource $ClusterName.$TaskDefinitionFamily.$TaskID.$ECSContainerName

[OUTPUT]
Name stdout
Expand Down Expand Up @@ -253,15 +253,15 @@ pipeline:
Skip_Long_Lines On
Refresh_Interval 10
Rotate_Wait 30
storage.type filesystem
Storage.Type filesystem
# Collect all logs on instance
Read_From_Head On

[FILTER]
Name ecs
Match *
cluster_metadata_only on
add cluster $ClusterName
Cluster_Metadata_Only on
Add cluster $ClusterName

[OUTPUT]
Name stdout
Expand Down
4 changes: 2 additions & 2 deletions pipeline/filters/geoip2-filter.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ pipeline:
Name geoip2
Match *
Database GeoLite2-City.mmdb
Lookup_key remote_addr
Lookup_Key remote_addr
Record country remote_addr %{country.names.en}
Record isocode remote_addr %{country.iso_code}

Expand All @@ -69,7 +69,7 @@ pipeline:
{% endtab %}
{% endtabs %}

Each `Record` parameter specifies the following triplet:
Each `record` parameter specifies the following triplet:

- `country`: The field name to be added to records.
- `remote_addr`: The lookup key to process.
Expand Down
2 changes: 1 addition & 1 deletion pipeline/filters/kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,8 +164,8 @@ The following annotations are available:

| Annotation | Description | Default |
| :--- | :--- | :--- |
| `fluentbit.io/parser[_stream][-container]` | Suggest a pre-defined parser. The parser must be registered already by Fluent Bit. This option will only be processed if Fluent Bit configuration (Kubernetes Filter) has enabled the option `K8S-Logging.Parser`. If present, the stream (stdout or stderr) will restrict that specific stream. If present, the container can override a specific container in a Pod. | _none_ |
| `fluentbit.io/exclude[_stream][-container]` | Define whether to request that Fluent Bit excludes the logs generated by the pod. This option will be processed only if the Fluent Bit configuration (Kubernetes Filter) has enabled the option `K8S-Logging.Exclude`. | `False` |
| `fluentbit.io/parser[_stream][-container]` | Suggest a pre-defined parser. The parser must be registered already by Fluent Bit. This option will only be processed if Fluent Bit configuration (Kubernetes Filter) has enabled the option `K8S-Logging.Parser`. If present, the stream (stdout or stderr) will restrict that specific stream. If present, the container can override a specific container in a Pod. | _none_ |

### Annotation examples in pod definition

Expand Down
146 changes: 73 additions & 73 deletions pipeline/filters/log_to_metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,8 @@ pipeline:

```text
[SERVICE]
flush 1
log_level info
Flush 1
Log_Level info

[INPUT]
Name dummy
Expand All @@ -99,18 +99,18 @@ pipeline:
Tag dummy.log2

[FILTER]
name log_to_metrics
match dummy.log*
tag test_metric
metric_mode counter
metric_name count_all_dummy_messages
metric_description This metric counts dummy messages
Name log_to_metrics
Match dummy.log*
Tag test_metric
Metric_Mode counter
Metric_Name count_all_dummy_messages
Metric_Description This metric counts dummy messages

[OUTPUT]
name prometheus_exporter
match *
host 0.0.0.0
port 9999
Name prometheus_exporter
Match *
Host 0.0.0.0
Port 9999
```

{% endtab %}
Expand Down Expand Up @@ -193,8 +193,8 @@ pipeline:

```text
[SERVICE]
flush 1
log_level info
Flush 1
Log_Level info

[INPUT]
Name dummy
Expand All @@ -207,24 +207,24 @@ pipeline:
Tag dummy.log2

[FILTER]
name log_to_metrics
match dummy.log*
tag test_metric
metric_mode gauge
metric_name current_duration
metric_description This metric shows the current duration
value_field duration
kubernetes_mode on
regex message .*el.*
add_label app $kubernetes['labels']['app']
label_field color
label_field shape
Name log_to_metrics
Match dummy.log*
Tag test_metric
Metric_Mode gauge
Metric_Name current_duration
Metric_Description This metric shows the current duration
Value_Field duration
Kubernetes_Mode on
Regex message .*el.*
Add_Label app $kubernetes['labels']['app']
Label_Field color
Label_Field shape

[OUTPUT]
name prometheus_exporter
match *
host 0.0.0.0
port 9999
Name prometheus_exporter
Match *
Host 0.0.0.0
Port 9999
```

{% endtab %}
Expand Down Expand Up @@ -317,8 +317,8 @@ pipeline:

```text
[SERVICE]
flush 1
log_level info
Flush 1
Log_Level info

[INPUT]
Name dummy
Expand All @@ -331,24 +331,24 @@ pipeline:
Tag dummy.log2

[FILTER]
name log_to_metrics
match dummy.log*
tag test_metric
metric_mode histogram
metric_name current_duration
metric_description This metric shows the request duration
value_field duration
kubernetes_mode on
regex message .*el.*
add_label app $kubernetes['labels']['app']
label_field color
label_field shape
Name log_to_metrics
Match dummy.log*
Tag test_metric
Metric_Mode histogram
Metric_Name current_duration
Metric_Description This metric shows the request duration
Value_Field duration
Kubernetes_Mode on
Regex message .*el.*
Add_Label app $kubernetes['labels']['app']
Label_Field color
Label_Field shape

[OUTPUT]
name prometheus_exporter
match *
host 0.0.0.0
port 9999
Name prometheus_exporter
Match *
Host 0.0.0.0
Port 9999
```

{% endtab %}
Expand Down Expand Up @@ -464,8 +464,8 @@ pipeline:

```text
[SERVICE]
flush 1
log_level info
Flush 1
Log_Level info

[INPUT]
Name dummy
Expand All @@ -478,31 +478,31 @@ pipeline:
Tag dummy.log2

[FILTER]
name log_to_metrics
match dummy.log*
tag test_metric
metric_mode histogram
metric_name current_duration
metric_description This metric shows the HTTP request duration as histogram in milliseconds
value_field duration
kubernetes_mode on
bucket 1
bucket 5
bucket 10
bucket 50
bucket 100
bucket 250
bucket 500
bucket 1000
regex message .*el.*
label_field color
label_field shape
Name log_to_metrics
Match dummy.log*
Tag test_metric
Metric_Mode histogram
Metric_Name current_duration
Metric_Description This metric shows the HTTP request duration as histogram in milliseconds
Value_Field duration
Kubernetes_Mode on
Bucket 1
Bucket 5
Bucket 10
Bucket 50
Bucket 100
Bucket 250
Bucket 500
Bucket 1000
Regex message .*el.*
Label_Field color
Label_Field shape

[OUTPUT]
name prometheus_exporter
match *
host 0.0.0.0
port 9999
Name prometheus_exporter
Match *
Host 0.0.0.0
Port 9999
```

{% endtab %}
Expand Down
14 changes: 7 additions & 7 deletions pipeline/filters/lua.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,8 @@ pipeline:
[FILTER]
Name lua
Match *
script test.lua
call cb_print
Script test.lua
Call cb_print

[OUTPUT]
Name null
Expand Down Expand Up @@ -316,9 +316,9 @@ pipeline:

```text
[SERVICE]
flush 1
daemon off
log_level debug
Flush 1
Daemon off
Log_Level debug

[INPUT]
Name random
Expand All @@ -329,7 +329,7 @@ pipeline:
Name lua
Match *
Call append_tag
code function append_tag(tag, timestamp, record) new_record = record new_record["tag"] = tag return 1, timestamp, new_record end
Code function append_tag(tag, timestamp, record) new_record = record new_record["tag"] = tag return 1, timestamp, new_record end

[OUTPUT]
Name stdout
Expand Down Expand Up @@ -556,7 +556,7 @@ pipeline:
[INPUT]
Name tail
Path /var/log/containers/*_istio-proxy-*.log
multiline.parser docker, cri
Multiline.Parser docker, cri
Tag istio.*
Mem_Buf_Limit 64MB
Skip_Long_Lines Off
Expand Down
Loading
Loading