Log collection through Filebeat daemonset

[solsson]

Pending native kafka support for fluent-bit (fluent/fluent-bit#94 or a go plugin) I'm evaluating filebeat which has kafka-support as an intermediate step for aggregation to ELK.

Candidate for merge to v3.0.0 (#84), with advantages over the tail-based solution:

• Resumes at previous position
• Discovers new files without restart
• Adds (some) kubernetes metadata.
• Every message contains source, compared to the delimeters approach from tail.
  • You'd can use jq to do like kubectl logs -f but for arbitrary filters on pods.

Sadly the kubernetes object lacks node name (zone name would be great too), a typical record having:

    "offset": 8770457,
    "source": "/var/lib/docker/containers/671a46ea4b400c73424a9f438fa1102f83b4db2d9fe3e229f61824df953b5b26/671a46ea4b400c73424a9f438fa1102f83b4db2d9fe3e229f61824df953b5b26-json.log",
    "kubernetes": {
      "container": {
        "name": "testcase"
      },
      "labels": {
        "test-target": "logs-streaming-raw",
        "pod-template-hash": "1087956622",
        "test-type": "readiness"
      },
      "namespace": "test-kafka",
      "pod": {
        "name": "logs-raw-54dcf9bb66-898h5"
      }
    }

[solsson]

Contrary to what 504911a#diff-81195533b894e72927dc12f0d5d58b35L14 suggests we still get lots and lots of INFO Same file found as symlink and originap. Skipping file: /var/lib/docker/containers/fa80887d5c1a580a35c298708c610d4c098e2b173d48d60172440303dd6df364/fa80887d5c1a580a35c298708c610d4c098e2b173d48d60172440303dd6df364-json.log in logs.

Combined with the fact that that 504911a#diff-81195533b894e72927dc12f0d5d58b35R13 breaks e04a403 we get lots and lots of logs.

[solsson]

Just need to note a few ideas, so I don't have to implement them now :)

• We'd prefer to have node name, and/or pod name, as key. Ordering is unimportant between pods, but within pods it's quite essential.
• Can we de-duplicate entries? Quite interesting for the raw impl of Addon: Collect logs to Kafka topic #40.
  • How is state storage implemented in filebeat? Does it produce first and then record position? Or does it record position when it has read a batch from the file, then produce?

[solsson]

I find Fluent Bit more promising. See #131.