Watch TLS certificates and reload on changes

[andsens]

See #9718 and #1678
Is your feature request related to a problem? Please describe.
Short lived client certificates are the norm. fluent-bit does not reload TLS certificates when they are renewed and simply fails at shipping logs.

Describe the solution you'd like
fluent-bit should reload the TLS configuration when any of the referenced files change.

Additional context
People are doing all kinds of workarounds (e.g. fluent/helm-charts#578) to do something that fluent-bit should do natively. Most other tooling I have encountered has done this for years. Two issues have been created for this over the last 6 years and they just went stale.
Please implement this.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[andsens]

Not stale

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[andsens]

Not stale

[tiagoinaba]

Hey, I want to work on this issue. Can I get some guidance on what the configuration for watching TLS should look like? Should it be default behavior, or should I add a new configuration field?

[andsens]

I have a hard time recalling any tls configuration where you even have the ability to disable automatic reloading. I don't see when that would be something you need. So I would recommend not implementing a setting like that.

[cosmo0920]

Hey, I tried to implement watching and reloading certificates capability. Is it fitting for your usage case? #11950

[tiagoinaba]

Sorry for the late response. Yes, I've been running this in some of our production containers for the last few days and it works perfectly. Thanks a lot!