Support Object Lock feature

[moroine]

Is your feature request related to a problem? Please describe.

In order to prevent removal of logs, it would be nice to add support for object lock. This would help with compliance, preventing anyone to be able to remove logs.

Describe the solution you'd like

Support new options such as:

object-lock-mode "COMPLIANCE" | "GOVERNANCE"
object-lock-retention 30d | 1y

https://docs.aws.amazon.com/cli/latest/reference/s3api/put-object-lock-configuration.html

Describe alternatives you've considered

Setting a default retention policy on the bucket itself, but nothing prevent it from being modified afterward.

Additional context

No response

[Watson1978]

@moroine
Thanks for the proposal! I'd like to understand the use case better, since this may already be doable on the S3 side.

Enabling Object Lock with a default retention on the bucket locks every uploaded object at PUT time without any plugin change, and already-locked objects aren't affected by later changes to the default. The concern about the default being modified can also be handled by denying s3:PutBucketObjectLockConfiguration via IAM/SCP, so only a limited role can change it.

Could you share where these fall short for you? Specifically:

1. Is the fluentd config managed more strictly than the bucket itself (e.g. IaC/CI), so setting the lock app-side is genuinely harder to tamper with than the bucket config?
2. Do you need different retention per object/data type within one bucket?

That would help us decide whether this adds real defense-in-depth beyond bucket-level config + IAM. Thanks!

[moroine]

Hi,

Yeah, that's mainly for a defense in depth. IAM configuration can be hard to manage, and scenario where an admin get his credentials stolen then this could be use to modify IAM policies and add let's say a lifecycle removal policy.

As we use this plugin to persist long-time the logs, then the server will directly use the COMPLIANCE mode so even an admin won't be able to delete neither the logs nor the bucket.

But I agree that's an extreme use case ^^ I was just wondering if this could make sense.

[Watson1978]

Thanks, that makes the intent much clearer.

Two things to align on first:

1. The lifecycle attack doesn't actually work against locked logs. Object Lock retention overrides Lifecycle expiration — a lifecycle rule can't delete an object version that's still within its retention period (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-managing.html). So a stolen-admin credential can't wipe logs that are already locked, no matter how the lock was applied.

2. For already-written logs, a bucket default retention in COMPLIANCE mode is equivalent to locking per-object from the plugin: in both cases no one (incl. root) can delete the objects, and the bucket can't be deleted while locked objects remain.

The one genuine delta for doing it plugin-side is independence from the bucket's default config for future writes — if the default gets changed/removed, app-side locking keeps protecting new objects. That's real defense-in-depth, though it's also closable by denying s3:PutBucketObjectLockConfiguration (ideally via an Organizations SCP a single compromised account admin can't override).

So my recommendation for most users would be bucket default retention + an SCP/IAM deny — it achieves the goal without extra config surface, and I'd want to document that as the primary approach.

That said, I do see the value of the plugin asserting the lock itself when the fluentd config is managed more strictly than the bucket (IaC/CI), so I'm not opposed to adding it as opt-in (defaulting to off).

If you'd like to pursue that, a PR is welcome — the main things I'd want are: clear docs that the bucket needs Object Lock + versioning enabled, and an explicit warning that COMPLIANCE is irreversible (objects can't be deleted until expiry even by mistake or for cost), which is especially important for long-retention logs.

Does that work for you?

[moroine]

Yup, that works for me !

I agree the scenario is very hard to hit. And yeah this should never become default on !!

I've never written ruby, but looks like the change is very small. Do you have any existing doc on how to test / run locally the plugin ?

[Watson1978]

Awesome, glad to have you give it a shot! Don't worry about the Ruby side — the change is small and the existing tests will guide you. Here's the local setup.

Prereqs

A recent Ruby and Bundler (gem install bundler).

Setup & run tests

git clone https://github.com/<your-fork>/fluent-plugin-s3.git
cd fluent-plugin-s3
bundle install
bundle exec rake test        # runs the whole suite

Run just the output-plugin tests:

bundle exec ruby -Ilib -Itest test/test_out_s3.rb
# single test method:
bundle exec ruby -Ilib -Itest test/test_out_s3.rb -n test_your_new_case

Write tests

You don't need a real S3 bucket or AWS credentials.
The tests stub the AWS SDK client (via the rr mocking lib), so they assert on the parameters the plugin would send to S3.
You'd add a test that sets your new object_lock_* options and checks that the put_object call receives the matching object_lock_mode / object_lock_retain_until_date params.

Where things live

• Output plugin code: lib/fluent/plugin/out_s3.rb (look for the config_param declarations near the top, and the put_object call in the write path)
• Tests: test/test_out_s3.rb