[Filestream] Ship data to logs.ecs instead of logs

[mohamedhamed-ahmed]

Summary

As part of some recent work in 9.4 logs endpoint split to logs.ecs and logs.otel, the logs stream is now marked as deprecated in favor of 2 new streams logs.ecs and logs.otel.

With this change in place, it would be best to have the Custom Logs(filestream) integration updated to reflect this change as well.

Acceptance Criteria

1. Change the wordings under the Custom Logs (filestream) settings to indicate shipping the data to logs.ecs instead of logs

2. Modify the underlying stream to actually be logs.ecs

Questions

1. How would this behave for old users with this setting already enabled on the old logs stream, should we show a warning or deprecation badge?

Checklist for integrations:

• Filestream Use the logs.ecs index instead of logs #17909
• UDP
• TCP

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[rdner]

I've checked and confirmed a couple of things:

1. The previously made change in Beats allows us to set any index, so setting logs.ecs in the integration should just work.
2. The Fleet permission change also enabled all logs.* indices. The documentation on the UI also remains reasonable to me and does not require a change.

How would this behave for old users with this setting already enabled on the old logs stream, should we show a warning or deprecation badge?

After making this change in the integration we will have to set the version constraint to ^9.4.0.

This means the users who upgrade to >= 9.4.0 will automatically upgrade the integration package as well and that will automatically change the target stream from logs to logs.ecs because it's a part of the template which would be re-rendered.

cc @kpollich in case I missed something on the Fleet side.

[mohamedhamed-ahmed]

This means the users who upgrade to >= 9.4.0 will automatically upgrade the integration package as well and that will automatically change the target stream from logs to logs.ecs because it's a part of the template which would be re-rendered.

The concern I have here is that these users will probably haven't yet re-enabled streams to create the 2 new ones. So making the integration here automatically send to logs.ecs might break things for them.

cc: @flash1293

[rdner]

@mohamedhamed-ahmed I expected this to be automatically handled on the Elasticsearch side.

While upgrading to 9.4 we should automatically move the existing logs stream to logs.ecs and maintain an alias logs for it.

[flash1293]

Do we have telemetry about existing usage? I expect it to be fairly low. I guess we can mention this in the release notes, seems like the appropriate level for a feature in this stage.

[mohamedhamed-ahmed]

I expected this to be automatically handled on the Elasticsearch side.

@rdner its not an automatic thing, if an old user had streams already enabled and are using the logs stream, they will see the toggle button disabled on kibana with a warning stating that they need to re-enable it to use the 2 new streams, if they didn't do so they are still on the logs one and the 2 new streams won't exist at this point.

seems like the appropriate level for a feature in this stage.

@flash1293 you suggest that in this case we automatically ship data from the integration to logs.ecs ? if so, then the logs.ecs will be created in this case if my understanding is correct and then these users won't be able to re-enable wired streams because of the conflict state they end up with.

[flash1293]

@mohamedhamed-ahmed right, but we would mention in the release notes how they need to upgrade:

• Upgrade stack
• Re-enable logs.ecs
• Upgrade integration

[mohamedhamed-ahmed]

@flash1293 sounds good to me as long as they can upgrade the integration separately after re-enabling wired streams, I am not sure how this part of integration upgrade works TBH.