Skip to content

x-pack/filebeat/input/httpjson: add httpjson-to-cel Redirector#49614

Draft
efd6 wants to merge 4 commits intoelastic:mainfrom
efd6:id6318-director-httpjson
Draft

x-pack/filebeat/input/httpjson: add httpjson-to-cel Redirector#49614
efd6 wants to merge 4 commits intoelastic:mainfrom
efd6:id6318-director-httpjson

Conversation

@efd6
Copy link
Copy Markdown
Contributor

@efd6 efd6 commented Mar 23, 2026 •
edited
Loading

Proposed commit message

x-pack/filebeat/input/httpjson: add httpjson-to-cel Redirector

Implement the Redirector interface on httpjson's InputManager so
that inputs with run_as_cel: true and a cel.program are transparently
redirected to the cel input type.

convertHttpjsonToCel translates shared fields (interval, URL, auth,
retry, redirect, keep_alive, tracer, TLS, proxy, timeout), cel-
namespaced fields (program, max_executions, state, regexp, xsd,
redact), and passthrough fields (id) into a cel input config.

migrateCursor reads the httpjson cursor from the persistent store
during Redirect and injects it into the translated config's
state.cursor so that the cel input continues from where httpjson
left off. Direct store writes are not viable because the Loader
initialises all plugin managers' in-memory state tables before any
Configure call where Redirect runs. The httpjson store entry is
preserved for rollback safety.

Note

Best reviewed commitwise.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
  • I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@efd6 efd6 self-assigned this Mar 23, 2026
@efd6 efd6 added enhancement Filebeat Filebeat backport-skip Skip notification from the automated backport with mergify Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team Team:Security-Service Integrations Security Service Integrations Team labels Mar 23, 2026
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Mar 23, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 23, 2026

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 23, 2026 •
edited
Loading

Vale Linting Results

Summary: 2 suggestions found

💡 Suggestions (2)
File Line Rule Message
docs/reference/filebeat/filebeat-input-httpjson.md 1772 Elastic.WordChoice Consider using 'run, start' instead of 'execute', unless the term is in the UI.
docs/reference/filebeat/filebeat-input-httpjson.md 1781 Elastic.WordChoice Consider using 'can, might' instead of 'May', unless the term is in the UI.

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 23, 2026 •
edited
Loading

🔍 Preview links for changed docs

@efd6 efd6 force-pushed the id6318-director-httpjson branch from 4cbe48d to 78c339c Compare March 23, 2026 22:48
@efd6
Copy link
Copy Markdown
Contributor Author

efd6 commented Mar 23, 2026 •
edited
Loading

re: vale linter

  1. Spelling changed to ASE.
  2. "execute" retained as it is conventional language for the input.
  3. The suggestion is semantically incorrect; the sense here is permission, not capacity or possibility.

@efd6 efd6 force-pushed the id6318-director-httpjson branch from 78c339c to 118dc97 Compare March 23, 2026 23:08
@efd6 efd6 force-pushed the id6318-director-httpjson branch from 118dc97 to cdb29b4 Compare March 23, 2026 23:20
@efd6 efd6 force-pushed the id6318-director-httpjson branch from cdb29b4 to f7641f4 Compare March 24, 2026 00:42
efd6 added 4 commits March 24, 2026 11:22
@efd6
filebeat/input/{v2,logv2}: add Redirector interface for general input…
acbaa75 
… migration

Add an optional Redirector interface to the v2 input system so the
Loader can transparently redirect one input type to another. This
replaces the bespoke approach where logv2 directly imported and
instantiated the filestream plugin.

The Loader checks for Redirector before calling Create. If Redirect
returns a non-empty target type, the Loader resolves the target from
its plugin registry and calls its Create with the translated config.
Only one redirect hop is allowed. Delete follows the same redirect
path so CheckConfig cleanup operates on the correct target.

Refactor the logv2 manager to implement Redirector instead of holding
a direct reference to the filestream manager. This removes logv2's
import of the filestream package, eliminates a duplicate filestream
manager instance (and its redundant Init call), and makes the pattern
reusable for future V2-to-V2 migrations such as httpjson to cel.
@efd6
x-pack/filebeat/input/httpjson: add httpjson-to-cel Redirector
a8a815d 
Implement the Redirector interface on httpjson's InputManager so
that when run_as_cel is true and cel.program is present, the Loader
redirects to the cel input with a translated config.

convertHttpjsonToCel maps the full set of shared fields between the
two inputs: interval, auth, request.url to resource.url, retry,
redirect, keep_alive, tracer, and inline transport settings (ssl,
timeout, proxy_url, proxy_headers, proxy_disable,
idle_connection_timeout). cel-namespaced fields (program, state,
regexp, xsd, max_executions, redact) are extracted to their
top-level equivalents. Fields specific to httpjson's declarative
config (method, body, transforms, response, cursor, chain) are
excluded since the CEL program handles them directly.

max_executions uses Int/SetInt to match its *int type in the cel
config rather than round-tripping through strings.
@efd6
x-pack/filebeat/input/httpjson: migrate cursor state during redirect
fcb8de0 
Read the httpjson cursor from the persistent store during Redirect
and inject it into the translated cel config's state.cursor field.
The cel input uses config.State["cursor"] as the initial value when
no stored cursor exists, so the first run after redirect continues
from where httpjson left off. Subsequent runs use the cel-stored
cursor and ignore the injected value.

Direct store writes are not viable because the Loader calls Init on
all plugin managers (loading the cel cursor manager's in-memory
table) before any Configure call where Redirect runs. A cel:: key
written to the persistent store during Configure would not appear
in the in-memory table, and findOrCreate would overwrite it with an
empty resource on first flush.

The httpjson store entry is preserved for rollback safety.
@efd6
docs/reference/filebeat: document run_as_cel and cel.* options
d2a44e8 
Add reference documentation for the run_as_cel option and the
cel.program, cel.state, cel.max_executions, cel.regexp, cel.xsd,
and cel.redact configuration fields. Includes a field mapping table,
a note on cursor state migration, and a worked example.
@efd6 efd6 force-pushed the id6318-director-httpjson branch from f7641f4 to d2a44e8 Compare March 24, 2026 00:53
@efd6 efd6 mentioned this pull request Mar 24, 2026
Open
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@efd6 efd6

Labels

backport-skip Skip notification from the automated backport with mergify enhancement Filebeat Filebeat Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team Team:Security-Service Integrations Security Service Integrations Team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@efd6