Skip to content

feat(sources): louder pipeline-embedded sources: deprecation, hide from docs#140

Merged
mostafa merged 5 commits into
mainfrom
feat/dynamic-sources-hide-pipeline
May 26, 2026
Merged

feat(sources): louder pipeline-embedded sources: deprecation, hide from docs#140
mostafa merged 5 commits into
mainfrom
feat/dynamic-sources-hide-pipeline

Conversation

@mostafa
Copy link
Copy Markdown
Member

@mostafa mostafa commented May 21, 2026

Closes #136.

Summary

  • Adds a loud stderr warning: line every time a pipeline file declares an inline sources: block. The structured tracing::warn! event stays in place for daemons that already parse it (now enriched with a path field).
  • Centralises the emission in a new public rsigma_runtime::warn_pipeline_inline_sources helper, so two paths share one warning text and one process-wide canonical-path dedup set:
    • CLI startup, via rsigma-cli's load_pipelines and pipeline resolve. Every command that loads a pipeline now surfaces the warning consistently (engine eval, engine daemon, rule validate, rule fields, backend convert, pipeline resolve).
    • Daemon hot-reload, via rsigma-runtime's RuntimeEngine::load_rules -> reload_pipelines. A SIGHUP, file-watcher event, or POST /api/v1/reload that re-reads a deprecated pipeline now surfaces the warning even though the reload path does not go back through the CLI's load_pipelines. Library consumers that drive RuntimeEngine themselves inherit the same behaviour.
  • The dedup set lives inside the runtime helper, so the daemon's hot-reload loop is a no-op after the first warning per canonical pipeline path. The CLI's load_pipelines shares the same set.
  • Rewrites every doc-site and README example for dynamic sources to use the external --source form. The pipeline-embedded form now lives only in a short "Deprecated" callout that points operators at rsigma rule migrate-sources and the v1.0 removal issue (sources: remove deprecated pipeline-level sources: (v1.0) #137).

Deprecation cycle

Stage Behaviour Tracked in
Phase 1 (v0.13.0, shipped #135) tracing::warn! at parse time, recommends --source #135
Phase 3 (this PR) adds stderr warning: line, plumbs warning through daemon hot-reload, all doc examples switch to --source #136
Phase 4 (v1.0) hard parse error, Pipeline.sources removed #137

This mirrors the CLI command groups deprecation cycle precedent: each post-deprecation step gets a tracking issue opened at the same time the deprecation lands, so the project has a public, dated commitment for the follow-up work.

Code surface

  • crates/rsigma-runtime/src/pipeline_deprecation.rs (new): hosts warn_pipeline_inline_sources, the process-wide canonical-path dedup set (OnceLock<Mutex<HashSet<PathBuf>>>), and two test-only helpers (reset_inline_sources_dedup_for_tests, tests_only_snapshot). Re-exported from rsigma_runtime crate root.
  • crates/rsigma-runtime/src/engine.rs: reload_pipelines now calls the helper for every reloaded pipeline that still declares an inline sources: block, so the daemon's hot-reload path covers the deprecation. Three new unit tests (load_rules_surfaces_inline_sources_deprecation_through_runtime, load_rules_does_not_warn_when_pipeline_has_no_inline_sources, hot_reload_dedups_inline_sources_warning_for_same_pipeline_path) drive RuntimeEngine::load_rules directly to pin the contract.
  • crates/rsigma-cli/src/main.rs: removes the duplicate CLI-local helper and dedup static; load_pipelines now calls into rsigma_runtime::warn_pipeline_inline_sources directly. The HashSet/Mutex/OnceLock/Path imports it brought along go too.
  • crates/rsigma-cli/src/commands/daemon.rs: the inline tracing::warn! loop that ran at daemon startup is removed in favour of the helper. A short comment in its place explains where the warning now lives.
  • crates/rsigma-cli/src/commands/resolve.rs: pipeline resolve parses pipeline files directly, so it calls rsigma_runtime::warn_pipeline_inline_sources inline.
  • crates/rsigma-cli/tests/cli_sources_deprecation.rs: integration-test suite with six subprocess-spawned tests pinning the stderr emission across rule validate / engine eval / pipeline resolve, the dedup invariant when the same pipeline is passed twice via -p, the negative case, and the migration-command suggestion (the warning embeds the actual pipeline path so the suggested rsigma rule migrate-sources invocation is copy-pasteable).

Docs and READMEs

  • docs/reference/dynamic-sources.md: collapses the two declaration sub-sections into a single "Source declaration" section whose body is the external-file shape. The deprecated form is a one-paragraph admonition.
  • docs/guide/processing-pipelines.md: same treatment in the dynamic-pipelines walkthrough; the sources file and the pipeline file are shown side by side with a single bash invocation that wires them together.
  • docs/guide/enrichers.md: enrich_ip_employee and enrich_cve_kev recipes now declare backing sources in sources.yml loaded via --source. Fixes the bogus mapping-form sources: block that snuck into the original copy.
  • docs/cli/engine/daemon.md: clarifies the uniqueness contract on --source and points at the deprecation note.
  • README.md (root) and crates/rsigma-cli/README.md: examples and recipe catalog mirror the docs-site form. The CLI README's recipe-catalog refresh values switch from the unsupported { interval: ... } mapping form to the literal-duration syntax (1h, 24h) that the parser actually accepts.
  • crates/rsigma-runtime/README.md: DaemonSourceRegistry description marks pipeline-embedded sources as deprecated rather than co-equal.
  • CHANGELOG.md: [Unreleased] entry covers the louder warning, the runtime/CLI helper split, the daemon hot-reload coverage, the doc/README sweep, and the new tests.

Test plan

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cargo test --workspace --all-features (1502 passed, 46 suites; +5 over pre-rebase baseline for the three new engine tests and two new pipeline_deprecation tests)
  • mkdocs build --strict
  • cargo test -p rsigma --test cli_sources_deprecation (6 passed; unchanged surface, now exercising the runtime helper through the CLI)
  • cargo test -p rsigma-runtime --lib engine::tests (3 passed; hot-reload path pinned)
  • cargo test -p rsigma-runtime --lib pipeline_deprecation::tests (2 passed; dedup primitive pinned)

Backward compatibility

Pipeline files that declare an inline sources: block continue to load successfully. The only behaviour change is the extra warning: line on stderr at every first load per canonical pipeline path, and the doc copy no longer documenting the embedded form as a recommended pattern.

mostafa added a commit that referenced this pull request May 21, 2026
#140)

Documents the louder stderr warning, the unified emission via
'load_pipelines', the canonical-path dedup, and the doc-site /
README sweep. The entry sits at the top of '[Unreleased]' since
this is the most recent change.
@mostafa mostafa marked this pull request as draft May 21, 2026 21:49
@mostafa
Copy link
Copy Markdown
Member Author

mostafa commented May 21, 2026

Held for post-v0.13.0 merge. Phases 1+2 of the detached-dynamic-sources cycle are on main but have not shipped yet (still in [Unreleased] as of this PR). Merging this Phase 3 PR before v0.13.0 cuts would collapse Phases 1+2+3 into a single release and skip the "visible-deprecated only" intermediate beat that the cycle is designed to give operators.

Marking as draft until v0.13.0 ships. After that, rebase onto post-v0.13.0 main and merge.

Rebase note for whoever picks this up after v0.13.0: the CHANGELOG commit (4af48b8) anchors its new entry above ### Deprecated CLI aliases hidden from --help (#125). When v0.13.0 cuts the current [Unreleased] section into ## [0.13.0] - <date>, that anchor moves with it. The rebase will silently land the Phase 3 entry under v0.13.0 instead of the new (empty) [Unreleased]. Fix is a small one-line move to relocate the entry into the new [Unreleased] block before merging.

cc: tracks #136.

mostafa added a commit that referenced this pull request May 25, 2026
The CHANGELOG follows the existing convention of pointing at the
GitHub PR/issue that ships the change (e.g. TLS PR #146, detached
sources PR #140). Roadmap item numbers stay in the plan files; the
release-facing surface uses the PR ID so readers can click through to
the diff and review thread.
mostafa added a commit that referenced this pull request May 25, 2026
The CHANGELOG follows the existing convention of pointing at the
GitHub PR/issue that ships the change (e.g. TLS PR #146, detached
sources PR #140). Roadmap item numbers stay in the plan files; the
release-facing surface uses the PR ID so readers can click through to
the diff and review thread.
@mostafa mostafa marked this pull request as ready for review May 26, 2026 21:31
mostafa added 5 commits May 26, 2026 23:37
Phase 3 of the detached-dynamic-sources deprecation cycle (issue #136):
the parser still accepts an inline 'sources:' block in a pipeline file
but now also prints a 'warning:' line on stderr in addition to the
existing 'tracing::warn!' event. The structured warning is unchanged so
log aggregators that already parse it keep working.

The emission moves out of 'commands/daemon.rs' into a new
'warn_pipeline_inline_sources' helper in 'main.rs' that 'load_pipelines'
calls for every pipeline file. This covers every CLI entry point that
loads a pipeline (engine eval, engine daemon, rule validate, rule
fields, backend convert) with one consistent message. The 'pipeline
resolve' command parses pipeline files directly, so it calls the helper
too.

Per-canonical-path deduplication via a 'OnceLock<Mutex<HashSet<PathBuf>>>'
keeps daemon hot-reloads (SIGHUP, file-watcher event, POST /api/v1/reload)
from re-spamming the message on every reload. One-shot commands only
load each pipeline once, so the dedup set is effectively a noop for
them.

Tests live in a new 'cli_sources_deprecation.rs' integration suite that
pins:

- The warning fires on 'rule validate -p <pipeline-with-sources>'.
- The warning fires on 'engine eval -p <pipeline-with-sources>'.
- The warning fires on 'pipeline resolve -p <pipeline-with-sources>'.
- A pipeline without inline sources does NOT warn.
- The warning fires exactly once when the same pipeline file is passed
  twice via '-p' (dedup invariant).
- The warning embeds the actual pipeline path so an operator can
  copy-paste the suggested 'rsigma rule migrate-sources' invocation.

Tracked in #136. Phase 4 (#137) replaces the warning with a hard parse
error at v1.0.
Phase 3 of the detached-dynamic-sources deprecation cycle: every doc
example for dynamic sources now declares them in a standalone YAML file
loaded via '--source'. The pipeline-embedded form is still mentioned but
only as a short 'Deprecated' callout that points operators at
'rsigma rule migrate-sources' and the v1.0 removal issue (#137).

Doc changes:

- docs/reference/dynamic-sources.md: collapses the two declaration
  sub-sections ('External source files (recommended)' /
  'Pipeline-embedded sources (deprecated)') into a single 'Source
  declaration' section whose body documents the external-file shape.
  The deprecated form is now a one-paragraph admonition. The template
  substitution example is split into separate 'sources.yml' and
  'pipeline.yml' code blocks so the reader sees the recommended
  two-file layout in one glance.
- docs/guide/processing-pipelines.md: same treatment. The dynamic-
  pipelines walkthrough now shows the external sources file first and
  the pipeline file second, with a single bash invocation that wires
  them together.
- docs/guide/enrichers.md: the 'enrich_ip_employee' and 'enrich_cve_kev'
  recipes now declare their backing sources in 'sources.yml' loaded
  via '--source' instead of using the bogus mapping-form 'sources:'
  block that snuck into the original copy.
- docs/cli/engine/daemon.md: the '--source' flag row clarifies that
  pipeline-embedded 'sources:' blocks are also subject to the
  uniqueness check, with a forward pointer to the deprecation note.

README changes (mirror the doc site, per CONTRIBUTING.md):

- README.md (root): swaps the pipeline-embedded example for the
  external-file form, drops the 'pipeline-embedded sources, deprecated'
  bash example, and references the v1.0 removal issue (#137).
- crates/rsigma-cli/README.md: collapses the two-subsection structure
  in 'Source declaration' the same way as the reference doc. Recipe
  catalog entries ('enrich_ip_employee', 'enrich_ip_geoip',
  'enrich_cve_kev') now use the external-file shape and the legitimate
  duration syntax ('1h', '24h') instead of the prior '{ interval: ... }'
  form, which the parser does not accept.
- crates/rsigma-runtime/README.md: 'DaemonSourceRegistry' description
  no longer presents pipeline-embedded sources as a co-equal site,
  marking them as deprecated and removed-in-v1.0 instead.

'mkdocs build --strict' passes; the doc site builds with zero warnings.
#140)

Documents the louder stderr warning, the unified emission via
'load_pipelines', the canonical-path dedup, and the doc-site /
README sweep. The entry sits at the top of '[Unreleased]' since
this is the most recent change.
…cation

The Phase 1 louder-warning commit (bdb4e52) kept the helper in
rsigma-cli's main.rs, which meant only the CLI startup path
(load_pipelines + pipeline resolve) ever called it. Daemon hot-reloads
go through RuntimeEngine::load_rules -> reload_pipelines in
rsigma-runtime, which never re-entered the helper, so a pipeline that
acquired an inline sources: block between reloads (or the reload-time
"someone forgot to migrate" reminder) was silently swallowed.

Move warn_pipeline_inline_sources into a new
crates/rsigma-runtime/src/pipeline_deprecation.rs module and re-export
it from the runtime crate root. The CLI's load_pipelines and resolve
helpers now call into the runtime helper, and reload_pipelines does the
same on every reload. The process-wide canonical-path dedup set lives
inside the runtime module, so both paths share one set; the daemon's
hot-reload loop is no-op after the first warning per pipeline path,
which is the dedup behaviour the original PR description already
promised but didn't actually deliver.

Library consumers that drive RuntimeEngine themselves inherit the
warning behaviour without needing to wire anything up.

Tests:
- Three unit tests in engine.rs drive RuntimeEngine::load_rules
  end-to-end: inline sources land in the dedup set, clean pipelines
  do not, and a hot-reload (second load_rules call) leaves the set
  unchanged. They share a per-binary mutex to serialize access to the
  process-wide set.
- Two unit tests in pipeline_deprecation.rs cover the dedup primitive
  directly (same canonical path skipped, distinct paths recorded
  separately).
- The existing cli_sources_deprecation.rs integration suite (6 tests)
  still passes unchanged; it exercises the CLI surface via subprocess
  spawns, which is layered above this change.

The helper's doc comment drops the speculative "v0.14" version label
for the current release in favour of phase-numbered cycle anchors
(Phase 1 / Phase 3 / Phase 4), since release tagging is decided
separately from the deprecation plan.
Refreshes the Phase 3 [Unreleased] entry to accurately describe the
shared rsigma_runtime::warn_pipeline_inline_sources helper, the two
call sites (CLI startup + daemon hot-reload via RuntimeEngine),
and the new runtime-side unit tests that pin the hot-reload coverage.
Drops the prior claim that the dedup alone covered hot-reloads (true
now that reload_pipelines calls the helper; previously misleading
because the runtime path didn't reach the helper at all).
@mostafa mostafa force-pushed the feat/dynamic-sources-hide-pipeline branch from e3e2a88 to cb39d0c Compare May 26, 2026 22:02
@mostafa mostafa changed the title sources: hide deprecated pipeline-embedded sources: from docs and make warning louder feat(sources): louder pipeline-embedded sources: deprecation, hide from docs May 26, 2026
@mostafa mostafa merged commit 4cc356a into main May 26, 2026
12 checks passed
@mostafa mostafa deleted the feat/dynamic-sources-hide-pipeline branch May 26, 2026 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sources: hide deprecated pipeline-level sources: from docs and examples

1 participant