feat(sources): louder pipeline-embedded sources: deprecation, hide from docs#140
Conversation
#140) Documents the louder stderr warning, the unified emission via 'load_pipelines', the canonical-path dedup, and the doc-site / README sweep. The entry sits at the top of '[Unreleased]' since this is the most recent change.
|
Held for post-v0.13.0 merge. Phases 1+2 of the detached-dynamic-sources cycle are on Marking as draft until v0.13.0 ships. After that, rebase onto post-v0.13.0 Rebase note for whoever picks this up after v0.13.0: the CHANGELOG commit ( cc: tracks #136. |
Phase 3 of the detached-dynamic-sources deprecation cycle (issue #136): the parser still accepts an inline 'sources:' block in a pipeline file but now also prints a 'warning:' line on stderr in addition to the existing 'tracing::warn!' event. The structured warning is unchanged so log aggregators that already parse it keep working. The emission moves out of 'commands/daemon.rs' into a new 'warn_pipeline_inline_sources' helper in 'main.rs' that 'load_pipelines' calls for every pipeline file. This covers every CLI entry point that loads a pipeline (engine eval, engine daemon, rule validate, rule fields, backend convert) with one consistent message. The 'pipeline resolve' command parses pipeline files directly, so it calls the helper too. Per-canonical-path deduplication via a 'OnceLock<Mutex<HashSet<PathBuf>>>' keeps daemon hot-reloads (SIGHUP, file-watcher event, POST /api/v1/reload) from re-spamming the message on every reload. One-shot commands only load each pipeline once, so the dedup set is effectively a noop for them. Tests live in a new 'cli_sources_deprecation.rs' integration suite that pins: - The warning fires on 'rule validate -p <pipeline-with-sources>'. - The warning fires on 'engine eval -p <pipeline-with-sources>'. - The warning fires on 'pipeline resolve -p <pipeline-with-sources>'. - A pipeline without inline sources does NOT warn. - The warning fires exactly once when the same pipeline file is passed twice via '-p' (dedup invariant). - The warning embeds the actual pipeline path so an operator can copy-paste the suggested 'rsigma rule migrate-sources' invocation. Tracked in #136. Phase 4 (#137) replaces the warning with a hard parse error at v1.0.
Phase 3 of the detached-dynamic-sources deprecation cycle: every doc example for dynamic sources now declares them in a standalone YAML file loaded via '--source'. The pipeline-embedded form is still mentioned but only as a short 'Deprecated' callout that points operators at 'rsigma rule migrate-sources' and the v1.0 removal issue (#137). Doc changes: - docs/reference/dynamic-sources.md: collapses the two declaration sub-sections ('External source files (recommended)' / 'Pipeline-embedded sources (deprecated)') into a single 'Source declaration' section whose body documents the external-file shape. The deprecated form is now a one-paragraph admonition. The template substitution example is split into separate 'sources.yml' and 'pipeline.yml' code blocks so the reader sees the recommended two-file layout in one glance. - docs/guide/processing-pipelines.md: same treatment. The dynamic- pipelines walkthrough now shows the external sources file first and the pipeline file second, with a single bash invocation that wires them together. - docs/guide/enrichers.md: the 'enrich_ip_employee' and 'enrich_cve_kev' recipes now declare their backing sources in 'sources.yml' loaded via '--source' instead of using the bogus mapping-form 'sources:' block that snuck into the original copy. - docs/cli/engine/daemon.md: the '--source' flag row clarifies that pipeline-embedded 'sources:' blocks are also subject to the uniqueness check, with a forward pointer to the deprecation note. README changes (mirror the doc site, per CONTRIBUTING.md): - README.md (root): swaps the pipeline-embedded example for the external-file form, drops the 'pipeline-embedded sources, deprecated' bash example, and references the v1.0 removal issue (#137). - crates/rsigma-cli/README.md: collapses the two-subsection structure in 'Source declaration' the same way as the reference doc. Recipe catalog entries ('enrich_ip_employee', 'enrich_ip_geoip', 'enrich_cve_kev') now use the external-file shape and the legitimate duration syntax ('1h', '24h') instead of the prior '{ interval: ... }' form, which the parser does not accept. - crates/rsigma-runtime/README.md: 'DaemonSourceRegistry' description no longer presents pipeline-embedded sources as a co-equal site, marking them as deprecated and removed-in-v1.0 instead. 'mkdocs build --strict' passes; the doc site builds with zero warnings.
#140) Documents the louder stderr warning, the unified emission via 'load_pipelines', the canonical-path dedup, and the doc-site / README sweep. The entry sits at the top of '[Unreleased]' since this is the most recent change.
…cation The Phase 1 louder-warning commit (bdb4e52) kept the helper in rsigma-cli's main.rs, which meant only the CLI startup path (load_pipelines + pipeline resolve) ever called it. Daemon hot-reloads go through RuntimeEngine::load_rules -> reload_pipelines in rsigma-runtime, which never re-entered the helper, so a pipeline that acquired an inline sources: block between reloads (or the reload-time "someone forgot to migrate" reminder) was silently swallowed. Move warn_pipeline_inline_sources into a new crates/rsigma-runtime/src/pipeline_deprecation.rs module and re-export it from the runtime crate root. The CLI's load_pipelines and resolve helpers now call into the runtime helper, and reload_pipelines does the same on every reload. The process-wide canonical-path dedup set lives inside the runtime module, so both paths share one set; the daemon's hot-reload loop is no-op after the first warning per pipeline path, which is the dedup behaviour the original PR description already promised but didn't actually deliver. Library consumers that drive RuntimeEngine themselves inherit the warning behaviour without needing to wire anything up. Tests: - Three unit tests in engine.rs drive RuntimeEngine::load_rules end-to-end: inline sources land in the dedup set, clean pipelines do not, and a hot-reload (second load_rules call) leaves the set unchanged. They share a per-binary mutex to serialize access to the process-wide set. - Two unit tests in pipeline_deprecation.rs cover the dedup primitive directly (same canonical path skipped, distinct paths recorded separately). - The existing cli_sources_deprecation.rs integration suite (6 tests) still passes unchanged; it exercises the CLI surface via subprocess spawns, which is layered above this change. The helper's doc comment drops the speculative "v0.14" version label for the current release in favour of phase-numbered cycle anchors (Phase 1 / Phase 3 / Phase 4), since release tagging is decided separately from the deprecation plan.
Refreshes the Phase 3 [Unreleased] entry to accurately describe the shared rsigma_runtime::warn_pipeline_inline_sources helper, the two call sites (CLI startup + daemon hot-reload via RuntimeEngine), and the new runtime-side unit tests that pin the hot-reload coverage. Drops the prior claim that the dedup alone covered hot-reloads (true now that reload_pipelines calls the helper; previously misleading because the runtime path didn't reach the helper at all).
e3e2a88 to
cb39d0c
Compare
sources: deprecation, hide from docs
Closes #136.
Summary
warning:line every time a pipeline file declares an inlinesources:block. The structuredtracing::warn!event stays in place for daemons that already parse it (now enriched with apathfield).rsigma_runtime::warn_pipeline_inline_sourceshelper, so two paths share one warning text and one process-wide canonical-path dedup set:rsigma-cli'sload_pipelinesandpipeline resolve. Every command that loads a pipeline now surfaces the warning consistently (engine eval,engine daemon,rule validate,rule fields,backend convert,pipeline resolve).rsigma-runtime'sRuntimeEngine::load_rules->reload_pipelines. A SIGHUP, file-watcher event, orPOST /api/v1/reloadthat re-reads a deprecated pipeline now surfaces the warning even though the reload path does not go back through the CLI'sload_pipelines. Library consumers that driveRuntimeEnginethemselves inherit the same behaviour.load_pipelinesshares the same set.--sourceform. The pipeline-embedded form now lives only in a short "Deprecated" callout that points operators atrsigma rule migrate-sourcesand the v1.0 removal issue (sources: remove deprecated pipeline-level sources: (v1.0) #137).Deprecation cycle
tracing::warn!at parse time, recommends--sourcewarning:line, plumbs warning through daemon hot-reload, all doc examples switch to--sourcePipeline.sourcesremovedThis mirrors the CLI command groups deprecation cycle precedent: each post-deprecation step gets a tracking issue opened at the same time the deprecation lands, so the project has a public, dated commitment for the follow-up work.
Code surface
crates/rsigma-runtime/src/pipeline_deprecation.rs(new): hostswarn_pipeline_inline_sources, the process-wide canonical-path dedup set (OnceLock<Mutex<HashSet<PathBuf>>>), and two test-only helpers (reset_inline_sources_dedup_for_tests,tests_only_snapshot). Re-exported fromrsigma_runtimecrate root.crates/rsigma-runtime/src/engine.rs:reload_pipelinesnow calls the helper for every reloaded pipeline that still declares an inlinesources:block, so the daemon's hot-reload path covers the deprecation. Three new unit tests (load_rules_surfaces_inline_sources_deprecation_through_runtime,load_rules_does_not_warn_when_pipeline_has_no_inline_sources,hot_reload_dedups_inline_sources_warning_for_same_pipeline_path) driveRuntimeEngine::load_rulesdirectly to pin the contract.crates/rsigma-cli/src/main.rs: removes the duplicate CLI-local helper and dedup static;load_pipelinesnow calls intorsigma_runtime::warn_pipeline_inline_sourcesdirectly. TheHashSet/Mutex/OnceLock/Pathimports it brought along go too.crates/rsigma-cli/src/commands/daemon.rs: the inlinetracing::warn!loop that ran at daemon startup is removed in favour of the helper. A short comment in its place explains where the warning now lives.crates/rsigma-cli/src/commands/resolve.rs:pipeline resolveparses pipeline files directly, so it callsrsigma_runtime::warn_pipeline_inline_sourcesinline.crates/rsigma-cli/tests/cli_sources_deprecation.rs: integration-test suite with six subprocess-spawned tests pinning the stderr emission acrossrule validate/engine eval/pipeline resolve, the dedup invariant when the same pipeline is passed twice via-p, the negative case, and the migration-command suggestion (the warning embeds the actual pipeline path so the suggestedrsigma rule migrate-sourcesinvocation is copy-pasteable).Docs and READMEs
docs/reference/dynamic-sources.md: collapses the two declaration sub-sections into a single "Source declaration" section whose body is the external-file shape. The deprecated form is a one-paragraph admonition.docs/guide/processing-pipelines.md: same treatment in the dynamic-pipelines walkthrough; the sources file and the pipeline file are shown side by side with a single bash invocation that wires them together.docs/guide/enrichers.md:enrich_ip_employeeandenrich_cve_kevrecipes now declare backing sources insources.ymlloaded via--source. Fixes the bogus mapping-formsources:block that snuck into the original copy.docs/cli/engine/daemon.md: clarifies the uniqueness contract on--sourceand points at the deprecation note.README.md(root) andcrates/rsigma-cli/README.md: examples and recipe catalog mirror the docs-site form. The CLI README's recipe-catalog refresh values switch from the unsupported{ interval: ... }mapping form to the literal-duration syntax (1h,24h) that the parser actually accepts.crates/rsigma-runtime/README.md:DaemonSourceRegistrydescription marks pipeline-embedded sources as deprecated rather than co-equal.CHANGELOG.md:[Unreleased]entry covers the louder warning, the runtime/CLI helper split, the daemon hot-reload coverage, the doc/README sweep, and the new tests.Test plan
cargo fmt --all -- --checkcargo clippy --workspace --all-targets --all-features -- -D warningscargo test --workspace --all-features(1502 passed, 46 suites; +5 over pre-rebase baseline for the three new engine tests and two newpipeline_deprecationtests)mkdocs build --strictcargo test -p rsigma --test cli_sources_deprecation(6 passed; unchanged surface, now exercising the runtime helper through the CLI)cargo test -p rsigma-runtime --lib engine::tests(3 passed; hot-reload path pinned)cargo test -p rsigma-runtime --lib pipeline_deprecation::tests(2 passed; dedup primitive pinned)Backward compatibility
Pipeline files that declare an inline
sources:block continue to load successfully. The only behaviour change is the extrawarning:line on stderr at every first load per canonical pipeline path, and the doc copy no longer documenting the embedded form as a recommended pattern.