pack.yml schema standartization

[esoadamo]

Hi,
this PR updates the pack.yaml format to be more extensible and compatible with Radegast use-cases.

pack.yaml format changes

• rules is now optional
  • if not specified, defaults to the current directory content
  • has three optional keys:
    • has, includes, excludes
      • each of them has three optional subkeys (sigma, yara, ioc)
    • has - same as the rules before, says which rules are included in this pack
    • includes - when extending, explicitly take only these rules
    • excludes - when extending, explicidly do not take these rules

For Radegast, we plan to use extension widely, but we often want to omit some high-noice rules.

• sources now has three optional subkeys
  • manual - same as sources now, manual URL address of the source
  • sigma - link e.g. to path in GitHub repo for automatic download of rules from that path
  • yara - same as sigma, but for yara

zip/pack folder structure change

• previously the strucure was pack.yaml and a rules subfolder
• now the contents of the rules subfolder was moved next to the pack yaml, which means that if you zip the whole pack folder you get a fully functional copy of the rustinel rules folder
  • previously, you would either have to unzip and move and rename the rules folder or you would have rustinel/rules/<package>/rules/ structure, which seems unnecessary

added tooling

• link_packs.py - converts the old pack.yml format to the new format + createst symlinks of all package rules for easier packaging
• upload_radegast.py - adds an automatic uploading of buitl rules directly into Radegast console for anybody to use, in future this can be handled by GitHub action

further notes

• currently rustinel does not support IoCs from the yaml format, so they are handled separately, which is little akward. Maybe rustinel update that would handle the IoCs like specified would be in order?

[Karib0u]

Thanks a lot for this PR. I really appreciate the work here, and I think the direction is useful.

I checked the Rustinel engine side as well to understand how this should fit. Today Rustinel does not load pack.yml directly. It only loads configured paths like:

• scanner.sigma_rules_path
• scanner.yara_rules_path
• the IOC text files under [ioc]

So I think the best contract for now is: rustinel-rules resolves pack metadata during the build step, then outputs a folder layout that Rustinel can load directly.

I like the idea of making the schema more flexible, especially rules.has, rules.includes, and rules.excludes. That seems useful for Radegast and for future pack curation.

The part I am less sure about is making generated symlinks and generated IOC .txt files inside packs/ part of the source tree. My initial preference is to keep rules/ as the canonical source of truth, keep pack.yml explicit, and let tools/build_packs.py generate the final runtime layout.

One concrete thing I noticed: the PR currently writes flat sigma/, yara/, and ioc/ folders, while the existing Rustinel defaults and docs use rules/sigma, rules/yara, and rules/ioc. I think we should agree on that contract before merging.

Would you be open to adjusting the PR in that direction: keep the schema improvements, but keep generated/runtime files as build output rather than source files?

[esoadamo]

Hey @Karib0u , thanks for the comment!

So I think the best contract for now is: rustinel-rules resolves pack metadata during the build step, then outputs a folder layout that Rustinel can load directly.
The part I am less sure about is making generated symlinks and generated IOC .txt files inside packs/ part of the source tree. My initial preference is to keep rules/ as the canonical source of truth, keep pack.yml explicit, and let tools/build_packs.py generate the final runtime layout.

That is fine with us, I can do the changes that remove the symlinks and generated files. This was more just a demo of how it could look like if the repo structure already was a zippable pack folder without any tool call needed.

One concrete thing I noticed: the PR currently writes flat sigma/, yara/, and ioc/ folders, while the existing Rustinel defaults and docs use rules/sigma, rules/yara, and rules/ioc. I think we should agree on that contract before merging.

Yes, as I explain in the PR description, this change is so that it is actually easier to take a zipped pack and make it a rustinel rules folder. With this change, your rules folder will look like rules/<pack-id>/sigma|yara|iocs, not rules/<pack-id>/rules/sigma|yara|iocs (we believe that the second /rules/ subdirectory is not needed and flatten structure is easier to understand.

Would you be open to adjusting the PR in that direction: keep the schema improvements, but keep generated/runtime files as build output rather than source files?

Definitely, as I wrote you in message, this PR is aimed mainly so that we sync the schema between our projects -- to make sure that all our needs and ideas are covered, so our packs are as compatible as possible.

Let me know what you think! Also, are there any plans to change the IoCs loading structure in rustinel? If it would support the IoCs .yaml files in the format as currently in this repo, it would make it more transparent, rather than having to convert to .txt every time (with metadata lost).

[Karib0u]

Thanks, that makes sense.

I am fine with removing the symlinks/generated files from packs/ and keeping them as build output only. That keeps the source tree clean while still allowing the final artifact to be directly usable.

On the layout: I see your point now. If the intended install path is:

rules/<pack-id>/sigma
rules/<pack-id>/yara
rules/<pack-id>/ioc

then the flat pack artifact is reasonable and probably nicer than:

rules/<pack-id>/rules/sigma

So I am okay with that direction, as long as we make the contract consistent everywhere:

• generated zip root contains pack.yml, sigma/, yara/, and ioc/
• index.json points to the flat paths
• docs/examples show rules/<pack-id>/sigma, rules/<pack-id>/yara, and rules/<pack-id>/ioc/*.txt
• packs/ remains source metadata only, not generated symlink/runtime files

For IOC loading in Rustinel: no concrete plan yet, but I agree with the idea. The current .txt format is simple and works well for runtime loading, but YAML IOC files would preserve metadata better and avoid the build-time flattening step.

I think the best approach would be backward-compatible support in Rustinel:

• keep the current .txt IOC files supported
• add optional loading from IOC YAML files/directories
• preserve fields like id, severity, references, and comments in alert metadata where possible

That probably belongs as a separate Rustinel engine change, not a blocker for this PR. For this PR, I think we can keep generating the .txt files for compatibility and align the schema/build layout first.

[esoadamo]

Allright. I have removed all the symlinks and pre-generated IoC .txt files. I have also added the IoC .yml files to the resulting pack to be future-proof.

The PR should now be clean and ready to merge!

[Karib0u]

Thanks! Removing the symlinks and generated files from packs/ addresses my main concern, and including the IOC YAML in the generated pack makes sense.

I approved and checked the workflows. A few issues remain before merging:

• Ruff formatting fails for tools/upload_radegast.py.
• Linux atomic tests report 0/13 detections.
• Windows atomic tests report 0/19 detections.

The atomic failures appear to come from tests/atomic/run_atomics.py still using <pack-id>/rules/sigma|yara|ioc, while the new generated layout is <pack-id>/sigma|yara|ioc. Ideally, the atomic harness should read these paths from dist/index.json to prevent them drifting again.

The old nested paths also remain in tools/build_catalog.py and the documentation, so the catalog, docs, and generated index currently disagree.

I also noticed a few smaller points:

• IOC entries appear twice in dist/index.json.
• Folder discovery still searches under pack/rules/.
• Local uv run ty check cannot resolve the optional requests dependency.
• The rules and sources schema changed incompatibly while pack_schema_version remains 1.

Once these are addressed and the workflows are green, I think it will be ready to merge.

[esoadamo]

Thanks for the review! I have done all the changes, all validations and formatters are passing locally and in our GH actions pipeline.
I have bumped the schema version to 2.