Skip to content

Refactor file collection logic to exclude lib/ directory#21639

Open
dwelch-r7 wants to merge 1 commit into
rapid7:masterfrom
dwelch-r7:exclude-lib-checksum
Open

Refactor file collection logic to exclude lib/ directory#21639
dwelch-r7 wants to merge 1 commit into
rapid7:masterfrom
dwelch-r7:exclude-lib-checksum

Conversation

@dwelch-r7

Copy link
Copy Markdown
Contributor

Description

Remove lib/ from the boot-time checksum validation scope in Msf::Modules::Metadata::Store.collect_files_to_check.

The checksum validation determines whether module files have changed since the metadata cache was last built. When files have changed, it forces immediate (non-deferred) module loading. Previously, this included all files under lib/ (~3000+ files), meaning any library change would invalidate the module metadata cache and trigger a full module reload — even though library changes don't affect the cached metadata (module name, author, platform, references, etc.).

This reduces the number of files checksummed on every boot from ~7000+ to ~4000 (modules only), and eliminates false cache invalidations caused by unrelated library edits during development.

Related Issue:

Breaking Changes

If a mixin change in lib/ alters module metadata (e.g., changes default targets or platform support), the metadata cache will not automatically invalidate. Users can run reload_all to force a rebuild.

Verification Steps

    • Start msfconsole normally, confirm it boots without error
    • Modify a file in lib/ (e.g., touch lib/msf/core/exploit.rb), restart msfconsole — it should still boot with deferred module loads (fast path) since lib changes no longer invalidate the checksum
    • Modify a file in modules/ (e.g., touch modules/exploits/multi/handler.rb), restart msfconsole — it should detect the change and force immediate module loading
    • Run bundle exec rspec spec/lib/msf/core/modules/metadata/store_spec.rb — all 5 examples pass

Test Evidence

Before

hyperfine --warmup 3 "bundle exec ruby ./msfconsole -q -x 'exit'"
Benchmark 1: bundle exec ruby ./msfconsole -q -x 'exit'
Time (mean ± σ): 4.393 s ± 0.066 s [User: 3.095 s, System: 1.362 s]
Range (min … max): 4.284 s … 4.483 s 10 runs

After

$ hyperfine --warmup 3 "bundle exec ruby ./msfconsole -q -x 'exit'"
Benchmark 1: bundle exec ruby ./msfconsole -q -x 'exit'
  Time (mean ± σ):      4.179 s ±  0.023 s    [User: 3.040 s, System: 1.176 s]
  Range (min … max):    4.154 s …  4.197 s    5 runs

Environment

Field Details
Operating System macOS (26.5.2)
Target Software/Hardware Metasploit Framework (Ruby 3.3.8)

AI Usage Disclosure

Kiro

Pre-Submission Checklist

  • No sensitive information (IP addresses, credentials, API keys, hashes) in code or documentation
  • Tested on the target environment specified in the Environment section above
  • Included RSpec tests for library changes
  • Read the CONTRIBUTING.md and module acceptance guidelines

@adfoster-r7

Copy link
Copy Markdown
Contributor

Are we sure we want this change? 👀

If a mixin change in lib/ alters module metadata (e.g., changes default targets or platform support), the metadata cache will no longer automatically invalidate - which doesn't seem correct

@dwelch-r7

Copy link
Copy Markdown
Contributor Author

If a mixin change in lib/ alters module metadata (e.g., changes default targets or platform support), the metadata cache will no longer automatically invalidate - which doesn't seem correct

Yep, that is true but a vast majority of the time it doesn't affect the metadata so almost everytime I boot up framework the cache needs to rebuild even though the metadata is the same
even in cases where the metadata does update it's not actually going to affect the ability to use the module, it'll still be searchable/useable, the only thing I can think of at least that it could possibly affect is if the metadata changes add or remove something that we can search on which I think is a worthwhile tradeoff 🤷

@dwelch-r7

Copy link
Copy Markdown
Contributor Author

I have another PR here #21645 which we can land along with this one or instead of if we don't want to skip lib files but I think at least one of these should land if not both

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

3 participants