[clangd] [C++20] [Modules] Build Database Support

[mwasplund]

DRAFT

Sending initial PR to get feedback on the high level direction for the change. Once we agree on the architecture I will do a polish pass and update this PR for final review.

What

Extending Support for Modules as outlined in clangd/clangd#1293. Add support for loading build database files as a preferred option for understanding compilation commands and module dependencies..

Why

For the first time C++ Modules introduced a dependency on compilation artifacts that must be up to date and compatible with the tooling that consumes them. Build systems are responsible for ensuring binary module interfaces are compiled before they are consumed and properly resolved during builds, however tooling also shares this problem. We can attempt to rebuild the dependency graph by parsing the compilation flags along with raw source scanning, however this information is incomplete and expensive to scan all files in a project. It is preferable to have the build system explicitly define these dependencies in a build_database.json as it is the single source of truth for compiling and resolving imports and the json file can function as a cache for fast LSP initialization.

How

• Extend the CompilationDatabse interface to expose a new ModuleManager interface. These functions mirror the functions that exists in the ProjectModules to effectively push down the logic to the database when possible (the database itself has the info required).
  • This is the design decision that I would like the most feedback on. I found the nullable getter to be the simplest first pass as it did not disrupt too much existing code and allowed me to easily check for the ability to resolve module references.
  • We could also extend the CompilationDatabase itself to handle module resolution and allow legacy database implementations to ignore this new functionality.
  • We could also extend the CompilationDatabase with a child interface for BuildDatabase that contains these new functions. This design would cause a lot of runtime checks to propagate down the usage flow for knowing which interface to use and there are a lot of wrappers for the CompilationDatabase interface.
• Create new BuildDatabaseProjectModules implementation that resolves module to source file lookups using the new ModuleManager interfaces if the CDB it references has a ModuleManager available.
• Update CompoundProjectModules to prioritize using the BuildDatabase over the CompileCommands and still fall back and validate using the scanner.
• Create new JSONBuildDatabase that builds on the JSONCompilationDatabase to parse the schema for build database json files.
• Update the DirectoryBasedGlobalCompilationDatabase to prioritize loading the build_database.json file in the root or build directory over the compile_commands.json file.

Tested

• Ran check-clangd
• Verified clangd in neovim for cmake generated sample
• Verified clangd in neovim for complex soup generated project. Can now resolve outside project structure.

TODO

• Cleanup format (my clang tidy seems wrong)
• Add tests
• Deduplicate shared logic in json loaders.

[github-actions]

Hello @mwasplund 👋

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

• All contributions to LLVM must follow our LLVM AI Tool Use Policy. In particular, if you used AI while working on this PR, remember to add a note to the PR description.
• The LLVM Code-Review Policy and Practices document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
• Our LLVM Developer Policy describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

Frequently asked questions

How do I add reviewers?

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the Reviewers section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using @ followed by their GitHub username.

What if there are no comments?

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

Are any special GitHub settings required to contribute to LLVM?

We only require contributors to have a public email address associated with their GitHub commits, see this section of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on LLVM Discord or LLVM Discourse.

Thank you,
The LLVM Community

[mwasplund]

FYI @mathstuf + @ChuanqiXu9