[HIPIFY][feature] Local header hipification via Clang PPCallbacks (experimental, opt-in) - Part 1 by ranapratap55 · Pull Request #2402 · ROCm/HIPIFY

ranapratap55 · 2026-02-23T09:33:45Z

Problem Statement

hipify-clang currently only transforms the main source file passed as input. Local (quoted) headers included in the main file remain same as CUDA code. These headers often cannot be hipified individually because they are not self-contained headers which requires types, operators, and definitions from preceding includes in the parent source.

Example: dxtc.cu includes CudaMath.h, which uses float3 operators (+=, *, -) defined in helper_math.h. Hipifying CudaMath.h alone fails because these operators are unavailable. Similar cases:

Solution

Two new opt-in flags enable local header hipification:

hipify-clang --local-headers main.cu -I ../Common/ 
hipify-clang --local-headers-recursive main.cu -I ../Common/

The feature is experimental and disabled by default. It will be promoted to default behavior in later release cycles with --skip-local-headers as the opt-out fallback.

Supersedes #2275

Summary of changes in this PR

Clang PPCallbacks instead of regex

Include discovery is using Clang's PPCallbacks::InclusionDirective via preprocessing pre-pass, not regex. This correctly handles conditional compilation (#ifdef), macro-computed includes (#include MACRO), compiler search paths (-I, -isystem), and comments.

Fixed Compilation Database fallback

Header files which have no entry in compile_commands.json falls back to a FixedCompilationDatabase rooted at the main source file's directory so that relative paths resolve correctly.

Cycle and duplicate detection

A processed set prevents re-hipification, and a queued set prevents duplicate work-queue entries during recursive traversal.

ToDo work (follow-up PRs):

Warning for non-self-contained headers: Warn (besides compiler’s error messaging) the user about imposibility to hipify a header file, which is not self-contained.
Folder-level hipification: Hipify everything in the provided folder.
Promote to default (add --skip-local-headers)
Angle-bracket local header hipification (#include <mylib/utils.h> via -I) — [HIPIFY][#2402] hipify project-local headers included with angle brackets (<>) #2505
Transitive preceding includes for deeply nested recursive headers — [HIPIFY][#2402] Transitive preceding includes for deeply nested recursive headers #2507

emankov · 2026-02-24T14:42:39Z

Can we say that the main purpose of this PR is to allow hipifying header files individually as main translation units?

ranapratap55 · 2026-02-25T06:08:55Z

Can we say that the main purpose of this PR is to allow hipifying header files individually as main translation units?

Yes, correct. The main purpose is to allow hipifying local header files as main translation units. We are doing it by injecting their preceding includes from the parent source file using Clang's -include ArgumentsAdjusters which is to provide the necessary compilation context.

This feature is enabled via --local-headers / --local-headers-recursive and only applies to local (quoted) headers referenced in the main source file.

emankov · 2026-02-25T18:18:07Z

the parent source

How do we guarantee finding the parent source(s) for a particular header file so we can hipify it as the main translation unit?

ranapratap55 · 2026-02-26T06:01:58Z

the parent source

How do we guarantee finding the parent source(s) for a particular header file so we can hipify it as the main translation unit?

We don't need to discover/guess the parent source file. The user provides it explicitly while hipifying. The main source file passed to hipify-clang is always the parent. --local-headers simply extends hipification to that source file's local headers, using the source file itself as the context for preceding includes.

For example: hipify-clang --local-headers sample.cu: here sample.cu is known parent for its local headers (header1.h, header2.h, etc.,).

emankov · 2026-02-26T13:20:37Z

hipify-clang --local-headers sample.cu

That is what worries me most.

The user should know or learn about the main source file, or, in general, the source files that include that particular header file.
sample.cu is not a header file. Here we have misleading information for the end users.
We need to explain somewhere, somehow, this tricky feature (actually a workaround).
Finally, it makes hipify-clang even more complicated in use, which we definitely do not want, as users already consider hipify-clang complicated enough to prefer hipify-perl usage instead. And we want the opposite. We want users to prefer hipify-clang. It means that we need to simplify it AMAP and do more work under the hood implicitly.

Technically, the main problem is that we are trying to treat a file, a header file, which is not self-contained, as a source file for hipification (actually, compilation).

I want us to think about other approaches here.
For instance:

Warn (besides compiler’s error messaging) the user about imposibility to hipify a header file, which is not self-contained.
Implement the ability to implicitly hipify all header files that are included in the main file.
Hipify everything in the provided folder.

ranapratap55 · 2026-03-04T06:47:11Z

hipify-clang --local-headers sample.cu

That is what worries me most.

The user should know or learn about the main source file, or, in general, the source files that include that particular header file.

sample.cu is not a header file. Here we have misleading information for the end users.

We need to explain somewhere, somehow, this tricky feature (actually a workaround).

Finally, it makes hipify-clang even more complicated in use, which we definitely do not want, as users already consider hipify-clang complicated enough to prefer hipify-perl usage instead. And we want the opposite. We want users to prefer hipify-clang. It means that we need to simplify it AMAP and do more work under the hood implicitly.

Technically, the main problem is that we are trying to treat a file, a header file, which is not self-contained, as a source file for hipification (actually, compilation).

I want us to think about other approaches here. For instance:

Warn (besides compiler’s error messaging) the user about imposibility to hipify a header file, which is not self-contained.

Implement the ability to implicitly hipify all header files that are included in the main file.

Hipify everything in the provided folder.

As per our discussions/suggestions, I've reworked this PR to implement suggestion 2.
Local header hipification is now implicit by default. hipify-clang sample.cu automatically detects and hipifies all local headers recursively, no flags needed. Added --skip-local-headers as the single opt-out mechanism.

I will raise 2 separate PRs as a follow up for:

Warn the user for non-self-contained headers
Folder-level hipification

Updated PR title and description to reflect the changes.

emankov · 2026-03-31T17:35:35Z

-cl::opt<bool> OptLocalHeadersRecursive("local-headers-recursive",
-  cl::desc("Enable hipification of quoted local headers recursively"),
+cl::opt<bool> SkipLocalHeaders("skip-local-headers",
+  cl::desc("Skip implicit hipification of local (quoted) headers included in the main source file"),


What if the "local" headers are included with <>?

I believe this is out of scope. Angled-bracket includes files are resolved using -I search paths. They represent system headers or external library dependencies.

But I do think there is edge case like, if a project uses #include <mylib/utils.h> via -I project/src flag, that would require resolving each angled-bracket against every -I path. That is a meaningful work and requires a separate PR of its own.

#2505: Angle-bracket local header hipification (#include <>)

emankov · 2026-03-31T18:18:42Z

+  bool collectPrecedingIncludes(const std::string &mainSourceAbspath,
+                                const std::string &targetHeaderAbspath,
+                                std::vector<std::string> &outIncludes) {
+    std::string mainSourceContent;
+    if (!readFile(mainSourceAbspath, mainSourceContent)) {
+      errs() << sHipify << sError << "Cannot read source files: "
+            << mainSourceAbspath << "\n";
+      return false;
+    }
+
+    std::string targetFileName = std::string(sys::path::filename(targetHeaderAbspath));
+    std::istringstream iss(mainSourceContent);
+    std::string line;
+    std::smatch m;
+
+    while (std::getline(iss, line)) {
+      if (std::regex_match(line, m, LocalIncludeRe)) {
+        std::string quotedName = m[1].str();
+        std::string quotedFileName = std::string(sys::path::filename(quotedName));
+        if (quotedFileName == targetFileName)
+          break;
+        std::string absPath;
+        if (resolveLocalIncludeInternal(mainSourceAbspath, quotedName, absPath))
+          outIncludes.push_back(absPath);
+      }
+
+      if (std::regex_search(line, m, SystemIncludeRe)) {
+        outIncludes.push_back(m[1].str());
+      }
+    }
+
+    return true;
+  }
+}


Looks like the collectPrecedingIncludes only scans the immediate parentPath.
What about deeply nested headers?

yes, correct.
collectPrecedingIncludes scans only one level (the immediate parent). This is intentional for this PR, it was designed for (main.cu → A.h → B.h) where B.h depends on explicitly included before it in A.h and collectPrecedingIncludes(A.h, B.h) finds exactly what B.h needs.

I considered collecting full ancestor chain recursively (for nested headers) but that results in over-injection. When two of those injected headers define same symbol, we get redefinition errors.

I would like to take this (deeply nested headers) as a follow-up PR. We track the full ancestors list and modify from {hdr, parentPath} to {hdr, vector<parentChain>}. And only injects that appear in ancestors(parents) up to the nearest file that explicitly includes the current target. This avoids over-injection issue.

#2507: Transitive preceding includes for deeply nested recursive headers

ranapratap55 · 2026-04-07T11:21:02Z

Follow up work

[HIPIFY][#2402] Transitive preceding includes for deeply nested recursive headers #2507: Transitive preceding includes for deeply nested recursive headers
[HIPIFY][#2402] hipify project-local headers included with angle brackets (<>) #2505: Angle-bracket local header hipification (#include <>)

emankov · 2026-04-16T20:50:19Z

The added tests cover basic recursion and depth-1 injection of “includes”. However, the test that verifies the transitive context is preserved during recursive traversal is still needed.

Please add a test that includes a non-self-contained header with a recursion depth of 2 or more. For example, the root .cu file includes <cmath> and parent.h; parent.h includes child.h; and child.h uses sqrtf but does not include <cmath> itself. This is essential to ensure that the inherited compilation context is correctly propagated down the inclusion chain and is not dropped when processing nested files.

emankov · 2026-04-16T20:54:28Z

I honestly believe we need to preserve -local-headers and --local-headers-recursive, since the feature you're introducing is experimental and shouldn't be used by default. What do you think, @ranapratap55?

ranapratap55 · 2026-04-20T10:18:20Z

The added tests cover basic recursion and depth-1 injection of “includes”. However, the test that verifies the transitive context is preserved during recursive traversal is still needed.

Please add a test that includes a non-self-contained header with a recursion depth of 2 or more. For example, the root .cu file includes <cmath> and parent.h; parent.h includes child.h; and child.h uses sqrtf but does not include <cmath> itself. This is essential to ensure that the inherited compilation context is correctly propagated down the inclusion chain and is not dropped when processing nested files.

I've adapted to use <algorithm> + std::sort

transitive_system_include.cu: includes <algorithm> and transitive_parent.h
transitive_parent.h: includes transitive_child.h and <cuda_runtime.h>, but no <algorithm>
transitive_child.h: uses std::sort and cudaMalloc, includes neither <algorithm> nor <cuda_runtime.h>

collectAncestorSystemIncludes walks the ancestor chain and collects <algorithm> from the main src file, then injects it before compiling transitive_child.h. If this propagation breaks, the test fails directly.

ranapratap55 · 2026-04-20T10:31:20Z

I honestly believe we need to preserve -local-headers and --local-headers-recursive, since the feature you're introducing is experimental and shouldn't be used by default. What do you think, @ranapratap55?

Thanks for raising this and it is a fair concern.

Based on our earlier discussions, we collectively moved toward implicit-by-default to lower the barrier for users. The current revision already has multiple layers in place for end users like

Opt-out flag: --skip-local-headers for users who don't want automatic hipification
Non-self-contained header warning: [HIPIFY] warning: non-self-contained header is emitted when a header cannot be hipified in isolation and with a note warning the user to run the parent source file instead

And I am thinking to add below

Auto-hipification notice: an info-level message [HIPIFY] note: local headers detected: hipifying automatically. Use --skip-local-headers to disable.

With these measures together I believe the implicit behavior is transparent. That said, I am happy to address any specific scenario where this causes harm directly rather than revert the design.

What do you think? @emankov

emankov · 2026-05-05T10:53:34Z

I honestly believe we need to preserve -local-headers and --local-headers-recursive, since the feature you're introducing is experimental and shouldn't be used by default. What do you think, @ranapratap55?

Thanks for raising this and it is a fair concern.

Based on our earlier discussions, we collectively moved toward implicit-by-default to lower the barrier for users. The current revision already has multiple layers in place for end users like

Opt-out flag: --skip-local-headers for users who don't want automatic hipification

Non-self-contained header warning: [HIPIFY] warning: non-self-contained header is emitted when a header cannot be hipified in isolation and with a note warning the user to run the parent source file instead

And I am thinking to add below

Auto-hipification notice: an info-level message [HIPIFY] note: local headers detected: hipifying automatically. Use --skip-local-headers to disable.

With these measures together I believe the implicit behavior is transparent. That said, I am happy to address any specific scenario where this causes harm directly rather than revert the design.

What do you think? @emankov

I still believe that experimental features should stay experimental for at least one release. Meanwhile, it might be thoroughly tested. Next step: inform end users in the documentation that, with the next release, this particular feature will be enabled by default, and for fallback functionality, here are the options to set explicitly.

emankov · 2026-05-05T20:07:31Z

[IMPORTANT]

After reviewing this regex-based solution for header tree calculation, I decided it is not a good solution. Using regex in hipify-perl is a forced decision, but using a regex-based approach with a C++ compiler under the hood can't be treated as a good solution.

[Reasons]

Parsing C/C++ includes with regular expressions is fundamentally brittle and serves as a temporary workaround.
Regex has major blind spots that cannot be solved without rewriting a full C++ preprocessor:
- Conditional Compilation: Regex blindly captures #include directives inside #if 0 or unmet #ifdef blocks, injecting phantom dependencies and false context.
- Macro Expansion: Regex completely fails on computed inclusions like #include MACRO_NAME.
- Path Resolution: Custom regex/path logic lacks knowledge of the compiler search flags (-I, -isystem) from compile_commands.json. Whereas Clang’s internal HeaderSearch engine perfectly applies these flags and resolves the exact absolute paths.
- Comments and Formatting: Regex breaks easily on multi-line comments /* ... */ or line continuations \ within the directive.
Tapping into Clang's built-in Preprocessor delegates all of this to the compiler, completely solving these issues out-of-the-box.

[Recommendations]

instead of regexs, use Clang's functionality, based on PPCallbacks and InclusionDirective.
It might need additional pre-pass.

…jection via ArgumentsAdjusters

…ntext test

…and restore opt-in flags

ranapratap55 · 2026-05-13T04:46:20Z

I still believe that experimental features should stay experimental for at least one release. Meanwhile, it might be thoroughly tested. Next step: inform end users in the documentation that, with the next release, this particular feature will be enabled by default, and for fallback functionality, here are the options to set explicitly.

Agreed, both --local-headers and --local-headers-recursive are preserved as explicit opt-in flags in latest revision.
For the graduation plan,

This release: Feature ships as opt-in behind --local-headers/--local-headers-recursive and document that it will become default. Test it thoroughly.
Next release: Enable by default with --skip-local-headers as fallback for opt-out

ranapratap55 · 2026-05-13T04:54:09Z

[IMPORTANT]

After reviewing this regex-based solution for header tree calculation, I decided it is not a good solution. Using regex in hipify-perl is a forced decision, but using a regex-based approach with a C++ compiler under the hood can't be treated as a good solution.

[Reasons]

Parsing C/C++ includes with regular expressions is fundamentally brittle and serves as a temporary workaround.

Regex has major blind spots that cannot be solved without rewriting a full C++ preprocessor:

Conditional Compilation: Regex blindly captures #include directives inside #if 0 or unmet #ifdef blocks, injecting phantom dependencies and false context.

Macro Expansion: Regex completely fails on computed inclusions like #include MACRO_NAME.

Path Resolution: Custom regex/path logic lacks knowledge of the compiler search flags (-I, -isystem) from compile_commands.json. Whereas Clang’s internal HeaderSearch engine perfectly applies these flags and resolves the exact absolute paths.

Comments and Formatting: Regex breaks easily on multi-line comments /* ... */ or line continuations \ within the directive.

Tapping into Clang's built-in Preprocessor delegates all of this to the compiler, completely solving these issues out-of-the-box.

[Recommendations]

instead of regexs, use Clang's functionality, based on PPCallbacks and InclusionDirective.

It might need additional pre-pass.

Thanks for this feedback. I've removed the entire regex-based include scanning system and replaced with Clang's PPCallbacks::InclusionDirective in the latest revision.

The new PPCallbacks approach uses IncludeCollectorCallbacks (a PPCallbacks subclass), IncludeCollectorAction (a PreprocessorFrontendAction), and collectIncludeTree() as an entry point

We are directly addressing below concerns:

Conditional compilation
Macro expansion
Path resolution
Comments/formatting

ranapratap55 requested review from emankov and searlmc1 as code owners February 23, 2026 09:33

ranapratap55 requested review from SyamaAmd and kzhuravl February 23, 2026 09:34

ranapratap55 self-assigned this Feb 23, 2026

ranapratap55 added fix It fixes bug feature Feature request or implementation labels Feb 23, 2026

ranapratap55 mentioned this pull request Feb 23, 2026

[HIPIFY][feature] Add header injection fallback for local headers hipification #2275

Draft

ranapratap55 force-pushed the users/ranapratap55/implict-local-headers branch from 5e6249f to 353ebef Compare March 4, 2026 06:20

ranapratap55 changed the title ~~[HIPIFY][feature] Local header hipification with preceding include injection via ArgumentsAdjusters~~ [HIPIFY][feature] Implicit hipification of local header files Mar 4, 2026

ranapratap55 added the partial Partial fix or implementation label Mar 20, 2026

ranapratap55 changed the title ~~[HIPIFY][feature] Implicit hipification of local header files~~ [HIPIFY][feature] Implicit hipification of local header files - Part 1 Mar 20, 2026

ranapratap55 force-pushed the users/ranapratap55/implict-local-headers branch from 353ebef to d10566f Compare March 24, 2026 07:43

ranapratap55 mentioned this pull request Mar 25, 2026

[HIPIFY][#2402] Warn users for non-self-contained header hipification in isolation - Part 2 #2447

Draft

emankov requested changes Mar 31, 2026

View reviewed changes

ranapratap55 force-pushed the users/ranapratap55/implict-local-headers branch from d10566f to 7d568b6 Compare April 6, 2026 17:10

ranapratap55 requested a review from emankov April 7, 2026 10:55

This was referenced Apr 7, 2026

[HIPIFY][#2402] hipify project-local headers included with angle brackets (<>) #2505

Open

[HIPIFY][#2402] Transitive preceding includes for deeply nested recursive headers #2507

Open

emankov requested changes Apr 9, 2026

View reviewed changes

Comment thread src/LocalHeader.cpp Outdated

Comment thread src/LocalHeader.cpp Outdated

ranapratap55 force-pushed the users/ranapratap55/implict-local-headers branch from 8d1113a to 8a1adf5 Compare April 13, 2026 05:54

ranapratap55 requested a review from emankov April 13, 2026 06:11

ranapratap55 force-pushed the users/ranapratap55/implict-local-headers branch from 8a1adf5 to e198b45 Compare April 20, 2026 10:12

ranapratap55 added 8 commits May 12, 2026 11:56

[HIPIFY][feature] Local header hipification with preceding include in…

c54c34e

…jection via ArgumentsAdjusters

[HIPIFY][doc][NFC] Updated the licence

6168267

[HIPIFY][feature] Implicit hipification of local header files

1b69ba8

[HIPIFY][fix] Fix regex, premature loop break, and duplicate warnings

d3b4bae

[HIPIFY][fix] Fix compile_commands json issue

73670f2

[HIPIFY][fix] Fix regex and structured bindings

bf687c1

[HIPIFY][fix] Refactor include scanning and add depth-2 transitive co…

36ead5b

…ntext test

[HIPIFY] Replace regex-based include scanning with Clang PPCallbacks …

7d31bff

…and restore opt-in flags

ranapratap55 force-pushed the users/ranapratap55/implict-local-headers branch from e198b45 to 7d31bff Compare May 12, 2026 08:37

ranapratap55 changed the title ~~[HIPIFY][feature] Implicit hipification of local header files - Part 1~~ [HIPIFY][feature] Local header hipification via Clang PPCallbacks (experimental, opt-in) - Part 1 May 13, 2026

Conversation

ranapratap55 commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem Statement

Solution

Summary of changes in this PR

Clang PPCallbacks instead of regex

Fixed Compilation Database fallback

Cycle and duplicate detection

Uh oh!

emankov commented Feb 24, 2026

Uh oh!

ranapratap55 commented Feb 25, 2026

Uh oh!

emankov commented Feb 25, 2026

Uh oh!

ranapratap55 commented Feb 26, 2026

Uh oh!

emankov commented Feb 26, 2026

Uh oh!

ranapratap55 commented Mar 4, 2026

Uh oh!

Uh oh!

Uh oh!

emankov Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

ranapratap55 Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

ranapratap55 Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

emankov Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

ranapratap55 Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

ranapratap55 Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ranapratap55 commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emankov commented Apr 16, 2026

Uh oh!

emankov commented Apr 16, 2026

Uh oh!

ranapratap55 commented Apr 20, 2026

Uh oh!

ranapratap55 commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emankov commented May 5, 2026

Uh oh!

emankov commented May 5, 2026

Uh oh!

ranapratap55 commented May 13, 2026

Uh oh!

ranapratap55 commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ranapratap55 commented Feb 23, 2026 •

edited

Loading

ranapratap55 commented Apr 7, 2026 •

edited

Loading

ranapratap55 commented Apr 20, 2026 •

edited

Loading