Re-encode IP-relative instructions when relocating into trampoline#813
Open
Re-encode IP-relative instructions when relocating into trampoline#813
Conversation
When the syscall rewriter copies pre-syscall or post-syscall instructions into the trampoline, any RIP-relative memory operands become incorrect because the instruction is now at a different virtual address. Detect this with is_ip_rel_memory_operand and re-encode affected instructions via iced_x86::Encoder at the correct trampoline IP. If re-encoding fails (e.g. the instruction changes size), the pre-syscall path falls back to hook_syscall_and_after; the post-syscall path rolls back the trampoline data to a checkpoint and returns InsufficientBytesBeforeOrAfter so the syscall is trapped instead.
Extract encode_instructions_for_trampoline() and reencode_instructions_at() so both pre-syscall and post-syscall paths use the same encode-first, append-on-success pattern. This eliminates the trampoline_data checkpoint/ rollback mechanism and fixes an O(n) skip_while scan by using direct index-based slicing from the backward/forward scan loops.
Instead of only re-encoding instructions with RIP-relative memory operands and raw-copying the rest, always run all relocated instructions through the encoder. This correctly handles IP-relative branch targets (call/jmp/jcc) in addition to RIP-relative memory, and allows the backward/forward scans to cross outgoing control transfers on x86_64 since the encoder fixes up relative displacements automatically. The x86_32 scan paths retain the control-transfer break since the encoder is 64-bit only; x86_32 support is removed in a separate PR.
Member
Author
|
This PR is ready for review, but it should NOT be merged untill PR #811 is merged so we can remove the temporary x86 test change in this PR. |
|
🤖 SemverChecks 🤖 No breaking API changes detected Note: this does not mean API is unchanged, or even that there are no breaking changes; simply, none of the detections triggered. |
wdcui
commented
Apr 25, 2026
Member
Author
There was a problem hiding this comment.
This change will be gone once the x86 removal in the syscall rewriter is merged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR improves the syscall rewriter by handling IP-relative instructions in pre or post syscall case (detect RIP-relative operands and re-encode them at the correct trampoline virtual address).