Release v0.2.0: VTable hooking, C++/COM support, and transaction serialization#11
Merged
Conversation
Introduce VtableInstanceHook, which clones an object's VTable, patches the selected slot in the clone, and redirects only that object's vptr to the clone. This complements the existing shared VtableHook (which patches a slot in place and therefore affects every object dispatching through that table). - Add attach_vtable_instance to TransactionCore and DetourTransaction, the detours_transaction_attach_vtable_instance C entry point, and the matching declarations in neohook.h / neohook.hpp. - Convert VTableHook into an RAII guard installed via install(), so commit rollback and Drop restore the slot consistently with the other hook types. - The instance hook clears its cloned-VTable pointer right after freeing it so that a failed protection-restore (which leaves the guard active) cannot make a subsequent Drop free the same allocation twice.
Add tests/vtable_hooks.rs with focused coverage for parameter validation, the explicit unhook() path, multi-slot clone correctness, shared-vs-instance semantics, and committing several VTable hooks in one transaction. Extend the api/ffi integration suites with instance-hook happy paths and add a runnable examples/vtable_instance_hook.rs.
Add a feature entry and example pointer for per-instance VTable hooks and mark the per-instance and shared VTable items as done in the roadmap.
Complete the remaining v0.2.0 roadmap item. Add tests/cpp_com_vtable.rs, which models objects that follow the C++ / COM ABI (vptr-first layout, extern system methods receiving `this`) and verifies shared and per-instance VTable hooks with correct `this` plumbing, chaining back into the original method, and an IUnknown AddRef intercept-and-forward. Add examples/com_vtable_hook.rs demonstrating a per-instance AddRef hook on a COM-style interface, and document it in the README.
test_macro_inline_simple and test_macro_helper_sets_original_and_hooks both hooked the same add_values function and ran in parallel. A hooked function is global mutable state, so the two tests raced: one could unhook/rehook while the other asserted its own detour result, causing intermittent failures. Give the inline-macro test its own target (add_values_inline) so the tests are fully isolated and parallel-safe.
Concurrent transactions on different threads could previously suspend each other and patch code at the same time, which races and can deadlock (two threads each suspending the other). Introduce a process-wide transaction lock, matching the one-transaction-at-a-time model used by Microsoft Detours. The lock is acquired before the first thread is suspended (so a transaction can never freeze the thread that holds the lock) and at the start of commit (so the patch phase is serialized even when no threads are suspended). It is released in cleanup_threads after all threads have been resumed, and recovers from poisoning so a panic in one transaction cannot wedge all future ones.
Add tests/concurrent_transactions.rs, in which several threads install, verify, and remove per-instance VTable hooks at the same time, exercising the new process-wide transaction lock under contention. Serialize the tests in iat_error_integration.rs: one of them briefly hooks a process-wide import (e.g. GetModuleHandleW) and redirects it to a stub returning zero, while another test calls that same function directly. Running them in parallel let one observe the other's active hook, causing intermittent failures. A shared lock keeps the IAT tests from overlapping.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Features
VTable hooking (af06379): rewrite a selected VTable slot to detour virtual calls, with original slot restore on unhook; exposed through the transaction API and the C/C++ FFI.
Per-instance VTable hooking (074acef): attach_vtable_instance clones an object's VTable, patches the clone, and redirects only that object's vptr so a single instance is detoured while every other object sharing the table is unaffected. VTableHook was reworked into an RAII guard installed via install(), consistent with the other hook types for commit-rollback and Drop.
Process-wide transaction lock (2005bb9): transactions are now serialized (the "one transaction at a time" model used by Microsoft Detours). The lock is taken before the first thread is suspended so a transaction can never freeze the thread holding the lock and at the start of commit(). It is released after threads are resumed and recovers from poisoning.
Bug fixes
Double free in VTableInstanceHook (a5acef6): if the protection restore after unhook failed (leaving the guard active), a subsequent Drop could free the cloned VTable a second time. The cloned table pointer is now cleared immediately after dealloc.
Tests
VTable hook coverage: validation, explicit unhook(), multi slot clones, shared-vs-instance semantics, multiple hooks per transaction, plus a per-instance example (1491da6).
C++/COM coverage and a runnable COM IUnknown (AddRef) example modeling the real interface ABI (24ac97f).
Concurrency test exercising the new transaction lock under contention (part of d3c7dab).
Flakiness fixes: macro tests no longer share one hook target (d63c2e7); IAT integration tests are serialized so one test's hook on a process-wide import (e.g. GetModuleHandleW) can't be observed by another (d3c7dab).