One important malware attacking vector is its hooking mechanism. Malicious programs implant hooks for many different purposes. Spyware may implant hooks to get notified of the arrival of new sensitive data. Rootkits may implant hooks to intercept and tamper with critical system information to conceal their presence in the system. A stealth backdoor may also place hooks on the network stack to establish a stealthy communication channel with remote attackers.
Several tools, such as VICE, SVV, and icesword, detect hooking behaviors by checking known memory regions for suspicious entries. However, they need prior knowledge of how existing malware implants hooks. Therefore, they become futile when malware uses new hooking mechanisms. This concern is not hypothetical. Recently, new stealthy kernel backdoors (e.g., deepdoor and uay) are reported to employ a novel hooking mechanism for intercepting the network stack.
We propose fine-grained impact analysis to automatically detect and analyze malware's hooking behaviors. Since this technique captures the intrinsic nature of hooking behaviors, it is well suited for identifying new hooking mechanisms.