One of the primary aims of an anti-virus (AV) engine is to monitor all process activity—while malware, on the other hand, wants to avoid detection by AV. The philosophy of most rootkits is to run silent and deep, which also aligns with the goals of advanced malware as it evades detection by most enterprise class host-based security solutions (HBSS) and AV.
So how does malware evade detection when starting new rogue processes? Easy—it directly attacks the operating system’s kernel. Microsoft provides the kernel routine PsSetCreateProcessNotifyRoutine to allow security software to monitor process creation and termination events in the Windows kernel, and security software calls the PsSetCreateProcessNotifyRoutine in order to be notified by the kernel when these events occur. These security solutions will then act on process creation and termination events in order to track system activity and protect critical resources.
The Windows OS internally maintains an array of callback objects with the starting address of PspCreateProcessNotifyRoutine. A maximum of eight callbacks may be registered on Windows XP SP2. Unfortunately for developers outside of Redmond, this internal pointer of the initial routine is not exported, and there is no publicly disclosed method for third-party applications to easily register for these notifications. (Read More)