Troubleshooting Procmon & Sysmon v3.32
When it comes to troubleshooting software errors, analyzing malware samples, or discovering security misconfigurations, the Sysinternals Suite can likely solve your problem. For many IT and Security professionals, this collection of Windows utilities is similar to the Swiss Army knife.
- Need a task manager on steroids? Process Explorer has your back.
- Want to execute a process on a remote host? Check out PsExec.
- Is there a sensitive file you’d like to remove? SDelete FTW.
But what happens when the SysInternals Suite is the software you need to troubleshoot? Well that’s exactly what happened during a recent breach investigation.
Process Monitor (Procmon) is one of most popular SysInternals utilities which provides users a simple interface to audit process, file, and network activity in ridiculously granular detail. Under the hood, it uses a driver to gather events of interest. Unfortunately, Procmon v3.32 was unable to load its driver on the 64-bit Windows 7 host we were analyzing.
Oddly enough, the same version of Procmon had worked like a charm on a 32-bit Windows 7 host earlier that day. Considering the previous error message and the different behavior between architectures, it led our team to believe there was a driver signing issue.
Since Windows Vista/Server 2008, Microsoft added Driver Signing Enforcement to their 64-bit operating systems to combat the surge of malicious drivers (rootkits). This makes it (slightly) harder for attackers to backdoor legitimate drivers and load custom code into the kernel. With this in mind, we decided to dig into the Windows Event Logs for additional clues.
Based on the screenshot above, we could clearly see that Windows refused to load Procmon’s driver due to “code integrity” issues. This information matched the user complaints on the SysInternal’s forum:
- https://forum.sysinternals.com/driver-verifier-violation-in-latest-procmon_topic32681.html
- https://forum.sysinternals.com/procmon-v332-fails-on-x64-win7_topic32654.html
So was Procmon backdoored? Did Mark forget to sign the driver? Or was there something else at the root of this problem? One easy way to find out involves examining the digital signature of the dropped driver. According to the Event Logs, the suspect file was located at:
\Device\HarddiskVolume1\Windows\System32\drivers\PROCMON23.SYS
This volume was assigned to drive letter C:. However, the PROCMON23.SYS file could not be found within the “drivers” directory.
Rather than call it quits, we decided to dig into the procmon.exe file. Procmon is typically distributed as a single 32-bit executable which determines the Operating System’s architecture and executes the appropriate version of itself. If the OS is 64-bit, a 64-bit version of Procmon will be extracted from the 32-bit EXE’s resources and placed at “%TEMP%\Procmon64.exe” as a hidden file.
When Procmon64.exe is run, it will extract a copy of the troublesome driver from its own PE resources to the %SYSTEMROOT%\System32\drivers directory. Unfortunately, PROCMON23.SYS wasn’t there for us to examine (likely deleted when the driver failed to load). From here, we could have attached a debugger to Procmon64.exe and stepped through until the driver was dropped -OR- we could have taken the easy route and extracted the driver with a resource editor. We chose the quickest path to victory and went with Resource Hacker.
As demonstrated in the above screenshot, Resource Hacker can easily browse the resources within Procmon64.exe. BINRES caught our eye as it likely stood for Binary Resource and contained a PE file labeled RCDRIVERNT. With these indicators, we were sure this was our suspect. To extract the driver, we right-clicked the resource and chose to save the file as Procmon332Driver.sys. To inspect the extracted file, we right-clicked on it, chose “Properties” from the context menu, and selected the “Details” tab.
Eureka! We’ve recovered the driver included with Procmon v3.32. To validate the digital signature of our driver, we used SysInternals Sigcheck.
The results suggested Procmon v3.32’s driver could not be verified all the way back to the certificate’s trusted root authority on this 64-bit Windows 7 host. To aide our comparison, we repeated the same driver extraction process on Procmon v3.31. With both drivers in-hand, we inspected the digital signatures used to sign each 64-bit driver.
Clearly, there was something wrong with this Windows 7 host’s Trusted Root Certificates. Unfortunately, we couldn’t risk tainting the forensic integrity of the system to get to the bottom of this issue while on-site.
Back in the Lab
After cleaning up our partner’s systems, we couldn’t help but wonder what was going on under the hood with Procmon. To reproduce the state of the infected system, we created a Windows 7 64-bit w/SP1 VM and brought it to the patch level of the original system. Thankfully, the issue was reproducible!
From here we tried several things, including manually trusting certificates and installing patches to update the Trusted Root Certificates. We were almost out of ideas until we recalled the Microsoft updates which added support for SHA-2 certificate signing (in preparation of the likely SHA-1 vulnerabilities).
After installing KB3033929, the extracted driver’s Certification Path was no longer failing validation. 😃
Just like that, the driver issues were resolved and Procmon was back up and running!
Finishing Notes
We hope you enjoyed this blog and learned something from it as well. If you’d like to see more step-by-step walkthroughs, let us know by sharing. If you have any questions, please don’t hesitate to give us a shout on Twitter via @HuntressLabs.
