Reading view

There are new articles available, click to refresh the page.

Next Windows Kernel Programming Remote Class

The next public remote Windows kernel Programming class I will be delivering is scheduled for April 15 to 18. It’s going to be very similar to the first one I did at the end of January (with some slight modifications and additions).

Cost: 1950 USD. Early bird (register before March 30th): 1650 USD

I have not yet finalized the time zone the class will be “targeting”. I will update in a few weeks on that.

If you’re interested in registering, please email [email protected] with the subject “Windows Kernel Programming class” and specify your name, company (if any) and time zone. I’ll reply by providing more information.

Feel free to contact me for questions using the email or through twitter (@zodiacon).

The complete syllabus is outlined below:

Duration: 4 Days
Target Audience: Experienced windows developers, interested in developing kernel mode drivers
Objectives: ·  Understand the Windows kernel driver programming model

·  Write drivers for monitoring processes, threads, registry and some types of objects

·  Use documented kernel hooking mechanisms

·  Write basic file system mini-filter drivers

Pre Requisites: ·  At least 2 years of experience working with the Windows API

·  Basic understanding of Windows OS concepts such as processes, threads, virtual memory and DLLs

Software requirements: ·  Windows 10 Pro 64 bit (latest stable version)
·  Visual Studio 2017 + latest update
·  Windows 10 SDK (latest)
·  Windows 10 WDK (latest)
·  Virtual Machine for testing and debugging

Instructor: Pavel Yosifovich

Abstract

The cyber security industry has grown considerably in recent years, with more sophisticated attacks and consequently more defenders. To have a fighting chance against these kinds of attacks, kernel mode drivers must be employed, where nothing (at least nothing from user mode) can escape their eyes.
The course provides the foundations for the most common software device drivers that are useful not just in cyber security, but also other scenarios, where monitoring and sometimes prevention of operations is required. Participants will write real device drivers with useful features they can then modify and adapt to their particular needs.

Syllabus

  • Module 1: Windows Internals quick overview
    • Processes
    • Virtual memory
    • Threads
    • System architecture
    • User / kernel transitions
    • Introduction to WinDbg
    • Windows APIs
    • Objects and handles
    • Summary

 

  • Module 2: The I/O System
    • I/O System overview
    • Device Drivers
    • The Windows Driver Model (WDM)
    • The Kernel Mode Driver Framework (KMDF)
    • Other device driver models
    • Driver types
    • Software drivers
    • Driver and device objects
    • I/O Processing and Data Flow
    • Accessing devices
    • Asynchronous I/O
    • Summary

 

  • Module 3: Kernel programming basics
    • Setting up for Kernel Development
    • Basic Kernel types and conventions
    • C++ in a kernel driver
    • Creating a driver project
    • Building and deploying
    • The kernel API
    • Strings
    • Linked Lists
    • The DriverEntry function
    • The Unload routine
    • Installation
    • Testing
    • Debugging
    • Summary
    • Lab: deploy a driver

 

  • Module 4: Building a simple driver
    • Creating a device object
    • Exporting a device name
    • Building a driver client
    • Driver dispatch routines
    • Introduction to I/O Request Packets (IRPs)
    • Completing IRPs
    • Handling DeviceIoControl calls
    • Testing the driver
    • Debugging the driver
    • Using WinDbg with a virtual machine
    • Summary
    • Lab: open a process for any access; zero driver; debug a driver

 

  • Module 5: Kernel mechanisms
    • Interrupt Request Levels (IRQLs)
    • Deferred Procedure Calls (DPCs)
    • Dispatcher objects
    • Low IRQL Synchronization
    • Spin locks
    • Work items
    • Summary

 

  • Module 6: Process and thread monitoring
    • Motivation
    • Process creation/destruction callback
    • Specifying process creation status
    • Thread creation/destruction callback
    • Notifying user mode
    • Writing a user mode client
    • Preventing potentially malicious processes from executing
    • Summary
    • Lab: monitoring process/thread activity; prevent specific processes from running; protecting processes

 

  • Module 7: Object and registry notifications
    • Process/thread object notifications
    • Pre and post callbacks
    • Registry notifications
    • Performance considerations
    • Reporting results to user mode
    • Summary
    • Lab: protect specific process from termination; simple registry monitor

 

  • Module 8: File system mini filters
    • File system model
    • Filters vs. mini filters
    • The Filter Manager
    • Filter registration
    • Pre and Post callbacks
    • File name information
    • Contexts
    • File system operations
    • Filter to user mode communication
    • Debugging mini-filters
    • Summary
    • Labs: protect a directory from file deletion; backup file before deletion

zodiacon

💾

Windows 10 Desktops vs. Sysinternals Desktops

One of the new Windows 10 features visible to users is the support for additional “Desktops”. It’s now possible to create additional surfaces on which windows can be used. This idea is not new – it has been around in the Linux world for many years (e.g. KDE, Gnome), where users have 4 virtual desktops they can use. The idea is that to prevent clutter, one desktop can be used for web browsing, for example, and another desktop can be used for all dev work, and yet a third desktop could be used for all social / work apps (outlook, WhatsApp, Facebook, whatever).

To create an additional virtual desktop on Windows 10, click on the Task View button on the task bar, and then click the “New Desktop” button marked with a plus sign.

newvirtualdesktop

Now you can switch between desktops by clicking the appropriate desktop button and then launch apps as usual. It’s even possible (by clicking Task View again) to move windows from desktop to desktop, or to request that a window be visible on all desktops.

The Sysinternals tools had a tool called “Desktops” for many years now. It too allows for creation of up to 4 desktops where applications can be launched. The question is – is this Desktops tool the same as the Windows 10 virtual desktops feature? Not quite.

First, some background information. In the kernel object hierarchy under a session object, there are window stations, desktops and other objects. Here’s a diagram summarizing this tree-like relationship:

Sessions

As can be seen in the diagram, a session contains a set of Window Stations. One window station can be interactive, meaning it can receive user input, and is always called winsta0. If there are other window stations, they are non-interactive.

Each window station contains a set of desktops. Each of these desktops can hold windows. So at any given moment, an interactive user can interact with a single desktop under winsta0. Upon logging in, a desktop called “Default” is created and this is where all the normal windows appear. If you click Ctrl+Alt+Del for example, you’ll be transferred to another desktop, called “Winlogon”, that was created by the winlogon process. That’s why your normal windows “disappear” – you have been switched to another desktop where different windows may exist. This switching is done by a documented function – SwitchDesktop.

And here lies the difference between the Windows 10 virtual desktops and the Sysinternals desktops tool. The desktops tool actually creates desktop objects using the CreateDesktop API. In that desktop, it launches Explorer.exe so that a taskbar is created on that desktop – initially the desktop has nothing on it. How can desktops launch a process that by default creates windows in a different desktop? This is possible to do with the normal CreateProcess function by specifying the desktop name in the STARTUPINFO structure’s lpDesktop member. The format is “windowstation\desktop”. So in the desktops tool case, that’s something like “winsta0\Sysinternals Desktop 1”. How do I know the name of the Sysinternals desktop objects? Desktops can be enumerated with the EnumDesktops API. I’ve written a small tool, that enumerates window stations and desktops in the current session. Here’s a sample output when one additional desktop has been created with “desktops”:

desktops1

In the Windows 10 virtual desktops feature, no new desktops are ever created. Win32k.sys just manipulates the visibility of windows and that’s it. Can you guess why? Why doesn’t Window 10 use the CreateDesktop/SwitchDesktop APIs for its virtual desktop feature?

The reason has to do with some limitations that exist on desktop objects. For one, a window (technically a thread) that is bound to a desktop cannot be switched to another; in other words, there is no way to transfer a windows from one desktop to another. This is intentional, because desktops provide some protection. For example, hooks set with SetWindowsHookEx can only be set on the current desktop, so cannot affect other windows in other desktops. The Winlogon desktop, as another example, has a strict security descriptor that prevents non system-level users from accessing that desktop. Otherwise, that desktop could have been tampered with.

The virtual desktops in Windows 10 is not intended for security purposes, but for flexibility and convenience (security always “contradicts” convenience). That’s why it’s possible to move windows between desktops, because there is no real “moving” going on at all. From the kernel’s perspective, everything is still on the same “Default” desktop.

 

 

 

zodiacon

💾

newvirtualdesktop

💾

Sessions

💾

desktops1

💾

Where did System Services 0 and 1 go?

System calls on Windows go through NTDLL.dll, where each system call is invoked by a syscall (x64) or sysenter (x86) CPU instruction, as can be seen from the following output of NtCreateFile from NTDLL:

0:000> u
ntdll!NtCreateFile:
00007ffc`c07fcb50 4c8bd1          mov     r10,rcx
00007ffc`c07fcb53 b855000000      mov     eax,55h
00007ffc`c07fcb58 f604250803fe7f01 test    byte ptr [SharedUserData+0x308 (00000000`7ffe0308)],1
00007ffc`c07fcb60 7503            jne     ntdll!NtCreateFile+0x15 (00007ffc`c07fcb65)
00007ffc`c07fcb62 0f05            syscall
00007ffc`c07fcb64 c3              ret
00007ffc`c07fcb65 cd2e            int     2Eh
00007ffc`c07fcb67 c3              ret

The important instructions are marked in bold. The value set to EAX is the system service number (0x55 in this case). The syscall instruction follows (the condition tested does not normally cause a branch). syscall causes transition to the kernel into the System Service Dispatcher routine, which is responsible for dispatching to the real system call implementation within the Executive. I will not go to the exact details here, but eventually, the EAX register must be used as a lookup index into the System Service Dispatch Table (SSDT), where each system service number (index) should point to the actual routine.

On x64 versions of Windows, the SSDT is available in the kernel debugger in the nt!KiServiceTable symbol:

lkd> dd nt!KiServiceTable
fffff804`13c3ec20  fced7204 fcf77b00 02b94a02 04747400
fffff804`13c3ec30  01cef300 fda01f00 01c06005 01c3b506
fffff804`13c3ec40  02218b05 0289df01 028bd600 01a98d00
fffff804`13c3ec50  01e31b00 01c2a200 028b7200 01cca500
fffff804`13c3ec60  02229b01 01bf9901 0296d100 01fea002

You might expect the values in the SSDT to be 64-bit pointers, pointing directly to the system services (this is the scheme used on x86 systems). On x64 the values are 32 bit, and are used as offsets from the start of the SSDT itself. However, the offset does not include the last hex digit (4 bits): this last value is the number of arguments to the system call.

Let’s see if this holds with NtCreateFile. Its service number is 0x55 as we’ve seen from user mode, so to get to the actual offset, we need to perform a simple calculation:

kd> dd nt!KiServiceTable+55*4 L1
fffff804`13c3ed74  020b9207

Now we need to take this offset (without the last hex digit), add it to the SSDT and this should point at NtCreateFile:

lkd> u nt!KiServiceTable+020b920
nt!NtCreateFile:
fffff804`13e4a540 4881ec88000000  sub     rsp,88h
fffff804`13e4a547 33c0            xor     eax,eax
fffff804`13e4a549 4889442478      mov     qword ptr [rsp+78h],rax
fffff804`13e4a54e c744247020000000 mov     dword ptr [rsp+70h],20h

Indeed – this is NtCreateFile. What about the argument count? The value stored is 7. Here is the prototype of NtCreateFile (documented in the WDK as ZwCreateFile):

NTSTATUS NtCreateFile(
  PHANDLE            FileHandle,
  ACCESS_MASK        DesiredAccess,
  POBJECT_ATTRIBUTES ObjectAttributes,
  PIO_STATUS_BLOCK   IoStatusBlock,
  PLARGE_INTEGER     AllocationSize,
  ULONG              FileAttributes,
  ULONG              ShareAccess,
  ULONG              CreateDisposition,
  ULONG              CreateOptions,
  PVOID              EaBuffer,
  ULONG              EaLength);

Clearly, there are 11 parameters, not just 7. Why the discrepency? The stored value is the number of parameters that are passed using the stack. In x64 calling convention, the first 4 arguments are passed using registers: RCX, RDX, R8, R9 (in this order).

Now back to the title of this post. Here are the first few entries in the SSDT again:

lkd> dd nt!KiServiceTable
fffff804`13c3ec20  fced7204 fcf77b00 02b94a02 04747400
fffff804`13c3ec30  01cef300 fda01f00 01c06005 01c3b506

The first two entries look different, with much larger numbers. Let’s try to apply the same logic for the first value (index 0):

kd> u nt!KiServiceTable+fced720
fffff804`2392c340 ??              ???
                    ^ Memory access error in 'u nt!KiServiceTable+fced720'

Clearly a bust. The value is in fact a negative value (in two’s complement), so we need to sign-extend it to 64 bit, and then perform the addition (leaving out the last hex digit as before):

kd> u nt!KiServiceTable+ffffffff`ffced720
nt!NtAccessCheck:
fffff804`1392c340 4c8bdc          mov     r11,rsp
fffff804`1392c343 4883ec68        sub     rsp,68h
fffff804`1392c347 488b8424a8000000 mov     rax,qword ptr [rsp+0A8h]

This is NtAccessCheck. The function’s implementation is in lower addresses than the SSDT itself. Let’s try the same exercise with index 1:

kd> u nt!KiServiceTable+ffffffff`ffcf77b0
nt!NtWorkerFactoryWorkerReady:
fffff804`139363d0 4c8bdc          mov     r11,rsp
fffff804`139363d3 49895b08        mov     qword ptr [r11+8],rbx

And we get system call number 1: NtWorkerFactoryWorkerReady.

For those fond of WinDbg scripting – write a script to display nicely all system call functions and their indices.

 

zodiacon

💾

Next Windows Internals (Remote) Training

It’s been a while since I gave the Windows Internals training, so it’s time for another class of my favorite topics!

This time I decided to make it more afordable, to allow more people to participate. The cost is based on whether paid by an individual vs. a company. The training includes lab exercises – some involve working with tools, while others involve coding in C/C++.

  • Public 5-day remote class
  • Dates: April 20, 22, 23, 27, 30
  • Time: 8 hours / day. Exact hours TBD
  • Price: 750 USD (payed by individual) / 1500 USD (payed by company)
  • Register by emailing [email protected] and specifying “Windows Internals Training” in the title
    • Provide names of participants (discount available for multiple participants from the same company), company name (if any) and preferred time zone.
    • You’ll receive instructions for payment and other details
  • Virtual space is limited!

The training time zone will be finalized closer to the start date.

Objectives: Understand the Windows system architectureExplore the internal workings of process, threads, jobs, virtual memory, the I/O system and other mechanisms fundamental to the way Windows works

Write a simple software device driver to access/modify information not available from user mode

Target Audience: Experienced windows programmers in user mode or kernel mode, interested in writing better programs, by getting a deeper understanding of the internal mechanisms of the windows operating system.Security researchers interested in gaining a deeper understanding of Windows mechanisms (security or otherwise), allowing for more productive research
Pre-Requisites: Basic knowledge of OS concepts and architecture.Power user level working with Windows

Practical experience developing windows applications is an advantage

C/C++ knowledge is an advantage

  • Module 1: System Architecture
    • Brief Windows NT History
    • Windows Versions
    • Tools: Windows, Sysinternals, Debugging Tools for Windows
    • Processes and Threads
    • Virtual Memory
    • User mode vs. Kernel mode
    • Architecture Overview
    • Key Components
    • User/kernel transitions
    • APIs: Win32, Native, .NET, COM, WinRT
    • Objects and Handles
    • Sessions
    • Introduction to WinDbg
    • Lab: Task manager, Process Explorer, WinDbg
  • Module 2: Processes & Jobs
    • Process basics
    • Creating and terminating processes
    • Process Internals & Data Structures
    • The Loader
    • DLL explicit and implicit linking
    • Process and thread attributes
    • Protected processes and PPL
    • UWP Processes
    • Minimal and Pico processes
    • Jobs
    • Nested jobs
    • Introduction to Silos
    • Server Silos and Docker
    • Lab: viewing process and job information; creating processes; setting job limits
  • Module 3: Threads
    • Thread basics
    • Thread Internals & Data Structures
    • Creating and terminating threads
    • Thread Stacks
    • Thread Priorities
    • Thread Scheduling
    • CPU Sets
    • Direct Switch
    • Deep Freeze
    • Thread Synchronization
    • Lab: creating threads; thread synchronization; viewing thread information; CPU sets
  • Module 4: Kernel Mechanisms
    • Trap Dispatching
    • Interrupts
    • Interrupt Request Level (IRQL)
    • Deferred Procedure Calls (DPCs)
    • Exceptions
    • System Crash
    • Object Management
    • Objects and Handles
    • Sharing Objects
    • Thread Synchronization
    • Synchronization Primitives (Mutex, Semaphore, Events, and more)
    • Signaled vs. Non-Signaled
    • High IRQL Synchronization
    • Windows Global Flags
    • Kernel Event Tracing
    • Wow64
    • Lab: Viewing Handles, Interrupts; creating maximum handles; Thread synchronization
  • Module 5: Memory Management
    • Overview
    • Small, large and huge pages
    • Page states
    • Memory Counters
    • Address Space Layout
    • Address Translation Mechanisms
    • Heaps
    • APIs in User mode and Kernel mode
    • Page Faults
    • Page Files
    • Commit Size and Commit Limit
    • Workings Sets
    • Memory Mapped Files (Sections)
    • Page Frame Database
    • Other memory management features
    • Lab: committing & reserving memory; using shared memory; viewing memory related information
  • Module 6: Management Mechanisms
    • The Registry
    • Services
    • Starting and controlling services
    • Windows Management Instrumentation
    • Lab: Viewing and configuring services; Process Monitor

  • Module 7: I/O System
    • I/O System overview
    • Device Drivers
    • Plug & Play
    • The Windows Driver Model (WDM)
    • The Windows Driver Framework (WDF)
    • WDF: KMDF and UMDF
    • Device and Driver Objects
    • I/O Processing and Data Flow
    • IRPs
    • Power Management
    • Driver Verifier
    • Writing a Software Driver
    • Labs: viewing driver and device information; writing a software driver
  • Module 8: Security
    • Security Components
    • Virtualization Based Security
    • Hyper-V
    • Protecting objects
    • SIDs
    • User Access Control (UAC)
    • Tokens
    • Integrity Levels
    • ACLs
    • Privileges
    • Access checks
    • AppContainers
    • Logon
    • Control Flow Guard (CFG)
    • Process mitigations
    • Lab: viewing security information

zodiacon

💾

How can I close a handle in another process?

Many of you are probably familiar with Process Explorer‘s ability to close a handle in any process. How can this be accomplished programmatically?

The standard CloseHandle function can close a handle in the current process only, and most of the time that’s a good thing. But what if you need, for whatever reason, to close a handle in another process?

There are two routes than can be taken here. The first one is using a kernel driver. If this is a viable option, then nothing can prevent you from doing the deed. Process Explorer uses that option, since it has a kernel driver (if launched with admin priveleges at least once). In this post, I will focus on user mode options, some of which are applicable to kernel mode as well.

The first issue to consider is how to locate the handle in question, since its value is unknown in advance. There must be some criteria for which you know how to identify the handle once you stumble upon it. The easiest (and probably most common) case is a handle to a named object.

Let take a concrete example, which I believe is now a classic, Windows Media Player. Regardless of what opnions you may have regarding WMP, it still works. One of it quirks, is that it only allows a single instance of itself to run. This is accomplished by the classic technique of creating a named mutex when WMP comes up, and if it turns out the named mutex already exists (presumabley created by an already existing instance of itself), send a message to its other instance and then exits.

The following screenshot shows the handle in question in a running WMP instance.

wmp1

This provides an opportunity to close that mutex’ handle “behind WMP’s back” and then being able to launch another instance. You can try this by manually closing the handle with Process Explorer and then launch another WMP instance successfully.

If we want to achieve this programmatically, we have to locate the handle first. Unfortunately, the documented Windows API does not provide a way to enumerate handles, not even in the current process. We have to go with the (officially undocumented) Native API if we want to enumerate handles. There two routes we can use:

  1. Enumerate all handles in the system with NtQuerySystemInformation, search for the handle in the PID of WMP.
  2. Enumerate all handles in the WMP process only, searching for the handle yet again.
  3. Inject code into the WMP process to query handles one by one, until found.

Option 3 requires code injection, which can be done by using the CreateRemoteThreadEx function, but requires a DLL that we inject. This technique is very well-known, so I won’t repeat it here. It has the advantage of not requring some of the native APIs we’ll be using shortly.

Options 1 and 2 look very similar, and for our purposes, they are. Option 1 retrieves too much information, so it’s probably better to go with option 2.

Let’s start at the beginning: we need to locate the WMP process. Here is a function to do that, using the Toolhelp API for process enumeration:

#include <windows.h>
#include <TlHelp32.h>
#include <stdio.h>

DWORD FindMediaPlayer() {
	HANDLE hSnapshot = ::CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
	if (hSnapshot == INVALID_HANDLE_VALUE)
		return 0;

	PROCESSENTRY32 pe;
	pe.dwSize = sizeof(pe);

	// skip the idle process
	::Process32First(hSnapshot, &pe);
	
	DWORD pid = 0;
	while (::Process32Next(hSnapshot, &pe)) {
		if (::_wcsicmp(pe.szExeFile, L"wmplayer.exe") == 0) {
			// found it!
			pid = pe.th32ProcessID;
			break;
		}
	}
	::CloseHandle(hSnapshot);
	return pid;
}


int main() {
	DWORD pid = FindMediaPlayer();
	if (pid == 0) {
		printf("Failed to locate media player\n");
		return 1;
	}
	printf("Located media player: PID=%u\n", pid);
	return 0;
}

Now that we have located WMP, let’s get all handles in that process. The first step is opening a handle to the process with PROCESS_QUERY_INFORMATION and PROCESS_DUP_HANDLE (we’ll see why that’s needed in a little bit):

HANDLE hProcess = ::OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_DUP_HANDLE,
	FALSE, pid);
if (!hProcess) {
	printf("Failed to open WMP process handle (error=%u)\n",
		::GetLastError());
	return 1;
}

If we can’t open a proper handle, then something is terribly wrong. Maybe WMP closed in the meantime?

Now we need to work with the native API to query the handles in the WMP process. We’ll have to bring in some definitions, which you can find in the excellent phnt project on Github (I added extern "C" declaration because we use a C++ file).

#include <memory>

#pragma comment(lib, "ntdll")

#define NT_SUCCESS(status) (status >= 0)

#define STATUS_INFO_LENGTH_MISMATCH      ((NTSTATUS)0xC0000004L)

enum PROCESSINFOCLASS {
	ProcessHandleInformation = 51
};

typedef struct _PROCESS_HANDLE_TABLE_ENTRY_INFO {
	HANDLE HandleValue;
	ULONG_PTR HandleCount;
	ULONG_PTR PointerCount;
	ULONG GrantedAccess;
	ULONG ObjectTypeIndex;
	ULONG HandleAttributes;
	ULONG Reserved;
} PROCESS_HANDLE_TABLE_ENTRY_INFO, * PPROCESS_HANDLE_TABLE_ENTRY_INFO;

// private
typedef struct _PROCESS_HANDLE_SNAPSHOT_INFORMATION {
	ULONG_PTR NumberOfHandles;
	ULONG_PTR Reserved;
	PROCESS_HANDLE_TABLE_ENTRY_INFO Handles[1];
} PROCESS_HANDLE_SNAPSHOT_INFORMATION, * PPROCESS_HANDLE_SNAPSHOT_INFORMATION;

extern "C" NTSTATUS NTAPI NtQueryInformationProcess(
	_In_ HANDLE ProcessHandle,
	_In_ PROCESSINFOCLASS ProcessInformationClass,
	_Out_writes_bytes_(ProcessInformationLength) PVOID ProcessInformation,
	_In_ ULONG ProcessInformationLength,
	_Out_opt_ PULONG ReturnLength);

The #include <memory> is for using unique_ptr<> as we’ll do soon enough. The #parma links the NTDLL import library so that we don’t get an “unresolved external” when calling NtQueryInformationProcess. Some people prefer getting the functions address with GetProcAddress so that linking with the import library is not necessary. I think using GetProcAddress is important when using a function that may not exist on the system it’s running on, otherwise the process will crash at startup, when the loader (code inside NTDLL.dll) tries to locate a function. It does not care if we check dynamically whether to use the function or not – it will crash. Using GetProcAddress will just fail and the code can handle it. In our case, NtQueryInformationProcess existed since the first Windows NT version, so I chose to go with the simplest route.

Our next step is to enumerate the handles with the process information class I plucked from the full list in the phnt project (ntpsapi.h file):

ULONG size = 1 << 10;
std::unique_ptr<BYTE[]> buffer;
for (;;) {
	buffer = std::make_unique<BYTE[]>(size);
	auto status = ::NtQueryInformationProcess(hProcess, ProcessHandleInformation, 
		buffer.get(), size, &size);
	if (NT_SUCCESS(status))
		break;
	if (status == STATUS_INFO_LENGTH_MISMATCH) {
		size += 1 << 10;
		continue;
	}
	printf("Error enumerating handles\n");
	return 1;
}

The Query* style functions in the native API request a buffer and return STATUS_INFO_LENGTH_MISMATCH if it’s not large enough or not of the correct size. The code allocates a buffer with make_unique<BYTE[]> and tries its luck. If the buffer is not large enough, it receives back the required size and then reallocates the buffer before making another call.

Now we need to step through the handles, looking for our mutex. The information returned from each handle does not include the object’s name, which means we have to make yet another native API call, this time to NtQyeryObject along with some extra required definitions:

typedef enum _OBJECT_INFORMATION_CLASS {
	ObjectNameInformation = 1
} OBJECT_INFORMATION_CLASS;

typedef struct _UNICODE_STRING {
	USHORT Length;
	USHORT MaximumLength;
	PWSTR  Buffer;
} UNICODE_STRING;

typedef struct _OBJECT_NAME_INFORMATION {
	UNICODE_STRING Name;
} OBJECT_NAME_INFORMATION, * POBJECT_NAME_INFORMATION;

extern "C" NTSTATUS NTAPI NtQueryObject(
	_In_opt_ HANDLE Handle,
	_In_ OBJECT_INFORMATION_CLASS ObjectInformationClass,
	_Out_writes_bytes_opt_(ObjectInformationLength) PVOID ObjectInformation,
	_In_ ULONG ObjectInformationLength,
	_Out_opt_ PULONG ReturnLength);

NtQueryObject has several information classes, but we only need the name. But what handle do we provide NtQueryObject? If we were going with option 3 above and inject code into WMP’s process, we could loop with handle values starting from 4 (the first legal handle) and incrementing the loop handle by four.

Here we are in an external process, so handing out the handles provided by NtQueryInformationProcess does not make sense. What we have to do is duplicate each handle into our own process, and then make the call. First, we set up a loop for all handles and duplicate each one:

auto info = reinterpret_cast<PROCESS_HANDLE_SNAPSHOT_INFORMATION*>(buffer.get());
for (ULONG i = 0; i < info->NumberOfHandles; i++) {
	HANDLE h = info->Handles[i].HandleValue;
	HANDLE hTarget;
	if (!::DuplicateHandle(hProcess, h, ::GetCurrentProcess(), &hTarget, 
		0, FALSE, DUPLICATE_SAME_ACCESS))
		continue;	// move to next handle
	}

We duplicate the handle from WMP’s process (hProcess) to our own process. This function requires the handle to the process opened with PROCESS_DUP_HANDLE.

Now for the name: we need to call NtQueryObject with our duplicated handle and buffer that should be filled with UNICODE_STRING and whatever characters make up the name.

BYTE nameBuffer[1 << 10];
auto status = ::NtQueryObject(hTarget, ObjectNameInformation, 
	nameBuffer, sizeof(nameBuffer), nullptr);
::CloseHandle(hTarget);
if (!NT_SUCCESS(status))
	continue;

Once we query for the name, the handle is not needed and can be closed, so we don’t leak handles in our own process. Next, we need to locate the name and compare it with our target name. But what is the target name? We see in Process Explorer how the name looks. It contains the prefix used by any process (except UWP processes): “\Sessions\<session>\BasedNameObjects\<thename>”. We need the session ID and the “real” name to build our target name:

WCHAR targetName[256];
DWORD sessionId;
::ProcessIdToSessionId(pid, &sessionId);
::swprintf_s(targetName,
	L"\\Sessions\\%u\\BaseNamedObjects\\Microsoft_WMP_70_CheckForOtherInstanceMutex", 
	sessionId);
auto len = ::wcslen(targetName);

This code should come before the loop begins, as we only need to build it once.

Not for the real comparison of names:

auto name = reinterpret_cast<UNICODE_STRING*>(nameBuffer);
if (name->Buffer && 
	::_wcsnicmp(name->Buffer, targetName, len) == 0) {
	// found it!
}

The name buffer is cast to a UNICODE_STRING, which is the standard string type in the native API (and the kernel). It has a Length member which is in bytes (not characters) and does not have to be NULL-terminated. This is why the function used is _wcsnicmp, which can be limited in its search for a match.

Assuming we find our handle, what do we do with it? Fortunately, there is a trick we can use that allows closing a handle in another process: call DuplicateHandle again, but add the DUPLICATE_CLOSE_SOURCE to close the source handle. Then close our own copy, and that’s it! The mutex is gone. Let’s do it:

// found it!
::DuplicateHandle(hProcess, h, ::GetCurrentProcess(), &hTarget,
	0, FALSE, DUPLICATE_CLOSE_SOURCE);
::CloseHandle(hTarget);
printf("Found it! and closed it!\n");
return 0;

This is it. If we get out of the loop, it means we failed to locate the handle with that name. The general technique of duplicating a handle and closing the source is applicable to kernel mode as well. It does require a process handle with PROCESS_DUP_HANDLE to make it work, which is not always possible to get from user mode. For example, protected and PPL (protected processes light) processes cannot be opened with this access mask, even by administrators. In kernel mode, on the other hand, any process can be opened with full access.

wmp1

💾

zodiacon

💾

Creating Registry Links

The standard Windows Registry contains some keys that are not real keys, but instead are symbolic links (or simply, links) to other keys. For example, the key HKEY_LOCAL_MACHINE\System\CurrentControlSet is a symbolic link to HKEY_LOCAL_MACHINE\System\ControlSet001 (in most cases). When working with the standard Registry editor, RegEdit.exe, symbolic links look like normal keys, in the sense that they behave as the link’s target. The following figure shows the above mentioned keys. They look exactly the same (and they are).

There are several other existing links in the Registry. As another example, the hive HKEY_CURRENT_CONFIG is a link to (HKLM is HKEY_LOCAL_MACHINE) HKLM\SYSTEM\CurrentControlSet\Hardware Profiles\Current.

But how to do you create such links yourself? The official Microsoft documentation has partial details on how to do it, and it misses two critical pieces of information to make it work.

Let’s see if we can create a symbolic link. One rule of Registry links, is that the link must point to somewhere within the same hive where the link is created; we can live with that. For demonstration purposes, we’ll create a link in HKEY_CURRENT_USER named DesktopColors that links to HKEY_CURRENT_USER\Control Panel\Desktop\Colors.

The first step is to create the key and specify it to be a link rather than a normal key (error handling omitted):

HKEY hKey;
RegCreateKeyEx(HKEY_CURRENT_USER, L"DesktopColors", 0, nullptr,
	REG_OPTION_CREATE_LINK, KEY_WRITE, nullptr, &hKey, nullptr);

The important part is that REG_OPTION_CREATE_LINK flag that indicates this is supposed to be a link rather than a standard key. The KEY_WRITE access mask is required as well, as we are about to set the link’s target.

Now comes the first tricky part. The documentation states that the link’s target should be written to a value named “SymbolicLinkValue” and it must be an absolute registry path. Sounds easy enough, right? Wrong. The issue here is the “absolute path” – you might think that it should be something like “HKEY_CURRENT_USER\Control Panel\Desktop\Colors” just like we want, but hey – maybe it’s supposed to be “HKCU” instead of “HKEY_CURRENT_USER” – it’s just a string after all.

It turns out both these variants are wrong. The “absolute path” required here is a native Registry path that is not visible in RegEdit.exe, but it is visible in my own Registry editing tool, RegEditX.exe, downloadable from https://github.com/zodiacon/AllTools. Here is a screenshot, showing the “real” Registry vs. the view we get with RegEdit.

This top view is the “real” Registry is seen by the Windows kernel. Notice there is no HKEY_CURRENT_USER, there is a USER key where subkeys exist that represent users on this machine based on their SIDs. These are mostly visible in the standard Registry under the HKEY_USERS hive.

The “absolute path” needed is based on the real view of the Registry. Here is the code that writes the correct path based on my (current user’s) SID:

WCHAR path[] = L"\\REGISTRY\\USER\\S-1-5-21-2575492975-396570422-1775383339-1001\\Control Panel\\Desktop\\Colors";
RegSetValueEx(hKey, L"SymbolicLinkValue", 0, REG_LINK, (const BYTE*)path,
    wcslen(path) * sizeof(WCHAR));

The above code shows the second (undocumented, as far as I can tell) piece of crucial information – the length of the link path (in bytes) must NOT include the NULL terminator. Good luck guessing that 🙂

And that’s it. We can safely close the key and we’re done.

Well, almost. If you try to delete your newly created key using RegEdit.exe – the target is deleted, rather than the link key itself! So, how do you delete the key link? (My RegEditX does not support this yet).

The standard RegDeleteKey and RegDeleteKeyEx APIs are unable to delete a link. Even if they’re given a key handle opened with REG_OPTION_OPEN_LINK – they ignore it and go for the target. The only API that works is the native NtDeleteKey function (from NtDll.Dll).

First, we add the function’s declaration and the NtDll import:

extern "C" int NTAPI NtDeleteKey(HKEY);

#pragma comment(lib, "ntdll")

Now we can delete a link key like so:

HKEY hKey;
RegOpenKeyEx(HKEY_CURRENT_USER, L"DesktopColors", REG_OPTION_OPEN_LINK, 
    DELETE, &hKey);
NtDeleteKey(hKey);

As a final note, RegCreateKeyEx cannot open an existing link key – it can only create one. This in contrast to standard keys that can be created OR opened with RegCreateKeyEx. This means that if you want to change an existing link’s target, you have to call RegOpenKeyEx first (with REG_OPTION_OPEN_LINK) and then make the change (or delete the link key and re-create it).

Isn’t Registry fun?

reglink2

zodiacon

Upcoming Public Remote Training

I have recently completed another successful iteration of the Windows Internals training – thank you those who participated!

I am announcing two upcoming training classes, Windows Internals and Windows Kernel Programming.

Windows Internals (5 days)

I promised some folks that the next Internals training would be convenient to US-based time zones. That said, all time zones are welcome!

Dates: Sep 29, Oct 1, 5, 7, 8
Times: 8am to 4pm Pacific time (11am to 7pm Eastern)

The syllabus can be found here. I may make small changes in the final topics, but the major topics remain the same.

Windows Kernel Programming (4 days)

Dates: Oct 13, 15, 19, 21
Times: TBA

The syllabus can be found here. Again, slight changes are possible. This is a development-heavy course, so be prepared to write lots of code!

The selected time zone will be based on the majority of participants’ preference.

Cost and Registration

The cost for each class is kept relatively low (as opposed to other, perhaps similar offerings), as I’ve done in the past year or so. This is to make these classes accessible to more people, especially in these challenging times. If you register for both classes, you get 10% off the second class. Previous students of my classes get 10% off as well.

Cost: 750 USD if paid by an individual, 1500 USD if paid by a company. Multiple participants from the same company are entitled to a discount (email me for the details).

To register, send an email to [email protected] and specify “Training” in the title. The email should include your name, company name (if any) and preferred time zone.

Please read carefully the pre-requisites of each class, especially for Windows Kernel Programming. In case of doubt, talk to me.

If you have any questions, feel free to shoot me an email, or DM me on twitter (@zodiacon) or Linkedin (https://www.linkedin.com/in/pavely/).

For Companies

Companies that are interested in such (or other) training classes receive special prices. Topics can also be customized according to specific needs.

Other classes I provide include: Modern C++ Programming, Windows System Programming, COM Programming, C#/.NET Programming (Basic and Advanced), Advanced Windows Debugging, and more. Contact me for detailed syllabi if interested.

Platform-Ico-n-Transparent-3-Windows

zodiacon

Next Public (Remote) Training Classes

I am announcing the next Windows Internals remote training to be held in January 2021 on the 19, 21, 25, 27, 28. Times: 11am to 7pm, London time.

The syllabus can be found here.

I am also announcing a new training, requested by quite a few people, Windows System Programming. The dates are in February 2021: 8, 9, 11, 15, 17. Times: 12pm to 8pm, London time.

The syllabus can be found here.

Cost and Registration

I’m keeping the cost of these training classes relatively low. This is to make these classes accessible to more people, especially in these challenging times. If you register for both classes, you get 10% off the second class. Previous students of my classes get 10% off as well.

Cost: 750 USD if paid by an individual, 1500 USD if paid by a company. Multiple participants from the same company are entitled to a discount (email me for the details).

To register, send an email to [email protected] and specify “Training” in the title. The email should include the training(s) you’re interested in, your name, contact email, company name (if any) and preferred time zone. The training times have already been selected, but it’s still good to know which time zone you live in.

As usual, if you have any questions, feel free to shoot me an email, or DM me on twitter (@zodiacon) or Linkedin (https://www.linkedin.com/in/pavely/).

Platform-Ico-n-Transparent-3-Windows

zodiacon

Parent Process vs. Creator Process

Normally, a process created by another process in Windows establishes a parent-child relationship between them. This relationship can be viewed in tools such as Process Explorer. Here is an example showing FoxitReader as a child process of Explorer, implying Explorer created FoxitReader. That must be the case, right?

First, it’s important to emphasize that there is no dependency between a child and a parent process in any way. For example, if the parent process terminates, the child is unaffected. The parent is used for inheriting many properties of the child process, such as current directory and environment variables (if not specified explicitly).

Windows stores the parent process ID in the process management object of the child in the kernel for posterity. This also means that the process ID, although correct, could point to a “wrong” process. This can happen if the parent dies, and the process ID is reused.

Process Explorer does not get confused in such a case, and left-justifies the process in its tree view. It knows to check the process creation time. If the “parent” was created after the child, there is no way it could be the real parent. The parent process ID can still be viewed in the process properties:

Notice the “<Non-existent Parent>” indication. Although the parent process ID is known (13636 in this case), there is no other information on that process, as it does not exist anymore, and its ID may or may not have been reused.

By the way, don’t be alarmed that Explorer has no living parent. This is expected, as it’s normally created by a process running the image UserInit.exe, that exits normally after finishing its duties.

Returning to the question raised earlier – is the parent process always the actual creator? Not necessarily.

The canonical example is when a process is launched elevated (for example, by right-clicking it in Explorer and selecting “Run as Administrator”. Here is a diagram showing the major components in an elevation procedure:

First, the user right-clicks in Explorer and asks to run some App.Exe elevated. Explorer calls ShellExecute(Ex) with the verb “runas” that requests this elevation. Next, The AppInfo service is contacted to perform the operation if possible. It launches consent.exe, which shows the Yes/No message box (for true administrators) or a username/password dialog to get an admin’s approval for the elevation. Assuming this is granted, the AppInfo service calls CreateProcessAsUser to launch the App.exe elevated. To maintain the illusion that Explorer created App.exe, it stores the parent ID of Explorer into App.exe‘s new process management block. This makes it look as if Explorer had created App.exe, but is not really what happened. However, that “re-parenting” is probably a good idea in this case, giving the user the expected result.

Is it possible to specify a different parent when creating a process with CreateProcess?

It is indeed possible by using a process attribute. Here is a function that does just that (with minimal error handling):

bool CreateProcessWithParent(DWORD parentId, PWSTR commandline) {
    auto hProcess = ::OpenProcess(PROCESS_CREATE_PROCESS, FALSE, parentId);
    if (!hProcess)
        return false;

    SIZE_T size;
    //
    // call InitializeProcThreadAttributeList twice
    // first, get required size
    //
    ::InitializeProcThreadAttributeList(nullptr, 1, 0, &size);

    //
    // now allocate a buffer with the required size and call again
    //
    auto buffer = std::make_unique<BYTE[]>(size);
    auto attributes = reinterpret_cast<PPROC_THREAD_ATTRIBUTE_LIST>(buffer.get());
    ::InitializeProcThreadAttributeList(attributes, 1, 0, &size);

    //
    // add the parent attribute
    //
    ::UpdateProcThreadAttribute(attributes, 0, 
        PROC_THREAD_ATTRIBUTE_PARENT_PROCESS, 
        &hProcess, sizeof(hProcess), nullptr, nullptr);

    STARTUPINFOEX si = { sizeof(si) };
    //
    // set the attribute list
    //
    si.lpAttributeList = attributes;
    PROCESS_INFORMATION pi;

    //
    // create the process
    //
    BOOL created = ::CreateProcess(nullptr, commandline, nullptr, nullptr, 
        FALSE, EXTENDED_STARTUPINFO_PRESENT, nullptr, nullptr, 
        (STARTUPINFO*)&si, &pi);

    //
    // cleanup
    //
    ::CloseHandle(hProcess);
    ::DeleteProcThreadAttributeList(attributes);

    return created;
}

The first step is to open a handle to the “prospected” parent by calling OpenProcess with the PROCESS_CREATE_PROCESS access mask. This means you cannot choose an arbitrary process, you must have at least that access to the process. Protected (and protected light) processes, for example, cannot be used as parents in this way, as PROCESS_CREATE_PROCESS cannot be obtained for such processes.

Next, an attribute list is created with a single attribute of type PROC_THREAD_ATTRIBUTE_PARENT_PROCESS by calling InitializeProcThreadAttributeList followed by UpdateProcThreadAttribute to set up the parent handle. Finally, CreateProcess is called with the extended STARTUPINFOEX structure where the attribute list is stored. CreateProcess must know about the extended version by specifying EXTENDED_STARTUPINFO_PRESENT in its flags argument.

Here is an example where Excel supposedly created Notepad (it really didn’t). The “real” process creator is nowhere to be found.

The obvious question perhaps is whether there is any way to know which process was the real creator?

The only way that seems to be available is for kernel drivers that register for process creation notifications with PsSetCreateProcessNotifyRoutineEx (or one of its variants). When such a notification is invoked in the driver, the following structure is provided:

typedef struct _PS_CREATE_NOTIFY_INFO {
  SIZE_T              Size;
  union {
    ULONG Flags;
    struct {
      ULONG FileOpenNameAvailable : 1;
      ULONG IsSubsystemProcess : 1;
      ULONG Reserved : 30;
    };
  };
  HANDLE              ParentProcessId;
  CLIENT_ID           CreatingThreadId;
  struct _FILE_OBJECT *FileObject;
  PCUNICODE_STRING    ImageFileName;
  PCUNICODE_STRING    CommandLine;
  NTSTATUS            CreationStatus;
} PS_CREATE_NOTIFY_INFO, *PPS_CREATE_NOTIFY_INFO;

On the one hand, there is the ParentProcessId member (although it’s typed as HANDLE, it actually the parent process ID). This is the parent process as set by CreateProcess based on the code snippet in CreateProcessWithParent.

However, there is also the CreatingThreadId member of type CLIENT_ID, which is a small structure containing a process and thread IDs. This is the real creator. In most cases it’s going to have the same process ID as ParentProcessId, but not in the case where a different parent has been specified.

After this point, that information goes away, leaving only the parent ID, as it’s the one stored in the EPROCESS kernel structure of the child process.

Update: (thanks to @jdu2600)

The creator is available by listening to the process create ETW event, where the creator is the one raising the event. Here is a snapshot from my own ProcMonXv2:

runelevated-1

zodiacon

Next Public Windows Internals training

I am announcing the next Windows Internals remote training to be held in July 2021 on the 12, 14, 15, 19, 21. Times: 11am to 7pm, London time.

The syllabus can be found here (slight changes are possible if new important topics come up).

Cost and Registration

I’m keeping the cost of these training classes relatively low. This is to make these classes accessible to more people, especially in these unusual and challenging times.

Cost: 800 USD if paid by an individual, 1500 USD if paid by a company. Multiple participants from the same company are entitled to a discount (email me for the details). Previous students of my classes are entitled to a 10% discount.

To register, send an email to [email protected] and specify “Windows Internals Training” in the title. The email should include your name, contact email, and company name (if any).

Later this year I plan a Windows Kernel Programming class. Stay tuned!

As usual, if you have any questions, feel free to send me an email, or DM me on twitter (@zodiacon) or Linkedin (https://www.linkedin.com/in/pavely/).

Platform-Ico-n-Transparent-3-Windows

zodiacon

Dynamic Symbolic Links

While teaching a Windows Internals class recently, I came across a situation which looked like a bug to me, but turned out to be something I didn’t know about – dynamic symbolic links.

Symbolic links are Windows kernel objects that point to another object. The weird situation in question was when running WinObj from Sysinternals and navigating to the KenrelObjects object manager directory.

WinObj from Sysinternals

You’ll notice some symbolic link objects that look weird: MemoryErrors, PhysicalMemoryChange, HighMemoryCondition, LowMemoryCondition and a few others. The weird thing that is fairly obvious is that these symbolic link objects have empty targets. Double-clicking any one of them confirms no target, and also shows a curious zero handles, as well as quota change of zero:

Symbolic link properties

To add to the confusion, searching for any of them with Process Explorer yields something like this:

It seems these objects are events, and not symbolic links!

My first instinct was that there is a bug in WinObj (I rewrote it recently for Sysinternals, so was certain I introduced a bug). I ran an old WinObj version, but the result was the same. I tried other tools with similar functionality, and still got the same results. Maybe a bug in Process Explorer? Let’s see in the kernel debugger:

lkd> !object 0xFFFF988110EC0C20
Object: ffff988110ec0c20  Type: (ffff988110efb400) Event
    ObjectHeader: ffff988110ec0bf0 (new version)
    HandleCount: 4  PointerCount: 117418
    Directory Object: ffff828b10689530  Name: HighCommitCondition

Definitely an event and not a symbolic link. What’s going on? I debugged it in WinObj, and indeed the reported object type is a symbolic link. Maybe it’s a bug in the NtQueryDirectoryObject used to query a directory object for an object.

I asked Mark Russinovich, could there be a bug in Windows? Mark remembered that this is not a bug, but a feature of symbolic links, where objects can be created/resolved dynamically when accessing the symbolic link. Let’s see if we can see something in the debugger:

lkd> !object \kernelobjects\highmemorycondition
Object: ffff828b10659510  Type: (ffff988110e9ba60) SymbolicLink
    ObjectHeader: ffff828b106594e0 (new version)
    HandleCount: 0  PointerCount: 1
    Directory Object: ffff828b10656ce0  Name: HighMemoryCondition
    Flags: 0x000010 ( Local )
    Target String is '*** target string unavailable ***'

Clearly, there is target, but notice the flags value 0x10. This is the flag indicating the symbolic link is a dynamic one. To get further information, we need to look at the object with a “symbolic link lenses” by using the data structure the kernel uses to represent symbolic links:

lkd> dt nt!_OBJECT_SYMBOLIC_LINK ffff828b10659510

   +0x000 CreationTime     : _LARGE_INTEGER 0x01d73d87`21bd21e5
   +0x008 LinkTarget       : _UNICODE_STRING "--- memory read error at address 0x00000000`00000005 ---"
   +0x008 Callback         : 0xfffff802`08512250     long  nt!MiResolveMemoryEvent+0

   +0x010 CallbackContext  : 0x00000000`00000005 Void
   +0x018 DosDeviceDriveIndex : 0
   +0x01c Flags            : 0x10
   +0x020 AccessMask       : 0x24

The Callback member shows the function that is being called (MiResolveMemoryEvent) that “resolves” the symbolic link to the relevant event. There are currently 11 such events, their names visible with the following:

lkd> dx (nt!_UNICODE_STRING*)&nt!MiMemoryEventNames,11
(nt!_UNICODE_STRING*)&nt!MiMemoryEventNames,11                 : 0xfffff80207e02e90 [Type: _UNICODE_STRING *]
    [0]              : "\KernelObjects\LowPagedPoolCondition" [Type: _UNICODE_STRING]
    [1]              : "\KernelObjects\HighPagedPoolCondition" [Type: _UNICODE_STRING]
    [2]              : "\KernelObjects\LowNonPagedPoolCondition" [Type: _UNICODE_STRING]
    [3]              : "\KernelObjects\HighNonPagedPoolCondition" [Type: _UNICODE_STRING]
    [4]              : "\KernelObjects\LowMemoryCondition" [Type: _UNICODE_STRING]
    [5]              : "\KernelObjects\HighMemoryCondition" [Type: _UNICODE_STRING]
    [6]              : "\KernelObjects\LowCommitCondition" [Type: _UNICODE_STRING]
    [7]              : "\KernelObjects\HighCommitCondition" [Type: _UNICODE_STRING]
    [8]              : "\KernelObjects\MaximumCommitCondition" [Type: _UNICODE_STRING]
    [9]              : "\KernelObjects\MemoryErrors" [Type: _UNICODE_STRING]
    [10]             : "\KernelObjects\PhysicalMemoryChange" [Type: _UNICODE_STRING]

Creating dynamic symbolic links is only possible from kernel mode, of course, and is undocumented anyway.

At least the conundrum is solved.

image

zodiacon

Next Windows Kernel Programming Training

Today I’m announcing the next public remote Windows Kernel Programming training. This is a 5-day training scheduled for October: 4, 5, 7, 11, 13. Times: 12pm to 8pm, London Time.

The syllabus can be found here. It may be slightly modified by the time the class starts, but not by much. This is a development-heavy course, so be prepared to write lots of code!

Cost: 800 USD if paid by an individual, 1500 USD if paid by a company. Previous participants of the my classes get 10% discount. Multiple participants from the same company are entitled to a discount (email me for the details).

To register, send an email to [email protected] and specify “Windows Kernel Programming Training” in the title. The email should include your name, preferred email for communication, and company name (if any).

The training sessions will be recorded and provided to the participants.

Please read carefully the pre-requisites for this class. You should especially be comfortable coding in C (any C++ used in the class will be explained). In case of any doubt, talk to me.
If you have any questions, feel free to shoot me an email, or DM me on twitter (@zodiacon) or Linkedin (https://www.linkedin.com/in/pavely/).

Kernel2

zodiacon

Processes, Threads, and Windows

The relationship between processes and threads is fairly well known – a process contains one or more threads, running within the process, using process wide resources, like handles and address space.

Where do windows (those usually-rectangular elements, not the OS) come into the picture?

The relationship between a process, its threads, and windows, along with desktop and Window Stations, can be summarized in the following figure:

A process, threads, windows, desktops and a Window Station

A window is created by a thread, and that thread is designated as the owner of the window. The creating thread becomes a UI thread (if it’s the first User/GDI call it makes), getting a message queue (provided internally by Win32k.sys in the kernel). Every occurrence in the created window causes a message to be placed in the message queue of the owner thread. This means that if a thread creates 100 windows, it’s responsible for all these windows.

“Responsible” here means that the thread must perform something known as “message pumping” – pulling messages from its message queue and processing them. Failure to do that processing would lead to the infamous “Not Responding” scenario, where all the windows created by that thread become non-responsive. Normally, a thread can read its own message queue only – other threads can’t do it for that thread – this is a single threaded UI scheme used by the Windows OS by default. The simplest message pump would look something like this:

MSG msg;
while(::GetMessage(&msg, nullptr, 0, 0)) {
    ::TranslateMessage(&msg);
    ::DispatchMessage(&msg);
}

The call to GetMessage does not return until there is a message in the queue (filling up the MSG structure), or the WM_QUIT message has been retrieved, causing GetMessage to return FALSE and the loop exited. WM_QUIT is the message typically used to signal an application to close. Here is the MSG structure definition:

typedef struct tagMSG {
  HWND   hwnd;
  UINT   message;
  WPARAM wParam;
  LPARAM lParam;
  DWORD  time;
  POINT  pt;
} MSG, *PMSG;

Once a message has been retrieved, the important call is to DispatchMessage, that looks at the window handle in the message (hwnd) and looks up the window class that this window instance is a member of, and there, finally, the window procedure – the callback to invoke for messages destined to these kinds of windows – is invoked, handling the message as appropriate or passing it along to default processing by calling DefWindowProc. (I may expand on that if there is interest in a separate post).

You can enumerate the windows owned by a thread by calling EnumThreadWindows. Let’s see an example that uses the Toolhelp API to enumerate all processes and threads in the system, and lists the windows created (owned) by each thread (if any).

We’ll start by creating a snapshot of all processes and threads (error handling mostly omitted for clarity):

#include <Windows.h>
#include <TlHelp32.h>
#include <unordered_map>
#include <string>

int main() {
	auto hSnapshot = ::CreateToolhelp32Snapshot(
        TH32CS_SNAPPROCESS | TH32CS_SNAPTHREAD, 0);

We’ll build a map of process IDs to names for easy lookup later on when we show details of a process:

PROCESSENTRY32 pe;
pe.dwSize = sizeof(pe);

// we can skip the idle process
::Process32First(hSnapshot, &pe);

std::unordered_map<DWORD, std::wstring> processes;
processes.reserve(500);

while (::Process32Next(hSnapshot, &pe)) {
	processes.insert({ pe.th32ProcessID, pe.szExeFile });
}

Now we loop through all the threads, calling EnumThreadWindows and showing the results:

THREADENTRY32 te;
te.dwSize = sizeof(te);
::Thread32First(hSnapshot, &te);

static int count = 0;

struct Data {
	DWORD Tid;
	DWORD Pid;
	PCWSTR Name;
};
do {
	if (te.th32OwnerProcessID <= 4) {
		// idle and system processes have no windows
		continue;
	}
	Data d{ 
        te.th32ThreadID, te.th32OwnerProcessID, 
        processes.at(te.th32OwnerProcessID).c_str() 
    };
	::EnumThreadWindows(te.th32ThreadID, [](auto hWnd, auto param) {
		count++;
		WCHAR text[128], className[32];
		auto data = reinterpret_cast<Data*>(param);
		int textLen = ::GetWindowText(hWnd, text, _countof(text));
		int classLen = ::GetClassName(hWnd, className, _countof(className));
		printf("TID: %6u PID: %6u (%ws) HWND: 0x%p [%ws] %ws\n",
			data->Tid, data->Pid, data->Name, hWnd,
			classLen == 0 ? L"" : className,
			textLen == 0 ? L"" : text);
		return TRUE;	// bring in the next window
		}, reinterpret_cast<LPARAM>(&d));
} while (::Thread32Next(hSnapshot, &te));
printf("Total windows: %d\n", count);

EnumThreadWindows requires the thread ID to look at, and a callback that receives a window handle and whatever was passed in as the last argument. Returning TRUE causes the callback to be invoked with the next window until the window list for that thread is complete. It’s possible to return FALSE to terminate the enumeration early.

I’m using a lambda function here, but only a non-capturing lambda can be converted to a C function pointer, so the param value is used to provide context for the lambda, with a Data instance having the thread ID, its process ID, and the process name.

The call to GetClassName returns the name of the window class, or type of window, if you will. This is similar to the relationship between objects and classes in object orientation. For example, all buttons have the class name “button”. Of course, the windows enumerated are only top level windows (have no parent) – child windows are not included. It is possible to enumerate child windows by calling EnumChildWindows.

Going the other way around – from a window handle to its owner thread and *its* owner process is possible with GetWindowThreadProcessId.

Desktops

A window is placed on a desktop, and is bound to it. More precisely, a thread is bound to a desktop once it creates its first window (or has any hook installed with SetWindowsHookEx). Before that happens, a thread can attach itself to a desktop by calling SetThreadDesktop.

Normally, a thread will use the “default” desktop, which is where a logged in user sees his or her windows appearing. It is possible to create additional desktops using the CreateDesktop API. A classic example is the desktops Sysinternals tool. See my post on desktops and windows stations for more information.

It’s possible to redirect a new process to use a different desktop (and optionally a window station) as part of the CreateProcess call. The STARTUPINFO structure has a member named lpDesktop that can be set to a string with the format “WindowStationName\DesktopName” or simply “DesktopName” to keep the parent’s window station. This could look something like this:

STARTUPINFO si = { sizeof(si) };
// WinSta0 is the interactive window station
WCHAR desktop[] = L"Winsta0\\MyOtherDesktop";
si.lpDesktop = desktop;
PROCESS_INFORMATION pi;
::CreateProcess(..., &si, &pi);

“WinSta0” is always the name of the interactive Window Station (one of its desktops can be visible to the user).

The following example creates 3 desktops and launches notepad in the default desktop and the other desktops:

void LaunchInDesktops() {
	HDESK hDesktop[4] = { ::GetThreadDesktop(::GetCurrentThreadId()) };
	for (int i = 1; i <= 3; i++) {
		hDesktop[i] = ::CreateDesktop(
			(L"Desktop" + std::to_wstring(i)).c_str(),
			nullptr, nullptr, 0, GENERIC_ALL, nullptr);
	}
	STARTUPINFO si = { sizeof(si) };
	WCHAR name[32];
	si.lpDesktop = name;
	DWORD len;
	PROCESS_INFORMATION pi;
	for (auto h : hDesktop) {
		::GetUserObjectInformation(h, UOI_NAME, name, sizeof(name), &len);
		WCHAR pname[] = L"notepad";
		if (::CreateProcess(nullptr, pname, nullptr, nullptr, FALSE,
			0, nullptr, nullptr, &si, &pi)) {
			printf("Process created: %u\n", pi.dwProcessId);
			::CloseHandle(pi.hProcess);
			::CloseHandle(pi.hThread);
		}
	}
}

If yoy run the above function, you’ll see one Notepad window on your desktop. If you dig deeper, you’ll find 4 notepad.exe processes in Task Manager. The other three have created their windows in different desktops. Task Manager gives no indication of that, nor does the process list in Process Explorer. However, we see a hint in the list of handles of these “other” notepad processes. Here is one:

Handle named \Desktop1

Curiously enough, desktop objects are not stored as part of the kernel’s Object Manager namespace. If you double-click the handle, copy its address, and look it up in the kernel debugger, this is the result:

lkd> !object 0xFFFFAC8C12035530
Object: ffffac8c12035530  Type: (ffffac8c0f2fdbc0) Desktop
    ObjectHeader: ffffac8c12035500 (new version)
    HandleCount: 1  PointerCount: 32701
    Directory Object: 00000000  Name: Desktop1

Notice the directory object address is zero – it’s not part of any directory. It has meaning in the context of its containing Window Station.

You might think that we can use the EnumThreadWindows as demonstrated at the beginning of this post to locate the other notepad windows. Unfortunately, access is restricted and those notepad windows are not listed (more on that possibly in a future post).

We can try a different approach by using yet another enumration function – EnumDesktopWindows. Given a powerful enough desktop handle, we can enumerate all top level windows in that desktop:

void ListDesktopWindows(HDESK hDesktop) {
	::EnumDesktopWindows(hDesktop, [](auto hWnd, auto) {
		WCHAR text[128], className[32];
		int textLen = ::GetWindowText(hWnd, text, _countof(text));
		int classLen = ::GetClassName(hWnd, className, _countof(className));
		DWORD pid;
		auto tid = ::GetWindowThreadProcessId(hWnd, &pid);
		printf("TID: %6u PID: %6u HWND: 0x%p [%ws] %ws\n",
			tid, pid, hWnd, 
			classLen == 0 ? L"" : className,
			textLen == 0 ? L"" : text);
		return TRUE;
		}, 0);
}

The lambda body is very similar to the one we’ve seen earlier. The difference is that we start with a window handle with no thread ID. But it’s fairly easy to get the owner thread and process with GetWindowThreadProcessId. Here is one way to invoke this function:

auto hDesktop = ::OpenDesktop(L"Desktop1", 0, FALSE, DESKTOP_ENUMERATE);
ListDesktopWindows(hDesktop);

Here is the resulting output:

TID:   3716 PID:  23048 HWND: 0x0000000000192B26 [Touch Tooltip Window] Tooltip
TID:   3716 PID:  23048 HWND: 0x0000000001971B1A [UAC_InputIndicatorOverlayWnd]
TID:   3716 PID:  23048 HWND: 0x00000000002827F2 [UAC Input Indicator]
TID:   3716 PID:  23048 HWND: 0x0000000000142B82 [CiceroUIWndFrame] CiceroUIWndFrame
TID:   3716 PID:  23048 HWND: 0x0000000000032C7A [CiceroUIWndFrame] TF_FloatingLangBar_WndTitle
TID:  51360 PID:  23048 HWND: 0x0000000000032CD4 [Notepad] Untitled - Notepad
TID:   3716 PID:  23048 HWND: 0x0000000000032C8E [CicLoaderWndClass]
TID:  51360 PID:  23048 HWND: 0x0000000000202C62 [IME] Default IME
TID:  51360 PID:  23048 HWND: 0x0000000000032C96 [MSCTFIME UI] MSCTFIME UI

A Notepad window is clearly there. We can repeat the same exercise with the other two desktops, which I have named “Desktop2” and “Desktop3”.

Can we view and interact with these “other” notepads? The SwitchDesktop API exists for that purpose. Given a desktop handle with the DESKTOP_SWITCHDESKTOP access, SwitchDesktop changes the input desktop to the requested desktop, so that input devices redirect to that desktop. This is exactly how the Sysinternals Desktops tool works. I’ll leave the interested reader to try this out.

Window Stations

The last piece of the puzzle is a Window Station. I assume you’ve read my earlier post and understand the basics of Window Stations.

A Thread can be associated with a desktop. A process is associated with a window station, with the constraint being that any desktop used by a thread must be within the process’ window station. In other words, a process cannot use one window station and any one of its threads using a desktop that belongs to a different window station.

A process is associated with a window station upon creation – the one specified in the STARTUPINFO structure or the creator’s window station if not specified. Still, a process can associate itself with a different window station by calling SetProcessWindowStation. The constraint is that the provided window station must be part of the current session. There is one window station per session named “WinSta0”, called the interactive window station. One of its desktop can be the input desktop and have user interaction. All other window stations are non-interactive by definition, which is perfectly fine for processes that don’t require any user interface. In return, they get better isolation, because a window station contains its own set of desktops, its own clipboard and its own atom table. Windows handles, by the way, have Window Station scope.

Just like desktops, window stations can be created by calling CreateWindowStation. Window stations can be enumerated as well (in the current session) with EnumerateWindowStations.

Summary

This discussion of threads, processes, desktops, and window stations is not complete, but hopefully gives you a good idea of how things work. I may elaborate on some advanced aspects of Windows UI system in future posts.

procthreadwin-1

zodiacon

New Class: COM Programming

Today I’m announcing a new training – COM (Component Object Model) Programming, to be held in November.

COM is a well established technology, debuted back in 1993, and is still very much in use. In fact, the Windows Runtime is based on an enhanced version of COM. There is quite a bit of confusion and misconceptions about COM, which is one reason I decided to offer this class.

The syllabus for the 3 day class can be found here. This is the first time I will be offering this class, and also will try a new format: 6 half-days instead of 3 full days.

When: November: 8, 9, 10, 11, 15, 16. 2pm to 6pm, UT. (Technically it’s more than 3 days, as with a full day there is a lunch break not present in half days). The class will be conducted remotely using Microsoft Teams or a similar platform.

What you need to know before the class: You should be comfortable using Windows on a Power User level. Concepts such as processes, threads, DLLs, and virtual memory should be understood fairly well. You should have experience writing code in C and some C++. You don’t have to be an expert, but you must know C and basic C++ to get the most out of the class. In case you have doubts, talk to me.

Obviously, participants in my Windows Internals and (especially) Windows System Programming classes have the required knowledge for the class.

We’ll start by looking at why COM was created in the first place, and then build clients and servers, digging into various mechanisms COM provides. See the syllabus for more details.

Registration will be different than previous classes:
Early bird (paid by September 30th): 500 USD (if paid by an individual), 1100 USD (if paid by a company).
Normal (paid after September 30th): 700 USD (if paid by an individual), 1300 USD (if paid by a company).

To register, send an email to [email protected] with the title “COM Training”, and write the name(s), email(s) and time zone(s) of the participants. Multiple participants from the same company get a discount. Previous participants (individuals) of my classes get 10% off. I will reply with further instructions.

I hope to see you in November!

COMReuse

zodiacon

Next Windows Internals Training

I am announcing the next 5 day Windows Internals remote training to be held in January 2022, starting on the 24th according to the followng schedule:

  • Jan 24 – 2pm to 10pm (all times are based on London time)
  • Jan 25, 26, 27 – 2pm to 6pm
  • Jan 31 – 2pm to 10pm
  • Feb 1, 2, 3 – 2pm to 6pm

The syllabus can be found here (slight changes are possible if new important topics come up).

Cost and Registration

I’m keeping the cost of these training classes relatively low. This is to make these classes accessible to more people, especially in these unusual and challenging times.

Cost: 800 USD if paid by an individual, 1500 USD if paid by a company. Multiple participants from the same company are entitled to a discount (email me for the details). Previous students of my classes are entitled to a 10% discount.

To register, send an email to [email protected] and specify “Windows Internals Training” in the title. The email should include your name, contact email, time zone, and company name (if any).

As usual, if you have any questions, feel free to send me an email, or DM me on twitter (@zodiacon) or Linkedin (https://www.linkedin.com/in/pavely/).

Windows11

zodiacon

Icon Handler with ATL

One of the exercises I gave at the recent COM Programming class was to build an Icon Handler that integrates with the Windows Shell, where DLLs should have an icon based on their “bitness” – whether they’re 64-bit or 32-bit Portable Executable (PE).

The Shell provides many opportunities for extensibility. An Icon Handler is one of the simplest, but still requires writing a full-fledged COM component that implements certain interfaces that the shell expects. Here is the result of using the Icon Handler DLL, showing the folders c:\Windows\System32 and c:\Windows\SysWow64 (large icons for easier visibility).

C:\Windows\System32
C:\Windows\SysWow64

Let’s see how to build such an icon handler. The full code is at zodiacon/DllIconHandler.

The first step is to create a new ATL project in Visual Studio. I’ll be using Visual Studio 2022, but any recent version would work essentially the same way (e.g. VS 2019, or 2017). Locate the ATL project type by searching in the terrible new project dialog introduced in VS 2019 and still horrible in VS 2022.

ATL (Active Template Library) is certainly not the only way to build COM components. “Pure” C++ would work as well, but ATL provides all the COM boilerplate such as the required exported functions, class factories, IUnknown implementations, etc. Since ATL is fairly “old”, it lacks the elegance of other libraries such as WRL and WinRT, as it doesn’t take advantage of C++11 and later features. Still, ATL has withstood the test of time, is robust, and full featured when it comes to COM, something I can’t say for these other alternatives.

If you can’t locate the ATL project, you may not have ATL installed propertly. Make sure the C++ Desktop development workload is installed using the Visual Studio Installer.

Click Next and select a project name and location:

Click Create to launch the ATL project wizard. Leave all defaults (Dynamic Link Library) and click OK. Shell extensions of all kinds must be DLLs, as these are loaded by Explorer.exe. It’s not ideal in terms of Explorer’s stability, as aun unhandled exception can bring down the entire process, but this is necessary to get good performance, as no inter-process calls are made.

Two projects are created, named DllIconHandler and DllIconHandlerPS. The latter is a proxy/stub DLL that maybe useful if cross-apartment COM calls are made. This is not needed for shell extensions, so the PS project should simply be removed from the solution.


A detailed discussion of COM is way beyond the scope of this post.


The remaining project contains the COM DLL required code, such as the mandatory exported function, DllGetClassObject, and the other optional but recommended exports (DllRegisterServer, DllUnregisterServer, DllCanUnloadNow and DllInstall). This is one of the nice benefits of working with ATL project for COM component development: all the COM boilerplate is implemented by ATL.

The next step is to add a COM class that will implement our icon handler. Again, we’ll turn to a wizard provided by Visual Studio that provides the fundamentals. Right-click the project and select Add Item… (don’t select Add Class as it’s not good enough). Select the ATL node on the left and ATL Simple Object on the right. Set the name to something like IconHandler:

Click Add. The ATL New Object wizard opens up. The name typed in the Add New Item dialog is used as a basis for generating names for source code elements (like the C++ class) and COM elements (that would be written into the IDL file and the resulting type library). Since we’re not going to define a new interface (we need to implement explorer-defined interfaces), there is no real need to tweak anything. You can click Finish to generate the class.

Three files are added with this last step: IconHandler.h, IconHandler.cpp and IconHandler.rgs. The C++ source files role is obvious – implementing the Icon Handler. The rgs file contains a script in an ATL-provided “language” indicating what information to write to the Registry when this DLL is registered (and what to remove if it’s unregistered).

The IDL (Interface Definition Language) file has also been modified, adding the definitions of the wizard generated interface (which we don’t need) and the coclass. We’ll leave the IDL alone, as we do need it to generate the type library of our component because the ATL registration code uses it internally.

If you look in IconHandler.h, you’ll see that the class implements the IIconHandler empty interface generated by the wizard that we don’t need. It even derives from IDispatch:

class ATL_NO_VTABLE CIconHandler :
	public CComObjectRootEx<CComSingleThreadModel>,
	public CComCoClass<CIconHandler, &CLSID_IconHandler>,
	public IDispatchImpl<IIconHandler, &IID_IIconHandler, &LIBID_DLLIconHandlerLib, /*wMajor =*/ 1, /*wMinor =*/ 0> {

We can leave the IDispatchImpl-inheritance, since it’s harmless. But it’s useless as well, so let’s delete it and also delete the interfaces IIconHandler and IDispatch from the interface map located further down:

class ATL_NO_VTABLE CIconHandler :
	public CComObjectRootEx<CComSingleThreadModel>,
	public CComCoClass<CIconHandler, &CLSID_IconHandler> {
public:
	BEGIN_COM_MAP(CIconHandler)
	END_COM_MAP()

(I have rearranged the code a bit). Now we need to add the interfaces we truly have to implement for an icon handler: IPersistFile and IExtractIcon. To get their definitions, we’ll add an #include for <shlobj_core.h> (this is documented in MSDN). We add the interfaces to the inheritance hierarchy, the COM interface map, and use the Visual Studio feature to add the interface members for us by right-clicking the class name (CIconHandler), pressing Ctrl+. (dot) and selecting Implement all pure virtuals of CIconHandler. The resulting class header looks something like this (some parts omitted for clarity) (I have removed the virtual keyword as it’s inherited and doesn’t have to be specified in derived types):

class ATL_NO_VTABLE CIconHandler :
	public CComObjectRootEx<CComSingleThreadModel>,
	public CComCoClass<CIconHandler, &CLSID_IconHandler>,
	public IPersistFile,
	public IExtractIcon {
public:
	BEGIN_COM_MAP(CIconHandler)
		COM_INTERFACE_ENTRY(IPersistFile)
		COM_INTERFACE_ENTRY(IExtractIcon)
	END_COM_MAP()

//...
	// Inherited via IPersistFile
	HRESULT __stdcall GetClassID(CLSID* pClassID) override;
	HRESULT __stdcall IsDirty(void) override;
	HRESULT __stdcall Load(LPCOLESTR pszFileName, DWORD dwMode) override;
	HRESULT __stdcall Save(LPCOLESTR pszFileName, BOOL fRemember) override;
	HRESULT __stdcall SaveCompleted(LPCOLESTR pszFileName) override;
	HRESULT __stdcall GetCurFile(LPOLESTR* ppszFileName) override;

	// Inherited via IExtractIconW
	HRESULT __stdcall GetIconLocation(UINT uFlags, PWSTR pszIconFile, UINT cchMax, int* piIndex, UINT* pwFlags) override;
	HRESULT __stdcall Extract(PCWSTR pszFile, UINT nIconIndex, HICON* phiconLarge, HICON* phiconSmall, UINT nIconSize) override;
};

Now for the implementation. The IPersistFile interface seems non-trivial, but fortunately we just need to implement the Load method for an icon handler. This is where we get the file name we need to inspect. To check whether a DLL is 64 or 32 bit, we’ll add a simple enumeration and a helper function to the CIconHandler class:

	enum class ModuleBitness {
		Unknown,
		Bit32,
		Bit64
	};
	static ModuleBitness GetModuleBitness(PCWSTR path);

The implementation of IPersistFile::Load looks something like this:

HRESULT __stdcall CIconHandler::Load(LPCOLESTR pszFileName, DWORD dwMode) {
    ATLTRACE(L"CIconHandler::Load %s\n", pszFileName);

    m_Bitness = GetModuleBitness(pszFileName);
    return S_OK;
}

The method receives the full path of the DLL we need to examine. How do we know that only DLL files will be delivered? This has to do with the registration we’ll make for the icon handler. We’ll register it for DLL file extensions only, so that other file types will not be provided. Calling GetModuleBitness (shown later) performs the real work of determining the DLL’s bitness and stores the result in m_Bitness (a data member of type ModuleBitness).

All that’s left to do is tell explorer which icon to use. This is the role of IExtractIcon. The Extract method can be used to provide an icon handle directly, which is useful if the icon is “dynamic” – perhaps generated by different means in each case. In this example, we just need to return one of two icons which have been added as resources to the project (you can find those in the project source code. This is also an opportunity to provide your own icons).

For our case, it’s enough to return S_FALSE from Extract that causes explorer to use the information returned from GetIconLocation. Here is its implementation:

HRESULT __stdcall CIconHandler::GetIconLocation(UINT uFlags, PWSTR pszIconFile, UINT cchMax, int* piIndex, UINT* pwFlags) {
    if (s_ModulePath[0] == 0) {
        ::GetModuleFileName(_AtlBaseModule.GetModuleInstance(), 
            s_ModulePath, _countof(s_ModulePath));
        ATLTRACE(L"Module path: %s\n", s_ModulePath);
    }
    if (s_ModulePath[0] == 0)
        return S_FALSE;

    if (m_Bitness == ModuleBitness::Unknown)
        return S_FALSE;

    wcscpy_s(pszIconFile, wcslen(s_ModulePath) + 1, s_ModulePath);
    ATLTRACE(L"CIconHandler::GetIconLocation: %s bitness: %d\n", 
        pszIconFile, m_Bitness);
    *piIndex = m_Bitness == ModuleBitness::Bit32 ? 0 : 1;
    *pwFlags = GIL_PERINSTANCE;

    return S_OK;
}

The method’s purpose is to return the current (our icon handler DLL) module’s path and the icon index to use. This information is enough for explorer to load the icon itself from the resources. First, we get the module path to where our DLL has been installed. Since this doesn’t change, it’s only retrieved once (with GetModuleFileName) and stored in a static variable (s_ModulePath).

If this fails (unlikely) or the bitness could not be determined (maybe the file was not a PE at all, but just had such an extension), then we return S_FALSE. This tells explorer to use the default icon for the file type (DLL). Otherwise, we store 0 or 1 in piIndex, based on the IDs of the icons (0 corresponds to the lower of the IDs).

Finally, we need to set a flag inside pwFlags to indicate to explorer that this icon extraction is required for every file (GIL_PERINSTANCE). Otherwise, explorer calls IExtractIcon just once for any DLL file, which is the opposite of what we want.

The final piece of the puzzle (in terms of code) is how to determine whether a PE is 64 or 32 bit. This is not the point of this post, as any custom algorithm can be used to provide different icons for different files of the same type. For completeness, here is the code with comments:

CIconHandler::ModuleBitness CIconHandler::GetModuleBitness(PCWSTR path) {
    auto bitness = ModuleBitness::Unknown;
    //
    // open the DLL as a data file
    //
    auto hFile = ::CreateFile(path, GENERIC_READ, FILE_SHARE_READ, 
        nullptr, OPEN_EXISTING, 0, nullptr);
    if (hFile == INVALID_HANDLE_VALUE)
        return bitness;

    //
    // create a memory mapped file to read the PE header
    //
    auto hMemMap = ::CreateFileMapping(hFile, nullptr, PAGE_READONLY, 0, 0, nullptr);
    ::CloseHandle(hFile);
    if (!hMemMap)
        return bitness;

    //
    // map the first page (where the header is located)
    //
    auto p = ::MapViewOfFile(hMemMap, FILE_MAP_READ, 0, 0, 1 << 12);
    if (p) {
        auto header = ::ImageNtHeader(p);
        if (header) {
            auto machine = header->FileHeader.Machine;
            bitness = header->Signature == IMAGE_NT_OPTIONAL_HDR64_MAGIC ||
                machine == IMAGE_FILE_MACHINE_AMD64 || machine == IMAGE_FILE_MACHINE_ARM64 ?
                ModuleBitness::Bit64 : ModuleBitness::Bit32;
        }
        ::UnmapViewOfFile(p);
    }
    ::CloseHandle(hMemMap);
    
    return bitness;
}

To make all this work, there is still one more concern: registration. Normal COM registration is necessary (so that the call to CoCreateInstance issued by explorer has a chance to succeed), but not enough. Another registration is needed to let explorer know that this icon handler exists, and is to be used for files with the extension “DLL”.

Fortunately, ATL provides a convenient mechanism to add Registry settings using a simple script-like configuration, which does not require any code. The added keys/values have been placed in DllIconHandler.rgs like so:

HKCR
{
	NoRemove DllFile
	{
		NoRemove ShellEx
		{
			IconHandler = s '{d913f592-08f1-418a-9428-cc33db97ed60}'
		}
	}
}

This sets an icon handler in HKEY_CLASSES_ROOT\DllFile\ShellEx, where the IconHandler value specifies the CLSID of our component. You can find the CLSID in the IDL file where the coclass element is defined:

[
	uuid(d913f592-08f1-418a-9428-cc33db97ed60)
]
coclass IconHandler {

Replace your own CLSID if you’re building this project from scratch. Registration itself is done with the RegSvr32 built-in tool. With an ATL project, a successful build also causes RegSvr32 to be invoked on the resulting DLL, thus performing registration. The default behavior is to register in HKEY_CLASSES_ROOT which uses HKEY_LOCAL_MACHINE behind the covers. This requires running Visual Studio elevated (or an elevated command window if called from outside VS). It will register the icon handler for all users on the machine. If you prefer to register for the current user only (which uses HKEY_CURRENT_USER and does not require running elevated), you can set the per-user registration in VS by going to project properties, clinking on the Linker element and setting per-user redirection:

If you’re registering from outside VS, the per-user registration is achieved with:

regsvr32 /n /i:user <dllpath>

This is it! The full source code is available here.

image-2

zodiacon

Planned Upcoming Classes

Some people asked me if I had a schedule of trainings I plan to do in the coming months. Well, here it is. At this point I would like to gauge interest, and plan the course hours (time zone) to accommodate the majority of participants.

I am moving to classes that are partly full days and partly half days. The full day sessions are going to be recorded for the participants (so that if anyone misses something because of urgent work, inconvenient time zone, etc., the recording should help). The half days are not recorded, and should be easier to handle, since they are only about 4 hours long.

Here is the planned course list with dates (f=full day, all others are half-days). The cost is in USD (paid by individual / paid by a company):

  • COM Programming with C++ (3 days): April 25, 26, 27, 28, May 2, 3 (Cost: 700/1300)
  • Windows System Programming (5 days): May 16 (f), 17, 18, 19, 23 (f), 24, 25, 26 (Cost: 800/1500)
  • Windows Kernel Programming (4 days): June 6 (f), 8, 9, 13 (f), 14 (Cost: 800/1500)
  • Windows Internals (5 days): July 11 (f), 12, 13, 14, 18 (f), 19, 20, 21 (Cost: 800/1500)
  • Advanced Kernel Programming (New!) (4 days): September 12 (f), 13, 14, 15, 19, 20, 21 (Cost: 800/1500)

“Advanced Kernel Programming” is a new class I’m planning, suitable for those who participated in “Windows Kernel Programming” (or have equivalent knowledge). This course will cover file system mini-filters, NDIS filters, and the Windows Filtering Platform (WFP), along with other advanced programming techniques.

I may add more classes after September, but it’s too far from now to make such a commitment.

If you are interested in one or more of these classes, please write an email to [email protected], and provide your name, preferred contact email, and your time zone. It’s not a commitment on your part, you may change your mind later on, but it should be genuine, where the dates and topics work for you.

Also, if you have other classes you would be happy if I deliver, you are welcome to suggest them. No promises, of course, but if there is enough interest, I will consider creating and delivering them.

If you’d like a private class for your team, get in touch. Syllabi can be customized as needed.

Have a great year ahead!

Rings

zodiacon

Next COM Programming Class

Update: the class is cancelled. I guess there weren’t that many people interested in COM this time around.

Today I’m opening registration for the COM Programming class to be held in April. The syllabus for the 3 day class can be found here. The course will be delivered in 6 half-days (4 hours each).

Dates: April (25, 26, 27, 28), May (2, 3).
Times: 2pm to 6pm, London time
Cost: 700 USD (if paid by an individual), 1300 USD (if paid by a company).

The class will be conducted remotely using Microsoft Teams or a similar platform.

What you need to know before the class: You should be comfortable using Windows on a Power User level. Concepts such as processes, threads, DLLs, and virtual memory should be understood fairly well. You should have experience writing code in C and some C++. You don’t have to be an expert, but you must know C and basic C++ to get the most out of this class. In case you have doubts, talk to me.

Participants in my Windows Internals and Windows System Programming classes have the required knowledge for the class.

We’ll start by looking at why COM was created in the first place, and then build clients and servers, digging into various mechanisms COM provides. See the syllabus for more details.

Previous students in my classes get 10% off. Multiple participants from the same company get a discount (email me for the details).

To register, send an email to [email protected] with the title “COM Training”, and write the name(s), email(s) and time zone(s) of the participants.

COMReuse

zodiacon

Registration is open for the Windows Internals training

My schedule has been a mess in recent months, and continues to be so for the next few months. However, I am opening registration today for the Windows Internals training with some date changes from my initial plan.

Here are the dates and times (all based on London time) – 5 days total:

  • July 6: 4pm to 12am (full day)
  • July 7: 4pm to 8pm
  • July 11: 4pm to 12am (full day)
  • July 12, 13, 14, 18, 19: 4pm to 8pm

Training cost is 800 USD, if paid by an individual, or 1500 USD if paid by a company. Participants from Ukraine (please provide some proof) are welcome with a 90% discount (paying 80 USD, individual payments only).

If you’d like to register, please send me an email to [email protected] with “Windows Internals training” in the title, provide your full name, company (if any), preferred contact email, and your time zone. The basic syllabus can be found here. if you’ve sent me an email before when I posted about my upcoming classes, you don’t have to do that again – I will send full details soon.

The sessions will be recorded, so can watch any part you may be missing, or that may be somewhat overwhelming in “real time”.

As usual, if you have any questions, feel free to send me an email, or DM me on twitter (@zodiacon) or Linkedin (https://www.linkedin.com/in/pavely/).

Kernel2

zodiacon

Threads, Threads, and More Threads

Looking at a typical Windows system shows thousands of threads, with process numbers in the hundreds, even though the total CPU consumption is low, meaning most of these threads are doing nothing most of the time. I typically rant about it in my Windows Internals classes. Why so many threads?

Here is a snapshot of my Task Manager showing the total number of threads and processes:

Showing processes details and sorting by thread count looks something like this:

The System process clearly has many threads. These are kernel threads created by the kernel itself and by device drivers. These threads are always running in kernel mode. For this post, I’ll disregard the System process and focus on “normal” user-mode processes.

There are other kernel processes that we should ignore, such as Registry and Memory Compression. Registry has few threads, but Memory Compression has many. It’s not shown in Task Manager (by design), but is shown in other tools, such as Process Explorer. While I’m writing this post, it has 78 threads. We should probably skip that process as well as being “out of our control”.

Notice the large number of threads in processes running the images Explorer.exe, SearchIndexer.exe, Nvidia Web helper.exe, Outlook.exe, Powerpnt.exe and MsMpEng.exe. Let’s write some code to calculate the average number of threads in a process and the standard deviation:

float ComputeStdDev(std::vector<int> const& values, float& average) {
	float total = 0;
	std::for_each(values.begin(), values.end(), 
		[&](int n) { total += n; });
	average = total / values.size();
	total = 0;
	std::for_each(values.begin(), values.end(), 
		[&](int n) { total += (n - average) * (n - average); });
	return std::sqrt(total / values.size());
}

int main() {
	auto hSnapshot = ::CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
	
	PROCESSENTRY32 pe;
	pe.dwSize = sizeof(pe);

	// skip the idle process
	::Process32First(hSnapshot, &pe);

	int processes = 0, threads = 0;
	std::vector<int> threads_per_process;
	threads_per_process.reserve(500);
	while (::Process32Next(hSnapshot, &pe)) {
		processes++;
		threads += pe.cntThreads;
		threads_per_process.push_back(pe.cntThreads);
	}
	::CloseHandle(hSnapshot);

	assert(processes == threads_per_process.size());

	printf("Process: %d Threads: %d\n", processes, threads);
	float average;
	auto sd = ComputeStdDev(threads_per_process, average);
	printf("Average threads/process: %.2f\n", average);
	printf("Std. Dev.: %.2f\n", sd);

	return 0;
}

The ComputeStdDev function computes the standard deviation and average of a vector of integers. The main function uses the ToolHelp API to enumerate processes in the system, which fortunately also provides the number of threads in each processes (stored in the threads_per_process vector. If I run this (no processes removed just yet), this is what I get:

Process: 525 Threads: 7810
Average threads/process: 14.88
Std. Dev.: 23.38

Almost 15 threads per process, with little CPU consumption in my Task Manager. The standard deviation is more telling – it’s big compared to the average, which suggests that many processes are far from the average in their thread consumption. And since a negative thread count is not possible (even zero is almost impossible), the the divergence is with higher thread numbers.

To be fair, let’s remove the System and Memory Compression processes from our calculations. Here are the changes to the while loop:

while (::Process32Next(hSnapshot, &pe)) {
	if (pe.th32ProcessID == 4 || _wcsicmp(pe.szExeFile, L"memory compression") == 0)
		continue;
//...

Here are the results:

Process: 521 Threads: 7412
Average threads/process: 14.23
Std. Dev.: 14.14

The standard deviation is definitely smaller, but still pretty big (close to the average), which does not invalidate the previous point. Some processes use lots of threads.

In an ideal world, the number of threads in a system would be the same as the number of logical processors – any more and threads might fight over processors, any less and you’re not using the full power of the machine. Obviously, each “normal” process must have at least one thread running whatever main function is available in the executable, so on my system 521 threads would be the minimum number of threads. Still – we have over 7000.

What are these threads doing, anyway? Let’s examine some processes. First, an Explorer.exe process. Here is the Threads tab shown in Process Explorer:

Thread list in Explorer.exe instance

93 threads. I’ve sorted the list by Start Address to get a sense of the common functions used. Let’s dig into some of them. One of the most common (in other processes as well) is ntdll!TppWorkerThread – this is a thread pool thread, likely waiting for work. Clicking the Stack button (or double clicking the entry in the list) shows the following call stack:

ntoskrnl.exe!KiSwapContext+0x76
ntoskrnl.exe!KiSwapThread+0x500
ntoskrnl.exe!KiCommitThreadWait+0x14f
ntoskrnl.exe!KeWaitForSingleObject+0x233
ntoskrnl.exe!KiSchedulerApc+0x3bd
ntoskrnl.exe!KiDeliverApc+0x2e9
ntoskrnl.exe!KiSwapThread+0x827
ntoskrnl.exe!KiCommitThreadWait+0x14f
ntoskrnl.exe!KeRemoveQueueEx+0x263
ntoskrnl.exe!IoRemoveIoCompletion+0x98
ntoskrnl.exe!NtWaitForWorkViaWorkerFactory+0x38e
ntoskrnl.exe!KiSystemServiceCopyEnd+0x25
ntdll.dll!ZwWaitForWorkViaWorkerFactory+0x14
ntdll.dll!TppWorkerThread+0x2f7
KERNEL32.DLL!BaseThreadInitThunk+0x14
ntdll.dll!RtlUserThreadStart+0x21

The system call NtWaitForWorkViaWorkerFactory is the one waiting for work (the name Worker Factory is the internal name of the thread pool type in the kernel, officially called TpWorkerFactory). The number of such threads is typically dynamic, growing and shrinking based on the amount of work provided to the thread pool(s). The minimum and maximum threads can be tweaked by APIs, but most processes are unlikely to do so.

Another function that appears a lot in the list is shcore.dll!_WrapperThreadProc. It looks like some generic function used by Explorer for its own threads. We can examine some call stacks to get a sense of what’s going on. Here is one:

ntoskrnl.exe!KiSwapContext+0x76
ntoskrnl.exe!KiSwapThread+0x500
ntoskrnl.exe!KiCommitThreadWait+0x14f
ntoskrnl.exe!KeWaitForSingleObject+0x233
ntoskrnl.exe!KiSchedulerApc+0x3bd
ntoskrnl.exe!KiDeliverApc+0x2e9
ntoskrnl.exe!KiSwapThread+0x827
ntoskrnl.exe!KiCommitThreadWait+0x14f
ntoskrnl.exe!KeWaitForSingleObject+0x233
ntoskrnl.exe!KeWaitForMultipleObjects+0x45b
win32kfull.sys!xxxRealSleepThread+0x362
win32kfull.sys!xxxSleepThread2+0xb5
win32kfull.sys!xxxRealInternalGetMessage+0xcfd
win32kfull.sys!NtUserGetMessage+0x92
win32k.sys!NtUserGetMessage+0x16
ntoskrnl.exe!KiSystemServiceCopyEnd+0x25
win32u.dll!NtUserGetMessage+0x14
USER32.dll!GetMessageW+0x2e
SHELL32.dll!_LocalServerThread+0x66
shcore.dll!_WrapperThreadProc+0xe9
KERNEL32.DLL!BaseThreadInitThunk+0x14
ntdll.dll!RtlUserThreadStart+0x21

This one seems to be waiting for UI messages, probably managing some user interface (GetMessage). We can verify with other tools. Here is my own WinSpy:

Apparently, I was wrong. This thread has the hidden window type used to receive messages targeting COM objects that leave in this Single Threaded Apartment (STA).

We can inspect WinSpy some more to see the threads and windows created by Explorer. I’ll leave that to the interested reader.

Other generic call stacks start with ucrtbase.dll!thread_start+0x42. Many of them have the following call stack (kernel part trimmed for brevity):

ntdll.dll!ZwWaitForMultipleObjects+0x14
KERNELBASE.dll!WaitForMultipleObjectsEx+0xf0
KERNELBASE.dll!WaitForMultipleObjects+0xe
cdp.dll!shared::CallbackNotifierListener::ListenerInternal::StartInternal+0x9f
cdp.dll!std::thread::_Invoke<std::tuple<<lambda_10793e1829a048bb2f8cc95974633b56> >,0>+0x2f
ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>+0x42
KERNEL32.DLL!BaseThreadInitThunk+0x14
ntdll.dll!RtlUserThreadStart+0x21

A function in CDP.dll is waiting for something (WaitForMultipleObjects). I count at least 12 threads doing just that. Perhaps all these waits could be consolidated to a smaller number of threads?

Let’s tackle a different process. Here is an instance of Teams.exe. My teams is minimized to the tray and I have not interacted with it for a while:

Teams threads

62 threads. Many have the same CRT wrapper for a thread created by Teams. Here are several call stacks I observed:

ntdll.dll!ZwRemoveIoCompletion+0x14
KERNELBASE.dll!GetQueuedCompletionStatus+0x4f
skypert.dll!rtnet::internal::SingleThreadIOCP::iocpLoop+0x116
skypert.dll!SplOpaqueUpperLayerThread::run+0x84
skypert.dll!auf::priv::MRMWTransport::process1+0x6c
skypert.dll!auf::ThreadPoolExecutorImp::workLoop+0x160
skypert.dll!auf::tpImpThreadTrampoline+0x47
skypert.dll!spl::threadWinDispatch+0x19
skypert.dll!spl::threadWinEntry+0x17b
ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>+0x42
KERNEL32.DLL!BaseThreadInitThunk+0x14
ntdll.dll!RtlUserThreadStart+0x21
ntdll.dll!ZwWaitForAlertByThreadId+0x14
ntdll.dll!RtlSleepConditionVariableCS+0x105
KERNELBASE.dll!SleepConditionVariableCS+0x29
Teams.exe!uv_cond_wait+0x10
Teams.exe!worker+0x8d
Teams.exe!uv__thread_start+0xa2
Teams.exe!thread_start<unsigned int (__cdecl*)(void *),1>+0x50
KERNEL32.DLL!BaseThreadInitThunk+0x14
ntdll.dll!RtlUserThreadStart+0x21

You can check more threads, but you get the idea. Most threads are waiting for something – this is not the ideal activity for a thread. A thread should run (useful) code.

Last example, Word:

57 threads. Word has been minimized for more than an hour now. The clearly common call stack looks like this:

ntdll.dll!ZwWaitForAlertByThreadId+0x14
ntdll.dll!RtlSleepConditionVariableSRW+0x131
KERNELBASE.dll!SleepConditionVariableSRW+0x29
v8jsi.dll!CrashForExceptionInNonABICompliantCodeRange+0x4092f6
v8jsi.dll!CrashForExceptionInNonABICompliantCodeRange+0x11ff2
v8jsi.dll!v8_inspector::V8StackTrace::topScriptIdAsInteger+0x43ad0
ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>+0x42
KERNEL32.DLL!BaseThreadInitThunk+0x14
ntdll.dll!RtlUserThreadStart+0x21

v8jsi.dll is the React Native v8 engine – it’s creating many threads, most of which are doing nothing. I found it in Outlook and PowerPoint as well.

Many applications today depend on various libraries and frameworks, some of which don’t seem to care too much about using threads economically – examples include Node.js, the Electron framework, even Java and .NET. Threads are not free – there is the ETHREAD and related data structures in the kernel, stack in kernel space, and stack in user space. Context switches and code run by the kernel scheduler when threads change states from Running to Waiting, and from Waiting to Ready are not free, either.

Many desktop/laptop systems today are very powerful and it might seem everything is fine. I don’t think so. Developers use so many layers of abstraction these days, that we sometimes forget there are actual processors that execute the code, and need to use memory and other resources. None of that is free.

image-1

zodiacon

Mysteries of the Registry

The Windows Registry is one of the most recognized aspects of Windows. It’s a hierarchical database, storing information on a machine-wide basis and on a per-user basis… mostly. In this post, I’d like to examine the major parts of the Registry, including the “real” Registry.

Looking at the Registry is typically done by launching the built-in RegEdit.exe tool, which shows the five “hives” that seem to comprise the Registry:

RegEdit showing the main hives

These so-called “hives” provide some abstracted view of the information in the Registry. I’m saying “abstracted”, because not all of these are true hives. A true hive is stored in a file. The full hive list can be found in the Registry itself – at HKLM\SYSTEM\CurrentControlSet\Control\hivelist (I’ll abbreviate HKEY_LOCAL_MACHINE as HKLM), mapping an internal key name to the file where it’s stored (more on these “internal” key names will be discussed soon):

The hive list

Let’s examine the so-called “hives” as seen in the root RegEdit’s view.

  • HKEY_LOCAL_MACHINE is the simplest to understand. It contains machine-wide information, most of it stored in files (persistent). Some details related to hardware is built when the system initializes and is only kept in memory while the system is running. Such keys are volatile, since their contents disappear when the system is shut down.
    There are many interesting keys within HKLM, but my goal is not to go over every key (that would take a full book), but highlight a few useful pieces. HKLM\System\CurrentControlSet\Services is the key where all services and device drivers are installed. Note that “CurrentControlSet” is not a true key, but in fact is a link key, connecting it to something like HKLM\System\ControlSet001. The reason for this indirection is beyond the scope of this post. Regedit does not show this fact directly – there is no way to tell whether a key is a true key or just points to a different key. This is one reason I created Total Registry (formerly called Registry Explorer), that shows these kind of nuances:
TotalRegistry showing HKLM\System\CurrentControlSet

The liked key seems to have a weird name starting with \REGISTRY\MACHINE\. We’ll get to that shortly.

Other subkeys of note under HKLM include SOFTWARE, where installed applications store their system-level information; SAM and SECURITY, where local security policy and local accounts information are managed. These two subkeys contents is not not visible – even administrators don’t get access – only the SYSTEM account is granted access. One way to see what’s in these keys is to use psexec from Sysinternals to launch RegEdit or TotalRegistry under the SYSTEM account. Here is a command you can run in an elevated command window that will launch RegEdit under the SYSTEM account (if you’re using RegEdit, close it first):

psexec -s -i -d RegEdit

The -s switch indicates the SYSTEM account. -i is critical as to run the process in the interactive session (the default would run it in session 0, where no interactive user will ever see it). The -d switch is optional, and simply returns control to the console while the process is running, rather than waiting for the process to terminate.

The other way to gain access to the SAM and SECURITY subkeys is to use the “Take Ownership” privilege (easy to do when the Permissions dialog is open), and transfer the ownership to an admin user – the owner can specify who can do what with an object, and allow itself full access. Obviously, this is not a good idea in general, as it weakens security.

The BCD00000000 subkey contains the Boot Configuration Data (BCD), normally accessed using the bcdedit.exe tool.

  • HKEY_USERS – this is the other hive that truly stores data. Its subkeys contain user profiles for all users that ever logged in locally to this machine. Each subkey’s name is a Security ID (SID), in its string representation:
HKEY_USERS

There are 3 well-known SIDs, representing the SYSTEM (S-1-5-18), LocalService (S-1-5-19), and NetworkService (S-1-5-20) accounts. These are the typical accounts used for running Windows Services. “Normal” users get ugly SIDs, such as the one shown – that’s my user’s local SID. You may be wondering what is that “_Classes” suffix in the second key. We’ll get to that as well.

  • HKEY_CURRENT_USER is a link key, pointing to the user’s subkey under HKEY_USERS running the current process. Obviously, the meaning of “current user” changes based on the process access token looking at the Registry.
  • HKEY_CLASSES_ROOT is the most curious of the keys. It’s not a “real” key in the sense that it’s not a hive – not stored in a file. It’s not a link key, either. This key is a “combination” of two keys: HKLM\Software\Classes and HKCU\Software\Classes. In other words, the information in HKEY_CLASSES_ROOT is coming from the machine hive first, but can be overridden by the current user’s hive.
    What information is there anyway? The first thing is shell-related information, such as file extensions and associations, and all other information normally used by Explorer.exe. The second thing is information related to the Component Object Model (COM). For example, the CLSID subkey holds COM class registration (GUIDs you can pass to CoCreateInstance to (potentially) create a COM object of that class). Looking at the CLSID subkey under HKLM\Software\Classes shows there are 8160 subkeys, or roughly 8160 COM classes registered on my system from HKLM:
HKLM\Software\Classes

Looking at the same key under HKEY_CURRENT_USER tells a different story:

HKCU\Software\Classes

Only 46 COM classes provide extra or overridden registrations. HKEY_CLASSES_ROOT combines both, and uses HKCU in case of a conflict (same key name). This explains the extra “_Classes” subkey within the HKEY_USERS key – it stores the per user stuff (in the file UsrClasses.dat in something like c:\Users\<username>\AppData\Local\Microsoft\Windows).

  • HKEY_CURRENT_CONFIG is a link to HKLM\SYSTEM\CurrentControlSet\Hardware\Profiles\Current

    The list of “standard” hives (the hives accessible by official Windows APIs such as RegOpenKeyEx contains some more that are not shown by Regedit. They can be viewed by TotalReg if the option “Extra Hives” is selected in the View menu. At this time, however, the tool needs to be restarted for this change to take effect (I just didn’t get around to implementing the change dynamically, as it was low on my priority list). Here are all the hives accessible with the official Windows API:
All hives

I’ll let the interested reader to dig further into these “extra” hives. On of these hives deserves special mentioning – HKEY_PERFORMANCE_DATA – it was used in the pre Windows 2000 days as a way to access Performance Counters. Registry APIs had to be used at the time. Fortunately, starting from Windows 2000, a new dedicated API is provided to access Performance Counters (functions starting with Pdh* in <pdh.h>).

Is this it? Is this the entire Registry? Not quite. As you can see in TotalReg, there is a node called “Registry”, that tells yet another story. Internally, all Registry keys are rooted in a single key called REGISTRY. This is the only named Registry key. You can see it in the root of the Object Manager’s namespace with WinObj from Sysinternals:

WinObj from Sysinternals showing the Registry key object

Here is the object details in a Local Kernel debugger:

lkd> !object \registry
Object: ffffe00c8564c860  Type: (ffff898a519922a0) Key
    ObjectHeader: ffffe00c8564c830 (new version)
    HandleCount: 1  PointerCount: 32770
    Directory Object: 00000000  Name: \REGISTRY
lkd> !trueref ffffe00c8564c860
ffffe00c8564c860: HandleCount: 1 PointerCount: 32770 RealPointerCount: 3

All other Registry keys are based off of that root key, the Configuration Manager (the kernel component in charge of the Registry) parses the remaining path as expected. This is the real Registry. The official Windows APIs cannot use this path format, but native APIs can. For example, using NtOpenKey (documented as ZwOpenKey in the Windows Driver Kit, as this is a system call) allows such access. This is how TotalReg is able to look at the real Registry.

Clearly, the normal user-mode APIs somehow map the “standard” hive path to the real Registry path. The simplest is the mapping of HKEY_LOCAL_MACHINE to \REGISTRY\MACHINE. Another simple one is HKEY_USERS mapped to \REGISTRY\USER. HKEY_CURRENT_USER is a bit more complex, and needs to be mapped to the per-user hive under \REGISTRY\USER. The most complex is our friend HKEY_CLASSES_ROOT – there is no simple mapping – the APIs have to check if there is per-user override or not, etc.

Lastly, it seems there are keys in the real Registry that cannot be reached from the standard Registry at all:

The real Registry

There is a key named “A” which seems inaccessible. This key is used for private keys in processes, very common in Universal Windows Application (UWP) processes, but can be used in other processes as well. They are not accessible generally, not even with kernel code – the Configuration Manager prevents it. You can verify their existence by searching for \Registry\A in tools like Process Explorer or TotalReg itself (by choosing Scan Key Handles from the Tools menu). Here is TotalReg, followed by Process Explorer:

TotalReg key handles
Process Explorer key handles

Finally, the WC key is used for Windows Container, internally called Silos. A container (like the ones created by Docker) is an isolated instance of a user-mode OS, kind of like a lightweight virtual machine, but the kernel is not separate (as would be with a true VM), but is provided by the host. Silos are very interesting, but outside the scope of this post.

Briefly, there are two main Silo types: An Application Silo, which is not a true container, and mostly used with application based on the Desktop Bridge technology. A classic example is WinDbg Preview. The second type is Server Silo, which is a true container. A true container must have its file system, Registry, and Object Manager namespace virtualized. This is exactly the role of the WC subkeys – provide the private Registry keys for containers. The Configuration Manager (as well as other parts of the kernel) are Silo-aware, and will redirect Registry calls to the correct subkey, having no effect on the Host Registry or the private Registry of other Silos.

You can examine some aspects of silos with the kernel debugger !silo command. Here is an example from a server 2022 running a Server Silo and the Registry keys under WC:

lkd> !silo
		Address          Type       ProcessCount Identifier
		ffff800f2986c2e0 ServerSilo 15           {1d29488c-bccd-11ec-a503-d127529101e4} (0n732)
1 active Silo(s)
lkd> !silo ffff800f2986c2e0

Silo ffff800f2986c2e0:
		Job               : ffff800f2986c2e0
		Type              : ServerSilo
		Identifier        : {1d29488c-bccd-11ec-a503-d127529101e4} (0n732)
		Processes         : 15

Server silo globals ffff800f27e65a40:
		Default Error Port: ffff800f234ee080
		ServiceSessionId  : 217
		Root Directory    : 00007ffcad26b3e1 '\Silos\732'
		State             : Running
A Server Silo’s keys

There you have it. The relatively simple-looking Registry shown in RegEdit is viewed differently by the kernel. Device driver writers find this out relatively early – they cannot use the “abstractions” provided by user mode even if these are sometimes convenient.


image-1

zodiacon

Zombie Processes

The term “Zombie Process” in Windows is not an official one, as far as I know. Regardless, I’ll define zombie process to be a process that has exited (for whatever reason), but at least one reference remains to the kernel process object (EPROCESS), so that the process object cannot be destroyed.

How can we recognize zombie processes? Is this even important? Let’s find out.

All kernel objects are reference counted. The reference count includes the handle count (the number of open handles to the object), and a “pointer count”, the number of kernel clients to the object that have incremented its reference count explicitly so the object is not destroyed prematurely if all handles to it are closed.

Process objects are managed within the kernel by the EPROCESS (undocumented) structure, that contains or points to everything about the process – its handle table, image name, access token, job (if any), threads, address space, etc. When a process is done executing, some aspects of the process get destroyed immediately. For example, all handles in its handle table are closed; its address space is destroyed. General properties of the process remain, however, some of which only have true meaning once a process dies, such as its exit code.

Process enumeration tools such as Task Manager or Process Explorer don’t show zombie processes, simply because the process enumeration APIs (EnumProcesses, Process32First/Process32Next, the native NtQuerySystemInformation, and WTSEnumerateProcesses) don’t return these – they only return processes that can still run code. The kernel debugger, on the other hand, shows all processes, zombie or not when you type something like !process 0 0. Identifying zombie processes is easy – their handle table and handle count is shown as zero. Here is one example:

kd> !process ffffc986a505a080 0
PROCESS ffffc986a505a080
    SessionId: 1  Cid: 1010    Peb: 37648ff000  ParentCid: 0588
    DirBase: 16484cd000  ObjectTable: 00000000  HandleCount:   0.
    Image: smartscreen.exe

Any kernel object referenced by the process object remains alive as well – such as a job (if the process is part of a job), and the process primary token (access token object). We can get more details about the process by passing the detail level “1” in the !process command:

lkd> !process ffffc986a505a080 1
PROCESS ffffc986a505a080
    SessionId: 1  Cid: 1010    Peb: 37648ff000  ParentCid: 0588
    DirBase: 16495cd000  ObjectTable: 00000000  HandleCount:   0.
    Image: smartscreen.exe
    VadRoot 0000000000000000 Vads 0 Clone 0 Private 16. Modified 7. Locked 0.
    DeviceMap ffffa2013f24aea0
    Token                             ffffa20147ded060
    ElapsedTime                       1 Day 15:11:50.174
    UserTime                          00:00:00.000
    KernelTime                        00:00:00.015
    QuotaPoolUsage[PagedPool]         0
    QuotaPoolUsage[NonPagedPool]      0
    Working Set Sizes (now,min,max)  (17, 50, 345) (68KB, 200KB, 1380KB)
    PeakWorkingSetSize                2325
    VirtualSize                       0 Mb
    PeakVirtualSize                   2101341 Mb
    PageFaultCount                    2500
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      20
    Job                               ffffc98672eea060

Notice the address space does not exist anymore (VadRoot is zero). The VAD (Virtual Address Descriptors) is a data structure managed as a balanced binary search tree that describes the address space of a process – which parts are committed, which parts are reserved, etc. No address space exists anymore. Other details of the process are still there as they are direct members of the EPROCESS structure, such as the kernel and user time the process has used, its start and exit times (not shown in the debugger’s output above).

We can ask the debugger to show the reference count of any kernel object by using the generic !object command, to be followed by !trueref if there are handles open to the object:

lkd> !object ffffc986a505a080
Object: ffffc986a505a080  Type: (ffffc986478ce380) Process
    ObjectHeader: ffffc986a505a050 (new version)
    HandleCount: 1  PointerCount: 32768
lkd> !trueref ffffc986a505a080
ffffc986a505a080: HandleCount: 1 PointerCount: 32768 RealPointerCount: 1

Clearly, there is a single handle open to the process and that’s the only thing keeping it alive.

One other thing that remains is the unique process ID (shown as Cid in the above output). Process and thread IDs are generated by using a private handle table just for this purpose. This explains why process and thread IDs are always multiples of four, just like handles. In fact, the kernel treats PIDs and TIDs with the HANDLE type, rather with something like ULONG. Since there is a limit to the number of handles in a process (16711680, the reason is not described here), that’s also the limit for the number of process and threads that could exist on a system. This is a rather large number, so probably not an issue from a practical perspective, but zombie processes still keep their PIDs “taken”, so it cannot be reused. This means that in theory, some code can create millions of processes, terminate them all, but not close the handles it receives back, and eventually new processes could not be created anymore because PIDs (and TIDs) run out. I don’t know what would happen then 🙂

Here is a simple loop to do something like that by creating and destroying Notepad processes but keeping handles open:

WCHAR name[] = L"notepad";
STARTUPINFO si{ sizeof(si) };
PROCESS_INFORMATION pi;
int i = 0;
for (; i < 1000000; i++) {	// use 1 million as an example
	auto created = ::CreateProcess(nullptr, name, nullptr, nullptr,
        FALSE, 0, nullptr, nullptr, &si, &pi);
	if (!created)
		break;
	::TerminateProcess(pi.hProcess, 100);
	printf("Index: %6d PID: %u\n", i + 1, pi.dwProcessId);
	::CloseHandle(pi.hThread);
}
printf("Total: %d\n", i);

The code closes the handle to the first thread in the process, as keeping it alive would create “Zombie Threads”, much like zombie processes – threads that can no longer run any code, but still exist because at least one handle is keeping them alive.

How can we get a list of zombie processes on a system given that the “normal” tools for process enumeration don’t show them? One way of doing this is to enumerate all the process handles in the system, and check if the process pointed by that handle is truly alive by calling WaitForSingleObject on the handle (of course the handle must first be duplicated into our process so it’s valid to use) with a timeout of zero – we don’t want to wait really. If the result is WAIT_OBJECT_0, this means the process object is signaled, meaning it exited – it’s no longer capable of running any code. I have incorporated that into my Object Explorer (ObjExp.exe) tool. Here is the basic code to get details for zombie processes (the code for enumerating handles is not shown but is available in the source code):

m_Items.clear();
m_Items.reserve(128);
std::unordered_map<DWORD, size_t> processes;
for (auto const& h : ObjectManager::EnumHandles2(L"Process")) {
	auto hDup = ObjectManager::DupHandle(
        (HANDLE)(ULONG_PTR)h->HandleValue , h->ProcessId, 
        SYNCHRONIZE | PROCESS_QUERY_LIMITED_INFORMATION);
	if (hDup && WAIT_OBJECT_0 == ::WaitForSingleObject(hDup, 0)) {
		//
		// zombie process
		//
		auto pid = ::GetProcessId(hDup);
		if (pid) {
			auto it = processes.find(pid);
			ZombieProcess zp;
			auto& z = it == processes.end() ? zp : m_Items[it->second];
			z.Pid = pid;
			z.Handles.push_back({ h->HandleValue, h->ProcessId });
			WCHAR name[MAX_PATH];
			if (::GetProcessImageFileName(hDup, 
                name, _countof(name))) {
				z.FullPath = 
                    ProcessHelper::GetDosNameFromNtName(name);
				z.Name = wcsrchr(name, L'\\') + 1;
			}
			::GetProcessTimes(hDup, 
                (PFILETIME)&z.CreateTime, (PFILETIME)&z.ExitTime, 
                (PFILETIME)&z.KernelTime, (PFILETIME)&z.UserTime);
			::GetExitCodeProcess(hDup, &z.ExitCode);
			if (it == processes.end()) {
				m_Items.push_back(std::move(z));
				processes.insert({ pid, m_Items.size() - 1 });
			}
		}
	}
	if (hDup)
		::CloseHandle(hDup);
}

The data structure built for each process and stored in the m_Items vector is the following:

struct HandleEntry {
	ULONG Handle;
	DWORD Pid;
};
struct ZombieProcess {
	DWORD Pid;
	DWORD ExitCode{ 0 };
	std::wstring Name, FullPath;
	std::vector<HandleEntry> Handles;
	DWORD64 CreateTime, ExitTime, KernelTime, UserTime;
};

The ObjectManager::DupHandle function is not shown, but it basically calls DuplicateHandle for the process handle identified in some process. if that works, and the returned PID is non-zero, we can go do the work. Getting the process image name is done with GetProcessImageFileName – seems simple enough, but this function gets the NT name format of the executable (something like \Device\harddiskVolume3\Windows\System32\Notepad.exe), which is good enough if only the “short” final image name component is desired. if the full image path is needed in Win32 format (e.g. “c:\Windows\System32\notepad.exe”), it must be converted (ProcessHelper::GetDosNameFromNtName). You might be thinking that it would be far simpler to call QueryFullProcessImageName and get the Win32 name directly – but this does not work, and the function fails. Internally, the NtQueryInformationProcess native API is called with ProcessImageFileNameWin32 in the latter case, which fails if the process is a zombie one.

Running Object Explorer and selecting Zombie Processes from the System menu shows a list of all zombie processes (you should run it elevated for best results):

Object Explorer showing zombie processes

The above screenshot shows that many of the zombie processes are kept alive by GameManagerService.exe. This executable is from Razer running on my system. It definitely has a bug that keeps process handle alive way longer than needed. I’m not sure it would ever close these handles. Terminating this process will resolve the issue as the kernel closes all handles in a process handle table once the process terminates. This will allow all those processes that are held by that single handle to be freed from memory.

I plan to add Zombie Threads to Object Explorer – I wonder how many threads are being kept “alive” without good reason.

image

zodiacon

Next Windows Kernel Programming Class

I’m happy to announce the next 5-day virtual Windows Kernel Programming class to be held in October. The syllabus for the class can be found here. A notable addition to the class is an introduction to the Kernel Mode Driver Framework (KMDF).

Dates and Times (all in October 2022), times based on London:
11 (full day): 4pm to 12am
12 (full day): 4pm to 12am
13 (half day): 4pm to 8pm
17 (half day): 4pm to 8pm
18 (full day): 4pm to 12am
19 (half day): 4pm to 8pm
20 (half day): 4pm to 8pm

The class will be recorded and provided to the participants.

Cost:
900 USD if paid by an individual
1700 USD if paid by a company
Previous participants of my classes get 10% off. Multiple participants from the same company get a discount as well (talk to me).

Registration
To register, send email to [email protected] and provide the name(s) and email(s) of the participant(s), the company name (if any), and your time zone (for my information, although I cannot change course times).

Feel free to contact me for any questions or comments via email, twitter (@zodiacon) or Linkedin.

Introduction to Monikers

The foundations of the Component Object Model (COM) are made of two principles:

  1. Clients program against interfaces, never concrete classes.
  2. Location transparency – clients need not know where the actual object is (in-process, out-of-process, another machine).

Although simple in principle, there are many details involved in COM, as those with COM experience are well aware. In this post, I’d like to introduce one extensibility aspect of COM called Monikers.

The idea of a moniker is to provide some way to identify and locate specific objects based on string names instead of some custom mechanism. Windows provides some implementations of monikers, most of which are related to Object Linking and Embedding (OLE), most notably used in Microsoft Office applications. For example, when an Excel chart is embedded in a Word document as a link, an Item moniker is used to point to that specific chart using a string with a specific format understood by the moniker mechanism and the specific monikers involved. This also suggests that monikers can be combined, which is indeed the case. For example, a cell in some Excel document can be located by going to a specific sheet, then a specific range, then a specific cell – each one could be pointed to by a moniker, that when chained together can locate the required object.

Let’s start with perhaps the simplest example of an existing moniker implementation – the Class moniker. This moniker can be used to replace a creation operation. Here is an example that creates a COM object using the “standard” mechanism of calling CoCreateInstance:

#include <shlobjidl.h>
//...
CComPtr<IShellWindows> spShell;
auto hr = spShell.CoCreateInstance(__uuidof(ShellWindows));

I use the ATL smart pointers (#include <atlcomcli.h> or <atlbase.h>). The interface and class I’m using is just an example – any standard COM class would work. The CoCreateInstance method calls the real CoCreateInstance. To make it clearer, here is the CoCreateInstance call without using the helper provided by the smart pointer:

CComPtr<IShellWindows> spShell;
auto hr = ::CoCreateInstance(__uuidof(ShellWindows), nullptr, 
    CLSCTX_ALL, __uuidof(IShellWindows), 
    reinterpret_cast<void**>(&spShell));

CoCreateInstance itself is a glorified wrapper for calling CoGetClassObject to retrieve a class factory, requesting the standard IClassFactory interface, and then calling CreateInstance on it:

CComPtr<IClassFactory> spCF;
auto hr = ::CoGetClassObject(__uuidof(ShellWindows), 
    CLSCTX_ALL, nullptr, __uuidof(IClassFactory), 
    reinterpret_cast<void**>(&spCF));
if (SUCCEEDED(hr)) {
    CComPtr<IShellWindows> spShell;
    hr = spCF->CreateInstance(nullptr, __uuidof(IShellWindows),
        reinterpret_cast<void**>(&spShell));
    if (SUCCEEDED(hr)) {
        // use spShell
    }
}

Here is where the Class moniker comes in: It’s possible to get a class factory directly using a string like so:

CComPtr<IClassFactory> spCF;
BIND_OPTS opts{ sizeof(opts) };
auto hr = ::CoGetObject(
    L"clsid:9BA05972-F6A8-11CF-A442-00A0C90A8F39", 
    &opts, __uuidof(IClassFactory), 
    reinterpret_cast<void**>(&spCF));

Using CoGetObject is the most convenient way in C++ to locate an object based on a moniker. The moniker name is the string provided to CoGetObject. It starts with a ProgID of sorts followed by a colon. The rest of the string is to be interpreted by the moniker behind the scenes. With the class factory in hand, the code can use IClassFactory::CreateInstance just as with the previous example.

How does it work? As is usual with COM, the Registry is involved. If you open RegEdit or TotalRegistry and navigate to HKYE_CLASSES_ROOT, ProgIDs are all there. One of them is “clsid” – yes, it’s a bit weird perhaps, but the entry point to the moniker system is that ProgID. Each ProgID should have a CLSID subkey pointing to the class ID of the moniker. So here, the key is HKCR\CLSID\CLSID!

Class Moniker Registration

Of course, other monikers have different names (not CLSID). If we follow the CLSID on the right to the normal location for COM CLSID registration (HKCR\CLSID), this is what we find:

Class moniker

And the InProcServer32 subkey points to Combase.dll, the DLL implementing the COM infrastructure:

Class Moniker Implementation

At this point, we know how the class moniker got discovered, but it’s still not clear what is that moniker and where is it anyway?

As mentioned earlier, CoGetObject is the simplest way to get an object from a moniker, as it hides the details of the moniker itself. CoGetObject is a shortcut for calling MkParseDisplayName – the real entry point to the COM moniker namespace. Here is the full way to get a class moniker by going through the moniker:

CComPtr<IMoniker> spClsMoniker;
CComPtr<IBindCtx> spBindCtx;
::CreateBindCtx(0, &spBindCtx);
ULONG eaten;
CComPtr<IClassFactory> spCF;
auto hr = ::MkParseDisplayName(
    spBindCtx,
    L"clsid:9BA05972-F6A8-11CF-A442-00A0C90A8F39",
    &eaten, &spClsMoniker);
if (SUCCEEDED(hr)) {
    spClsMoniker->BindToObject(spBindCtx, nullptr,
        __uuidof(IClassFactory), reinterpret_cast<void**>(&spCF));

MkParseDisplayName takes a “display name” – a string, and attempts to locate the moniker based on the information in the Registry (it actually has some special code for certain OLE stuff which is not interesting in this context). The Bind Context is a helper object that can (in the general case) contain an arbitrary set of properties that can be used by the moniker to customize the way it interprets the display name. The class moniker does not use any property, but it’s still necessary to provide the object even if it has no interesting data in it. If successful, MkParseDisplayName returns the moniker interface pointer, implementing the IMoniker interface that all monikers must implement. IMoniker is somewhat a scary interface, having 20 methods (excluding IUnknown). Fortunately, not all have to be implemented. We’ll get to implementing our own moniker soon.

The primary method in IMoniker is BindToObject, which is tasked of interpreting the display name, if possible, and returning the real object that the client is trying to locate. The client provides the interface it expects the target object to implement – IClassFactory in the case of a class moniker.

You might be wondering what’s the point of the class moniker if you could simply create the required object directly with the normal class factory. One advantage of the moniker is that a string is involved, which allows “late binding” of sorts, and allows other languages, such as scripting languages, to create COM objects indirectly. For example, VBScript provides the GetObject function that calls CoGetObject.

Implementing a Moniker

Some details are still missing, such as how does the moniker object itself gets created? To show that, let’s implement our own moniker. We’ll call it the Process Moniker – its purpose is to locate a COM process object we’ll implement that allows working with a Windows Process object.

Here is an example of something a client would do to find a process object based on its PID, and then display its executable path:

BIND_OPTS opts{ sizeof(opts) };
CComPtr<IWinProcess> spProcess;
auto hr = ::CoGetObject(L"process:3284", 
    &opts, __uuidof(IWinProcess), 
    reinterpret_cast<void**>(&spProcess));
if (SUCCEEDED(hr)) {
    CComBSTR path;
    if (S_OK == spProcess->get_ImagePath(&path)) {
        printf("Image path: %ws\n", path.m_str);
    }
}

The IWinProcess is the interface our process object implements, but there is no need to know its CLSID (in fact, it has none, and is created privately by the moniker). The display name “prcess:3284” identifies the string “process” as the moniker name, meaning there must be a subkey under HKCR named “process” for this to have any chance of working. And under the “process” key there must be the CLSID of the moniker. Here is the final result:

process moniker

The CLSID of the process moniker must be registered normally like all COM classes. The text after the colon is passed to the moniker which should interpret it in a way that makes sense for that moniker (or fail trying). In our case, it’s supposed to be a PID of an existing process.

Let’s see the main steps needed to implement the process moniker. From a technical perspective, I created an ATL DLL project in Visual Studio (could be an EXE as well), and then added an “ATL Simple Object” class template to get the boilerplate code the ATL template provides. We just need to implement IMoniker – no need for some custom interface. Here is the layout of the class:

class ATL_NO_VTABLE CProcessMoniker :
	public CComObjectRootEx<CComMultiThreadModel>,
	public CComCoClass<CProcessMoniker, &CLSID_ProcessMoniker>,
	public IMoniker {
public:
	DECLARE_REGISTRY_RESOURCEID(106)
	DECLARE_CLASSFACTORY_EX(CMonikerClassFactory)

	BEGIN_COM_MAP(CProcessMoniker)
		COM_INTERFACE_ENTRY(IMoniker)
	END_COM_MAP()

	DECLARE_PROTECT_FINAL_CONSTRUCT()
	HRESULT FinalConstruct() {
		return S_OK;
	}
	void FinalRelease() {
	}

public:
	// Inherited via IMoniker
	HRESULT __stdcall GetClassID(CLSID* pClassID) override;
	HRESULT __stdcall IsDirty(void) override;
	HRESULT __stdcall Load(IStream* pStm) override;
	HRESULT __stdcall Save(IStream* pStm, BOOL fClearDirty) override;
	HRESULT __stdcall GetSizeMax(ULARGE_INTEGER* pcbSize) override;
	HRESULT __stdcall BindToObject(IBindCtx* pbc, IMoniker* pmkToLeft, REFIID riidResult, void** ppvResult) override;
    // other IMoniker methods...
	std::wstring m_DisplayName;
};

OBJECT_ENTRY_AUTO(__uuidof(ProcessMoniker), CProcessMoniker)

Those familiar with the typical code the ATL wizard generates might notice one important difference from the standard template: the class factory. It turns out that monikers are not created by an IClassFactory when called by a client invoking MkParseDisplayName (or its CoGetObject wrapper), but instead must implement the interface IParseDisplayName, which we’ll tackle in a moment. This is why DECLARE_CLASSFACTORY_EX(CMonikerClassFactory) is used to instruct ATL to use a custom class factory which we must implement.

MkParseDisplayName operation

Before we get to that, let’s implement the “main” method – BindToObject. We have to assume that the m_DisplayName member already has the process ID – it will be provided by our class factory that creates our moniker. First, we’ll convert the display name to a number:

HRESULT __stdcall CProcessMoniker::BindToObject(IBindCtx* pbc, IMoniker* pmkToLeft, REFIID riidResult, void** ppvResult) {
	auto pid = std::stoul(m_DisplayName);

Next, we’ll attempt to open a handle to the process:

auto hProcess = ::OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, 
    FALSE, pid);
if (!hProcess)
    return HRESULT_FROM_WIN32(::GetLastError());

If we fail, we just return a failed HRESULT and we’re done. If successful, we can create the WinProcess object, pass the handle and return the interface requested by the client (if supported):

	CComObject<CWinProcess>* pProcess;
	auto hr = pProcess->CreateInstance(&pProcess);
	pProcess->SetHandle(hProcess);
	pProcess->AddRef();
	
	hr = pProcess->QueryInterface(riidResult, ppvResult);
	pProcess->Release();
	return hr;
}

The creation of the object is internal via CComObject<>. The WinProcess COM class is not registered, which is just a matter of choice. I decided, a WinProcess object can only be obtained through the Process Moniker.

The calls to AddRef/Release may be puzzling, but there is a good reason for using them. When creating a CComObject<> object, the reference count of the object is zero. Then, the call to AddRef increments it to 1. Next, if the QueryInterface call succeeds, the ref count is incremented to 2. Then, the Release call decrements it to 1, as that is the correct count when the object is returned to the client. If, however, the call to QI fails, the ref count remains at 1, and the Release call will destroy the object! More elegant than calling delete.

SetHandle is a function in CWinProcess (outside the IWinProcess interface) that passes the handle to the object.

The WinProcess COM class is the uninteresting part in all of these, so I created a bare minimum class like so:

class ATL_NO_VTABLE CWinProcess :
	public CComObjectRootEx<CComMultiThreadModel>,
	public IDispatchImpl<IWinProcess> {
public:
	DECLARE_NO_REGISTRY()

	BEGIN_COM_MAP(CWinProcess)
		COM_INTERFACE_ENTRY(IWinProcess)
		COM_INTERFACE_ENTRY(IDispatch)
		COM_INTERFACE_ENTRY_AGGREGATE(IID_IMarshal, m_pUnkMarshaler.p)
	END_COM_MAP()

	DECLARE_PROTECT_FINAL_CONSTRUCT()
	DECLARE_GET_CONTROLLING_UNKNOWN()

	HRESULT FinalConstruct() {
		return CoCreateFreeThreadedMarshaler(
			GetControllingUnknown(), &m_pUnkMarshaler.p);
	}

	void FinalRelease() {
		m_pUnkMarshaler.Release();
		if (m_hProcess)
			::CloseHandle(m_hProcess);
	}

	void SetHandle(HANDLE hProcess);

private:
	HANDLE m_hProcess{ nullptr };
	CComPtr<IUnknown> m_pUnkMarshaler;

	// Inherited via IWinProcess
	HRESULT get_Id(DWORD* pId);
	HRESULT get_ImagePath(BSTR* path);
	HRESULT Terminate(DWORD exitCode);
};

The two properties and one method look like this:

void CWinProcess::SetHandle(HANDLE hProcess) {
	m_hProcess = hProcess;
}

HRESULT CWinProcess::get_Id(DWORD* pId) {
	ATLASSERT(m_hProcess);
	return *pId = ::GetProcessId(m_hProcess), S_OK;
}

HRESULT CWinProcess::get_ImagePath(BSTR* pPath) {
	WCHAR path[MAX_PATH];
	DWORD size = _countof(path);
	if (::QueryFullProcessImageName(m_hProcess, 0, path, &size))
		return CComBSTR(path).CopyTo(pPath);

	return HRESULT_FROM_WIN32(::GetLastError());
}

HRESULT CWinProcess::Terminate(DWORD exitCode) {
	HANDLE hKill;
	if (::DuplicateHandle(::GetCurrentProcess(), m_hProcess, 
		::GetCurrentProcess(), &hKill, PROCESS_TERMINATE, FALSE, 0)) {
		auto success = ::TerminateProcess(hKill, exitCode);
		auto error = ::GetLastError();
		::CloseHandle(hKill);
		return success ? S_OK : HRESULT_FROM_WIN32(error);
	}
	return HRESULT_FROM_WIN32(::GetLastError());
}

The APIs used above are fairly straightforward and of course fully documented.

The last piece of the puzzle is the moniker’s class factory:

class ATL_NO_VTABLE CMonikerClassFactory : 
	public ATL::CComObjectRootEx<ATL::CComMultiThreadModel>,
	public IParseDisplayName {
public:
	BEGIN_COM_MAP(CMonikerClassFactory)
		COM_INTERFACE_ENTRY(IParseDisplayName)
	END_COM_MAP()

	// Inherited via IParseDisplayName
	HRESULT __stdcall ParseDisplayName(IBindCtx* pbc, LPOLESTR pszDisplayName, ULONG* pchEaten, IMoniker** ppmkOut) override;
};

Just one method to implement:

HRESULT __stdcall CMonikerClassFactory::ParseDisplayName(
    IBindCtx* pbc, LPOLESTR pszDisplayName, 
    ULONG* pchEaten, IMoniker** ppmkOut) {
    auto colon = wcschr(pszDisplayName, L':');
    ATLASSERT(colon);
    if (colon == nullptr)
        return E_INVALIDARG;

    //
    // simplistic, assume all display name consumed
    //
    *pchEaten = (ULONG)wcslen(pszDisplayName);

    CComObject<CProcessMoniker>* pMon;
    auto hr = pMon->CreateInstance(&pMon);
    if (FAILED(hr))
        return hr;

    //
    // provide the process ID
    //
    pMon->m_DisplayName = colon + 1;
    pMon->AddRef();
    hr = pMon->QueryInterface(ppmkOut);
    pMon->Release();
    return hr;
}

First, the colon is searched for, as the display name looks like “process:xxxx”. The “xxxx” part is stored in the resulting moniker, created with CComObject<>, similarly to the CWinProcess earlier. The pchEaten value reports back how many characters were consumed – the moniker factory should parse as much as it understands, because moniker composition may be in play. Hopefully, I’ll discuss that in a future post.

Finally, registration must be added for the moniker. Here is ProcessMoniker.rgs, where the lower part was added to connect the “process” ProgId/moniker name to the CLSID of the process moniker:

HKCR
{
	NoRemove CLSID
	{
		ForceRemove {6ea3a80e-2936-43be-8725-2e95896da9a4} = s 'ProcessMoniker class'
		{
			InprocServer32 = s '%MODULE%'
			{
				val ThreadingModel = s 'Both'
			}
			TypeLib = s '{97a86fc5-ffef-4e80-88a0-fa3d1b438075}'
			Version = s '1.0'
		}
	}
	process = s 'Process Moniker Class'
	{
		CLSID = s '{6ea3a80e-2936-43be-8725-2e95896da9a4}'
	}
}

And that is it. Here is an example client that terminates a process given its ID:

void Kill(DWORD pid) {
	std::wstring displayName(L"process:");
	displayName += std::to_wstring(pid);
	BIND_OPTS opts{ sizeof(opts) };
	CComPtr<IWinProcess> spProcess;
	auto hr = ::CoGetObject(displayName.c_str(), &opts, 
		__uuidof(IWinProcess), reinterpret_cast<void**>(&spProcess));
	if (SUCCEEDED(hr)) {
		auto hr = spProcess->Terminate(1);
		if (SUCCEEDED(hr))
			printf("Process %u terminated.\n", pid);
		else
			printf("Error terminating process: hr=0x%X\n", hr);
	}
}

All the code can be found in this Github repo: zodiacon/MonikerFun: Demonstrating a simple moniker. (github.com)

Here is VBScript example (this works because WinProcess implements IDispatch):

set process = GetObject("process:25520")
MsgBox process.ImagePath

How about .NET or PowerShell? Here is Powershell:

PS> $p = [System.Runtime.InteropServices.Marshal]::BindToMoniker("process:25520")
PS> $p | Get-Member                                                                                             

   TypeName: System.__ComObject#{3ab0471f-2635-429d-95e9-f2baede2859e}

Name      MemberType Definition
----      ---------- ----------
Terminate Method     void Terminate (uint)
Id        Property   uint Id () {get}
ImagePath Property   string ImagePath () {get}


PS> $p.ImagePath
C:\Windows\System32\notepad.exe

The DisplayWindows function just displays names of Explorer windows obtained by using IShellWindows:

void DisplayWindows(IShellWindows* pShell) {
	long count = 0;
	pShell->get_Count(&count);
	for (long i = 0; i < count; i++) {
		CComPtr<IDispatch> spDisp;
		pShell->Item(CComVariant(i), &spDisp);
		CComQIPtr<IWebBrowserApp> spWin(spDisp);
		if (spWin) {
			CComBSTR name;
			spWin->get_LocationName(&name);
			printf("Name: %ws\n", name.m_str);
		}
	}
}

Happy Moniker day!

Next Windows Internals Training

I’m happy to open registration for the next 5 day Windows Internals training to be conducted in November in the following dates and from 11am to 7pm, Eastern Standard Time (EST) (8am to 4pm PST): 21, 22, 28, 29, 30.

The syllabus can be found here (some modifications possible, but the general outline should remain).

Training cost is 900 USD if paid by an individual, or 1800 USD if paid by a company. Participants in any of my previous training classes get 10% off.

If you’d like to register, please send me an email to [email protected] with “Windows Internals training” in the title, provide your full name, company (if any), preferred contact email, and your time zone.

The sessions will be recorded, so you can watch any part you may be missing, or that may be somewhat overwhelming in “real time”.

As usual, if you have any questions, feel free to send me an email, or DM on twitter (@zodiacon) or Linkedin (https://www.linkedin.com/in/pavely/).

Upcoming COM Programming Class

Today I’m happy to announce the next COM Programming class to be held in February 2023. The syllabus for the 3 day class can be found here. The course will be delivered in 6 half-days (4 hours each).

Dates: February (7, 8, 9, 14, 15, 16).
Times: 11am to 3pm EST (8am to 12pm PST) (4pm to 8pm UT)
Cost: 750 USD (if paid by an individual), 1400 USD (if paid by a company).

Half days should make it comfortable enough even if you’re not in an ideal time zone.

The class will be conducted remotely using Microsoft Teams.

What you need to know before the class: You should be comfortable using Windows on a Power User level. Concepts such as processes, threads, DLLs, and virtual memory should be understood fairly well. You should have experience writing code in C and some C++. You don’t have to be an expert, but you must know C and basic C++ to get the most out of this class. In case you have doubts, talk to me.

Participants in my Windows Internals and Windows System Programming classes have the required knowledge for the class.

We’ll start by looking at why COM was created in the first place, and then build clients and servers, digging into various mechanisms COM provides. See the syllabus for more details.

Previous students in my classes get 10% off. Multiple participants from the same company get a discount (email me for the details).

To register, send an email to [email protected] with the title “COM Programming Training”, and write the name(s), email(s) and time zone(s) of the participants.

Unnamed Directory Objects

A lot of the functionality in Windows is based around various kernel objects. One such object is a Directory, not to be confused with a directory in a file system. A Directory object is conceptually simple: it’s a container for other kernel objects, including other Directory objects, thus creating a hierarchy used by the kernel’s Object Manager to manage named objects. This arrangement can be easily seen with tools like WinObj from Sysinternals:

The left part of WinObj shows object manager directories, where named objects are “stored” and can be located by name. Clear and simple enough.

However, Directory objects can be unnamed as well as named. How can this be? Here is my Object Explorer tool (similar functionality is available with my System Explorer tool as well). One of its views is a “statistical” view of all object types, some of their properties, such as their name, type index, number of objects and handles, peak number of objects and handles, generic access mapping, and the pool type they’re allocated from.

If you right-click the Directory object type and select “All Objects”, you’ll see another view that shows all Directory objects in the system (well, not necessarily all, but most*).

If you scroll a bit, you’ll see many unnamed Directory objects that have no name:

It seems weird, as a Directory with no name doesn’t make sense. These directories, however, are “real” and serve an important purpose – managing a private object namespace. I blogged about private object namespaces quite a few years ago (it was in my old blog site that is now unfortunately lost), but here is the gist of it:

Object names are useful because they allow easy sharing between processes. For example, if two or more processes would like to share memory, they can create a memory mapped file object (called Section within the kernel) with a name they are all aware of. Calling CreateFileMapping (or one of its variants) with the same name will create the object (by the first caller), where subsequent callers get handles to the existing object because it was looked up by name.

This is easy and useful, but there is a possible catch: since the name is “visible” using tools or APIs, other processes can “interfere” with the object by getting their own handle using that visible name and “meddle” with the object, maliciously or accidentally.

The solution to this problem arrived in Windows Vista with the idea of private object namespaces. A set of cooperating processes can create a private namespace only they can use, protected by a “secret” name and more importantly a boundary descriptor. The details are beyond the scope of this post, but it’s all documented in the Windows API functions such as CreateBoundaryDescriptor, CreatePrivateNamespace and friends. Here is an example of using these APIs to create a private namespace with a section object in it (error handling omitted):

HANDLE hBD = ::CreateBoundaryDescriptor(L"MyDescriptor", 0);
BYTE sid[SECURITY_MAX_SID_SIZE];
auto psid = reinterpret_cast<PSID>(sid);
DWORD sidLen;
::CreateWellKnownSid(WinBuiltinUsersSid, nullptr, psid, &sidLen);
::AddSIDToBoundaryDescriptor(&m_hBD, psid);

// create the private namespace
hNamespace = ::CreatePrivateNamespace(nullptr, hBD, L"MyNamespace");
if (!hNamespace) { // maybe created already?
	hNamespace = ::OpenPrivateNamespace(hBD, L"MyNamespace");
namespace");
}

HANDLE hSharedMem = ::CreateFileMapping(INVALID_HANDLE_VALUE, nullptr, PAGE_READWRITE, 0, 1 << 12, L"MyNamespace\\MySharedMem"));

This snippet is taken from the PrivateSharing code example from the Windows 10 System Programming part 1 book.

If you run this demo application, and look at the resulting handle (hSharedMem) in the above code in a tool like Process Explorer or Object Explorer you’ll see the name of the object is not given:

The full name is not shown and cannot be retrieved from user mode. And even if it could somehow be located, the boundary descriptor provides further protection. Let’s examine this object in the kernel debugger. Copying its address from the object’s properties:

Pasting the address into a local kernel debugger – first using the generic !object command:

lkd> !object 0xFFFFB3068E162D10
Object: ffffb3068e162d10  Type: (ffff9507ed78c220) Section
    ObjectHeader: ffffb3068e162ce0 (new version)
    HandleCount: 1  PointerCount: 32769
    Directory Object: ffffb3069e8cbe00  Name: MySharedMem

The name is there, but the directory object is there as well. Let’s examine it:

lkd> !object ffffb3069e8cbe00
Object: ffffb3069e8cbe00  Type: (ffff9507ed6d0d20) Directory
    ObjectHeader: ffffb3069e8cbdd0 (new version)
    HandleCount: 3  PointerCount: 98300

    Hash Address          Type                      Name
    ---- -------          ----                      ----
     19  ffffb3068e162d10 Section                   MySharedMem

There is one object in this directory. What’s the directory’s name? We need to examine the object header for that – its address is given in the above output:

lkd> dt nt!_OBJECT_HEADER ffffb3069e8cbdd0
   +0x000 PointerCount     : 0n32769
   +0x008 HandleCount      : 0n1
   +0x008 NextToFree       : 0x00000000`00000001 Void
   +0x010 Lock             : _EX_PUSH_LOCK
   +0x018 TypeIndex        : 0x53 'S'
   +0x019 TraceFlags       : 0 ''
   +0x019 DbgRefTrace      : 0y0
   +0x019 DbgTracePermanent : 0y0
   +0x01a InfoMask         : 0x8 ''
   +0x01b Flags            : 0 ''
   +0x01b NewObject        : 0y0
   +0x01b KernelObject     : 0y0
   +0x01b KernelOnlyAccess : 0y0
   +0x01b ExclusiveObject  : 0y0
   +0x01b PermanentObject  : 0y0
   +0x01b DefaultSecurityQuota : 0y0
   +0x01b SingleHandleEntry : 0y0
   +0x01b DeletedInline    : 0y0
   +0x01c Reserved         : 0x301
   +0x020 ObjectCreateInfo : 0xffff9508`18f2ba40 _OBJECT_CREATE_INFORMATION
   +0x020 QuotaBlockCharged : 0xffff9508`18f2ba40 Void
   +0x028 SecurityDescriptor : 0xffffb305`dd0d56ed Void
   +0x030 Body             : _QUAD

Getting a kernel’s object name is a little tricky, and will not be fully described here. The first requirement is the InfoMask member must have bit 1 set (value of 2), as this indicates a name is present. Since it’s not (the value is 8), there is no name to this directory. We can examine the directory object in more detail by looking at the real data structure underneath given the object’s original address:

kd> dt nt!_OBJECT_DIRECTORY ffffb3069e8cbe00
   +0x000 HashBuckets      : [37] (null) 
   +0x128 Lock             : _EX_PUSH_LOCK
   +0x130 DeviceMap        : (null) 
   +0x138 ShadowDirectory  : (null) 
   +0x140 NamespaceEntry   : 0xffffb306`9e8cbf58 Void
   +0x148 SessionObject    : (null) 
   +0x150 Flags            : 1
   +0x154 SessionId        : 0xffffffff

The interesting piece is the NamespaceEntry member, which is not-NULL. This indicates the purpose of this directory: to be a container for a private namespace’s objects. You can also click on HasBuckets and locate the single section object there.

Going back to Process Explorer, enabling unnamed object handles (View menu, Show Unnamed Handles and Mappings) and looking for unnamed directory objects:

The directory’s address is the same one we were looking at!

The pointer at NamespaceEntry points to an undocumented structure that is not currently provided with the symbols. But just looking a bit beyond the directory’s object structure shows a hint:

lkd> db ffffb3069e8cbe00+158
ffffb306`9e8cbf58  d8 f9 a3 55 06 b3 ff ff-70 46 12 66 07 f8 ff ff  ...U....pF.f....
ffffb306`9e8cbf68  00 be 8c 9e 06 b3 ff ff-48 00 00 00 00 00 00 00  ........H.......
ffffb306`9e8cbf78  00 00 00 00 00 00 00 00-0b 00 00 00 00 00 00 00  ................
ffffb306`9e8cbf88  01 00 00 00 02 00 00 00-48 00 00 00 00 00 00 00  ........H.......
ffffb306`9e8cbf98  01 00 00 00 20 00 00 00-4d 00 79 00 44 00 65 00  .... ...M.y.D.e.
ffffb306`9e8cbfa8  73 00 63 00 72 00 69 00-70 00 74 00 6f 00 72 00  s.c.r.i.p.t.o.r.
ffffb306`9e8cbfb8  02 00 00 00 18 00 00 00-01 02 00 00 00 00 00 05  ................
ffffb306`9e8cbfc8  20 00 00 00 21 02 00 00-00 00 00 00 00 00 00 00   ...!...........

The name “MyDescriptor” is clearly visible, which is the name of the boundary descriptor in the above code.

The kernel debugger’s documentation indicates that the !object command with a -p switch should show the private namespaces. However, this fails:

lkd> !object -p
00000000: Unable to get value of ObpPrivateNamespaceLookupTable

The debugger seems to fail locating a global kernel variable. This is probably a bug in the debugger command, because object namespaces scope has changed since the introduction of Server Silos in Windows 10 version 1607 (for example, Docker uses these when running Windows containers). Each silo has its own object manager namespace, so the old global variable does not exist anymore. I suspect Microsoft has not updated this command switch to support silos. Even with no server silos running, the host is considered to be in its own (global) silo, called host silo. You can see its details by utilizing the !silo debugger command:

kd> !silo -g host
Server silo globals fffff80766124540:
		Default Error Port: ffff950815bee140
		ServiceSessionId  : 0
		OB Root Directory : 
		State             : Running

Clicking the “Server silo globals” link, shows more details:

kd> dx -r1 (*((nt!_ESERVERSILO_GLOBALS *)0xfffff80766124540))
(*((nt!_ESERVERSILO_GLOBALS *)0xfffff80766124540))                 [Type: _ESERVERSILO_GLOBALS]
    [+0x000] ObSiloState      [Type: _OBP_SILODRIVERSTATE]
    [+0x2e0] SeSiloState      [Type: _SEP_SILOSTATE]
    [+0x310] SeRmSiloState    [Type: _SEP_RM_LSA_CONNECTION_STATE]
    [+0x360] EtwSiloState     : 0xffff9507edbc9000 [Type: _ETW_SILODRIVERSTATE *]
    [+0x368] MiSessionLeaderProcess : 0xffff95080bbdb040 [Type: _EPROCESS *]
    [+0x370] ExpDefaultErrorPortProcess : 0xffff950815bee140 [Type: _EPROCESS *]
<truncated>

ObSiloState is the root object related to the object manager. Clicking this one shows:

lkd> dx -r1 (*((ntkrnlmp!_OBP_SILODRIVERSTATE *)0xfffff80766124540))
(*((ntkrnlmp!_OBP_SILODRIVERSTATE *)0xfffff80766124540))                 [Type: _OBP_SILODRIVERSTATE]
    [+0x000] SystemDeviceMap  : 0xffffb305c8c48720 [Type: _DEVICE_MAP *]
    [+0x008] SystemDosDeviceState [Type: _OBP_SYSTEM_DOS_DEVICE_STATE]
    [+0x078] DeviceMapLock    [Type: _EX_PUSH_LOCK]
    [+0x080] PrivateNamespaceLookupTable [Type: _OBJECT_NAMESPACE_LOOKUPTABLE]

PrivateNamespaceLookupTable is the root object for the private namespaces for this Silo (in this example it’s the host silo).

The interested reader is welcome to dig into this further.

The list of private namespaces is provided with the WinObjEx64 tool if you run it elevated and have local kernel debugging enabled, as it uses the kernel debugger’s driver to read kernel memory.

* Most objects, because the way Object Explorer works is by enumerating handles and associating them with objects. However, some objects are held using references from the kernel with zero handles. Such objects cannot be detected by Object Explorer.

Introduction to the Windows Filtering Platform

As part of the second edition of Windows Kernel Programming, I’m working on chapter 13 to describe the basics of the Windows Filtering Platform (WFP). The chapter will focus mostly on kernel-mode WFP Callout drivers (it is a kernel programming book after all), but I am also providing a brief introduction to WFP and its user-mode API.

This introduction (with some simplifications) is what this post is about. Enjoy!

The Windows Filtering Platform (WFP) provides flexible ways to control network filtering. It exposes user-mode and kernel-mode APIs, that interact with several layers of the networking stack. Some configuration and control is available directly from user-mode, without requiring any kernel-mode code (although it does require administrator-level access). WFP replaces older network filtering technologies, such as Transport Driver Interface (TDI) filters some types of NDIS filters.

If examining network packets (and even modification) is required, a kernel-mode Callout driver can be written, which is what we’ll be concerned with in this chapter. We’ll begin with an overview of the main pieces of WFP, look at some user-mode code examples for configuring filters before diving into building simple Callout drivers that allows fine-grained control over network packets.

WFP is comprised of user-mode and kernel-mode components. A very high-level architecture is shown here:

In user-mode, the WFP manager is the Base Filtering Engine (BFE), which is a service implemented by bfe.dll and hosted in a standard svchost.exe instance. It implements the WFP user-mode API, essentially managing the platform, talking to its kernel counterpart when needed. We’ll examine some of these APIs in the next section.

User-mode applications, services and other components can utilize this user-mode management API to examine WFP objects state, and make changes, such as adding or deleting filters. A classic example of such “user” is the Windows Firewall, which is normally controllable by leveraging the Microsoft Management Console (MMC) that is provided for this purpose, but using these APIs from other applications is just as effective.

The kernel-mode filter engine exposes various logical layers, where filters (and callouts) can be attached. Layers represent locations in the network processing of one or more packets. The TCP/IP driver makes calls to the WFP kernel engine so that it can decide which filters (if any) should be “invoked”.

For filters, this means checking the conditions set by the filter against the current request. If the conditions are satisfied, the filter’s action is applied. Common actions include blocking a request from being further processed, allowing the request to continue without further processing in this layer, continuing to the next filter in this layer (if any), and invoking a callout driver. Callouts can perform any kind of processing, such as examining and even modifying packet data.
The relationship between layers, filters, and callouts is shown here:

As you can see the diagram, each layer can have zero or more filters, and zero or more callouts. The number and meaning of the layers is fixed and provided out of the box by Windows. On most system, there are about 100 layers. Many of the layers are sets of pairs, where one is for IPv4 and the other (identical in purpose) is for IPv6.

The WFP Explorer tool I created provides some insight into what makes up WFP. Running the tool and selecting View/Layers from the menu (or clicking the Layers tool bar button) shows a view of all existing layers.

You can download the WFP Explorer tool from its Github repository
(https://github.com/zodiacon/WFPExplorer) or the AllTools repository
(https://github.com/zodiacon/AllTools).

Each layer is uniquely identified by a GUID. Its Layer ID is used internally by the kernel engine as an identifier rather than the GUID, as it’s smaller and so is faster (layer IDs are 16-bit only). Most layers have fields that can be used by filters to set conditions for invoking their actions. Double-clicking a layer shows its properties. The next figure shows the general properties of an example layer. Notice it has 382 filters and 2 callouts attached to it.

Clicking the Fields tab shows the fields available in this layer, that can be used by filters to set conditions.

The meaning of the various layers, and the meaning of the fields for the layers are all documented in the official WFP documentation.

The currently existing filters can be viewed in WFP Explorer by selecting Filters from the View menu. Layers cannot be added or removed, but filters can. Management code (user or kernel) can add and/or remove filters dynamically while the system is running. You can see that on the system the tool is running on there are currently 2978 filters.

Each filter is uniquely identified by a GUID, and just like layers has a “shorter” id (64-bit) that is used by the kernel engine to more quickly compare filter IDs when needed. Since multiple filters can be assigned to the same layer, some kind of ordering must be used when assessing filters. This is where the filter’s weight comes into play. A weight is a 64-bit value that is used to sort filters by priority. As you can see in figure 13-7, there are two weight properties – weight and effective weight. Weight is what is specified when adding the filter, but effective weight is the actual one used. There are three possible values to set for weight:

  • A value between 0 and 15 is interpreted by WFP as a weight index, which simply means that the effective weight is going to start with 4 bits having the specified weight value and generate the other 60 bit. For example, if the weight is set to 5, then the effective weight is going to be between 0x5000000000000000 and 0x5FFFFFFFFFFFFFFF.
  • An empty value tells WFP to generate an effective weight somewhere in the 64-bit range.
  • A value above 15 is taken as is to become the effective weight.

What is an “empty” value? The weight is not really a number, but a FWP_VALUE type can hold all sorts of values, including holding no value at all (empty).

Double-clicking a filter in WFP Explorer shows its general properties:

The Conditions tab shows the conditions this filter is configured with. When all the conditions are met, the action of the filter is going to fire.

The list of fields used by a filter must be a subset of the fields exposed by the layer this filter is attached to. There are six conditions shown in figure 13-9 out of the possible 39 fields supported by this layer (“ALE Receive/Accept v4 Layer”). As you can see, there is a lot of flexibility in specifying conditions for fields – this is evident in the matching enumeration, FWPM_MATCH_TYPE:

typedef enum FWP_MATCH_TYPE_ {
    FWP_MATCH_EQUAL    = 0,
    FWP_MATCH_GREATER,
    FWP_MATCH_LESS,
    FWP_MATCH_GREATER_OR_EQUAL,
    FWP_MATCH_LESS_OR_EQUAL,
    FWP_MATCH_RANGE,
    FWP_MATCH_FLAGS_ALL_SET,
    FWP_MATCH_FLAGS_ANY_SET,
    FWP_MATCH_FLAGS_NONE_SET,
    FWP_MATCH_EQUAL_CASE_INSENSITIVE,
    FWP_MATCH_NOT_EQUAL,
    FWP_MATCH_PREFIX,
    FWP_MATCH_NOT_PREFIX,
    FWP_MATCH_TYPE_MAX
} FWP_MATCH_TYPE;

The WFP API exposes its functionality for user-mode and kernel-mode callers. The header files used are different, to cater for differences in API expectations between user-mode and kernel-mode, but APIs in general are identical. For example, kernel APIs return NTSTATUS, whereas user-mode APIs return a simple LONG, that is the error value that is returned normally from GetLastError. Some APIs are provided for kernel-mode only, as they don’t make sense for user mode.

W> The user-mode WFP APIs never set the last error, and always return the error value directly. Zero (ERROR_SUCCESS) means success, while other (positive) values mean failure. Do not call GetLastError when using WFP – just look at the returned value.

WFP functions and structures use a versioning scheme, where function and structure names end with a digit, indicating version. For example, FWPM_LAYER0 is the first version of a structure describing a layer. At the time of writing, this was the only structure for describing a layer. As a counter example, there are several versions of the function beginning with FwpmNetEventEnum: FwpmNetEventEnum0 (for Vista+), FwpmNetEventEnum1 (Windows 7+), FwpmNetEventEnum2 (Windows 8+), FwpmNetEventEnum3 (Windows 10+), FwpmNetEventEnum4 (Windows 10 RS4+), and FwpmNetEventEnum5 (Windows 10 RS5+). This is an extreme example, but there are others with less “versions”. You can use any version that matches the target platform. To make it easier to work with these APIs and structures, a macro is defined with the base name that is expanded to the maximum supported version based on the target compilation platform. Here is part of the declarations for the macro FwpmNetEventEnum:

DWORD FwpmNetEventEnum0(
   _In_ HANDLE engineHandle,
   _In_ HANDLE enumHandle,
   _In_ UINT32 numEntriesRequested,
   _Outptr_result_buffer_(*numEntriesReturned) FWPM_NET_EVENT0*** entries,
   _Out_ UINT32* numEntriesReturned);
#if (NTDDI_VERSION >= NTDDI_WIN7)
DWORD FwpmNetEventEnum1(
   _In_ HANDLE engineHandle,
   _In_ HANDLE enumHandle,
   _In_ UINT32 numEntriesRequested,
   _Outptr_result_buffer_(*numEntriesReturned) FWPM_NET_EVENT1*** entries,
   _Out_ UINT32* numEntriesReturned);
#endif // (NTDDI_VERSION >= NTDDI_WIN7)
#if (NTDDI_VERSION >= NTDDI_WIN8)
DWORD FwpmNetEventEnum2(
   _In_ HANDLE engineHandle,
   _In_ HANDLE enumHandle,
   _In_ UINT32 numEntriesRequested,
   _Outptr_result_buffer_(*numEntriesReturned) FWPM_NET_EVENT2*** entries,
   _Out_ UINT32* numEntriesReturned);
#endif // (NTDDI_VERSION >= NTDDI_WIN8)

You can see that the differences in the functions relate to the structures returned as part of these APIs (FWPM_NET_EVENTx). It’s recommended you use the macros, and only turn to specific versions if there is a compelling reason to do so.

The WFP APIs adhere to strict naming conventions that make it easier to use. All management functions start with Fwpm (Filtering Windows Platform Management), and all management structures start with FWPM. The function names themselves use the pattern <prefix><object type><operation>, such as FwpmFilterAdd and FwpmLayerGetByKey.

It’s curious that the prefixes used for functions, structures, and enums start with FWP rather than the (perhaps) expected WFP. I couldn’t find a compelling reason for this.

WFP header files start with fwp and end with u for user-mode or k for kernel-mode. For example, fwpmu.h holds the management functions for user-mode callers, whereas fwpmk.h is the header for kernel callers. Two common files, fwptypes.h and fwpmtypes.h are used by both user-mode and kernel-mode headers. They are included by the “main” header files.

User-Mode Examples

Before making any calls to specific APIs, a handle to the WFP engine must be opened with FwpmEngineOpen:

DWORD FwpmEngineOpen0(
   _In_opt_ const wchar_t* serverName,  // must be NULL
   _In_ UINT32 authnService,            // RPC_C_AUTHN_DEFAULT
   _In_opt_ SEC_WINNT_AUTH_IDENTITY_W* authIdentity,
   _In_opt_ const FWPM_SESSION0* session,
   _Out_ HANDLE* engineHandle);

Most of the arguments have good defaults when NULL is specified. The returned handle must be used with subsequent APIs. Once it’s no longer needed, it must be closed:

DWORD FwpmEngineClose0(_Inout_ HANDLE engineHandle);

Enumerating Objects

What can we do with an engine handle? One thing provided with the management API is enumeration. These are the APIs used by WFP Explorer to enumerate layers, filters, sessions, and other object types in WFP. The following example displays some details for all the filters in the system (error handling omitted for brevity, the project wfpfilters has the full source code):

#include <Windows.h>
#include <fwpmu.h>
#include <stdio.h>
#include <string>

#pragma comment(lib, "Fwpuclnt")

std::wstring GuidToString(GUID const& guid) {
    WCHAR sguid[64];
    return ::StringFromGUID2(guid, sguid, _countof(sguid)) ? sguid : L"";
}

const char* ActionToString(FWPM_ACTION const& action) {
    switch (action.type) {
        case FWP_ACTION_BLOCK:               return "Block";
        case FWP_ACTION_PERMIT:              return "Permit";
        case FWP_ACTION_CALLOUT_TERMINATING: return "Callout Terminating";
        case FWP_ACTION_CALLOUT_INSPECTION:  return "Callout Inspection";
        case FWP_ACTION_CALLOUT_UNKNOWN:     return "Callout Unknown";
        case FWP_ACTION_CONTINUE:            return "Continue";
        case FWP_ACTION_NONE:                return "None";
        case FWP_ACTION_NONE_NO_MATCH:       return "None (No Match)";
    }
    return "";
}

int main() {
    //
    // open a handle to the WFP engine
    //
    HANDLE hEngine;
    FwpmEngineOpen(nullptr, RPC_C_AUTHN_DEFAULT, nullptr, nullptr, &hEngine);

    //
    // create an enumeration handle
    //
    HANDLE hEnum;
    FwpmFilterCreateEnumHandle(hEngine, nullptr, &hEnum);

    UINT32 count;
    FWPM_FILTER** filters;
    //
    // enumerate filters
    //
    FwpmFilterEnum(hEngine, hEnum, 
        8192,       // maximum entries, 
        &filters,   // returned result
        &count);    // how many actually returned

    for (UINT32 i = 0; i < count; i++) {
        auto f = filters[i];
        printf("%ws Name: %-40ws Id: 0x%016llX Conditions: %2u Action: %s\n",
            GuidToString(f->filterKey).c_str(),
            f->displayData.name,
            f->filterId,
            f->numFilterConditions,
            ActionToString(f->action));
    }
    //
    // free memory allocated by FwpmFilterEnum
    //
    FwpmFreeMemory((void**)&filters);

    //
    // close enumeration handle
    //
    FwpmFilterDestroyEnumHandle(hEngine, hEnum);

    //
    // close engine handle
    //
    FwpmEngineClose(hEngine);

    return 0;
}

The enumeration pattern repeat itself with all other WFP object types (layers, callouts, sessions, etc.).

Adding Filters

Let’s see if we can add a filter to perform some useful function. Suppose we want to prevent network access from some process. We can add a filter at an appropriate layer to make it happen. Adding a filter is a matter of calling FwpmFilterAdd:

DWORD FwpmFilterAdd0(
   _In_ HANDLE engineHandle,
   _In_ const FWPM_FILTER0* filter,
   _In_opt_ PSECURITY_DESCRIPTOR sd,
   _Out_opt_ UINT64* id);

The main work is to fill a FWPM_FILTER structure defined like so:

typedef struct FWPM_FILTER0_ {
    GUID filterKey;
    FWPM_DISPLAY_DATA0 displayData;
    UINT32 flags;
    /* [unique] */ GUID *providerKey;
    FWP_BYTE_BLOB providerData;
    GUID layerKey;
    GUID subLayerKey;
    FWP_VALUE0 weight;
    UINT32 numFilterConditions;
    /* [unique][size_is] */ FWPM_FILTER_CONDITION0 *filterCondition;
    FWPM_ACTION0 action;
    /* [switch_is] */ /* [switch_type] */ union 
        {
        /* [case()] */ UINT64 rawContext;
        /* [case()] */ GUID providerContextKey;
        }     ;
    /* [unique] */ GUID *reserved;
    UINT64 filterId;
    FWP_VALUE0 effectiveWeight;
} FWPM_FILTER0;

The weird-looking comments are generated by the Microsoft Interface Definition Language (MIDL) compiler when generating the header file from an IDL file. Although IDL is most commonly used by Component Object Model (COM) to define interfaces and types, WFP uses IDL to define its APIs, even though no COM interfaces are used; just plain C functions. The original IDL files are provided with the SDK, and they are worth checking out, since they may contain developer comments that are not “transferred” to the resulting header files.

Some members in FWPM_FILTER are necessary – layerKey to indicate the layer to attach this filter, any conditions needed to trigger the filter (numFilterConditions and the filterCondition array), and the action to take if the filter is triggered (action field).

Let’s create some code that prevents the Windows Calculator from accessing the network. You may be wondering why would calculator require network access? No, it’s not contacting Google to ask for the result of 2+2. It’s using the Internet for accessing current exchange rates.

Clicking the Update Rates button causes Calculator to consult the Internet for the updated exchange rate. We’ll add a filter that prevents this.

We’ll start as usual by opening handle to the WFP engine as was done in the previous example. Next, we need to fill the FWPM_FILTER structure. First, a nice display name:

FWPM_FILTER filter{};   // zero out the structure
WCHAR filterName[] = L"Prevent Calculator from accessing the web";
filter.displayData.name = filterName;

The name has no functional part – it just allows easy identification when enumerating filters. Now we need to select the layer. We’ll also specify the action:

filter.layerKey = FWPM_LAYER_ALE_AUTH_CONNECT_V4;
filter.action.type = FWP_ACTION_BLOCK;

There are several layers that could be used for blocking access, with the above layer being good enough to get the job done. Full description of the provided layers, their purpose and when they are used is provided as part of the WFP documentation.

The last part to initialize is the conditions to use. Without conditions, the filter is always going to be invoked, which will block all network access (or just for some processes, based on its effective weight). In our case, we only care about the application – we don’t care about ports or protocols. The layer we selected has several fields, one of with is called ALE App ID (ALE stands for Application Layer Enforcement).

This field can be used to identify an executable. To get that ID, we can use FwpmGetAppIdFromFileName. Here is the code for Calculator’s executable:

WCHAR filename[] = LR"(C:\Program Files\WindowsApps\Microsoft.WindowsCalculator_11.2210.0.0_x64__8wekyb3d8bbwe\CalculatorApp.exe)";
FWP_BYTE_BLOB* appId;
FwpmGetAppIdFromFileName(filename, &appId);

The code uses the path to the Calculator executable on my system – you should change that as needed because Calculator’s version might be different. A quick way to get the executable path is to run Calculator, open Process Explorer, open the resulting process properties, and copy the path from the Image tab.

The R"( and closing parenthesis in the above snippet disable the “escaping” property of backslashes, making it easier to write file paths (C++ 14 feature).

The return value from FwpmGetAppIdFromFileName is a BLOB that needs to be freed eventually with FwpmFreeMemory.

Now we’re ready to specify the one and only condition:

FWPM_FILTER_CONDITION cond;
cond.fieldKey = FWPM_CONDITION_ALE_APP_ID;      // field
cond.matchType = FWP_MATCH_EQUAL;
cond.conditionValue.type = FWP_BYTE_BLOB_TYPE;
cond.conditionValue.byteBlob = appId;

filter.filterCondition = &cond;
filter.numFilterConditions = 1;

The conditionValue member of FWPM_FILTER_CONDITION is a FWP_VALUE, which is a generic way to specify many types of values. It has a type member that indicates the member in a big union that should be used. In our case, the type is a BLOB (FWP_BYTE_BLOB_TYPE) and the actual value should be passed in the byteBlob union member.

The last step is to add the filter, and repeat the exercise for IPv6, as we don’t know how Calculator connects to the currency exchange server (we can find out, but it would be simpler and more robust to just block IPv6 as well):

FwpmFilterAdd(hEngine, &filter, nullptr, nullptr);

filter.layerKey = FWPM_LAYER_ALE_AUTH_CONNECT_V6;   // IPv6
FwpmFilterAdd(hEngine, &filter, nullptr, nullptr);

We didn’t specify any GUID for the filter. This causes WFP to generate a GUID. We didn’t specify weight, either. WFP will generate them.

All that’s left now is some cleanup:

FwpmFreeMemory((void**)&appId);
FwpmEngineClose(hEngine);

Running this code (elevated) should and trying to refresh the currency exchange rate with Calculator should fail. Note that there is no need to restart Calculator – the effect is immediate.

We can locate the filters added with WFP Explorer:

Double-clicking one of the filters and selecting the Conditions tab shows the only condition where the App ID is revealed to be the full path of the executable in device form. Of course, you should not take any dependency on this format, as it may change in the future.

You can right-click the filters and delete them using WFP Explorer. The FwpmFilterDeleteByKey API is used behind the scenes. This will restore Calculator’s exchange rate update functionality.

Upcoming Public Training Classes for April/May

Today I’m happy to announce two training classes to take place in April and May. These classes will be in 4-hour session chunks, so that it’s easier to consume even for uncomfortable time zones.

The first is Advanced Windows Kernel Programming, a class I was promising for quite some time now… it will be held on the following dates:

  • April: 18, 20, 24, 27 and May: 1, 4, 8, 11 (4 days total)
  • Times: 11am to 3pm ET (8am-12pm PT, 4pm to 8pm UT/GMT)

The course will include advanced topics in Windows kernel development, and is recommended for those that were in my Windows Kernel Programming class or have equivalent knowledge; for example, by reading my book Windows Kernel Programming.

Example topics include: deep dive into Windows’ kernel design, working with APCs, Windows Filtering Platform callout drivers, advanced memory management techniques, plug & play filter drivers, and more!

The second class is Windows Internals to be held on the following dates:

  • May: 2, 3, 9, 10, 15, 18, 22, 24, 30 and June: 1, 5 (5.5 days)
  • Times: 11am to 3pm ET (8am-12pm PT, 4pm to 8pm UT/GMT)

The syllabus can be found here (some modifications possible, but the general outline remains).

Cost
950 USD (if paid by an individual), 1900 USD (if paid by a company). The cost is the same for these training classes. Previous students in my classes get 10% off.
Multiple participants from the same company get a discount as well (contact me for the details).

If you’d like to register, please send me an email to [email protected] with the name of the training in the email title, provide your full name, company (if any), preferred contact email, and your time zone.

The sessions will be recorded, so you can watch any part you may be missing, or that may be somewhat overwhelming in “real time”.

As usual, if you have any questions, feel free to send me an email, or DM on twitter (@zodiacon) or Linkedin (https://www.linkedin.com/in/pavely/).


❌