This method utilizes TLS callbacks to execute a payload without spawning any threads in a remote process. This method is inspired by Threadless Injection as RemoteTLSCallbackInjection does not invoke any API calls to trigger the injected payload.
Create a suspended process using the CreateProcessViaWinAPIsW function (i.e. RuntimeBroker.exe).
Fetch the remote process image base address followed by reading the process's PE headers.
Fetch an address to a TLS callback function.
Patch a fixed shellcode (i.e. g_FixedShellcode) with runtime-retrieved values. This shellcode is responsible for restoring both original bytes and memory permission of the TLS callback function's address.
Inject both shellcodes: g_FixedShellcode and the main payload.
Patch the TLS callback function's address and replace it with the address of our injected payload.
Resume process.
The g_FixedShellcode shellcode will then make sure that the main payload executes only once by restoring the original TLS callback's original address before calling the main payload. A TLS callback can execute multiple times across the lifespan of a process, therefore it is important to control the number of times the payload is triggered by restoring the original code path execution to the original TLS callback function.
Demo
The following image shows our implementation, RemoteTLSCallbackInjection.exe, spawning a cmd.exe as its main payload.
TL;DR - Debugging an application can provide the insight to understanding strange runtime application behaviour or help troubleshoot a subtle bug in your software. Normally, when debugging, you have source code and data type information (aka symbols) to help navigate your application. In the world of Reverse Engineering closed source applications, you won’t have the needed information to debu...
A while back I blogged about the differences between the virtual desktop feature exposed to users on Windows 10/11, and the Desktops tool from Sysinternals. In this post, I’d like to shed some more light on Window Stations, desktops, and windows. I assume you have read the aforementioned blog post before continuing.
We know that Window Stations are contained in sessions. Can we enumerate these? The EnumWindowStations API is available in the Windows API, but it only returns the Windows Stations in the current session. There is no “EnumSessionWindowStations”. Window Stations, however, are named objects, and so are visible in tools such as WinObj (running elevated):
Window stations in session 0
The Window Stations in session 0 are at \Windows\WindowStations The Window Stations in session x are at \Sessions\x\Windows\WindowStations
The OpenWindowStation API only accepts a “local” name, under the callers session. The native NtUserOpenWindowStation API (from Win32u.dll) is more flexible, accepting a full object name:
Here is an example that opens the “msswindowstation” Window Station:
#include <Windows.h>
#include <winternl.h>
#pragma comment(lib, "ntdll")
HWINSTA NTAPI _NtUserOpenWindowStation(_In_ POBJECT_ATTRIBUTES attr, _In_ ACCESS_MASK access);
int main() {
// force Win32u.DLL to load
::LoadLibrary(L"user32");
auto NtUserOpenWindowStation = (decltype(_NtUserOpenWindowStation)*)
::GetProcAddress(::GetModuleHandle(L"win32u"), "NtUserOpenWindowStation");
UNICODE_STRING winStaName;
RtlInitUnicodeString(&winStaName, L"\\Windows\\WindowStations\\msswindowstation");
OBJECT_ATTRIBUTES winStaAttr;
InitializeObjectAttributes(&winStaAttr, &winStaName, 0, nullptr, nullptr);
auto hWinSta = NtUserOpenWindowStation(&winStaAttr, READ_CONTROL);
if (hWinSta) {
// do something with hWinSta
::CloseWindowStation(hWinSta);
}
You may or may not have enough power to open a handle with the required access – depending on the Window Station in question. Those in session 0 are hardly accessible from non-session 0 processes, even with the SYSTEM account. You can examine their security descriptor with the kernel debugger (as other tools will return access denied):
You can become SYSTEM to help with access by using PsExec from Sysinternals to launch a command window (or whatever) as SYSTEM but still run in the interactive session:
psexec -s -i -d cmd.exe
If all else fails, you may need to use the “Take Ownership” privilege to make yourself the owner of the object and change its DACL to allow yourself full access. Apparently, even that won’t work, as getting something from a Window Station in another session seems to be blocked (see replies in Twitter thread). READ_CONTROL is available to get some basic info.
Here is a screenshot of Object Explorer running under SYSTEM that shows some details of the “msswindowstation” Window Station:
Guess which processes hold handles to this hidden Windows Station?
Once you are able to get a Window Station handle, you may be able to go one step deeper by enumerating desktops, if you managed to get at least WINSTA_ENUMDESKTOPS access mask:
::EnumDesktops(hWinSta, [](auto deskname, auto param) -> BOOL {
printf(" Desktop: %ws\n", deskname);
auto h = (HWINSTA)param;
return TRUE;
}, (LPARAM)hWinSta);
Going one level deeper, you can enumerate the top-level windows in each desktop (if any). For that you will need to connect the process to the Window Station of interest and then call EnumDesktopWindows:
The desktops in the Window Station “Service-0x0-45193$” above don’t seem to have top-level visible windows.
You can also access the clipboard and atom table of a given Windows Station, if you have a powerful enough handle. I’ll leave that as an exercise as well.
Finally, what about session enumeration? That’s the easy part – no need to call NtOpenSession with Session objects that can be found in the “\KernelObjects” directory in the Object Manager’s namespace – the WTS family of functions can be used. Specifically, WTSEnumerateSessionsExcan provide some important properties of a session:
What about creating a process to use a different Window Station and desktop? One member of the STARTUPINFO structure passed to CreateProcess (lpDesktop) allows setting a desktop name and an optional Windows Station name separated by a backslash (e.g. “MyWinSta\MyDesktop”).
There is more to Window Stations and Desktops that meets the eye… this should give interested readers a head start in doing further research.
Doing any kind of research into the Windows kernel requires working with a kernel debugger, mostly WinDbg (or WinDbg Preview). There are at least 3 “levels” of debugging the kernel.
Level 1: Local Kernel Debugging
The first is using a local kernel debugger, which means configuring WinDbg to look at the kernel of the local machine. This can be configured by running the following command in an elevated command window, and restarting the system:
bcdedit -debug on
You must disable Secure Boot (if enabled) for this command to work, as Secure Boot protects against putting the machine in local kernel debugging mode. Once the system is restarted, WinDbg launched elevated, select File/Kernel Debug and go with the “Local” option (WinDbg Preview shown):
If all goes well, you’ll see the “lkd>” prompt appearing, confirming you’re in local kernel debugging mode.
What can you in this mode? You can look at anything in kernel and user space, such as listing the currently existing processes (!process 0 0), or examining any memory location in kernel or user space. You can even change kernel memory if you so desire, but be careful, any “bad” change may crash your system.
The downside of local kernel debugging is that the system is a moving target, things change while you’re typing commands, so you don’t want to look at things that change quickly. Additionally, you cannot set any breakpoint; you cannot view any CPU registers, since these are changing constantly, and are on a CPU-basis anyway.
The upside of local kernel debugging is convenience – setting it up is very easy, and you can still get a lot of information with this mode.
Level 2: Remote Debugging of a Virtual Machine
The next level is a full kernel debugging experience of a virtual machine, which can be running locally on your host machine, or perhaps on another host somewhere. Setting this up is more involved. First, the target VM must be set up to allow kernel debugging and set the “interface” to the host debugger. Windows supports several interfaces, but for a VM the best to use is network (supported on Windows 8 and later).
First, go to the VM and ping the host to find out its IP address. Then type the following:
bcdedit /dbgsettings net hostip:172.17.32.1 port:55000 key:1.2.3.4
Replace the host IP with the correct address, and select an unused port on the host. The key can be left out, in which case the command will generate something for you. Since that key is needed on the host side, it’s easier to select something simple. If the target VM is not local, you might prefer to let the command generate a random key and use that.
Next, launch WinDbg elevated on the host, and attach to the kernel using the “Net” option, specifying the correct port and key:
Restart the target, and it should connect early in its boot process:
Microsoft (R) Windows Debugger Version 10.0.25200.1003 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Using NET for debugging
Opened WinSock 2.0
Waiting to reconnect...
Connected to target 172.29.184.23 on port 55000 on local IP 172.29.176.1.
You can get the target MAC address by running .kdtargetmac command.
Connected to Windows 10 25309 x64 target at (Tue Mar 7 11:38:18.626 2023 (UTC - 5:00)), ptr64 TRUE
Kernel Debugger connection established. (Initial Breakpoint requested)
************* Path validation summary **************
Response Time (ms) Location
Deferred SRV*d:\Symbols*https://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*d:\Symbols*https://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 10 Kernel Version 25309 MP (1 procs) Free x64
Edition build lab: 25309.1000.amd64fre.rs_prerelease.230224-1334
Machine Name:
Kernel base = 0xfffff801`38600000 PsLoadedModuleList = 0xfffff801`39413d70
System Uptime: 0 days 0:00:00.382
nt!DebugService2+0x5:
fffff801`38a18655 cc int 3
Enter the g command to let the system continue. The prompt is “kd>” with the current CPU number on the left. You can break at any point into the target by clicking the “Break” toolbar button in the debugger. Then you can set up breakpoints, for whatever you’re researching. For example:
In this “level” of debugging you have full control of the system. When in a breakpoint, nothing is moving. You can view register values, call stacks, etc., without anything changing “under your feet”. This seems perfect, so do we really need another level?
Some aspects of a typical kernel might not show up when debugging a VM. For example, looking at the list of interrupt service routines (ISRs) with the !idt command on my Hyper-V VM shows something like the following (truncated):
Some things are missing, such as the keyboard interrupt handler. This is due to certain things handled “internally” as the VM is “enlightened”, meaning it “knows” it’s a VM. Normally, it’s a good thing – you get nice support for copy/paste between the VM and the host, seamless mouse and keyboard interaction, etc. But it does mean it’s not the same as another physical machine.
Level 3: Remote debugging of a physical machine
In this final level, you’re debugging a physical machine, which provides the most “authentic” experience. Setting this up is the trickiest. Full description of how to set it up is described in the debugger documentation. In general, it’s similar to the previous case, but network debugging might not work for you depending on the network card type your target and host machines have.
If network debugging is not supported because of the limited list of network cards supported, your best bet is USB debugging using a dedicated USB cable that you must purchase. The instructions to set up USB debugging are provided in the docs, but it may require some trial and error to locate the USB ports that support debugging (not all do). Once you have that set up, you’ll use the “USB” tab in the kernel attachment dialog on the host. Once connected, you can set breakpoints in ISRs that may not exist on a VM:
Today I’m happy to announce the next COM Programming class to be held in February 2023. The syllabus for the 3 day class can be found here. The course will be delivered in 6 half-days (4 hours each).
Dates: February (7, 8, 9, 14, 15, 16). Times: 11am to 3pm EST (8am to 12pm PST) (4pm to 8pm UT) Cost: 750USD (if paid by an individual), 1400 USD (if paid by a company).
Half days should make it comfortable enough even if you’re not in an ideal time zone.
The class will be conducted remotely using Microsoft Teams.
What you need to know before the class: You should be comfortable using Windows on a Power User level. Concepts such as processes, threads, DLLs, and virtual memory should be understood fairly well. You should have experience writing code in C and some C++. You don’t have to be an expert, but you must know C and basic C++ to get the most out of this class. In case you have doubts, talk to me.
Participants in my Windows Internals and Windows System Programming classes have the required knowledge for the class.
We’ll start by looking at why COM was created in the first place, and then build clients and servers, digging into various mechanisms COM provides. See the syllabus for more details.
Previous students in my classes get 10% off. Multiple participants from the same company get a discount (email me for the details).
To register, send an email to [email protected] with the title “COM Programming Training”, and write the name(s), email(s) and time zone(s) of the participants.
I’m happy to announce the next 5-day virtual Windows Kernel Programming class to be held in October. The syllabus for the class can be found here. A notable addition to the class is an introduction to the Kernel Mode Driver Framework (KMDF).
Dates and Times (all in October 2022), times based on London: 11 (full day): 4pm to 12am 12 (full day): 4pm to 12am 13 (half day): 4pm to 8pm 17 (half day): 4pm to 8pm 18 (full day): 4pm to 12am 19 (half day): 4pm to 8pm 20 (half day): 4pm to 8pm
The class will be recorded and provided to the participants.
Cost: 900 USD if paid by an individual 1700 USD if paid by a company Previous participants of my classes get 10% off. Multiple participants from the same company get a discount as well (talk to me).
Registration To register, send email to [email protected] and provide the name(s) and email(s) of the participant(s), the company name (if any), and your time zone (for my information, although I cannot change course times).
Feel free to contact me for any questions or comments via email, twitter (@zodiacon) or Linkedin.
I created this application as a little challenge and some practice at manually obfuscating an application at the assembly level.
I wrote the application in IA32 assembly and then manually obfuscated it using a couple of different methods.
Here I will show how to solve the challenge in 2 different ways.
Lastly I will show how the obfuscation could have been done better so that it would have been a lot more difficult to solve this using a simple static disassembly.
The Challenge
We are given the static disassembly below of a 32bit linux application which says whether or not the author is going to some event:
root@dev:~# nasm -felf32 -o going-or-not-obf-test1 going-or-not-obf-test1.nasm going-or-not-obf-test1.nasm:16: error: comma, colon or end of line expectedgoing-or-not-obf-test1.nasm:29: error: comma, colon or end of line expectedgoing-or-not-obf-test1.nasm:47: error: comma, colon or end of line expectedgoing-or-not-obf-test1.nasm:49: error: comma, colon or end of line expectedgoing-or-not-obf-test1.nasm:50: error: comma, colon or end of line expectedgoing-or-not-obf-test1.nasm:51: error: comma, colon or end of line expectedgoing-or-not-obf-test1.nasm:59: error: comma, colon or end of line expectedgoing-or-not-obf-test1.nasm:63: error: comma, colon or end of line expectedgoing-or-not-obf-test1.nasm:67: error: comma, colon or end of line expectedgoing-or-not-obf-test1.nasm:75: error: comma, colon or end of line expectedgoing-or-not-obf-test1.nasm:78: error: comma, colon or end of line expected
You can see that its all lines that have [SIZE] PTR, we will remove any DWORD PTR and BYTE PTR and for the lines that had BYTE put that before the first operand, so they end up like this:
1 2 3 4 5 6 7 8 9101112
root@dev:~# for i in1629474950515963677578;do cat -n going-or-not-obf-test2.nasm | grep "^[ ]*$i";done 16 mov BYTE al, [esi] 29 jmp [esp] 47 and ebp, gs:[edx] 49 sub ebp, [edx] 50 sub fs:0x654e0d64,esp 51 and ebp, [edx] 59 mov BYTE [esp],cl 63 cmp BYTE [edx],al 67 xor BYTE [edx],bl 75 mov BYTE cl,[esi+edx*1] 78 mov BYTE [esp],cl
Now we try to assemble it again:
123
root@dev:~# nasm -felf32 -o going-or-not-obf-test2 going-or-not-obf-test2.nasm going-or-not-obf-test2.nasm:47: error: invalid combination of opcode and operandsgoing-or-not-obf-test2.nasm:50: error: invalid combination of opcode and operands
So there is still a problem with 2 lines, it looks as if these instructions are invalid, this could possibly be data, what we shall do is replace these 2 instructions with the raw opcodes from the disassembly, so our application ends up like this:
root@dev:~# gdb -q ./going-or-not-obf-test3Reading symbols from /root/going-or-not-obf-test3...(no debugging symbols found)...done.(gdb)rStarting program: /root/going-or-not-obf-test3 Program received signal SIGSEGV, Segmentation fault.0x080480b6 in _start ()(gdb)x/i $eip=> 0x80480b6 <_start+86>: add BYTE PTR [eax],al(gdb)print/x $eax$1= 0x0(gdb)disassemble Dump of assembler code for function _start: 0x08048060 <+0>: mov edx,eax 0x08048062 <+2>: mov edi,0x25 0x08048067 <+7>: jmp 0x80480b6 <_start+86> 0x0804806c <+12>: mov bl,0x32 0x0804806e <+14>: pop esi 0x0804806f <+15>: xor eax,eax 0x08048071 <+17>: je 0x80480dc <_start+124> 0x08048077 <+23>: mov bh,0x6a 0x08048079 <+25>: call 0x804808e <_start+46> 0x0804807e <+30>: mov cl,0x4 0x08048080 <+32>: mov al,BYTE PTR [esi] 0x08048082 <+34>: sub esp,ecx 0x08048084 <+36>: inc ecx 0x08048085 <+37>: xor al,cl 0x08048087 <+39>: xor ecx,ecx 0x08048089 <+41>: cmp eax,0x4 0x0804808c <+44>: je 0x8048099 <_start+57> 0x08048092 <+50>: lea ecx,[ebp-0xf] 0x08048095 <+53>: mov dl,0x10 0x08048097 <+55>: jmp 0x8048097 <_start+55> 0x0804809c <+60>: xor ebx,ebx 0x0804809e <+62>: xor ecx,ecx 0x080480a0 <+64>: mov edx,ecx 0x080480a2 <+66>: jmp DWORD PTR [esp] 0x080480a5 <+69>: jmp 0x804809e <_start+62> 0x080480aa <+74>: lea ecx,[ebp-0x1b] 0x080480ad <+77>: mov dl,0xc 0x080480af <+79>: xor eax,eax 0x080480b1 <+81>: mov al,0x8 0x080480b3 <+83>: mov ebx,0x4 0x080480b8 <+88>: sub eax,ebx 0x080480ba <+90>: sub ebx,eax 0x080480bc <+92>: inc ebx 0x080480bd <+93>: int 0x80 0x080480bf <+95>: xor eax,eax 0x080480c1 <+97>: xor ebx,ebx 0x080480c3 <+99>: inc al 0x080480c5 <+101>: int 0x80---Type <return> to continue, or q <return> to quit--- 0x080480c7 <+103>: call 0x8048069 <_start+9> 0x080480cc <+108>: in eax,dx 0x080480cd <+109>: dec esi 0x080480ce <+110>: and ebp,DWORD PTR gs:[edx] 0x080480d1 <+113>: sub eax,0x3064232b 0x080480d6 <+118>: sub ebp,DWORD PTR [edx] 0x080480d8 <+120>: sub DWORD PTR fs:0x654e0d64,esp 0x080480df <+127>: and ebp,DWORD PTR [edx] 0x080480e1 <+129>: sub eax,0x2964232b 0x080480e6 <+134>: and eax,0x89ee0d64 0x080480eb <+139>: mov ebp,eax 0x080480ed <+141>: mov al,0xc9 0x080480ef <+143>: add eax,edi 0x080480f1 <+145>: jmp 0x8048101 <_start+161> 0x080480f6 <+150>: lea edx,[ebp+0x0] 0x080480f9 <+153>: mov BYTE PTR [esp],cl 0x080480fc <+156>: dec esp 0x080480fd <+157>: push 0x80480e9 0x08048102 <+162>: test edx,edx 0x08048104 <+164>: cmp BYTE PTR [edx],al 0x08048106 <+166>: je 0x8048070 <_start+16> 0x0804810c <+172>: mov ebx,edi 0x0804810e <+174>: add ebx,0x1f 0x08048111 <+177>: xor BYTE PTR [edx],bl 0x08048113 <+179>: dec edx 0x08048114 <+180>: ret 0x08048115 <+181>: xor ebp,ebp 0x08048117 <+183>: xor ecx,ecx 0x08048119 <+185>: xor edx,edx 0x0804811b <+187>: inc edx 0x0804811c <+188>: lea ebp,[esp+ecx*1] 0x0804811f <+191>: mov cl,BYTE PTR [esi+edx*1] 0x08048122 <+194>: cmp cl,al 0x08048124 <+196>: je 0x80480e2 <_start+130> 0x0804812a <+202>: mov BYTE PTR [esp],cl 0x0804812d <+205>: sub esp,0x1 0x08048130 <+208>: inc edx 0x08048131 <+209>: mov esp,esp 0x08048133 <+211>: cmp ecx,0x0---Type <return> to continue, or q <return> to quit--- 0x08048136 <+214>: jg 0x804810b <_start+171> 0x0804813c <+220>: mov ebp,ebp 0x0804813e <+222>: ret End of assembler dump.
So it looks as if we've landed in the middle of an instruction.
Near the start of the application (on line 16 above), it jumps it a certain memory address which is the middle of an instruction. The resulting instruction, as seen on line 9, tries to move a value to the address pointed to by the EAX register.
On line 11 you can see that the value in EAX is 0, which is what caused the segfault, 0 is an invalid memory address.
The reason for this is because the original application jumped to static memory addresses, in the application the memory addresses are different so this will need to be fixed for the application to work.
What we need to do is replace any fixed memory addresses with labels. We can find where in the application the memory addresses are meant to go by looking at the original disassembly.
Once we have done this the resulting application is as follows:
There are a couple of values here (on lines 55, 59 and 60) which look like memory addresses but they aren't valid memory addresses in the original disassembly so they could just be normal values or, as its in the same section as the invalid instructions, part of some data.
With this done we can test this application:
1234
root@dev:~# nasm -felf32 -o going-or-not-obf-test4.o going-or-not-obf-test4.nasmroot@dev:~# ld -o going-or-not-obf-test4 going-or-not-obf-test4.oroot@dev:~# ./going-or-not-obf-test4I am not going!
So we have our answer, the author is not going :-)
Method 2: The Hard Way
Here we will attempt to understand the application and figure out what the application does without building and running it.
Although you would have needed some understanding of IA32 to do the previous method, obviously you will need a better understanding of it to do this.
The first step would be what we have already done. Well, there would be no need for the ability to assemble the application, or even have a valid nasm file but we would need to replace any known addresses with labels because this will make the disassembly significantly easier to read.
For this will we just use the nasm file above (going-or-not-obf-test4.nasm), just because it will make this post a little shorter :-)
What we do now is follow the control flow of the application and simplfy it as we go by replacing more complex sequencies with less complex 1's or even only 1 instruction in some cases and removing any dead instructions (instructions which have no effect on the application at all) altogether.
This process is manual deobfuscation and can be applied to small sections of applications instead of just full applications like the last method.
Let's start with the first instruction mov edx,eax, this looks like it is a junk line (or dead code) mainly because this is the first instruction of the application, if this was just a code segment instead of a full application this code would be more likely to be meaningful.
The second instruction mov edi,0x25, is also very difficult to quickly determine its usefulness to the application, what we need to do here is take note of the value inside the EDI register.
The next 4 instructions do something interesting, if you follow the control flow of the application and line the instructions sequentially you get:
123456
jmpOneOne:callTwoTwo:movbl,0x32popesi
So the 3rd instruction (on line 5) is not related here, and is similar to the previous mov instruction, just make a note that bl contains 0x32.
The other 3 instructions are using a technique used in some shellcode to get the an address in memory when the code might start at a different point in memory.
Its called the JMP-CALL-POP technique and gets the address of the address immediately following the call instruction into the register used in the pop instruction.
Knowing this we can replace the entire code above with:
12
movbl,0x32movesi,One
Let's look at the next 4 instructions:
12345
xoreax,eaxjeThreeThree:moval,0xc9addeax,edi
So here, on line 5, we use the EDI register, we zero EAX, set it to 0xc9 (201), adds it to EDI (0x25 or 37) and stores the result in EAX, this series of instructions are what is called constant unfolding where a series of instructions are done to work out the actual required value instead of just assigning the value to begin with.
We could use the opposite, a common compiler optimization constant folding, to decrease the complexity of this code, so these 4 instructions could be replaced by:
1
moveax,0xee
The next 5 instructions are:
123456
jmpFourFour:xorebp,ebpxorecx,ecxxoredx,edxincedx
This set of instructions just sets EBP and ECX to 0 and EDX to 1. Now its obvious that the instrction at the beginning was dead code because EDX hasn't been used at all and now it has been overwritten.
We can rewrite the application so far in a much more simplfied way:
As you can see, this is much easier to read than the previous code that was jumping about all over the place.
I kept the assignment to EDI (on line 2) there because, although I've removed the need for it in assigning the value of EAX (on line 5), it still might be used in the future.
Also, the assignment to bl (on line 3) still might not be needed but we shall keep it there just incase.
So this first moves a byte at ESI + EDX * 1, which is basically just ESI + EDX, into the cl register. We know at this point the value inside EDX is 1 and that ESI points to some address in the middle of the application, so our loop will start getting data 1 byte after that address.
This byte is them compared with al, which we know is 0xee, and if they are the same execution will jump to Six.
Providing the jump to Six isn't taken, the byte is moved to the top of the stack (which ESP points to), ESP is adjusted accordingly, EDX is incremented by 1 and the loop is rerun.
The mov instruction on line 8 doesn't do anything, dead code which can be removed.
Now we can find all of the data that is being worked on here:
The starting address of this data is 80480bc in the original disassembly, which is 1 byte after the address of the instruction following the call instruction in the jmp-call-pop routine at the start of the application.
It ends with the ee value because this is the point at which the jump to Six is taken.
Also, notice that nowhere here is a 0x0 (or 00) byte, this means that the jg (jump if greater than) instruction on line 10 will always be taken, every byte there is above 0 so the 2 instructions after are dead code and can be removed from the analysis and the jg can be replaced with a jmp.
It is clear that this data, which is sitting in the middle of the application, is being put on the stack for some reason, the lea instruction right before the loop just saved the address pointing to the beginning of the new location of the data on the stack into the EBP register.
We could try to figure out how meaningful this data is now but it would be best to have a look to see what the application does with it first.
Now let's take the jump to Six:
123
leaedx,[ebp+0x0]movBYTE[esp],cldecesp
First it loads the address of the data on the stack, currently in EBP, into EDX.
cl, which is currently 0xee, is put onto the stack and ESP is adjusted accordingly.
This is a very unusual loop, you will only see this type of code when reversing obfuscated code.
It started by pushing its own address to the stack, this allows the ret on line 10 to return to Seven.
The test instruction on line 3 is dead code because all test does is set EFLAGS, but they are immediately overwritten by the cmp instruction that follows.
Lines 4 and 5 again test the value of a byte in the data, this time pointed to by EDX, against 0xee and jump's to Eight when its reached.
The next 2 instructions, lines 6 and 7, move the value from EDI into EBX and add's 0x1f to it. We already know that 0x25 is currently in EDI, so EBX = 0x25 + 0x1f or EBX = 0x44.
The byte in the data is then xor'd with bl (or 0x44) and EDX is decremented.
Clearly this is a simply xor encoding of the data, I wrote a python script a while ago to xor a number of bytes with 1 byte and output both the resulting bytes as ascii characters, and the same but with the characters reversed (due to little endian architectures), here is the script:
1 2 3 4 5 6 7 8 91011121314151617
#!/usr/bin/env pythonimportsysstring=sys.argv[1]xor=sys.argv[2]decoded=""forcinstring:decoded+=chr(ord(c)^ord(xor))print"String as is:"printdecodedprint"\n\nString reversed:"printdecoded[::-1]
This script is very simple, 1 thing to bare in mind though is that, because we are dealing with data outside of the printable ascii range (0x20 - 0x7e), we can just type the characters on the command line.
So we run the script like this:
1 2 3 4 5 6 7 8 910
root@dev:~# python xor-and-ascii.py $(python -c 'print "\x4e\x65\x23\x2a\x2d\x2b\x23\x64\x30\x2b\x2a\x64\x29\x25\x64\x0d\x4e\x65\x23\x2a\x2d\x2b\x23\x64\x29\x25\x64\x0d"')$(python -c 'print "\x44"')String as is:!gniog ton ma I!gniog ma IString reversed:I am going!I am not going!
So now we know what that data is in the middle of the application, clearly it was done like this to confuse but we have reversed enough of the application now to figure out what this is.
With this is mind, we no longer need those 2 loops, or any of the code aimed at moving and decoding the data, we can simply put it in as is.
Let's review our rewritten application:
1 2 3 4 5 6 7 8 9101112
_start:movedi,0x25movesi,Onemovebp,not+0xfmovebx,0x44movecx,0xeemoveax,ecxmovedx,amOne:db0xedam:db"I am going!",0xanot:db"I am not going!",0xa
I have obviously removed most of the code because it simply isn't needed now, I've made sure that EBP still points to the end of the data and EDX to the beginning just incase there is some reason for this, but most of the code so far was devoted to decoding the data which is no longer needed.
Firstly there is an assignment to bh (the second 8 bits of the EBX register) but then, on line 5, the whole EBX register is cleared using xor so line 2 is dead code.
The call instruction on line 3 and the jmp instruction on line 8 seem to be used just to confuse the reverser, there is no reason for this, but bare in mind that this would have stuck 4 bytes on the stack, next to the decoded data, which hasn't been cleaned up (this could effect the application in some way).
The rest of this code just zero's out EBX, ECX and EDX.
Lines 1 and 3 fix the value of ESP after the call, jmp sequence earlier.
The rest xor's 0x5 with the byte at One and compares the result with 0x4. We can test this out in python, we know the byte at One is 0xed, so:
12345678
root@dev:~# pythonPython 2.7.3 (default, Mar 14 2014, 11:57:14) [GCC 4.7.2] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> a = "\xed">>> b = "\x05">>> hex(ord(a) ^ ord(b))'0xe8'
This isn't equal to 0x4 so the jump on line 8 will not be taken.
The next instruction lea ecx,[ebp-0xf] loads EBP - 16 into ECX, ECX will now point to somewhere in the middle of the data (it will actually point 16 characters from the end, which is the start of the string I am not going!).
We can probably guess at what this is going to do from here but let's finish the analysis.
0x10 is then loaded into EDX and then 2 unconditional jumps are taken:
123
jmpTwelveTwelve:jmpTen
The only reason for these jumps is to confuse the reverser, we can just ignore them.
The next 7 lines is a very important part of the application:
So lines 1-4 set EAX to 0x4, lines 5 and 6 set EBX to 0x1 and then the interrupt *0x80 is initiated.
Interrupt 0x80 is a special interrupt which initiates a system call, the system call number has to be stored in EAX, which is 0x4 at this moment in time.
This makes sense, the prototype for this syscall is:
1
ssize_twrite(intfd,constvoid*buf,size_tcount);
Each of the arguments go in EBX, ECX and EDX. So to write to stdout, EBX should be 1 which it is.
ECX should point to the string, which it currently points to I am not going!, and EDX should contain the number of characters to print which it does.
The last 4 instructions just run another syscall, exit, you can check this yourself if you wish:
1234
xoreax,eaxxorebx,ebxincalint0x80
Obviously we can now wrtie this in a much simpler way, but there is no need, we know exactly what this application does and how it does it.
Improving Obfuscation
As I mentioned earlier, the obfuscation could have been done better to make the reversing process harder. I actually purposefully made the obfuscation weaker than I could have to make the challenge easier.
Inserting more junk data inbetween some instructions could make the static disassembly significantly more difficult to read and understand.
I have to actually add a byte (0x89) at the end of the data section because the next few instructions were being obfuscated in a way that made them unreadable:
The disassembly shown here has had the last byte of the data removed and is the last line of the data section; and a few lines after.
As you can see the byte following the data section has been moved to the data section and as a result the next few instructions have been incorrectly disassembled.
This method can be implemented throughout the whole application, making most of the instructions disassemble incorrectly.
Constant unfolding could be improved here, for instance:
They both do the same thing but the second is a little harder to read, you could obviously keep extending this by implementing more and more complex algorithms to work out your required value.
This can also be applied to references to memory addresses, for instance, if you want to jump to a certain memory address, do some maths to work out the memory address before jumping there.
More advanced instructions could be used like imul, idiv, cmpsb, rol, stosb, rep, movsx, fadd, fcom... The list goes on...
The MMX and other unusual registers could have been taken advantage of.
Also, the key to decrypt the data could have been a command line argument or somehow retreived from outside of the application, this way it would have been extremely difficult decode the data.
Conclusion
There are sometimes easier ways to get a result other than reversing the whole application, maybe just understanding a few bits might be enough.
Although there are ways to make the reversers job more difficult, its never possible to make it impossible to reverse, providing the reverser is able to run the application (if the CPU can see the instructions, then so can the reverser).
A good knowledge of assembly is needed to do any type of indepth reverse engineering.
Further Reading
Reversing: Secrets of Reverse Engineering by Eldad Eilam
I have started the Windows kernel hacking section with a simple explaination of the setup and a quick analysis of the crackme, that we analysed here, using the kd.exe kernel debugger.
I chose to do this instead of any actual Windows kernel stuff because its a steep learning experience learning how to use KD so its probably best to look at something you have already seen.
Setting Up The Environment
For this post I will be using a total of 4 machines, 3 virtual machines using VMware Player (you probably could use Virtualbox for this also though) hosted on a reasonably powerful machine and a laptop.
You can however do all of this with just 1 physical machine, hosting 1 virtual machine and I will explain the differences in the setup afterwards but I'll first explain the setup I am using.
Here is a visual representation of the network:
So I have 3 virtual machines on my machine running VMware Player:
1 Kali Linux, 1 Windows XP Professional and 1 Windows 7 Home Edition. All 3 of these are 32bit, although it doesn't matter but to follow along you would probably want the debuggee (the Windows 7 machine in my setup) to be 32bit. In my 2 machine setup described below the host (and debugger) is a Windows 7 64bit machine.
The Kali machine has 2 network interfaces, 1 setup in Bridged mode (so that I can SSH directly to it):
And the other setup in Host-only mode (So that it has access to the other 2 machines):
The Windows XP machine has 1 network interface setup in Host-only mode:
And the same for the Windows 7 machine:
The Windows XP and Windows 7 machines are also connected via a virtual serial cable, this is for the debugger connection.
The Windows XP machine will be the client (or the debugger):
And the Windows 7 machine will be the server (or the debuggee):
The Windows 7 machine needs both Visual Studio Express 2013 for Windows Desktop and the Windows Driver Kit (WDK) installed on it. You can get them both here.
The Windows XP machine needs Microsoft Windows SDK for Windows 7 installed, which you can get here. To install this you need to install the full version of Microsoft .NET Framework 4, which you can get here (Bare in mind that you might need an internet connection while you install these so just change the network adaptor configuration to NAT and then once it is installed change it back to Host-only again).
If the debugger is a Windows 7 machine then you will need to install the same software as on the debuggee.
Once these are installed, its best to add the path to the kd.exe application to the PATH variable.
You do this by going in to the properties of My Computer and, on Windows 7 going to Advanced system settings->Environment Variables... or on Windows XP going to Advanced->Environment Variables... and scroll down the Path and click Edit.
The path on Windows 7 should be something like C:\Program Files\Windows Kits\8.1\Debuggers\x86 and on Windows XP C:\Program Files\Debugging Tools for Windows (x86).
For remote administration I've installed TightVNC on both of the Windows machines.
I set it up with access through a Kali machine so that I can setup SSH tunnels and get VNC access to the Windows machines without giving them access to the outside network.
After TightVNC is up and running on your Windows machines, you can setup the SSH tunnels like this (For this explaination we'll imagine that the Windows XP machine is on the VMware virutal network with an IP of 172.16.188.130, the Windows 7 machine is on 172.16.188.131 and that our Kali machine is also on this network):
Now if you VNC to 127.0.0.1 you will have access to the Windows XP machine and to 127.0.0.1:1 you will have access to the Windows 7 machine.
1 VM Setup
You can also setup this up with 2 machines, the VMware host (running Windows, which will be the debugger) and the VMware guest (also running Windows, which will be the debuggee).
The serial port configuration for the debuggee in VMware in this setup should look like this:
Notice the different file path and name for Windows, the other end should be set to The other end is an application and Yeild CPU on poll should be checked.
The only other thing that is different is the command you will use to launch KD on the debugger (we haven't got to that but it is shown below for my 4 machine setup), you should instead use kd -k com:port=\\.\pipe\com_1,pipe.
Using KD
On Windows 7 (the debuggee) you will need to tell it to lanuch the debugger on boot, for this you need to run an Administrator command prompt and:
The DEBUGPORT:2 option here is the port number of the COM port that you are going to use, for me it was COM2 hence the number 2.
Now we launch the kernel debugger on the Windows XP machine (this is the command that is different on the 2 machine setup):
1234567
C:\Documents and Settings\User>kd -k com:port=1,baud=115200Microsoft (R) Windows Debugger Version 6.12.0002.633 X86Copyright (c) Microsoft Corporation. All rights reserved.Opened \\.\com1Waiting to reconnect...
Again the port=1 option here is the COM port that you are going to be using, I will be using COM1 on this machine hence the 1.
Then reboot the Windows 7 machine and watch the KD terminal on the Windows XP machine:
1 2 3 4 5 6 7 8 910111213
Connected to Windows 7 7601 x86 compatible target at (Fri Sep 26 14:43:59.625 2014 (UTC + 1:00)), ptr64 FALSEKernel Debugger connection established.Symbol search path is: SRV*C:\websymbols*http://msdl.microsoft.com/download/symbolsExecutable search path is:Windows 7 Kernel Version 7601 (Service Pack 1) MP (1 procs) Free x86 compatibleProduct: WinNt, suite: TerminalServer SingleUserTS PersonalBuilt by: 7601.18409.x86fre.win7sp1_gdr.140303-2144Machine Name:Kernel base = 0x82814000 PsLoadedModuleList = 0x8295d5b0Debug session time: Sun Dec 29 22:42:59.976 1985 (UTC + 1:00)System Uptime: 0 days 0:02:14.490
Now run the crackme application on the debuggee (Windows 7):
Go back to the Windows XP machine and in the debugger terminal window press Control + C:
1 2 3 4 5 6 7 8 9101112131415161718
Break instruction exception - code 80000003 (first chance)******************************************************************************** ** You are seeing this message because you pressed either ** CTRL+C (if you run kd.exe) or, ** CTRL+BREAK (if you run WinDBG), ** on your debugger machine's keyboard. ** ** THIS IS NOT A BUG OR A SYSTEM CRASH ** ** If you did not intend to break into the debugger, press the "g" key, then ** press the "Enter" key now. This message might immediately reappear. If it ** does, press "g" and "Enter" again. ** ********************************************************************************nt!RtlpBreakWithStatusInstruction:8288e7b8 cc int 3kd>
Now we have broken into the kernel, this means that anything we do will be in the context of the kernel, we can see this in the debugger:
On line 1 I run the .process command without any parameters and it tells us the process we are currently in (844bdae8 is the EPROCESS number).
On line 3 I run the !process extension with 0 0 as its arguments, this lists all of the running processes and some details about them, as you can see from lines 5-7, EPROCESS 844bdae8 is the System process, or the kernel.
What we want to do is change the context to our crackme application, which you can see from lines 141-143 has the EPROCESS of 85abfd40:
1 2 3 4 5 6 7 8 91011
kd> .process /i /r /p 85abfd40You need to continue execution (press 'g' <enter>) for the contextto be switched. When the debugger breaks in again, you will be inthe new process context.kd> gBreak instruction exception - code 80000003 (first chance)nt!RtlpBreakWithStatusInstruction:828c97b8 cc int 3kd> .processImplicit process is now 85abfd40kd>
On line 1 I use the .process command to change the context to our crackme application but before the context can be changed execution needs to be resumed (which is done on line 5).
Now we can set a breakpoint anywhere in the crackme's virtual memory address space, we want to break with them calls to GetDlgItemTextA that were responsible for getting the text in the textboxes of the application (If you are unsure about what I am talking about, please go back and review the previous post):
1234
kd> bp USER32!GetDlgItemTextAkd> bl 0 e 76213d14 0001 (0001) user32!GetDlgItemTextAkd>
Now that the breakpoint is set we can resume execution, wait for it to be hit and inspect the memory.
Remember that the prototype for GetDlgItemText is:
kd> gBreakpoint 0 hituser32!GetDlgItemTextA:001b:76213d14 8bff mov edi,edikd> dd esp L40012fb6c 0040127f 0002014e 000003e9 0012fc40kd> da 12fc400012fc40 "Enter your name..."kd>
On line 5 I use the dd command to display 4 double words on the top of the stack. The first dword will be the return address (as you will see in a minute), then we have the first 3 arguments.
The 3rd argument is the address where the buffer for the string is, on line 7 I use the da command to display the ascii value at that address.
Keep in mind that this is the start of the function so the value hasn't been fetched yet, we can see the returned value by tracing through until we are in the calling function using the ug command and checking again:
As you can see the value is the same (because we haven't changed the text in the textbox), you can also see the address which it returned back to after executing GetDlgItemTextA was 0040127f, which was the top value on the stack.
Lastly let's resume and make sure it does the same with the other textbox:
1 2 3 4 5 6 7 8 9101112
kd> gBreakpoint 0 hituser32!GetDlgItemTextA:001b:76213d14 8bff mov edi,edikd> dd esp L40012fb6c 00401291 0002014e 000003ea 0012fc00kd> da 12fc000012fc00 "Enter your serial..."kd> gu001b:00401291 8d44240c lea eax,[esp+0Ch]kd> da 12fc000012fc00 "Enter your serial..."
Conclusion
This was only a simple tutorial to get the environment set up and get a basic grasp of kd.exe and some of its commands.
This was by no means an exhaustive list of commands and extensions, the debugger comes with many and has very good documentation.
Hopefully you now have a better understanding of how to debug using kd.exe and you now have the environment to do it in.
Further Reading
The Debugging and Automation chapter in Practical Reverse Engineering by Bruce Dang, Alexandre Gazet and Elias Bachaalany.
Also the kd.exe documentation that ships with the WDK or SDK.
pBefore I go into some of the protections that are commonly in place, I thought it would be best to show how to detect these 2 basic vulnerabilities using a href="https://en.wikipedia.org/wiki/Reverse_engineering" target="_blank"reverse engineering/a (as opposed to randomly a href="https://en.wikipedia.org/wiki/Fuzz_testing" target="_blank"fuzzing/a inputs as we did in parts a href="/x86-32-linux/2014/05/08/plain-buffer-overflow/"1/a, a href="/x86-32-linux/2014/05/20/plain-format-string-vulnerability/"2/a and a href="/x86-32-linux/2014/06/12/remote-exploitation/"3/a)./p
pReverse engineering (reversing) is an extremely powerful tool in the hackers arsenal and when there is no source code for the application that you are targeting nothing is better./p
!-- more --
pa href="https://en.wikipedia.org/wiki/Assembly_language" target="_blank"Assembly/a is the language of reversing and a a href="https://en.wikipedia.org/wiki/Debugger" target="_blank"debugger/a is the most important tool./p
pAssembly is essentially the language of the processor, the actual "machine code" that people think of what the computer deals with (whether viewed as binary or hex) is just a different representation of assembly language, so this is the lowest level programming language possible to those outside of processor firmware development./p
pA debugger is an application that allows you to view an applications a href="https://en.wikipedia.org/wiki/Virtual_memory" target="_blank"virtual memory segment/a as the application itself views it, as well as change the values in sections of memory or a href="https://en.wikipedia.org/wiki/Processor_register" target="_blank"CPU registers/a at run time./p
pAnother important feature of a debugger is the ability to set a href="https://en.wikipedia.org/wiki/Breakpoint" target="_blank"breakpoints/a so you can force the application to stop execution at a specific part of the application and view values or a href="https://en.wikipedia.org/wiki/Stepping_%28debugging%29" target="_blank"step through/a the application instruction by instruction./p
h2The App/h2
pWe will use the same basic application we used in parts a href="/x86-32-linux/2014/05/08/plain-buffer-overflow/"1/a and a href="/x86-32-linux/2014/05/20/plain-format-string-vulnerability/"2/a:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal" 1/span/span
span class="code-line"span class="normal" 2/span/span
span class="code-line"span class="normal" 3/span/span
span class="code-line"span class="normal" 4/span/span
span class="code-line"span class="normal" 5/span/span
span class="code-line"span class="normal" 6/span/span
span class="code-line"span class="normal" 7/span/span
span class="code-line"span class="normal" 8/span/span
span class="code-line"span class="normal" 9/span/span
span class="code-line"span class="normal"10/span/span
span class="code-line"span class="normal"11/span/span
span class="code-line"span class="normal"12/span/span
span class="code-line"span class="normal"13/span/span
span class="code-line"span class="normal"14/span/span
span class="code-line"span class="normal"15/span/span
span class="code-line"span class="normal"16/span/span
span class="code-line"span class="normal"17/span/span
span class="code-line"span class="normal"18/span/span
span class="code-line"span class="normal"19/span/span
span class="code-line"span class="normal"20/span/span
span class="code-line"span class="normal"21/span/span
span class="code-line"span class="normal"22/span/span
span class="code-line"span class="normal"23/span/span
span class="code-line"span class="normal"24/span/span
span class="code-line"span class="normal"25/span/span
span class="code-line"span class="normal"26/span/span
span class="code-line"span class="normal"27/span/span
span class="code-line"span class="normal"28/span/span
span class="code-line"span class="normal"29/span/span
span class="code-line"span class="normal"30/span/span
span class="code-line"span class="normal"31/span/span
span class="code-line"span class="normal"32/span/span
span class="code-line"span class="normal"33/span/span
span class="code-line"span class="normal"34/span/span
span class="code-line"span class="normal"35/span/span
span class="code-line"span class="normal"36/span/span
span class="code-line"span class="normal"37/span/span
span class="code-line"span class="normal"38/span/span
span class="code-line"span class="normal"39/span/span
span class="code-line"span class="normal"40/span/span
span class="code-line"span class="normal"41/span/span
span class="code-line"span class="normal"42/span/span
span class="code-line"span class="normal"43/span/span
span class="code-line"span class="normal"44/span/span
span class="code-line"span class="normal"45/span/span
span class="code-line"span class="normal"46/span/span
span class="code-line"span class="normal"47/span/span
span class="code-line"span class="normal"48/span/span
span class="code-line"span class="normal"49/span/span
span class="code-line"span class="normal"50/span/span
span class="code-line"span class="normal"51/span/span
span class="code-line"span class="normal"52/span/span
span class="code-line"span class="normal"53/span/span
span class="code-line"span class="normal"54/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="cp"#include/spanspan class="w" /spanspan class="cpf"lt;stdio.hgt;/spanspan class="cp"/span/span
span class="code-line"span class="cp"#include/spanspan class="w" /spanspan class="cpf"lt;string.hgt;/spanspan class="cp"/span/span
span class="code-line"span class="cp"#include/spanspan class="w" /spanspan class="cpf"lt;stdlib.hgt;/spanspan class="cp"/span/span
span class="code-line"/span
span class="code-line"span class="cp"#define PASS quot;topsecretpasswordquot;/span/span
span class="code-line"/span
span class="code-line"span class="cp"#define SFILE quot;secret.txtquot;/span/span
span class="code-line"/span
span class="code-line"span class="kt"int/spanspan class="w" /spanspan class="nf"checkpass/spanspan class="p"(/spanspan class="kt"char/spanspan class="w" /spanspan class="o"*/spanspan class="n"p/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="kt"void/spanspan class="w" /spanspan class="nf"printfile/spanspan class="p"();/spanspan class="w"/span/span
span class="code-line"/span
span class="code-line"span class="kt"int/spanspan class="w" /spanspan class="nf"main/spanspan class="p"(/spanspan class="kt"int/spanspan class="w" /spanspan class="n"argc/spanspan class="p",/spanspan class="w" /spanspan class="kt"char/spanspan class="w" /spanspan class="o"**/spanspan class="n"argv/spanspan class="p")/spanspan class="w"/span/span
span class="code-line"span class="p"{/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="kt"int/spanspan class="w" /spanspan class="n"r/spanspan class="p";/spanspan class="w"/span/span
span class="code-line"/span
span class="code-line"span class="w" /spanspan class="k"if/spanspan class="w" /spanspan class="p"(/spanspan class="n"argc/spanspan class="w" /spanspan class="o"lt;/spanspan class="w" /spanspan class="mi"2/spanspan class="p")/spanspan class="w" /spanspan class="p"{/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printf/spanspan class="p"(/spanspan class="s"quot;Usage: quot;/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printf/spanspan class="p"(/spanspan class="n"argv/spanspan class="p"[/spanspan class="mi"0/spanspan class="p"]);/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printf/spanspan class="p"(/spanspan class="s"quot; lt;passwordgt;/spanspan class="se"\n/spanspan class="s"quot;/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"exit/spanspan class="p"(/spanspan class="mi"1/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="p"}/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"r/spanspan class="w" /spanspan class="o"=/spanspan class="w" /spanspan class="n"checkpass/spanspan class="p"(/spanspan class="n"argv/spanspan class="p"[/spanspan class="mi"1/spanspan class="p"]);/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="k"if/spanspan class="w" /spanspan class="p"(/spanspan class="n"r/spanspan class="w" /spanspan class="o"!=/spanspan class="w" /spanspan class="mi"0/spanspan class="p")/spanspan class="w" /spanspan class="p"{/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printf/spanspan class="p"(/spanspan class="s"quot;Wrong password: quot;/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printf/spanspan class="p"(/spanspan class="n"argv/spanspan class="p"[/spanspan class="mi"1/spanspan class="p"]);/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printf/spanspan class="p"(/spanspan class="s"quot;/spanspan class="se"\n/spanspan class="s"quot;/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"exit/spanspan class="p"(/spanspan class="mi"1/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="p"}/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printfile/spanspan class="p"();/spanspan class="w"/span/span
span class="code-line"span class="p"}/spanspan class="w"/span/span
span class="code-line"/span
span class="code-line"span class="kt"int/spanspan class="w" /spanspan class="nf"checkpass/spanspan class="p"(/spanspan class="kt"char/spanspan class="w" /spanspan class="o"*/spanspan class="n"a/spanspan class="p")/spanspan class="w"/span/span
span class="code-line"span class="p"{/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="kt"char/spanspan class="w" /spanspan class="n"p/spanspan class="p"[/spanspan class="mi"512/spanspan class="p"];/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="kt"int/spanspan class="w" /spanspan class="n"r/spanspan class="p";/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"strncpy/spanspan class="p"(/spanspan class="n"p/spanspan class="p",/spanspan class="w" /spanspan class="n"a/spanspan class="p",/spanspan class="w" /spanspan class="n"strlen/spanspan class="p"(/spanspan class="n"a/spanspan class="p")/spanspan class="o"+/spanspan class="mi"1/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"r/spanspan class="w" /spanspan class="o"=/spanspan class="w" /spanspan class="n"strcmp/spanspan class="p"(/spanspan class="n"p/spanspan class="p",/spanspan class="w" /spanspan class="n"PASS/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="k"return/spanspan class="w" /spanspan class="n"r/spanspan class="p";/spanspan class="w"/span/span
span class="code-line"span class="p"}/spanspan class="w"/span/span
span class="code-line"/span
span class="code-line"span class="kt"void/spanspan class="w" /spanspan class="nf"printfile/spanspan class="p"()/spanspan class="w"/span/span
span class="code-line"span class="p"{/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="kt"FILE/spanspan class="w" /spanspan class="o"*/spanspan class="n"f/spanspan class="p";/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="kt"int/spanspan class="w" /spanspan class="n"c/spanspan class="p";/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"f/spanspan class="w" /spanspan class="o"=/spanspan class="w" /spanspan class="n"fopen/spanspan class="p"(/spanspan class="n"SFILE/spanspan class="p",/spanspan class="w" /spanspan class="s"quot;rquot;/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="k"if/spanspan class="w" /spanspan class="p"(/spanspan class="n"f/spanspan class="p")/spanspan class="w" /spanspan class="p"{/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="k"while/spanspan class="w" /spanspan class="p"((/spanspan class="n"c/spanspan class="w" /spanspan class="o"=/spanspan class="w" /spanspan class="n"getc/spanspan class="p"(/spanspan class="n"f/spanspan class="p"))/spanspan class="w" /spanspan class="o"!=/spanspan class="w" /spanspan class="n"EOF/spanspan class="p")/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"putchar/spanspan class="p"(/spanspan class="n"c/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"fclose/spanspan class="p"(/spanspan class="n"f/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="p"}/spanspan class="w" /spanspan class="k"else/spanspan class="w" /spanspan class="p"{/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printf/spanspan class="p"(/spanspan class="s"quot;Error opening file: quot;/spanspan class="w" /spanspan class="n"SFILE/spanspan class="w" /spanspan class="s"quot;/spanspan class="se"\n/spanspan class="s"quot;/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"exit/spanspan class="p"(/spanspan class="mi"1/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="p"}/spanspan class="w"/span/span
span class="code-line"span class="p"}/spanspan class="w"/span/span
span class="code-line"/code/pre/div
/td/tr/table
pThis time we will not exploit this application (we've done that already), instead we'll just use the debugger it figure out that these vulnerabilities exist./p
h2Setting Up The Environment/h2
pThis is the same as in part a href="/x86-32-linux/2014/05/08/plain-buffer-overflow/"1/a and a href="/x86-32-linux/2014/05/20/plain-format-string-vulnerability/"2/a so please refer to the strongSetting Up The Environment/strong section of 1 of those./p
h2Looking For The Juicy Bits/h2
pFirst we'll test the application as usual:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/span
span class="code-line"span class="normal"3/span/span
span class="code-line"span class="normal"4/span/span
span class="code-line"span class="normal"5/span/span
span class="code-line"span class="normal"6/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="gp"testuser@dev:~$ /span./app/span
span class="code-line"span class="go"Usage: ./app lt;passwordgt;/span/span
span class="code-line"span class="gp"testuser@dev:~$ /span./app span class="nb"test/span/span
span class="code-line"span class="go"Wrong password: test/span/span
span class="code-line"span class="gp"testuser@dev:~$ echo $/span?/span
span class="code-line"span class="go"1/span/span
span class="code-line"/code/pre/div
/td/tr/table
pNothing unusual there but we now know that the application takes 1 argument. If we open this using codegdb/code we can have a closer look at it:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal" 1/span/span
span class="code-line"span class="normal" 2/span/span
span class="code-line"span class="normal" 3/span/span
span class="code-line"span class="normal" 4/span/span
span class="code-line"span class="normal" 5/span/span
span class="code-line"span class="normal" 6/span/span
span class="code-line"span class="normal" 7/span/span
span class="code-line"span class="normal" 8/span/span
span class="code-line"span class="normal" 9/span/span
span class="code-line"span class="normal"10/span/span
span class="code-line"span class="normal"11/span/span
span class="code-line"span class="normal"12/span/span
span class="code-line"span class="normal"13/span/span
span class="code-line"span class="normal"14/span/span
span class="code-line"span class="normal"15/span/span
span class="code-line"span class="normal"16/span/span
span class="code-line"span class="normal"17/span/span
span class="code-line"span class="normal"18/span/span
span class="code-line"span class="normal"19/span/span
span class="code-line"span class="normal"20/span/span
span class="code-line"span class="normal"21/span/span
span class="code-line"span class="normal"22/span/span
span class="code-line"span class="normal"23/span/span
span class="code-line"span class="normal"24/span/span
span class="code-line"span class="normal"25/span/span
span class="code-line"span class="normal"26/span/span
span class="code-line"span class="normal"27/span/span
span class="code-line"span class="normal"28/span/span
span class="code-line"span class="normal"29/span/span
span class="code-line"span class="normal"30/span/span
span class="code-line"span class="normal"31/span/span
span class="code-line"span class="normal"32/span/span
span class="code-line"span class="normal"33/span/span
span class="code-line"span class="normal"34/span/span
span class="code-line"span class="normal"35/span/span
span class="code-line"span class="normal"36/span/span
span class="code-line"span class="normal"37/span/span
span class="code-line"span class="normal"38/span/span
span class="code-line"span class="normal"39/span/span
span class="code-line"span class="normal"40/span/span
span class="code-line"span class="normal"41/span/span
span class="code-line"span class="normal"42/span/span
span class="code-line"span class="normal"43/span/span
span class="code-line"span class="normal"44/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="gp"testuser@dev:~$ /spangdb -q ./app/span
span class="code-line"span class="go"Reading symbols from /home/testuser/app...(no debugging symbols found)...done./span/span
span class="code-line"span class="gp gp-VirtualEnv"(gdb)/span span class="go"set disassembly-flavor intel/span/span
span class="code-line"span class="gp gp-VirtualEnv"(gdb)/span span class="go"info functions/span/span
span class="code-line"span class="go"All defined functions:/span/span
span class="code-line"/span
span class="code-line"span class="go"Non-debugging symbols:/span/span
span class="code-line"span class="go"0x0804842e _init/span/span
span class="code-line"span class="go"0x08048460 strcmp/span/span
span class="code-line"span class="go"0x08048460 strcmp@plt/span/span
span class="code-line"span class="go"0x08048470 printf/span/span
span class="code-line"span class="go"0x08048470 printf@plt/span/span
span class="code-line"span class="go"0x08048480 fclose/span/span
span class="code-line"span class="go"0x08048480 fclose@plt/span/span
span class="code-line"span class="go"0x08048490 _IO_getc/span/span
span class="code-line"span class="go"0x08048490 _IO_getc@plt/span/span
span class="code-line"span class="go"0x080484a0 puts/span/span
span class="code-line"span class="go"0x080484a0 puts@plt/span/span
span class="code-line"span class="go"0x080484b0 __gmon_start__/span/span
span class="code-line"span class="go"0x080484b0 __gmon_start__@plt/span/span
span class="code-line"span class="go"0x080484c0 exit/span/span
span class="code-line"span class="go"0x080484c0 exit@plt/span/span
span class="code-line"span class="go"0x080484d0 strlen/span/span
span class="code-line"span class="go"0x080484d0 strlen@plt/span/span
span class="code-line"span class="go"0x080484e0 __libc_start_main/span/span
span class="code-line"span class="go"0x080484e0 __libc_start_main@plt/span/span
span class="code-line"span class="go"0x080484f0 fopen/span/span
span class="code-line"span class="go"0x080484f0 fopen@plt/span/span
span class="code-line"span class="go"0x08048500 putchar/span/span
span class="code-line"span class="go"0x08048500 putchar@plt/span/span
span class="code-line"span class="go"0x08048510 strncpy/span/span
span class="code-line"span class="go"0x08048510 strncpy@plt/span/span
span class="code-line"span class="go"0x08048520 _start/span/span
span class="code-line"span class="go"0x08048550 deregister_tm_clones/span/span
span class="code-line"span class="go"0x08048580 register_tm_clones/span/span
span class="code-line"span class="go"0x080485c0 __do_global_dtors_aux/span/span
span class="code-line"span class="go"0x080485e0 frame_dummy/span/span
span class="code-line"span class="go"0x0804860c main/span/span
span class="code-line"span class="go"0x080486a2 checkpass/span/span
span class="code-line"span class="go"0x080486f0 printfile/span/span
span class="code-line"span class="go"0x08048760 __libc_csu_fini/span/span
span class="code-line"span class="go"0x08048770 __libc_csu_init/span/span
span class="code-line"span class="go"0x080487ca __i686.get_pc_thunk.bx/span/span
span class="code-line"span class="go"0x080487d0 _fini/span/span
span class="code-line"/code/pre/div
/td/tr/table
pHere we can tell that the application was written in a href="https://en.wikipedia.org/wiki/C_%28programming_language%29" target="_blank"C/a because it includes code__libc_start_main/code on lines 25 and 26. This means we have a codemain/code function which is the start of our application (shown on line 38)./p
pThere are a couple of other functions of interest here but let's leave them for a bit and look at the codemain/code function:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal" 1/span/span
span class="code-line"span class="normal" 2/span/span
span class="code-line"span class="normal" 3/span/span
span class="code-line"span class="normal" 4/span/span
span class="code-line"span class="normal" 5/span/span
span class="code-line"span class="normal" 6/span/span
span class="code-line"span class="normal" 7/span/span
span class="code-line"span class="normal" 8/span/span
span class="code-line"span class="normal" 9/span/span
span class="code-line"span class="normal"10/span/span
span class="code-line"span class="normal"11/span/span
span class="code-line"span class="normal"12/span/span
span class="code-line"span class="normal"13/span/span
span class="code-line"span class="normal"14/span/span
span class="code-line"span class="normal"15/span/span
span class="code-line"span class="normal"16/span/span
span class="code-line"span class="normal"17/span/span
span class="code-line"span class="normal"18/span/span
span class="code-line"span class="normal"19/span/span
span class="code-line"span class="normal"20/span/span
span class="code-line"span class="normal"21/span/span
span class="code-line"span class="normal"22/span/span
span class="code-line"span class="normal"23/span/span
span class="code-line"span class="normal"24/span/span
span class="code-line"span class="normal"25/span/span
span class="code-line"span class="normal"26/span/span
span class="code-line"span class="normal"27/span/span
span class="code-line"span class="normal"28/span/span
span class="code-line"span class="normal"29/span/span
span class="code-line"span class="normal"30/span/span
span class="code-line"span class="normal"31/span/span
span class="code-line"span class="normal"32/span/span
span class="code-line"span class="normal"33/span/span
span class="code-line"span class="normal"34/span/span
span class="code-line"span class="normal"35/span/span
span class="code-line"span class="normal"36/span/span
span class="code-line"span class="normal"37/span/span
span class="code-line"span class="normal"38/span/span
span class="code-line"span class="normal"39/span/span
span class="code-line"span class="normal"40/span/span
span class="code-line"span class="normal"41/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="gp gp-VirtualEnv"(gdb)/span span class="go"disassemble main/span/span
span class="code-line"span class="go"Dump of assembler code for function main:/span/span
span class="code-line"span class="go" 0x0804860c lt;+0gt;: push ebp/span/span
span class="code-line"span class="go" 0x0804860d lt;+1gt;: mov ebp,esp/span/span
span class="code-line"span class="go" 0x0804860f lt;+3gt;: and esp,0xfffffff0/span/span
span class="code-line"span class="go" 0x08048612 lt;+6gt;: sub esp,0x20/span/span
span class="code-line"span class="go" 0x08048615 lt;+9gt;: cmp DWORD PTR [ebp+0x8],0x1/span/span
span class="code-line"span class="go" 0x08048619 lt;+13gt;: jg 0x804864c lt;main+64gt;/span/span
span class="code-line"span class="go" 0x0804861b lt;+15gt;: mov DWORD PTR [esp],0x80487f0/span/span
span class="code-line"span class="go" 0x08048622 lt;+22gt;: call 0x8048470 lt;printf@pltgt;/span/span
span class="code-line"span class="go" 0x08048627 lt;+27gt;: mov eax,DWORD PTR [ebp+0xc]/span/span
span class="code-line"span class="go" 0x0804862a lt;+30gt;: mov eax,DWORD PTR [eax]/span/span
span class="code-line"span class="go" 0x0804862c lt;+32gt;: mov DWORD PTR [esp],eax/span/span
span class="code-line"span class="go" 0x0804862f lt;+35gt;: call 0x8048470 lt;printf@pltgt;/span/span
span class="code-line"span class="go" 0x08048634 lt;+40gt;: mov DWORD PTR [esp],0x80487f8/span/span
span class="code-line"span class="go" 0x0804863b lt;+47gt;: call 0x80484a0 lt;puts@pltgt;/span/span
span class="code-line"span class="go" 0x08048640 lt;+52gt;: mov DWORD PTR [esp],0x1/span/span
span class="code-line"span class="go" 0x08048647 lt;+59gt;: call 0x80484c0 lt;exit@pltgt;/span/span
span class="code-line"span class="go" 0x0804864c lt;+64gt;: mov eax,DWORD PTR [ebp+0xc]/span/span
span class="code-line"span class="go" 0x0804864f lt;+67gt;: add eax,0x4/span/span
span class="code-line"span class="go" 0x08048652 lt;+70gt;: mov eax,DWORD PTR [eax]/span/span
span class="code-line"span class="go" 0x08048654 lt;+72gt;: mov DWORD PTR [esp],eax/span/span
span class="code-line"span class="go" 0x08048657 lt;+75gt;: call 0x80486a2 lt;checkpassgt;/span/span
span class="code-line"span class="go" 0x0804865c lt;+80gt;: mov DWORD PTR [esp+0x1c],eax/span/span
span class="code-line"span class="go" 0x08048660 lt;+84gt;: cmp DWORD PTR [esp+0x1c],0x0/span/span
span class="code-line"span class="go" 0x08048665 lt;+89gt;: je 0x804869b lt;main+143gt;/span/span
span class="code-line"span class="go" 0x08048667 lt;+91gt;: mov DWORD PTR [esp],0x8048804/span/span
span class="code-line"span class="go" 0x0804866e lt;+98gt;: call 0x8048470 lt;printf@pltgt;/span/span
span class="code-line"span class="go" 0x08048673 lt;+103gt;: mov eax,DWORD PTR [ebp+0xc]/span/span
span class="code-line"span class="go" 0x08048676 lt;+106gt;: add eax,0x4/span/span
span class="code-line"span class="go" 0x08048679 lt;+109gt;: mov eax,DWORD PTR [eax]/span/span
span class="code-line"span class="go" 0x0804867b lt;+111gt;: mov DWORD PTR [esp],eax/span/span
span class="code-line"span class="go" 0x0804867e lt;+114gt;: call 0x8048470 lt;printf@pltgt;/span/span
span class="code-line"span class="go" 0x08048683 lt;+119gt;: mov DWORD PTR [esp],0xa/span/span
span class="code-line"span class="go" 0x0804868a lt;+126gt;: call 0x8048500 lt;putchar@pltgt;/span/span
span class="code-line"span class="go" 0x0804868f lt;+131gt;: mov DWORD PTR [esp],0x1/span/span
span class="code-line"span class="go" 0x08048696 lt;+138gt;: call 0x80484c0 lt;exit@pltgt;/span/span
span class="code-line"span class="go" 0x0804869b lt;+143gt;: call 0x80486f0 lt;printfilegt;/span/span
span class="code-line"span class="go" 0x080486a0 lt;+148gt;: leave /span/span
span class="code-line"span class="go" 0x080486a1 lt;+149gt;: ret /span/span
span class="code-line"span class="go"End of assembler dump./span/span
span class="code-line"/code/pre/div
/td/tr/table
pThe first 4 instructions are the a href="https://en.wikipedia.org/wiki/Function_prologue" target="_blank"function prologue/a (lines 3, 4, 5 and 6). Here the a href="http://en.citizendium.org/wiki/Stack_frame" target="_blank"stack frame/a is set up./p
pThe last 2 instructions are the a href="https://en.wikipedia.org/wiki/Function_prologue#Epilogue" target="_blank"function epilogue/a (lines 39 and 40). Here the codeleave/code instruction preforms the inverse of what the prologue did./p
pLooking at the prologue and epilogue we can see that the a href="https://en.wikipedia.org/wiki/Calling_convention" target="_blank"calling convention/a is probably a href="https://en.wikipedia.org/wiki/X86_calling_conventions#cdecl" target="_blank"cdecl/a./p
pI will not go into calling conventions much here, because it isn't terribly relevant although its important to know what they are and the differences, but a calling convention basically defines how a function is called./p
pBack on topic, initially when looking for a vulnerability we should check some of the known vulnerable functions commonly used by developers. The main 1's are the codeprintf/code family of functions and the string copying/moving functions./p
pLooking back at our list of functions, a couple of interest are being used. Mainly codeprintf/code and codestrncpy/code. In the main function though only codeprintf/code out of those 2 is being used. Let's examine them a little closer./p
pThe first, on line 10, is set up on line 9 with an argument:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="go" 0x0804861b lt;+15gt;: mov DWORD PTR [esp],0x80487f0/span/span
span class="code-line"span class="go" 0x08048622 lt;+22gt;: call 0x8048470 lt;printf@pltgt;/span/span
span class="code-line"/code/pre/div
/td/tr/table
pWhat the first instruction is doing here is moving the address code0x80487f0/code into the address strongpointed to/strong by the a href="http://www.c-jump.com/CIS77/ASM/Stack/S77_0040_esp_register.htm" target="_blank"ESP register/a. These 2 lines relate to line 17 in our source code above./p
pThe ESP register points to the top of the a href="https://en.wikipedia.org/wiki/Stack_%28abstract_data_type%29" target="_blank"stack/a and in the cdecl calling convension, before the actual call to the function, its arguments are strongpushed/strong onto the stack in reverse order. As there is only 1 argument to this call only 1 is put on the stack./p
pTo be honest, this call doesn't look like its going to be of interest as the argument is a static address and it points to the a href="https://en.wikipedia.org/wiki/Code_segment" target="_blank"text segment/a of memory which isn't writable, but we can check the value of this just to make sure:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="gp gp-VirtualEnv"(gdb)/span span class="go"x/s 0x80487f0/span/span
span class="code-line"span class="go"0x80487f0: quot;Usage: quot;/span/span
span class="code-line"/code/pre/div
/td/tr/table
pSo it looks to be part of an error message. The next call to codeprintf/code looks more interesting but first we need to understand how a stack frame is arranged in an application like this./p
h2Stack Frames/h2
pBelow is the top of an example stack frame which is getting ready for a function call:/p
pimg src="/assets/images/x86-32-linux/stack1.jpg" width="300"/p
pHere we are unable to see the base pointer (EBP) but we can see the stack pointer (ESP) which always points to the top of the stack./p
pPutting arguments on the stack can be done in a number of ways. Firstly it can be done using the codepush/code instruction as follows:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/span
span class="code-line"span class="normal"3/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="nf"push/spanspan class="w" /spanspan class="nb"eax/spanspan class="w"/span/span
span class="code-line"span class="nf"push/spanspan class="w" /spanspan class="mh"0x80487f0/spanspan class="w"/span/span
span class="code-line"span class="nf"push/spanspan class="w" /spanspan class="p"[/spanspan class="nb"ebp/spanspan class="o"+/spanspan class="nv"c/spanspan class="p"]/spanspan class="w"/span/span
span class="code-line"/code/pre/div
/td/tr/table
pHere the value is the EAX register is being strongpushed/strong onto the stack as the third argument (or "ARG 3" in our diagram), then the static value code0x80487f0/code as the second argument and finally EBP+c (or EBP+12, which is usually the second argument to the current function) as the first argument./p
pThe codepush/code instruction automatically adjusts the value of ESP accordingly but it can also be done manually:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/span
span class="code-line"span class="normal"3/span/span
span class="code-line"span class="normal"4/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="nf"sub/spanspan class="w" /spanspan class="nb"esp/spanspan class="p",/spanspan class="w" /spanspan class="mh"0xc/spanspan class="w"/span/span
span class="code-line"span class="nf"mov/spanspan class="w" /spanspan class="p"[/spanspan class="nb"esp/spanspan class="o"+/spanspan class="mi"8/spanspan class="p"],/spanspan class="w" /spanspan class="nb"eax/spanspan class="w"/span/span
span class="code-line"span class="nf"mov/spanspan class="w" /spanspan class="p"[/spanspan class="nb"esp/spanspan class="o"+/spanspan class="mi"4/spanspan class="p"],/spanspan class="w" /spanspan class="mh"0x80487f0/spanspan class="w"/span/span
span class="code-line"span class="nf"mov/spanspan class="w" /spanspan class="p"[/spanspan class="nb"esp/spanspan class="p"],/spanspan class="w" /spanspan class="p"[/spanspan class="nb"ebp/spanspan class="o"+/spanspan class="nv"c/spanspan class="p"]/spanspan class="w"/span/span
span class="code-line"/code/pre/div
/td/tr/table
pThis set of instructions are functionally the same as the previous. These are followed by a codecall/code instruction and after the call instruction our stack looks like this:/p
pimg src="/assets/images/x86-32-linux/stack2.jpg" width="300"/p
pThe codecall/code instruction autmatically strongpushes/strong the memory address of the next instruction onto the stack. This is done so that when a function returns the application knows where to start executing instructions./p
pInside the function that we have just called we start executing that functions prologue. First there is a codepush ebp/code instruction which does this to the stack:/p
pimg src="/assets/images/x86-32-linux/stack3.jpg" width="300"/p
pAfter that it executes codemov ebp, esp/code:/p
pimg src="/assets/images/x86-32-linux/stack4.jpg" width="300"/p
pLastly any space for needed for local variables is subtracted from ESP (codesub esp, 0x8/code), so our stack ends up like this:/p
pimg src="/assets/images/x86-32-linux/stack5.jpg" width="300"/p
pEBP always points to the start of the current functions stack frame and ESP to the top of the stack so if we call another function inside the current function the same process would happen./p
pThe functions epilogue does the opposite, in the application we are debugging it just have to codeleave/code instruction. The codeleave/code instruction automates the cleanup of the stack frame./p
pIn our example stack, the codeleave/code function would be equivalent to:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="nf"add/spanspan class="w" /spanspan class="nb"esp/spanspan class="p",/spanspan class="w" /spanspan class="mh"0x8/spanspan class="w"/span/span
span class="code-line"span class="nf"pop/spanspan class="w" /spanspan class="nb"ebp/spanspan class="w"/span/span
span class="code-line"/code/pre/div
/td/tr/table
pThis would bring our stack frame back to this:/p
pimg src="/assets/images/x86-32-linux/stack2.jpg" width="300"/p
pAnd then the final coderet/code instruction would remove the strongRET ADDR/strong from the stack setting everything back to how it was before the function call, coderet/code essentially does codepop eip/code./p
h2Juicy Bits Continued/h2
pNow that we understand how the stack works we can have a look at that second call to codeprintf/code. The first argument to codeprintf/code is always the format string so when looking for a format string vulnerability we are trying to figure out if we can control the first argument./p
pThe relevant lines that setup and call codeprintf/code are:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/span
span class="code-line"span class="normal"3/span/span
span class="code-line"span class="normal"4/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="x" 0x08048627 lt;+27gt;: mov eax,DWORD PTR [ebp+0xc]/span/span
span class="code-line"span class="x" 0x0804862a lt;+30gt;: mov eax,DWORD PTR [eax]/span/span
span class="code-line"span class="x" 0x0804862c lt;+32gt;: mov DWORD PTR [esp],eax/span/span
span class="code-line"span class="x" 0x0804862f lt;+35gt;: call 0x8048470 lt;printf@pltgt;/span/span
span class="code-line"/code/pre/div
/td/tr/table
pThese 4 lines of code is actually line 18 in the source of the application. Line 1 moves the second argument (codeebp+0xc/code) (the second argument is always +C or +12 because EBP points to the old EBP, +4 points to the return address and +8 points to the first argument) into EAX./p
pIn C the second argument to the main function is a list of pointers to the actual application arguments./p
pBecause this argument is an array of pointers, line 2 moves the first pointer in this array into EAX (this normally points to the path of the application itself)./p
pThis pointer is moved to the address pointed to by ESP (the top of the stack) and finally codeprintf/code is called. This shows that only 1 argument was given and that argument is the application path./p
pWe can check this using codegdb/code but first there was a conditional statement which determined if this code got executed:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="x" 0x08048615 lt;+9gt;: cmp DWORD PTR [ebp+0x8],0x1/span/span
span class="code-line"span class="x" 0x08048619 lt;+13gt;: jg 0x804864c lt;main+64gt;/span/span
span class="code-line"/code/pre/div
/td/tr/table
pThis is the codeif/code statement on line 16 of the source code./p
pLine 1 compares the first argument codeebp+0x8/code, with 1 and jumps to code0x804864c/code if the first argument is greater than 1. As you can see the assembly condition is the opposite to what is in the source code, this is often the case./p
pIn C the first argument to the main function is the number of arguments give to the application on the command line so to enter the section of code we want to analyse we just need to give the application 1 argument (the name of the application is considered the first argument so there is always at least 1)./p
h3Integer Overflow/h3
pThe codejg/code instruction means that the numbers that are being compared are signed (it would be codeja/code if they were unsigned) and because there is no bound checking done on codeebp+0x8/code, it is vulnerable to an integer overflow:/p
pI wanted to demostrate this as soon as I realised but because it is an integer I need to send at least 2147483647 arguments, I couldn't do this on my test machine because there just isn't enough RAM./p
pSo in the name of science, I rewrote the application so that the codeargc/code argument (or the number of arguments passed to the main function) is a codechar/code instead, here is my new application:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal" 1/span/span
span class="code-line"span class="normal" 2/span/span
span class="code-line"span class="normal" 3/span/span
span class="code-line"span class="normal" 4/span/span
span class="code-line"span class="normal" 5/span/span
span class="code-line"span class="normal" 6/span/span
span class="code-line"span class="normal" 7/span/span
span class="code-line"span class="normal" 8/span/span
span class="code-line"span class="normal" 9/span/span
span class="code-line"span class="normal"10/span/span
span class="code-line"span class="normal"11/span/span
span class="code-line"span class="normal"12/span/span
span class="code-line"span class="normal"13/span/span
span class="code-line"span class="normal"14/span/span
span class="code-line"span class="normal"15/span/span
span class="code-line"span class="normal"16/span/span
span class="code-line"span class="normal"17/span/span
span class="code-line"span class="normal"18/span/span
span class="code-line"span class="normal"19/span/span
span class="code-line"span class="normal"20/span/span
span class="code-line"span class="normal"21/span/span
span class="code-line"span class="normal"22/span/span
span class="code-line"span class="normal"23/span/span
span class="code-line"span class="normal"24/span/span
span class="code-line"span class="normal"25/span/span
span class="code-line"span class="normal"26/span/span
span class="code-line"span class="normal"27/span/span
span class="code-line"span class="normal"28/span/span
span class="code-line"span class="normal"29/span/span
span class="code-line"span class="normal"30/span/span
span class="code-line"span class="normal"31/span/span
span class="code-line"span class="normal"32/span/span
span class="code-line"span class="normal"33/span/span
span class="code-line"span class="normal"34/span/span
span class="code-line"span class="normal"35/span/span
span class="code-line"span class="normal"36/span/span
span class="code-line"span class="normal"37/span/span
span class="code-line"span class="normal"38/span/span
span class="code-line"span class="normal"39/span/span
span class="code-line"span class="normal"40/span/span
span class="code-line"span class="normal"41/span/span
span class="code-line"span class="normal"42/span/span
span class="code-line"span class="normal"43/span/span
span class="code-line"span class="normal"44/span/span
span class="code-line"span class="normal"45/span/span
span class="code-line"span class="normal"46/span/span
span class="code-line"span class="normal"47/span/span
span class="code-line"span class="normal"48/span/span
span class="code-line"span class="normal"49/span/span
span class="code-line"span class="normal"50/span/span
span class="code-line"span class="normal"51/span/span
span class="code-line"span class="normal"52/span/span
span class="code-line"span class="normal"53/span/span
span class="code-line"span class="normal"54/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="cp"#include/spanspan class="w" /spanspan class="cpf"lt;stdio.hgt;/spanspan class="cp"/span/span
span class="code-line"span class="cp"#include/spanspan class="w" /spanspan class="cpf"lt;string.hgt;/spanspan class="cp"/span/span
span class="code-line"span class="cp"#include/spanspan class="w" /spanspan class="cpf"lt;stdlib.hgt;/spanspan class="cp"/span/span
span class="code-line"/span
span class="code-line"span class="cp"#define PASS quot;topsecretpasswordquot;/span/span
span class="code-line"/span
span class="code-line"span class="cp"#define SFILE quot;secret.txtquot;/span/span
span class="code-line"/span
span class="code-line"span class="kt"int/spanspan class="w" /spanspan class="nf"checkpass/spanspan class="p"(/spanspan class="kt"char/spanspan class="w" /spanspan class="o"*/spanspan class="n"p/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="kt"void/spanspan class="w" /spanspan class="nf"printfile/spanspan class="p"();/spanspan class="w"/span/span
span class="code-line"/span
span class="code-line"span class="kt"int/spanspan class="w" /spanspan class="nf"main/spanspan class="p"(/spanspan class="kt"char/spanspan class="w" /spanspan class="n"argc/spanspan class="p",/spanspan class="w" /spanspan class="kt"char/spanspan class="w" /spanspan class="o"**/spanspan class="n"argv/spanspan class="p")/spanspan class="w"/span/span
span class="code-line"span class="p"{/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="kt"int/spanspan class="w" /spanspan class="n"r/spanspan class="p";/spanspan class="w"/span/span
span class="code-line"/span
span class="code-line"span class="w" /spanspan class="k"if/spanspan class="w" /spanspan class="p"(/spanspan class="n"argc/spanspan class="w" /spanspan class="o"lt;/spanspan class="w" /spanspan class="mi"2/spanspan class="p")/spanspan class="w" /spanspan class="p"{/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printf/spanspan class="p"(/spanspan class="s"quot;Usage: quot;/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printf/spanspan class="p"(/spanspan class="n"argv/spanspan class="p"[/spanspan class="mi"0/spanspan class="p"]);/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printf/spanspan class="p"(/spanspan class="s"quot; lt;passwordgt;/spanspan class="se"\n/spanspan class="s"quot;/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"exit/spanspan class="p"(/spanspan class="mi"1/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="p"}/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"r/spanspan class="w" /spanspan class="o"=/spanspan class="w" /spanspan class="n"checkpass/spanspan class="p"(/spanspan class="n"argv/spanspan class="p"[/spanspan class="mi"1/spanspan class="p"]);/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="k"if/spanspan class="w" /spanspan class="p"(/spanspan class="n"r/spanspan class="w" /spanspan class="o"!=/spanspan class="w" /spanspan class="mi"0/spanspan class="p")/spanspan class="w" /spanspan class="p"{/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printf/spanspan class="p"(/spanspan class="s"quot;Wrong password: quot;/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printf/spanspan class="p"(/spanspan class="n"argv/spanspan class="p"[/spanspan class="mi"1/spanspan class="p"]);/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printf/spanspan class="p"(/spanspan class="s"quot;/spanspan class="se"\n/spanspan class="s"quot;/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"exit/spanspan class="p"(/spanspan class="mi"1/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="p"}/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printfile/spanspan class="p"();/spanspan class="w"/span/span
span class="code-line"span class="p"}/spanspan class="w"/span/span
span class="code-line"/span
span class="code-line"span class="kt"int/spanspan class="w" /spanspan class="nf"checkpass/spanspan class="p"(/spanspan class="kt"char/spanspan class="w" /spanspan class="o"*/spanspan class="n"a/spanspan class="p")/spanspan class="w"/span/span
span class="code-line"span class="p"{/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="kt"char/spanspan class="w" /spanspan class="n"p/spanspan class="p"[/spanspan class="mi"512/spanspan class="p"];/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="kt"int/spanspan class="w" /spanspan class="n"r/spanspan class="p";/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"strncpy/spanspan class="p"(/spanspan class="n"p/spanspan class="p",/spanspan class="w" /spanspan class="n"a/spanspan class="p",/spanspan class="w" /spanspan class="n"strlen/spanspan class="p"(/spanspan class="n"a/spanspan class="p")/spanspan class="o"+/spanspan class="mi"1/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"r/spanspan class="w" /spanspan class="o"=/spanspan class="w" /spanspan class="n"strcmp/spanspan class="p"(/spanspan class="n"p/spanspan class="p",/spanspan class="w" /spanspan class="n"PASS/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="k"return/spanspan class="w" /spanspan class="n"r/spanspan class="p";/spanspan class="w"/span/span
span class="code-line"span class="p"}/spanspan class="w"/span/span
span class="code-line"/span
span class="code-line"span class="kt"void/spanspan class="w" /spanspan class="nf"printfile/spanspan class="p"()/spanspan class="w"/span/span
span class="code-line"span class="p"{/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="kt"FILE/spanspan class="w" /spanspan class="o"*/spanspan class="n"f/spanspan class="p";/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="kt"int/spanspan class="w" /spanspan class="n"c/spanspan class="p";/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"f/spanspan class="w" /spanspan class="o"=/spanspan class="w" /spanspan class="n"fopen/spanspan class="p"(/spanspan class="n"SFILE/spanspan class="p",/spanspan class="w" /spanspan class="s"quot;rquot;/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="k"if/spanspan class="w" /spanspan class="p"(/spanspan class="n"f/spanspan class="p")/spanspan class="w" /spanspan class="p"{/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="k"while/spanspan class="w" /spanspan class="p"((/spanspan class="n"c/spanspan class="w" /spanspan class="o"=/spanspan class="w" /spanspan class="n"getc/spanspan class="p"(/spanspan class="n"f/spanspan class="p"))/spanspan class="w" /spanspan class="o"!=/spanspan class="w" /spanspan class="n"EOF/spanspan class="p")/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"putchar/spanspan class="p"(/spanspan class="n"c/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"fclose/spanspan class="p"(/spanspan class="n"f/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="p"}/spanspan class="w" /spanspan class="k"else/spanspan class="w" /spanspan class="p"{/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"printf/spanspan class="p"(/spanspan class="s"quot;Error opening file: quot;/spanspan class="w" /spanspan class="n"SFILE/spanspan class="w" /spanspan class="s"quot;/spanspan class="se"\n/spanspan class="s"quot;/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="n"exit/spanspan class="p"(/spanspan class="mi"1/spanspan class="p");/spanspan class="w"/span/span
span class="code-line"span class="w" /spanspan class="p"}/spanspan class="w"/span/span
span class="code-line"span class="p"}/spanspan class="w"/span/span
span class="code-line"/code/pre/div
/td/tr/table
pHere is the quick demonstration:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/span
span class="code-line"span class="normal"3/span/span
span class="code-line"span class="normal"4/span/span
span class="code-line"span class="normal"5/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="gp"root@dev:/home/testuser# /spangcc -z execstack -fno-stack-protector -o app-intof app-intof.c /span
span class="code-line"span class="gp"root@dev:/home/testuser# ./app-intof $/spanspan class="o"(/spanpython -c span class="s1"#39;print quot;A quot;*126#39;/spanspan class="o")/span/span
span class="code-line"span class="go"Wrong password: A/span/span
span class="code-line"span class="gp"root@dev:/home/testuser# ./app-intof $/spanspan class="o"(/spanpython -c span class="s1"#39;print quot;A quot;*127#39;/spanspan class="o")/span/span
span class="code-line"span class="go"Usage: ./app-intof lt;passwordgt;/span/span
span class="code-line"/code/pre/div
/td/tr/table
pWhat is happening here is that the argument codeargc/code is being interpreted as a signed char and the max value for this type of variable is 127:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/span
span class="code-line"span class="normal"3/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="gp"root@dev:/home/testuser# /spangrep SCHAR_MAX /usr/include/limits.h /span
span class="code-line"span class="gp"# /spandefine SCHAR_MAX span class="m"127/span/span
span class="code-line"span class="gp"# /spandefine CHAR_MAX SCHAR_MAX/span
span class="code-line"/code/pre/div
/td/tr/table
pAs the application is the first argument, we can have another 126 argument before the variable overflows and becomes -128, which is obviously smaller than 2./p
h2Back To The Juicy Bits/h2
pSo now we know how to get to the code we want to analyse, which is:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/span
span class="code-line"span class="normal"3/span/span
span class="code-line"span class="normal"4/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="x" 0x08048627 lt;+27gt;: mov eax,DWORD PTR [ebp+0xc]/span/span
span class="code-line"span class="x" 0x0804862a lt;+30gt;: mov eax,DWORD PTR [eax]/span/span
span class="code-line"span class="x" 0x0804862c lt;+32gt;: mov DWORD PTR [esp],eax/span/span
span class="code-line"span class="x" 0x0804862f lt;+35gt;: call 0x8048470 lt;printf@pltgt;/span/span
span class="code-line"/code/pre/div
/td/tr/table
pLet's set a breakpoint on line 1 here (or code0x08048627/code) and run the application without any arguments./p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal" 1/span/span
span class="code-line"span class="normal" 2/span/span
span class="code-line"span class="normal" 3/span/span
span class="code-line"span class="normal" 4/span/span
span class="code-line"span class="normal" 5/span/span
span class="code-line"span class="normal" 6/span/span
span class="code-line"span class="normal" 7/span/span
span class="code-line"span class="normal" 8/span/span
span class="code-line"span class="normal" 9/span/span
span class="code-line"span class="normal"10/span/span
span class="code-line"span class="normal"11/span/span
span class="code-line"span class="normal"12/span/span
span class="code-line"span class="normal"13/span/span
span class="code-line"span class="normal"14/span/span
span class="code-line"span class="normal"15/span/span
span class="code-line"span class="normal"16/span/span
span class="code-line"span class="normal"17/span/span
span class="code-line"span class="normal"18/span/span
span class="code-line"span class="normal"19/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="gp gp-VirtualEnv"(gdb)/span span class="go"break *0x08048627/span/span
span class="code-line"span class="go"Breakpoint 1 at 0x8048627/span/span
span class="code-line"span class="gp gp-VirtualEnv"(gdb)/span span class="go"r/span/span
span class="code-line"span class="go"Starting program: /home/testuser/app /span/span
span class="code-line"/span
span class="code-line"span class="go"Breakpoint 1, 0x08048627 in main ()/span/span
span class="code-line"span class="gp gp-VirtualEnv"(gdb)/span span class="go"disassemble $eip,+10/span/span
span class="code-line"span class="go"Dump of assembler code from 0x8048627 to 0x8048631:/span/span
span class="code-line"span class="go"=gt; 0x08048627 lt;main+27gt;: mov eax,DWORD PTR [ebp+0xc]/span/span
span class="code-line"span class="go" 0x0804862a lt;main+30gt;: mov eax,DWORD PTR [eax]/span/span
span class="code-line"span class="go" 0x0804862c lt;main+32gt;: mov DWORD PTR [esp],eax/span/span
span class="code-line"span class="go" 0x0804862f lt;main+35gt;: call 0x8048470 lt;printf@pltgt;/span/span
span class="code-line"span class="go"End of assembler dump./span/span
span class="code-line"span class="gp gp-VirtualEnv"(gdb)/span span class="go"x/xw $ebp+0xc/span/span
span class="code-line"span class="go"0xbfc674f4: 0xbfc67594/span/span
span class="code-line"span class="gp gp-VirtualEnv"(gdb)/span span class="go"x/xw 0xbfc67594/span/span
span class="code-line"span class="go"0xbfc67594: 0xbfc6795f/span/span
span class="code-line"span class="gp gp-VirtualEnv"(gdb)/span span class="go"x/s 0xbfc6795f/span/span
span class="code-line"span class="go"0xbfc6795f: quot;/home/testuser/appquot;/span/span
span class="code-line"/code/pre/div
/td/tr/table
pThis shows that our assumptions were correct and that there is likely a format string vulnerability here which we can exploit by chaning the name of the application (or creating a symlink as in a href="/x86-32-linux/2014/05/20/plain-format-string-vulnerability/"part 2/a./p
pWe also have a very similar set of codeprintf/code calls towards the end of the application:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/span
span class="code-line"span class="normal"3/span/span
span class="code-line"span class="normal"4/span/span
span class="code-line"span class="normal"5/span/span
span class="code-line"span class="normal"6/span/span
span class="code-line"span class="normal"7/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="x" 0x08048667 lt;+91gt;: mov DWORD PTR [esp],0x8048804/span/span
span class="code-line"span class="x" 0x0804866e lt;+98gt;: call 0x8048470 lt;printf@pltgt;/span/span
span class="code-line"span class="x" 0x08048673 lt;+103gt;: mov eax,DWORD PTR [ebp+0xc]/span/span
span class="code-line"span class="x" 0x08048676 lt;+106gt;: add eax,0x4/span/span
span class="code-line"span class="x" 0x08048679 lt;+109gt;: mov eax,DWORD PTR [eax]/span/span
span class="code-line"span class="x" 0x0804867b lt;+111gt;: mov DWORD PTR [esp],eax/span/span
span class="code-line"span class="x" 0x0804867e lt;+114gt;: call 0x8048470 lt;printf@pltgt;/span/span
span class="code-line"/code/pre/div
/td/tr/table
pWe are interested in the second codeprintf/code here but to figure out how to get to it we need to have a look at the memory at code0x8048804/code which is printed just before./p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="gp gp-VirtualEnv"(gdb)/span span class="go"x/s 0x8048804/span/span
span class="code-line"span class="go"0x8048804: quot;Wrong password: quot;/span/span
span class="code-line"/code/pre/div
/td/tr/table
pSo we get to this section of code when we give a wrong password. The call to the codeprintf/code in question is the same as previous except 4 is added to EAX before the pointer is followed. This suggests the second argument is being printed (also the previous codeprintf/code supports our theory), but let's check./p
pLet's set a breakpoint and examine the memory again:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal" 1/span/span
span class="code-line"span class="normal" 2/span/span
span class="code-line"span class="normal" 3/span/span
span class="code-line"span class="normal" 4/span/span
span class="code-line"span class="normal" 5/span/span
span class="code-line"span class="normal" 6/span/span
span class="code-line"span class="normal" 7/span/span
span class="code-line"span class="normal" 8/span/span
span class="code-line"span class="normal" 9/span/span
span class="code-line"span class="normal"10/span/span
span class="code-line"span class="normal"11/span/span
span class="code-line"span class="normal"12/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="gp gp-VirtualEnv"(gdb)/span span class="go"info breakpoints/span/span
span class="code-line"span class="go"Num Type Disp Enb Address What/span/span
span class="code-line"span class="go"1 breakpoint keep y 0x08048627 lt;main+27gt;/span/span
span class="code-line"span class="gp gp-VirtualEnv"(gdb)/span span class="go"delete 1/span/span
span class="code-line"span class="gp gp-VirtualEnv"(gdb)/span span class="go"break *0x0804867b/span/span
span class="code-line"span class="go"Breakpoint 2 at 0x804867b/span/span
span class="code-line"span class="gp gp-VirtualEnv"(gdb)/span span class="go"r ABC/span/span
span class="code-line"span class="go"Starting program: /home/testuser/app ABC/span/span
span class="code-line"/span
span class="code-line"span class="go"Breakpoint 2, 0x0804867b in main ()/span/span
span class="code-line"span class="gp gp-VirtualEnv"(gdb)/span span class="go"x/s $eax/span/span
span class="code-line"span class="go"0xbffff96d: quot;ABCquot;/span/span
span class="code-line"/code/pre/div
/td/tr/table
pThis is the second format string vulnerability./p
h2Buffer Overflow/h2
pSo far we have found an integer overflow and 2 format string vulnerabilities./p
pNext we should look over the codecheckpass/code function which is called on line 23 of the disassembly above. Here is the relevant instructions related to the call to codecheckpass/code:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/span
span class="code-line"span class="normal"3/span/span
span class="code-line"span class="normal"4/span/span
span class="code-line"span class="normal"5/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="x" 0x0804864c lt;+64gt;: mov eax,DWORD PTR [ebp+0xc]/span/span
span class="code-line"span class="x" 0x0804864f lt;+67gt;: add eax,0x4/span/span
span class="code-line"span class="x" 0x08048652 lt;+70gt;: mov eax,DWORD PTR [eax]/span/span
span class="code-line"span class="x" 0x08048654 lt;+72gt;: mov DWORD PTR [esp],eax/span/span
span class="code-line"span class="x" 0x08048657 lt;+75gt;: call 0x80486a2 lt;checkpassgt;/span/span
span class="code-line"/code/pre/div
/td/tr/table
pWe've already seen a set of instructions that were exactly the same as this, the second call to codeprintf/code, so this function takes 1 argument, the second argument to the application./p
pHere is the disassembly of codecheckpass/code:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal" 1/span/span
span class="code-line"span class="normal" 2/span/span
span class="code-line"span class="normal" 3/span/span
span class="code-line"span class="normal" 4/span/span
span class="code-line"span class="normal" 5/span/span
span class="code-line"span class="normal" 6/span/span
span class="code-line"span class="normal" 7/span/span
span class="code-line"span class="normal" 8/span/span
span class="code-line"span class="normal" 9/span/span
span class="code-line"span class="normal"10/span/span
span class="code-line"span class="normal"11/span/span
span class="code-line"span class="normal"12/span/span
span class="code-line"span class="normal"13/span/span
span class="code-line"span class="normal"14/span/span
span class="code-line"span class="normal"15/span/span
span class="code-line"span class="normal"16/span/span
span class="code-line"span class="normal"17/span/span
span class="code-line"span class="normal"18/span/span
span class="code-line"span class="normal"19/span/span
span class="code-line"span class="normal"20/span/span
span class="code-line"span class="normal"21/span/span
span class="code-line"span class="normal"22/span/span
span class="code-line"span class="normal"23/span/span
span class="code-line"span class="normal"24/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="gp gp-VirtualEnv"(gdb)/span span class="go"disassemble checkpass/span/span
span class="code-line"span class="go"Dump of assembler code for function checkpass:/span/span
span class="code-line"span class="go" 0x080486a2 lt;+0gt;: push ebp/span/span
span class="code-line"span class="go" 0x080486a3 lt;+1gt;: mov ebp,esp/span/span
span class="code-line"span class="go" 0x080486a5 lt;+3gt;: sub esp,0x228/span/span
span class="code-line"span class="go" 0x080486ab lt;+9gt;: mov eax,DWORD PTR [ebp+0x8]/span/span
span class="code-line"span class="go" 0x080486ae lt;+12gt;: mov DWORD PTR [esp],eax/span/span
span class="code-line"span class="go" 0x080486b1 lt;+15gt;: call 0x80484d0 lt;strlen@pltgt;/span/span
span class="code-line"span class="go" 0x080486b6 lt;+20gt;: add eax,0x1/span/span
span class="code-line"span class="go" 0x080486b9 lt;+23gt;: mov DWORD PTR [esp+0x8],eax/span/span
span class="code-line"span class="go" 0x080486bd lt;+27gt;: mov eax,DWORD PTR [ebp+0x8]/span/span
span class="code-line"span class="go" 0x080486c0 lt;+30gt;: mov DWORD PTR [esp+0x4],eax/span/span
span class="code-line"span class="go" 0x080486c4 lt;+34gt;: lea eax,[ebp-0x20c]/span/span
span class="code-line"span class="go" 0x080486ca lt;+40gt;: mov DWORD PTR [esp],eax/span/span
span class="code-line"span class="go" 0x080486cd lt;+43gt;: call 0x8048510 lt;strncpy@pltgt;/span/span
span class="code-line"span class="go" 0x080486d2 lt;+48gt;: mov DWORD PTR [esp+0x4],0x8048815/span/span
span class="code-line"span class="go" 0x080486da lt;+56gt;: lea eax,[ebp-0x20c]/span/span
span class="code-line"span class="go" 0x080486e0 lt;+62gt;: mov DWORD PTR [esp],eax/span/span
span class="code-line"span class="go" 0x080486e3 lt;+65gt;: call 0x8048460 lt;strcmp@pltgt;/span/span
span class="code-line"span class="go" 0x080486e8 lt;+70gt;: mov DWORD PTR [ebp-0xc],eax/span/span
span class="code-line"span class="go" 0x080486eb lt;+73gt;: mov eax,DWORD PTR [ebp-0xc]/span/span
span class="code-line"span class="go" 0x080486ee lt;+76gt;: leave /span/span
span class="code-line"span class="go" 0x080486ef lt;+77gt;: ret /span/span
span class="code-line"span class="go"End of assembler dump./span/span
span class="code-line"/code/pre/div
/td/tr/table
pIn the prologue, 0x228 bytes (or 552 bytes) are reserved for local variables and function call arguments./p
pThe interesting call here is the call to codestrncpy/code but we need to examine the call to codestrlen/code first because it looks like output is the third argument to codestrncpy/code./p
pThe call to codestrlen/code:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/span
span class="code-line"span class="normal"3/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="x" 0x080486ab lt;+9gt;: mov eax,DWORD PTR [ebp+0x8]/span/span
span class="code-line"span class="x" 0x080486ae lt;+12gt;: mov DWORD PTR [esp],eax/span/span
span class="code-line"span class="x" 0x080486b1 lt;+15gt;: call 0x80484d0 lt;strlen@pltgt;/span/span
span class="code-line"/code/pre/div
/td/tr/table
pIt's clear the first argument is being used as the argument to codestrlen/code. Return values are normally passed using the EAX register./p
pHere is the call to codestrncpy/code:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/span
span class="code-line"span class="normal"2/span/span
span class="code-line"span class="normal"3/span/span
span class="code-line"span class="normal"4/span/span
span class="code-line"span class="normal"5/span/span
span class="code-line"span class="normal"6/span/span
span class="code-line"span class="normal"7/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="x" 0x080486b6 lt;+20gt;: add eax,0x1/span/span
span class="code-line"span class="x" 0x080486b9 lt;+23gt;: mov DWORD PTR [esp+0x8],eax/span/span
span class="code-line"span class="x" 0x080486bd lt;+27gt;: mov eax,DWORD PTR [ebp+0x8]/span/span
span class="code-line"span class="x" 0x080486c0 lt;+30gt;: mov DWORD PTR [esp+0x4],eax/span/span
span class="code-line"span class="x" 0x080486c4 lt;+34gt;: lea eax,[ebp-0x20c]/span/span
span class="code-line"span class="x" 0x080486ca lt;+40gt;: mov DWORD PTR [esp],eax/span/span
span class="code-line"span class="x" 0x080486cd lt;+43gt;: call 0x8048510 lt;strncpy@pltgt;/span/span
span class="code-line"/code/pre/div
/td/tr/table
pYou can see that 1 is added to the return value and it is put on the stack as the third argument to codestrncpy/code./p
pThe pointer to the function argument is then put on the stack as the second argument (on line 3 and 4)./p
pLastly the address of the local variable is then put on the stack as the first argument (on lines 5 and 6)./p
pHere we can see that the local variable is 0x20c bytes (524 bytes) away from EBP, meaning that we'll need to write 528 bytes until we overwrite EIP using an overflow here, 4 bytes are added for the old EBP saved during the prologue./p
pLooking at the prototype for codestrncpy/code (using codeman strncpy/code), we can see that the first argument is the destination, second the source and third the maximum characters to copy:/p
table class="highlighttable"trtd class="linenos"div class="linenodiv"prespan class="code-line"span class="normal"1/span/pre/div/tdtd class="code"div class="highlight"prespan class="code-line"span/spancodespan class="go" char *strncpy(char *dest, const char *src, size_t n);/span/span
span class="code-line"/code/pre/div
/td/tr/table
pKnowing all of this, its easy to see that there is in fact a buffer overflow here because the developer has used the length of the input buffer to bound the copy function. We can even see how many bytes we have until we overwrite EIP./p
h2Conclusion/h2
pWhile its technically possible to just fuzz all of the application inputs, the more complex the application gets the more infeasible it becomes./p
pThis is also true for reverse engineering every section of an application so its important that you know how to focus on the important parts of the application./p
pUltimately reverse engineering is much more powerful than fuzzing but both should be used in combination to increase efficiency./p
pHappy Hacking :-)/p
I am announcing the next 5 day Windows Internals remote training to be held in January 2022, starting on the 24th according to the followng schedule:
Jan 24 – 2pm to 10pm (all times are based on London time)
Jan 25, 26, 27 – 2pm to 6pm
Jan 31 – 2pm to 10pm
Feb 1, 2, 3 – 2pm to 6pm
The syllabus can be found here (slight changes are possible if new important topics come up).
Cost and Registration
I’m keeping the cost of these training classes relatively low. This is to make these classes accessible to more people, especially in these unusual and challenging times.
Cost: 800 USD if paid by an individual, 1500 USD if paid by a company. Multiple participants from the same company are entitled to a discount (email me for the details). Previous students of my classes are entitled to a 10% discount.
To register, send an email to [email protected] and specify “Windows Internals Training” in the title. The email should include your name, contact email, time zone, and company name (if any).
A while ago, WinDbg added support for a new debugger data model, a change that completely changed the way we can use WinDbg. No more horrible MASM commands and obscure syntax. No more copying addresses or parameters to a Notepad file so that you can use them in the next commands without scrolling up. No more running the same command over and over with different addresses to iterate over a list or an array.
This is part 1 of this guide, because I didn’t actually think anyone would read through 8000 words of me explaining WinDbg commands. So you get 2 posts of 4000 words! That’s better, right?
In this first post we will learn the basics of how to use this new data model — using custom registers and new built-in registers, iterating over objects, searching them and filtering them and customizing them with anonymous types. And finally we will learn how to parse arrays and lists in a much nicer and easier way than you’re used to.
And in the net post we’ll learn the more complicated and fancier methods and features that this data model gives us. Now that we all know what to expect and grabbed another cup of coffee, let’s start!
This data model, accessed in WinDbg through the dx command, is an extremely powerful tool, able to define custom variables, structures, functions and use a wide range of new capabilities. It also lets us search and filter information with LINQ — a natural query language built on top of database languages such as SQL.
This data model is documented and even has usage examples on GitHub. Additionally, all of its modules have documentation that can be viewed in the debugger with dx -v <method> (though you will get the same documentation if you run dx <method> without the -v flag):
dx -v Debugger.Utility.Collections.FromListEntry
Debugger.Utility.Collections.FromListEntry [FromListEntry(ListEntry, [<ModuleName | ModuleObject>], TypeName, FieldExpression) — Method which converts a LIST_ENTRY specified by the ‘ListEntry’ parameter of types whose name is specified by the string ‘TypeName’ and whose embedded links within that type are accessed via an expression specified by the string ‘FieldExpression’ into a collection object. If an optional module name or object is specified, the type name is looked up in the context of such module]
There has also been some external documentation, but I felt like there were things that needed further explanation and that this feature is worth more attention than it receives.
Custom Registers
First, NatVis adds the option for custom registers. Kind of like MASM had @$t1, @$t2, @$t3 , etc. Only now you can call them whatever name you want, and they can have a type of your choice:
dx @$myString = “My String” dx @$myInt = 123
We can see all our variables with dx @$vars and remove them with dx @$vars.Remove("var name"), or clear all with @$vars.Clear(). We can also use dx to show handle more complicated structures, such as an EPROCESS. As you might know, symbols in public PDBs don’t have type information. With the old debugger, this wasn’t always a problem, since in MASM, there’s no types anyway, and we could use the poi command to dereference a pointer.
We first have to use explicit MASM operators to get the address of PsIdleProcess and then print it as an EPROCESS. With dx we can be smarter and cast symbols directly, using c-style casts. But when we try to cast nt!PsInitialSystemProcess to a pointer to an EPROCESS:
dx @$systemProc = (nt!_EPROCESS*)nt!PsInitialSystemProcess Error: No type (or void) for object at Address 0xfffff8074ef843a0
We get an error.
Like I mentioned, symbols have no type. And we can’t cast something with no type. So we need to take the address of the symbol, and cast it to a pointer to the type we want (In this case, PsInitialSystemProcess is already a pointer to an EPROCESS so we need to cast its address to a pointer to a pointer to an EPROCESS).
We can also use ToDisplayString to cast it from a char* to a string. We have two options — ToDisplayString("s"), which will cast it to a string and keep the quotes as part of the string, or ToDisplayString("sb"), which will remove them:
dx ((char*)@$systemProc->ImageFileName).ToDisplayString("sb") ((char*)@$systemProc->ImageFileName).ToDisplayString("sb") : System Length : 0x6
Built-in Registers
This is fun, but for processes (and a few other things) there is an even easier way. Together with NatVis’ implementation in WinDbg we got some “free” registers already containing some useful information — curframe, curprocess, cursession, curstack and curthread. It’s not hard to guess their contents by their names, but let’s take a look:
@$curframe
Gives us information about the current frame. I never actually used it myself, but it might be useful:
A container with information about the current process. This is not an EPROCESS (though it does contain it). It contains easily accessible information about the current process, like its threads, loaded modules, handles, etc.
dx @$curprocess
@$curprocess : System [Switch To] KernelObject [Type: _EPROCESS] Name : System Id : 0x4 Handle : 0xf0f0f0f0 Threads Modules Environment Devices Io
In KernelObject we have the EPROCESS, but we can also use the other fields. For example, we can access all the handles held by the process through @$curprocess.Io.Handles, which will lead us to an array of handles, indexed by their handle number:
System has a lot of handles, these are just the first few! Let’s just take a look at the first one (which we can also access through @$curprocess.Io.Handles[0x4]):
We can see the handle, the type of object the handle is for, its granted access, and we even have a pointer to the object itself (or its object header, to be precise)!
There are plenty more things to find under this register, and I encourage you to investigate them, but I will not show all of them.
By the way, have we mentioned already that dx allows tab completion?
@$cursession
As its name suggests, this register gives us information about the current debugger session:
So, we can get information about our debugger session, which is always fun. But there are more useful things to be found here, such as the Processes field, which is an array of all processes, indexed by their PID. Let’s pick one of them:
Now we can get all that useful information about every single process! We can also search through processes by filtering them based on a search (such as by their name, specific modules loaded into them, strings in their command line, etc. But I will explain all of that later.
@$curstack
This register contains a single field — frames — which shows us the current stack in an easily-handled way:
A very useful thing that NatVis allows us to do, which we briefly mentioned before, is searching, filtering and ordering information in an SQL-like way through Select, Where, OrderBy and more.
For example, let’s try to find all the processes that don’t enable high entropy ASLR. This information is stored in the EPROCESS->MitigationFlags field, and the value for HighEntropyASLREnabled is 0x20 (all values can be found here and in the public symbols).
First, we’ll declare a new register with that value, just to make things more readable:
We can also see everything in decimal by adding , d to the end of the command, to specify the output format (we can also use b for binary, o for octal or s for string):
dx @$cursession.Processes.Select(p => p.Threads.Count()), d
Or, in a slightly more complicated example, see the ideal processor for each thread running in a certain process (I chose a process at random, just to see something that is not the System process):
If we want them in a descending order, we can use OrderByDescending.
But what if we want to pick more than one attribute to see? There is a solution for that too.
Anonymous Types
We can declare a type of our own, that will be unnamed and only used in the scope of our query, using this syntax: Select(x => new { var1 = x.A, var2 = x.B, ...}).
We’ll try it out on one of our previous examples. Let’s say for each process we want to show a process name and its thread count:
dx @$cursession.Processes.Select(p => new {Name = p.Name, ThreadCount = p.Threads.Count()})
But now we only see the process container, not the actual information. To see the information itself we need to go one layer deeper, by using -r2. The number specifies the output recursion level. The default is -r1, -r0 will show no output, -r2 will show two levels, etc.
DX also gives us a new, much easier way, to handle arrays and lists with new syntax. Let’s look at arrays first, where the syntax is dx *(TYPE(*)[Size])<pointer to array start>.
For this example, we will dump the contents on PsInvertedFunctionTable, which contains an array of up to 256 cached modules in its TableEntry field.
First, we will get the pointer of this symbol and cast it to _INVERTED_FUNCTION_TABLE:
Now we can create our array. Unfortunately, the size of the array has to be static and can’t use a register, so we need to input it manually, based on CurrentSize (or just set it to 256, which is the size of the whole array). And we can use the grid view to print it nicely:
We can also do the same thing to see the UserInvertedFunctionTable (right after we switch to user that’s not System), starting from nt!KeUserInvertedFunctionTable:
And of course we can use Select() , Where() or other functions to filter, sort or select only specific fields for our output and get tailored results that fit exactly what we need.
The next thing to handle is lists — Windows is full of linked lists, you can find them everywhere. Linking processes, threads, modules, DPCs, IRPs, and more.
Fortunately the new data model has a very useful Debugger method - Debugger.Utiilty.Collections.FromListEntry, which takes in a linked list head, type and name of the field in this type containing the LIST_ENTRY structure, and will return a container of all the list contents.
So, for our example let’s dump all the handle tables in the system. Our starting point will be the symbol nt!HandleTableListHead, the type of the objects in the list is nt!_HANDLE_TABLE and the field linking the list is HandleTableList:
See the QuotaProcess field? That field points to the process that this handle table belongs to. Since every process has a handle table, this allows us to enumerate all the processes on the system in a way that’s not widely known. This method has been used by rootkits in the past to enumerate processes without being detected by EDR products. So to implement this we just need to Select() the QuotaProcess from each entry in our handle table list. To create a nicer looking output we can also create an anonymous container with the process name, PID and EPROCESS pointer:
dx -r2 (Debugger.Utility.Collections.FromListEntry(*(nt!_LIST_ENTRY*)&nt!HandleTableListHead, "nt!_HANDLE_TABLE", "HandleTableList")).Select(h => new { Object = h.QuotaProcess, Name = ((char*)h.QuotaProcess->ImageFileName).ToDisplayString("s"), PID = (__int64)h.QuotaProcess->UniqueProcessId})
(Debugger.Utility.Collections.FromListEntry(*(nt!_LIST_ENTRY*)&nt!HandleTableListHead, "nt!_HANDLE_TABLE", "HandleTableList")).Select(h => new { Object = h.QuotaProcess, Name = ((char*)h.QuotaProcess->ImageFileName).ToDisplayString("s"), PID = (__int64)h.QuotaProcess->UniqueProcessId})
The first result is the table belonging to the System process and it does not have a QuotaProcess, which is the reason this query returns an error for it. But it should work perfectly for every other entry in the array. If we want to make our output prettier, we can filter out entries where QuotaProcess == 0 before we do the Select():
As we already showed before, we can also print this list in a graphic view or use any LINQ queries to make the output match our needs.
This is the end of our first part, but don’t worry, the second part is right here, and it contains all the fancy new dx methods such as a new disassembler, defining our own methods, conditional breakpoints that actually work, and more.
Welcome to part 2 of me trying to make you enjoy debugging on Windows (wow, I’m a nerd)!
In the first part we got to know the basics of the new debugger data model — Using the new objects, having custom registers, searching and filtering output, declaring anonymous types and parsing lists and arrays. In this part we will learn how to use legacy commands with dx, get to know the amazing new disassembler, create synthetic methods and types, see the fancy changes to breakpoints and use the filesystem from within the debugger.
This sounds like a lot. Because it is. So let’s start!
Legacy Commands
This new data model completely changes the debugging experience. But sometimes you do need to use one of the old commands or extensions that we all got used to, and that don’t have a matching functionality under dx.
But we can still use these under dx with Debugger.Utility.Control.ExecuteCommand, which lets us run a legacy command as part of a dx query. For example, we can use the legacy u command to unassemble the address that is pointed to by RIP in our second stack frame.
Since dx output is decimal by default and legacy commands only take hex input we first need to convert it to hex using ToDisplayString("x"):
Another useful legacy command is !irp. This command supplies us with a lot of information about IRPs, so no need to work hard to recreate it with dx.
So we will try to run !irp for all IRPs in lsass.exe process. Let’s walk through that:
First, we need to find the process container for lsass.exe. We already know how to do that using Where(). Then we’ll pick the first process returned. Usually there should only be one lsass anyway, unless there are server silos on the machine:
Then we need to iterate over IrpList for each thread in the process and get the IRPs themselves. We can easily do that with FromListEntry() that we’ve seen already. Then we only pick the threads that have IRPs in their list:
@$irpThreads = @$lsass.Threads.Select(t => new {irp = Debugger.Utility.Collections.FromListEntry(t.KernelObject.IrpList, "nt!_IRP", "ThreadListEntry")}).Where(t => t.irp.Count() != 0) [0x384] irp [0x0] [Type: _IRP] [<Raw View>] [Type: _IRP] IoStack : Size = 12, Current IRP_MJ_DIRECTORY_CONTROL / 0x2 for Device for "\FileSystem\Ntfs" CurrentThread : 0xffffb90a59477080 [Type: _ETHREAD *] [0x1] [Type: _IRP] [<Raw View>] [Type: _IRP] IoStack : Size = 12, Current IRP_MJ_DIRECTORY_CONTROL / 0x2 for Device for "\FileSystem\Ntfs" CurrentThread : 0xffffb90a59477080 [Type: _ETHREAD *]
We can stop here for a moment, click on IoStack for one of the IRPs (or run with -r5 to see all of them) and get the stack in a nice container we can work with:
dx @$irpThreads.First().irp[0].IoStack
@$irpThreads.First().irp[0].IoStack : Size = 12, Current IRP_MJ_DIRECTORY_CONTROL / 0x2 for Device for "\FileSystem\Ntfs" [0] : IRP_MJ_CREATE / 0x0 for {...} [Type: _IO_STACK_LOCATION] [1] : IRP_MJ_CREATE / 0x0 for {...} [Type: _IO_STACK_LOCATION] [2] : IRP_MJ_CREATE / 0x0 for {...} [Type: _IO_STACK_LOCATION] [3] : IRP_MJ_CREATE / 0x0 for {...} [Type: _IO_STACK_LOCATION] [4] : IRP_MJ_CREATE / 0x0 for {...} [Type: _IO_STACK_LOCATION] [5] : IRP_MJ_CREATE / 0x0 for {...} [Type: _IO_STACK_LOCATION] [6] : IRP_MJ_CREATE / 0x0 for {...} [Type: _IO_STACK_LOCATION] [7] : IRP_MJ_CREATE / 0x0 for {...} [Type: _IO_STACK_LOCATION] [8] : IRP_MJ_CREATE / 0x0 for {...} [Type: _IO_STACK_LOCATION] [9] : IRP_MJ_CREATE / 0x0 for {...} [Type: _IO_STACK_LOCATION] [10] : IRP_MJ_DIRECTORY_CONTROL / 0x2 for Device for "\FileSystem\Ntfs" [Type: _IO_STACK_LOCATION] [11] : IRP_MJ_DIRECTORY_CONTROL / 0x2 for Device for "\FileSystem\FltMgr" [Type: _IO_STACK_LOCATION]
And as the final step we will iterate over every thread, and over every IRP in them, and ExecuteCommand !irp <irp address>. Here too we need casting and ToDisplayString("x") to match the format expected by legacy commands (the output of !irp is very long so we trimmed it down to focus on the interesting data):
@$irpThreads.Select(t => t.irp.Select(i => Debugger.Utility.Control.ExecuteCommand("!irp " + ((__int64)&i).ToDisplayString("x")))) [0x384] [0x0] [0x0] : Irp is active with 12 stacks 11 is current (= 0xffffb90a5b8f4d40) [0x1] : No Mdl: No System Buffer: Thread ffffb90a59477080: Irp stack trace. [0x2] : cmd flg cl Device File Completion-Context [0x3] : [N/A(0), N/A(0)] ... [0x34] : Irp Extension present at 0xffffb90a5b8f4dd0: [0x1] [0x0] : Irp is active with 12 stacks 11 is current (= 0xffffb90a5bd24840) [0x1] : No Mdl: No System Buffer: Thread ffffb90a59477080: Irp stack trace. [0x2] : cmd flg cl Device File Completion-Context [0x3] : [N/A(0), N/A(0)] ... [0x34] : Irp Extension present at 0xffffb90a5bd248d0:
Most of the information given to us by !irp we can get by parsing the IRPs with dx and dumping the IoStack for each. But there are a few things we might have a harder time to get but receive from the legacy command such as the existence and address of an IrpExtension and information about a possible Mdl linked to the Irp.
Disassembler
We used the u command as an example, though in this case there actually is functionality implementing this in dx, through Debugger.Utility.Code.CreateDisassember and DisassembleBlock, creating iterable and searchable disassembly:
Another functionality that we get with this debugger data model is to create functions of our own and use them, with this syntax:
0: kd> dx @$multiplyByThree = (x => x * 3)
@$multiplyByThree = (x => x * 3)
0: kd> dx @$multiplyByThree(5)
@$multiplyByThree(5) : 15
Or we can have functions taking multiple arguments:
0: kd> dx @$add = ((x, y) => x + y)
@$add = ((x, y) => x + y)
0: kd> dx @$add(5, 7)
@$add(5, 7) : 12
Or if we want to really go a few levels up, we can apply these functions to the disassembly output we saw earlier to find all writes into memory in ZwSetInformationProcess. For that there are a few checks we need to apply to each instruction to know whether or not it’s a write into memory:
· Does it have at least 2 operands? For example, ret will have zero and jmp <address> will have one. We only care about cases where one value is being written into some location, which will always require two operands. To verify that we will check for each instruction Operands.Count() > 1.
· Is this a memory reference? We are only interested in writes into memory and want to ignore instructions like mon r10, rcx. To do that, we will check for each instruction its Operands[0].Attributes.IsMemoryReference == true. We check Operands[0] because that will be the destination. If we wanted to find memory reads we would have checked the source, which is in Operands[1].
· Is the destination operand an output? We want to filter out instructions where memory is referenced but not written into. To check that we will use Operands[0].IsOutput == true.
· As our last filter we want to ignore memory writes into the stack, which will look like mov [rsp+0x18], 1 or mov [rbp-0x10], rdx. We will check the register of the first operand and make sure its index is not the rsp index (0x14) or rbp index (0x15).
We will write a function, @$isMemWrite, that receives a block and only returns the instructions that contain a memory write, based in these checks. Then we can create a disassembler, disassemble our target function and only print the memory writes in it:
As another project combining almost everything mentioned here, we can try to create a version of !apc using dx. To simplify we will only look for kernel APCs. To do that, we have a few steps:
Iterate over all the processes using @$cursession.Processes to find the ones containing threads where KTHREAD.ApcState.KernelApcPending is set to 1.
Make a container in the process with only the threads that have pending kernel APCs. Ignore the rest.
For each of these threads, iterate over KTHREAD.ApcState.ApcListHead[0] (contains the kernel APCs) and gather interesting information about them. We can do that with the FromListHead() method we’ve seen earlier. To make our container as similar as possible to !apc, we will only get KernelRoutine and RundownRoutine, though in your implementation you might find there are other fields that interest you as well.
To make the container easier to navigate, collect process name, ID and EPROCESS address, and thread ID and ETHREAD address.
In our implementation we implemented a few helper functions: @$printLn — runs the legacy command ln with the supplied address, to get information about the symbol @$extractBetween — extract a string between two other strings, will be used for getting a substring from the output of @$printLn @$printSymbol — Sends an address to @$printLn and extracts the symbol name only using @$extractSymbol @$apcsForThread — Finds all kernel APCs for a thread and creates a container with their KernelRoutine and RundownRoutine.
We then got all the processes that have threads with pending kernel APCs and saved it into the @$procWithKernelApcs register, and then in a separate command got the APC information using @$apcsForThread. We also cast the EPPROCESS and ETHREAD pointers to void* so dx doesn’t print the whole structure when we print the final result.
This was our way of solving this problem, but there can be others, and yours doesn’t have to be identical to ours!
The script we came up with is:
dx -r0 @$printLn = (a => Debugger.Utility.Control.ExecuteCommand(“ln “+((__int64)a).ToDisplayString(“x”)))
We can also print it as a table, receive information about the processes and be able to explore the APCs of each process separately:
dx -g @$procWithKernelApc.Select(p => new { Name = p.Name, PID = p.PID, Object = p.Object, ApcThreads = p.ApcThreads.Select(t => @$apcsForThread(t))})
But wait, what are all these APCs withnt!EmpCheckErrataList? And why does SearchUI.exe have all of them? What does this process have to do with erratas?
The secret is that there are not actually APCs meant to callnt!EmpCheckErrataList. And no, the symbols are not wrong.
The thing we see here is happening because the compiler is being smart — when it sees a few different functions that have the same code, it makes them all point to the same piece of code, instead of duplicating this code multiple times. You might think that this is not a thing that would happen very often, but lets look at the disassembly for nt!EmpCheckErrataList (the old way this time):
u EmpCheckErrataList
nt!EmpCheckErrataList: fffff807`4eb86010 c20000 ret 0 fffff807`4eb86013 cc int 3 fffff807`4eb86014 cc int 3 fffff807`4eb86015 cc int 3 fffff807`4eb86016 cc int 3
This is actually just a stub. It might be a function that has not been implemented yet (probably the case for this one) or a function that is meant to be a stub for a good reason. The function that is the real KernelRoutine/RundownRoutine of these APCs is nt!KiSchedulerApcNop, and is meant to be a stub on purpose, and has been for many years. And we can see it has the same code and points to the same address:
u nt!KiSchedulerApcNop
nt!EmpCheckErrataList: fffff807`4eb86010 c20000 ret 0 fffff807`4eb86013 cc int 3 fffff807`4eb86014 cc int 3 fffff807`4eb86015 cc int 3 fffff807`4eb86016 cc int 3
So why do we see so many of these APCs?
When a thread is being suspended, the system creates a semaphore and queues an APC to the thread that will wait on that semaphore. The thread will be waiting until someone asks the resume it, and then the system will free the semaphore and the thread will stop waiting and will resume. The APC itself doesn’t need to do much, but it must have a KernelRoutine and a RundownRoutine, so the system places a stub there. In the symbols this stub receives the name of one of the functions that have this “code”, this time nt!EmpCheckErrataList, but it can be a different one in the next version.
Anyone interested in the suspension mechanism can look at ReactOS. The code for these functions changed a bit since, and the stub function was renamed from KiSuspendNop to KiSchedulerApcNop, but the general design stayed similar.
But I got distracted, this is not what this blog was supposed to be talking about. Let’s get back to WinDbg and synthetic functions:
Synthetic Types
After covering synthetic methods, we can also add our own named types and use them to parse data where the type is not available to us.
For example, let’s try to print the PspCreateProcessNotifyRoutine array, which holds all the registered process notify routines — function that are registered by drivers and will receive a notification whenever a process starts. But this array doesn’t contain pointers to the registered routines. Instead it contains pointers to the non-documented EX_CALLBACK_ROUTINE_BLOCK structure.
So to parse this array, we need to make sure WinDbg knows this type — to do that we use Synthetic Types. We start by creating a header file containing all the types we want to define (I used c:\temp\header.h). In this case it’s just EX_CALLBACK_ROUTINE_BLOCK, that we can find in ReactOS:
It’s important to notice that CreateInstance only takes __int64 inputs so any other type has to be cast. It’s good to know this in advance because the error messages these modules return are not always easy to understand.
Now, if we look at our output, and specifically at Context, something seems weird. And actually if we try to dump Function we will see it doesn’t point to any code:
So what happened? The problem is not our cast to EX_CALLBACK_ROUTINE_BLOCK, but the address we are casting. If we dump the values in PspCreateProcessNotifyRoutine we might see what it is:
dx ((void**[0x40])&nt!PspCreateProcessNotifyRoutine).Where(a => a != 0)
The lower half-byte in all of these is 0xF, while we know that pointers in x64 machines are always aligned to 8 bytes, and usually to 0x10. This is because I oversimplified it earlier — these are not pointers to EX_CALLBACK_ROUTINE_BLOCK, they are actually EX_CALLBACK structures (another type that is not in the public pdb), containing an EX_RUNDOWN_REF. But to make this example simpler we will treat them as simple pointers that have been ORed with 0xF, since this is good enough for our purposes. If you ever choose to write a driver that will handle PspCreateProcessNotifyRoutine please do not use this hack, look into ReactOS and do things properly. 😊 So to fix our command we just need to align the addresses to 0x10 before casting them. To do that we do:
But this will be more fun if we could see the symbols instead of the addresses. We already know how to get the symbols by executing the legacy command ln, but this time we will do it with .printf. First we will write a helper function @$getsym which will run the command printf "%y", <address>:
Conditional breakpoints are a huge pain-point when debugging. And with the old MASM syntax they’re almost impossible to use. I spent hours trying to get them to work the way I wanted to, but the command turns out to be so awful that I can’t even understand what I was trying to do, not to mention why it doesn’t filter anything or how to fix it.
Well, these days are over. We can now use dx queries for conditional breakpoints with the following syntax: bp /w “dx query" <address>.
For example, let’s say we are trying to debug an issue involving file opens by Wow64 processes. The function NtOpenProcess is called all the time, but we only care about calls done by Wow64 processes, which are not the majority of processes on modern systems. So to avoid helplessly going through 100 debugger breaks until we get lucky or struggle with MASM-style conditional breakpoints, we can do this instead:
bp /w "@$curprocess.KernelObject.WoW64Process != 0" nt!NtOpenProcess
We then let the machine run, and when the breakpoint is hit we can check if it worked:
Breakpoint 3 hit nt!NtOpenProcess: fffff807`2e96b7e0 4883ec38 sub rsp,38h
The process that triggered our breakpoint is a WoW64 process! For anyone who has ever tried using conditional breakpoints with MASM, this is a life-changing addition.
Other Breakpoint Options
There are a few other interesting breakpoint options found under Debugger.Utility.Control:
SetBreakpointAtSourceLocation — allowing us to set a breakpoint in a module whose source file is available to us, with this syntax: dx Debugger.Utility.Control.SetBreakpointAtSourceLocation("MyModule!myFile.cpp", “172”)
SetBreakpointAtOffset — sets a breakpoint at an offset inside a function — dx Debugger.Utility.Control.SetBreakpointAtOffset("NtOpenFile", 8, “nt")
SetBreakpointForReadWriteFile — similar to the legacy ba command but with more readable syntax, this lets us set a breakpoint to issue a debug break whenever anyone reads or writes to an address. It has default configuration of type = Hardware Write and size = 1. For example, let’s try to break on every read of Ci!g_CiOptions, a variable whose size is 4 bytes:
We let the machine keep running and almost immediately our breakpoint is hit:
0: kd> g Breakpoint 0 hit CI!CiValidateImageHeader+0x51b: fffff807`2f6fcb1b 740c je CI!CiValidateImageHeader+0x529 (fffff807`2f6fcb29)
CI!CiValidateImageHeader read this global variable when validating an image header. In this specific example, we will see reads of this variable very often and writes into it are the more interesting case, as it can show us an attempt to tamper with signature validation.
An interesting thing to notice about these commands in that they don’t just set a breakpoint, they actually return it as an object we can control, which has attributes like IsEnabled, Condition (allowing us to set a condition), PassCount (telling us how many times this breakpoint has been hit) and more.
FileSystem
Under Debugger.Utility we have the FileSystem module, letting us query and control the file system on the host machine (not the machine we are debugging) from within the debugger:
dx -r1 Debugger.Utility.FileSystem
Debugger.Utility.FileSystem
CreateFile [CreateFile(path, [disposition]) - Creates a file at the specified path and returns a file object. 'disposition' can be one of 'CreateAlways' or 'CreateNew']
CreateTempFile [CreateTempFile() - Creates a temporary file in the %TEMP% folder and returns a file object]
CreateTextReader [CreateTextReader(file | path, [encoding]) - Creates a text reader over the specified file. If a path is passed instead of a file, a file is opened at the specified path. 'encoding' can be 'Utf16', 'Utf8', or 'Ascii'. 'Ascii' is the default]
CreateTextWriter [CreateTextWriter(file | path, [encoding]) - Creates a text writer over the specified file. If a path is passed instead of a file, a file is created at the specified path. 'encoding' can be 'Utf16', 'Utf8', or 'Ascii'. 'Ascii' is the default]
CurrentDirectory : C:\WINDOWS\system32
DeleteFile [DeleteFile(path) - Deletes a file at the specified path]
FileExists [FileExists(path) - Checks for the existance of a file at the specified path]
OpenFile [OpenFile(path) - Opens a file read/write at the specified path]
We can create files, open them, write into them, delete them or check if a file exists in a certain path. To see a simple example, let’s dump the contents of our current directory — C:\Windows\System32:
Notice that in this module paths have to have double backslash (“\\”), as they would if we had called the Win32 API ourselves.
As a last exercise we’ll put together a few of the things we learned here — we’re going to create a breakpoint on a kernel variable, get the symbol that accessed it from the stack and write the symbol the accessed it into a file on our host machine.
Let’s break it down into steps:
Open a file to write the results to.
Create a text writer, which we will use to write into the file.
Create a breakpoint for access into a variable. In this case we’ll choose nt!PsInitialSystemProcess and set a breakpoint for read access. We will use the old MASM syntax to run a dx command every time the breakpoint is hit and move on: ba r4 <address> "dx <command>; g" Our command will use @$curstack to get the address that accessed the variable, and then use the @$getsym helper function we wrote earlier to find the symbol for it. We’ll use our text writer to write the result into the file.
ba r4 nt!PsInitialSystemProcess "dx @$txtWriter.WriteLine(@$getsym(@$curstack.Frames[0].Attributes.InstructionOffset)); g"
We let the machine run for as long as we want, and when we want to stop the logging we can disable or clear the breakpoint and close the file with dx @$tmpFile.Close().
Now we can open our @$tmpFile and look at the results:
That’s it! What an amazingly easy way to log information about the debugger!
So that’s the end of our WinDbg series! All the scripts in this series will be uploaded to a github repo, as well as some new ones not included here. I suggest you investigate this data model further, because we didn’t even cover all the different methods it contains. Write cool tools of your own and share them with the world :)
And as long as this guide was, these are not even all the possible options in the new data model. And I didn’t even mention the new support for Javascript! You can get more information about using Javascript in WinDbg and the new and exciting support for TTD (time travel debugging) in this excellent post.
While teaching a Windows Internals class recently, I came across a situation which looked like a bug to me, but turned out to be something I didn’t know about – dynamic symbolic links.
Symbolic links are Windows kernel objects that point to another object. The weird situation in question was when running WinObj from Sysinternals and navigating to the KenrelObjects object manager directory.
WinObj from Sysinternals
You’ll notice some symbolic link objects that look weird: MemoryErrors, PhysicalMemoryChange, HighMemoryCondition, LowMemoryCondition and a few others. The weird thing that is fairly obvious is that these symbolic link objects have empty targets. Double-clicking any one of them confirms no target, and also shows a curious zero handles, as well as quota change of zero:
Symbolic link properties
To add to the confusion, searching for any of them with Process Explorer yields something like this:
It seems these objects are events, and not symbolic links!
My first instinct was that there is a bug in WinObj (I rewrote it recently for Sysinternals, so was certain I introduced a bug). I ran an old WinObj version, but the result was the same. I tried other tools with similar functionality, and still got the same results. Maybe a bug in Process Explorer? Let’s see in the kernel debugger:
Definitely an event and not a symbolic link. What’s going on? I debugged it in WinObj, and indeed the reported object type is a symbolic link. Maybe it’s a bug in the NtQueryDirectoryObject used to query a directory object for an object.
I asked Mark Russinovich, could there be a bug in Windows? Mark remembered that this is not a bug, but a feature of symbolic links, where objects can be created/resolved dynamically when accessing the symbolic link. Let’s see if we can see something in the debugger:
Clearly, there is target, but notice the flags value 0x10. This is the flag indicating the symbolic link is a dynamic one. To get further information, we need to look at the object with a “symbolic link lenses” by using the data structure the kernel uses to represent symbolic links:
The Callback member shows the function that is being called (MiResolveMemoryEvent) that “resolves” the symbolic link to the relevant event. There are currently 11 such events, their names visible with the following:
The main goal of today's post is to show a bit more of what is now possible with the latest Windbg (currently branded "WinDbg Preview" in the Microsoft store) and the time travel debugging tools that Microsoft released a few months ago. When these finally got released, a bit after cppcon2017 this year, I expected a massive pick-up from the security / reverse-engineering industry with a bunch of posts, tools, scripts, etc. To my surprise, this has not happened yet so I have waited patiently for my vacation to write a little something about it myself. So, here goes!
Obviously, one of the most noticeable change in this debugger is the new UI.. but this is not something we will talk about. The second big improvement is .. a decent scripting engine! Until recently, I always had to use pyKD to write automation scripts. This has worked fairly well for years, but I’m glad to move away from it and embrace the new extension model provided by Windbg & Javascript (yes, you read this right). One of the biggest pain point I’ve to deal with with pyKD (aside from the installation process!) is that you had to evaluate many commands and then parse their outputs to extract the bits and pieces you needed. Thankfully, the new debugger data model solves this (or part of this anyway). The third new change is the integration of the time travel debugging (TTD) features discussed in this presentation: Time Travel Debugging: Root Causing Bugs in Commercial Scale Software
.
The goal of this post is to leverage all the nifty stuff we will learn to enumerate x64 try/except handlers in Javascript.
So grab yourself a cup of fine coffee and read on :).
What is being called the debugger data model is a hierarchy of objects (methods, properties, values) that are accessible both directly from the debugger's command window and through a Javascript API. The debugger exposes a bunch of information that it is responsible: thread related information, register values, stack trace information, etc. As an extension writer, you can go and expose your feature through the node of your choosing in the hierarchy. Once it is plugged in into the model, it is available for consumption by another script, or through the debugger's command window.
One really interesting property of this exposed information is that it becomes queryable via operators that have been highly inspired from C#’s LINQ operators. For those who are unfamiliar with them I would suggest looking at Basic LINQ query operations.
First query
Say you would like to find what modules the current @rip is pointing into, you can easily express this through a query using LINQ operators and the data model:
In the previous two samples, there are several interesting points to highlight:
1) dx is the operator to access the data model which is not available through the ?? / ? operators
2) @$name is how you access a variable that you have defined during a debugging session. The debugger itself defines several variables right off the bat just to make querying the model easier: @$curprocess is equivalent to host.currentProcess in Javascript, @cursession is host.currentSession, and @$curthread is host.currentThread. You can also define custom variables yourself, for example:
0:001> dx @$doare = "Diary of a reverse-engineer"
@$doare = "Diary of a reverse-engineer" : Diary of a reverse-engineer
Length : 0x1b
0:001> dx "Hello, " + @$doare
"Hello, " + @$doare : Hello, Diary of a reverse-engineer
Length : 0x22
0:001> ?? @$doare
Bad register error at '@$doare'
0:001> ? @$doare
Bad register error at '@$doare'
3) To query all the nodes in the @$curprocess hierarchy (if you want to wander through the data model you can just use dx Debugger and click through the DML links):
0:001> dx @$curprocess
@$curprocess : cmd.exe [Switch To]
Name : cmd.exe
Id : 0x874
Threads
Modules
Environment
You can also check Debugger.State.DebuggerVariables where you can see the definitions for the variables we just mentioned:
0:001> dx Debugger.State.DebuggerVariables
Debugger.State.DebuggerVariables
cursession : Live user mode: <Local>
curprocess : cmd.exe [Switch To]
curthread : ntdll!DbgUiRemoteBreakin (00007ffc`98675320) [Switch To]
scripts
scriptContents : [object Object]
vars
curstack
curframe : ntdll!DbgBreakPoint [Switch To]
0:001> dx Debugger.State.DebuggerVariables.vars
Debugger.State.DebuggerVariables.vars
doare : Diary of a reverse-engineer
4) Last but not least, most of (all?) the iterable objects can be queried through LINQ-style operators. If you’ve never used these it can be a bit weird at the beginning but at some point it will click and then it is just goodness.
Here is the list of the currently available operators on iterable objects in the data model:
Aggregate [Aggregate(AggregateMethod) | Aggregate(InitialSeed, AggregateMethod) | Aggregate(InitialSeed, AggregateMethod, ResultSelectorMethod) - LINQ equivalent method which iterates through the items in the given collection, running the aggregate method on each one and storing the returned result as the current aggregate value. Once the collection has been exhausted, the final accumulated value is returned. An optional result selector method can be specified which transforms the final accumulator value before returning it.]
All [All(PredicateMethod) - LINQ equivalent method which returns whether all elements in the collection match a given predicate]
AllNonError [AllNonError(PredicateMethod) - LINQ equivalent method which returns whether all elements in the collection match a given predicate. Errors are ignored if all non-error results match the predicate.]
Any [Any(PredicateMethod) - LINQ equivalent method which returns whether any element in the collection matches a given predicate]
Average [Average([ProjectionMethod]) - LINQ equivalent method which finds the average of all values in the enumeration. An optional projection method can be specified that transforms each value before the average is computed.]
Concat [Concat(InnerCollection) - LINQ equivalent method which returns all elements from both collections, including duplicates.]
Contains [Contains(Object, [ComparatorMethod]) - LINQ equivalent method which searches for the given element in the sequence using default comparator rules. An optional comparator method can be provided that will be called each time the element is compared against an entry in the sequence.]
Count [Count() - LINQ equivalent method which returns the number of objects in the collection]
Distinct [Distinct([ComparatorMethod]) - LINQ equivalent method which returns all distinct objects from the given collection, using default comparison rules. An optional comparator method can be provided to be called each time objects in the collection must be compared.]
Except [Except(InnerCollection, [ComparatorMethod]) - LINQ equivalent method which returns all distinct objects in the given collection that are NOT found in the inner collection. An optional comparator method can also be specified.]
First [First([PredicateMethod]) - LINQ equivalent method which returns the first element in the collection or the first which matches an optional predicate]
FirstNonError [FirstNonError([PredicateMethod]) - LINQ equivalent method which returns the first element in the collection or the first which matches an optional predicate. Any errors encountered are ignored if a valid element is found.]
Flatten [Flatten([KeyProjectorMethod]) - Method which flattens a tree of collections (or a tree of keys that project to collections via an optional projector method) into a single collection]
GroupBy [GroupBy(KeyProjectorMethod, [KeyComparatorMethod]) - LINQ equivalent method which groups the collection by unique keys defined via a key projector and optional key comparator]
Intersect [Intersect(InnerCollection, [ComparatorMethod]) - LINQ equivalent method which returns all distinct objects in the given collection that are also found in the inner collection. An optional comparator method can also be specified.]
Join [Join(InnerCollection, Outer key selector method, Inner key selector method, Result selector method, [ComparatorMethod]) - LINQ equivalent method which projects a key for each element of the outer collection and each element of the inner collection using the methods provided. If the projected keys from both these elements match, then the result selector method is called with both those values and its output is returned to the user. An optional comparator method can also be specified.]
Last [Last([PredicateMethod]) - LINQ equivalent method which returns the last element in the collection or the last which matches an optional predicate]
LastNonError [LastNonError([PredicateMethod]) - LINQ equivalent method which returns the last element in the collection or the last which matches an optional predicate. Any errors are ignored.]
Max [Max([ProjectionMethod]) - LINQ equivalent method which returns the maximum element using standard comparison rules. An optional projection method can be specified to project the elements of a sequence before comparing them with each other.]
Min [Min([ProjectionMethod]) - LINQ equivalent method which returns the minimum element using standard comparison rules. An optional projection method can be specified to project the elements of a sequence before comparing them with each other.]
OrderBy [OrderBy(KeyProjectorMethod, [KeyComparatorMethod]) - LINQ equivalent method which orders the collection via a key projector and optional key comparator in ascending order]
OrderByDescending [OrderByDescending(KeyProjectorMethod, [KeyComparatorMethod]) - LINQ equivalent method which orders the collection via a key projector and optional key comparator in descending order]
Reverse [Reverse() - LINQ equivalent method which returns the reverse of the supplied enumeration.]
Select [Select(ProjectionMethod) - LINQ equivalent method which projects the collection to a new collection via calling a projection method on every element]
SequenceEqual [SequenceEqual(InnerCollection, [ComparatorMethod]) - LINQ equivalent method which goes through the outer and inner collections and makes sure that they are equal (incl. sequence length). An optional comparator can be specified.]
Single [Single([PredicateMethod]) - LINQ equivalent method which returns the only element in a list, or, if a predicate was specified, the only element that satisfies the predicate. If there are multiple elements that match the criteria, an error is returned.]
Skip [Skip(Count) - LINQ equivalent method which skips the specified number of elements in the collection and returns all the rest.]
SkipWhile [SkipWhile(PredicateMethod) - LINQ equivalent method which runs the predicate for each element and skips it as long as it keeps returning true. Once the predicate fails, the rest of the collection is returned.]
Sum [Sum([ProjectionMethod]) - LINQ equivalent method which sums all the elements in the collection. Can optionally specify a projector method to transform the elements before summation occurs.]
Take [Take(Count) - LINQ equivalent method which takes the specified number of elements from the collection.]
TakeWhile [TakeWhile(PredicateMethod) - LINQ equivalent method which runs the predicate for each element and returns it only if the result is successful. Once the predicate fails, no more elements will be taken.]
Union [Union(InnerCollection, [ComparatorMethod]) - LINQ equivalent method which returns all distinct objects from the given and inner collection. An optional comparator method can also be specified.]
Where [Where(FilterMethod) - LINQ equivalent method which filters elements in the collection according to when a filter method returns true for a given element]
Now you may be wondering if the model is available with every possible configuration of Windbg? By configuration I mean that you can use the debugger live in user-mode attached to a process, offline looking at a crash-dump of a process, live in kernel-mode, offline looking at a system crash-dump, or off-line looking at a TTD trace.
And yes, the model is accessible with all the previous configurations, and this is awesome. This allows you to, overall, write very generic scripts as long as the information you are mining / exposing is not tied to a specific configuration.
Scripting the model in Javascript
As we described a bit earlier, you can now access programmatically everything that is exposed through the model via Javascript. No more eval or string parsing to extract the information you want, just go find the node exposing what you are after. If this node doesn’t exist, add your own to expose the information you want :)
Javascript integers and Int64
The first thing you need to be aware with Javascript is the fact that integers are encoded in C doubles.. which means your integers are stored in 53 bits. This is definitely a problem as most of the data we deal with are 64 bit integers. In order to address this problem, Windbg exposes a native type to Javascript that is able to store 64 bit integers. The type is called Int64 and most (all?) information available in the data model is through Int64 instances. This type exposes various methods to perform arithmetic and binary operations (if you use the native operators, the Int64 gets converted back to an integer and throws if data is lost during this conversion; cf Auto-conversion). It takes a bit of time to get used to this, but feels natural pretty quickly. Note that the Frida framework exposes a very similar type to address the same issue, which means it will be even easier for you if you have played with Frida in the past!
You can construct an Int64 directly using a native Javascript integers (so at most 53 bits long as described above), or you can use the host.parseInt64 method that takes a string as input. The other very important method you are going to need is Int64.compareTo which returns 1 if the instance is bigger than the argument, 0 if equal and -1 if smaller. The below script shows a summary of the points we touched on:
// Int64.js"use strict";letlogln=function(e){host.diagnostics.debugLog(e+'\n');}functioninvokeScript(){leta=host.Int64(1337);letaplusone=a+1;// 53alogln(aplusone.toString(16));letb=host.parseInt64('0xdeadbeefbaadc0de',16);letbplusone=b.add(1);// 0xdeadbeefbaadc0dflogln(bplusone.toString(16));letbplusonenothrow=b.convertToNumber()+1;// 16045690984229355000logln(bplusonenothrow);try{letbplusonethrow=b+1;}catch(e){// Error: 64 bit value loses precision on conversion to numberlogln(e);}// 1logln(a.compareTo(1));// 0logln(a.compareTo(1337));// -1logln(a.compareTo(1338));}
Registers are accessible in the host.currentThread.Registers object. You can access the classical GPRs in the User node, but you can also access the xmm/ymm registers via SIMD and Floating Point nodes. As you may have guessed, the registers are all instances of the Int64 object we just talked about.
Reading memory
You can read raw memory via the host.memory.readMemoryValues function. It allows you to read memory as an array of items whose size you can specify. You can also use host.memory.readString and host.memory.readWideString for reading (narrow/wide) strings directly from memory.
Even though a bunch of data is accessible programmatically via the data model, not everything is exposed today in the model. For example, you cannot access the same amount of information that kp shows you with the Frame model object. Specifically, the addresses of the frames or the saved return addresses are not currently available in the object unfortunately :-( As a result, being able to evaluate commands can still be important.
The API call ExecuteCommand evaluates a command and returns the output of the command as a string:
There is at least one pitfall with this function to be aware of: the API executes until it completes. So, if you use ExecuteCommand to execute let's say gc the call will return only when you encounter any sort of break. If you don't encounter any break, the call will never end.
Setting breakpoints
Settings breakpoints is basically handled by three different APIs: SetBreakpointAtSourceLocation, SetBreakpointAtOffset, and SetBreakpointForReadWrite. The names are pretty self-explanatory so I will not spend much time describing them. Unfortunately, as far as I can tell there is no easy way to bind a breakpoint to a Javascript function that could handle it when it is hit. The objects returned by these APIs have a Command field you can use to trigger a command when the breakpoint fires, as opposed to a function invocation. In essence, it is pretty much the same than when you do bp foo "command".
Hopefully these APIs will become more powerful and more suited for scripting in future versions with the possibility of invoking a Javascript function when triggered, that would pass an object to the function describing why and where the breakpoint triggered, etc.
Here is a simple example:
//breakpoint.js"use strict";letlogln=function(e){host.diagnostics.debugLog(e+'\n');}functionhandle_bp(){letRegs=host.currentThread.Registers.User;letArgs=[Regs.rcx,Regs.rdx,Regs.r8];letArgsS=Args.map(c=>c.toString(16));letHeapHandle=ArgsS[0];letFlags=ArgsS[1];letSize=ArgsS[2];logln('RtlAllocateHeap: HeapHandle: '+HeapHandle+', Flags: '+Flags+', Size: '+Size);}functioninvokeScript(){letControl=host.namespace.Debugger.Utility.Control;letRegs=host.currentThread.Registers.User;letCurrentProcess=host.currentProcess;letBreakpointAlreadySet=CurrentProcess.Debug.Breakpoints.Any(c=>c.OffsetExpression=='ntdll!RtlAllocateHeap+0x0');if(BreakpointAlreadySet==false){letBp=Control.SetBreakpointAtOffset('RtlAllocateHeap',0,'ntdll');Bp.Command='.echo doare; dx @$scriptContents.handle_bp(); gc';}else{logln('Breakpoint already set.');}logln('Press "g" to run the target.');// let Lines = Control.ExecuteCommand('gc');// for(let Line of Lines) {// logln('Line: ' + Line);// }}
This gives:
0:000>
Press "g" to run the target.
0:000> g-
doare
RtlAllocateHeap: HeapHandle: 0x21b5dcd0000, Flags: 0x140000, Size: 0x82
@$scriptContents.handle_bp()
doare
RtlAllocateHeap: HeapHandle: 0x21b5dcd0000, Flags: 0x140000, Size: 0x9a
@$scriptContents.handle_bp()
doare
RtlAllocateHeap: HeapHandle: 0x21b5dcd0000, Flags: 0x140000, Size: 0x40
@$scriptContents.handle_bp()
doare
RtlAllocateHeap: HeapHandle: 0x21b5dcd0000, Flags: 0x140000, Size: 0x38
@$scriptContents.handle_bp()
doare
RtlAllocateHeap: HeapHandle: 0x21b5dcd0000, Flags: 0x0, Size: 0x48
@$scriptContents.handle_bp()
...
Now, I find this interface not well suited for scenarios where you need to have a breakpoint that just dumps stuff and keep going, but hopefully in the future this will improve. Let's say you have a function and you’re interested in dumping its arguments/state every time it gets called. If you attempt to do this with the above code, every time the breakpoint is hit the debugger will execute your callback and stop. At this point you have to tell it to keep executing. (Also, feel free to uncomment the last lines of the script to see what happens if you ExecuteCommand('gc') :-)).
One way I found around this limitation is to use evaluation and the bp command:
//breakpoint2.js"use strict";letlogln=function(e){host.diagnostics.debugLog(e+'\n');}functionhandle_bp(){letRegs=host.currentThread.Registers.User;letArgs=[Regs.rcx,Regs.rdx,Regs.r8];letArgsS=Args.map(c=>c.toString(16));letHeapHandle=ArgsS[0];letFlags=ArgsS[1];letSize=ArgsS[2];logln('RtlAllocateHeap: HeapHandle: '+HeapHandle+', Flags: '+Flags+', Size: '+Size);if(Args[2].compareTo(0x100)>0){// stop execution if the allocation size is bigger than 0x100returntrue;}// keep the execution going if it's a small sizereturnfalse;}functioninvokeScript(){letControl=host.namespace.Debugger.Utility.Control;letRegs=host.currentThread.Registers.User;letCurrentProcess=host.currentProcess;letHeapAlloc=host.getModuleSymbolAddress('ntdll','RtlAllocateHeap');letBreakpointAlreadySet=CurrentProcess.Debug.Breakpoints.Any(c=>c.Address==HeapAlloc);if(BreakpointAlreadySet==false){logln('RltAllocateHeap @ '+HeapAlloc.toString(16));Control.ExecuteCommand('bp /w "@$scriptContents.handle_bp()" '+HeapAlloc.toString(16));}else{logln('Breakpoint already set.');}logln('Press "g" to run the target.');}
Which gives this output:
0:000>
RltAllocateHeap @ 0x7fffc07587a0
Press "g" to run the target.
0:000> g
RtlAllocateHeap: HeapHandle: 0x21b5dcd0000, Flags: 0x0, Size: 0x48
RtlAllocateHeap: HeapHandle: 0x21b5dcd0000, Flags: 0x140000, Size: 0x38
...
RtlAllocateHeap: HeapHandle: 0x21b5dcd0000, Flags: 0x140000, Size: 0x34a
Breakpoint 0 hit
Time Travel Position: 2A51:314
ntdll!RtlAllocateHeap:
00007fff`c07587a0 48895c2408 mov qword ptr [rsp+8],rbx ss:000000b8`7f39e9a0=000000b87f39e9b0
Of course, yet another way of approaching this problem would be to wrap the script invocation into the command of a breakpoint like this:
bp ntdll!RtlAllocateHeap ".scriptrun c:\foo\script.js"
TTD
For those who are not familiar with Microsoft’s "Time Travel Debugging" toolset, in a nutshell it allows you to record the execution of a process. Once the recording is done, you end up with a trace file written to disk that you can load into the debugger to replay what you just recorded -- a bit like a camera / VCR. If you want to learn more about it, I would highly recommend checking out this presentation: Time Travel Debugging: root causing bugs in commercial scale software.
Even though I won’t cover how recording and replaying a TTD trace in this article, I just wanted to show you in this part how powerful such features can be once coupled with the data model. As you have probably realized by now, the data model is all about extensibility: you can access specific TTD features via the model when you have a trace loaded in the debugger. This section tries to describe them.
TTD.Calls
The first feature I wanted to talked about is TTD.Calls. This API goes through an entire execution trace and finds every unique point in the trace where an API has been called.
0:000> dx -v @$cursession.TTD
@$cursession.TTD : [object Object]
Calls [Returns call information from the trace for the specified set of symbols: TTD.Calls("module!symbol1", "module!symbol2", ...)]
For each of those points, you have an object describing the call: time travel position (that you can travel to: see TimeStart and TimeEnd below), parameters (leveraging symbols if you have any to know how many parameters the API expects), return value, the thread id, etc.
Obviously, the collection returned by TTD.Calls can be queried via the same LINQ-like operators we mentioned earlier which is awesome. As an example, asking the following question has never been easier: "How many times did the allocator fail to allocate memory?":
Note that because the API has been designed in a way that abstracts away ABI-specific details, you can have your query / code working on both x86 & x64 seamlessly. Another important point is that this is much faster than setting a breakpoint manually and running the trace forward to collect this information yourself.
TTD.Memory
The other very powerful feature that was announced fairly recently in version 1.1712.15003 is TTD.Memory. A bit like TTD.Calls, this feature lets you go and find every memory accesses that happened in an execution trace on a specific memory range. And again, it returns to the user a nice object that has all the information you could be potentially interested in (time travel positions, access type, the instruction pointer address, the address of the memory accessed, etc.):
Here is how you would go and ask it to find out every piece of code that write-accessed (read and execute are also other valid type of access you can query for and combine) the TEB region of the current thread:
In the above example, you can see that the TppWorkCallbackPrologRelease function is zeroing the ActivityId GUID of the current TEB - magical.
TTD.Utility.GetHeapAddress
The two previous features were mostly building blocks; this utility consumes the TTD.Calls API in order to show the lifetime of a heap chunk in a trace session. What does that mean exactly? Well, the utility looks for every heap related operation that happened on a chunk (start address, size) and show them to you.
This is extremely useful when debugging or root-causing issues, and here is what it looks like on a dummy trace:
The attentive reader has probably noticed something maybe unexpected with entries 0x59 and entries 0x276 where we are seeing two different allocations of the same chunk without any free in between. The answer to this question lies in the way the GetHeapAddress function is implemented (check out the TTD\Analyzers\HeapAnalysis.js file) - it basically looks for every heap related operation and only shows you the ones where address + size is a range containing the argument you passed. In this example we gave the function the address 0x21b5dce40a0, 0x59 is an allocation and 0x21b5dce40a0 is in the range 0x21b5dce4030 + 0xAA so we display it. Now, a free does not know the size of the chunk, the only thing it knows is the base pointer. In this case if we have a free of 0x21b5dce4030 the utility function would just not display it to us which explains how we can have two heap chunks allocated without a free in the following time frame: ED:7D7, E59:3A7.
We can even go ahead and prove this by finding the free by running the below command:
As expected, the entry 0x64 is our free operation and it also happens in between the two allocation operations we were seeing earlier - solved.
Pretty neat uh?
It is nice enough to ask the utility for a specific heap address, but it would also be super nice if we had access to the whole heap activity that has happened during the session and that is what TTD.Data.Heap gives you:
And of course do not forget that all these collections are queryable. We can easily find out what are all the other heap operations that are not alloc or free with the below query:
After consuming all the various features available in the data model, I am sure you guys are wondering how you can go and add your own node and extend it. In order to do this, you can use the API host.namedModelParent.
class host.namedModelParent
An object representing a modification of the object model of the debugger.
This links together a JavaScript class (or prototype) with a data model.
The JavaScript class (or prototype) becomes a parent data model (e.g.: similar to a prototype)
to the data model registered under the supplied name.
An instance of this object can be returned in the array of records returned from
the initializeScript method.
Let's say we would like to add a node that is associated with a Process called DiaryOfAReverseEngineer which has the following properties:
DiaryOfAReverseEngineer
Foo - string
Bar - string
Add - function
Sub
SubBar - string
SubFoo - string
Step 1: Attach a node to the Process model
Using host.namedModelParent you get the opportunity to link a Javascript class to the model of your choice. The other thing to understand is that this feature is made to be used by extension (as opposed to imperative) scripts.
Extension and imperative scripts are basically the same but they have different entry points: extensions use initializeScript (the command .scriptload invokes this entry point) and imperative scripts use invokeScript (the command .scriptrun invokes both the initializeScript and invokeScript). The small difference is that in an extension script you are expected to return an array of registration objects if you want to modify the data model, which is exactly what we want to do.
Anyway, let's attach a node called DiaryOfAReverseEngineer to the Process model:
//extendmodel_1.js"use strict";classProcessModelParent{getDiaryOfAReverseEngineer(){return'hello from '+this.Name;}}functioninitializeScript(){return[newhost.namedModelParent(ProcessModelParent,'Debugger.Models.Process')];}
Once loaded you can go ahead and check that the node has been added:
0:000> dx @$curprocess
@$curprocess : PING.EXE [Switch To]
Name : PING.EXE
Id : 0x1cfc
Threads
Modules
Environment
TTD
DiaryOfAReverseEngineer : hello from PING.EXE
One important thing to be aware of in the previous example is that the this pointer is effectively an instance of the data model you attached to. In our case it is an instance of the Process model and as a result you can access every property available on this node, like its Name for example.
Step 2: Add the first level to the node
What we want to do now is to have our top node exposing two string properties and one function (we’ll deal with Sub later). This is done by creating a new Javascript class that represents this level, and we can return an instance of this said class in the DiaryOfReverseEngineer property. Simple enough uh?
//extendmodel_2.js"use strict";classDiaryOfAReverseEngineer{constructor(Process){this.process=Process;}getFoo(){return'Foo from '+this.process.Name;}getBar(){return'Bar from '+this.process.Name;}Add(a,b){returna+b;}}classProcessModelParent{getDiaryOfAReverseEngineer(){returnnewDiaryOfAReverseEngineer(this);}}functioninitializeScript(){return[newhost.namedModelParent(ProcessModelParent,'Debugger.Models.Process')];}
Which gives:
0:000> dx @$curprocess
@$curprocess : PING.EXE [Switch To]
Name : PING.EXE
Id : 0x1cfc
Threads
Modules
Environment
TTD
DiaryOfAReverseEngineer : [object Object]
0:000> dx @$curprocess.DiaryOfAReverseEngineer
@$curprocess.DiaryOfAReverseEngineer : [object Object]
process : PING.EXE [Switch To]
Foo : Foo from PING.EXE
Bar : Bar from PING.EXE
From the previous dumps there are at least two things we can do better:
1) The DiaryOfAReverseEngineer node has a string representation of [object Object] which is not great. In order to fix that we can just define our own toString method and return what we want.
2) When displaying the DiaryOfAReverseEngineer node, it displays the instance properties process where we keep a copy of the Process model we attached to. Now, this might be something you want to hide to the user as it has nothing to do with whatever this node is supposed to be about. To solve that, we just have to prefix the field with __.
(If you are wondering why we do not see the method Add you can force dx to display it with the -v flag.)
After fixing the two above points, here is what we have:
// extendmodel_2_1.js"use strict";classDiaryOfAReverseEngineer{constructor(Process){this.__process=process;}getFoo(){return'Foo from '+this.__process.Name;}getBar(){return'Bar from '+this.__process.Name;}Add(a,b){returna+b;}toString(){return'Diary of a reverse-engineer';}}classProcessModelParent{getDiaryOfAReverseEngineer(){returnnewDiaryOfAReverseEngineer(this);}}functioninitializeScript(){return[newhost.namedModelParent(ProcessModelParent,'Debugger.Models.Process')];}
And now if we display the Process model:
0:000> dx @$curprocess
@$curprocess : PING.EXE [Switch To]
Name : PING.EXE
Id : 0x1cfc
Threads
Modules
Environment
TTD
DiaryOfAReverseEngineer : Diary of a reverse-engineer
0:000> dx @$curprocess.DiaryOfAReverseEngineer
@$curprocess.DiaryOfAReverseEngineer : Diary of a reverse-engineer
Foo : Foo from PING.EXE
Bar : Bar from PING.EXE
0:000> dx @$curprocess.DiaryOfAReverseEngineer.Add(1, 2)
@$curprocess.DiaryOfAReverseEngineer.Add(1, 2) : 0x3
Step 3: Adding another level and an iterable class
At this stage, I am pretty sure that you guys are starting to get the hang of it. In order to add a new level, you can just define yet another class, define a property in the DiaryOfAReverseEngineer class and return an instance of it. And that's basically it.
The last concept I wanted to touch on before moving on is how to add the iterable property on one of your data model classes. Let's say you have a class called Attribute that stores a key and a value, and let's also say you have another class called Attributes that is an Attribute store. The thing is, you might have noticed that one class instance usually corresponds to a node with its own properties in the data model view. This is not great for our Attributes class as it is basically an array of Attribute objects, meaning that we will have two copies of everything..
If you want to have the debugger be able to iterate on your instance you can define a *[Symbol.iterator]() method like this:
// extendmodel.js"use strict";classAttribute{constructor(Process,Name,Value){this.__process=Process;this.Name=Name;this.Value=Value;}toString(){letS='Process: '+this.__process.Name+', ';S+='Name: '+this.Name+', ';S+='Value: '+this.Value;returnS;}}classAttributes{constructor(){this.__attrs=[];}push(Attr){this.__attrs.push(Attr);}*[Symbol.iterator](){for(letAttrofthis.__attrs){yieldAttr;}}toString(){return'Attributes';}}classSub{constructor(Process){this.__process=Process;}getSubFoo(){return'SubFoo from '+this.__process.Name;}getSubBar(){return'SubBar from '+this.__process.Name;}getAttributes(){letAttrs=newAttributes();Attrs.push(newAttribute(this.__process,'attr0','value0'));Attrs.push(newAttribute(this.__process,'attr1','value0'));returnAttrs;}toString(){return'Sub module';}}classDiaryOfAReverseEngineer{constructor(Process){this.__process=Process;}getFoo(){return'Foo from '+this.__process.Name;}getBar(){return'Bar from '+this.__process.Name;}Add(a,b){returna+b;}getSub(){returnnewSub(this.__process);}toString(){return'Diary of a reverse-engineer';}}classProcessModelParent{getDiaryOfAReverseEngineer(){returnnewDiaryOfAReverseEngineer(this);}}functioninitializeScript(){return[newhost.namedModelParent(ProcessModelParent,'Debugger.Models.Process')];}
And we can play with the node in the model:
0:000> dx @$curprocess
@$curprocess : PING.EXE [Switch To]
Name : PING.EXE
Id : 0x1cfc
Threads
Modules
Environment
TTD
DiaryOfAReverseEngineer : Diary of a reverse-engineer
0:000> dx @$curprocess.DiaryOfAReverseEngineer
@$curprocess.DiaryOfAReverseEngineer : Diary of a reverse-engineer
Foo : Foo from PING.EXE
Bar : Bar from PING.EXE
Sub : Sub module
0:000> dx @$curprocess.DiaryOfAReverseEngineer.Sub
@$curprocess.DiaryOfAReverseEngineer.Sub : Sub module
SubFoo : SubFoo from PING.EXE
SubBar : SubBar from PING.EXE
Attributes : Attributes
0:000> dx @$curprocess.DiaryOfAReverseEngineer.Sub.Attributes
@$curprocess.DiaryOfAReverseEngineer.Sub.Attributes : Attributes
[0x0] : Process: PING.EXE, Name: attr0, Value: value0
[0x1] : Process: PING.EXE, Name: attr1, Value: value0
0:000> dx @$curprocess.DiaryOfAReverseEngineer.Sub.Attributes[0]
@$curprocess.DiaryOfAReverseEngineer.Sub.Attributes[0] : Process: PING.EXE, Name: attr0, Value: value0
Name : attr0
Value : value0
Another simpler example is available in Determining process architecture with JavaScript and LINQ where the author adds a node to the Process node that tells you with which bitness the process is running on, either 64 or 32 bits.
If you want to extend the data model with best practices you should also have a look at Debugger Data Model Design Considerations which sort of lays down various guidelines.
Misc
In this section I will try to answer a bunch of other questions and share various tricks that have been useful for me - you might learn a thing or two!
Try and play with host.* API from the command window
One of the things I quickly was bothered with at first is not being able to run my Javascript from the command window. Let's say that you want to play with a host.* API: these are not really directly accessible.
A way to work around that is to load a script and to use the @$scriptContents variable from where you can access the host object.
Note that this is also super useful if you want to wander around and get a feel for the various features / APIs that have not been documented yet (or you were just not aware of).
How to load an extension script
The .scriptload command is available in both Windbg Preview and the Windbg from the SDK.
How to run an imperative script
Similar to above, you can use the .scriptrun command for that.
Is the Javascript engine only available in Windbg Preview?
Nope it is not! You can load your Javascript scripts from the latest SDK's Windbg. You can use the .scriptproviders command to know what the various script providers currently loaded are, and if you do not see the Javascript provider you can just run .load jsprovider.dll to load it.
One thing I have not experimented with yet is the .scriptdebug command that lets you debug a script. This is a very important feature as without it it can be a bit of a pain to figure out what is going wrong and where. If you want to know more about this, please refer to Script Debugging Walkthrough from Andy Luhrs.
How to do Nat-Vis style visualizer in Javascript?
I did not cover how to write custom visualizer in Javascript but you should look at host.typeSignatureRegistration to register a class that is responsible for visualizing a type (every property of the class will be used as the main visualizers for the type).
Get a value out of a typed object
Sometimes you are accessing a Javascript object that behaves like a structure instance -- you can access its various fields seamlessly (e.g. you want to access the TEB through the EnvironmentBlock object). This is great. However, for various reasons you might need to get the raw value of a field (e.g. for doing arithmetic) and for that you can use the address property:
Another interesting function I wanted to mention is host.evaluateExpression. As the name suggests, it allows you to evaluate an expression; it is similar to when you use the dx operator but you can only use the language syntax (this means no ‘!’). Any expression you can evaluate through dx, you can evaluate through host.evaluateExpression. The neat thing about this, is that the resulting expression keeps the type information and as a result the Javascript object behaves like the type of the expression.
Here is a small example showing what I am trying to explain:
// host.evaluateExpression"use strict";letlogln=function(e){host.diagnostics.debugLog(e+'\n');}functioninvokeScript(){logln(host.evaluateExpression('(unsigned __int64)0'));logln(host.evaluateExpression('(unsigned __int64*)0'));logln(host.evaluateExpression('(_TEB*)0xb87f4e4000').FlsData);logln(host.evaluateExpression('(_TEB*)0xb87f4e4000').FlsData.address);try{logln(host.evaluateExpression('(unsigned __int64*)0').dereference());}catch(e){logln(e);}// not valid: @$ is not part of the language - logln(host.evaluateExpression('@$teb'));// not valid: @rsp is not part of the language - logln(host.evaluateExpression('(unsigned __int64)@rsp'));// not valid: '!' is not part of the language - logln(host.evaluateExpression('((ntdll!_TEB*)0)'))}
Resulting in:
0:000>
0
[object Object]
[object Object]
2316561115408
Error: Unable to read memory at Address 0x0
How to access global from modules
If you need to get access to a global in a specific module, you can use the function host.getModuleSymbol which returns one of those magic Javascript object behaving like a structure. You can check out an example in the following article: Implementation logic for the COM global interface table.
x64 exception handling vs Javascript
Phew, you made it to the last part! This one is more about trying to do something useful with all the small little things we have learned throughout this article.
I am sure you guys all already know all of this but Windows revisited how exception handling and frame unwinding work on its 64 bit operating systems. Once upon a time the exception handlers could be found directly onto the stack and they formed some sort of linked list. Today, the compiler encodes every static exception handler at compile / link time into various tables embedded into the final binary image.
Anyway, you might know about Windbg's !exchains command that displays the current exception handler chain. This is what the output looks like:
(9a0.14d4): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
except!Fault+0x3d:
00007ff7`a900179d 48c70001000000 mov qword ptr [rax],1 ds:00000000`00000001=????????????????
0:000> !exchain
8 stack frames, scanning for handlers...
Frame 0x01: except!main+0x59 (00007ff7`a9001949)
ehandler except!ILT+900(__GSHandlerCheck_SEH) (00007ff7`a9001389)
Frame 0x03: except!__scrt_common_main_seh+0x127 (00007ff7`a9002327)
ehandler except!ILT+840(__C_specific_handler) (00007ff7`a900134d)
Frame 0x07: ntdll!RtlUserThreadStart+0x21 (00007ff8`3802efb1)
ehandler ntdll!_C_specific_handler (00007ff8`38050ef0)
And here is the associated C code:
// except.c__declspec(noinline)voidFault(uintptr_t*x){printf("I'm about to fault!");*(uintptr_t*)x=1;}intmain(intargc,char**argv){__try{printf("Yo!\n");Fault((uintptr_t*)argc);}__except(Filter()){printf("Exception!");}returnEXIT_SUCCESS;}
As you can see, it is not obvious from the dump above to identify the Filter function and the __except code block.
I figured it would be a good exercise to parse those tables (at least partially) from Javascript, expose the information inside the data model, and write a command similar to !exchain - so let's do it.
A few words about ImageRuntimeFunctionEntries, UnwindInfos, SehScopeTables and CSpecificHandlerDatas
Before giving you the script, I would just like to spend a bit of time to give you a brief overview of how this information is encoded and embedded inside a PE32+ binary. Note that I am only interested by x64 binaries coded in C; in other words I am focusing on SEH (__try / __except) as opposed to C++ EH (try / catch).
The first table we need to look at is the ENTRY_EXCEPTION table that resides in the DataDirectory of the OptionalHeader. This directory is an array of IMAGE_RUNTIME_FUNCTION_ENTRY that describes the boundary of functions (handy for IDA!) and their unwinding information which is stored at the end of this structure.
The unwinding information is mainly described by the UNWIND_INFO structure in which the frame unwinder can find what is necessary to unwind a stack-frame associated to this function. The array of UNWIND_CODE structures basically tells you how to do an epilogue.
What follows this array is variable though (documented here): if the Flags field of UNWIND_INFO specifies the EHHANDLER flag then we have what I call a UNWIND_INFO_END structure defined like this:
This is basically where !exchains stops -- the ehhandler address in the output is the ExceptionHandler field. This is basically an RVA to a function that encapsulates the exception handling for this function. This is not to be confused with either your Filter function or your __except block, this is a generic entry-point that the compiler generates and can be used for other functions too. This function is invoked by the exception dispatching / handling code with an argument that is the value of ExceptionData. ExceptionData is basically an RVA to a blob of memory that the ExceptionHandler function knows how to read and takes actions on. This is where the information we are after is stored.
This is also where it was a bit of a surprise to me, as you basically cannot really tell for sure what type of structure is referenced by ExceptionData. For that, you would have to analyze the ExceptionHandler function to understand what and how this data is used. That is also most likely, why the !exchains command stops here and does not bother trying to parse the exception data blob.
Obviously we can easily make an assumption and assume that the ExceptionData is the structure we would like it to be, and verify that it looks right. In addition, the fact that the code you are most likely looking at has been emitted by a well behaved compiler and that the binary has not been tampered with combined have given me good enough results. But keep in mind that in theory, you could place your own function and have your own ExceptionData format in which case reverse-engineering the handler would be mandatory - in practice this is an unlikely scenario if you are dealing with normal binaries.
The type of ExceptionData that we are interested in is what I call a SEH_SCOPE_TABLE which is an array of SCOPE_RECORDs that are defined like this:
BeginAddress and EndAddress give you the __try block RVA, HandlerAddress encodes either the Filter function or the start of the __finally block. The JumpTarget field tells you if you are looking at either a __try / __except or a __try / __finally. Also, the current heuristic I use to know if the SCOPE_RECORD looks legit or not is to ensure that the __try block resides in between the boundaries of the function the handler is defined in. This has been working well so far - at least on the binaries I have tried it on, but I would not be that surprised if there exists some edge cases to this; if you know any feel free to hit me up!
Putting it all together
All right, so now that we sort of know how to dig out the information we are interested in, you can check the script I came up with: parse_eh_win64.js.
This extends both the Process and the Module models. In both of those models it adds a Functions node as well as a ExceptionHandlers node. Each node under Functions has an ExceptionHandlers node too.
This basically means that you can now:
Get every exception handler registered in the process regardless of which module it is coming from (using Process.ExceptionHandlers)
Get every exception handler registered by a specific module (using Module.ExceptionHandlers)
Get every function in the process address space (using Process.Functions)
Get every function in a specific module (using Module.Functions)
Get every exception handler defined by a specific function (using either Module.Functions[x].ExceptionHandlers or Process.Functions[x].ExceptionHandlers)
With the same source of information we can easily filter and shape the way we want it displayed through the data model. There is no need to display every exception handler from the Module node as it would not be information related to a Module -- this is why we choose to filter it out and display only the ones concerning this Module. Same thing reasoning applies to Functions as well. The model is something you should explore step by step, it is not something where you have all the available information displayed at once - it is meant to be scoped and not overwhelming.
And just in case you forgot about it, all this information is now accessible from the command window for query purposes. You can ask things like Which function defines the most exception handlers? very easily:
In this example, the function KERNEL32!LoadModule seems to be the function that has registered the largest number of exception handlers (12 of them).
Now that we have this new source of information, we can also push it a bit further and implement a command that does a very similar job than !exchain by just mining information from the nodes we just added to the data model:
We could even push it a bit more and have our command returns structured data instead of displaying text on the output so that other commands and extensions could build on top of it.
EOF
Wow, sounds like you made it to the end :-) I hope you enjoyed the post and ideally it will allow you to start scripting Windbg with Javascript pretty quickly. I hope to see more people coming up with new scripts and/or tools based on the various technologies I touched on today.
As usual, big thanks to my buddy yrp604 for proofreading and edits.
If you are still thirsty for more information, here is a collection of links you should probably check out:
Yesterday afternoon, I was peacefully coding some stuff you know but I couldn't make my code working.
As usual, in those type of situations you fire up your debugger in order to understand what is going on under the hood.
That was a bit weird, to give you a bit of context I was doing some inline x86 assembly, and I've put on purpose an int3 just
before the piece of assembly code I thought was buggy. Once my file loaded in OllyDbg2, I hit F9 in order to reach quickly the int3 I've slipped into the inline assembly code. A bit of single-stepping, and BOOM I got a nasty crash. It happens sometimes, and that's uncool.
Then, I relaunch my binary and try to reproduce the bug: same actions and BOOM again. OK, this time it's cool, I got a reproducible crash in OllyDbg2.
I like when things like that happens to me (remember the crashes I've found in OllyDbg/IDA here: PDB Ain't PDD), it's always a nice exercise for me where I've to:
pinpoint the bug in the application: usually not trivial when it's a real/big application
reverse-engineer the codes involved in the bug in order to figure out why it's happening (sometimes I got the sources, sometimes I don't like this time)
In this post, I will show you how I've manage to pinpoint where the bug was, using GFlags, PageHeap and WinDbg. Then, we will reverse-engineer the buggy code in order to understand why the bug is happening, and how we can code a clean trigger.
The first thing I did was to launch WinDbg to debug OllyDbg2 to debug my binary (yeah.). Once OllyDbg2 has been started up, I reproduced exactly the same steps as previously to trigger the bug and here is what WinDbg was telling me:
HEAP[ollydbg.exe]: Heap block at 00987AB0 modified at 00987D88 past
requested size of 2d0
(a60.12ac): Break instruction exception - code 80000003 (first chance)
eax=00987ab0 ebx=00987d88 ecx=76f30b42 edx=001898a5 esi=00987ab0 edi=000002d0
eip=76f90574 esp=00189aec ebp=00189aec iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00200202
ntdll!RtlpBreakPointHeap+0x23:
76f90574 cc int 3
We got a debug message from the heap allocator informing us the process has written outside of its heap buffer. The thing is, this message and the breakpoint are not triggered when the faulty write is done but triggered like after, when another call to the allocator has been made. At this moment, the allocator is checking the chunks are OK and if it sees something weird, it outputs a message and breaks. The stack-trace should confirm that:
As we said just above, the message from the heap allocator has been probably triggered when OllyDbg2 wanted to free a chunk of memory.
Basically, the problem with our issue is the fact we don't know:
where the heap chunk has been allocated
where the faulty write has been made
That's what makes our bug not trivial to debug without the suitable tools. If you want to have more information about debugging heap issues efficiently, you should definitely read the heap chapter in Advanced Windows Debugging (cheers `Ivan).
Pinpointing the heap issue: introducing full PageHeap
In a nutshell, the full PageHeap option is really powerful to diagnostic heap issues, here are at least two reasons why:
it will save where each heap chunk has been allocated
it will allocate a guard page at the end of our chunk (thus when the faulty write occurs, we might have a write access exception)
To do so, this option changes a bit how the allocator works (it adds more meta-data for each heap chunk, etc.) ; if you want more information, try at home allocating stuff with/without page heap and compare the allocated memory. Here is how looks like a heap chunk when PageHeap full is enabled:
To enable it for ollydbg.exe, it's trivial. We just launch the gflags.exe binary (it's in Windbg's directory) and you tick the features you want to enable.
Now, you just have to relaunch your target in WinDbg, reproduce the bug and here is what I get now:
(f48.1140): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=000000b4 ebx=0f919abc ecx=0f00ed30 edx=00000b73 esi=00188694 edi=005d203c
eip=004ce769 esp=00187d60 ebp=00187d80 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
ollydbg!Findfreehardbreakslot+0x21d9:
004ce769 891481 mov dword ptr [ecx+eax*4],edx ds:002b:0f00f000=????????
Woot, this is very cool, because now we know exactly where something is going wrong. Let's get more information about the heap chunk now:
With this really handy command we got a lot of relevant information:
This chunk has a size of 0x2d0 bytes. Thus, starting from 0xf00ed30 to 0xf00efff.
The faulty write now makes sense: the application tries to write 4 bytes outside of its heap buffer (off-by-one on an unsigned array I guess).
The memory has been allocated in ollydbg!Memalloc (called by ollydbg!Getsourceline, PDB related ?). We will study that routine later in the post.
The faulty write occurs at address 0x4ce769.
Looking inside OllyDbg2
We are kind of lucky, the routines involved with this bug are quite simple to reverse-engineer, and Hexrays works just like a charm. Here is the C code (the interesting part at least) of the buggy function:
//ollydbg!buggy @ 0x004CE424signedintbuggy(struct_a1*u){intfile_size;unsignedintnbchar;unsigned__int8*file_content;intnb_lines;intidx;// ...file_content=(unsigned__int8*)Readfile(&u->sourcefile,0,&file_size);// ...nbchar=0;nb_lines=0;while(nbchar<file_size){// doing stuff to count all the char, and all the lines in the file// ...}u->mem1_ov=(unsignedint*)Memalloc(12*(nb_lines+1),3);u->mem2=Memalloc(8*(nb_lines+1),3);if(u->mem1_ov&&u->mem2){nbchar=0;nb_lines2=0;while(nbchar<file_size&&file_content[nbchar]){u->mem1_ov[3*nb_lines2]=nbchar;u->mem1_ov[3*nb_lines2+1]=-1;if(nbchar<file_size){while(file_content[nbchar]){// Consume a line, increment stuff until finding a '\r' or '\n' sequence// ..}}++nb_lines2;}// BOOM!u->mem1_ov[3*nb_lines2]=nbchar;// ...}}
So, let me explain what this routine does:
This routine is called by OllyDbg2 when it finds a PDB database for your binary and, more precisely, when in this database it finds the path of your application's source codes. It's useful to have those kind of information when you are debugging, OllyDbg2 is able to tell you at which line of your C code you're currently at.
* At line 10: "u->Sourcefile" is a string pointer on the path of your source code (found in the PDB database). The routine is just reading the whole file, giving you its size, and a pointer on the file content now stored memory.
* From line 12 to 18: we have a loop counting the total number of lines in your source code.
* At line 20: we have the allocation of our chunk. It allocates 12*(nb_lines + 1) bytes. We saw previously in WinDbg that the size of the chunk was 0x2d0: it should means we have exactly ((0x2d0 / 12) - 1) = 59 lines in our source code:
From line 24 to 39: we have a loop similar to previous one. It's basically counting lines again and initializing the memory we just allocated with some information.
At line 41: we have our bug. Somehow, we can manage to get out of the loop with "nb_lines2 = nb_lines + 1". That means the line 41 will try to write one cell outside of our buffer. In our case, if we have "nb_lines2 = 60" and our heap buffer starting at 0xf00ed30, it means we're going to try to write at (0xf00ed30+6034)=0xf00f000. That's exactly what we saw earlier.
At this point, we have fully explained the bug. If you want to do some dynamic analysis in order to follow important routines, I've made several breakpoints, here they are:
Download the last version of OllyDbg2 here, extract the files
Download the three files from odb2-oob-write-heap, put them in the same directory than ollydbg.exe is
Launch WinDbg and open the last version of OllyDbg2
Set your breakpoints (or not), F5 to launch
Open the trigger in OllyDbg2
Press F9 when the binary is fully loaded
BOOM :). Note that you may not have a visible crash (remember, that's what made our bug not trivial to debug without full pageheap). Try to poke around with the debugger: restarting the binary or closing OllyDbg2 should be enough to get the message from the heap allocator in your debugger.
Fun fact
You can even trigger the bug with only the binary and the PDB database. The trick is to tamper the PDB, and more precisely where it keeps the path to your source code. That way, when OllyDbg2 will load the PDB database, it will read that same database like it's the source code of the application. Awesome.
Conclusion
Those kind of crashes are always an occasion to learn new things. Either it's trivial to debug/repro and you won't waste much of your time, or it's not and you will improve your debugger/reverse-engineer-fu on a real example. So do it!
By the way, I doubt the bug is exploitable and I didn't even try to exploit it ; but if you succeed I would be really glad to read your write-up! But if we assume it's exploitable for a second, you would still have to distribute the PDB file, the source file (I guess it would give you more control than with the PDB) and the binary to your victim. So no big deal.
If you are too lazy to debug your crashes, send them to me, I may have a look at it!
I am announcing the next Windows Internals remote training to be held in July 2021 on the 12, 14, 15, 19, 21. Times: 11am to 7pm, London time.
The syllabus can be found here (slight changes are possible if new important topics come up).
Cost and Registration
I’m keeping the cost of these training classes relatively low. This is to make these classes accessible to more people, especially in these unusual and challenging times.
Cost: 800 USD if paid by an individual, 1500 USD if paid by a company. Multiple participants from the same company are entitled to a discount (email me for the details). Previous students of my classes are entitled to a 10% discount.
To register, send an email to [email protected] and specify “Windows Internals Training” in the title. The email should include your name, contact email, and company name (if any).
Later this year I plan a Windows Kernel Programming class. Stay tuned!
I have recently completed another successful iteration of the Windows Internals training – thank you those who participated!
I am announcing two upcoming training classes, Windows Internals and Windows Kernel Programming.
Windows Internals (5 days)
I promised some folks that the next Internals training would be convenient to US-based time zones. That said, all time zones are welcome!
Dates: Sep 29, Oct 1, 5, 7, 8 Times: 8am to 4pm Pacific time (11am to 7pm Eastern)
The syllabus can be found here. I may make small changes in the final topics, but the major topics remain the same.
Windows Kernel Programming (4 days)
Dates: Oct 13, 15, 19, 21 Times: TBA
The syllabus can be found here. Again, slight changes are possible. This is a development-heavy course, so be prepared to write lots of code!
The selected time zone will be based on the majority of participants’ preference.
Cost and Registration
The cost for each class is kept relatively low (as opposed to other, perhaps similar offerings), as I’ve done in the past year or so. This is to make these classes accessible to more people, especially in these challenging times. If you register for both classes, you get 10% off the second class. Previous students of my classes get 10% off as well.
Cost: 750 USD if paid by an individual, 1500 USD if paid by a company. Multiple participants from the same company are entitled to a discount (email me for the details).
To register, send an email to [email protected] and specify “Training” in the title. The email should include your name, company name (if any) and preferred time zone.
Please read carefully the pre-requisites of each class, especially for Windows Kernel Programming. In case of doubt, talk to me.
Companies that are interested in such (or other) training classes receive special prices. Topics can also be customized according to specific needs.
Other classes I provide include: Modern C++ Programming, Windows System Programming, COM Programming, C#/.NET Programming (Basic and Advanced), Advanced Windows Debugging, and more. Contact me for detailed syllabi if interested.
It’s been a while since I gave the Windows Internals training, so it’s time for another class of my favorite topics!
This time I decided to make it more afordable, to allow more people to participate. The cost is based on whether paid by an individual vs. a company. The training includes lab exercises – some involve working with tools, while others involve coding in C/C++.
Public 5-day remote class
Dates: April 20, 22, 23, 27, 30
Time: 8 hours / day. Exact hours TBD
Price: 750 USD (payed by individual) / 1500 USD (payed by company)
Register by emailing [email protected] and specifying “Windows Internals Training” in the title
Provide names of participants (discount available for multiple participants from the same company), company name (if any) and preferred time zone.
You’ll receive instructions for payment and other details
Virtual space is limited!
The training time zone will be finalized closer to the start date.
Objectives:
Understand the Windows system architectureExplore the internal workings of process, threads, jobs, virtual memory, the I/O system and other mechanisms fundamental to the way Windows works
Write a simple software device driver to access/modify information not available from user mode
Target Audience:
Experienced windows programmers in user mode or kernel mode, interested in writing better programs, by getting a deeper understanding of the internal mechanisms of the windows operating system.Security researchers interested in gaining a deeper understanding of Windows mechanisms (security or otherwise), allowing for more productive research
Pre-Requisites:
Basic knowledge of OS concepts and architecture.Power user level working with Windows
Practical experience developing windows applications is an advantage
C/C++ knowledge is an advantage
Module 1: System Architecture
Brief Windows NT History
Windows Versions
Tools: Windows, Sysinternals, Debugging Tools for Windows
Processes and Threads
Virtual Memory
User mode vs. Kernel mode
Architecture Overview
Key Components
User/kernel transitions
APIs: Win32, Native, .NET, COM, WinRT
Objects and Handles
Sessions
Introduction to WinDbg
Lab: Task manager, Process Explorer, WinDbg
Module 2: Processes & Jobs
Process basics
Creating and terminating processes
Process Internals & Data Structures
The Loader
DLL explicit and implicit linking
Process and thread attributes
Protected processes and PPL
UWP Processes
Minimal and Pico processes
Jobs
Nested jobs
Introduction to Silos
Server Silos and Docker
Lab: viewing process and job information; creating processes; setting job limits
Module 3: Threads
Thread basics
Thread Internals & Data Structures
Creating and terminating threads
Thread Stacks
Thread Priorities
Thread Scheduling
CPU Sets
Direct Switch
Deep Freeze
Thread Synchronization
Lab: creating threads; thread synchronization; viewing thread information; CPU sets
Module 4: Kernel Mechanisms
Trap Dispatching
Interrupts
Interrupt Request Level (IRQL)
Deferred Procedure Calls (DPCs)
Exceptions
System Crash
Object Management
Objects and Handles
Sharing Objects
Thread Synchronization
Synchronization Primitives (Mutex, Semaphore, Events, and more)
Signaled vs. Non-Signaled
High IRQL Synchronization
Windows Global Flags
Kernel Event Tracing
Wow64
Lab: Viewing Handles, Interrupts; creating maximum handles; Thread synchronization
Module 5: Memory Management
Overview
Small, large and huge pages
Page states
Memory Counters
Address Space Layout
Address Translation Mechanisms
Heaps
APIs in User mode and Kernel mode
Page Faults
Page Files
Commit Size and Commit Limit
Workings Sets
Memory Mapped Files (Sections)
Page Frame Database
Other memory management features
Lab: committing & reserving memory; using shared memory; viewing memory related information
Module 6: Management Mechanisms
The Registry
Services
Starting and controlling services
Windows Management Instrumentation
Lab: Viewing and configuring services; Process Monitor
Module 7: I/O System
I/O System overview
Device Drivers
Plug & Play
The Windows Driver Model (WDM)
The Windows Driver Framework (WDF)
WDF: KMDF and UMDF
Device and Driver Objects
I/O Processing and Data Flow
IRPs
Power Management
Driver Verifier
Writing a Software Driver
Labs: viewing driver and device information; writing a software driver
System calls on Windows go through NTDLL.dll, where each system call is invoked by a syscall (x64) or sysenter (x86) CPU instruction, as can be seen from the following output of NtCreateFile from NTDLL:
0:000> u
ntdll!NtCreateFile:
00007ffc`c07fcb50 4c8bd1 mov r10,rcx
00007ffc`c07fcb53 b855000000 mov eax,55h
00007ffc`c07fcb58 f604250803fe7f01 test byte ptr [SharedUserData+0x308 (00000000`7ffe0308)],1
00007ffc`c07fcb60 7503 jne ntdll!NtCreateFile+0x15 (00007ffc`c07fcb65)
00007ffc`c07fcb62 0f05 syscall
00007ffc`c07fcb64 c3 ret
00007ffc`c07fcb65 cd2e int 2Eh
00007ffc`c07fcb67 c3 ret
The important instructions are marked in bold. The value set to EAX is the system service number (0x55 in this case). The syscall instruction follows (the condition tested does not normally cause a branch). syscall causes transition to the kernel into the System Service Dispatcher routine, which is responsible for dispatching to the real system call implementation within the Executive. I will not go to the exact details here, but eventually, the EAX register must be used as a lookup index into the System Service Dispatch Table (SSDT), where each system service number (index) should point to the actual routine.
On x64 versions of Windows, the SSDT is available in the kernel debugger in the nt!KiServiceTable symbol:
You might expect the values in the SSDT to be 64-bit pointers, pointing directly to the system services (this is the scheme used on x86 systems). On x64 the values are 32 bit, and are used as offsets from the start of the SSDT itself. However, the offset does not include the last hex digit (4 bits): this last value is the number of arguments to the system call.
Let’s see if this holds with NtCreateFile. Its service number is 0x55 as we’ve seen from user mode, so to get to the actual offset, we need to perform a simple calculation:
Indeed – this is NtCreateFile. What about the argument count? The value stored is 7. Here is the prototype of NtCreateFile (documented in the WDK as ZwCreateFile):
Clearly, there are 11 parameters, not just 7. Why the discrepency? The stored value is the number of parameters that are passed using the stack. In x64 calling convention, the first 4 arguments are passed using registers: RCX, RDX, R8, R9 (in this order).
Now back to the title of this post. Here are the first few entries in the SSDT again:
The first two entries look different, with much larger numbers. Let’s try to apply the same logic for the first value (index 0):
kd> u nt!KiServiceTable+fced720
fffff804`2392c340 ?? ???
^ Memory access error in 'u nt!KiServiceTable+fced720'
Clearly a bust. The value is in fact a negative value (in two’s complement), so we need to sign-extend it to 64 bit, and then perform the addition (leaving out the last hex digit as before):
kd> u nt!KiServiceTable+ffffffff`ffced720
nt!NtAccessCheck:
fffff804`1392c340 4c8bdc mov r11,rsp
fffff804`1392c343 4883ec68 sub rsp,68h
fffff804`1392c347 488b8424a8000000 mov rax,qword ptr [rsp+0A8h]
This is NtAccessCheck. The function’s implementation is in lower addresses than the SSDT itself. Let’s try the same exercise with index 1:
With 2020 over, I’ll be releasing a bunch of new anti-debug methods that you most likely have never seen. To start off, we’ll take a look at two new methods, both relating to thread suspension. They aren’t the most revolutionary or useful, but I’m keeping the best for last.
Bypassing process freeze
This one is a cute little thread creation flag that Microsoft added into 19H1. Ever wondered why there is a hole in thread creation flags? Well, the hole has been filled with a flag that I’ll call THREAD_CREATE_FLAGS_BYPASS_PROCESS_FREEZE (I have no idea what it’s actually called) whose value is, naturally, 0x40.
To demonstrate what it does, I’ll show how PsSuspendProcess works:
NTSTATUSPsSuspendProcess(_EPROCESS*Process){constautocurrentThread=KeGetCurrentThread();KeEnterCriticalRegionThread(currentThread);NTSTATUSstatus=STATUS_SUCCESS;if(ExAcquireRundownProtection(&Process->RundownProtect)){autotargetThread=PsGetNextProcessThread(Process,nullptr);while(targetThread){// Our flag in actionif(!targetThread->Tcb.MiscFlags.BypassProcessFreeze)PsSuspendThread(targetThread,nullptr);targetThread=PsGetNextProcessThread(Process,targetThread);}ExReleaseRundownProtection(&Process->RundownProtect);}elsestatus=STATUS_PROCESS_IS_TERMINATING;if(Process->Flags3.EnableThreadSuspendResumeLogging)EtwTiLogSuspendResumeProcess(status,Process,Process,0);KeLeaveCriticalRegionThread(currentThread);returnstatus;}
So as you can see, NtSuspendProcess that calls PsSuspendProcess will simply ignore the thread with this flag. Another bonus is that the thread also doesn’t get suspended by NtDebugActiveProcess! As far as I know, there is no way to query or disable the flag once a thread has been created with it, so you can’t do much against it.
As far as its usefulness goes, I’d say this is just a nice little extra against dumping and causes confusion when you click suspend in Processhacker, and the process continues to chug on as if nothing happened.
Example
For example, here is a somewhat funny code that will keep printing I am running. I am sure that seeing this while reversing would cause a lot of confusion about why the hell one would suspend his own process.
#define THREAD_CREATE_FLAGS_BYPASS_PROCESS_FREEZE 0x40
NTSTATUSprinter(void*){while(true){std::puts("I am running\n");Sleep(1000);}returnSTATUS_SUCCESS;}HANDLEhandle;NtCreateThreadEx(&handle,MAXIMUM_ALLOWED,nullptr,NtCurrentProcess(),&printer,nullptr,THREAD_CREATE_FLAGS_BYPASS_PROCESS_FREEZE,0,0,0,nullptr);NtSuspendProcess(NtCurrentProcess());
Suspend me more
Continuing the trend of NtSuspendProcess being badly behaved, we’ll again abuse how it works to detect whether our process was suspended.
The trick lies in the fact that suspend count is a signed 8-bit value. Just like for the previous one, here’s some code to give you an understanding of the inner workings:
If you take a look at the first code sample with PsSuspendProcess it has no error checking and doesn’t care if you can’t suspend a thread anymore. So what happens when you call NtResumeProcess? It decrements the suspend count! All we need to do is max it out, and when someone decides to suspend and resume us, they’ll actually leave the count in a state it wasn’t previously in.
Example
The simple code below is rather effective:
Visual Studio - prevents it from pausing the process once attached.
WinDbg - gets detected on attach.
x64dbg - pause button becomes sketchy with error messages like “Program is not running” until you manually switch to the main thread.
ScyllaHide - older versions used NtSuspendProcess and caused it to be detected, but it was fixed once I reported it.
for(size_ti=0;i<128;++i)NtSuspendThread(thread,nullptr);while(true){if(NtSuspendThread(thread,nullptr)!=STATUS_SUSPEND_COUNT_EXCEEDED)std::puts("I was suspended\n");Sleep(1000);}
Conclusion
If anything, I hope that this demonstrated that it’s best not to rely on NtSuspendProcess to work as well as you’d expect for tools dealing with potentially malicious or protected code. Hope you liked this post and expect more content to come out in the upcoming weeks.
If you'd like to follow along, I'll be using system files from Windows x64 10.0.15063 (Creator's Update). All pseudo-source and disassembly is reconstructed from that specific release.
Don't have a kernel debugging environment set up? Don't fret. You can follow our tutorial on how to setup basic kernel debugging using WinDbg and VMware here.
Without further ado, let's begin.
What do these callbacks do?
These callbacks can be used by driver developers to gain notifications when certain events happen. For example, the basic process creation callback, nt!PsSetCreateProcessNotifyRoutine, registers a user-defined function pointer ("NotifyRoutine") that will be invoked by Windows each time a process is created or deleted. As part of the event notification, the supplied handler gets a wealth of information. In our example, this will include the parent process' (if one exists) PID, the actual process' PID, and a boolean value that will let us know if the process is being created or if it's terminating.
Security software leverages these callbacks to be able to carefully inspect code running on the machine.
Divin' deep
The documented APIs
Our investigation has to begin somewhere. What better place than at the start of a documented function? We turn to nt!PsSetCreateProcessNotifyRoutine. MSDN claims that this routine has been around since Windows 2000. Even our friends at ReactOS seem to have implemented this functionality a long time ago. We'll see exactly how (if at all) things have changed in the 17 years from Windows 2000 until now.
The only difference is in the second parameter being passed to nt!PspSetCreateProcessNotifyRoutine. These are effectively flags. In the base case (nt!PsSetCreateProcessNotifyRoutine), these flags can either be 1 or 0 depending on the state of the "Remove" parameter. If "Remove" is TRUE, Flags=1. If "Remove" is FALSE, Flags=0. In the extended case (nt!PsSetCreateProcessNotifyRoutineEx), the flags can take on the value 2 or 3:
Finally, for nt!PsSetCreateProcessNotifyRoutineEx2, these flags will take on the value 6 or 7:
Therefore, one can imply that the flags passed to nt!PspSetCreateProcessNotifyRoutine have this definition:
The undocumented world
nt!PspSetCreateProcessNotifyRoutine is slightly complicated. I've defined it below, but I strongly recommend opening it in another window and following the text to ease understanding.
Luckily for us, a lot of the internal data structures related to callback routines haven't changed since Windows 2000. The trailblazers at ReactOS have been spot-on with their structure definitions so we'll use them, when possible, to avoid duplicating work.
For each callback, there's a global array that can contain up to 64 entries. In our case, the start of this array for process creation callbacks is located at nt!PspCreateProcessNotifyRoutine. Each entry in this array is of type _EX_CALLBACK:
To avoid synchronization problems, nt!ExReferenceCallBackBlock is used which will safely acquire a reference to the underlying callback object, _EX_CALLBACK_ROUTINE_BLOCK (documented below). We can effectively reproduce the same behavior in a non-thread safe way via:
If we're deleting a callback object ("Remove" is TRUE), we need to make sure that we can find the appropriate _EX_CALLBACK_ROUTINE_BLOCK in the array. This is done by checking first if the target "NotifyRoutine" matches that of the current _EX_CALLBACK_ROUTINE with nt!ExGetCallBackBlockRoutine:
Then, we check to see if it's the right type (created with the correct version of (nt!PsSetCreateProcessNotifyRoutine/Ex/Ex2), by using nt!ExGetCallBackBlockContext:
At this point, we've found the entry in the array. We will erase it by setting the _EX_CALLBACK value to NULL via nt!ExCompareExchangeCallback, decrementing the appropriate global counter (nt!PspCreateProcessNotifyRoutineExCount or nt!PspCreateProcessNotifyRoutineCount), dereferencing the _EX_CALLBACK_ROUTINE_BLOCK with nt!ExDereferenceCallBackBlock, waiting for any other code using the _EX_CALLBACK (nt!ExWaitForCallBacks), and finally freeing memory (nt!ExFreePoolWithTag). As you can see, great care is taken by Microsoft to not free a callback object that is in use.
If we can't find the entry to remove in the nt!PspCreateProcessNotifyRoutine array after exhausting all 64 possibilities, the STATUS_PROCEDURE_NOT_FOUND error message is returned.
On the other hand, if we're adding a new entry into the callback array, things are a little easier. A sanity check is performed by nt!MmVerifyCallbackFunctionCheckFlags to ensure that the "NotifyRoutine" is present in a loaded module. This helps avoid unlinked drivers (or shellcode) from receiving callback events:
After we pass the sanity check, an _EX_CALLBACK_ROUTINE_BLOCK is allocated via nt!ExAllocateCallBack. This routine confirms the size and layout of the _EX_CALLBACK_ROUTINE_BLOCK structure:
To wrap up, the newly allocated _EX_CALLBACK_ROUTINE_BLOCK is added to a free (NULL) location in the nt!PspCreateProcessNotifyRoutine array using nt!ExCompareExchangeCallBack (ensuring that it doesn't overflow the 64 limit maximum). Finally, the appropriate global counter is incremented and a global flag is set in nt!PspNotifyEnableMask denoting that there are callbacks of the user-specified type registered on the system.
The other callbacks
Thankfully, thread and image creation callbacks are very similar to process callbacks. They utilize the same underlying data structures. The only difference is that thread creation/termination callbacks are stored in the nt!PspCreateThreadNotifyRoutine array and that image load notification callbacks are stored in nt!PspLoadImageNotifyRoutine.
The script
It's finally time to put what we know to good use. Using WinDbg, we can create a simple script to automagically enumerate process, thread, and image callback routines.
Instead of leveraging WinDbg's built-in scripting engine, I've elected to use something a little less disgusting. There's a great 3rd party extension for WinDbg called PyKd that enables Python scripting in WinDbg. Installing it is very straightforward. You'll need a copy of the appropriate bitness (e.g. 64-bit for 64-bit install of WinDbg) of Python for this to work.
The script should be easy to follow. I tried to document it as best I could. It should also be compatible, at a minimum, with all forms of Windows from XP and up (both 32-bit and 64-bit flavors).
After running the script using the "!py" command, you should see output similar to this:
Final thoughts
Knowing how the callback system functions in Windows allows us to do very interesting things. As seen above, we're able to programmatically iterate through each callback array and discover all registered callbacks. This is very useful for forensic purposes.
Furthermore, these underlying array lists aren't under the protection of PatchGuard. Since registering callbacks is more-or-less a requirement for anti-virus products in order to develop a useful driver that plays nicely with PatchGuard on x64 systems, malware could dynamically disable (or replace) these registered callbacks to thwart security protection solutions. The possibilities are endless.
Special thanks to the folks at ReactOS for their meticulous documentation. In particular, most of the structures I used were identified by Alex Ionescu for ReactOS a long time ago. Additionally, kudos to the folks that make PyKd. It's a much better alternative to the native scripting interface for WinDbg, in my opinion!
As always, if y'all have any questions or comments, please feel free to comment below. Suggestions are greatly appreciated too!
This blog post will go over an assumption made over a decade ago by Microsoft when dealing with software breakpoints that can be used to reveal the presence of most (all publicly available?) usermode and kernelmode debuggers.
The x86 architecture can potentially encode a particular assembly instruction in multiple ways. For example, adding two registers, eax and ebx, and storing the result in eax takes the following mnemonic form: add eax, ebx. This can be encoded as the byte sequence 0x03 0xC3or 0x01 0xD8. Fundamentally, the machine code represents the same assembly operation.
If you're just interested in the anti-debug trick (without any context on why it works the way it does), scroll to the bottom of this post. For the rest of you brave enough to read this article in its entirety... buckle up.
The "long form" of int 3
An int 3 can be encoded as either a single-byte 0xCC or via the more unconventional way as the multi-byte sequence 0xCD 0x03:
So, what happens when Windows encounters a multi-byte int 3? We create a simple C++ program to find out:
After you run this application, you should see output similar to this:
A single-byte int 3 (0xCC) works as expected. The start of the stub is located at 0x000001BE94B90000. When the stub is executed, the exception handler fires and we see that both the _EXCEPTION_RECORD.ExceptionAddress and _CONTEXT.Rip are located at 0x000001BE94B90000. This is the start of the int 3 instruction. Excellent!
The multi-byte int 3 (0xCD 0x03) is located at address 0x000001BE94B90002. When this stub executes, the exception handler proclaims that the _EXCEPTION_RECORD.ExceptionAddress and _CONTEXT.Rip are located at 0x000001BE94B90003. This is in the middle of the int 3 instruction. Why? What went wrong?
The assumption
NOTE: From this point on, all disassembly and pseudo-source is reconstructed from system files that are provided with Windows x64 10.0.15063 (Creator's Update). If you'd like to follow along, make sure you use the same version I'm using!
Microsoft assumes that all int 3's result from the single-byte variant.
This assumption occurs very early during interrupt processing. Namely, when any interrupt occurs, such as when an int 3 is executed by the processor, control is redirected by the CPU to a handler registered in the appropriate position of the IDT (Interrupt Descriptor Table). In Windows, the handler for software breakpoints can be found at the symbol nt!KiBreakpointTrap:
The first thing nt!KiBreakpointTrap does is generate a trap frame (_KTRAP_FRAME) on the stack that it passes to subsequent routines. A definition of this structure can be found below:
Parts of this structure are automatically filled by the CPU when the interrupt fires, in particular, the range from +0x160 (_KTRAP_FRAME.ErrorCode) to +0x188 (_KTRAP_FRAME.SegSs):
The _KTRAP_FRAME is essentially an extension of the elements saved on the stack by the CPU. It's purpose is to provide a place to store volatile registers which can be clobbered when calling into functions that are compiled in C.
A very important thing to note is that the instruction pointer (EIP) saved by the CPU on the stack (_KTRAP_FRAME.Rip) will be set to the instruction immediately following the one that caused entry into the handler. In our scenario, this means that the _KTRAP_FRAME.Rip member will be the instruction following our int 3, which will be ret (0xC3) in the example code above.
After the volatile registers have been saved off, nt!KiBreakpointTrap performs a quick check to see whether the interrupt fired from usermode (ring3) or kernelmode code (ring0). If execution is coming from ring3, a swapgs needs to occur as well as some other bookkeeping with debug registers.
Eventually, control flow will reconvene and the volatile floating point registers will also be stored off into the _KTRAP_FRAME. Before entering into more exception handling logic, the instruction pointer will be extracted from _KTRAP_FRAME.Rip (saved by the CPU upon entering nt!KiBreakpointTrap), decremented by one, and passed as an argument to nt!KiExceptionDispatch. Additionally, the exception code, EXCEPTION_BREAKPOINT (0x80000003), will also be passed in. The prototype for nt!KiExceptionDispatch:
It's important to note that nt!KiExceptionDispatch (like nt!KiBreakpointTrap) is written in hand-ASM. It assumes that ecx contains the exception code, edx is the number of exception parameters (up to 3), r8 contains the address of the exception, r9 is the first exception parameter (if one exists), r10 is the second exception parameter (if one exists), r11 is the third exception parameter (if one exists), and rbp points to a segment in the _KTRAP_FRAME structure (at offset +0x80).
Upon entry of nt!KiExceptionDispatch, the first thing that occurs is the generation of a _KEXCEPTION_FRAME. Whereas the _KTRAP_FRAME was used to store volatile registers, the _KEXCEPTION_FRAME provides a place to save all nonvolatile registers:
nt!KiExceptionDispatch also creates an _EXCEPTION_RECORD structure on the stack. If you've done any error handling in Windows (in either usermode or kernelmode), you'll be familiar with this data structure as it is contained as a child within the _EXCEPTION_POINTERS data structure. We use both of these structures in our example above.
Furthermore, this explains the first part of our mystery, namely, why the _EXCEPTION_RECORD.ExceptionAddress is incorrect. Recall that the _EXCEPTION_RECORD.ExceptionAddress is populated by the 3rd (r8) argument to nt!KiExceptionDispatch. This was passed in from nt!KiBreakpointTrap. This argument is a copy of the _KTRAP_FRAME.Rip member decremented by one.
To figure out where the _CONTEXT.Rip member is populated, we need to go deeper down the rabbit hole.
nt!KiExceptionDispatch will call into nt!KiDispatchException (yes, the ordering of the words are intentionally flipped) passing in the recently created _EXCEPTION_RECORD and _KEXCEPTION_FRAME:
This function will build a _CONTEXT out of the _KTRAP_FRAME and _KEXCEPTION_FRAME by invoking the helper routine KeContextFromKFrame. After the _CONTEXT is created, a check is made against the _EXCEPTION_RECORD.ExceptionCode (received as an argument from nt!KiExceptionDispatch) for STATUS_BREAKPOINT (0x80000003). If it's true, the _CONTEXT.Rip member will be decremented:
This solves the last part of the mystery and causes the value in _CONTEXT.Rip to be tainted.
The anti-debug trick
Knowing what we know about how Windows handles the different types of int 3s, is it possible to leverage this discrepancy in a useful way? The answer is yes.
Debuggers display the state of the program at the time of an exception. Since Windows will incorrectly assume that our int 3 exception was generated from the single-byte variant, it is possible to confuse the debugger into reading "extra" memory. We leverage this inconsistency to trip a "guard page" of sorts.
As we saw in our first example (at the start of the article), when a multi-byte int 3 occurs, the _EXCEPTION_RECORD.ExceptionAddress and _CONTEXT.Rip values will lie in the middle of our multi-byte instruction instead of at the start. This means that the debugger will incorrectly determine that the instruction which threw the software breakpoint begins with the opcode 0x03. Referring to the trusty Intel manual, we can see that this opcode represents a 2-byte add instruction:
What would happen if we positioned our multi-byte int 3 near the end of a page of memory?
When the operating system notifies our attached debugger of the breakpoint exception, the instruction pointer will point to memory that will be misinterpreted as the start of an add (0x03) instruction. This will cause the debugger to disassemble data on the adjacent page (since this instruction is 2 bytes long), and effectively read one byte past our "valid" memory range.
Our trick relies on the fact that Windows, as an optimization, will not commit virtual memory to physical RAM unless it absolutely needs it. That is to say that most memory, especially in usermode, is paged. When memory needs to be made available for use that is not currently in physical RAM, a page fault occurs. To learn more about memory management, check out the following articles on our site: Introduction to IA-32e hardware paging and Exploring Windows virtual memory management.
So, we can detect the memory read on this adjacent page by inspecting the corresponding PTE (Page Table Entry) using the QueryWorkingSetEx API. If the page is resident in our process' working set (e.g. mapped into our process by the debugger), the Valid bit in the _PSAPI_WORKING_SET_EX_BLOCK will be set.
PoC||GTFO
A full example can be found below:
As always, if you have any questions or comments, please feel free to send us a message below. Happy hacking 😎.
In a previous post, we discussed the IA-32e 64-bit paging structures, and how they can be used to turn virtual addresses into physical addresses. They're a simple but elegant way to manage virtual address mappings as well as page permissions with varying granularity of page sizes. All of which is provided by the architecture. But as one might expect, once you add an operating system like Windows into the mix, things get a little more interesting.
The problem of per-process memory
In Windows, a process is nothing more than a simple container of threads and metadata that represents a user-mode application. It has its own memory so that it can manage the different pieces of data and code that make the process do something useful. Let's consider, then, two processes that both try to read and write from the memory located at the virtual address 0x00000000`11223344. Based on what we know about paging, we expect that the virtual address is going to end up translating into the same physical address (let's say 0x00000001`ff003344 as an example) in both processes. There is, after all, only one CR3 register per processor, and the hardware dictates that the paging structure root is located in that register.
Figure 1: If the two process' virtual addresses would translate to the same physical address, then we expect that they would both see the same memory, right?
Of course, in reality we know that it can't work that way. If we use one process to write to a virtual memory address, and then use another process to read from that address, we shouldn't get the same value. That would be devastating from a security and stability standpoint. In fact, the same permissions may not even be applied to that virtual memory in both processes.
But how does Windows accomplish this separation? It's actually pretty straightforward: when switching threads in kernel-mode or user-mode (called a context switch), Windows stores off or loads information about the current thread including the state of all of the registers. Because of this, Windows is able to swap out the root of the paging structures when the thread context is switched by changing the value of CR3, effectively allowing it to manage an entirely separate set of paging structures for each process on the system. This gives each process a unique mapping of virtual memory to physical memory, while still using the same virtual address ranges as another process. The PML4 table pointer for each user-mode process is stored in the DirectoryTableBase member of an internal kernel structure called the EPROCESS, which also manages a great deal of other state and metadata about the process.
Figure 2: In reality, each process has its own set of paging structures, and Windows swaps out the value of the CR3 register when it executes within that process. This allows virtual addresses in each process to map to different physical addresses.
We can see the paging structure swap between processes for ourselves if we do a little bit of exploration using WinDbg. If you haven't already set up kernel debugging, you should check out this article to get yourself started. Then follow along below.
Let's first get a list of processes running on the target system. We can do that using the !process command. For more details on how to use this command, consider checking out the documentation using .hh !process. In our case, we pass parameters of zero to show all processes on the system.
We can use notepad.exe as our target process, but you should be able to follow along with virtually any process of your choice. The next thing we need to do is attach ourselves to this process - simply put, we need to be in this process' context. This lets us access the virtual memory of notepad.exe by remapping the paging structures. We can verify that the context switch is happening by watching what happens to the CR3 register. If the virtual memory we have access to is going to change, we expect that the value of CR3 will change to new paging structures that represent notepad.exe's virtual memory. Let's take a look at the value of CR3 before the context switch.
We know that this value should change to the DirectoryTableBase member of the EPROCESS structure that represents notepad.exe when we make the switch. As a matter of interest, we can take a look at that structure and see what it contains. The PROCESS fffffa8019218b10 line emitted by the debugger when we listed all processes is actually the virtual address of that process' EPROCESS structure.
The fully expanded EPROCESS structure is massive, so everything after what we're interested in has been omitted from the results above. We can see, though, that the DirectoryTableBase is a member at +0x028 of the process control block (KPROCESS) structure that's embedded as part of the larger EPROCESS structure.
According to this output, we should expect that CR3 will change to 0x00000006`52e89000 when we switch to this process' context in WinDbg.
To perform the context swap, we use the .process command and indicate that we want an invasive swap (/i) which will remap the virtual address space and allow us to do things like set breakpoints in user-mode memory. Also, in order for the process context swap to complete, we need to allow the process to execute once using the g command. The debugger will then break again, and we're officially in the context of notepad.exe.
Okay! Now that we're in the context we need to be in, let's check the CR3 register to verify that the paging structures have been changed to the DirectoryTableBase member we saw earlier.
Looks like it worked as we expected. We would find a unique set of paging structures at 0x00000006`52e89000 that represented the virtual to physical address mappings within notepad.exe. This is essentially the same kind of swap that occurs each time Windows switches to a thread in a different process.
Virtual address ranges
While each process gets its own view of virtual memory and can re-use the same virtual address range as another process, there are some consistent rules of thumb that Windows abides by when it comes to which virtual address ranges store certain kinds of information.
To start, each user-mode process is allowed a user-mode virtual address space ranging from 0x000`00000000 to 0x7ff`ffffffff, giving each process a theoretical maximum of 8TB of virtual memory that it can access. Then, each process also has a range of kernel-mode virtual memory that is split up into a number of different subsections. This much larger range gives the kernel a theoretical maximum of 248TB of virtual memory, ranging from 0xffff0800`00000000 to 0xffffffff`ffffffff. The remaining address space is not actually used by Windows, though, as we can see below.
Figure 3: All possible virtual memory, divided into the different ranges that Windows enforces. The virtual addresses for the kernel-mode regions may not be true on Windows 10, where these regions are subject to address space layout randomization (ASLR). Credits to Alex Ionescu for specific kernel space mappings.
Currently, there is an extremely large “no man's land” of virtual memory space between the user-mode and kernel-mode ranges of virtual memory. This range of memory isn't wasted, though, it's just not addressable due to the current architecture constraint of 48-bit virtual addresses, which we discussed in our previous article. If there existed a system with 16EB of physical memory - enough memory to address all possible 64-bit virtual memory - the extra physical memory would simply be used to hold the pages of other processes, so that many processes' memory ranges could be resident in physical memory at once.
As an aside, one other interesting property of the way Windows handles virtual address mapping is being able to quickly tell kernel pointers from user-mode pointers. Memory that is mapped as part of the kernel has the highest order bits of the address (the 16 bits we didn't use as part of the linear address translation) set to 1, while user-mode memory has them set to 0. This ensures that kernel-mode pointers begin with 0xFFFF and user-mode pointers begin with 0x0000.
A tree of virtual memory: the VAD
We can see that the kernel-mode virtual memory is nicely divided into different sections. But what about user-mode memory? How does the memory manager know which portions of virtual memory have been allocated, which haven't, and details about each of those ranges? How can it know if a virtual address within a process is valid or invalid? It could walk the process' paging structures to figure this out every time the information was needed, but there is another way: the virtual address descriptor (VAD) tree.
Each process has a VAD tree that can be located in the VadRoot member of the aforementioned EPROCESS structure. The tree is a balanced binary search tree, with each node representing a region of virtual memory within the process.
Figure 4: The VAD tree is balanced with lower virtual page numbers to the left, and each node providing some additional details about the memory range.
Each node gives details about the range of addresses, the memory protection of that region, and some other metadata depending on the state of the memory it is representing.
We can use our friend WinDbg to easily list all of the entries in the VAD tree of a particular process. Let's have a look at the VAD entries from notepad.exe using !vad.
The range of addresses supported by a given VAD entry are stored as virtual page numbers - similar to a PFN, but simply in virtual memory. This means that an entry representing a starting VPN of 0x7f and an ending VPN of 0x8f would actually be representing virtual memory from address 0x00000000`0007f000 to 0x00000000`0008ffff.
There are a number of complexities of the VAD tree that are outside the scope of this article. For example, each node in the tree can be one of three different types depending on the state of the memory being represented. In addition, a VAD entry may contain information about the backing PTEs for that region of memory if that memory is shared. We will touch more on that concept in a later section.
Let's get physical
So we now know that Windows maintains separate paging structures for each individual process, and some details about the different virtual memory ranges that are defined. But the operating system also needs a central mechanism to keep track of each individual page of physical memory. After all, it needs to know what's stored in each physical page, whether it can write that data out to a paging file on disk to free up memory, how many processes are using that page for the purposes of shared memory, and plenty of other details for proper memory management
That's where the page frame number (PFN) database comes in. A pointer to the base of this very large structure can be located at the symbol nt!MmPfnDatabase, but we know based on the kernel-mode memory ranges that it starts at the virtual address 0xfffffa80`00000000, except on Windows 10 where this is subject to ASLR. (As an aside, WinDbg has a neat extension for dealing with the kernel ASLR in Windows 10 - !vm 0x21 will get you the post-KASLR regions). For each physical page available on the system, there is an nt!_MMPFN structure allocated in the database to provide details about the page.
Figure 5: Each physical page in the system is represented by a PFN entry structure in this very large, contiguous data structure.
Though some of the bits of the nt!_MMPFN structure can vary depending on the state of the page, that structure generally looks something like this:
A page represented in the PFN database can be in a number of different states. The state of the page will determine what the memory manager does with the contents of that page.
We won't be focusing on the different states too much in this article, but there are a few of them: active, transition, modified, free, and bad, to name several. It is definitely worth mentioning that for efficiency reasons, Windows manages linked lists that are comprised of all of the nt!_MMPFN entries that are in a specific state. This makes it much easier to traverse all pages that are in a specific state, rather than having to walk the entire PFN database. For example, it can allow the memory manager to quickly locate all of the free pages when memory needs to be paged in from disk.
Figure 6: Different linked lists make it easier to walk the PFN database according to the state of the pages, e.g. walk all of the free pages contiguously.
Another purpose of the PFN database is to help facilitate the translation of physical addresses back to their corresponding virtual addresses. Windows uses the PFN database to accomplish this during calls such as nt!MmGetVirtualForPhysical. While it is technically possible to search all of the paging structures for every process on the system in order to work backwards up the paging structures to get the original virtual address, the fact that the nt!_MMPFN structure contains a reference to the backing PTE coupled with some clever allocation rules by Microsoft allow them to easily convert back to a virtual address using the PTE and some bit shifting.
For a little bit of practical experience exploring the PFN database, let's find a region of memory in notepad.exe that we can take a look at. One area of memory that could be of interest is the entry point of our application. We can use the !dh command to display the PE header information associated with a given module in order to track down the address of the entry point.
Because we've switched into a user-mode context in one of our previous examples, WinDbg will require us to reload our symbols so that it can make sense of everything again. We can do that using the .reload /f command. Then we can look at notepad.exe's headers:
Again, the output is quite verbose, so the section information at the bottom is omitted from the above snippet. We're interested in the address of entry point member of the optional header, which is listed as 0x3acc. That value is called a relative virtual address (RVA), and it's the number of bytes from the base address of the notepad.exe image. If we add that relative address to the base of notepad.exe, we should see the code located at our entry point.
And we do see that the address resolves to notepad!WinMainCRTStartup, like we expected. Now we have the address of our target process' entry point: 00000000`ffd53acc.
While the above steps were a handy exercise in digging through parts of a loaded image, they weren't actually necessary since we had symbols loaded. We could have simply used the ? qualifier in combination with the symbol notepad!WinMainCRTStartup, as demonstrated below, or gotten the value of a handy pseudo-register that represents the entry point with r $exentry.
In any case, we now have the address of our entry point, which from here on we'll refer to as our “target” or the “target page”. We can now start taking a look at the different paging structures that support our target, as well as the PFN database entry for it.
Let's first take a look at the PFN database. We know the virtual address where this structure is supposed to start, but let's look for it the long way, anyway. We can easily find the beginning of this structure by using the ? qualifier and poi on the symbol name. The poi command treats its parameter as a pointer and retrieves the value located at that pointer.
Knowing that the PFN database begins at 0xfffffa80`00000000, we should be able to index easily to the entry that represents our target page. First we need to figure out the page frame number in physical memory that the target's PTE refers to, and then we can index into the PFN database by that number.
Looking back on what we learned from the previous article, we can grab the PTE information about the target page very easily using the handy !pte command.
The above result would indicate that the backing page frame number for the target is 0x65207b. That should be the index into the PFN database that we'll need to use. Remember that we'll need to multiply that index by the size of an nt!_MMPFN structure, since we're essentially trying to skip that many PFN entries.
This looks like a valid PFN entry. We can verify that we've done everything correctly by first doing the manual calculation to figure out what the address of the PFN entry should be, and then comparing it to where WinDbg thinks it should be.
So based on the above, we know that the nt!_MMPFN entry for the page we're interested in it should be located at 0xfffffa80`12f61710, and we can use a nice shortcut to verify if we're correct. As always in WinDbg, there is an easier way to obtain information from the PFN database. This can be done by using the !pfn command with the page frame number.
Here we can see that WinDbg also indicates that the PFN entry is at 0xfffffa8012f61710, just like our calculation, so it looks like we did that correctly.
An interlude about working sets
Phew - we've done some digging around in the PFN database now, and we've seen how each entry in that database stores some information about the physical page itself. Let's take a step back for a moment, back into the world of virtual memory, and talk about working sets.
Each process has what's called a working set, which represents all of the process' virtual memory that is subject to paging and is accessible without incurring a page fault. Some parts of the process' memory may be paged to disk in order to free up RAM, or in a transition state, and therefore accessing those regions of memory will generate a page fault within that process. In layman's terms, a page fault is essentially the architecture indicating that it can't access the specified virtual memory, because the PTEs needed for translation weren't found inside the paging structures, or because the permissions on the PTEs restrict what the application is attempting to do. When a page fault occurs, the page fault handler must resolve it by adding the page back into the process' working set (meaning it also gets added back into the process' paging structures), mapping the page back into memory from disk and then adding it back to the working set, or indicating that the page being accessed is invalid.
Figure 7: An example working set of a process, where some rarely accessed pages were paged out to disk to free up physical memory.
It should be noted that other regions of virtual memory may be accessible to the process which do not appear in the working set, such as Address Windowing Extensions (AWE) mappings or large pages; however, for the purposes of this article we will be focusing on memory that is part of the working set.
Occasionally, Windows will trim the working set of a process in response to (or to avoid) memory pressure on the system, ensuring there is memory available for other processes.
If the working set of a process is trimmed, the pages being trimmed have their backing PTEs marked as “not valid” and are put into a transition state while they await being paged to disk or given away to another process. In the case of a “soft” page fault, the page described by the PTE is actually still resident in physical memory, and the page fault handler can simply mark the PTE as valid again and resolve the fault efficiently. Otherwise, in the case of a “hard” page fault, the page fault handler needs to fetch the contents of the page from the paging file on disk before marking the PTE as valid again. If this kind of fault occurs, the page fault handler will likely also have to alter the page frame number that the PTE refers to, since the page isn't likely to be loaded back into the same location in physical memory that it previously resided in.
Sharing is caring
It's important to remember that while two processes do have different paging structures that map their virtual memory to different parts of physical memory, there can be portions of their virtual memory which map to the same physical memory. This concept is called shared memory, and it's actually quite common within Windows. In fact, even in our previous example with notepad.exe's entry point, the page of memory we looked at was shared. Examples of regions in memory that are shared are system modules, shared libraries, and files that are mapped into memory with CreateFileMapping() and MapViewOfFile().
In addition, the kernel-mode portion of a process' memory will also point to the same shared physical memory as other processes, because a shared view of the kernel is typically mapped into every process. Despite the fact that a view of the kernel is mapped into their memory, user-mode applications will not be able to access pages of kernel-mode memory as Windows sets the UserSupervisor bit in the kernel-mode PTEs. The hardware uses this bit to enforce ring0-only access to those pages.
Figure 8: Two processes may have different views of their user space virtual memory, but they get a shared view of the kernel space virtual memory.
In the case of memory that is not shared between processes, the PFN database entry for that page of memory will point to the appropriate PTE in the process that owns that memory.
Figure 9: When not sharing memory, each process will have PTE for a given page, and that PTE will point to a unique member of the PFN database.
When dealing with memory that is shareable, Windows creates a kind of global PTE - known as a prototype PTE - for each page of the shared memory. This prototype always represents the real state of the physical memory for the shared page. If marked as Valid, this prototype PTE can act as a hardware PTE just as in any other case. If marked as Not Valid, the prototype will indicate to the page fault handler that the memory needs to be paged back in from disk. When a prototype PTE exists for a given page of memory, the PFN database entry for that page will always point to the prototype PTE.
Figure 10: Even though both processes still have a valid PTE pointing to their shared memory, Windows has created a prototype PTE which points to the PFN entry, and the PFN entry now points to the prototype PTE instead of a specific process.
Why would Windows create this special PTE for shared memory? Well, imagine for a moment that in one of the processes, the PTE that describes a shared memory location is stripped out of the process' working set. If the process then tries to access that memory, the page fault handler sees that the PTE has been marked as Not Valid, but it has no idea whether that shared page is still resident in physical memory or not.
For this, it uses the prototype PTE. When the PTE for the shared page within the process is marked as Not Valid, the Prototype bit is also set and the page frame number is set to the location of the prototype PTE for that page.
Figure 11: One of the processes no longer has a valid PTE for the shared memory, so Windows instead uses the prototype PTE to ascertain the true state of the physical page.
This way, the page fault handler is able to examine the prototype PTE to see if the physical page is still valid and resident or not. If it is still resident, then the page fault handler can simply mark the process' version of the PTE as valid again, resolving the soft fault. If the prototype PTE indicates it is Not Valid, then the page fault handler must fetch the page from disk.
We can continue our adventures in WinDbg to explore this further, as it can be a tricky concept. Based on what we know about shared memory, that should mean that the PTE referenced by the PFN entry for the entry point of notepad.exe is a prototype PTE. We can already see that it's a different address (0xfffff8a0`09e25a00) than the PTE that we were expecting from the !pte command (0xfffff680007fea98). Let's look at the fully expanded nt!_MMPTE structure that's being referenced in the PFN entry.
We can compare that with the nt!_MMPTE entry that was referenced when we did the !pte command on notepad.exe's entry point.
It looks like the Prototype bit is not set on either of them, and they're both valid. This makes perfect sense. The shared page still belongs to notepad.exe's working set, so the PTE in the process' paging structures is still valid; however, the operating system has proactively allocated a prototype PTE for it because the memory may be shared at some point and the state of the page will need to be tracked with the prototype PTE. The notepad.exe paging structures also point to a valid hardware PTE, just not the same one as the PFN database entry.
The same isn't true for a region of memory that can't be shared. For example, if we choose another memory location that was allocated as MEM_PRIVATE, we will not see the same results. We can use the !vad command to give us all of the virtual address regions (listed by virtual page frame) that are mapped by the current process.
We can take a look at a MEM_PRIVATE page, such as 0x1cf0, and see if the PTE from the process' paging structures matches the PTE from the PFN database.
As we can see, it does match, with both addresses referring to 0xfffff680`0000e780. Because this memory is not shareable, the process' paging structures are able to manage the hardware PTE directly. In the case of shareable pages allocated with MEM_MAPPED, though, the PFN database maintains its own copy of the PTE.
It's worth exploring different regions of memory this way, just to see how the paging structures and PFN entries are set up in different cases. As mentioned above, the VAD tree is another important consideration when dealing with user-mode memory as in many cases, it will actually be a VAD node which indicates where the prototype PTE for a given shared memory region resides. In these cases, the page fault handler will need to refer to the process' VAD tree and walk the tree until it finds the node responsible for the shared memory region.
Figure 12: If the invalid PTE points to the process' VAD tree, a VAD walk must be performed to locate the appropriate _MMVAD node that represents the given virtual memory.
The FirstPrototypePte member of the VAD node will indicate the starting virtual address of a region of memory that contains prototype PTEs for each shared page in the region. The list of prototype PTEs is terminated with the LastContiguousPte member of the VAD node. The page fault handler must then walk this list of prototype PTEs to find the PTE that backs the specific page that has faulted.
Figure 13: The FirstPrototypePte member of the VAD node points to a region of memory that has a contiguous block of prototype PTEs that represent shared memory within that virtual address range.
One more example to bring it all together
It would be helpful to walk through each of these scenarios with a program that we control, and that we can change, if needed. That's precisely what we're going to do with the memdemo project. You can follow along by compiling the application yourself, or you can simply take a look at the code snippets that will be posted throughout this example.
To start off, we'll load our memdemo.exe and then attach the kernel debugger. We then need to get a list of processes that are currently running on the system.
Let's quickly switch back to the application so that we can let it create our initial buffer. To do this, we're simply allocating some memory and then accessing it to make sure it's resident.
Upon running the code, we see that the application has created a buffer for us (in the current example) at 0x000001fe`151c0000. Your buffer may differ.
We should hop back into our debugger now and check out that memory address. As mentioned before, it's important to remember to switch back into the process context of memdemo.exe when we break back in with the debugger. We have no idea what context we could have been in when we interrupted execution, so it's important to always do this step.
When we wrote memdemo.exe, we could have used the __debugbreak() compiler intrinsic to avoid having to constantly switch back to our process' context. It would ensure that when the breakpoint was hit, we were already in the correct context. For the purposes of this article, though, it's best to practice swapping back into the correct process context, as during most live analysis we would not have the liberty of throwing int3 exceptions during the program's execution.
We can now check out the memory at 0x000001fe`151c0000 using the db command.
Looks like that was a success - we can even see the 0xff byte that we wrote to it. Let's have a look at the backing PTE for this page using the !pte command.
That's good news. It seems like the Valid (V) bit is set, which is what we expect. The memory is Writeable (W), as well, which makes sense based on our PAGE_READWRITE permissions. Let's look at the PFN database entry using !pfn for page 0xa1dd0.
We can see that the PFN entry points to the same PTE structure we were just looking at. We can go to the address of the PTE at 0xffffed00ff0a8e00 and cast it as an nt!_MMPTE.
We see that it's Valid, Dirty, Accessed, and Writeable, which are all things that we expect. The Accessed bit is set by the hardware when the page table entry is used for translation. If that bit is set, it means that at some point the memory has been accessed because the PTE was used as part of an address translation. Software can reset this value in order to track accesses to certain memory. Similarly, the Dirty bit shows that the memory has been written to, and is also set by the hardware. We see that it's set for us because we wrote our 0xff byte to the page.
Now let's let the application execute using the g command. We're going to let the program page out the memory that we were just looking at, using the following code:
Once that's complete, don't forget to switch back to the process context again. We need to do that every time we go back into the debugger! Now let's check out the PTE with the !pte command after the page has been supposedly trimmed from our working set.
We see now that the PTE is no longer valid, because the page has been trimmed from our working set; however, it has not been paged out of RAM yet. This means it is in a transition state, as shown by WinDbg. We can verify this for ourselves by looking at the actual PTE structure again.
In the _MMPTE_TRANSITION version of the structure, the Transition bit is set. So because the memory hasn't yet been paged out, if our program were to access that memory, it would cause a soft page fault that would then simply mark the PTE as valid again. If we examine the PFN entry with !pfn, we can see that the page is still resident in physical memory for now, and still points to our original PTE.
Now let's press g again and let the app continue. It'll create a shared section of memory for us. In order to do so, we need to create a file mapping and then map a view of that file into our process.
Let's take a look at the shared memory (at 0x000001fe`151d0000 in this example) using db. Don't forget to change back to our process context when you switch back into the debugger.
And look! There's the 0xff that we wrote to this memory region as well. We're going to follow the same steps that we did with the previous allocation, but first let's take a quick look at our process' VAD tree with the !vad command.
You can see the first allocation we did, starting at virtual page number 0x1fe151c0. It's a Private region that has the PAGE_READWRITE permissions applied to it. You can also see the shared section allocated at VPN 0x1fe151d0. It has the same permissions as the non-shared region; however, you can see that it's Mapped rather than Private.
Let's take a look at the PTE information that's backing our shared memory.
This region, too, is Valid and Writeable, just like we'd expect. Now let's take a look at the !pfn.
We see that the Share Count now actually shows us how many times the page has been shared, and the page also has the Shared property. In addition, we see that the PTE address referenced by the PFN entry is not the same as the PTE that we got from the !pte command. That's because the PFN database entry is referencing a prototype PTE, while the PTE within our process is acting as a hardware PTE because the memory is still valid and mapped in.
Let's take a look at the PTE structure that's in our process' paging structures, that was originally found with the !pte command.
We can see that it's Valid, so it will be used by the hardware for address translation. Let's see what we find when we take a look at the prototype PTE being referenced by the PFN entry.
This PTE is also valid, because it's representing the true state of the physical page. Something interesting to note, though, is that you can see that the Dirty bit is not set. Because this bit is only set by the hardware in the context of whatever process is doing the writing, you can theoretically use this bit to actually detect which process on a system wrote to a shared memory region.
Now let's run the app more and let it page out the shared memory using the same technique we used with the private memory. Here's what the code looks like:
Let's take a look at the memory with db now.
We see now that it's no longer visible in our process. If we do !pte on it, let's see what we get.
The PTE that's backing our page is no longer valid. We still get an indication of what the page permissions were, but the PTE now tells us to refer to the process' VAD tree in order to get access to the prototype PTE that contains the real state. If you recall from when we used the !vad command earlier in our example, the address of the VAD node for our shared memory is 0xffffa50d`d2313a20. Let's take a look at that memory location as an nt!_MMVAD structure.
The FirstPrototypePte member contains a pointer to a location in virtual memory that stores contiguous prototype PTEs for the region of memory represented by this VAD node. Since we only allocated (and subsequently paged out) one page, there's only one prototype PTE in this list. The LastContiguousPte member shows that our prototype PTE is both the first and last element in the list. Let's take a look at this prototype PTE as an nt!_MMPTE structure.
We can see that the prototype indicates that the memory is no longer valid. So what can we do to force this page back into memory? We access it, of course. Let's let the app run one more step so that it can try to access this memory again.
Remember to switch back into the context of the process after the application has executed the next step, and then take a look at the PTE from the PFN entry again.
Looks like it's back, just like we expected!
Exhausted yet? Compared to the 64-bit paging scheme we talked about in our last article, Windows memory management is significantly more complex and involves a lot of moving parts. But at it's core, it's not too daunting. Hopefully, now with a much stronger grasp of how things work under the hood, we can put our memory management knowledge to use in something practical in a future article.
If you're interested in getting your hands on the code used in this article, you can check it out on GitHub and experiment on your own with it.
Microsoft has a great track record of maintaining support for legacy software running under Windows. There is an entire compatibility layer baked into the OS that is dedicated to fixing issues with decades old software running on modern iterations of Windows. To learn more about this application compatibility infrastructure, I'd recommend swinging over to Alex Ionescu's blog. He has a great set of posts describing the technical details on how user (even kernel) mode shimming is implemented.
With all of that said, it's an understatement to say that Microsoft takes backwards compatibility seriously. Occasionally, the humans at Microsoft make mistakes. Usually, though, they're very quick to address these problems.
This blog post will go over an unnoticed bug that was introduced in Windows 8 with a documented Win32 API. At the time of this post, this bug is still present in Windows 10 (Creator's Update) and has been around for over 5 years.
Forgotten Win32 APIs
There is a set of Win32 APIs that were introduced in Windows XP to monitor the working set of a process. A process' working set is a collection of pages, chunks of memory, that are currently in RAM (physical memory) and are accessible to that process without inducing a page fault. In particular, the APIs of interest for us are InitializeProcessForWsWatch and GetWsChanges/GetWsChangeEx.
After reading the MSDN documentation, it's easy to discover what the intended use for these APIs were. These APIs profile the number of page faults that occur within a process' address space.
What's a page fault? A quick recap.
There are 3 general categories of page faults.
A hard page fault occurs when memory is accessed that's not currently in RAM (physical). In situations like this, the OS will need to retrieve the memory from disk (e.g. pagefile.sys) and make it accessible to the faulting process.
A soft page fault occurs when memory is in RAM (physical), but not currently accessible to the process that induced the fault. This memory might be shared amongst multiple processes and the process that caused the page fault might not have it mapped into its working set. These types of page faults are much more performant than hard page faults as there is no disk I/O conducted.
The last and final type of page fault is known formally as an invalid fault. These can also be referred to as access violations. This can be caused when a program, for example, tries to access unallocated memory or tries to write to memory that's marked read-only.
Paging is necessary to make modern operating systems work. You probably have many processes running on your system, but not nearly enough RAM to hold all the possible contents of each process into physical memory. To learn more about paging, I strongly recommend this article posted by my colleague.
Demo
The best way to illustrate what's broken is through an example. I created two simple programs.
The first application, WorkingSetWatch.exe, implements the InitializeProcessForWsWatch and GetWsChangeEx APIs. This application logs when a specific memory region is paged into our process' working set:
The second application, ReadProcessMemory.exe, implements reading of an arbitrary memory blob from another target process' memory space:
The basic idea is to use ReadProcessMemory.exe to read from the monitored memory address inside of WorkingSetWatch.exe. This will induce a page fault.
Windows 7: Build 7601 (SP1)
The WorkingSetWatch.exe application works as expected. We're able to read any (valid) sized buffer using ReadProcessMemory.exe and log it.
Windows 10: Build 15063 (Creator's Update)
Unfortunately, WorkingSetWatch.exe does not seem to log the page fault that occurs when our remote application, ReadProcessMemory.exe, reads a buffer greater than or equal to 512 bytes; however, it does seem to work as expected when a read occurs that's less than 512 bytes.
This renders these working set APIs useless for profiling reasons on Windows 8+.
What went wrong?
To determine what went wrong, we'll need to reverse engineer parts of Windows and see exactly how the implementation changed in Windows 8+ from Windows 7.
All disassembly and pseudo-source is reconstructed from system files that are provided with Windows x64 10.0.15063 (Creator's Update).
Enabling process working set logging
To enable working set logging for a process, we need to call InitializeProcessForWsWatch. From the MSDN documentation, we're told that on newer versions of Windows this API is exported as K32InitializeProcessForWsWatch within kernel32.dll. Our analysis begins there:
This function is very simple. It invokes an import from another library. In this case, it executes a function of the same name (K32InitializeProcessForWsWatch), but contained within a different library, api-ms-win-core-psapi-l1-1-0.dll. This library doesn't exist on disk, but rather resolves to an API Set mapping corresponding to kernelbase.dll (which does exist on disk) for this version of Windows. A look into kernelbase.dll's implementation shows that a call to NtSetInformationProcess is performed without any parameter marshalling:
Our next target is NtSetInformationProcess within ntdll.dll:
This is just a simplistic syscall stub that will eventually make its way into the implementation contained within ntoskrnl.exe, the Windows kernel. nt!NtSetInformationProcess is a massive function that contains a huge switch statement that supports all the different PROCESSINFOCLASS that can be passed to it.
We're interested in the PROCESSINFOCLASS for ProcessWorkingSetWatch. This is case 15 (0xF). A snippet of the relevant parts (with the cleaned-up disassembly):
It's interesting to note that you're able to start monitoring on a process' working set with either a class of ProcessWorkingSetWatch (15) or ProcessWorkingSetWatchEx (42). This can be achieved by invoking nt!NtSetInformationProcess directly instead of going through the documented route with kernel32!InitializeProcessForWsWatch. The latter utilizes only the ProcessWorkingSetWatch class.
The actual logic of nt!NtSetInformationProcess is pretty trivial to understand. A blob of memory is allocated per process that we're monitoring. This blob of memory is a _PAGEFAULT_HISTORY structure and contains up to 1024 _PROCESS_WS_WATCH_INFORMATION structures internally. Each _PROCESS_WS_WATCH_INFORMATION structure is an entry that describes a page fault. These entries will be cycled through as the array fills up. Recall from the MSDN documentation (the "Remarks" section) that you must call GetWsChanges/Ex with enough frequency to avoid record loss. This makes perfect sense because we can see that there are a fixed number of these records (1024) allocated. I took the liberty of documenting these structures:
The union at the beginning of the _PAGEFAULT_HISTORY structure may be a little confusing, but it'll be explained later.
On successful execution of this routine, the monitored process object will have an internal member (_EPROCESS.WorkingSetWatch) updated to include this recently allocated _PAGEFAULT_HISTORY pointer. Additionally, the PsWatchEnabled global will be set. This value informs the system to track page faults for processes. It will remain set until the system reboots (even if there are no processes running that have working sets tracked). There are only 2 references to PsWatchEnabled and we've already inspected the one in nt!NtSetInformationProcess.
Our investigation leads us to nt!KiPageFault.
Logging a page fault
When a page fault occurs, the CPU transfers execution to nt!KiPageFault:
If the PsWatchEnabled global is set, that means we've enabled working set logging for processes on the system and execution is passed to nt!PsWatchWorkingSet. This function is documented below:
As I mentioned above, there are 3 types of page faults. Access violations are not logged to our process' working set due to an early out by nt!MmAccessFault in nt!KiPageFault. Since this function is executed for the other 2 types of page faults (hard and soft) on the system, it will be accessed heavily by the operating system. Luckily, one of the first things the routine does is check whether or not a working set watch was enabled on the process where the page fault occurred. If there is no working set watch on the process, the routine completes.
As per the documentation, nt!PsWatchWorkingSet will not function while records are being processed (EntrySelector.Busy). We'll describe this part in depth at a later time. Since higher priority interrupts can preempt our working set monitor, most of the logic in this routine needs to have adequate sanity (safety) checks and complete as atomically (Interlocked*** operations) as possible. The first part of the function will safely select a free index in the _PAGEFAULT_HISTORY.WatchInfo array that it can use for logging purposes. If the array is full (there can be at most 1024 entries), a "miss" is recorded (_PAGEFAULT_HISTORY.MissingRecords) and the routine completes. If everything is successful, a page fault event is recorded in a free slot in the _PAGEFAULT_HISTORY.WatchInfo array. An interesting (and undocumented) feature changes the entry's _PROCESS_WS_WATCH_INFORMATION.FaultingVa least significant bit to 0 if a hard page fault occurred and 1 if a soft page fault occurred.
Ultimately, there doesn't seem to be any apparent bugs with this code. Additionally, this code matches very closely to the Windows 7 version which we know works. Our investigation leads us to the working set watch retrieval functions: GetWsChanges/Ex.
Querying working set logging
For article brevity, I'll give a quick summary of the call-flow of kernel32!GetWsChanges (kernel32!K32GetWsChanges) and kernel32!GetWsChangesEx (kernel32!K32GetWsChangesEx). These functions will call into their kernelbase.dll variants. From there, they will branch into kernelbase!GetWsChangesInternal which will invoke ntdll!NtQueryInformationProcess with the appropriate PROCESSINFOCLASS. In particular, the ProcessWorkingSetWatch class will be used for the GetWsChanges family of functions and ProcessWorkingSetWatchEx will be used for the others. From ntdll!NtQueryInformationProcess, a syscall will be made. This makes it to the implementation of NtQueryInformationProcess within the kernel. A massive switch statement awaits:
The part that interests us resides one level deeper within nt!PspQueryWorkingSetWatch:
There's some input validation (e.g. alignment checks) and a safety check (nt!ExIsRestrictedCaller) to avoid kernel pointer leaks in low integrity processes. After that, the process object is retrieved from the supplied process handle. The operating system checks to see that the _EPROCESS.WorkingSetWatch member is set. Just like the documentation states, at most one query can access a process' working set buffer at a time (EntrySelector.Busy). Additionally, while the buffer is being accessed, logging (by nt!PsWatchWorkingSet in nt!KiPageFault) will produce misses.
As long as there's enough space in the user supplied buffer, the operating system will copy over the entry array to the user supplied buffer. The data will be structured in the appropriate way for the appropriate PROCESSINFOCLASS. The last entry in the user supplied buffer (PSAPI_WS_WATCH_INFORMATION/EX) will be terminated with a FaultingPc member of NULL. Additionally, the number of "misses" will be recorded in the FaultingVa member of the last entry.
Finally, the _PAGEFAULT_HISTORY.WatchInfo array of the _EPROCESS.WorkingSetWatch will be reset after a successful call.
/rant.
The InitializeProcessForWsWatch and GetWsChanges/Ex APIs are surprisingly very finicky. There are many weird restrictions and caveats which make it surprisingly difficult for developers to retrieve information regarding the complete set of page faults that occurred within a process.
There is a very good chance that you will run into situations where records will wind up missing especially in a multi-processor and multi-threaded environment. For example, if a thread is querying the working set of a process, but a page fault occurs on another thread within that same process, a miss could be recorded since the _PAGEFAULT_HISTORY.Busy member will be acquired by nt!PspQueryWorkingSetWatch. This will prevent the page fault logging logic in nt!PsWatchWorkingSet. Functionally, this weakens the usability of the API for profiling purposes. To compound this problem, only 1024 entries can be stored in the array between calls of GetWsChanges/Ex. That's at most 4 MB (1024*PAGE_SIZE) of page fault history. This really isn't enough for modern applications which can be very complex.
In our specific situation, we ran our tests on a VM that had 1 processor allocated to it. Furthermore, our application was simple enough that it had 1 thread. This mitigates the chance of page fault "misses". Additionally, after a thorough investigation of the working set APIs, we've concluded that we've still not discovered where the bug is. In particular, why does the buffer size play a role in the success of these APIs? In our demo, we were unable to log page faults on Windows 10 when the buffer size was greater than or equal to 512 bytes. Is it possible that the bug is not within WorkingSetWatch.exe, but rather ReadProcessMemory.exe?
To continue our investigation, we need to turn to ReadProcessMemory.exe.
Reading memory
The ReadProcessMemory.exe application is simple enough to understand. We know that we're not logging a page fault when we're reading a buffer that is greater than or equal to 512 bytes. Since there is no apparent bug in the working set APIs, the problem most likely resides in kernel32!ReadProcessMemory.
I'll step past the irrelevant details, but the same strategy is applied as was in the previous parts. In particular, kernel32!ReadProcessMemory calls into kernelbase!ReadProcessMemory. These functions do nothing special and more-or-less directly issue a system call by invoking ntdll!NtReadVirtualMemory. This takes us to the implementation of nt!ReadVirtualMemory in the kernel:
This function just invokes nt!MiReadWriteVirtualMemory. On some versions of ntoskrnl, this routine may just be inlined into the caller's body.
Aside from a check that prevents reading and writing to protected processes (ProcessObject->Pcb.SecurePid), this function is nearly identical to the one in the Windows 7 kernel. We need to go deeper. We traverse into nt!MmCopyVirtualMemory.
This function is massive. It contains many subfunctions that have been inlined. For article brevity, the important parts of nt!MmCopyVirtualMemory will be highlighted. One of the first things that this routine does is search for VAD entries that corresponds to the input addresses (FromAddress and ToAddress). The idea is to leverage the "region size" information for memory, but this isn't really relevant to our bug. We'll leave the discussion of the VAD (Virtual Address Descriptor) to another time.
nt!MmCopyVirtualMemory's next task is to determine the input buffer's length. In particular, there are a couple checks against the buffer length and the value 512. This is significant to us because we know the bug only seems to manifest when the buffer size is greater than or equal to 512 bytes.
Basically, it seems that if the buffer is greater than or equal to 512 bytes, nt!MmCopyVirtualMemory will utilize nt!MmProbeAndLockPages and nt!MmMapLockedPagesSpecifyCache followed by a memcpy to clone over memory.
If the buffer is less than 512 bytes, nt!MmCopyVirtualMemory will just leverage memcpy directly by using a buffer on the stack or a buffer allocated in dynamic memory (based on buffer size) via nt!ExAllocatePoolWithTag.
This is probably done for performance reasons. Larger memory copies probably benefit from direct mapping instead of memory pool copying. If we do leverage memory pool copying (buffers that are less than 512 bytes in size), we trigger a page fault and the event is logged by our WorkingSetWatch.exe application. On the other hand, if we leverage a direct mapping to copy memory, we do not trigger a page fault.
One incorrect assumption is to believe that on Windows 7 this optimization did not exist. On the contrary, there is very similar logic inside of the older version of nt!MmCopyVirtualMemory. However, something did change, otherwise we would not have any discrepancies with our WorkingSetWatch program. Our investigation leads us into nt!MmProbeAndLockPages.
The bug: an optimization in nt!MmProbeAndLockPages
The implementation of nt!MmProbeAndLockPages underwent drastic changes between Windows 7 to now. If you looked at these two functions side-by-side, you'd quickly notice that the Windows 7 implementation was in some ways much simpler.
The purpose of nt!MmProbeAndLockPages (per the documentation) is to ensure that the specified virtual pages (in the argument contained within MemoryDescriptorList) are backed by physical memory. Additionally, there is a series of permission checks to ensure that the virtual pages permit the user-specified access rights. In Windows 7, to perform this access check, the routine actually "probed" the memory by directly accessing it. This would induce a page fault in the context of the correct process and therefore we'd be able to log it using our WorkingSetWatch.exe application.
On Windows 10, this process was optimized. Instead of accessing the memory directly, a PTE (Page Table Entry) walk is performed to ensure that the correct permissions exist. This change makes the process more efficient especially since the PTEs are leveraged to lock the memory into physical pages anyway.
OS development isn't easy
One seemingly inconspicuous change can break functionality in an entirely unrelated part of the operating system. In our case, an optimization in the underlying logic of how nt!MmProbeAndLockPages functioned broke backwards compatibility of the working set APIs. This bug seems to be entirely unnoticed, but it unfortunately renders the performance profiling nature of the GetWsChanges/Ex APIs useless.
A potential fix for Microsoft is to simply just throw a page fault for "invalid" pages if the PsWatchEnabled global is set or, more granularly, if a process' _EPROCESS.WorkingSetWatch is set.
This article assumes you've read the first part of the series. In particular, at this point you should have successfully setup VMware's GDB stub and IDA Pro's GDB debugger. You should now be in a connected state and broken into IDA Pro's debugger GUI.
Furthermore, the focus of this post is going to be exclusively on loading kernel symbols for 64-bit editions of Windows (AMD64). Different operating systems (and different architectures of Windows) require slight modifications to the article's logic.
Where's Waldo ntoskrnl?
The end goal
The first and most important thing is to discover where the NT Kernel (ntoskrnl.exe) is loaded in memory since it's not at any fixed (static) address thanks to address space layout randomization (ASLR).
We are then able to force IDA Pro to load symbol data (PDBs) at ntoskrnl's base address to have useful debugging information. From there, we can enumerate the linked list, nt!PsLoadedModuleList, to figure out where other kernel mode components are located. However, this isn't trivial. When you break in to IDA Pro's GDB debugger, it's difficult to know what state you'll be in on any given processor. You might be executing code in a usermode process, or you might be busy servicing a system call. Additionally, you're further restricted to the functionality the GDB stub exposes.
Enter the _KPCR
On all architectures and versions of Windows, each processor maintains a control structure dubbed as the _KPCR (Kernel Processor Control Region). This structure is massive and it can be used to infer exactly what the processor is doing. On Windows 10 (15063.0.amd64fre.rs2_release.170317-1834), the _KPCR is 0x6bc0 bytes large. It contains many kernel pointers that we can leverage to figure out exactly where the base of ntoskrnl is in memory. A link detailing the members of the _KPCR can be found here.
This structure can be accessed through its virtual address or through the fs segment on x86 and the gs segment on x64. In fact, if you've done any reverse engineering of the Windows kernel, you should have seen many examples of Windows itself accessing members of the _KPCR through the segment selector.
For example, when an int 3 (a software breakpoint; 0xCC) is executed by the processor, control is redirected by the CPU to a handler registered in the appropriate position of the IDT (Interrupt Descriptor Table). We'll touch more on this process later. In Windows, the handler for software breakpoints is nt!KiBreakpointTrap. Here is a snippet of the assembly code of the handler under AMD64:
In particular, at address 0x00000001401749FD we see a swapgs instruction. Since the gs selector means different things in user-mode (_TEB) and the kernel (_KPCR), this instruction is utilized to ensure that we're operating on the kernel-mode construct (_KPCR). Immediately following that instruction at address 0x0000000140174A00, we have an access of the gs segment with a mov r10, gs:188h. The astute reader will realize that upon execution of this instruction, r10 will contain the pointer from the _KPCR.Prcb.CurrentThread. This is discerned from the definition of the structure's members posted above. A breakdown of this process can be illustrated below:
We don't know the _KPCR's exact linear address (it too isn't allocated at a fixed location), but we should be able to access it through the segment selector, though, just like the Windows kernel does. This approach might seem like the ideal one, but, unfortunately, we're further restricted by the functionality of the GDB stub. Let's see what the GDB stub exposes by issuing help:
There are only three major commands available: help, r, and linuxoffsets. We've just executed help, and linuxoffsets isn't relevant to us since we're debugging a Windows kernel. The only other command is r. At first, r looks very useful to us. However, on closer examination, we can see that the GDB stub is unable to read arbitrary offsets off of the gs selector, e.g. the _KPCR.Prcb.CurrentThread from gs:188h by executing r gs:188h.
At least executing r gs without an offset produces data:
This command should get us the base of the gs selector. We then should be able to define a _KPCR structure at that location using IDA Pro. According to the GDB stub, though, our base is 0. If we go to that memory location in the "IDA View - RIP" tab by pressing 'G' and entering 0 in the "Jump to address" window, we don't see anything there:
What changed from x86 to x64?
If you ran this test on a VM running on an x86 (32-bit) version of Windows and substituted fs for gs, the base of the fs selector would not be 0. It would be a valid memory location. You would then have the address of the _KPCR and could continue on your merry way.
Unfortunately, you're a sucker for pain and are following this tutorial down to the T. In 64-bit (long) mode on x64, the cs, ss, ds, and es segment selectors have a zero-forced base address. gs and fs are the exceptions and have a non-zero base address. So, how is it possible that the base of the gs selector is 0 when Windows itself uses the segment selector to retrieve processor state?
The answer is in the model-specific registers, MSRs. MSRs are per-processor registers that can be read via rdmsr and written via wrmsr instructions. On x64, the IA32_GS_BASE (0xC0000101) and IA32_KERNEL_GS_BASE (0xC0000102) MSRs are used for storage of the base address of the gs selector. swapgs was introduced to exchange the address of the current gs base register with the value contained in the IA32_KERNEL_GS_BASE MSR.
This means that we could, theoretically, read the IA32_GS_BASE MSR if we're executing code in CPL0 (ring0/kernel-mode). This would get us the base address of the gs segment. However, that's not directly possible through the VMware GDB stub. There is no support for reading or writing to MSRs directly.
A shimmer in the shadows
Nevertheless, through persistence, we come up with an approach that plays nicely given our constraints. There are multiple ways to skin a cat and this approach may not be the most elegant solution, but it should work nicely for all x64 Windows kernels.
The basic idea is to leverage the IDT, the interrupt descriptor table, to find a symbol that's in the address space of ntoskrnl. We can access the idtr, a register that houses the IDT, through the GDB stub:
Once we have the base of the IDT, in our case 0xfffff802c4850000, we can access the first entry of the IDT. This should resolve to a symbol within ntoskrnl (nt!KiDivideErrorFault):
From there, we can walk kernel memory backwards until we get to a valid PE header. Since the symbol is contained within ntoskrnl's address space, the first valid PE header should belong to ntoskrnl:
Figure 1: Layout of kernel memory.
Writing an IDA script using IDAPython
It'd be nice to programmatically implement the algorithm described above so we don't need to manually go through it each time we're trying to discover the base address of ntoskrnl. We'll do this by writing a script for IDA Pro to run. I chose to do this with IDAPython instead of IDC (IDA's C-like bindings) because of the niceties that Python provides (like string manipulation).
The basics
We'll start by switching the input from "GDB" to "Python" in the "Output window". If your "Output window" is missing, you can restore it by selecting "Windows" and then "Output window" from the menu bar:
We can see all the functionality exposed by IDAPython by executing the Python command dir() in the text box. If you try to do this, you'll see lots of output. It's easy to feel overwhelmed. Luckily, there exists amble documentation on the Hex-Rays website that can help us navigate these murky waters.
I try to find useful things by searching for it first in the dir() listing. You can position your cursor in the "Output window" and press Alt+T to search for a keyword. To find the next occurrence, you can hit Ctrl+T. If this fails, I move on to the documentation.
Sending a command to the GDB stub
Our first task is to figure out how to send a command to the GDB stub. If you search for the "command" keyword in the "Output window" you'll find something labeled "SendDbgCommand". Let's see what this function does by executing help(SendDbgCommand):
It seems very relevant to us. Let's give it a try:
Looks like it's working. This is the same output we received from the GDB stub when we issued the help command.
Parsing the response from the GDB stub
Now that we know how to send a command to the GDB stub, we need to issue a command to retrieve the contents of the idtr. We then parse and extract the base address from the resulting string.
It's important to tell Python that we're working with an integer object by "casting" the string to an integer-type:
Easy!
Getting the first IDT entry's handler
We have the base of the IDT in idt_base. Our next task is to retrieve the first entry in the IDT. The IDT is effectively an array that contains 256 IDT entries (0-0xFF) on x64. The format of the IDT is dictated by the architecture of the processor (e.g. Intel x64). Each IDT entry on x64 takes the following form:
To get to the handler (e.g. where the processor moves control to when an interrupt occurs), the target address is built from the OffsetHigh, OffsetMiddle, and OffsetLow fields of this structure using the following algorithm: HandlerAddress = ((OffsetHigh << 32) + (OffsetMiddle << 16) + OffsetLow).
We'll leverage the Dbg* commands to read virtual memory from IDAPython. Since we're extracting the first IDT entry, we can just read directly from the start of our idt_base:
This shows us that the handler for the first IDT entry (nt!KiDivideErrorFault) is loaded at 0xfffff802c27f4300. If we wanted to read the N'th IDT entry, we'd have to index into the array by adding 0x10, the size of a _KIDTENTRY64, times the location in the array (in this case N). So, to index into the 3rd IDT entry, we'd apply the following math: idt_entry = idt_base + (0x10 * 2).
Finding the base address from a symbol within ntoskrnl
First, we'll define a simple helper function that will align addresses to their page boundaries. This will help speed up our lookup because we know that the base address of ntoskrnl will be on a page boundary:
We'll then create a very simple loop to walk memory backwards (on a page-aligned boundary) searching for the magical value 0x5A4D, commonly known as 'MZ' (IMAGE_DOS_SIGNATURE). This value signifies the start of the IMAGE_DOS_HEADER which is also the base address of an image:
Voila! The base address of ntoskrnl is discovered at 0xfffff802c2680000.
Creating the final version of the script
After some refactoring and code tidying (including error checking), we produce a much better version of the script. This does the same thing as the commands we inserted in the IDAPython "Output window":
Save a copy of the script to your local drive. We are then able to run it at any time by going to "File" and then "Script file..." in the IDA Pro GUI. A sample of the output is listed below:
The important line appears on the bottom; the base address of ntoskrnl is displayed. It checks out with the work we did by hand too.
Loading ntoskrnl at its base address
We mustn't forget the final objective: loading kernel symbols. We're almost at the finish line. Let's tell IDA to load ntoskrnl at the base address our script found.
First, we'll need to grab a copy of ntoskrnl on the VM. Don't use the version on your host as this may not match with what's on the VM. This'll be found in your guest's system directory:
You might need to resume your VM if you're currently active in IDA's GDB debugger by selecting "Debugger" and then "Continue process" (or by hitting F9) from the menu bar.
After you've pulled ntoskrnl from your VM, break into IDA's GDB debugger by selecting "Suspend". Now, we must load it by selecting "File" then "Load file" and finally "PDB file..."
Find where you copied ntoskrnl to on your host and use the address that the script found:
It'll take IDA at least a couple of minutes to fully finish the loading process. You can see IDA's progress in the bottom left corner:
You'll know IDA's finished when the status changes to "AU: idle".
Quick validation
We should make sure that the symbols are loaded correctly. Navigate to "Jump" and then "Jump to address" (or press "G"). Enter PsLoadedModuleList (case sensitive) and hit "OK".
From there, double click the address immediately to the right of the PsLoadedModuleList symbol. This takes you to the first entry in the list.
Each entry in this list is of type _LDR_DATA_TABLE_ENTRY. You might be familiar with this structure from usermode programming. It's also used in the kernel.
We'll need to add the definition of the _LDR_DATA_TABLE_ENTRY to IDA's structures. Luckily, we have symbols loaded and this is a pretty straightforward process.
After the structure was added, you'll see a window similar to this.
Go back to the "Debug View". Impose the _LDR_DATA_TABLE_ENTRY structure on that memory region:
Let's follow the FullName.Buffer field:
And now let's convert this to a readable string:
You should see the characters \SystemRoot\system32\ntoskrnl.exe. We did it!
Final thoughts
Now that symbols are loaded for ntoskrnl, it would be wise to iterate through the PsLoadedModuleList and load symbols for all the other kernel mode components. This can be scripted using IDAPython too, however, it's beyond the scope of this article.
Sometimes you'll run into a situation that you can't analyze with a traditional kernel debugger like WinDbg. An example of such is trying to troubleshoot the runtime logic of PatchGuard (Microsoft's Kernel Patch Protection). In situations like this, you need to bust out the heavy tools. VMware has built in support for remote debugging of virtual machines running inside it through a GDB stub. IDA Pro, the defacto disassembler that most reverse engineers have, includes a GDB debugger. Together these make for a very powerful combo.
This article goes over how to setup VMware's GDB stub and how to connect to it using IDA Pro's GDB debugger.
Requirements
A copy of VMware Workstation (free 30-day trial). I'll be using VMware Workstation 12.5.7 (build-5813279).
Unfortunately, VMware Player (entirely free for non-commercial use) does not expose the GDB stub interface.
You can use either the Linux or Windows build of VMware. I'll be using the 64-bit Windows build.
A Windows operating system installed on your host and guest (VM). These do not have to be the same versions of Windows. My host and guest OS are both running Windows x64 10.0.15063 (Version 1703).
This can be any OS supported by VMware such as Ubuntu.
The second part of this tutorial (loading kernel symbols) assumes you're running a Windows 64-bit VM (AMD64).
Enabling the GDB stub within VMware
Select the VM you wish to enable GDB stub debugging on within VMware.
VMs should be listed in the "Library" pane on the left of the GUI. If the "Library" pane is missing, you can restore it by selecting "View" then "Customize" and choosing "Library" (or hit F9).
Ensure that the VM is currently not running. If it's currently active, power it off via the menu bar: "VM" then "Power" then "Shut Down Guest" (or Ctrl+E).
Select "Edit virtual machine settings". Ensure that you are on the "Options" tab.
Find the "Working directory" text field and copy the string to your clipboard. 'Cancel' out of the prompt.
Go to the working directory.
Right-click on the *.vmx file and "Open with" your favorite text editor. I'll be using Notepad++.
Add one of the following lines to the end of the file, based on preference.
If your VM is 32-bit and you want to debug locally: debugStub.listen.guest32 = "TRUE"
If your VM is 64-bit and you want to debug locally: debugStub.listen.guest64 = "TRUE"
If your VM is 32-bit and you want to debug remotely: debugStub.listen.guest32.remote = "TRUE"
If your VM is 64-bit and you want to debug remotely: debugStub.listen.guest64.remote = "TRUE"
The default port for the GDB stub is 8864 for 64-bit guests and 8832 for 32-bit guests. If you'd like to change what port the VMware GDB stub listens on (e.g. 55555), add one of the following lines to the file:
If your VM is 32-bit: debugStub.port.guest32 = "55555"
If your VM is 64-bit: debugStub.port.guest64 = "55555"
If you want to start debugging immediately on BIOS load add one of the following lines to your file:
If your VM is 32-bit: monitor.debugOnStartGuest32 = "TRUE"
If your VM is 64-bit: monitor.debugOnStartGuest64 = "TRUE"
To make it difficult to detect breakpoints that you've set using GDB, it's strongly recommended to add the following option too:
debugStub.hideBreakpoints = "TRUE"
An important thing to note is that this option is restricted by the number of hardware breakpoints available to the processor (usually 4).
Save the *.vmx file via "File" and then "Save" from the menu bar (or hit Control+S). Here's a copy of the contents of my *.vmx file:
Close the file.
Run the VM corresponding to the *.vmx file you just edited. Validate that the GDB stub is currently running by opening the vmware.log file in the same directory as the *.vmx file:
If you see a message from "Debug stub" that tells you VMware is "listening" for a debug connection on a certain port number, you're in a good state.
If you are missing that log line or have an error, ensure that your *.vmx file has the appropriate settings. Remember: you must edit the *.vmx file when the Virtual Machine is off or your changes may be lost.
Configuring the GDB debugger within IDA Pro
Launch the 64-bit version of IDA Pro if you're debugging a 64-bit VM and the 32-bit version of IDA Pro if you're debugging a 32-bit VM.
Skip the "Welcome" dialog (by hitting "Go") and go to the main disassembler window. Choose "Debugger" and then "Attach" and finally "Remote GDB debugger" from the menu bar.
Enter the appropriate "Hostname" and "Port". These were set up by you during steps 7 and 8 of the "Enabling the GDB stub within VMware" section. Furthermore, these can be validated in the vmware.log file (this was done in step 12 of the same section). Then hit "Debug options".
In the "Debugger setup" window, select "Set specific options".
Ensure that the right "Processor" is set in the drop down box. If you're debugging a 64-bit edition of Windows (AMD64), select "Intel x64". If you're debugging a 32-bit edition of Windows (X86), select "Intel x86".
Select 'OK' in the "GDB configuration" window. And then select 'OK' in the "Debugger setup" window. Finally, select 'OK' in the "Debug application setup" window.
The VM will become suspended and a green "play" button will appear. At this point, IDA should bring up a window with the title "Choose process to attach to".
Select "<attach to the process started on target>" and hit 'OK'.
If you see this window, that means you're almost done.
Select "Debugger" and then "Manual memory regions" from the menu bar.
Inside of the "Manual memory regions" tab, right click and select "Insert" (or just press "Insert" on your keyboard).
A new window will pop up. Enter in the "Start address" as 0 and the "End address" as -2. Make sure the right "segment" is selected (e.g. 64-bit segment for 64-bit VM debugging) and hit 'OK'.
This essentially maps all virtual memory from 0 to 0xFFFFFFFFFFFFFFFE (on x64).
"-1" is not an acceptable boundary for IDA Pro as the "End address".
Find the "General Registers" window and find the IP register (RIP on x64, EIP on x86).
If the "General Registers" window is gone, select "Debugger" and then "Debugger windows" and finally "General Registers" from the menu bar.
Right click on the IP register and select "Jump" from the context menu that appears.
Your memory view will become synched to the IP register. If there are raw bytes listed and not code, don't panic. Place your cursor on the address of the IP and hit "C".
Congratulations. At this point you've successfully set up VMware's GDB stub and IDA Pro's GDB debugger. You are now able to debug the VM and apply breakpoints through the IDA Pro GUI just as you would normally through a kernel debugger. Most of the functionality of the GDB debugger can be accessed through the "Debugger" menu bar.
This type of debugging is transparent to the kernel and therefore "debugger" checks like "KdDebuggerEnabled" and "KdDebuggerNotPresent" will not trigger. Furthermore, if the debugStub.hideBreakpoints option was enabled, breakpoints (up until the hardware maximum) will not make any inline code edits!
Final thoughts
Ultimately, the GDB debugger is not very useful without kernel symbols being loaded. One option, albeit a naive one, is to attach WinDbg as a kernel debugger while running IDA's GDB stub in the background. A tutorial on how to setup kernel debugging using WinDbg and VMware can be found here. You are then able to use the symbolic data that is provided from WinDbg to power debugging in IDA's GDB debugger. This is very cumbersome and has many disadvantages such as not being able to avoid kernel debugger checks.
Luckily, there is a better way. In the second part of this series, we'll discover how to load kernel symbols in IDA Pro's GDB debugger.
In this article, we explore the complexities and concepts behind Intel's 64-bit paging scheme, why we need paging in the first place, and some practical analysis of paging structures.
Why do we need paging?
In any application, whether it's a student's first program or a complicated operating system, instructions executed by the computer that involve memory use a virtual address. In fact, even when the CPU fetches the next instruction to execute, it uses a virtual address. A virtual address represents a specific location in the application's view of memory; however, it does not represent a location within physical RAM. Paging, or linear address translation, is the mechanism that converts a linear address accessible by the CPU to a physical address that the memory management unit (MMU) can use to access physical memory.
Technically, a linear address and a virtual address are not the same. For the purposes of this article, though, we will consider them to be the same, since we do not need to consider segmentation. Older architectures would first need to convert a virtual address to a linear address using segmentation.
Figure 1: An application with different parts of virtual memory mapping to different parts of physical memory.
Paging modes
In this article, we will focus on IA-32e 4-level paging (64-bit paging) on Intel architectures. It is worth noting, though, that there are other paging modes supported by Intel.
There are three mechanisms which control paging and the currently enabled paging mode. The first is the PG flag (bit 31) in control register 0 (CR0). If this bit is set, paging is enabled on the processor. If this bit is not set, no paging is enabled. In the latter case, the virtual address and physical address are considered equivalent and no translation is necessary.
If paging is enabled on the processor, then control register 4 (CR4) is checked for the Physical Address Extension (PAE) bit (bit 5) being set. If it is not, then 32-bit paging is used. If it is set, then the final condition that is checked is the Extended Feature Enable Register, or IA32_EFER MSR. If the Long Mode Enable (LME) bit (bit 8) of this register is not set, the processor is in PAE 36-bit paging mode. If the LME bit is set, the processor is in 4-level paging mode, which is the 64-bit mode that we plan to explore in this article. This mode translates 48-bit virtual addresses into 52-bit physical addresses, though because the virtual addresses are limited to 48-bits, the maximum addressable space is limited to 256TB.
Paging structures
Regardless of which paging mode is enabled, a series of paging structures are used to facilitate the translation from a virtual address to a physical address. The format and depth of these paging structures will depend on the paging mode chosen. Generally speaking, each entry in the paging structure is the size of a pointer and contains a series of control bits, as well as a page frame number.
In our case, 64-bit mode structures are 4,096 bytes in size (the size of the smallest architecture page - we will touch more on that later), containing 512 entries each. Every entry is 8 bytes.
Figure 2: A paging structure containing 512 pointer-size entries in 64-bit mode.
The first paging structure is always located at the physical address specified in control register 3 (CR3). As an aside, this is also the only place that stores the fully qualified physical address to a paging structure - in all other cases, we need to multiply a page frame number by the size of a page to get the real physical address. Each entry within the paging structure will contain a page frame number which either references a child paging structure for that region of memory, or maps directly to a page of physical memory that the original virtual address translates to. Again, in both cases, the page frame number is simply an index of a physical page in memory, and needs to be multiplied by the size of a page to get a meaningful physical address. Each paging structure entry also describes the the different memory access protections that are applied to the memory region they describe - whether the code is writable, executable, etc - as well as some more interesting properties such as whether or not that specific structure has previously been used for a translation.
While the nested paging structures are being walked, the translation can be considered complete either by identifying a page frame at the lowest level of paging structure or by an early termination caused by the configuration of a paging structure. For example, if a paging structure is marked as not present (bit 0 of the structure is not set) or if a reserved bit is set, the translation fails and the virtual address is considered invalid. Additionally, a paging structure can set its Page Size bit to indicate that it is the lowest paging structure for that region of memory, which we will touch more on later.
Figure 3: Some paging structures may not map to a physical page because the virtual address range they represent is invalid.
Anatomy of a virtual address
Information is encoded in a virtual address that makes the translation to a physical address possible. In 64-bit mode, we use 4-level paging, which means that any given virtual address can be divided into 6 sections with 4 of them associated with the different paging structures.
The different paging structures are as follows: a PML4 table (located in CR3), a Page Directory Pointer Table (PDPT), a Page Directory (PD), and a Page Table (PT). The figure below illustrates which bits of a given virtual address map to these different paging structures.
A single entry in the PML4 table (a PML4E) can address up to 512GB of memory, while an entry in the PDPT (a PDPTE) can address 1GB (parent granularity divided by 512) of memory, and so on. This is how we get the granularity of the paging structures down to 4KB at the lowest level.
Figure 4: The anatomy of a virtual address in 64-bit mode.
In the example above, we see that the highest bits (bits 63-48) are reserved. We will talk more about these bits in a future article, but for the purposes of address translation they are not used.
The next 9 bits (bits 47-39) are used to identify the index into the PML4 table that contains the entry (PML4E) that's next in our paging structure walk. For example, if these 9 bits evaluate to the number 16, then the 16th entry in the table (PML4[15]) is selected to be used for the address translation.
Once we have the PML4E entry from the given index, we can use that entry to provide us the address of the start of the next paging structure to walk to. Here is an example of what a PML4E structure would look like in C++.
Using the page frame number (PFN) member of the structure (in this case, it actually refers to the page frame where the next structure is located), we can now walk to the next structure in the hierarchy by multiplying that number by the size of a page (0x1000). The result of that multiplication is the physical address where the next paging structure is located. The PML4E points to a Page Directory Pointer Table (PDPT). We use the next 9 bits of our original virtual address (bits 38-30) to determine the index in the PDPT that we want to look at. At that index, we will find a PDPTE structure, like the one defined below.
It's worth noting at this point that paging structures other than those in the PML4 table contain a Page Size (PS) bit (bit 7). If this bit is set, then the current entry represents the physical page. This means that page sizes as large as 1GB can be supported, if the associated PDPTE indicates that it is a 1GB page by setting the PS bit. Otherwise, 2MB pages can be supported if the PS bit is set in the PDE structure. Not all processors support the PS bit being set in a PDPTE; therefore, not all processors will support 1GB pages.
Moving along in our example, we can assume that the PS bit is not set in the PDPTE that we just referenced. So, we will look at the page frame number of this structure and multiply by the page size again to get the physical address of the next paging structure root.
Figure 5: Our walk so far, from the CR3 register, through the PML4 and PDPT structures.
Using the PFN stored in the PDPTE structure, we're able to locate the Page Directory paging structure, which is next in the hierarchy. As before, we use the next 9 bits (bits 29-21) of the original virtual address to get the index into this structure where our entry of interest (a PDE, in this case) resides. The PDE structure is defined similarly to the previous structures, as shown below.
Again, we can use the PFN member of this structure multiplied by the size of a page to locate the next, and final, paging structure that facilitates the translation - the Page Table. The next 9 bits (bits 20-12) of our original virtual address are the index into the Page Table where the associated entry (PTE) is located. This PTE structure is defined below, and once again has similar characteristics to its predecessors.
The PFN member of this structure indicates the real page frame of the backing physical memory. Because our example went the full depth of the paging structures, the size of a page frame is 4KB, or 0x1000. Thus, in order to get the location in physical memory where the backing page begins, we multiply the page frame number from the PTE by 0x1000 as we had been doing previously. The remaining 12 bits (bits 11-0) of the original virtual address are the offset into the physical page where the actual data resides. Had our example not used the full depth of the paging structures, and had instead used 2MB page sizes (stopping at the Page Directory level), that PDE would have contained the page frame number of interest, and we would have multiplied that number by the size of a page frame, which in that case would be 2MB or 0x20000. We would then add the offset into the page, which would be the remaining bits (bits 20-0) of the original virtual address since we did not need to use the usual 9 bits for indexing into a Page Table structure.
Figure 6: Here we have a full traversal of the paging structures from CR3 all the way to the final PTE. We use the PFN from the PTE to calculate the backing physical page.
Practical exploration with WinDbg
We can use WinDbg to explore what this structure hierarchy looks like in practice. Windows does some things differently (such as per-process CR3 to keep the virtual address spaces of processes separate) and there are certain complexities that we will cover in a future article, but we will choose a simple example that demonstrates what we've just learned.
Attach an instance of WinDbg as a kernel debugger to the virtual machine or physical box of your choice to get started. Check out this article for instructions on how to do so.
Once we've broken in, use the lm command to list the modules that have been loaded by the current process.
We'll use the image base of ntdll.dll as our example. It's located at 0x00000000`771d0000. We can view the memory at that virtual address by using db (or dX, where X is your desired format specifier).
Here we can see the signature 'MZ' as we would expect from a DOS header. But where are these bytes located in physical memory? There are two ways we can find out.
The first way is the hard way - we can get the value stored in CR3 which gives us the beginning of our PML4 paging structure, and begin our manual walk like we described above.
This means that the start of our PML4 table is located at physical address 0x187000. We can take a look at the physical memory at that location using !dq (or !dX, again where X is the format specifier you want to use). We're aligning on a quad-word because the size of each entry in any paging structure in 64-bit mode is 8 bytes.
Here we see that we have one PML4E structure, with 0x00700007`ddc82867 as the value. For a paging structure entry, we know that bits 47-12 represent the page frame number of the next paging structure. So we extract those bits to get 0x7ddc82, then multiply it by the size of a page frame on this architecture (4KB) to get a physical address of 0x00000007`ddc82000.
If we navigate to that physical address, let's see what we get.
Sure enough, there are two PDPTE entries (or potentially more, off-screen, since there can be up to 512 listed) here in this PDPT that we've walked to. In order to figure out which PDPTE we need to reference, we'd need to refer to the 9 bits in the original virtual address that map to the PDPT (bits 39-30), which in the case of our example works out to 0x1. That means we want the second entry of the PDPT structure, at index 1.
We can extract the page frame number from that PDPTE entry using the same bits we used in the last example (bits 47-12), resulting in 0x7d96b8. Let's multiply that number by 4KB, and see what we've got at that physical address.
You may be wondering at this point: what's going on? Why is there nothing in the PD structure that was referenced by our PDPTE? Remember, not all memory is valid and mapped, so the fact that we are seeing a bunch of zero-value PDE entries isn't a surprise. It just means that those regions of virtual memory aren't currently mapped to a physical page. In order to get to the PDE we care about, we need to take the next 9 bits of the original virtual address as we did before, this time getting a value of 0x1b8 after extracting the bits. That will get us the index into the PD structure where our PDE of interest is located. We can navigate to that memory location now, remembering to multiply the index by the size of a paging structure entry, which is 8 bytes.
That gets us 0x67e00007`d96b9867 as our PDE value. Once again, we extract the bits that are relevant to the page frame number, and we come up with 0x7d96b9.
We can repeat the steps we've taken previously to multiply that page frame number by 4KB, add the PT index using the next 9 bits of the original virtual address (0x1d0 in this case), then navigate to the correct physical address.
We've gotten the value 0xe7d00007`d9cc0025 for our PTE entry. We're almost there! We just need to do the same steps we've been doing one more time - extract the PFN from that value (0x7d9cc0), multiply by the size of a page (0x1000), but this time, we need to add the page offset (bits 11-0) from our original virtual address to the result. This should get us to 0x00000007`d9cc0000 since our page offset in this example was actually zero. Let's look at the memory!
There's the header, just like we expected. That's a cumbersome amount of work, though, and we don't want to have to be doing that manually every time we try to translate an address. Luckily, there's an easier way.
WinDbg provides the !pte command to illustrate the entire walk down the paging structures and what each entry contains. It is important to note, though, that the addresses of the paging structures are converted to virtual addresses before being displayed, so they will look different from the physical addresses we extrapolated on our own, but they point to the same memory.
You can see that WinDbg gives us the address of the paging structure used, what it contained, and the page frame number for each. You can verify that the PFN on the PXE (PML4E) entry matches up with what we calculated, too. The most important part of all of this information is the PFN that's within the lowermost entry, the PTE. In our case it's 0x7d9cc0.
So, we can multiply that page frame number by 0x1000 to get 0x00000007`d9cc0000, and that should be the physical address of the DOS header of ntdll.dll! This checks out based on the manual calculations we did previously, but let's take a look again to make sure.
And there it is! We can test this by editing the DOS header in WinDbg and seeing if those changes are reflected on the physical page.
Let's check it out using the virtual address...
...and the physical address...
And there you have it! We now know how to successfully walk the IA-32e paging structures to convert a virtual address into a physical address.
Setting up WinDbg for kernel-mode debugging is a fairly trivial process, however, it's easy to miss (or incorrectly configure) a step causing you to waste precious time. In this post, I have written a tutorial that goes through the entire process of setting up WinDbg (and configuring symbol lookup) for kernel-mode debugging with VMware using a named pipe and a virtual serial connection.
Serial port debugging was chosen for compatibility reasons. Other debugging modes like ethernet/network, while quicker, require special hardware (e.g. certain network interface cards are compatible and many are not) and are only supported on newer versions of Windows.
Requirements
A copy of either VMware Workstation (free 30-day trial) or VMware Player (entirely free for non-commercial use) for Windows. I'll be using VMware Workstation 12.5.7 (build-5813279).
A Windows operating system installed on your host and guest (VM). These do not have to be the same versions of Windows, but should be running at least Windows XP or later. My host and guest OS are both running Windows x64 10.0.15063 (Version 1703).
A free copy of Windows 10 can be found here as long as the tool is run on a machine that has a valid Windows license (of any version). Follow the steps to create an ISO file. Use the ISO file to install the OS on the Virtual Machine (helpful documentation can be found on the VMware website and WikiHow).
WinDbg.
The latest and greatest version can be downloaded from this page (direct link). This requires installation through the Windows SDK, however, you can unselect all components except "Debugging Tools for Windows" if you do not plan on doing any software development. I'll be using WinDbg x64 10.0.15063.400.
Setting up symbols on your host
Microsoft provides stripped ("redacted") PDBs (commonly referred to as "symbols") for most of their software releases. This includes the kernel components that power the operating system. In order to leverage this very useful information, we'll need to setup WinDbg so it can access these resources.
Locate your WinDbg installation.
For most people, this will be located in the following directory: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64
Right-click on the windbg.exe file and select "Create shortcut". This shortcut should be placed on the desktop or another convenient place.
Right-click on the shortcut that you just created. Select "Properties". In the "Shortcut" tab, you'll see a window similar to this:
Select the "Target:" text field and append a string of the following format: -y "srv*c:\symbols*https://msdl.microsoft.com/download/symbols"
The syntax of this command string is: srv*[local cache]*[private symbol server]*https://msdl.microsoft.com/download/symbols
This will download all available symbols, as necessary, from the Microsoft Symbol Server to your local symbol directory at c:\symbols. If you prefer to place your downloaded symbols somewhere else, choose another local path instead.
This command supports multiple symbol servers. For example, if you wish to pull symbols from a remote share, you can append to this path, e.g:
srv*c:\symbols*\\mainserver\symbols*https://msdl.microsoft.com/download/symbols
Example of a fully qualified "Target:" text field:
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\windbg.exe" -y "srv*c:\symbols*https://msdl.microsoft.com/download/symbols"
Save your shortcut changes by pushing 'OK'. Now that we've instructed WinDbg to pull symbol information from the Microsoft Symbol Server, let's test it to ensure that everything is working.
We pass the symbol path via a command line parameter to WinDbg for reliability reasons. We could have, alternatively, configured an environment variable, _NT_SYMBOL_PATH, to achieve the same functionality, but it's a less elegant solution.
Run the shortcut and a copy of the pre-installed application Notepad.
Select "File" then "Attach to a Process..." (or hit F6) in WinDbg. Scroll down all the way and select "notepad.exe" from your process list. Then hit 'OK'.
A "Command" window will appear. We'll use this to issue commands to WinDbg.
I like to expand my "Command" window so it takes up the full view in the debugger. You can do this by right clicking on the "Command" window title and selecting "Dock":
Let's try to load symbols for all the running modules (executable and DLLs). First, let's list what modules are currently loaded in our process by using the lm (list modules) command in the text box immediately to the right of ">". This is in the bottom left corner of the "Command" window:
If your list looks different from mine, don't worry. Different versions of Windows and different versions of Notepad will have different modules loaded.
Next, let's force a symbol load of all modules within our process by executing .reload /f:
Pro-tip: WinDbg has a great manual. To access it, you can type the command .hh within the debugger (or select "Help" and then "Search" from the menu bar). Typing .hh search terms go here automatically runs a search for the user supplied argument.
.hh .reload documents the .reload command. In particular, it explains why the /f argument is supplied.
It may take WinDbg a few moments to load all symbol information. You can see the status of WinDbg in the bottom left corner (next to where commands are inserted). After WinDbg has loaded symbols, run the lm command again.
Pro-tip: If WinDbg stays "BUSY" for a long time, you can force it to stop its current task by pushing Ctrl+Break on your keyboard or by selecting "Debug" and then "Break" from the menu bar.
As you can see, most modules now have a local symbol path listed to the right of their module name. It's very possible that there may be some modules that still do not have symbols loaded. These modules are most likely not distributed by Microsoft (e.g. 3rd party antivirus vendors).
For validation, go to the directory that you've setup for your local symbol cache, e.g. C:\symbols. If the folder contains data, you're set and can skip the troubleshooting step.
Troubleshooting
Verbose output
The easiest way to troubleshoot problems with symbol loading is to enable verbose output with the !sym noisy command:
Next, issue the .reload /f command.
In my example, it's easy to see that I mistyped the URL to the Microsoft Symbol Server in my shortcut's "Target:" field. After applying the right URL to my "Target:" field, I can restart WinDbg and try again.
The lazy fix
WinDbg may be able to fix the problem for you automagically if you issue .symfix and then .reload /f. In this case, WinDbg will alter your symbol path to the Microsoft Symbol Server. Your downloaded symbols will be stored, locally, in WinDbg's current working directory (C:\Program Files (x86)\Windows Kits\10\Debuggers\x64) or C:\ProgramData\dbg.
Setting up VMware on your host
Select the VM you wish to enable kernel-mode debugging on within VMware.
VMs should be listed in the "Library" pane on the left of the GUI. If the "Library" pane is missing, you can restore it by selecting "View" then "Customize" and choosing "Library" (or hit F9).
If your VM is not listed in the "Library" pane, you can manually navigate to it's .vmx file via "File" and then "Open..." (or Control+O).
Ensure that the VM is currently not running. If it's currently active, power it off via the menu bar: "VM" then "Power" then "Shut Down Guest" (or Ctrl+E).
Select "Edit virtual machine settings". Ensure that you are on the "Hardware" tab.
Select the "Add" button and choose "Serial Port" from the "Add Hardware Wizard". Hit "Next >".
Ensure that the "Serial port" checkbox is targeting "Output to named pipe" and then hit "Next >".
On the final screen, you should see similar settings to this. Make a note of the "Named pipe" field and then hit "Finish".
Ensure that your settings match those above. In particular, output to a "Named pipe" at \\.\pipe\com_1 and ensure that the first drop down box has "This end is the server" selected and the last drop down box has "The other end is a virtual machine" selected. Finally, make sure that you've selected "Connect at power on".
The com_1 substring can be changed to something else (e.g. kdebug), but it needs to be remembered and the exact name should be used within WinDbg too.
The "Add Hardware Wizard" will now close and a new "Serial port" will be added to your "Hardware" tab. Ensure the "Yield CPU on poll" checkbox is selected in "Virtual Machine Settings". Make a note of the number to the right of "Serial Port" (if there is no number, it's assumed to be 1).
In my example, my serial port is number 2.
The 'Printer' is using "Serial Port 1".
In the guest (Virtual Machine) context
For guests (VMs) running Windows Vista and later.
Start the VM.
After Windows is finished loading, run "Command Prompt" (Start+R > cmd.exe) as an Administrator.
In Windows 10, you can right-click on the Windows logo in the taskbar (bottom-left) and select "Command Prompt (Admin)".
Input the following commands in this elevated prompt:
bcdedit /debug on
bcdedit /dbgsettings serial debugport:2 baudrate:115200
Make sure your debugport argument matches your serial port number from step 7 in the "Setting up VMware" section. My serial port number is 2 because my VM has a printer that is using serial port number 1.
Pro-tip: You can add the /noumex switch to the the dbgsettings command, e.g. bcdedit /dbgsettings serial debugport:2 baudrate:115200 /noumex. This avoids user mode exceptions from causing the system to break into the kernel debugger.
Now validate that the settings have been successfully applied:
bcdedit /dbgsettings
bcdedit
You should see similar command prompt output to this:
Finally, shutdown Windows cleanly. You can do this via the traditional route (the start menu) or by executing the shutdown -s -t 0 command in command prompt.
For guests (VMs) running Windows XP.
Start the VM.
bcdedit does not exist on Windows XP. To enable kernel debugging, you must alter the boot.ini file. The easiest way to do this is by clicking on Start and then Run (Start+R). Enter C:\boot.ini as the argument and hit 'OK'.
You might have to change the drive letter (from C:\) if your operating system is installed on a different drive.
This file is hidden (and considered a protected operating system file). Therefore, it won't be displayed in Windows Explorer by default.
Append the string /debug /debugport=COM2 /baudrate=115200 to the end of the first entry in the [operating systems] section.
Make sure your debugport argument matches your serial port number from step 7 in the "Setting up VMware" section. My serial port number is 2 (hence COM2) because my VM has a printer that is using serial port number 1.
Save the boot.ini via "File" and then "Save" from the menu bar (or hit Control+S). Close the file.
Finally, shutdown Windows cleanly via the traditional route (the start menu).
Finalizing WinDbg on your host
Open the shortcut to your WinDbg that you created in step 2 in the "Setting up symbols on your host" section.
Click on "File" and then "Kernel Debug..." (or press Ctrl+K). Select the "COM" tab and use your settings from the previous sections. If you've been following the tutorial verbatim, you can just use these settings:
Finally, hit 'OK' and launch your Virtual Machine. WinDbg should automatically establish a connection to VMware when Windows begins loading.
Break into the debugger by pressing Ctrl+Break or by selecting "Debug" and then "Break" from the menu bar. At this point, the Virtual Machine will be in a suspended state (e.g. Windows will stop loading).
Load your kernel symbols with a .reload /f command. Then list the loaded modules via lm. If you're having troubles loading symbols, review the "Setting up symbols on your host" section above and work through the "Troubleshooting" tips if all else fails.
Congratulations. At this point you've successfully set up kernel debugging using WinDbg and VMware over a virtual serial connection.
Extra special bonus stage
Modifying the shortcut to start kernel debugging immediately
Having to manually configure WinDbg each time for kernel debugging is a real pain. Luckily, there is a better way.
Right-click on the shortcut that you created for WinDbg. Select "Properties". In the "Shortcut" tab, you'll see a window similar to this:
Append the following string to the "Target:" textbox: -k com:pipe,port=\\.\pipe\com_1,resets=0,reconnect
You might have to change the pipe name from com_1 to whatever you selected in step 6 in the "Setting up VMware on your host" section.
The final "Target:" argument should look similar to this: "C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\windbg.exe" -y "srv*c:\symbols*https://msdl.microsoft.com/download/symbols" -k com:pipe,port=\\.\pipe\com_1,resets=0,reconnect
Hit 'OK' and you should be all set. Now when you run this shortcut of WinDbg, it will correctly configure your symbol path (without having to use yucky environment variables) and will automatically start kernel debugging the first active named pipe.
Login
