This blog post continues the FLARE script series with a discussion of
patching IDA Pro database files (IDBs) to interactively emulate code.
While the fastest way to analyze or unpack malware is often to run it,
malware won’t always successfully execute in a VM. I use IDA
Pro’s Bochs integration in IDB mode to sidestep tedious
debugging scenarios and get quick results. Bochs emulates the opcodes
directly from your IDB in a Bochs VM with no OS.
Bochs IDB mode eliminates distractions like switching VMs, debugger
setup, neutralizing anti-analysis measures, and navigating the program
counter to the logic of interest. Alas, where there is no OS, there
can be no loader or dynamic imports. Execution is constrained to
opcodes found in the IDB. This precludes emulating routines that call
imported string functions or memory allocators. Tom Bennett’s flare-emu
ships with emulated versions of these, but for off-the-cuff analysis
(especially when I don’t know if there will be a payoff), I prefer
interactively examining registers and memory to adjust my tactics ad hoc.
What if I could bring my own imported functions to Bochs like
flare-emu does? I’ve devised such a technique, and I call it code
grafting. In this post I’ll discuss the particulars of statically
linking stand-ins for common functions into an IDB to get more mileage
out of Bochs. I’ll demonstrate using this on an EVILNEST sample to
unpack and dump next-stage payloads from emulated memory. I’ll also
show how I copied a tricky call sequence from one IDB to another IDB
so I could keep the unpacking process all in a single Bochs debug session.
EVILNEST Scenario
My sample (MD5 hash 37F7F1F691D42DCAD6AE740E6D9CAB63 which is
available on VirusTotal)
was an EVILNEST variant that populates the stack with configuration
data before calling an intermediate payload. Figure 1 shows this
unusual call site.
Figure 1: Call site for intermediate payload
The code in Figure 1 executes in a remote thread within a
hollowed-out iexplore.exe process; the
malware uses anti-analysis tactics as well. I had the intermediate
payload stage and wanted to unpack next-stage payloads without
managing a multi-process debugging scenario with anti-analysis. I knew
I could stub out a few function calls in the malware to run all of the
relevant logic in Bochs. Here’s how I did it.
Code Carving
I needed opcodes for a few common functions to inject into my IDBs
and emulate in Bochs. I built simple C implementations of selected
functions and compiled them into one binary. Figure 2 shows some of
these stand-ins.
Figure 2: Simple implementations of common functions
I compiled this and then used IDAPython code similar to Figure 3 to
extract the function opcode bytes.
Figure 3: Function extraction
I curated a library of function opcodes in an IDAPython script as
shown in Figure 4. The nonstandard function opcodes at the bottom of
the figure were hand-assembled as tersely as possible to generically
return specific values and manipulate the stack (or not) in
conformance with calling conventions.
Figure 4: Extracted function opcodes
On top of simple functions like memcpy, I
implemented a memory allocator. The allocator referenced global state
data, meaning I couldn’t just inject it into an IDB and expect it to
work. I read the disassembly to find references to global operands and
templatize them for use with Python’s format
method. Figure 5 shows an example for malloc.
Figure 5: HeapAlloc template code
I organized the stubs by name as shown in Figure 6 both to call out
functions I would need to patch, and to conveniently add more function
stubs as I encounter use cases for them. The mangled name I specified
as an alias for free is operator delete.
Figure 6: Function stubs and associated names
To inject these functions into the binary, I wrote code to find the
next available segment of a given size. I avoided occupying low memory
because Bochs places its loader segment below 0x10000. Adjacent to the code in my code segment,
I included space for the data used by my memory allocator. Figure 7
shows the result of patching these functions and data into the IDB and
naming each location (stub functions are prefixed with stub_).
Figure 7: Data and code injected into IDB
The script then iterates all the relevant calls in the binary and
patches them with calls to their stub implementations in the newly
added segment. As shown in Figure 8, IDAPython’s Assemble function saved the effort of calculating
the offset for the call operand manually.
Note that the Assemble function worked well
here, but for bigger tasks, Hex-Rays
recommends a dedicated assembler such as Keystone Engine and its
Keypatch
plugin for IDA Pro.
Figure 8: Abbreviated routine for
assembling a call instruction and patching a call site to an import
The Code Grafting script updated all the relevant call sites to
resemble Figure 9, with the target functions being replaced by calls
to the stub_ implementations injected
earlier. This prevented Bochs in IDB mode from getting derailed when
hitting these call sites, because the call operands now pointed to
valid code inside the IDB.
Figure 9: Patched operator new() call site
Dealing with EVILNEST
The debug scenario for the dropper was slightly inconvenient, and
simultaneously, it was setting up a very unusual call site for the
payload entry point. I used Bochs to execute the dropper until it
placed the configuration data on the stack, and then I used
IDAPython’s idc.get_bytes function to
extract the resulting stack data. I wrote IDAPython script code to
iterate the stack data and assemble push instructions into the payload
IDB leading up to a call instruction pointing to the DLL’s export.
This allowed me to debug the unpacking process from Bochs within a
single session.
I clicked on the beginning of my synthesized call site and hit F4 to
run it in Bochs. I was greeted with the warning in Figure 10
indicating that the patched IDB would not match the depictions made by
the debugger (which is untrue in the case of Bochs IDB mode). Bochs
faithfully executed my injected opcodes producing exactly the desired result.
Figure 10: Patch warning
I watched carefully as the instruction pointer approached and passed
the IsDebuggerPresent check. Because of the
stub I injected (stub_IsDebuggerPresent), it
passed the check returning zero as shown in Figure 11.
Figure 11: Passing up IsDebuggerPresent
I allowed the program counter to advance to address 0x1A1538, just beyond the unpacking routine.
Figure 12 shows the register state at this point which reflects a
value in EAX that was handed out by my fake
heap allocator and which I was about to visit.
Figure 12: Running to the end of the
unpacker and preparing to view the result
Figure 13 shows that there was indeed an IMAGE_DOS_SIGNATURE (“MZ”) at this location. I
used idc.get_bytes() to dump the unpacked
binary from the fake heap location and saved it for analysis.
Figure 13: Dumping the unpacked binary
Through Bochs IDB mode, I was also able to use the interactive
debugger interface of IDA Pro to experiment with manipulating
execution and traversing a different branch to unpack another payload
for this malware as well.
Conclusion
Although dynamic analysis is sometimes the fastest road, setting it
up and navigating minutia detract from my focus, so I’ve developed an
eye for routines that I can likely emulate in Bochs to dodge those
distractions while still getting answers. Injecting code into an IDB
broadens the set of functions that I can do this with, letting me get
more out of Bochs. This in turn lets me do more on-the-fly
experimentation, one-off string decodes, or validation of hypotheses
before attacking something at scale. It also allows me to experiment
dynamically with samples that won’t load correctly anyway, such as
unpacked code with damaged or incorrect PE headers.
I’ve shared the Code Grafting tools as part of the flare-ida GitHub
repository. To use this for your own analyses:
In IDA Pro’s IDAPython prompt, run code_grafter.py or import it as a module.
Instantiate a CodeGrafter object and
invoke its graftCodeToIdb() method:
CodeGrafter().graftCodeToIdb()
Use Bochs in IDB mode to conveniently execute your
modified sample and experiment away!
This post makes it clear just how far I’ll go to avoid breaking eye
contact with IDA. If you’re a fan of using Bochs with IDA too, then
this is my gift to you. Enjoy!
We’re proud to release a new plug-in for IDA Pro users –
SimplifyGraph – to help automate creation of groups of nodes in the
IDA’s disassembly graph view. Code and binaries are available from the
FireEye GitHub
repo. Prior to this release we submitted it in the 2017
Hex-Rays plugin contest, where it placed third overall.
My personal preference is to use IDA’s graph mode when doing the
majority of my reverse engineering. It provides a graphical
representation of the control flow graph and gives visual cues about
the structure of the current function that helps me better understand
the disassembly.
Graph mode is great until the function becomes complex. IDA is often
forced to place adjacent nodes relatively far apart, or have edges in
the graph cross and have complex paths. Using the overview graph
becomes extremely difficult due to the density of nodes and edges, as
seen in Figure 1.
Figure 1: An annoying function
IDA has a built-in mechanism to help simplify graphs: creating
groups of nodes, which replaces all of the selected nodes with a new
group node representative. This is done by selecting one or more
nodes, right-clicking, and selecting “Group nodes”, as shown in Figure
2. Doing this manually is certainly possible, but it becomes tedious
to follow edges in complex graphs and correctly select all of the
relevant nodes without missing any, and without making mistakes.
Figure 2: Manual group creation
The SimplifyGraph IDA Pro plug-in we’re releasing is built to
automate IDA’s node grouping capability. The plug-in is
source-compatible with the legacy IDA SDK in 6.95, and has been ported
to the new SDK for IDA 7.0. Pre-built binaries for both are available
on the Releases tab
for the project repository.
The plug-in has several parts, which are introduced below. By
combining these together it’s possible to isolate parts of a control
flow graph for in-depth reverse engineering, allowing you to look at
Figure 3 instead of Figure 1.
Figure 3: Isolated subgraph to focus on
Create Unique-Reachable (UR) Subgraph
Unique-Reachable nodes are all nodes reachable in the graph from a
given start node and that are not reachable from any nodes not
currently in the UR set. For example, in Figure 4, all of the
Unique-Reachable nodes starting at the green node are highlighted in
blue. The grey node is reachable from the green node, but because it
is reachable from other nodes not in the current UR set it is pruned
prior to group creation.
Figure 4: Example Unique Reachable selection
The plug-in allows you to easily create a new group based on the UR
definition. Select a node in IDA's graph view to be the start of the
reachable search. Right click and select "SimplifyGraph ->
Create unique-reachable group". The plug-in performs a graph
traversal starting at this node, identifies all reachable nodes, and
prunes any nodes (and their reachable nodes) that have predecessor
nodes not in the current set. It then prompts you for the node text to
appear in the new group node.
If you select more than one node (by holding the Ctrl key when
selecting nodes) for the UR algorithm, each additional node acts as a
sentry node. Sentry nodes will not be included in the new group, and
they halt the graph traversal when searching for reachable nodes. For
example, in Figure 5, selecting the green node first treats it as the
starting node, and selecting the red node second treats it as a sentry
node. Running the “Create unique-reachable group” plug-in option
creates a new group made of the green node and all blue nodes. This
can be useful when you are done analyzing a subset of the current
graph, and wish to hide the details behind a group node so you can
concentrate on the rest of the graph.
Figure 5: Unique reachable with sentry
The UR algorithm operates on the currently visible graph, meaning
that you can run the UR algorithm repeatedly and nest groups.
Switch Case Groups Creation
Switch statements implemented as jump tables appear in the graph as
nodes with a large fan-out, as shown in Figure 6. The SimplifyGraph
plug-in detects when the currently selected node has more than two
successor nodes and adds a right-click menu option “SimplifyGraph
-> Create switch case subgraphs”. Selecting this runs the
Unique-Reachable algorithm on each separate case branch and
automatically uses IDA’s branch label as the group node text.
Figure 6: Switch jumptable use
Figure 7 shows a before and after graph overview of the same
function when the switch-case grouping is run.
Figure 7: Before and after of switch
statement groupings
Isolated Subgraphs
Running Edit -> Plugins -> SimplifyGraph brings up a new
chooser named "SimplifyGraph - Isolated subgraphs" that
begins showing what I call isolated subgraphs of the current graph, as
seen in Figure 8.
Figure 8: Example isolated subgraphs chooser
A full definition appears later in the appendix including how these
are calculated, but the gist is that an isolated subgraph in a
directed graph is a subset of nodes and edges such that there is a
single entrance node, a single exit node, and none of the nodes (other
than the subgraph entry node) is reachable by nodes not in the subgraph.
Finding isolated subgraphs was originally researched to help
automatically identify inline functions. It does this, but it turns
out that this graph construct occurs naturally in code without inline
functions. This isn’t a bad thing as it shows a natural grouping of
nodes that could be a good candidate to group to help simplify the
overall graph and make analysis easier.
Once the chooser is active, you can double click (or press Enter) on
a row in the chooser to highlight the nodes that make up the subgraph,
as seen in Figure 9.
Figure 9: Highlighted isolated subgraph
You can create a group for an isolated subgraph by:
Right-clicking on the
chooser row and selecting "Create group", or pressing
Insert while a row is selected.
Right-clicking in a
highlighted isolated subgraph node and selecting "SimplifyGraph
-> Create isolated subgraph".
Doing either of these prompts you for text for the new graph node to create.
If you manually create/delete groups using IDA you may need to
refresh the chooser's knowledge of the current function groups
(right-click and select "Refresh groups" in the chooser).
You can right click in the chooser and select "Clear
highlights" to remove the current highlights. As you navigate to
new functions the chooser updates to show isolated subgraphs in the
current function. Closing the chooser removes any active highlights.
Any custom colors you applied prior to running the plug-in are
preserved and reapplied when the current highlights are removed.
Isolated subgraph calculations operates on the original control flow
graph, so isolated subgroups can't be nested. As you create groups,
rows in the chooser turn red indicating a group already exists, or
can't be created because there is an overlap with an existing group.
Another note: this calculation does not currently work on functions
that do not return (those with an infinite loop). See the Appendix for details.
Graph Complement
Creating groups to simplify the overall control flow graph is nice,
but it doesn’t help understand the details of a group that you create.
To assist with this, the last feature of the plug-in hides everything
but the group you’re interested in allowing you to focus on your
reverse engineering. Right clicking on a collapsed group node, or a
node that that belongs to an uncollapsed group (as highlighted by IDA
in yellow), brings up the plug-in option “Complement & expand
group” and “Complement group”, respectively. When this runs the
plug-in creates a group of all nodes other than the group you’re
interested in. This has the effect of hiding all graph nodes that you
aren’t currently examining and allows you to better focus on analysis
of the current group. As you can see, we’re abusing group creation a
bit so that we can avoid creating a custom graph viewer, and instead
stay within the built-in IDA graph disassembly view which allows us to
continue to markup the disassembly as you’re used to.
Complementing the graph gives you the view seen in Figure 10, where
the entire graph is grouped into a node named “Complement of group X”.
When you’re done analyzing the current group, right click on the
complement node and select IDA’s “Ungroup nodes” command.
Figure 10: Group complement
Example Workflow
As an example that exercises the plug-in, let’s revisit the function
in Figure 1. This is a large command-and-control dispatch function for
a piece of malware. It contains a large if-else-if series of inlined
strcmp comparisons that branch to the logic for each command when the
input string matches the expected command.
Find all of the inline
strcmp’s and create groups for those. Run Edit -> Plugins ->
SimplifyGraph to bring up the plug-in chooser. In this function
nearly every isolated subgraph is a 7-node inlined strcmp
implementation. Go through in the chooser to verify, and create a
group. This results in a graph similar to Figure 11.
Figure 11: Grouped strcmp
When an input string matches a command string, the
malware branches to code that implements the command. To further
simplify the graph and make analysis easier, run the
Unique-Reachable algorithm on each separate command by right
clicking on the first node after each string-comparison and
selecting SimplifyGraph -> Create unique-reachable group. After
this we now have a graph as in Figure 12.
Figure 12: Grouped command logic
Now perform your reverse engineering on each
separate branch in the dispatch function. For each command handler
group node that we created, right click that node and select
“SimplifyGraph -> Complement & expand group”. A result of
complementing a single command handler node is shown in Figure 13,
which is much easier to analyze.
Figure 13: Group complement
When done analyzing the current command handler,
delete the complement group by right clicking the “Complement of
group X” node and use IDA’s built-in “Ungroup nodes” command. Repeat
for the remaining command handler grouped nodes.
Config
You can tweak some of the configuration by entering data in a file
named %IDAUSR%/SimplifyGraph.cfg, where %IDAUSR% is typically
%APPDATA%/Hex-Rays/IDA Pro/ unless explicitly set to something else.
All of the config applies to the isolated subgraph component. Options:
* SUBGRAPH_HIGHLIGHT_COLOR: Default 0xb3ffb3: The color to apply to
nodes when you double click/press enter in the chooser to show nodes
that make up the currently selected isolated subgraph. Not everyone
agrees that my IDA color scheme is best, so you can set your own
highlight color here.
* MINIMUM_SUBGRAPH_NODE_COUNT: Default 3: The minimum number of
nodes for a valid isolated subgraph. If a discovered subgraph has
fewer nodes than this number it is not included in the shown list.
This prevents trivial two-node subgraphs from being shown.
* MAXIMUM_SUBGRAPH_NODE_PERCENTAGE: Default 95: The maximum percent
of group nodes (100.0 *(subgroup_node_count /
total_function_node_count)) allowed. This filters out isolated
subgraphs that make up (nearly) the entire function, which are
typically not interesting.
Example SimplifyGraph.cfg contents
```
"MINIMUM_SUBGRAPH_NODE_COUNT"=5
"MAXIMUM_SUBGRAPH_NODE_PERCENTAGE"=75
"SUBGRAPH_HIGHLIGHT_COLOR"=0x00aa1111
```
Prior work:
I came across semi-related work while working on this: GraphSlick from
the 2014
Hex-Rays contest. That plug-in had different goals to
automatically identifying (nearly) identical inline functions via CFG
and basic block analysis, and patching the program to force mock
function calls to the explicit function. It had a separate viewer to
present information to the user.
SimplifyGraph is focused on automating tasks when doing manual
reverse engineering (group creation) to reduce the complexity of
disassembly in graph mode. Future work may incorporate the same
prime-products calculations to help automatically identify isolated subgraphs.
Installation
Prebuilt Windows binaries are available from the Releases tab of the
GitHub project page. The ZIP files contain both IDA 32 and IDA
64 plug-ins for each of the new IDA 7.0 SDK and for the legacy IDA
6.95 SDK. Copy the two plug-ins for your version of IDA to the
%IDADIR%\plugins directory.
Building
This plug-in & related files were built using Visual Studio 2013
Update 5.
Environment Variables Referenced by project:
* IDASDK695: path to the extracted IDA 6.95 SDK. This should have
`include` and `lib` paths beneath it.
* IDASDK: path to the extracted IDA 7.0 (or newer) SDK. This Should
have `include` and `lib` paths beneath it.
* BOOSTDIR: path to the extracted Boost library. Should have `boost`
and `libs` paths beneath it.
The easiest way is to use the Microsoft command-line build tools:
* For IDA7.0: Launch VS2013 x64 Native Tools Command Prompt, then run:
I hope this blog has shown the power of automatically grouping nodes
within a disassembly graph view, and viewing these groups in isolation
to help with your analysis. This plug-in has become a staple of my
workflow, and we’re releasing it to the community with the hope that
others find it useful as well.
Appendix: Isolated Subgraphs
Finding isolated subgraphs relies on calculating the immediate
dominator and immediate post-dominator trees for a given function graph.
A node d dominates n if every path to n must go through d.
The immediate dominator p of node n is basically the closest
dominator to n, where there is no node t where p dominates t, and t
dominates n.
A node z post-dominates a node n if every path from n to the exit
node must go through z.
The immediate post-dominator x of node n is the closest
post-dominator, where there is no node t where t post-dominates n and
x post-dominates t.
The immediate dominator relationship forms a tree of nodes, where
every node has an immediate dominator other than the entry node.
The Lengauer-Tarjan algorithm can efficiently calculate the
immediate dominator tree of a graph. It can also calculate the
immediate post-dominator tree by reversing the direction of each edge
in the same graph.
The plug-in calculates the immediate dominator tree and immediate
post-dominator tree of the function control flow graph and looks for
the situations where the (idom[i] == j) and (ipdom[j] == i). This
means all paths from the function start to node i must go through node
j, and all paths from j to the function terminal must go through i. A
candidate isolated subgraph thus starts at node j and ends at node i.
For each candidate isolated subgraph, the plug-in further verifies
only the entry node has predecessor nodes not in the candidate
subgraph. The plug-in also filters out candidate subgraphs by making
sure they have a minimum node count and cover a maximum percentage of
nodes (see MINIMUM_SUBGRAPH_NODE_COUNT and
MAXIMUM_SUBGRAPH_NODE_PERCENTAGE in the config section).
One complication is that functions often have more than one terminal
node – programmers can arbitrarily return from the current function at
any point. The immediate post-dominator tree is calculated for every
terminal node, and any inconsistencies are marked as indeterminate and
are not possible candidates for use. Functions with infinite loops do
not have terminal nodes, and are not currently handled.
For a simple example, consider the graph in Figure 14. It has the
following immediate dominator and post-dominator trees:
Figure 14: Example graph
Node
idom
0
None
1
0
2
1
3
1
4
3
5
3
6
3
7
6
8
0
Node
ipdom
0
8
1
3
2
3
3
6
4
6
5
6
6
7
7
8
8
None
Looking for pairs of (idom[i] == j) and (ipdom[j] == i) gives the following:
(0, 8) (1, 3) (3, 6) (6,7)
(0, 8) is filtered because it makes up all of the nodes of the graph.
(1,3) and (6, 7) are filtered out because they contain nodes
reachable from nodes not in the set: for (1, 3) node 2 is reachable
from node 6, and for (6, 7) node 2 is reachable from node 1.
This leaves (3, 6) as the only isolate subgraph in this example,
shown in Figure 15.
The VulFi (Vulnerability Finder) tool is a plugin to IDA Pro which can be used to assist during bug hunting in binaries. Its main objective is to provide a single view with all cross-references to the most interesting functions (such as strcpy, sprintf, system, etc.). For cases where a Hexrays decompiler can be used, it will attempt to rule out calls to these functions which are not interesting from a vulnerability research perspective (think something like strcpy(dst,"Hello World!")). Without the decompiler, the rules are much simpler (to not depend on architecture) and thus only rule out the most obvious cases.
Installation
Place the vulfi.py, vulfi_prototypes.json and vulfi_rules.json files in the IDA plugin folder (cp vulfi* <IDA_PLUGIN_FOLDER>).
Preparing the Database File
Before you run VulFi make sure that you have a good understanding of the binary that you work with. Try to identify all standard functions (strcpy, memcpy, etc.) and name them accordingly. The plugin is case insensitive and thus MEMCPY, Memcpy and memcpy are all valid names. However, note that the search for the function requires exact match. This means that memcpy? or std_memcpy (or any other variant) will not be detected as a standard function and therefore will not be considered when looking for potential vulnerabilities. If you are working with an unknown binary you need to set the compiler options first Options > Compiler. After that VulFi will do its best to filter all obvious false positives (such as call to printf with constant string as a first parameter). Please note that while the plugin is made without any ties to a specific ar chitecture some processors do not have full support for specifying types and in such case VulFi will simply mark all cross-references to potentially dangerous standard functions to allow you to proceed with manual analysis. In these cases, you can benefit from the tracking features of the plugin.
Usage
Scanning
To initiate the scan, select Search > VulFi option from the top bar menu. This will either initiate a new scan, or it will read previous results stored inside the idb/i64 file. The data are automatically saved whenever you save the database.
Once the scan is completed or once the previous results are loaded a table will be presented with a view containing following columns:
IssueName - Used as a title for the suspected issue.
FunctionName - Name of the function.
FoundIn - The function that contains the potentially interesting reference.
Address - The address of the detected call.
Status - The review status, initial Not Checked is assigned to every new item. The other statuses are False Positive, Suspicious and Vulnerable. Those can be set using a right-click menu on a given item and should reflect the results of the manual review of the given function call.
Priority - An attempt to prioritize more interesting calls over the less interesting ones. Possible values are High, Medium and Low. The priorities are defined along with other rules in vulfi_rules.json file.
Comment - A user defined comment for the given item.
In case that there are no data inside the idb/i64 file or user decides to perform a new scan. The plugin will ask whether it should run the scan using the default included rules or whether it should use a custom rules file. Please note that running a new scan with already existing data does not overwrite the previously found items identified by the rule with the same name as the one with previously stored results. Therefore, running the scan again does not delete existing comments and status updates.
In the right-click context menu within the VulFi view, you can also remove the item from the results or remove all items. Please note that any comments or status updates will be lost after performing this operation.
Investigation
Whenever you would like to inspect the detected instance of a possible vulnerable function, just double-click anywhere in the desired row and IDA will take you to the memory location which was identified as potentially interesting. Using a right-click and option Set Vulfi Comment allows you to enter comment for the given instance (to justify the status for example).
Adding More Functions
The plugin also allows for creating custom rules. These rules could be defined in the IDA interface (ideal for single functions) or supplied as a custom rule file (ideal for rules that aim to cover multiple functions).
Within the Interface
When you would like to trace a custom function, which was identified during the analysis, just switch the IDA View to that function, right-click anywhere within its body and select Add current function to VulFi.
Custom Set of Rules
It is also possible to load a custom file with set of multiple rules. To create a custom rule file with the below structure you can use the included template file here.
[ // An array of rules { "name": "RULE NAME", // The name of the rule "alt_names":[ "function_name_to_look_for" // List of all function names that should be matched against the conditions defined in this rule ], "wrappers":true, // Look for wrappers of the above functions as well (note that the wrapped function has to also match the rule) "mark_if":{ "High":"True", // If evaluates to True, mark with priority High (see Rules below) "Medium":"False", // If evaluates to True, mark with priority Medium (see Rules below) "Low": "False" // If evaluates to True, mark with priority Low (see Rules below) } } ]
An example rule that looks for all cross-references to function malloc and checks whether its paramter is not constant and whether the return value of the function is checked is shown below:
param[<index>]: Used to access the parameter to a function call (index starts at 0)
function_call: Used to access the function call event
param_count: Holds the count of parameters that were passed to a function
Available Functions
Is parameter a constant: param[<index>].is_constant()
Get numeric value of parameter: param[<index>].number_value()
Get string value of parameter: param[<index>].string_value()
Is parameter set to null after the call: param[<index>].set_to_null_after_call()
Is return value of a function checked: function_call.return_value_checked(<constant_to_check>)
Examples
Mark all calls to a function where third parameter is > 5: param[2].number_value() > 5
Mark all calls to a function where the second parameter contains "%s": "%s" in param[1].string_value()
Mark all calls to a function where the second parameter is not constant: not param[1].is_constant()
Mark all calls to a function where the return value is validated against the value that is equal to the number of parameters: function_call.return_value_checked(param_count)
Mark all calls to a function where the return value is validated against any value: function_call.return_value_checked()
Mark all calls to a function where none of the parameters starting from the third are constants: all(not p.is_constant() for p in param[2:])
Mark all calls to a function where any of the parameters are constant: any(p.is_constant() for p in param)
Mark all calls to a function: True
Issues and Warnings
When you request the parameter with index that is out of bounds any call to a function will be marked as Low priority. This is a way to avoid missing cross references where it was not possible to correctly get all parameters (this mainly applies to disassembly mode).
When you search within the VulFi view and change context out of the view and come back, the view will not load. You can solve this either by terminating the search operation before switching the context, moving the VulFi view to the side-view so that it is always visible or by closing and re-opening the view (no data will be lost).
Scans for more exotic architectures end with a lot of false positives.
If you'd like to follow along, I'll be using system files from Windows x64 10.0.15063 (Creator's Update). All pseudo-source and disassembly is reconstructed from that specific release.
Don't have a kernel debugging environment set up? Don't fret. You can follow our tutorial on how to setup basic kernel debugging using WinDbg and VMware here.
Without further ado, let's begin.
What do these callbacks do?
These callbacks can be used by driver developers to gain notifications when certain events happen. For example, the basic process creation callback, nt!PsSetCreateProcessNotifyRoutine, registers a user-defined function pointer ("NotifyRoutine") that will be invoked by Windows each time a process is created or deleted. As part of the event notification, the supplied handler gets a wealth of information. In our example, this will include the parent process' (if one exists) PID, the actual process' PID, and a boolean value that will let us know if the process is being created or if it's terminating.
Security software leverages these callbacks to be able to carefully inspect code running on the machine.
Divin' deep
The documented APIs
Our investigation has to begin somewhere. What better place than at the start of a documented function? We turn to nt!PsSetCreateProcessNotifyRoutine. MSDN claims that this routine has been around since Windows 2000. Even our friends at ReactOS seem to have implemented this functionality a long time ago. We'll see exactly how (if at all) things have changed in the 17 years from Windows 2000 until now.
The only difference is in the second parameter being passed to nt!PspSetCreateProcessNotifyRoutine. These are effectively flags. In the base case (nt!PsSetCreateProcessNotifyRoutine), these flags can either be 1 or 0 depending on the state of the "Remove" parameter. If "Remove" is TRUE, Flags=1. If "Remove" is FALSE, Flags=0. In the extended case (nt!PsSetCreateProcessNotifyRoutineEx), the flags can take on the value 2 or 3:
Finally, for nt!PsSetCreateProcessNotifyRoutineEx2, these flags will take on the value 6 or 7:
Therefore, one can imply that the flags passed to nt!PspSetCreateProcessNotifyRoutine have this definition:
The undocumented world
nt!PspSetCreateProcessNotifyRoutine is slightly complicated. I've defined it below, but I strongly recommend opening it in another window and following the text to ease understanding.
Luckily for us, a lot of the internal data structures related to callback routines haven't changed since Windows 2000. The trailblazers at ReactOS have been spot-on with their structure definitions so we'll use them, when possible, to avoid duplicating work.
For each callback, there's a global array that can contain up to 64 entries. In our case, the start of this array for process creation callbacks is located at nt!PspCreateProcessNotifyRoutine. Each entry in this array is of type _EX_CALLBACK:
To avoid synchronization problems, nt!ExReferenceCallBackBlock is used which will safely acquire a reference to the underlying callback object, _EX_CALLBACK_ROUTINE_BLOCK (documented below). We can effectively reproduce the same behavior in a non-thread safe way via:
If we're deleting a callback object ("Remove" is TRUE), we need to make sure that we can find the appropriate _EX_CALLBACK_ROUTINE_BLOCK in the array. This is done by checking first if the target "NotifyRoutine" matches that of the current _EX_CALLBACK_ROUTINE with nt!ExGetCallBackBlockRoutine:
Then, we check to see if it's the right type (created with the correct version of (nt!PsSetCreateProcessNotifyRoutine/Ex/Ex2), by using nt!ExGetCallBackBlockContext:
At this point, we've found the entry in the array. We will erase it by setting the _EX_CALLBACK value to NULL via nt!ExCompareExchangeCallback, decrementing the appropriate global counter (nt!PspCreateProcessNotifyRoutineExCount or nt!PspCreateProcessNotifyRoutineCount), dereferencing the _EX_CALLBACK_ROUTINE_BLOCK with nt!ExDereferenceCallBackBlock, waiting for any other code using the _EX_CALLBACK (nt!ExWaitForCallBacks), and finally freeing memory (nt!ExFreePoolWithTag). As you can see, great care is taken by Microsoft to not free a callback object that is in use.
If we can't find the entry to remove in the nt!PspCreateProcessNotifyRoutine array after exhausting all 64 possibilities, the STATUS_PROCEDURE_NOT_FOUND error message is returned.
On the other hand, if we're adding a new entry into the callback array, things are a little easier. A sanity check is performed by nt!MmVerifyCallbackFunctionCheckFlags to ensure that the "NotifyRoutine" is present in a loaded module. This helps avoid unlinked drivers (or shellcode) from receiving callback events:
After we pass the sanity check, an _EX_CALLBACK_ROUTINE_BLOCK is allocated via nt!ExAllocateCallBack. This routine confirms the size and layout of the _EX_CALLBACK_ROUTINE_BLOCK structure:
To wrap up, the newly allocated _EX_CALLBACK_ROUTINE_BLOCK is added to a free (NULL) location in the nt!PspCreateProcessNotifyRoutine array using nt!ExCompareExchangeCallBack (ensuring that it doesn't overflow the 64 limit maximum). Finally, the appropriate global counter is incremented and a global flag is set in nt!PspNotifyEnableMask denoting that there are callbacks of the user-specified type registered on the system.
The other callbacks
Thankfully, thread and image creation callbacks are very similar to process callbacks. They utilize the same underlying data structures. The only difference is that thread creation/termination callbacks are stored in the nt!PspCreateThreadNotifyRoutine array and that image load notification callbacks are stored in nt!PspLoadImageNotifyRoutine.
The script
It's finally time to put what we know to good use. Using WinDbg, we can create a simple script to automagically enumerate process, thread, and image callback routines.
Instead of leveraging WinDbg's built-in scripting engine, I've elected to use something a little less disgusting. There's a great 3rd party extension for WinDbg called PyKd that enables Python scripting in WinDbg. Installing it is very straightforward. You'll need a copy of the appropriate bitness (e.g. 64-bit for 64-bit install of WinDbg) of Python for this to work.
The script should be easy to follow. I tried to document it as best I could. It should also be compatible, at a minimum, with all forms of Windows from XP and up (both 32-bit and 64-bit flavors).
After running the script using the "!py" command, you should see output similar to this:
Final thoughts
Knowing how the callback system functions in Windows allows us to do very interesting things. As seen above, we're able to programmatically iterate through each callback array and discover all registered callbacks. This is very useful for forensic purposes.
Furthermore, these underlying array lists aren't under the protection of PatchGuard. Since registering callbacks is more-or-less a requirement for anti-virus products in order to develop a useful driver that plays nicely with PatchGuard on x64 systems, malware could dynamically disable (or replace) these registered callbacks to thwart security protection solutions. The possibilities are endless.
Special thanks to the folks at ReactOS for their meticulous documentation. In particular, most of the structures I used were identified by Alex Ionescu for ReactOS a long time ago. Additionally, kudos to the folks that make PyKd. It's a much better alternative to the native scripting interface for WinDbg, in my opinion!
As always, if y'all have any questions or comments, please feel free to comment below. Suggestions are greatly appreciated too!
This blog post will go over an assumption made over a decade ago by Microsoft when dealing with software breakpoints that can be used to reveal the presence of most (all publicly available?) usermode and kernelmode debuggers.
The x86 architecture can potentially encode a particular assembly instruction in multiple ways. For example, adding two registers, eax and ebx, and storing the result in eax takes the following mnemonic form: add eax, ebx. This can be encoded as the byte sequence 0x03 0xC3or 0x01 0xD8. Fundamentally, the machine code represents the same assembly operation.
If you're just interested in the anti-debug trick (without any context on why it works the way it does), scroll to the bottom of this post. For the rest of you brave enough to read this article in its entirety... buckle up.
The "long form" of int 3
An int 3 can be encoded as either a single-byte 0xCC or via the more unconventional way as the multi-byte sequence 0xCD 0x03:
So, what happens when Windows encounters a multi-byte int 3? We create a simple C++ program to find out:
After you run this application, you should see output similar to this:
A single-byte int 3 (0xCC) works as expected. The start of the stub is located at 0x000001BE94B90000. When the stub is executed, the exception handler fires and we see that both the _EXCEPTION_RECORD.ExceptionAddress and _CONTEXT.Rip are located at 0x000001BE94B90000. This is the start of the int 3 instruction. Excellent!
The multi-byte int 3 (0xCD 0x03) is located at address 0x000001BE94B90002. When this stub executes, the exception handler proclaims that the _EXCEPTION_RECORD.ExceptionAddress and _CONTEXT.Rip are located at 0x000001BE94B90003. This is in the middle of the int 3 instruction. Why? What went wrong?
The assumption
NOTE: From this point on, all disassembly and pseudo-source is reconstructed from system files that are provided with Windows x64 10.0.15063 (Creator's Update). If you'd like to follow along, make sure you use the same version I'm using!
Microsoft assumes that all int 3's result from the single-byte variant.
This assumption occurs very early during interrupt processing. Namely, when any interrupt occurs, such as when an int 3 is executed by the processor, control is redirected by the CPU to a handler registered in the appropriate position of the IDT (Interrupt Descriptor Table). In Windows, the handler for software breakpoints can be found at the symbol nt!KiBreakpointTrap:
The first thing nt!KiBreakpointTrap does is generate a trap frame (_KTRAP_FRAME) on the stack that it passes to subsequent routines. A definition of this structure can be found below:
Parts of this structure are automatically filled by the CPU when the interrupt fires, in particular, the range from +0x160 (_KTRAP_FRAME.ErrorCode) to +0x188 (_KTRAP_FRAME.SegSs):
The _KTRAP_FRAME is essentially an extension of the elements saved on the stack by the CPU. It's purpose is to provide a place to store volatile registers which can be clobbered when calling into functions that are compiled in C.
A very important thing to note is that the instruction pointer (EIP) saved by the CPU on the stack (_KTRAP_FRAME.Rip) will be set to the instruction immediately following the one that caused entry into the handler. In our scenario, this means that the _KTRAP_FRAME.Rip member will be the instruction following our int 3, which will be ret (0xC3) in the example code above.
After the volatile registers have been saved off, nt!KiBreakpointTrap performs a quick check to see whether the interrupt fired from usermode (ring3) or kernelmode code (ring0). If execution is coming from ring3, a swapgs needs to occur as well as some other bookkeeping with debug registers.
Eventually, control flow will reconvene and the volatile floating point registers will also be stored off into the _KTRAP_FRAME. Before entering into more exception handling logic, the instruction pointer will be extracted from _KTRAP_FRAME.Rip (saved by the CPU upon entering nt!KiBreakpointTrap), decremented by one, and passed as an argument to nt!KiExceptionDispatch. Additionally, the exception code, EXCEPTION_BREAKPOINT (0x80000003), will also be passed in. The prototype for nt!KiExceptionDispatch:
It's important to note that nt!KiExceptionDispatch (like nt!KiBreakpointTrap) is written in hand-ASM. It assumes that ecx contains the exception code, edx is the number of exception parameters (up to 3), r8 contains the address of the exception, r9 is the first exception parameter (if one exists), r10 is the second exception parameter (if one exists), r11 is the third exception parameter (if one exists), and rbp points to a segment in the _KTRAP_FRAME structure (at offset +0x80).
Upon entry of nt!KiExceptionDispatch, the first thing that occurs is the generation of a _KEXCEPTION_FRAME. Whereas the _KTRAP_FRAME was used to store volatile registers, the _KEXCEPTION_FRAME provides a place to save all nonvolatile registers:
nt!KiExceptionDispatch also creates an _EXCEPTION_RECORD structure on the stack. If you've done any error handling in Windows (in either usermode or kernelmode), you'll be familiar with this data structure as it is contained as a child within the _EXCEPTION_POINTERS data structure. We use both of these structures in our example above.
Furthermore, this explains the first part of our mystery, namely, why the _EXCEPTION_RECORD.ExceptionAddress is incorrect. Recall that the _EXCEPTION_RECORD.ExceptionAddress is populated by the 3rd (r8) argument to nt!KiExceptionDispatch. This was passed in from nt!KiBreakpointTrap. This argument is a copy of the _KTRAP_FRAME.Rip member decremented by one.
To figure out where the _CONTEXT.Rip member is populated, we need to go deeper down the rabbit hole.
nt!KiExceptionDispatch will call into nt!KiDispatchException (yes, the ordering of the words are intentionally flipped) passing in the recently created _EXCEPTION_RECORD and _KEXCEPTION_FRAME:
This function will build a _CONTEXT out of the _KTRAP_FRAME and _KEXCEPTION_FRAME by invoking the helper routine KeContextFromKFrame. After the _CONTEXT is created, a check is made against the _EXCEPTION_RECORD.ExceptionCode (received as an argument from nt!KiExceptionDispatch) for STATUS_BREAKPOINT (0x80000003). If it's true, the _CONTEXT.Rip member will be decremented:
This solves the last part of the mystery and causes the value in _CONTEXT.Rip to be tainted.
The anti-debug trick
Knowing what we know about how Windows handles the different types of int 3s, is it possible to leverage this discrepancy in a useful way? The answer is yes.
Debuggers display the state of the program at the time of an exception. Since Windows will incorrectly assume that our int 3 exception was generated from the single-byte variant, it is possible to confuse the debugger into reading "extra" memory. We leverage this inconsistency to trip a "guard page" of sorts.
As we saw in our first example (at the start of the article), when a multi-byte int 3 occurs, the _EXCEPTION_RECORD.ExceptionAddress and _CONTEXT.Rip values will lie in the middle of our multi-byte instruction instead of at the start. This means that the debugger will incorrectly determine that the instruction which threw the software breakpoint begins with the opcode 0x03. Referring to the trusty Intel manual, we can see that this opcode represents a 2-byte add instruction:
What would happen if we positioned our multi-byte int 3 near the end of a page of memory?
When the operating system notifies our attached debugger of the breakpoint exception, the instruction pointer will point to memory that will be misinterpreted as the start of an add (0x03) instruction. This will cause the debugger to disassemble data on the adjacent page (since this instruction is 2 bytes long), and effectively read one byte past our "valid" memory range.
Our trick relies on the fact that Windows, as an optimization, will not commit virtual memory to physical RAM unless it absolutely needs it. That is to say that most memory, especially in usermode, is paged. When memory needs to be made available for use that is not currently in physical RAM, a page fault occurs. To learn more about memory management, check out the following articles on our site: Introduction to IA-32e hardware paging and Exploring Windows virtual memory management.
So, we can detect the memory read on this adjacent page by inspecting the corresponding PTE (Page Table Entry) using the QueryWorkingSetEx API. If the page is resident in our process' working set (e.g. mapped into our process by the debugger), the Valid bit in the _PSAPI_WORKING_SET_EX_BLOCK will be set.
PoC||GTFO
A full example can be found below:
As always, if you have any questions or comments, please feel free to send us a message below. Happy hacking 😎.
Microsoft has a great track record of maintaining support for legacy software running under Windows. There is an entire compatibility layer baked into the OS that is dedicated to fixing issues with decades old software running on modern iterations of Windows. To learn more about this application compatibility infrastructure, I'd recommend swinging over to Alex Ionescu's blog. He has a great set of posts describing the technical details on how user (even kernel) mode shimming is implemented.
With all of that said, it's an understatement to say that Microsoft takes backwards compatibility seriously. Occasionally, the humans at Microsoft make mistakes. Usually, though, they're very quick to address these problems.
This blog post will go over an unnoticed bug that was introduced in Windows 8 with a documented Win32 API. At the time of this post, this bug is still present in Windows 10 (Creator's Update) and has been around for over 5 years.
Forgotten Win32 APIs
There is a set of Win32 APIs that were introduced in Windows XP to monitor the working set of a process. A process' working set is a collection of pages, chunks of memory, that are currently in RAM (physical memory) and are accessible to that process without inducing a page fault. In particular, the APIs of interest for us are InitializeProcessForWsWatch and GetWsChanges/GetWsChangeEx.
After reading the MSDN documentation, it's easy to discover what the intended use for these APIs were. These APIs profile the number of page faults that occur within a process' address space.
What's a page fault? A quick recap.
There are 3 general categories of page faults.
A hard page fault occurs when memory is accessed that's not currently in RAM (physical). In situations like this, the OS will need to retrieve the memory from disk (e.g. pagefile.sys) and make it accessible to the faulting process.
A soft page fault occurs when memory is in RAM (physical), but not currently accessible to the process that induced the fault. This memory might be shared amongst multiple processes and the process that caused the page fault might not have it mapped into its working set. These types of page faults are much more performant than hard page faults as there is no disk I/O conducted.
The last and final type of page fault is known formally as an invalid fault. These can also be referred to as access violations. This can be caused when a program, for example, tries to access unallocated memory or tries to write to memory that's marked read-only.
Paging is necessary to make modern operating systems work. You probably have many processes running on your system, but not nearly enough RAM to hold all the possible contents of each process into physical memory. To learn more about paging, I strongly recommend this article posted by my colleague.
Demo
The best way to illustrate what's broken is through an example. I created two simple programs.
The first application, WorkingSetWatch.exe, implements the InitializeProcessForWsWatch and GetWsChangeEx APIs. This application logs when a specific memory region is paged into our process' working set:
The second application, ReadProcessMemory.exe, implements reading of an arbitrary memory blob from another target process' memory space:
The basic idea is to use ReadProcessMemory.exe to read from the monitored memory address inside of WorkingSetWatch.exe. This will induce a page fault.
Windows 7: Build 7601 (SP1)
The WorkingSetWatch.exe application works as expected. We're able to read any (valid) sized buffer using ReadProcessMemory.exe and log it.
Windows 10: Build 15063 (Creator's Update)
Unfortunately, WorkingSetWatch.exe does not seem to log the page fault that occurs when our remote application, ReadProcessMemory.exe, reads a buffer greater than or equal to 512 bytes; however, it does seem to work as expected when a read occurs that's less than 512 bytes.
This renders these working set APIs useless for profiling reasons on Windows 8+.
What went wrong?
To determine what went wrong, we'll need to reverse engineer parts of Windows and see exactly how the implementation changed in Windows 8+ from Windows 7.
All disassembly and pseudo-source is reconstructed from system files that are provided with Windows x64 10.0.15063 (Creator's Update).
Enabling process working set logging
To enable working set logging for a process, we need to call InitializeProcessForWsWatch. From the MSDN documentation, we're told that on newer versions of Windows this API is exported as K32InitializeProcessForWsWatch within kernel32.dll. Our analysis begins there:
This function is very simple. It invokes an import from another library. In this case, it executes a function of the same name (K32InitializeProcessForWsWatch), but contained within a different library, api-ms-win-core-psapi-l1-1-0.dll. This library doesn't exist on disk, but rather resolves to an API Set mapping corresponding to kernelbase.dll (which does exist on disk) for this version of Windows. A look into kernelbase.dll's implementation shows that a call to NtSetInformationProcess is performed without any parameter marshalling:
Our next target is NtSetInformationProcess within ntdll.dll:
This is just a simplistic syscall stub that will eventually make its way into the implementation contained within ntoskrnl.exe, the Windows kernel. nt!NtSetInformationProcess is a massive function that contains a huge switch statement that supports all the different PROCESSINFOCLASS that can be passed to it.
We're interested in the PROCESSINFOCLASS for ProcessWorkingSetWatch. This is case 15 (0xF). A snippet of the relevant parts (with the cleaned-up disassembly):
It's interesting to note that you're able to start monitoring on a process' working set with either a class of ProcessWorkingSetWatch (15) or ProcessWorkingSetWatchEx (42). This can be achieved by invoking nt!NtSetInformationProcess directly instead of going through the documented route with kernel32!InitializeProcessForWsWatch. The latter utilizes only the ProcessWorkingSetWatch class.
The actual logic of nt!NtSetInformationProcess is pretty trivial to understand. A blob of memory is allocated per process that we're monitoring. This blob of memory is a _PAGEFAULT_HISTORY structure and contains up to 1024 _PROCESS_WS_WATCH_INFORMATION structures internally. Each _PROCESS_WS_WATCH_INFORMATION structure is an entry that describes a page fault. These entries will be cycled through as the array fills up. Recall from the MSDN documentation (the "Remarks" section) that you must call GetWsChanges/Ex with enough frequency to avoid record loss. This makes perfect sense because we can see that there are a fixed number of these records (1024) allocated. I took the liberty of documenting these structures:
The union at the beginning of the _PAGEFAULT_HISTORY structure may be a little confusing, but it'll be explained later.
On successful execution of this routine, the monitored process object will have an internal member (_EPROCESS.WorkingSetWatch) updated to include this recently allocated _PAGEFAULT_HISTORY pointer. Additionally, the PsWatchEnabled global will be set. This value informs the system to track page faults for processes. It will remain set until the system reboots (even if there are no processes running that have working sets tracked). There are only 2 references to PsWatchEnabled and we've already inspected the one in nt!NtSetInformationProcess.
Our investigation leads us to nt!KiPageFault.
Logging a page fault
When a page fault occurs, the CPU transfers execution to nt!KiPageFault:
If the PsWatchEnabled global is set, that means we've enabled working set logging for processes on the system and execution is passed to nt!PsWatchWorkingSet. This function is documented below:
As I mentioned above, there are 3 types of page faults. Access violations are not logged to our process' working set due to an early out by nt!MmAccessFault in nt!KiPageFault. Since this function is executed for the other 2 types of page faults (hard and soft) on the system, it will be accessed heavily by the operating system. Luckily, one of the first things the routine does is check whether or not a working set watch was enabled on the process where the page fault occurred. If there is no working set watch on the process, the routine completes.
As per the documentation, nt!PsWatchWorkingSet will not function while records are being processed (EntrySelector.Busy). We'll describe this part in depth at a later time. Since higher priority interrupts can preempt our working set monitor, most of the logic in this routine needs to have adequate sanity (safety) checks and complete as atomically (Interlocked*** operations) as possible. The first part of the function will safely select a free index in the _PAGEFAULT_HISTORY.WatchInfo array that it can use for logging purposes. If the array is full (there can be at most 1024 entries), a "miss" is recorded (_PAGEFAULT_HISTORY.MissingRecords) and the routine completes. If everything is successful, a page fault event is recorded in a free slot in the _PAGEFAULT_HISTORY.WatchInfo array. An interesting (and undocumented) feature changes the entry's _PROCESS_WS_WATCH_INFORMATION.FaultingVa least significant bit to 0 if a hard page fault occurred and 1 if a soft page fault occurred.
Ultimately, there doesn't seem to be any apparent bugs with this code. Additionally, this code matches very closely to the Windows 7 version which we know works. Our investigation leads us to the working set watch retrieval functions: GetWsChanges/Ex.
Querying working set logging
For article brevity, I'll give a quick summary of the call-flow of kernel32!GetWsChanges (kernel32!K32GetWsChanges) and kernel32!GetWsChangesEx (kernel32!K32GetWsChangesEx). These functions will call into their kernelbase.dll variants. From there, they will branch into kernelbase!GetWsChangesInternal which will invoke ntdll!NtQueryInformationProcess with the appropriate PROCESSINFOCLASS. In particular, the ProcessWorkingSetWatch class will be used for the GetWsChanges family of functions and ProcessWorkingSetWatchEx will be used for the others. From ntdll!NtQueryInformationProcess, a syscall will be made. This makes it to the implementation of NtQueryInformationProcess within the kernel. A massive switch statement awaits:
The part that interests us resides one level deeper within nt!PspQueryWorkingSetWatch:
There's some input validation (e.g. alignment checks) and a safety check (nt!ExIsRestrictedCaller) to avoid kernel pointer leaks in low integrity processes. After that, the process object is retrieved from the supplied process handle. The operating system checks to see that the _EPROCESS.WorkingSetWatch member is set. Just like the documentation states, at most one query can access a process' working set buffer at a time (EntrySelector.Busy). Additionally, while the buffer is being accessed, logging (by nt!PsWatchWorkingSet in nt!KiPageFault) will produce misses.
As long as there's enough space in the user supplied buffer, the operating system will copy over the entry array to the user supplied buffer. The data will be structured in the appropriate way for the appropriate PROCESSINFOCLASS. The last entry in the user supplied buffer (PSAPI_WS_WATCH_INFORMATION/EX) will be terminated with a FaultingPc member of NULL. Additionally, the number of "misses" will be recorded in the FaultingVa member of the last entry.
Finally, the _PAGEFAULT_HISTORY.WatchInfo array of the _EPROCESS.WorkingSetWatch will be reset after a successful call.
/rant.
The InitializeProcessForWsWatch and GetWsChanges/Ex APIs are surprisingly very finicky. There are many weird restrictions and caveats which make it surprisingly difficult for developers to retrieve information regarding the complete set of page faults that occurred within a process.
There is a very good chance that you will run into situations where records will wind up missing especially in a multi-processor and multi-threaded environment. For example, if a thread is querying the working set of a process, but a page fault occurs on another thread within that same process, a miss could be recorded since the _PAGEFAULT_HISTORY.Busy member will be acquired by nt!PspQueryWorkingSetWatch. This will prevent the page fault logging logic in nt!PsWatchWorkingSet. Functionally, this weakens the usability of the API for profiling purposes. To compound this problem, only 1024 entries can be stored in the array between calls of GetWsChanges/Ex. That's at most 4 MB (1024*PAGE_SIZE) of page fault history. This really isn't enough for modern applications which can be very complex.
In our specific situation, we ran our tests on a VM that had 1 processor allocated to it. Furthermore, our application was simple enough that it had 1 thread. This mitigates the chance of page fault "misses". Additionally, after a thorough investigation of the working set APIs, we've concluded that we've still not discovered where the bug is. In particular, why does the buffer size play a role in the success of these APIs? In our demo, we were unable to log page faults on Windows 10 when the buffer size was greater than or equal to 512 bytes. Is it possible that the bug is not within WorkingSetWatch.exe, but rather ReadProcessMemory.exe?
To continue our investigation, we need to turn to ReadProcessMemory.exe.
Reading memory
The ReadProcessMemory.exe application is simple enough to understand. We know that we're not logging a page fault when we're reading a buffer that is greater than or equal to 512 bytes. Since there is no apparent bug in the working set APIs, the problem most likely resides in kernel32!ReadProcessMemory.
I'll step past the irrelevant details, but the same strategy is applied as was in the previous parts. In particular, kernel32!ReadProcessMemory calls into kernelbase!ReadProcessMemory. These functions do nothing special and more-or-less directly issue a system call by invoking ntdll!NtReadVirtualMemory. This takes us to the implementation of nt!ReadVirtualMemory in the kernel:
This function just invokes nt!MiReadWriteVirtualMemory. On some versions of ntoskrnl, this routine may just be inlined into the caller's body.
Aside from a check that prevents reading and writing to protected processes (ProcessObject->Pcb.SecurePid), this function is nearly identical to the one in the Windows 7 kernel. We need to go deeper. We traverse into nt!MmCopyVirtualMemory.
This function is massive. It contains many subfunctions that have been inlined. For article brevity, the important parts of nt!MmCopyVirtualMemory will be highlighted. One of the first things that this routine does is search for VAD entries that corresponds to the input addresses (FromAddress and ToAddress). The idea is to leverage the "region size" information for memory, but this isn't really relevant to our bug. We'll leave the discussion of the VAD (Virtual Address Descriptor) to another time.
nt!MmCopyVirtualMemory's next task is to determine the input buffer's length. In particular, there are a couple checks against the buffer length and the value 512. This is significant to us because we know the bug only seems to manifest when the buffer size is greater than or equal to 512 bytes.
Basically, it seems that if the buffer is greater than or equal to 512 bytes, nt!MmCopyVirtualMemory will utilize nt!MmProbeAndLockPages and nt!MmMapLockedPagesSpecifyCache followed by a memcpy to clone over memory.
If the buffer is less than 512 bytes, nt!MmCopyVirtualMemory will just leverage memcpy directly by using a buffer on the stack or a buffer allocated in dynamic memory (based on buffer size) via nt!ExAllocatePoolWithTag.
This is probably done for performance reasons. Larger memory copies probably benefit from direct mapping instead of memory pool copying. If we do leverage memory pool copying (buffers that are less than 512 bytes in size), we trigger a page fault and the event is logged by our WorkingSetWatch.exe application. On the other hand, if we leverage a direct mapping to copy memory, we do not trigger a page fault.
One incorrect assumption is to believe that on Windows 7 this optimization did not exist. On the contrary, there is very similar logic inside of the older version of nt!MmCopyVirtualMemory. However, something did change, otherwise we would not have any discrepancies with our WorkingSetWatch program. Our investigation leads us into nt!MmProbeAndLockPages.
The bug: an optimization in nt!MmProbeAndLockPages
The implementation of nt!MmProbeAndLockPages underwent drastic changes between Windows 7 to now. If you looked at these two functions side-by-side, you'd quickly notice that the Windows 7 implementation was in some ways much simpler.
The purpose of nt!MmProbeAndLockPages (per the documentation) is to ensure that the specified virtual pages (in the argument contained within MemoryDescriptorList) are backed by physical memory. Additionally, there is a series of permission checks to ensure that the virtual pages permit the user-specified access rights. In Windows 7, to perform this access check, the routine actually "probed" the memory by directly accessing it. This would induce a page fault in the context of the correct process and therefore we'd be able to log it using our WorkingSetWatch.exe application.
On Windows 10, this process was optimized. Instead of accessing the memory directly, a PTE (Page Table Entry) walk is performed to ensure that the correct permissions exist. This change makes the process more efficient especially since the PTEs are leveraged to lock the memory into physical pages anyway.
OS development isn't easy
One seemingly inconspicuous change can break functionality in an entirely unrelated part of the operating system. In our case, an optimization in the underlying logic of how nt!MmProbeAndLockPages functioned broke backwards compatibility of the working set APIs. This bug seems to be entirely unnoticed, but it unfortunately renders the performance profiling nature of the GetWsChanges/Ex APIs useless.
A potential fix for Microsoft is to simply just throw a page fault for "invalid" pages if the PsWatchEnabled global is set or, more granularly, if a process' _EPROCESS.WorkingSetWatch is set.
This article assumes you've read the first part of the series. In particular, at this point you should have successfully setup VMware's GDB stub and IDA Pro's GDB debugger. You should now be in a connected state and broken into IDA Pro's debugger GUI.
Furthermore, the focus of this post is going to be exclusively on loading kernel symbols for 64-bit editions of Windows (AMD64). Different operating systems (and different architectures of Windows) require slight modifications to the article's logic.
Where's Waldo ntoskrnl?
The end goal
The first and most important thing is to discover where the NT Kernel (ntoskrnl.exe) is loaded in memory since it's not at any fixed (static) address thanks to address space layout randomization (ASLR).
We are then able to force IDA Pro to load symbol data (PDBs) at ntoskrnl's base address to have useful debugging information. From there, we can enumerate the linked list, nt!PsLoadedModuleList, to figure out where other kernel mode components are located. However, this isn't trivial. When you break in to IDA Pro's GDB debugger, it's difficult to know what state you'll be in on any given processor. You might be executing code in a usermode process, or you might be busy servicing a system call. Additionally, you're further restricted to the functionality the GDB stub exposes.
Enter the _KPCR
On all architectures and versions of Windows, each processor maintains a control structure dubbed as the _KPCR (Kernel Processor Control Region). This structure is massive and it can be used to infer exactly what the processor is doing. On Windows 10 (15063.0.amd64fre.rs2_release.170317-1834), the _KPCR is 0x6bc0 bytes large. It contains many kernel pointers that we can leverage to figure out exactly where the base of ntoskrnl is in memory. A link detailing the members of the _KPCR can be found here.
This structure can be accessed through its virtual address or through the fs segment on x86 and the gs segment on x64. In fact, if you've done any reverse engineering of the Windows kernel, you should have seen many examples of Windows itself accessing members of the _KPCR through the segment selector.
For example, when an int 3 (a software breakpoint; 0xCC) is executed by the processor, control is redirected by the CPU to a handler registered in the appropriate position of the IDT (Interrupt Descriptor Table). We'll touch more on this process later. In Windows, the handler for software breakpoints is nt!KiBreakpointTrap. Here is a snippet of the assembly code of the handler under AMD64:
In particular, at address 0x00000001401749FD we see a swapgs instruction. Since the gs selector means different things in user-mode (_TEB) and the kernel (_KPCR), this instruction is utilized to ensure that we're operating on the kernel-mode construct (_KPCR). Immediately following that instruction at address 0x0000000140174A00, we have an access of the gs segment with a mov r10, gs:188h. The astute reader will realize that upon execution of this instruction, r10 will contain the pointer from the _KPCR.Prcb.CurrentThread. This is discerned from the definition of the structure's members posted above. A breakdown of this process can be illustrated below:
We don't know the _KPCR's exact linear address (it too isn't allocated at a fixed location), but we should be able to access it through the segment selector, though, just like the Windows kernel does. This approach might seem like the ideal one, but, unfortunately, we're further restricted by the functionality of the GDB stub. Let's see what the GDB stub exposes by issuing help:
There are only three major commands available: help, r, and linuxoffsets. We've just executed help, and linuxoffsets isn't relevant to us since we're debugging a Windows kernel. The only other command is r. At first, r looks very useful to us. However, on closer examination, we can see that the GDB stub is unable to read arbitrary offsets off of the gs selector, e.g. the _KPCR.Prcb.CurrentThread from gs:188h by executing r gs:188h.
At least executing r gs without an offset produces data:
This command should get us the base of the gs selector. We then should be able to define a _KPCR structure at that location using IDA Pro. According to the GDB stub, though, our base is 0. If we go to that memory location in the "IDA View - RIP" tab by pressing 'G' and entering 0 in the "Jump to address" window, we don't see anything there:
What changed from x86 to x64?
If you ran this test on a VM running on an x86 (32-bit) version of Windows and substituted fs for gs, the base of the fs selector would not be 0. It would be a valid memory location. You would then have the address of the _KPCR and could continue on your merry way.
Unfortunately, you're a sucker for pain and are following this tutorial down to the T. In 64-bit (long) mode on x64, the cs, ss, ds, and es segment selectors have a zero-forced base address. gs and fs are the exceptions and have a non-zero base address. So, how is it possible that the base of the gs selector is 0 when Windows itself uses the segment selector to retrieve processor state?
The answer is in the model-specific registers, MSRs. MSRs are per-processor registers that can be read via rdmsr and written via wrmsr instructions. On x64, the IA32_GS_BASE (0xC0000101) and IA32_KERNEL_GS_BASE (0xC0000102) MSRs are used for storage of the base address of the gs selector. swapgs was introduced to exchange the address of the current gs base register with the value contained in the IA32_KERNEL_GS_BASE MSR.
This means that we could, theoretically, read the IA32_GS_BASE MSR if we're executing code in CPL0 (ring0/kernel-mode). This would get us the base address of the gs segment. However, that's not directly possible through the VMware GDB stub. There is no support for reading or writing to MSRs directly.
A shimmer in the shadows
Nevertheless, through persistence, we come up with an approach that plays nicely given our constraints. There are multiple ways to skin a cat and this approach may not be the most elegant solution, but it should work nicely for all x64 Windows kernels.
The basic idea is to leverage the IDT, the interrupt descriptor table, to find a symbol that's in the address space of ntoskrnl. We can access the idtr, a register that houses the IDT, through the GDB stub:
Once we have the base of the IDT, in our case 0xfffff802c4850000, we can access the first entry of the IDT. This should resolve to a symbol within ntoskrnl (nt!KiDivideErrorFault):
From there, we can walk kernel memory backwards until we get to a valid PE header. Since the symbol is contained within ntoskrnl's address space, the first valid PE header should belong to ntoskrnl:
Figure 1: Layout of kernel memory.
Writing an IDA script using IDAPython
It'd be nice to programmatically implement the algorithm described above so we don't need to manually go through it each time we're trying to discover the base address of ntoskrnl. We'll do this by writing a script for IDA Pro to run. I chose to do this with IDAPython instead of IDC (IDA's C-like bindings) because of the niceties that Python provides (like string manipulation).
The basics
We'll start by switching the input from "GDB" to "Python" in the "Output window". If your "Output window" is missing, you can restore it by selecting "Windows" and then "Output window" from the menu bar:
We can see all the functionality exposed by IDAPython by executing the Python command dir() in the text box. If you try to do this, you'll see lots of output. It's easy to feel overwhelmed. Luckily, there exists amble documentation on the Hex-Rays website that can help us navigate these murky waters.
I try to find useful things by searching for it first in the dir() listing. You can position your cursor in the "Output window" and press Alt+T to search for a keyword. To find the next occurrence, you can hit Ctrl+T. If this fails, I move on to the documentation.
Sending a command to the GDB stub
Our first task is to figure out how to send a command to the GDB stub. If you search for the "command" keyword in the "Output window" you'll find something labeled "SendDbgCommand". Let's see what this function does by executing help(SendDbgCommand):
It seems very relevant to us. Let's give it a try:
Looks like it's working. This is the same output we received from the GDB stub when we issued the help command.
Parsing the response from the GDB stub
Now that we know how to send a command to the GDB stub, we need to issue a command to retrieve the contents of the idtr. We then parse and extract the base address from the resulting string.
It's important to tell Python that we're working with an integer object by "casting" the string to an integer-type:
Easy!
Getting the first IDT entry's handler
We have the base of the IDT in idt_base. Our next task is to retrieve the first entry in the IDT. The IDT is effectively an array that contains 256 IDT entries (0-0xFF) on x64. The format of the IDT is dictated by the architecture of the processor (e.g. Intel x64). Each IDT entry on x64 takes the following form:
To get to the handler (e.g. where the processor moves control to when an interrupt occurs), the target address is built from the OffsetHigh, OffsetMiddle, and OffsetLow fields of this structure using the following algorithm: HandlerAddress = ((OffsetHigh << 32) + (OffsetMiddle << 16) + OffsetLow).
We'll leverage the Dbg* commands to read virtual memory from IDAPython. Since we're extracting the first IDT entry, we can just read directly from the start of our idt_base:
This shows us that the handler for the first IDT entry (nt!KiDivideErrorFault) is loaded at 0xfffff802c27f4300. If we wanted to read the N'th IDT entry, we'd have to index into the array by adding 0x10, the size of a _KIDTENTRY64, times the location in the array (in this case N). So, to index into the 3rd IDT entry, we'd apply the following math: idt_entry = idt_base + (0x10 * 2).
Finding the base address from a symbol within ntoskrnl
First, we'll define a simple helper function that will align addresses to their page boundaries. This will help speed up our lookup because we know that the base address of ntoskrnl will be on a page boundary:
We'll then create a very simple loop to walk memory backwards (on a page-aligned boundary) searching for the magical value 0x5A4D, commonly known as 'MZ' (IMAGE_DOS_SIGNATURE). This value signifies the start of the IMAGE_DOS_HEADER which is also the base address of an image:
Voila! The base address of ntoskrnl is discovered at 0xfffff802c2680000.
Creating the final version of the script
After some refactoring and code tidying (including error checking), we produce a much better version of the script. This does the same thing as the commands we inserted in the IDAPython "Output window":
Save a copy of the script to your local drive. We are then able to run it at any time by going to "File" and then "Script file..." in the IDA Pro GUI. A sample of the output is listed below:
The important line appears on the bottom; the base address of ntoskrnl is displayed. It checks out with the work we did by hand too.
Loading ntoskrnl at its base address
We mustn't forget the final objective: loading kernel symbols. We're almost at the finish line. Let's tell IDA to load ntoskrnl at the base address our script found.
First, we'll need to grab a copy of ntoskrnl on the VM. Don't use the version on your host as this may not match with what's on the VM. This'll be found in your guest's system directory:
You might need to resume your VM if you're currently active in IDA's GDB debugger by selecting "Debugger" and then "Continue process" (or by hitting F9) from the menu bar.
After you've pulled ntoskrnl from your VM, break into IDA's GDB debugger by selecting "Suspend". Now, we must load it by selecting "File" then "Load file" and finally "PDB file..."
Find where you copied ntoskrnl to on your host and use the address that the script found:
It'll take IDA at least a couple of minutes to fully finish the loading process. You can see IDA's progress in the bottom left corner:
You'll know IDA's finished when the status changes to "AU: idle".
Quick validation
We should make sure that the symbols are loaded correctly. Navigate to "Jump" and then "Jump to address" (or press "G"). Enter PsLoadedModuleList (case sensitive) and hit "OK".
From there, double click the address immediately to the right of the PsLoadedModuleList symbol. This takes you to the first entry in the list.
Each entry in this list is of type _LDR_DATA_TABLE_ENTRY. You might be familiar with this structure from usermode programming. It's also used in the kernel.
We'll need to add the definition of the _LDR_DATA_TABLE_ENTRY to IDA's structures. Luckily, we have symbols loaded and this is a pretty straightforward process.
After the structure was added, you'll see a window similar to this.
Go back to the "Debug View". Impose the _LDR_DATA_TABLE_ENTRY structure on that memory region:
Let's follow the FullName.Buffer field:
And now let's convert this to a readable string:
You should see the characters \SystemRoot\system32\ntoskrnl.exe. We did it!
Final thoughts
Now that symbols are loaded for ntoskrnl, it would be wise to iterate through the PsLoadedModuleList and load symbols for all the other kernel mode components. This can be scripted using IDAPython too, however, it's beyond the scope of this article.
Sometimes you'll run into a situation that you can't analyze with a traditional kernel debugger like WinDbg. An example of such is trying to troubleshoot the runtime logic of PatchGuard (Microsoft's Kernel Patch Protection). In situations like this, you need to bust out the heavy tools. VMware has built in support for remote debugging of virtual machines running inside it through a GDB stub. IDA Pro, the defacto disassembler that most reverse engineers have, includes a GDB debugger. Together these make for a very powerful combo.
This article goes over how to setup VMware's GDB stub and how to connect to it using IDA Pro's GDB debugger.
Requirements
A copy of VMware Workstation (free 30-day trial). I'll be using VMware Workstation 12.5.7 (build-5813279).
Unfortunately, VMware Player (entirely free for non-commercial use) does not expose the GDB stub interface.
You can use either the Linux or Windows build of VMware. I'll be using the 64-bit Windows build.
A Windows operating system installed on your host and guest (VM). These do not have to be the same versions of Windows. My host and guest OS are both running Windows x64 10.0.15063 (Version 1703).
This can be any OS supported by VMware such as Ubuntu.
The second part of this tutorial (loading kernel symbols) assumes you're running a Windows 64-bit VM (AMD64).
Enabling the GDB stub within VMware
Select the VM you wish to enable GDB stub debugging on within VMware.
VMs should be listed in the "Library" pane on the left of the GUI. If the "Library" pane is missing, you can restore it by selecting "View" then "Customize" and choosing "Library" (or hit F9).
Ensure that the VM is currently not running. If it's currently active, power it off via the menu bar: "VM" then "Power" then "Shut Down Guest" (or Ctrl+E).
Select "Edit virtual machine settings". Ensure that you are on the "Options" tab.
Find the "Working directory" text field and copy the string to your clipboard. 'Cancel' out of the prompt.
Go to the working directory.
Right-click on the *.vmx file and "Open with" your favorite text editor. I'll be using Notepad++.
Add one of the following lines to the end of the file, based on preference.
If your VM is 32-bit and you want to debug locally: debugStub.listen.guest32 = "TRUE"
If your VM is 64-bit and you want to debug locally: debugStub.listen.guest64 = "TRUE"
If your VM is 32-bit and you want to debug remotely: debugStub.listen.guest32.remote = "TRUE"
If your VM is 64-bit and you want to debug remotely: debugStub.listen.guest64.remote = "TRUE"
The default port for the GDB stub is 8864 for 64-bit guests and 8832 for 32-bit guests. If you'd like to change what port the VMware GDB stub listens on (e.g. 55555), add one of the following lines to the file:
If your VM is 32-bit: debugStub.port.guest32 = "55555"
If your VM is 64-bit: debugStub.port.guest64 = "55555"
If you want to start debugging immediately on BIOS load add one of the following lines to your file:
If your VM is 32-bit: monitor.debugOnStartGuest32 = "TRUE"
If your VM is 64-bit: monitor.debugOnStartGuest64 = "TRUE"
To make it difficult to detect breakpoints that you've set using GDB, it's strongly recommended to add the following option too:
debugStub.hideBreakpoints = "TRUE"
An important thing to note is that this option is restricted by the number of hardware breakpoints available to the processor (usually 4).
Save the *.vmx file via "File" and then "Save" from the menu bar (or hit Control+S). Here's a copy of the contents of my *.vmx file:
Close the file.
Run the VM corresponding to the *.vmx file you just edited. Validate that the GDB stub is currently running by opening the vmware.log file in the same directory as the *.vmx file:
If you see a message from "Debug stub" that tells you VMware is "listening" for a debug connection on a certain port number, you're in a good state.
If you are missing that log line or have an error, ensure that your *.vmx file has the appropriate settings. Remember: you must edit the *.vmx file when the Virtual Machine is off or your changes may be lost.
Configuring the GDB debugger within IDA Pro
Launch the 64-bit version of IDA Pro if you're debugging a 64-bit VM and the 32-bit version of IDA Pro if you're debugging a 32-bit VM.
Skip the "Welcome" dialog (by hitting "Go") and go to the main disassembler window. Choose "Debugger" and then "Attach" and finally "Remote GDB debugger" from the menu bar.
Enter the appropriate "Hostname" and "Port". These were set up by you during steps 7 and 8 of the "Enabling the GDB stub within VMware" section. Furthermore, these can be validated in the vmware.log file (this was done in step 12 of the same section). Then hit "Debug options".
In the "Debugger setup" window, select "Set specific options".
Ensure that the right "Processor" is set in the drop down box. If you're debugging a 64-bit edition of Windows (AMD64), select "Intel x64". If you're debugging a 32-bit edition of Windows (X86), select "Intel x86".
Select 'OK' in the "GDB configuration" window. And then select 'OK' in the "Debugger setup" window. Finally, select 'OK' in the "Debug application setup" window.
The VM will become suspended and a green "play" button will appear. At this point, IDA should bring up a window with the title "Choose process to attach to".
Select "<attach to the process started on target>" and hit 'OK'.
If you see this window, that means you're almost done.
Select "Debugger" and then "Manual memory regions" from the menu bar.
Inside of the "Manual memory regions" tab, right click and select "Insert" (or just press "Insert" on your keyboard).
A new window will pop up. Enter in the "Start address" as 0 and the "End address" as -2. Make sure the right "segment" is selected (e.g. 64-bit segment for 64-bit VM debugging) and hit 'OK'.
This essentially maps all virtual memory from 0 to 0xFFFFFFFFFFFFFFFE (on x64).
"-1" is not an acceptable boundary for IDA Pro as the "End address".
Find the "General Registers" window and find the IP register (RIP on x64, EIP on x86).
If the "General Registers" window is gone, select "Debugger" and then "Debugger windows" and finally "General Registers" from the menu bar.
Right click on the IP register and select "Jump" from the context menu that appears.
Your memory view will become synched to the IP register. If there are raw bytes listed and not code, don't panic. Place your cursor on the address of the IP and hit "C".
Congratulations. At this point you've successfully set up VMware's GDB stub and IDA Pro's GDB debugger. You are now able to debug the VM and apply breakpoints through the IDA Pro GUI just as you would normally through a kernel debugger. Most of the functionality of the GDB debugger can be accessed through the "Debugger" menu bar.
This type of debugging is transparent to the kernel and therefore "debugger" checks like "KdDebuggerEnabled" and "KdDebuggerNotPresent" will not trigger. Furthermore, if the debugStub.hideBreakpoints option was enabled, breakpoints (up until the hardware maximum) will not make any inline code edits!
Final thoughts
Ultimately, the GDB debugger is not very useful without kernel symbols being loaded. One option, albeit a naive one, is to attach WinDbg as a kernel debugger while running IDA's GDB stub in the background. A tutorial on how to setup kernel debugging using WinDbg and VMware can be found here. You are then able to use the symbolic data that is provided from WinDbg to power debugging in IDA's GDB debugger. This is very cumbersome and has many disadvantages such as not being able to avoid kernel debugger checks.
Luckily, there is a better way. In the second part of this series, we'll discover how to load kernel symbols in IDA Pro's GDB debugger.
Login
