🔒
There are new articles available, click to refresh the page.
Before yesterdayThreat Research

Mandiant Discloses Critical Vulnerability Affecting Millions of IoT Devices

17 August 2021 at 12:00

Today, Mandiant disclosed a critical risk vulnerability in coordination with the Cybersecurity and Infrastructure Security Agency (“CISA”) that affects millions of IoT devices that use the ThroughTek “Kalay” network. This vulnerability, discovered by researchers on Mandiant’s Red Team in late 2020, would enable adversaries to remotely compromise victim IoT devices, resulting in the ability to listen to live audio, watch real time video data, and compromise device credentials for further attacks based on exposed device functionality. These further attacks could include actions that would allow an adversary to remotely control affected devices.

At the time of writing this blog post, ThroughTek advertises having more than 83 million active devices and over 1.1 billion monthly connections on their platform. ThroughTek’s clients include IoT camera manufacturers, smart baby monitors, and Digital Video Recorder (“DVR”) products. Unlike the vulnerability published by researchers from Nozomi Networks in May 2021 (also in coordination with CISA), this latest vulnerability allows attackers to communicate with devices remotely. As a result, further attacks could include actions that would allow an adversary to remotely control affected devices and could potentially lead to remote code execution.

The Kalay protocol is implemented as a Software Development Kit (“SDK”) which is built into client software (e.g. a mobile or desktop application) and networked IoT devices, such as smart cameras. Due to how the Kalay protocol is integrated by original equipment manufacturers (“OEMs”) and resellers before devices reach consumers, Mandiant is unable to determine a complete list of products and companies affected by the discovered vulnerability.

This vulnerability has been assigned a CVSS3.1 base score of 9.6 and is tracked as CVE-2021-28372 and FEYE-2021-0020. This blog post discusses the Kalay network and CVE-2021-28372 at a high level. It also includes recommendations from ThroughTek and Mandiant, along with mitigation options.

Mandiant would like to thank both CISA and ThroughTek for their coordination and support in releasing this advisory.

FAQ

What devices are affected, and (potentially) how many devices are affected?

The vulnerabilities described in this post affect a core component of the Kalay platform. Mandiant was not able to create a comprehensive list of affected devices; however, ThroughTek’s website reports more than 83 million active devices on the Kalay platform at the time of writing this post.

How is the issue being addressed?

Mandiant worked with ThroughTek and CISA to disclose this vulnerability and would strongly recommend companies using the Kalay platform follow the guidance provided by ThroughTek and Mandiant:

  • If the implemented SDK is below version 3.1.10, upgrade the library to version 3.3.1.0 or version 3.4.2.0 and enable the Authkey and Datagram Transport Layer Security (“DTLS”) features provided by the Kalay platform.
  • If the implemented SDK is version 3.1.10 and above, enable Authkey and DTLS.
  • Review security controls in place on APIs or other services that return Kalay unique identifiers (“UIDs”).

How would an attacker exploit these vulnerabilities?

An attacker would require comprehensive knowledge of the Kalay protocol and the ability to generate and send messages. The attacker would also need to obtain Kalay UIDs through social engineering or other vulnerabilities in APIs or services that return Kalay UIDs. From there, an attacker would be able to remotely compromise affected devices that correspond to the obtained UIDs.

How do I know if a device I own is affected? How do I protect myself if I own an affected device?

Mandiant was not able to create a comprehensive list of devices using the Kalay platform, but strongly encourages users of IoT devices to keep device software and applications up to date and use complex, unique passwords for any accounts associated with these devices.

Device owners should avoid connecting to affected devices from untrusted networks such as public wireless networks.

Who discovered this vulnerability?

Jake Valletta, Erik Barzdukas, and Dillon Franke.

CVE-2021-28372: Device Impersonation

Mandiant researchers analyzed ThroughTek’s Kalay protocol using two different approaches. First, the researchers selectively downloaded and disassembled applications from both the Google Play Store and Apple App Store that included ThroughTek libraries. These libraries typically did not contain debugging symbols, which required the team to also perform dynamic analysis with tools such as Frida, gdb, and Wireshark.

In addition, Mandiant purchased various Kalay-enabled devices. The team performed local and hardware-based attacks to obtain shell access, recover firmware images, and perform additional dynamic testing. These techniques included identifying UART/JTAG interfaces, performing chip-off attacks, and exploiting other debugging functionality present on the devices.

Over the course of several months, the researchers developed a fully functional implementation of ThroughTek’s Kalay protocol, which enabled the team to perform key actions on the network, including device discovery, device registration, remote client connections, authentication, and most importantly, process audio and video (“AV”) data. Equally as important as processing AV data, the Kalay protocol also implements remote procedure call (“RPC”) functionality. This varies from device to device but typically is used for device telemetry, firmware updates, and device control.

Having written a flexible interface for creating and manipulating Kalay requests and responses, Mandiant researchers focused on identifying logic and flow vulnerabilities in the Kalay protocol. The vulnerability discussed in this post affects how Kalay-enabled devices access and join the Kalay network. The researchers determined that the device registration process requires only the device’s 20-byte uniquely assigned identifier (called a “UID” here) to access the network. In Mandiant’s testing, this UID was typically provided to a Kalay-enabled client (such as a mobile application) from a web API hosted by the company that markets and sells a device model. Mandiant investigated the viability of brute forcing ThroughTek UIDs and found it to be infeasible due to the necessary time and resources.

Figure 1 shows a typical device registration and client connection process on the Kalay network. The example scenario is a user remotely accessing their camera feed on a mobile application from a remote network, (e.g. a coffee shop or mobile phone network) with their Kalay-enabled camera located on their home network.


Figure 1: Normal device registration and connection process

If an attacker obtains a UID of a victim Kalay device, they can maliciously register a device with the same UID on the network and cause the Kalay servers to overwrite the existing Kalay device. Once an attacker has maliciously registered a UID, any client connection attempts to access the victim UID will be directed to the attacker. The attacker can then continue the connection process and obtain the authentication materials (a username and password) needed to access the device. Figure 2 shows a client connection attempt when both a victim device and a malicious device with the same UID exist simultaneously on the network. In this example, the malicious registration is overwriting the existing registration on the network, causing client connections to be mistakenly routed to the malicious device.


Figure 2: Attacker exploiting device personation vulnerability to capture credentials

With the compromised credentials, an attacker can use the Kalay network to remotely connect to the original device, access AV data, and execute RPC calls. Vulnerabilities in the device-implemented RPC interface can lead to fully remote and complete device compromise. Mandiant observed that the binaries on IoT devices processing Kalay data typically ran as the privileged user root and lacked common binary protections such as Address Space Layout Randomization (“ASLR”), Platform Independent Execution (“PIE”), stack canaries, and NX bits.

Figure 3 shows a hypothetical attack utilizing the captured Kalay username and password to stage a further attack by abusing vulnerabilities in the Kalay RPC interface.


Figure 3: Attacker using captured credentials to fetch audio/video data

The following video (Figure 4) demonstrates a functional proof of concept for CVE-2021-28372. Note that Mandiant is not releasing any public exploit code.

Mandiant Discloses Critical Vulnerability Affecting Millions of IoT Devices

17 August 2021 at 12:00

Today, Mandiant disclosed a critical risk vulnerability in coordination with the Cybersecurity and Infrastructure Security Agency (“CISA”) that affects millions of IoT devices that use the ThroughTek “Kalay” network. This vulnerability, discovered by researchers on Mandiant’s Red Team in late 2020, would enable adversaries to remotely compromise victim IoT devices, resulting in the ability to listen to live audio, watch real time video data, and compromise device credentials for further attacks based on exposed device functionality. These further attacks could include actions that would allow an adversary to remotely control affected devices.

At the time of writing this blog post, ThroughTek advertises having more than 83 million active devices and over 1.1 billion monthly connections on their platform. ThroughTek’s clients include IoT camera manufacturers, smart baby monitors, and Digital Video Recorder (“DVR”) products. Unlike the vulnerability published by researchers from Nozomi Networks in May 2021 (also in coordination with CISA), this latest vulnerability allows attackers to communicate with devices remotely. As a result, further attacks could include actions that would allow an adversary to remotely control affected devices and could potentially lead to remote code execution.

The Kalay protocol is implemented as a Software Development Kit (“SDK”) which is built into client software (e.g. a mobile or desktop application) and networked IoT devices, such as smart cameras. Due to how the Kalay protocol is integrated by original equipment manufacturers (“OEMs”) and resellers before devices reach consumers, Mandiant is unable to determine a complete list of products and companies affected by the discovered vulnerability.

This vulnerability has been assigned a CVSS3.1 base score of 9.6 and is tracked as CVE-2021-28372 and FEYE-2021-0020. This blog post discusses the Kalay network and CVE-2021-28372 at a high level. It also includes recommendations from ThroughTek and Mandiant, along with mitigation options.

Mandiant would like to thank both CISA and ThroughTek for their coordination and support in releasing this advisory.

FAQ

What devices are affected, and (potentially) how many devices are affected?

The vulnerabilities described in this post affect a core component of the Kalay platform. Mandiant was not able to create a comprehensive list of affected devices; however, ThroughTek’s website reports more than 83 million active devices on the Kalay platform at the time of writing this post.

How is the issue being addressed?

Mandiant worked with ThroughTek and CISA to disclose this vulnerability and would strongly recommend companies using the Kalay platform follow the guidance provided by ThroughTek and Mandiant:

  • If the implemented SDK is below version 3.1.10, upgrade the library to version 3.3.1.0 or version 3.4.2.0 and enable the Authkey and Datagram Transport Layer Security (“DTLS”) features provided by the Kalay platform.
  • If the implemented SDK is version 3.1.10 and above, enable Authkey and DTLS.
  • Review security controls in place on APIs or other services that return Kalay unique identifiers (“UIDs”).

How would an attacker exploit these vulnerabilities?

An attacker would require comprehensive knowledge of the Kalay protocol and the ability to generate and send messages. The attacker would also need to obtain Kalay UIDs through social engineering or other vulnerabilities in APIs or services that return Kalay UIDs. From there, an attacker would be able to remotely compromise affected devices that correspond to the obtained UIDs.

How do I know if a device I own is affected? How do I protect myself if I own an affected device?

Mandiant was not able to create a comprehensive list of devices using the Kalay platform, but strongly encourages users of IoT devices to keep device software and applications up to date and use complex, unique passwords for any accounts associated with these devices.

Device owners should avoid connecting to affected devices from untrusted networks such as public wireless networks.

Who discovered this vulnerability?

Jake Valletta, Erik Barzdukas, and Dillon Franke.

CVE-2021-28372: Device Impersonation

Mandiant researchers analyzed ThroughTek’s Kalay protocol using two different approaches. First, the researchers selectively downloaded and disassembled applications from both the Google Play Store and Apple App Store that included ThroughTek libraries. These libraries typically did not contain debugging symbols, which required the team to also perform dynamic analysis with tools such as Frida, gdb, and Wireshark.

In addition, Mandiant purchased various Kalay-enabled devices. The team performed local and hardware-based attacks to obtain shell access, recover firmware images, and perform additional dynamic testing. These techniques included identifying UART/JTAG interfaces, performing chip-off attacks, and exploiting other debugging functionality present on the devices.

Over the course of several months, the researchers developed a fully functional implementation of ThroughTek’s Kalay protocol, which enabled the team to perform key actions on the network, including device discovery, device registration, remote client connections, authentication, and most importantly, process audio and video (“AV”) data. Equally as important as processing AV data, the Kalay protocol also implements remote procedure call (“RPC”) functionality. This varies from device to device but typically is used for device telemetry, firmware updates, and device control.

Having written a flexible interface for creating and manipulating Kalay requests and responses, Mandiant researchers focused on identifying logic and flow vulnerabilities in the Kalay protocol. The vulnerability discussed in this post affects how Kalay-enabled devices access and join the Kalay network. The researchers determined that the device registration process requires only the device’s 20-byte uniquely assigned identifier (called a “UID” here) to access the network. In Mandiant’s testing, this UID was typically provided to a Kalay-enabled client (such as a mobile application) from a web API hosted by the company that markets and sells a device model. Mandiant investigated the viability of brute forcing ThroughTek UIDs and found it to be infeasible due to the necessary time and resources.

Figure 1 shows a typical device registration and client connection process on the Kalay network. The example scenario is a user remotely accessing their camera feed on a mobile application from a remote network, (e.g. a coffee shop or mobile phone network) with their Kalay-enabled camera located on their home network.


Figure 1: Normal device registration and connection process

If an attacker obtains a UID of a victim Kalay device, they can maliciously register a device with the same UID on the network and cause the Kalay servers to overwrite the existing Kalay device. Once an attacker has maliciously registered a UID, any client connection attempts to access the victim UID will be directed to the attacker. The attacker can then continue the connection process and obtain the authentication materials (a username and password) needed to access the device. Figure 2 shows a client connection attempt when both a victim device and a malicious device with the same UID exist simultaneously on the network. In this example, the malicious registration is overwriting the existing registration on the network, causing client connections to be mistakenly routed to the malicious device.


Figure 2: Attacker exploiting device personation vulnerability to capture credentials

With the compromised credentials, an attacker can use the Kalay network to remotely connect to the original device, access AV data, and execute RPC calls. Vulnerabilities in the device-implemented RPC interface can lead to fully remote and complete device compromise. Mandiant observed that the binaries on IoT devices processing Kalay data typically ran as the privileged user root and lacked common binary protections such as Address Space Layout Randomization (“ASLR”), Platform Independent Execution (“PIE”), stack canaries, and NX bits.

Figure 3 shows a hypothetical attack utilizing the captured Kalay username and password to stage a further attack by abusing vulnerabilities in the Kalay RPC interface.


Figure 3: Attacker using captured credentials to fetch audio/video data

The following video (Figure 4) demonstrates a functional proof of concept for CVE-2021-28372. Note that Mandiant is not releasing any public exploit code.

Shining a Light on SolarCity: Practical Exploitation of the X2e IoT Device (Part Two)

17 February 2021 at 13:00

In this post, we continue our analysis of the SolarCity ConnectPort X2e Zigbee device (referred to throughout as X2e device). In Part One, we discussed the X2e at a high level, performed initial network-based attacks, then discussed the hardware techniques used to gain a remote shell on the X2e device as a non-privileged system user. In this segment, we’ll cover how we obtained a privileged shell on the device locally using power glitching attacks, and explore CVE-2020-12878, a vulnerability we discovered that permitted remote privilege escalation to the root user. Combined with CVE-2020-9306 (discussed in Part One), this would result in a complete remote compromise of the X2e device.

Technical Analysis

Recap

Before we dive into next steps, let’s recap where we left off:

  • The X2e has an exposed universal asynchronous transmit/receive (UART) interface, which allows a physically connected user to view (but not interrupt) the Das U-Boot (U-Boot) boot process, and given proper credentials, authenticate to the Linux operating system. Since we do not have root credentials, we put this thread on the backburner.
  • We have a full NAND dump of the Spansion raw flash, which includes boot configuration, bootloader firmware, filesystems, and the Linux kernel image. This was used previously in Part One to obtain the hardcoded credential for the python user.

Knowing that UART is present and access to the bootloader would be extremely valuable, we decided to revisit that thread.

Gaining Privileged Access Locally

Revisiting the Bootloader

Figure 1 shows the U-Boot boot process displayed while connected via UART connection. In some cases, it is possible to send keyboard input to the device during a set period (usually one to four seconds) when the bootloader presents the message, “Hit any key to stop autoboot,” which interrupts the boot process and drops the user into a U-Boot shell. On the X2e, this feature has been disabled by setting the U-Boot configuration parameter CONFIG_BOOTDELAY to 0.


Figure 1: Uninterruptable U-Boot bootloader output

One attack that has been documented to be successful to disrupt autoboot is to manipulate the bootloader’s ability to access the flash storage during the boot process. In certain circumstances where the U-Boot bootloader is unable to access its own configuration, it fails into a default environment, which may be less restricted. We decided to see if this would be possible on the X2e.

These attacks, known as glitch attacks (or more officially known as fault-injection), are a type of side channel attack that attempts to cause a microcontroller unit (MCU) to skip instructions, perform wrong instructions, or fail to access flash memory. Various types of glitching attacks exist including electrical, thermal, and radiation. Based on our objective, we opted to try glitching the power between the MCU and the Spansion NAND flash. Note that glitch attacks can often cause damage to the components on a board or put the device in an unusable state. These types of attacks should be tested as either a last resort or against a secondary device you are comfortable with damaging.

Glitching the Bootloader

Based on previous research in this domain, we opted to target the data lines (I/O) between the MCU and NAND flash. Recall from Part One that the NAND flash on the X2e was the Spansion S34ML01G1, which was a 63-pin ball grid array (BGA) package. This chip is capable of supporting both 8-bit and 16-bit bus width, which corresponds to the number of I/O lines utilized. By using the datasheet for the flash and then querying the ONFI Device ID of our chip, we determined our chip was utilizing the 8-bit configuration, meaning eight I/O lines were present between the NAND flash and the MCU. For this attack, we focused on manipulating the power on the first (I/O0) data line. Figure 2 shows the configuration of the BGA-63 pins, with I/O0 highlighted.


Figure 2: Identifying I/O0 for NAND chip in the Spansion datasheet

Because the pins are actually underneath the flash package, we needed to find an exposed lead that corresponded to I/O0 elsewhere on the PCB. One such method for tracing connections across a PCB is a continuity test. A continuity test (using a multimeter) sends a low current electrical signal across two points and produces an audible beep if the points are connected. Using this technique, we located an exposed test point (known as a via) on the bottom of the PCB. Figure 3 shows the I/O0 pin on the top of the PCB (under the NAND chip), and Figure 4 shows the I/O0 pin exposed on the bottom of the PCB.


Figure 3: I/O0 on top of PCB (under NAND chip)


Figure 4: I/O0 on bottom of PCB

With exposed access to I/O0 located, we experimented with connecting this pin directly to a known ground (GND) pin at various points during the boot process. Figure 5 shows the device powering on with the metal tweezers connecting I/O0 to GND.


Figure 5: Shorting I/O0 to GND

While connected to the UART interface, we noted several different outcomes. When shorting the pin immediately after powering on, the device failed to produce any output or boot. When shorting after the bootloader finished loading (and handing off to the Linux kernel), the device would also force reboot. However, when timed perfectly between the bootloader loading and attempting to read its configuration, we noted that the bootloader would present different output, and the option to interrupt the boot process was possible with a four-second delay. By pressing keyboard input, we were successfully able to drop into a U-Boot shell, which is shown in Figure 6.


Figure 6: Access to U-Boot bootloader shell

While this was great progress, we noted that the current failback bootloader configuration was completely inoperable and certain NAND blocks had been marked as bad (as expected). To get our device back to a working state, we needed to revisit the NAND dump we generated in Part One.

Repairing the Bootloader Configuration

While the current configuration provided us a working shell, we needed to fix the damage we had done. This was performed in two steps: fixing the mistakenly marked bad blocks and then rebuilding the configuration. In our case, the nand utility and its sub-commands read, write, and scrub allowed us to inspect and manipulate pages and blocks of the NAND. The nand scrub command with a valid offset and size could be used to completely reset a segment of the NAND, which removed any bad block markers. The next challenge was determining what needed to be replaced in the damaged blocks and rebuilding the configuration.

Since we had a valid NAND image, we revisited the sections read by the bootloader to determine what changes were needed. The format did not match a known format, so we wrote a simple parser in Python to read the binary structure, shown in Figure 7.


Figure 7: Parsing bootloader nvram configuration from flash

With details of how the configuration should look, we used the nand write to rebuild this section, byte by byte with the correct details. We also set the boot delay to be four seconds, so that we could always interrupt the bootloader once the new configuration was committed. Once we confirmed our changes were stable, we saved the configuration to flash and could access the bootloader without performing the aforementioned glitch attack.

Accessing Linux as root User

Now that we have unrestricted access to the bootloader, we can finally influence the rest of the boot process and achieve a privileged shell. We alluded to this in Part One, but the easiest way to turn an unlocked U-Boot shell into a root Linux shell is to adjust the boot arguments that U-Boot passes to the Linux kernel. In our case, this was accomplished by using the setenv utility to change the std_bootarg environment variable to be init=/bin/sh and instructing U-Boot to resume the standard boot process. Figure 8 shows the Linux shell presented over UART.


Figure 8: root shell after bootloader

At this point, we’ve demonstrated a repeatable method for achieving local privilege escalation. In the final segment, we’ll complete our attack by exploring an avenue to remotely escalate privileges.

Gaining Privileged Access Remotely

Since the X2e has only two available listening network services, it makes sense to reinvestigate these services. During Part One, we identified hardcoded credentials for the limited user python. This was useful for initial probing of the device while it was running, but where do we go from here?

Embedded devices typically only have a handful of users, with a majority of functionality being performed by the root user. This presents an interesting opportunity for us to abuse overlap between actions performed by the root user on contents owned and controlled by the python user.

By reviewing the boot process, we noted a large number of custom init scripts in the /etc/init.d/ directory. These scripts are executed at system start by the root user and were responsible for starting daemons and ensuring directories or files exist. One file in particular, /etc/init.d/S50dropbear.sh, was interesting to us, as it appeared to perform a number of actions on files within the directory specified by the $PYTHON_HOME variable, which was /WEB/python/, shown in Figure 9.


Figure 9: Unsafe operations on $PYTHON_HOME directory

At first glance this may seem benign but considering that the /WEB/python/ directory is controllable by the python user, it means that we can potentially control actions taken by root. More specifically, the chown operation is dangerous, as the previous mkdir command can fail silently and result in an unsafe chown operation. To weaponize this, we can use symbolic links to point the /WEB/python/.ssh/ to other areas of the filesystem and coerce the root process into chown’ing these files to be owned by the python user. The process we took to exploit this was as follows:

  1. Authenticate over SSH using hardcoded python user credentials.
  2. Create a symbolic link, /WEB/python/.ssh, that points to /etc/init.d/.
  3. Reboot the X2e, forcing the system to re-execute /etc/init.d/S50dropbear.sh.
  4. After boot completes, create a malicious init script in /etc/init.d/ as the python user.
  5. Reboot the X2e, forcing the system to execute the new init script.

While not the cleanest approach (it requires two reboots), it accomplishes the goal of achieving code execution as root. Figure 10 shows the output of our proof of concept. In this case, our malicious init script spawned a bind shell on TCP port 8080, so that we could connect in as root.


Figure 10: Exploiting chown vulnerability to gain shell as user root

And there we have it: a remote connection as root, by abusing two separate vulnerabilities. While not explored in this series, another viable avenue of attack would be to explore potential vulnerabilities in the web server listening on TCP ports 80 and 443; however, this was not an approach that we took.

Conclusion

We covered a wide variety of topics in this two-part series, including:

  • Physical device inspection
  • Identifying and exploring physical debugging interfaces (UART)
  • Chip-off techniques to remove the NAND storage
  • Binary analysis of the filesystems and bootloader configurations
  • Power glitch attacks against the U-Boot bootloader
  • Linux user space privilege escalation

We hope that readers were able to learn from our experiences with the X2e and will be inspired to use these techniques in their own analysis. Finally, Mandiant would like to thank both Tesla/SolarCity and Digi International for their efforts to remediate these vulnerabilities and for their cooperation with releasing this blog series.

Shining a Light on SolarCity: Practical Exploitation of the X2e IoT Device (Part One)

17 February 2021 at 13:00

In 2019, Mandiant’s Red Team discovered a series of vulnerabilities present within Digi International’s ConnectPort X2e device, which allows for remote code execution as a privileged user. Specifically, Mandiant’s research focused on SolarCity’s (now owned by Tesla) rebranded ConnectPort X2e device, which is used in residential solar installations. Mandiant performs this type of work both for research purposes and in a professional capacity for their global clients.

Mandiant collaborated with Digi International and SolarCity/Tesla to responsibly disclose the results of the research, resulting in the following two CVEs:

Technical details can be found in Digi International’s 3.2.30.6 software release, and on FireEye’s Vulnerability Disclosures GitHub project (FEYE-2020-0019 and FEYE-2020-0020).

This two-part blog series will discuss our analysis at a high level, explore the novel techniques used to gain initial access to the ConnectPort X2e device, and share the technical details of the vulnerabilities discovered. Topics to be covered will include physical device inspection, debugging interface probing, chip-off techniques, firmware analysis, glitch attacks, and software exploitation.

If you’re interested in continuing the story in Part Two, you can read it now.

FAQ

What devices are affected, and (potentially) how many devices are affected?

The vulnerabilities described in this post affect ConnectPort X2e devices as well as the SolarCity rebranded variant. Other vendor devices may also be vulnerable. It is unclear how many ConnectPort X2e devices are deployed in the wild.

How is the issue being addressed?

Mandiant worked independently with Digi International and Tesla to remediate the vulnerabilities. Mandiant would like to thank Digi International and Tesla for their cooperation and dedication to improving the security of their products.

How would an attacker exploit these vulnerabilities?

An attacker with local network access (such as being connected to an individual’s home network via Ethernet) to a vulnerable X2e device can exploit CVE-2020-9306 and CVE-2020-12878 to gain privileged access to the device.

Who discovered these vulnerabilities?

Jake Valletta (@jake_valletta), Sam Sabetan (@samsabetan)

More information such as videos and datasheets on Mandiant’s Embedded Device Assessments can be found here.

Technical Analysis

Device Overview

Before diving into the details, we’ll discuss the ConnectPort X2e device (referred to as X2e device throughout the post) at a high level. The X2e device is a programmable gateway that connects to and collects data from ZigBee devices. It is commonly used as a Smart Energy gateway to interpret and send energy readings from a residential Solar Inverter. Vendors will often purchase an X2e device and configure it to read power consumption generated by a customer’s Solar Inverter. Figure 1 outlines a typical residential solar installation and highlights the X2e’s role.


Figure 1: Typical X2e residential deployment

For our research, we focused on the X2e device used by SolarCity, now Tesla, to retrieve data from residential solar installations. A typical setup would involve SolarCity providing a customer with a gateway that would be connected to the Internet via an Ethernet cable on the customer’s home network. Figure 2 shows one of the SolarCity branded X2e devices that we tested.


Figure 2: X2e device

Without even plugging in the X2e device, we know of at least two separate interfaces to explore: the Ethernet interface and the ZigBee radio. Note that we did not review the ZigBee interface between the X2e and a solar invertor, and that interface will not be covered in either Part One or Part Two of this series.

Initial Analysis and Physical Inspection

Network Reconnaissance

We started our research by assessing the X2e device from a network perspective. By using nmap, we discovered that the device exposed both SSH and HTTP/HTTPS, shown in Figure 3.


Figure 3: Port scan results from the X2e

Upon accessing these services remotely, we noted that both services required authentication. We also performed limited brute force attempts, which were unsuccessful. Additionally, the underlying services were not vulnerable to any public exploits. With not many network-based leads to follow, we shifted our analysis to a hardware perspective to determine if any local attacks may be possible to gain initial access onto the device.

Physical Board Inspection

To begin our hardware analysis, we removed the plastic casing from the device and mapped out the various integrated circuit (IC) components and searched for potential debugging interfaces. Inventorying the components present on the circuit board (also known as a PCB) is a crucial step in understanding how the device was designed and what can be expected down the road. Figure 4 shows the mapped-out components as well as a cluster of pins that resembled a typical 3-pin universal asynchronous transmit/receive (UART) connection, a common debugging interface on embedded devices.


Figure 4: X2e components and suspicious cluster of pins

Without a remote connection to the X2e device, UART is an attractive target. UART typically provides the equivalent functionality of a service like SSH or Telnet and the added benefit of watching verbose output during system boot. To determine if the cluster of pins was a UART interface, we first soldered a 3-pin through-hole header to the PCB. Using a combination of continuity tests with a multimeter and the digital logic analyzer Saleae, it became apparent that we were in fact dealing with a UART interface. The Figure 5 shows the three pins (Ground, TX, RX) connected to the header. Attached to the other end of the three wires was a FTDI serial TTL-232 to USB adapter, which was connected to a Linux virtual machine.


Figure 5: Connecting to potential UART interface

In addition to correctly identifying the UART pins and a UART to USB adapter, we also needed software to read/write from the interface as well as knowledge of the baud rate. Baud rates vary but typically follow standard values, including 9600, 14400, 19200, 38400, 57600, and 115200. Using the python module pySerial, we connected to the USB adapter and tried standard baud rates until one of the rates produced readable ASCII output (an incorrect baud rate will typically produce non-readable output), and determined the X2e used a baud rate of 115200.

Upon booting the X2e, we noted output from the BootROM, bootloader (which was Das U-Boot 2009.8, a common embedded bootloader), as well as output from the Linux kernel transmitted over the UART connection, shown in Figure 6.


Figure 6: UART boot messages

Many configurations of U-Boot allow a physically connected user (using an interface such as UART) the ability to interrupt the boot process; however, this configuration explicitly disabled that feature, shown in Figure 7.


Figure 7: Uninterruptable U-Boot bootloader on the X2e

Interrupting a bootloader is attractive to an attacker, as often the boot parameters passed to the Linux operating system can be manipulated to control how it will load, such as booting into single user mode (typically a recover shell) or mounting filesystems as read-write. In the case of the X2e, the UART connection was mapped to a Linux TTY which required username and password authentication, shown in Figure 8.


Figure 8: User authentication to Linux over UART

Without any ability to interrupt the boot process or credentials to authenticate to the X2e, we were faced with another dead end. We then shifted our analysis to obtaining the firmware stored on the X2e’s non-volatile storage.

Chip Removal and Data Extraction

In this section, we’ll cover the basics of non-volatile memory, often referred to as “flash memory”, present on embedded devices as well as the process used to extract content from the chip. As mentioned, taking inventory of the components on the PCB is an important first step. Figure 9 shows the suspected flash chip present on the PCB magnified under a digital microscope.


Figure 9: Closeup of Spansion flash

The visible markings seen in Figure 9 are important as they allow us to determine the manufacturer and model of the flash, which will assist us with obtaining the datasheet for the chip. In our case, the NAND we were dealing with was a Spansion S34ML01G1, and its datasheet could be found here.

NAND Overview

Before we talk about acquiring the firmware from the NAND chip, it’s important to first understand the various scenarios that embedded devices typically follow.

NAND verses NOR: These fundamentally different technologies each have their own benefits and drawbacks. NAND is cheap but suffers from high probability of “bad blocks,” or areas that are corrupt sometimes directly from the factory. As such, protections and considerations need to be present to be able to protect against this. NAND is also much faster to erase and write, making it ideal for storing file systems, kernels, and other pieces of code that may need to be reset or changed. NOR has significantly faster read times but is not as flexible with accessing data and has low erase and write speeds. NOR is usually used for low-level bootloaders, hardcoded firmware blobs, and other areas that are not expected to change frequently. The X2e uses a NAND flash.

Serial verses Parallel: This refers to how the data is accessed and is typically visually identifiable. If there are a large number of pins, the flash is likely parallel. Serial NOR chips can be small in size and typically need eight or fewer pins to function. Common serial interfaces are Serial Peripheral Interface (SPI) or Inter-Integrated Circuit (I2C), while a common parallel interface for NAND is Open NAND Flash Interface (ONFI2.0, ONFI3.0). The X2e is a parallel flash.

IC Form Factor: Another visually identifiable trait—form factor (or “package”)—refers to how the chip is attached to the PCB. There is a long list of options here, but common surface-mount flash packages include small outline package (SOP), thin outline small package (TOSP), or a variant of ball grid array (*BGA). The key distinction here is SOP and TOSP expose the pins, while BGA conceals the pins under the package. The X2e is BGA63, also referred to as a 63-pin BGA package.

Managed verses Unmanaged Flash: This one is more applicable to NAND, for reasons alluded to in the NAND verses NOR section. As stated, NAND needs help to manage the integrity of the data. With unmanaged NAND, the IC reserves sections of the flash (often called “spare” area) for someone else to manage the data. This is typically implemented as either a kernel driver or an external NAND controller. Managed NAND means that the IC package includes the controller and transparently manages the data. This is extremely common in embedded devices, as either embedded MMC (eMMC) or universal flash storage (UFS). The X2e uses unmanaged flash and is controlled by the main microcontroller present on the PCB.

With the basics out of the way, we proceeded with physically removing the chip from the PCB.

Chip Removal

Physical chip removal is considered a destructive approach but can certainly be performed without damaging the PCB or the flash chip itself. When presented with removal of BGA packages, the two most common removal techniques are either hot air or infrared light (IR). Commercial solutions exist for both hot air and IR, but cheaper options exist with hot air removal. We opted to use hot air on the X2e.

To minimize damage to the PCB and flash, a PCB heater or oven can be used to slowly bring the entire PCB to a temperature right below the solder melting point. This will reduce the amount of time we need to focus our hot air directly onto the flash IC and help with reducing the heat dissipation into the PCB throughout the process.

One final trick that can be used to minimize nearby chips from being damaged or lost (due to the air pressure) is the use of high-heat resistant tape, commonly referred to as Kapton tape. Figure 10 shows the PCB wrapped in Kapton tape to protect nearby components.


Figure 10: High-heat resistant tape on PCB

Figure 11 shows an example setup with the X2e PCB inserted into a PCB heater, with a hot air gun suspended over the IC.


Figure 11: Hot air rework/reflow station

While using the hot air to warm the IC and surrounding areas, we gently nudged the flash to see if the solder had become molten. Once the chip appeared to be floating, we quickly removed the chip and let it cool for about 30 seconds. Figure 12 shows the IC flash removed from the PCB, with the solder still present on the BGA pads.


Figure 12: NAND removed from X2e

Before inserting the NAND into a clam-shell chip reader, the leftover solder must be removed from the flash. This can be accomplished using a soldering iron, high-quality flux, and de-soldering wick. Once removed, isopropyl alcohol and a toothbrush are highly effective at removing the leftover flux residue and cleaning the chip.

In the next section, we’ll attempt to extract the data from the NAND chip using a multi-purpose chip programmer.

Data Extraction

With the cleaned flash chip in hand, we can now explore options for reading the raw contents. Commercial forensic acquisition devices exist, but a quick eBay or AliExpress search will produce a multitude of generic chip readers. One such device, the XGecu Pro, supports a variety of adapters and chipsets and connects to a Windows machine using USB. It also comes with software to interface with the XGecu Pro and can even auto-detect flash. To connect the Spansion NAND to the XGecu Pro, we also purchased a clamshell BGA63 adapter. Figure 13 shows the NAND inserted into the clamshell reader, and Figure 14 shows the clamshell adapter connected to the XGecu Pro device.


Figure 13: Spansion NAND in BGA clamshell adapter


Figure 14: NAND adapter connected to XGecu

Using the XGecu Pro software, we can read the entire contents of the flash to a binary file for further analysis. Since these are not commercial solutions, it is a good idea to perform two or three reads and then diff the extraction to confirm the content was read without errors.

Firmware Analysis

Cleaning and Mounting

With our fresh NAND dump in hand, the next step was to parse out any relevant firmware blobs, configurations, or filesystems. The go-to tool for starting this process is binwalk. binwalk does a fantastic job of detecting filesystems, bootloaders, and kernels. In addition, binwalk can calculate entropy (detecting packed or encrypted data) and identify assembly opcodes. Figure 15 shows partial output of running binwalk against the NAND dump.


Figure 15: Initial binwalk scan against NAND dump

We can see from the output that binwalk successfully identified what it believes are U-Boot uImage headers, Linux kernel images, and more than a dozen Journaling Flash File System version 2 (JFFS2) filesystems. JFFS2 is a common filesystem used in embedded devices; Unsorted Block Image File System (UBIFS) and SquashFS would also be common.

At first glance, the output appears to be promising; however, it is highly unlikely that there are actually that many JFFS2 filesystems present on our NAND. Another indication that something isn’t quite right are the hexadecimal offsets – they don’t appear to be clean, uniform offsets. It is far more common that the offsets of the items identified by binwalk would align with NAND page offsets, which are a multiple of 2048.

In order to understand what is occurring here, we need to revisit a characteristic of unmanaged (or “raw”) NAND ICs described in the NAND Overview section. To recap, raw NAND requires additional bytes per page for use by higher-level components to attest to the validity of the page, typically implemented as a defined “bad block” marker and a per-page (or subpage) Error-Correcting Code (ECC). Without going too deep into ECC fundamentals, ECC provides the ability for higher-level processes to detect n number of bad bits on a page and to correct m number of bits.

Since our goal here is not to perform forensics on the raw NAND, our immediate objective is to remove any ECC bytes or other non-data related bytes from the NAND dump. The MCU is ultimately the system manipulating the raw NAND, so understanding how our MCU, which was an NXP iMX28 series MCU, manages NAND is critical to being able to perform this.

Fortunately for us, this process has already been explored by the security community, and iMX parsing libraries exist to manipulate the raw NAND dump and remove existing extraneous data. Figure 16 shows the results of re-running binwalk on the output of the imx-nand-convert script.


Figure 16: binwalk scan of fixed NAND dump

This time, we see only one JFFS2 filesystem, at the very round offset of 0x880000. Using the extraction (-e) feature of binwalk, we can now obtain parsed versions of the U-Boot bootloader, Linux kernel, and JFFS2 system.

The final hurdle we need to overcome is mounting the extracted JFFS2 filesystem in a way that allows us to explore the contents. On Linux, the easiest way to perform this is to use the mtd, mtdblock, and nandsim kernel modules. The nandsim module simulates a given NAND device and uses the mtd and JFFS2 subsystems to parse and manage appropriately. The key piece of information that needs to be passed to the nandsim module is the ONFI chip identifier, which can be obtained from the NAND datasheet or by requesting the ID from the IC using a generic reader (like the XGecu Pro used in the Data Extraction section). A list of supported IDs is also provided by the mtd maintainers. Getting the parameters correct is a bit of luck and magic and may require you to compile your own version of the nandsim module; that process will not be covered in this post.

Figure 17 shows the steps required to simulate the correct Spansion NAND and mount the JFFS2 filesystem in the form of a Makefile target.


Figure 17: Makefile target to mount JFFS2 filesystem

By running make mount-jffs2, we can quickly prep and mount the JFFS2 filesystem and explore the contents as we would any filesystem.

Accessing the Filesystem

In the last section of this post, we’ll walk through our analysis of the JFFS2 filesystem. Remember that our end goal is to obtain a remotely exploitable bug that will permit privileged code execution. With that in mind, some areas of interest are running daemons/processes, system startup logic, and credentials for services listening on the network. The first stop was reviewing the /etc/shadow file to see if there were password hashes for the root user as well as other system users. A quick check of this file determined there was no password hash for the root user, which indicated we would not be able to authenticate using password authentication. We noticed that two other password hashes were present, for the addpd and python users, shown in Figure 18.


Figure 18: Connects of /etc/shadow

The addpd user had a weak default password but was unable to authenticate using remote methods, and we were ultimately unable to crack the python user’s hash using internal GPU-based servers.

Additionally, we were interested in processes that are launched during system boot or post-boot. The directory /WEB/python/ contained a ZIP archive called _x2e.zip, which contained over 200 compiled Python scripts (PYC files), which were loaded on system boot. Using the decompiler uncompyle2, we unpacked these files for review. One file that stood out by name was password_manager.pyc, a file used to reset the login password upon successful boot-up. The file contained five hardcoded and plaintext credentials that mapped to the python system user. These credentials could be used to access the web interface and SSH, shown in Figure 19. Mandiant confirmed different passwords were used for different versions and connectivity states. Mandiant reported this to SolarCity and was assigned the CVE number CVE-2020-9306.


Figure 19: Hardcoded credentials in password_manager.pyc

With the correct password, we were finally able to connect to the web and SSH ports on a running X2e, but unfortunately only as the less-privileged python system user. While this was a great start, it didn’t satisfy our final objective, which was to remotely compromise the X2e as a privileged user. In Part Two of this blog series, we will explore additional avenues to further compromise the X2e.

Conclusion

In Part One of this two-part blog series, we covered an overview of the X2e, our initial network-based reconnaissance, PCB inspection techniques, physical debugging interface probing, chip-off techniques, and firmware analysis. Using these methodologies, we were successfully able to remotely compromise the X2e device as a non-administrative user due to hardcoded credentials (CVE-2020-9306). In Part Two, we will re-investigate physical attacks against the X2e in the form of glitch attacks, re-explore the U-Boot bootloader, and finally demonstrate an attack to remotely compromise the X2e device as a privileged user.

To continue reading, check out Part Two now

Shining a Light on SolarCity: Practical Exploitation of the X2e IoT Device (Part One)

17 February 2021 at 13:00

In 2019, Mandiant’s Red Team discovered a series of vulnerabilities present within Digi International’s ConnectPort X2e device, which allows for remote code execution as a privileged user. Specifically, Mandiant’s research focused on SolarCity’s (now owned by Tesla) rebranded ConnectPort X2e device, which is used in residential solar installations. Mandiant performs this type of work both for research purposes and in a professional capacity for their global clients.

Mandiant collaborated with Digi International and SolarCity/Tesla to responsibly disclose the results of the research, resulting in the following two CVEs:

Technical details can be found in Digi International’s 3.2.30.6 software release, and on FireEye’s Vulnerability Disclosures GitHub project (FEYE-2020-0019 and FEYE-2020-0020).

This two-part blog series will discuss our analysis at a high level, explore the novel techniques used to gain initial access to the ConnectPort X2e device, and share the technical details of the vulnerabilities discovered. Topics to be covered will include physical device inspection, debugging interface probing, chip-off techniques, firmware analysis, glitch attacks, and software exploitation.

If you’re interested in continuing the story in Part Two, you can read it now.

FAQ

What devices are affected, and (potentially) how many devices are affected?

The vulnerabilities described in this post affect ConnectPort X2e devices as well as the SolarCity rebranded variant. Other vendor devices may also be vulnerable. It is unclear how many ConnectPort X2e devices are deployed in the wild.

How is the issue being addressed?

Mandiant worked independently with Digi International and Tesla to remediate the vulnerabilities. Mandiant would like to thank Digi International and Tesla for their cooperation and dedication to improving the security of their products.

How would an attacker exploit these vulnerabilities?

An attacker with local network access (such as being connected to an individual’s home network via Ethernet) to a vulnerable X2e device can exploit CVE-2020-9306 and CVE-2020-12878 to gain privileged access to the device.

Who discovered these vulnerabilities?

Jake Valletta (@jake_valletta), Sam Sabetan (@samsabetan)

More information such as videos and datasheets on Mandiant’s Embedded Device Assessments can be found here.

Technical Analysis

Device Overview

Before diving into the details, we’ll discuss the ConnectPort X2e device (referred to as X2e device throughout the post) at a high level. The X2e device is a programmable gateway that connects to and collects data from ZigBee devices. It is commonly used as a Smart Energy gateway to interpret and send energy readings from a residential Solar Inverter. Vendors will often purchase an X2e device and configure it to read power consumption generated by a customer’s Solar Inverter. Figure 1 outlines a typical residential solar installation and highlights the X2e’s role.


Figure 1: Typical X2e residential deployment

For our research, we focused on the X2e device used by SolarCity, now Tesla, to retrieve data from residential solar installations. A typical setup would involve SolarCity providing a customer with a gateway that would be connected to the Internet via an Ethernet cable on the customer’s home network. Figure 2 shows one of the SolarCity branded X2e devices that we tested.


Figure 2: X2e device

Without even plugging in the X2e device, we know of at least two separate interfaces to explore: the Ethernet interface and the ZigBee radio. Note that we did not review the ZigBee interface between the X2e and a solar invertor, and that interface will not be covered in either Part One or Part Two of this series.

Initial Analysis and Physical Inspection

Network Reconnaissance

We started our research by assessing the X2e device from a network perspective. By using nmap, we discovered that the device exposed both SSH and HTTP/HTTPS, shown in Figure 3.


Figure 3: Port scan results from the X2e

Upon accessing these services remotely, we noted that both services required authentication. We also performed limited brute force attempts, which were unsuccessful. Additionally, the underlying services were not vulnerable to any public exploits. With not many network-based leads to follow, we shifted our analysis to a hardware perspective to determine if any local attacks may be possible to gain initial access onto the device.

Physical Board Inspection

To begin our hardware analysis, we removed the plastic casing from the device and mapped out the various integrated circuit (IC) components and searched for potential debugging interfaces. Inventorying the components present on the circuit board (also known as a PCB) is a crucial step in understanding how the device was designed and what can be expected down the road. Figure 4 shows the mapped-out components as well as a cluster of pins that resembled a typical 3-pin universal asynchronous transmit/receive (UART) connection, a common debugging interface on embedded devices.


Figure 4: X2e components and suspicious cluster of pins

Without a remote connection to the X2e device, UART is an attractive target. UART typically provides the equivalent functionality of a service like SSH or Telnet and the added benefit of watching verbose output during system boot. To determine if the cluster of pins was a UART interface, we first soldered a 3-pin through-hole header to the PCB. Using a combination of continuity tests with a multimeter and the digital logic analyzer Saleae, it became apparent that we were in fact dealing with a UART interface. The Figure 5 shows the three pins (Ground, TX, RX) connected to the header. Attached to the other end of the three wires was a FTDI serial TTL-232 to USB adapter, which was connected to a Linux virtual machine.


Figure 5: Connecting to potential UART interface

In addition to correctly identifying the UART pins and a UART to USB adapter, we also needed software to read/write from the interface as well as knowledge of the baud rate. Baud rates vary but typically follow standard values, including 9600, 14400, 19200, 38400, 57600, and 115200. Using the python module pySerial, we connected to the USB adapter and tried standard baud rates until one of the rates produced readable ASCII output (an incorrect baud rate will typically produce non-readable output), and determined the X2e used a baud rate of 115200.

Upon booting the X2e, we noted output from the BootROM, bootloader (which was Das U-Boot 2009.8, a common embedded bootloader), as well as output from the Linux kernel transmitted over the UART connection, shown in Figure 6.


Figure 6: UART boot messages

Many configurations of U-Boot allow a physically connected user (using an interface such as UART) the ability to interrupt the boot process; however, this configuration explicitly disabled that feature, shown in Figure 7.


Figure 7: Uninterruptable U-Boot bootloader on the X2e

Interrupting a bootloader is attractive to an attacker, as often the boot parameters passed to the Linux operating system can be manipulated to control how it will load, such as booting into single user mode (typically a recover shell) or mounting filesystems as read-write. In the case of the X2e, the UART connection was mapped to a Linux TTY which required username and password authentication, shown in Figure 8.


Figure 8: User authentication to Linux over UART

Without any ability to interrupt the boot process or credentials to authenticate to the X2e, we were faced with another dead end. We then shifted our analysis to obtaining the firmware stored on the X2e’s non-volatile storage.

Chip Removal and Data Extraction

In this section, we’ll cover the basics of non-volatile memory, often referred to as “flash memory”, present on embedded devices as well as the process used to extract content from the chip. As mentioned, taking inventory of the components on the PCB is an important first step. Figure 9 shows the suspected flash chip present on the PCB magnified under a digital microscope.


Figure 9: Closeup of Spansion flash

The visible markings seen in Figure 9 are important as they allow us to determine the manufacturer and model of the flash, which will assist us with obtaining the datasheet for the chip. In our case, the NAND we were dealing with was a Spansion S34ML01G1, and its datasheet could be found here.

NAND Overview

Before we talk about acquiring the firmware from the NAND chip, it’s important to first understand the various scenarios that embedded devices typically follow.

NAND verses NOR: These fundamentally different technologies each have their own benefits and drawbacks. NAND is cheap but suffers from high probability of “bad blocks,” or areas that are corrupt sometimes directly from the factory. As such, protections and considerations need to be present to be able to protect against this. NAND is also much faster to erase and write, making it ideal for storing file systems, kernels, and other pieces of code that may need to be reset or changed. NOR has significantly faster read times but is not as flexible with accessing data and has low erase and write speeds. NOR is usually used for low-level bootloaders, hardcoded firmware blobs, and other areas that are not expected to change frequently. The X2e uses a NAND flash.

Serial verses Parallel: This refers to how the data is accessed and is typically visually identifiable. If there are a large number of pins, the flash is likely parallel. Serial NOR chips can be small in size and typically need eight or fewer pins to function. Common serial interfaces are Serial Peripheral Interface (SPI) or Inter-Integrated Circuit (I2C), while a common parallel interface for NAND is Open NAND Flash Interface (ONFI2.0, ONFI3.0). The X2e is a parallel flash.

IC Form Factor: Another visually identifiable trait—form factor (or “package”)—refers to how the chip is attached to the PCB. There is a long list of options here, but common surface-mount flash packages include small outline package (SOP), thin outline small package (TOSP), or a variant of ball grid array (*BGA). The key distinction here is SOP and TOSP expose the pins, while BGA conceals the pins under the package. The X2e is BGA63, also referred to as a 63-pin BGA package.

Managed verses Unmanaged Flash: This one is more applicable to NAND, for reasons alluded to in the NAND verses NOR section. As stated, NAND needs help to manage the integrity of the data. With unmanaged NAND, the IC reserves sections of the flash (often called “spare” area) for someone else to manage the data. This is typically implemented as either a kernel driver or an external NAND controller. Managed NAND means that the IC package includes the controller and transparently manages the data. This is extremely common in embedded devices, as either embedded MMC (eMMC) or universal flash storage (UFS). The X2e uses unmanaged flash and is controlled by the main microcontroller present on the PCB.

With the basics out of the way, we proceeded with physically removing the chip from the PCB.

Chip Removal

Physical chip removal is considered a destructive approach but can certainly be performed without damaging the PCB or the flash chip itself. When presented with removal of BGA packages, the two most common removal techniques are either hot air or infrared light (IR). Commercial solutions exist for both hot air and IR, but cheaper options exist with hot air removal. We opted to use hot air on the X2e.

To minimize damage to the PCB and flash, a PCB heater or oven can be used to slowly bring the entire PCB to a temperature right below the solder melting point. This will reduce the amount of time we need to focus our hot air directly onto the flash IC and help with reducing the heat dissipation into the PCB throughout the process.

One final trick that can be used to minimize nearby chips from being damaged or lost (due to the air pressure) is the use of high-heat resistant tape, commonly referred to as Kapton tape. Figure 10 shows the PCB wrapped in Kapton tape to protect nearby components.


Figure 10: High-heat resistant tape on PCB

Figure 11 shows an example setup with the X2e PCB inserted into a PCB heater, with a hot air gun suspended over the IC.


Figure 11: Hot air rework/reflow station

While using the hot air to warm the IC and surrounding areas, we gently nudged the flash to see if the solder had become molten. Once the chip appeared to be floating, we quickly removed the chip and let it cool for about 30 seconds. Figure 12 shows the IC flash removed from the PCB, with the solder still present on the BGA pads.


Figure 12: NAND removed from X2e

Before inserting the NAND into a clam-shell chip reader, the leftover solder must be removed from the flash. This can be accomplished using a soldering iron, high-quality flux, and de-soldering wick. Once removed, isopropyl alcohol and a toothbrush are highly effective at removing the leftover flux residue and cleaning the chip.

In the next section, we’ll attempt to extract the data from the NAND chip using a multi-purpose chip programmer.

Data Extraction

With the cleaned flash chip in hand, we can now explore options for reading the raw contents. Commercial forensic acquisition devices exist, but a quick eBay or AliExpress search will produce a multitude of generic chip readers. One such device, the XGecu Pro, supports a variety of adapters and chipsets and connects to a Windows machine using USB. It also comes with software to interface with the XGecu Pro and can even auto-detect flash. To connect the Spansion NAND to the XGecu Pro, we also purchased a clamshell BGA63 adapter. Figure 13 shows the NAND inserted into the clamshell reader, and Figure 14 shows the clamshell adapter connected to the XGecu Pro device.


Figure 13: Spansion NAND in BGA clamshell adapter


Figure 14: NAND adapter connected to XGecu

Using the XGecu Pro software, we can read the entire contents of the flash to a binary file for further analysis. Since these are not commercial solutions, it is a good idea to perform two or three reads and then diff the extraction to confirm the content was read without errors.

Firmware Analysis

Cleaning and Mounting

With our fresh NAND dump in hand, the next step was to parse out any relevant firmware blobs, configurations, or filesystems. The go-to tool for starting this process is binwalk. binwalk does a fantastic job of detecting filesystems, bootloaders, and kernels. In addition, binwalk can calculate entropy (detecting packed or encrypted data) and identify assembly opcodes. Figure 15 shows partial output of running binwalk against the NAND dump.


Figure 15: Initial binwalk scan against NAND dump

We can see from the output that binwalk successfully identified what it believes are U-Boot uImage headers, Linux kernel images, and more than a dozen Journaling Flash File System version 2 (JFFS2) filesystems. JFFS2 is a common filesystem used in embedded devices; Unsorted Block Image File System (UBIFS) and SquashFS would also be common.

At first glance, the output appears to be promising; however, it is highly unlikely that there are actually that many JFFS2 filesystems present on our NAND. Another indication that something isn’t quite right are the hexadecimal offsets – they don’t appear to be clean, uniform offsets. It is far more common that the offsets of the items identified by binwalk would align with NAND page offsets, which are a multiple of 2048.

In order to understand what is occurring here, we need to revisit a characteristic of unmanaged (or “raw”) NAND ICs described in the NAND Overview section. To recap, raw NAND requires additional bytes per page for use by higher-level components to attest to the validity of the page, typically implemented as a defined “bad block” marker and a per-page (or subpage) Error-Correcting Code (ECC). Without going too deep into ECC fundamentals, ECC provides the ability for higher-level processes to detect n number of bad bits on a page and to correct m number of bits.

Since our goal here is not to perform forensics on the raw NAND, our immediate objective is to remove any ECC bytes or other non-data related bytes from the NAND dump. The MCU is ultimately the system manipulating the raw NAND, so understanding how our MCU, which was an NXP iMX28 series MCU, manages NAND is critical to being able to perform this.

Fortunately for us, this process has already been explored by the security community, and iMX parsing libraries exist to manipulate the raw NAND dump and remove existing extraneous data. Figure 16 shows the results of re-running binwalk on the output of the imx-nand-convert script.


Figure 16: binwalk scan of fixed NAND dump

This time, we see only one JFFS2 filesystem, at the very round offset of 0x880000. Using the extraction (-e) feature of binwalk, we can now obtain parsed versions of the U-Boot bootloader, Linux kernel, and JFFS2 system.

The final hurdle we need to overcome is mounting the extracted JFFS2 filesystem in a way that allows us to explore the contents. On Linux, the easiest way to perform this is to use the mtd, mtdblock, and nandsim kernel modules. The nandsim module simulates a given NAND device and uses the mtd and JFFS2 subsystems to parse and manage appropriately. The key piece of information that needs to be passed to the nandsim module is the ONFI chip identifier, which can be obtained from the NAND datasheet or by requesting the ID from the IC using a generic reader (like the XGecu Pro used in the Data Extraction section). A list of supported IDs is also provided by the mtd maintainers. Getting the parameters correct is a bit of luck and magic and may require you to compile your own version of the nandsim module; that process will not be covered in this post.

Figure 17 shows the steps required to simulate the correct Spansion NAND and mount the JFFS2 filesystem in the form of a Makefile target.


Figure 17: Makefile target to mount JFFS2 filesystem

By running make mount-jffs2, we can quickly prep and mount the JFFS2 filesystem and explore the contents as we would any filesystem.

Accessing the Filesystem

In the last section of this post, we’ll walk through our analysis of the JFFS2 filesystem. Remember that our end goal is to obtain a remotely exploitable bug that will permit privileged code execution. With that in mind, some areas of interest are running daemons/processes, system startup logic, and credentials for services listening on the network. The first stop was reviewing the /etc/shadow file to see if there were password hashes for the root user as well as other system users. A quick check of this file determined there was no password hash for the root user, which indicated we would not be able to authenticate using password authentication. We noticed that two other password hashes were present, for the addpd and python users, shown in Figure 18.


Figure 18: Connects of /etc/shadow

The addpd user had a weak default password but was unable to authenticate using remote methods, and we were ultimately unable to crack the python user’s hash using internal GPU-based servers.

Additionally, we were interested in processes that are launched during system boot or post-boot. The directory /WEB/python/ contained a ZIP archive called _x2e.zip, which contained over 200 compiled Python scripts (PYC files), which were loaded on system boot. Using the decompiler uncompyle2, we unpacked these files for review. One file that stood out by name was password_manager.pyc, a file used to reset the login password upon successful boot-up. The file contained five hardcoded and plaintext credentials that mapped to the python system user. These credentials could be used to access the web interface and SSH, shown in Figure 19. Mandiant confirmed different passwords were used for different versions and connectivity states. Mandiant reported this to SolarCity and was assigned the CVE number CVE-2020-9306.


Figure 19: Hardcoded credentials in password_manager.pyc

With the correct password, we were finally able to connect to the web and SSH ports on a running X2e, but unfortunately only as the less-privileged python system user. While this was a great start, it didn’t satisfy our final objective, which was to remotely compromise the X2e as a privileged user. In Part Two of this blog series, we will explore additional avenues to further compromise the X2e.

Conclusion

In Part One of this two-part blog series, we covered an overview of the X2e, our initial network-based reconnaissance, PCB inspection techniques, physical debugging interface probing, chip-off techniques, and firmware analysis. Using these methodologies, we were successfully able to remotely compromise the X2e device as a non-administrative user due to hardcoded credentials (CVE-2020-9306). In Part Two, we will re-investigate physical attacks against the X2e in the form of glitch attacks, re-explore the U-Boot bootloader, and finally demonstrate an attack to remotely compromise the X2e device as a privileged user.

To continue reading, check out Part Two now

Shining a Light on SolarCity: Practical Exploitation of the X2e IoT Device (Part Two)

17 February 2021 at 13:00

In this post, we continue our analysis of the SolarCity ConnectPort X2e Zigbee device (referred to throughout as X2e device). In Part One, we discussed the X2e at a high level, performed initial network-based attacks, then discussed the hardware techniques used to gain a remote shell on the X2e device as a non-privileged system user. In this segment, we’ll cover how we obtained a privileged shell on the device locally using power glitching attacks, and explore CVE-2020-12878, a vulnerability we discovered that permitted remote privilege escalation to the root user. Combined with CVE-2020-9306 (discussed in Part One), this would result in a complete remote compromise of the X2e device.

Technical Analysis

Recap

Before we dive into next steps, let’s recap where we left off:

  • The X2e has an exposed universal asynchronous transmit/receive (UART) interface, which allows a physically connected user to view (but not interrupt) the Das U-Boot (U-Boot) boot process, and given proper credentials, authenticate to the Linux operating system. Since we do not have root credentials, we put this thread on the backburner.
  • We have a full NAND dump of the Spansion raw flash, which includes boot configuration, bootloader firmware, filesystems, and the Linux kernel image. This was used previously in Part One to obtain the hardcoded credential for the python user.

Knowing that UART is present and access to the bootloader would be extremely valuable, we decided to revisit that thread.

Gaining Privileged Access Locally

Revisiting the Bootloader

Figure 1 shows the U-Boot boot process displayed while connected via UART connection. In some cases, it is possible to send keyboard input to the device during a set period (usually one to four seconds) when the bootloader presents the message, “Hit any key to stop autoboot,” which interrupts the boot process and drops the user into a U-Boot shell. On the X2e, this feature has been disabled by setting the U-Boot configuration parameter CONFIG_BOOTDELAY to 0.


Figure 1: Uninterruptable U-Boot bootloader output

One attack that has been documented to be successful to disrupt autoboot is to manipulate the bootloader’s ability to access the flash storage during the boot process. In certain circumstances where the U-Boot bootloader is unable to access its own configuration, it fails into a default environment, which may be less restricted. We decided to see if this would be possible on the X2e.

These attacks, known as glitch attacks (or more officially known as fault-injection), are a type of side channel attack that attempts to cause a microcontroller unit (MCU) to skip instructions, perform wrong instructions, or fail to access flash memory. Various types of glitching attacks exist including electrical, thermal, and radiation. Based on our objective, we opted to try glitching the power between the MCU and the Spansion NAND flash. Note that glitch attacks can often cause damage to the components on a board or put the device in an unusable state. These types of attacks should be tested as either a last resort or against a secondary device you are comfortable with damaging.

Glitching the Bootloader

Based on previous research in this domain, we opted to target the data lines (I/O) between the MCU and NAND flash. Recall from Part One that the NAND flash on the X2e was the Spansion S34ML01G1, which was a 63-pin ball grid array (BGA) package. This chip is capable of supporting both 8-bit and 16-bit bus width, which corresponds to the number of I/O lines utilized. By using the datasheet for the flash and then querying the ONFI Device ID of our chip, we determined our chip was utilizing the 8-bit configuration, meaning eight I/O lines were present between the NAND flash and the MCU. For this attack, we focused on manipulating the power on the first (I/O0) data line. Figure 2 shows the configuration of the BGA-63 pins, with I/O0 highlighted.


Figure 2: Identifying I/O0 for NAND chip in the Spansion datasheet

Because the pins are actually underneath the flash package, we needed to find an exposed lead that corresponded to I/O0 elsewhere on the PCB. One such method for tracing connections across a PCB is a continuity test. A continuity test (using a multimeter) sends a low current electrical signal across two points and produces an audible beep if the points are connected. Using this technique, we located an exposed test point (known as a via) on the bottom of the PCB. Figure 3 shows the I/O0 pin on the top of the PCB (under the NAND chip), and Figure 4 shows the I/O0 pin exposed on the bottom of the PCB.


Figure 3: I/O0 on top of PCB (under NAND chip)


Figure 4: I/O0 on bottom of PCB

With exposed access to I/O0 located, we experimented with connecting this pin directly to a known ground (GND) pin at various points during the boot process. Figure 5 shows the device powering on with the metal tweezers connecting I/O0 to GND.


Figure 5: Shorting I/O0 to GND

While connected to the UART interface, we noted several different outcomes. When shorting the pin immediately after powering on, the device failed to produce any output or boot. When shorting after the bootloader finished loading (and handing off to the Linux kernel), the device would also force reboot. However, when timed perfectly between the bootloader loading and attempting to read its configuration, we noted that the bootloader would present different output, and the option to interrupt the boot process was possible with a four-second delay. By pressing keyboard input, we were successfully able to drop into a U-Boot shell, which is shown in Figure 6.


Figure 6: Access to U-Boot bootloader shell

While this was great progress, we noted that the current failback bootloader configuration was completely inoperable and certain NAND blocks had been marked as bad (as expected). To get our device back to a working state, we needed to revisit the NAND dump we generated in Part One.

Repairing the Bootloader Configuration

While the current configuration provided us a working shell, we needed to fix the damage we had done. This was performed in two steps: fixing the mistakenly marked bad blocks and then rebuilding the configuration. In our case, the nand utility and its sub-commands read, write, and scrub allowed us to inspect and manipulate pages and blocks of the NAND. The nand scrub command with a valid offset and size could be used to completely reset a segment of the NAND, which removed any bad block markers. The next challenge was determining what needed to be replaced in the damaged blocks and rebuilding the configuration.

Since we had a valid NAND image, we revisited the sections read by the bootloader to determine what changes were needed. The format did not match a known format, so we wrote a simple parser in Python to read the binary structure, shown in Figure 7.


Figure 7: Parsing bootloader nvram configuration from flash

With details of how the configuration should look, we used the nand write to rebuild this section, byte by byte with the correct details. We also set the boot delay to be four seconds, so that we could always interrupt the bootloader once the new configuration was committed. Once we confirmed our changes were stable, we saved the configuration to flash and could access the bootloader without performing the aforementioned glitch attack.

Accessing Linux as root User

Now that we have unrestricted access to the bootloader, we can finally influence the rest of the boot process and achieve a privileged shell. We alluded to this in Part One, but the easiest way to turn an unlocked U-Boot shell into a root Linux shell is to adjust the boot arguments that U-Boot passes to the Linux kernel. In our case, this was accomplished by using the setenv utility to change the std_bootarg environment variable to be init=/bin/sh and instructing U-Boot to resume the standard boot process. Figure 8 shows the Linux shell presented over UART.


Figure 8: root shell after bootloader

At this point, we’ve demonstrated a repeatable method for achieving local privilege escalation. In the final segment, we’ll complete our attack by exploring an avenue to remotely escalate privileges.

Gaining Privileged Access Remotely

Since the X2e has only two available listening network services, it makes sense to reinvestigate these services. During Part One, we identified hardcoded credentials for the limited user python. This was useful for initial probing of the device while it was running, but where do we go from here?

Embedded devices typically only have a handful of users, with a majority of functionality being performed by the root user. This presents an interesting opportunity for us to abuse overlap between actions performed by the root user on contents owned and controlled by the python user.

By reviewing the boot process, we noted a large number of custom init scripts in the /etc/init.d/ directory. These scripts are executed at system start by the root user and were responsible for starting daemons and ensuring directories or files exist. One file in particular, /etc/init.d/S50dropbear.sh, was interesting to us, as it appeared to perform a number of actions on files within the directory specified by the $PYTHON_HOME variable, which was /WEB/python/, shown in Figure 9.


Figure 9: Unsafe operations on $PYTHON_HOME directory

At first glance this may seem benign but considering that the /WEB/python/ directory is controllable by the python user, it means that we can potentially control actions taken by root. More specifically, the chown operation is dangerous, as the previous mkdir command can fail silently and result in an unsafe chown operation. To weaponize this, we can use symbolic links to point the /WEB/python/.ssh/ to other areas of the filesystem and coerce the root process into chown’ing these files to be owned by the python user. The process we took to exploit this was as follows:

  1. Authenticate over SSH using hardcoded python user credentials.
  2. Create a symbolic link, /WEB/python/.ssh, that points to /etc/init.d/.
  3. Reboot the X2e, forcing the system to re-execute /etc/init.d/S50dropbear.sh.
  4. After boot completes, create a malicious init script in /etc/init.d/ as the python user.
  5. Reboot the X2e, forcing the system to execute the new init script.

While not the cleanest approach (it requires two reboots), it accomplishes the goal of achieving code execution as root. Figure 10 shows the output of our proof of concept. In this case, our malicious init script spawned a bind shell on TCP port 8080, so that we could connect in as root.


Figure 10: Exploiting chown vulnerability to gain shell as user root

And there we have it: a remote connection as root, by abusing two separate vulnerabilities. While not explored in this series, another viable avenue of attack would be to explore potential vulnerabilities in the web server listening on TCP ports 80 and 443; however, this was not an approach that we took.

Conclusion

We covered a wide variety of topics in this two-part series, including:

  • Physical device inspection
  • Identifying and exploring physical debugging interfaces (UART)
  • Chip-off techniques to remove the NAND storage
  • Binary analysis of the filesystems and bootloader configurations
  • Power glitch attacks against the U-Boot bootloader
  • Linux user space privilege escalation

We hope that readers were able to learn from our experiences with the X2e and will be inspired to use these techniques in their own analysis. Finally, Mandiant would like to thank both Tesla/SolarCity and Digi International for their efforts to remediate these vulnerabilities and for their cooperation with releasing this blog series.

  • There are no more articles
❌