This article is in no way affiliated, sponsored, or endorsed with/by Citrix Systems, Inc. All graphics are being displayed under fair use for the purposes of this article.
Hacking Citrix Storefront Users
With the substantial shift from traditional work environments to remote/telework capable infrastructures due to COVID-19, products like Citrix Storefront have seen a significant boost in deployment and usage. Due to this recent shift, we thought we’d present a subtle configuration point in Citrix Receiver that can be exploited for lateral movement across disjoint networks. More plainly, this (mis)configuration can allow an attacker that has compromised the virtual Citrix Storefront environment to compromise the systems of the users that connect to it using Citrix Receiver.
For those that aren’t familiar with Citrix Storefront, it is made up of multiple components. It is often associated with other Citrix products like Citrix Receiver\Workspace, XenApp, and XenDesktop. An oversimplification of what it provides is the ability for users to remotely access shared virtual machines or virtual applications.
To be able to remote in to these virtual environments, a user has to download and install Citrix Workspace (formerly Receiver). Upon install, the user is greeted with the following popup and the choice is stored in the registry for that user.
What we’ve found is that more often than not, end-users as well as group policy managed systems have this configuration set to “Permit All Access”. Likely because it isn’t very clear what you are permitting all access to, and whether it is necessary for proper usage of the application. I for one can admit to having clicked “Permit All Access” prior to researching what this setting actually means.
So what exactly does this setting do? It mounts a share to the current user’s drives on the remote Citrix virtual machine. If the user selects “Permit All Access”, it enables the movement of files from the remote system to the user’s shared drive.
Ok, so a user can copy files from the remote system, why is this a security issue? This is a security issue because there is now no security boundary between the user’s system and the remote Citrix environment. If the remote Citrix virtual machine is compromised, an attacker can freely write files to the connecting user’s shared drive without authentication.
Giving an attacker the ability to write files on your computer doesn’t sound that bad right? Especially if you are a low privileged user on the system. What could they possibly do with that? They could overwrite binaries that are executed by the user or operating system. A simple example of trivial code execution on Windows 10 is overwriting the OneDriveStandaloneUpdater binary that is located in the user’s AppData directory. This binary is called daily as a scheduled task.
Use the principle of least privilege when using Citrix Workspace to remote into a shared Citrix virtual environments. By default set the file security permissions for Citrix Workspace to “No Access” and only change it temporarily when it is necessary to copy files to or from the remote virtual environment. The following Citrix article explains how to change these settings in the registry. https://support.citrix.com/article/CTX133565
**Securifera is in no way affiliated, sponsored, or endorsed with/by BMC. All graphics produced are in no way associated with BMC or it’s products and were created solely for this blog post. All uses of the terms BMC, PATROL, and any other BMC product trademarks is intended only for identification purposes and is to be considered fair use throughout this commentary. Securifera is offering no competing products or services with the BMC products being referenced.
A little over 2 years ago I wrote a blog post about a red team engagement I participated in for a customer that utilized BMC PATROL for remote administration on the network. The assessment culminated with our team obtaining domain admin privileges on the network by exploiting a critical vulnerability in the BMC PATROL software. After coordinating with the vendor we provided several mitigations to the customer. The vendor characterized the issue as a misconfiguration and guidance was given to how to better lock down the software. Two years later we executed a retest for the customer and this blog post will describe what we found.
From a red teamers perspective, the BMC PATROL software can be described as a remote administration tool. The vulnerability discovered in the previous assessment allowed an unprivileged domain user to execute commands on any Windows PATROL client as SYSTEM. If this doesn’t seem bad enough, it should be noted that this software was running on each of the customer’s domain controllers.
The proposed mitigation to the vulnerability was a couple of configuration changes that ensured the commands were executed on the client systems under the context of the authenticated user.
A specific PATROL Agent configuration parameter (/AgentSetup/pemCommands_policy = “U” ) can be enabled that ensures the PATROL Agent executes the command with (or using) the PATROL CLI connected user.
Restricted mode. Only users from Administrators group can connect and perform operations (“/AgentSetup/accessControlList” = “:Administrators/*/CDOPSR”):
Unprivileged Remote Command Execution
Given the results from our previous assessment, as soon as we secured a domain credential I decided to test out PATROL again. I started up the PatrolCli application and tried to send a command to test whether it would be executed as my user or a privileged one. (In the screenshot, the IP shows loopback because I was sending traffic through an SSH port forward)
The output suggested the customer had indeed implemented the mitigations suggested by the vendor. The command was no longer executed with escalated privileges on the target, but as the authenticated user. The next thing to verify was whether the domain implemented authorization checks were in place. To give a little background here, in most Windows Active Directory implementations, users are added to specific groups to define what permissions they have across the domain. Often times these permissions specify which systems a user can login to/execute commands on. This domain was no different in that very stringent access control lists were defined on the domain for each user.
A simple way to test whether or not authorization checks were being performed properly was to attempt to login/execute commands with a user on a remote Windows system using RDP, SMB, or WMI. Next, the same test would be performed using BMC PATROL and see if the results were the same. To add further confidence to my theory, I decided to test against the most locked down system on the domain, the domain controller. Minimal reconnaissance showed the DC only allowed a small group of users remote access and required an RSA token for 2FA. Not surprisingly, I was able to execute commands directly on the domain controller with an unprivileged user that did not have the permissions to login or remotely execute on the system with standard Windows methods.
As this result wasn’t largely unexpected based on my previous research, the next question to answer was whether or not I could do anything meaningful on a domain controller as an unprivileged user that had no defined permissions on the system. The first thing that stood out to me was the absence of a writable user folder since PATROL had undermined the OS’s external access permissions. This meant my file system permissions would be bound to those set for the “Users”, “Authenticated Users”, and “Everyone” groups. To make things just a little bit harder, I discovered that a policy was in place that only allowed the execution of trusted signed executables.
Escalation of Privilege
With unprivileged remote command execution using PATROL, the next logical step was to try and escalate privileges on the remote system. As a red teamer, the need to escalate privileges for a unprivileged user to SYSTEM occurs pretty often. It is also quite surprising how common it is to find vulnerabilities that can be exploited to escalate privileges in Windows services and scheduled tasks. I spent a fair amount of time hunting for these types of bugs following research by James Forshaw and others several years back.
The first thing I usually check for when I’m looking for Windows privilege escalation bugs is if there are any writable folders defined in the current PATH environmental variable. For such an old and well known misconfiguration, I come across this ALL THE TIME. A writable folder in the PATH is not a guaranteed win. It is one of two requirements for escalating privileges. The second is finding a privileged process that insecurely loads a DLL or executes a binary. When I say insecurely, I am referring to not specifying the absolute path to the resource. When this happens, Windows attempts to locate the binary by searching the folders defined in the PATH variable. If an attacker has the ability to write to a folder in the PATH, they can drop a malicious binary that will be loaded or executed by the privileged process, thus escalating privileges.
Listing the environmental variables with “set” on the target reveals that it does indeed have a custom folder on the root of the drive in the PATH. At a glance I already have a good chance that it is writable because by default any new folder on the root of the drive is writable based on permission inheritance. A quick test confirms it.
With the first requirement for my privilege escalation confirmed, I then moved on to searching for a hijackable DLL or binary. The most common technique is to simply open up Sysinternals ProcessMonitor and begin restarting all the services and scheduled tasks on the system. This isn’t really a practical approach in our situation since one already has to be in a privileged context to be able to restart these processes and you need to be in an interactive session.
What we can do is attempt to model the target system in a test environment and perform this technique in hopes that any vulnerabilities will map to the target. The obvious first privileged service to investigate is BMC PATROL. After loading up process monitor and restarting the PatrolAgent service I add a filter to look for “NO SUCH FILE” and “NAME NOT FOUND” results. Unfortunately I don’t see any relative loading of DLLs. I do see something else interesting though.
What we’re seeing here is the PatrolAgent service executing “cmd /c bootime” whenever it is started. Since an absolute path is not specified, the operating system attempts to locate the application using the PATH. An added bonus is that the developers didn’t even bother to add an extension so we aren’t limited to an executable (This will be important later). In order for this to be successful, our writable folder has to be listed earlier in the search order than the actual path of the real bootime binary. Fortunate for me, the target system lists the writable folder first in the PATH search order. To confirm I can actually get execution, I drop a “boottime.bat” file in my test environment and watch as it is successfully selected from a folder in the PATH earlier in the search order.
So that’s it right? Time to start raining shells all over the network? Not quite yet. As most are probably aware, an unprivileged user doesn’t typically have the permissions necessary to restart a service. This means the most certain way to get execution is each time the system reboots. Unfortunately, on a server that could be weeks or longer, especially for a domain controller. Another possibility could be to try and crash the service and hope it is configured to restart. Before I capitulated to these ideas, I decided to research whether the application in its complex, robustness actually provided this feature in some way. A little googling later I came across the following link. Supposedly I could just run the following command from an adjacent system with PATROL and the remote service would restart.
pconfig +RESTART -host
Sure enough, it worked. I didn’t take the time to reverse engineer what other possibilities existed with this new “pconfig” application that apparently had the ability to perform at least some privileged operations, without authentication. I’ll leave that for PART 3 if the opportunity arises.
Combining all of this together, I now had all of the necessary pieces to again, achieve domain admin with only a low privileged domain user using BMC Patrol. I wrote “net user Administrators /add” to C:\Scripts\boottime.bat using PATROL and then executed “pconfig +RESTART -host “ to restart the service and add my user to the local administrators group. I chose to go with “boottime.bat” rather than “boottime.exe” because it provided me with privileged command execution while also evading the execution policy that required trusted signed executables. It was almost to good to be true.
Following the assessment, I reached out to BMC to responsibly disclose the binary hijack in the PatrolAgent service. They were quick to reply and issue a patch. The vulnerability is being tracked as CVE-2020-35593.
The main lesson to be learned from this example is to always be cognizant of the security implications each piece of software introduces into your network. In this instance, the customer had invested significant time and resources to lock down their network. It had stringent access controls, group policies, and multiple 2 factor authentication mechanisms (smart card and RSA tokens). Unfortunately however, they also installed a remote administration suite that subverted almost all of these measures. While there are a myriad of third party remote administration tools for IT professionals at their disposal, often times it is much safer to just use the built-in mechanisms supported by the operating system for remote administration. At least this way there is a higher probability that it was designed to properly utilize the authentication and authorization systems in place.
A 3D Printed Shell
With 3D printers getting a lot of attention with the COVID-19 pandemic, I thought I’d share a post about an interesting handful of bugs I discovered last year. The bugs were found in a piece of software that is used for remotely managing 3D printers. Chaining these vulnerabilities together enabled me to remotely exploit the Windows server hosting the software with SYSTEM level privileges. Let me introduce “Repetier-Server”, the remote 3D printer management software.
Like many of my past targets, I came across this software while performing a network penetration test for a customer. I came across the page above while reviewing screenshots of all of the web servers in scope of the assessment. Having never encountered this software before, I loaded it up in my browser and started checking it out. After exploring some of the application’s features, I googled the application to see if I could find some documentation, or better, download a copy of the software to install. I was happy to find that not only could I download a free version of the software, but they also provided a nice user manual that detailed all of the features.
In scenarios where I can obtain the software, my approach to vulnerability discovery is slightly different than the typical black-box web application. Since I had access to the code, I had the ability to disassemble/decompile the software and directly search for vulnerabilities. With time constraints being a concern, I started with the low hanging fruit and worked towards the more complex vulnerabilities. I reviewed the documentation looking for mechanisms where the software might execute commands against the operating system. Often times, simple web applications are nothing more than a web-based wrapper around a set of shell commands.
I discovered the following blurb in the “Advanced Setup” section of the documentation that describes how a user can define “external” commands that can be executed by the web application.
As I had hoped, the application already had the ability to execute system commands, I just had to find a way to abuse it. The documentation provided the syntax and XML structure for the external command config file.
The video below demonstrates the steps necessary to define an external command, load it into the application, and execute it. These steps would become requirements for the exploit primitives I needed to discover in order to achieve remote code execution.
Now that I had a feature to target, external commands, I needed to identify what the technical requirements were to reach that function. The first and primary goal was to find a way to write a file to disk from the web application. The second goal was ensuring I had sufficient control over the content of the file to pass any XML parsing checks. The remaining goals were nice to haves: a way to trigger a reboot/service restart, ability to read external command output, and file system navigation for debugging.
I started up Sysinternals Process Monitor to help me identify the different ways I could induce a file write from the web application. I then added a filter to only display file write operations by the RepetierServer.exe process.
Bug: 1 – File Upload & Download – Arbitrary Content – Constant PATH
The first file write opportunity I found was in the custom watermark upload feature in the “Global Settings – > Timelapse” menu. Process Monitor shows the RepetierServer process writes the file to “C:\ProgramData\Repetier-Server\database\watermark.png”. I had to tweak my process monitor filters because the file first gets written to a temp file called upload1.bin and then renamed to watermark.png.
Manually manipulating each of the fields, I found a couple interesting results. It appears the only security check being performed server-side is a file extension check on the filename field in the request form. This check isn’t really necessary since the destination file on disk is constant. However, I did find that the file content can be whatever I want. The web application also provided another endpoint that allows for the retrieval of the watermark file. While this isn’t immediately useful, it means if I can write arbitrary data to the watermark file location, I can read it back remotely. I’ll save this away for later in case we need it.
Bug: 2 – File Upload – Uncontrolled Content – Partially Controlled PATH (Directory Traversal), Controlled File Name, Uncontrolled Extension
Continuing with my mission of identifying file upload possibilities, I started to investigate the flow for adding a new printer to be managed by the web application. The printer creation wizard is pretty straightforward. The following video demonstrates how to create a fake printer on a Windows host running in VMware Workstation.
Based on the process monitor output, it appears that when a new printer is created, an XML file named after the printer is created in the “C:\ProgramData\Repetier-Server\configs” directory, as well as a matching directory in the “C:\ProgramData\Repetier-Server\printer” with additional subdirectories and files.
Attempting to identify the request responsible for creating the new printer in Burp proved elusive at first until I figured out that the web application utilizes websockets for much of the communication to the server. After some trial and error I identified the websocket request that creates the printer configuration file on disk.
From here I began modifying the different fields of the request to see what interesting effects might happen. Since the configuration file name mirrored the printer name, the first thing I tried was prepending a directory traversal string to the printer name in the websocket request to see if I could alter the path. Given my goal of creating an external command configuration file, I named my printer “..\\database\\extcommands”. To my surprise, it worked!!
At this point I could write to the file location necessary to load a external command, getting me substantially closer to full remote code execution. However, I still could not control the file contents. I decided to go ahead and script up a quick POC to reliably exploit the vulnerability and move on.
Bug: 3 – File Upload & Download – Partially Controlled Content – Uncontrolled PATH – Insufficient Validation
Starting from where I left off with the directory traversal bug, I began investigating ways I could try and modify the printer configuration file that I had written as the external configuration file. Luckily for me, the web application provided a feature for downloading the current configuration file or replacing it with a new one.
Coming off the high from my last bug I figured why not just try and use this feature to upload the external command configuration file for the win. Nope… Still more work to do.
Since both files were XML, I began trying different combinations of elements from each configuration file to try and satisfy whatever validation checks were happening. After spending a fair amount of time on this I just decided to open the binary up in IDA Pro and look for myself. Rather than bore you with disassembly and the tedium that followed, I’ll skip right to the end. Given a lack of full validation being performed on each element of the printer configuration file and the external command configuration file, a single XML file could be constructed that passed validation for both by including the necessary elements that were being checked when each file was parsed. This means I was able to use the “Replace Printer Configuration” feature to add an external command to our extcommands.xml file.
Bug: 4 (BONUS) – Remote File System Enumeration
Digging further into the web application, I also discovered an interesting “feature” located in the “Global Settings – > Folders” menu. The web application allows a user to add a registered folder to import files for 3D printing. The first thing I noticed about this feature is that it is not constrained to a particular folder and can be used to navigate the folder structure of the entire target file system. This can be achieved by simply clicking the “Browse” button.
Since this feature references the ability to print from locations on disk, I decided to investigate further by creating a Folder at C:\ and seeing if I could find where the Folder is referenced. After creating a printer and selecting it from the main page, a menu can be selected that looks like a triangle in the top right of the page.
When I select the Folder the following window is displayed. If I deselect the “Filter Wrong File Types” checkbox, the dialog basically becomes a remote file browser for the system. The great thing about this feature from an attacker’s perspective is it gives me the ability to confirm exploitation of the directory traversal file upload vulnerability identified earlier.
Using the vulnerabilities discovered above, I mapped out the different stages of the exploit chain that needed to be implemented. The only piece that I lacked for the exploit chain was the ability to remotely restart the ReptierServer service or the system. Since the target system was a user’s workstation, I would just have to hope that they would reboot the system at some point in the near future. This also meant that replacing the external command would be impractical since it required a service restart each time. I would need to ensure that whatever external command I created was reliable and flexible enough to support the execution of subsequent system commands. Fortunately for me, I had just the bug for this. I could use the watermark file upload & download vulnerability as a medium for storing the commands I wanted to execute, and the resulting output. The following external command achieves this goal by reading from the watermark file, executing its contents, and then piping the output to the watermark file.
Putting this all together, I came up with the following exploit flow that needed to be implemented.
I implemented each step in this python POC. The following video demonstrates it in action against my test RepetierServer installation.
After successfully testing the POC, I executed it against the target server on the customer’s network. It took ~3 days until the system was rebooted, but I was ultimately able to remotely compromise the target. When the penetration test was complete, I reached out to the vendor to report the vulnerabilities and they were quick to patch the software and release an update. I also coordinated the findings with MITRE and two CVEs were issued, CVE-2019-14450 & CVE-2019-14451.
403 to RCE in XAMPP
Some of the best advice I was ever given at how to become more successful at vulnerability discovery is to always try and dig a little deeper. Whether you are a penetration tester, red teamer, or bug bounty hunter, this advice has always proven true. Far too often it is easy to become reliant on the latest “hacker” toolsets and other peoples exploits or research. When those fail, we often just move on to the next low hanging fruit rather than digging in.
On a recent assessment, I was performing my usual network recon and came across the following webpage while reviewing the website screenshots I had taken.
The page displayed a list of significantly outdated software that was running behind this webserver. Having installed XAMPP before, I was also familiar with the very manual and tedious process of updating each of the embedded services that are bundled with it. My first step was to try and enumerate any web applications that were being hosted on the webserver. Right now my tool of choice is dirsearch, mainly just because I’ve gotten used to its syntax and haven’t found a need to find something better.
After having zero success enumerating any endpoints on the webserver, I decided to setup my own XAMPP installation mirroring the target system. The download page for XAMPP can be found here. It has versions dating all the way back to 2003. From the 403 error page we can piece together what we need to download the right version of XAMPP. We know it’s a Windows install (Win32). If we lookup the release date for the listed PHP version we can see it was released in 2011.
Based on the release date we can reliably narrow it down to a couple of candidate XAMPP installations.
After installing the software, I navigated to the apache configuration file directory to see what files were being served by default. The default configuration is pretty standard with the root directory being served out of C:\xampp\htdocs. What grabbed my attention was the “supplemental configurations” that were included at the bottom of the file.
The main thing to pay attention to in these configuration files is the lines that start with ScriptAlias as they map a directory on disk to one reachable from the web server. There are only two that show up. /cgi-bin/ and /php-cgi/. What is this php-cgi.exe? This seems awful interesting…
After a few searches on google, it seems the php-cgi binary has the ability to execute php code directly. I stumbled across an exploit that lists the version of the target as vulnerable, but it is targeting Linux instead of Windows. Since php is cross platform I can only assume the Windows version is also affected. The exploit also identifies the vulnerability as CVE-2012-1823.
Did I hit the jackpot??? Did XAMPP slide under the radar as being affected by this bug when it was disclosed? With this CVE in hand, I googled a little bit more and found an article by Praetorian that mentions the same php-cgi binary and conveniently includes a Metasploit module for exploiting it. Loading it up into metasploit, I changed the TARGETURI to /php-cgi/php-cgi.exe and let it fly. To my surprise, remote code execution as SYSTEM.
Bugs like this remind me to always keep an eye out for frameworks and software packages that are collections of other software libraries and services. XAMPP is a prime example because it has no built-in update mechanism and requires manual updates. Hopefully examples like this will help encourage others to always dig a little deeper on interesting targets.
Defcon 2020 Red Team CTF – Seeding Part 1 & 2
Last month was Defcon and with it came the usual rounds of competitions and CTFs. With work and family I didn’t have a ton of time to dedicate to the Defcon CTF so I decided to check out the Red Team Village CTF. The challenges for the qualifier ranged pretty significantly in difficulty as well as category but a couple challenges kept me grinding. The first was the fuzzing of a custom C2 server to retrieve a crash dump, which I could never get to crash (Feel free to leave comments about the solution). The second was a two part challenge called “Seeding” in the programming category that this post is about.
Connecting to the challenge service returns the following instructions:
We are also provided with the following code snippet from the server that shows how the random string is generated and how the PRNG is seeded.
The challenge seemed pretty straight forward. With the given seed and code for generating the random string, we should be able to recover the key given enough examples. The thing that made this challenge a little different than other “seed” based crypto challenges I’ve seen is that the string is constructed using random.choice() over the key rather than just generating numbers. A little tinkering with my solution script shows that the sequence of characters generated by random.choice() varies based on the length of the parameter provided, aka the key.
This means the first objective we have is to determine the length of the key. We can pretty easily determine the minimal key length by finding the complete keyspace by sampling outputs from the service until we stop getting new characters in the oracle’s output. However, this does not account for multiples of the same character in the key. So how do we get the full length of the key? We have to leverage the determinism of the sequence generated by random. If we relate random.choice() to random.randint() we see they are actually very similar except that random.choice() just maps the next number in the random sequence to an index in the string. This means if we select a key with unique characters, we should be able to identify the sequence generated by the PRNG by noting the indexes of the generated random characters in the key. It also means these indexes, or key positions, should be consistent across keys of the same length with the same seed.
Applying this logic we create a key index map using our custom key and then apply it to the sample fourth iteration string provided by the server to reveal the positions of each character in the unknown key. Assuming the key is longer than our keyspace, we will replace any unknown characters with “_” until we deduce them from each sample string.
Now we have the ability to derive a candidate key based on the indexes we’ve mapped given our key and the provided seed. Unfortunately this alone doesn’t bring us any closer to determining the unknown key length. What happens if we change the seed? If we change the seed we get a different set of indexes and a different sampling of key characters.
In the example above, you’ll notice that no characters in our derived keys conflict. This is because we know that the key length is 10, since we generated it. What happens if we try to derive a candidate key that is not 10 characters long using the generated 4th iteration random string from a 10 character key?
It appears if the length of the key used to generate the random string is not the same length as our local key, then characters in our derived keys do not match for each index. This is great news because that means we can find the server key length by incrementing our key length from the length of our key space until our derived keys don’t conflict.
Unfortunately, this is where I got stumped during the CTF. When I looped through the different key lengths I never got matching derived keys for the server key. After pouring over my code for several hours I finally gave up and moved on to other challenges. After the CTF was over I reached out to the challenge creator and he confirmed my approach was the right one. He was also kind enough to provide me with the challenge source code so I could troubleshoot my code. Executing the python challenge server and running my solution code yielded the following output.
So what gives??? Now it works??? I chalked it up to some coding mistake I must have magically fixed and decided to go ahead and finish out the solution. The next step is to derive the full server key by sampling the random output strings from different seeds. I simply added a loop around my previous code with an exit condition when there are no more underscores (“_”) in our key array. Unfortunately when I submitted the key I got an socket error instead of the flag.
Taking a look at the server code I see the author already added debugging that I can use to troubleshoot the issue. The logs show a familiar python3 error in regards to string encoding/decoding.
Well that’s an easy fix. I’ll just run the server with python3 and we’ll be back in business. To my surprise re-running my script displays the following.
This challenge just doesn’t want to be solved. Why don’t my derived keys match-up anymore? This feels familiar. Is it possible that different versions of python affect the sequences produced by random for the same seed?
Well there ya have it. Depending on the version of python you are running you will get different outputs from random for the same seed. I’m going to assume this wasn’t intentional. Either that or the author wanted to inflict some pain on all of us late adopters Finishing up the solution, and running the server and solution code with python3 finally gave me the flags.
Obligatory statement: This blog post is in no way affiliated, sponsored, or endorsed with/by Synack, Inc. All graphics are being displayed under fair use for the purposes of this article.
Over the last few months Synack has been running a user engagement based competition called Red vs Fed. As can be deduced from the name, the competition was focused on penetration testing Synack’s federal customers. For those of you unfamiliar with Synack, it is a crowd-sourced hacking platform with a bug bounty type payment system. Hackers (consultants) are recruited from all over the planet to perform penetration testing services through Synack’s VPN-based platform. Some of the key differences marketed by Synack between other bounty platforms are a defined payout schedule based on vulnerability type and a 48 hour triage time.
Red Vs Fed
This section is going to be a general overview of my experience participating in my first hacking competition with Synack, Red Vs Fed. At times it may come off as a diatribe so feel free to jump forward to the technical notes that follow. In order for any of this to make sense we first have to start off with the competition rules and scoring. Points were awarded per “accepted” vulnerability based on the CVSS score determined by Synack on a scale of 1-10. There were also additional multipliers added once you passed a certain number of bugs accepted. The important detail here is the word “accepted”, which means you have to pass the myriad of exceptions, loopholes, and flat-out dismissal of submitted bugs as it goes through the Synack triage process. The information behind all of these “rules” is scattered across various help pages accessible by red team members. Example of some of these written, unwritten, and observed rules that will be referenced in this section:
- Shared code: If a report falls within about 10 different permutations of what may be guessed as shared code/same root issue, the report will be accepted at a substantially discounted payout or forced to be combined into a single report.
- 24 hour rule: In the first 24 hours of a new listing, any duplicate reports will be compared by triage staff and they will choose which report they feel has the highest “quality“. This is by far the most controversial and abused “feature” as it has led to report stuffing, favoritism, and even rewards given after clear violation of the Rules of Engagement.
- Customer rejected: Even though vulnerability acceptance and triage is marketed as being performed internally by Synack experts within 48 hours, randomly some reports may be sent to the customer for determination of rejection.
- Low Impact: Depending on the listing type, age of the listing, or individual triage staff, some bugs will be marked as low impact and rejected. This rule is specific to bugs that seem to be some-what randomly accepted on targets even though they all fall into the low impact category.
- Dynamic Out-of-Scope: Bug types, domains, and entire targets can be taken out of scope abruptly depending on report volume. There are loopholes to this rule if you happen to find load balancers or CNAME records for the same domain targets.
- Target Analytics: This is a feature not a rule but it seemed fitting for this list. When a vulnerability is accepted by Synack on a target, details like the bug type and location are released to all currently enrolled on the target.
A bug hunter can’t ask for an easier SQL injection vulnerability, verbose error messages, the actual SQL query with database names, tables names, and column names. I whipped up a POC, (discussed further down) and starting putting together a report for the bug. After a little more recon, I was able to uncover and report a handful of additional SQLi vulns over the next couple days. While continuing to discover and “prove” SQLi on more endpoints, I was first introduced to (5) as the entire domain was taken out of scope for SQLi. No worries, at least my bugs were submitted so no one else should be able to swoop in and cleanup my leftovers. Well not exactly, it just so happened that there was a load balancer in front of my target as well as a defined CNAME (alias) for my domain. This meant thanks to (6) another competitor saw my bugs, knew about this loophole and was able to submit an additional 5 SQLi on this aliased domain before another dynamic OOS was issued.
At this point, the race was on to report as many vulnerabilities on this new web application now that my discovery was now public to the rest of the competitors via analytics. I managed to get in a pretty wide range of vulnerabilities on the target but they were pretty well split with the individual that was already in second place, putting him into first place. With this web application pretty well picked through I began looking for additional vhosts on new subdomains in hopes of finding similar vulnerable applications. I also began to strategize about how best to submit vulnerabilities in regards to timing and grouping given my last outcome realizing these nuances could be the difference between other competitors scooping them up, Synack taking them out of scope, or forcing reports to be combined.
A couple weeks later I caught a lucky break. I came across a couple more web applications on a different subdomain that appeared to be using the same web technologies as my previous find and hopefully similarly buggy code. As I started enumerating the targets the bugs started stacking up. This time I took a different approach. Knowing that as soon as my reports hit analytics it would be a feeding frenzy amongst the other competitors, I began queuing up reports but not submitting them. After I had exhausted all the bugs I could find, I began to submit the reports in chunks. Chunks sized close to what I expected the tolerance for Synack to issue the vulnerability class OOS but also in small enough windows that hopefully other competitors wouldn’t be able to beat me to submission by watching analytics. Thankfully, I was able to slip in all of the reports just before the entire parent domain was taken OOS by Synack.
Back to the grind… After the last barrage of vulnerability submissions I managed to get dozens of websites taken out of scope so I had to set my sights on a new target. Luckily I had managed to position myself in first place after all of the reports had been triaged.
Nothing new here, lots of recon and endpoint enumeration. I began brute-forcing vhosts on domains that were shown to host multiple web applications in an attempt to find new ones. After some time I stumbled across a new web application on a new vhost. This application appeared to have functionality broken out into 3 distinct groupings. Similar to my previous finds I came across various bug types related to access control issues, PII disclosure, and unauthorized data modification. I followed my last approach and began to line-up submissions until I had finished assessing the web application. I then began to stagger the submissions based on the groupings I had identified.
Unfortunately things didn’t go as well this time. The triage time started dragging on so I got nervous I wouldn’t be able to get all of my reports in before the target was put out of scope. I decided to go ahead and submit everything. This was a mistake. When the second group of reports hit, the triage team decided that they considered all 6 of the reports stemmed from an access control issue. They accepted a couple as (1) shared code for 10% payout, then pushed back a couple more to be combined into one report. I pleaded my case to more than one of the triage team members but was overruled. After it was all said and done they distilled dozens of vulnerabilities across over 20 endpoints into 2 reports that would count towards the competition and then put the domain OOS (5). Oh well… I was already in first so I probably shouldn’t make a big stink right?
Over the next month it would seem I was introduced to pretty much any way reports could be rejected unless it was a clear CVSS 10 (This is obviously an exaggeration but these bug types are regularly accepted on the platform).
1. User account state modification (Self Activation/Auth Bypass, Arbitrary Account Lockout) – initially rejected, appealed it and got it accepted
2. PII Disclosure – Rejected as (3). Claimed as customer won’t fix
3. User Login Enumeration on target with NIST SP 800-53 requirement – Rejected as (3) and (4) – Even though a email submission captcha bypass was accepted on the same target… Really???
4. Phpinfo page – Rejected as (4) even though these are occasionally accepted on web targets
5. Auth Bypass/Access Control issue – Endpoint behind a site’s login portal allowed for the live viewing of CC feeds – Rejected as (4) with the following
6. Unauthenticated, On-demand reboot of network protection device – Rejected as (3) even though the exact same vulnerability had been accepted on a previous listing literally 2 days prior with points awarded toward the competition. The dialog I had with the organizer about the issue below:
At this point it really started to feel like I was fighting a unwinnable battle against the Synack triage team. 6 of my last 8 submissions had been rejected for some reason or another and the other 2 I had to fight for. Luckily the competition would be over shortly and there were very few viable targets eligible for the competition. With 2 days remaining in the competition I was still winning by a couple bugs.
I wish I could say everything ended uneventfully 2 days later but what good Synack sob story doesn’t talk about the infamous 24 hour rule (2). To make things interesting a web target was released roughly 2 days before the end of the competition. I mention “web” because the acceptance criteria for “low impact” (4) bugs is typically lowered for these assessment types which means it could really shake up the scoreboard. Well here we go…
I approached the last target like a CTF, only 48 more hours to hopefully secure the win. Ironically, the last target had a bit of a CTF feel to it. After navigating the application a little, it became obvious that the target was a test environment that had already undergone similar pentests. It was littered with persistent XSS payloads that had been dropped from over a year prior. These “former” bugs served as red herrings as the application was no longer vulnerable to them even though the payloads persisted. I presume they also served as a time suck for many as under the 24 hour rule (1) any red teamer could win the bug if their report was deemed the “best” submission. Unfortunately however, with only a few hours into the test the target became very unstable. The server didn’t go down but it became nearly impossible to login to, regularly returning an unavailable message as if being DOS-ed. Rather than continue to fight with it I hit the sack with hopes that it would be up in the morning.
First thing in the morning I was back on it. With only 12 hours left in the 24 hr rule ( or so I thought) I needed to get some bugs submitted. I came a across an arbitrary file upload with a preview function that looked buggy.
The file preview endpoint contained an argument that specified the filename and mimetype. Modifying these parameters changed the response Content-Type and Content-Disposition (whether the response is displayed or downloaded) headers.
I got the vulnerability written up and submitted before the end of the 24 hour rule. Since persistent XSS has a higher payout and higher CVSS score on Synack, I chose this vulnerability type rather than arbitrary file upload. Several hours after my submission, an announcement was made that due to the unavailability of the application the night previous ( possible DOS) that the 24 hour rule would be extended 12 hours. Well that sux… that means I will now be competing with more reports of possibly better quality for the bug I just submitted because of the extension. Time to go look for more bugs and hopefully balance it out.
After some more trolling around, I found an endpoint that imported records into a database using a custom XML format. Features like this are prime for XXE vulnerabilities. After some testing I found it was indeed vulnerable to blind XXE. XXE bugs are very interesting because of the various exploit primitives they can provide. Depending on the XML parser implementation, the application configuration, system platform, and network connectivity, these bugs can be used for arbitrary file read, SSRF, and even RCE. The most effective way to exploit XXEs is if you have the XML specification for the object being parsed. Unfortunately, for me the documentation for the XML file format I was attacking was behind a login page that was out-of-scope. I admit I probably spent too much time on this bug as the red teamer in me wanted RCE. I could get it to download my DTD file and open files but I couldn’t get it to leak the data.
Since I couldn’t get LFI working, I used the XXE to perform a port scan on the internal server to at least ensure I could submit the bug as SSRF. I plugged in the payload to Burp Intruder and set it on its way to enumerate all the open ports.
After I got the report written up for SSRF I held off on submission in hopes I could get the arbitrary file read (LFI) to work and maybe pull off remote code execution. Unfortunately about this time the server started to get unstable again. For the second night in a row it appeared as if someone was DOS-ing the target and it was between crawling and unavailable. Again I decided to go to bed in hopes by morning it would be usable.
Bright and early I got back to it, only 18 hours left in the competition and the target was up albeit slow. I spent a couple more hours on my XXE but with no real progress. I decided to go ahead and submit before the extended 24 hour rule expired to again hopefully ensure at least a chance at winning a possible “best” report award. Unsurprisingly, a couple hours later the 24 hour rule was extended again (because of the DOS) with it ending shortly before the end of the competition. On this announcement I decided I was done. While the reason behind the extension made sense, this could effectively put the results of the competition in the hands of the triage team as they arbitrarily chose best reports for any duplicated submissions.
The competition ended and we awaited the results. Based on analytics and my report numbers I determined that the bug submission count on the new target was pretty low. As the bugs started to roll in, this theory was confirmed. There were roughly 8 or so reports after accounting for combined endpoints. My XXE got awarded as a unique finding and my persistent XSS got duped under the 24 hour rule that was extended to 48. Shocking, extra time leads to better reports. Based on the scoreboard, the person in second must have scored every other bug on the board except 1, largely all reflective XSS. For those, familiar with Synack, this would actually be quite a feat because XSS is typically the most duped bug on the platform meaning they likely won best report for several other collisions. I’d like to say good game but certainly didn’t feel like it.
Technical Notes of Interest
The techniques used to discover and exploit most of the vulnerabilities I found are not new. That said, I did learn a few new tricks and wrote a couple new scripts that seemed worth sharing.
SQL injections bugs were my most prevalent find in the competition. Unlike other bug bounty platforms, Synack typically requires full exploitation of vulnerabilities found for full payout. With SQL injection, typically database names, columns, tables, and table dumps are requested to “prove” exploitation rather than basic injection tests or SQL errors. While I was able to get some easy wins with sqlmap on several of the endpoints, some of the more complex queries required custom scripts to dump data from the databases. This necessity produced two new POCs I will describe below.
In my experience a large percentage of modern SQL injection bugs are blind. By blind, I’m referring to SQLi bugs that are not able to return data from the database in the request response. That said, there are also different kinds of blind SQLi. There are bugs that return error messages and those that do not ( completely blind). The first POC is an example of a HTTP GET, time-based boolean exploit for a completely blind SQL injection vulnerability on an Oracle database. The main difference between this POC and previous examples in my github repository is the following SQL query.
The returned table is two columns of type string and this query performs a boolean test against a supplied character limit, if the test passes, then the database will sleep for a specified timeout. The response time is then measured to verify the boolean test.
The second POC is an example of a HTTP GET, error-based boolean exploit for a partially blind SQL injection vulnerability on an Oracle database. This POC is targeting a vulnerability that can be forced to produce two different error messages.
The error message used for the boolean test is forced by the payload in the event that the “UTL_INADDR.get_host_name” function is disabled on the target system. Both of the POCs extract strings from the database a character at a time using the binary search algorithm below.
Remote Code Execution (Unrestricted File Upload)
Probably one of the more interesting of my findings was a file extension blacklist bypass I found for a file upload endpoint. This particular endpoint was on a ColdFusion server and appeared to have no allowedExtensions defined as I could upload almost any file types. I say almost because a small subset of extensions were blocked with the following error.
Further research found that the ability to upload arbitrary file types was allowed up until recently when someone reported it as a vulnerability and it was issued CVE-2019-7816. The patch released by Adobe created a blacklist of dangerous file extensions that could no longer be uploaded using cffile upload.
This got me thinking, where there is a blacklist, there is the possibility of a bypass. I googled around for a large file extension list and loaded it up into Burp Intruder. After burning through the list I reviewed the results to see a 500 error on the “.ashx” extension with a message indicating the content of my file was being executed. A little googling later, I replaced my file with this simple ASHX webshell from the internet and passed it a command. Bingo.
Overall the competition proved to be a rewarding experience, both financially and academically. It also helped knowing that each vulnerability found was one less that could be used to attack US federal agencies. The prevailing criticism I have for the competition is that it was unfortunate that winning became more about exploiting the platform, its policies, and people more than finding vulnerabilities. In CTF this isn’t particularly unusual for newer contests which seem to forget (or not care) about the “meta” surrounding the competition itself. Hopefully, in future competitions some of the nuanced issues surrounding vulnerability acceptance and payout can be detached from the competition scoring. My total tally of vulnerabilities against Federal targets is displayed below.
|Bug Type||Count||CVSS (Competition Metric)|
|Remote Code Execution||6||10|
|Local File Include||2||8|
On June 8th it was announced that I had taken 2nd place in the competition and won $15,000 in prize money.
Given the current global pandemic, we decided to donate the prize money to 3 organizations who focus on helping the less fortunate: Water Mission, Low Country Foodbank, and Build Up. For any of you reading this that may have a little extra in these trying times, consider donating to your local charities to support your communities.
A Year of Windows Privilege Escalation Bugs
Earlier last year I came across an article by Provadys (now Almond) highlighting several bugs they had discovered based on research by James Forshaw of Google’s Project Zero. The research focused on the exploitation of Windows elevation of privilege (EOP) vulnerabilities using NTFS junctions, hard links, and a combination of the two Forshaw coined as Windows symlinks. James also released a handy toolset to ease the exploitation of these vulnerabilities called the symbolic testing toolkit. Since they have done such an excellent job describing these techniques already, I won’t rehash their inner workings. The main purpose of this post is to showcase some of our findings and how we exploited them.
My initial target set was software covered under a bug bounty program. After I had exhausted that group I moved on to Windows services and scheduled tasks. The table below details the vulnerabilities discovered and any additional information regarding the bugs.
|Vendor||Arbitrary File||ID||Date Reported||Reference||Reward|
Before I get started I want to clearly state that I am in no way affiliated, sponsored, or endorsed with/by Palo Alto Networks. All graphics are being displayed under fair use for the purposes of this article.
I recently encountered several unpatched Palo Alto firewall devices during a routine red team engagement. These particular devices were internet facing and configured as Global Protect gateways. As a red teamer/bug bounty rookie, I am often asked by customers to prove the exploitability of vulnerabilities I report. With bug bounty, this will regularly be a stipulation for payment, something I don’t think is always necessary or safe in production. If a vulnerability has been proven to be exploitable by the general security community, CVE issued, and patch developed, that should be sufficient for acceptance as a finding. I digress…
The reason an outdated Palo Alto Global Protect gateway caught my eye was because of a recent blog post by DEVCORE team members Orange Tsai(@orange_8361) and Meh Chang(@mehqq_). They identified a pre-authentication format string vulnerability (CVE-2019-1579) that had been silently patched by Palo Alto a little over a year ago (June 2018). The post also provided instructions for safely checking the existence of the vulnerability as well as a generic POC exploit.
Virtual vs Physical Appliance
If you’ve made it this far, you’re probably wondering why there’s need for another blog post if DEVCORE already covered things. According to the post, “the exploitation is easy”, given you have the right offsets in the PLT and GOT, and the assumption that the stack is aligned consistently across versions. In reality however, I found that obtaining these offsets and determining the correct instance type and version to be the hard part.
Palo Alto currently markets several next generation firewall deployments, that can be broadly categorized into physical or virtual. The focus of the exploitation details in the article are based on a virtual instance, AWS in this scenario. How then do you determine whether or not the target device you are investigating is virtual or physical? In my experience, one of the easy ways is based on the IP address. Often times companies do not setup reverse DNS records for their virtual instances. If the IP/DNS belongs to a major cloud provider, then chances are it’s a virtual appliance.
If you determine that the firewall type is an AWS instance, then head over to AWS marketplace and spin up a Palo Alto VM. One of the first things you’ll notice here is that you are limited to only the newest releases of 8.0.x, 8.1.x, 9.0.x.
Don’t worry, you can actually upgrade and downgrade the firmware from the management web interface once it has been launched… if you have a valid support license for the appliance. There are some nuances however, if you launch 9.0.x, it can only be downgraded to 8.1.x. Another very important detail is to ensure you select a “m4” instance or when you downgrade to 8.x.x or the AWS instance will be unreachable and thus unusable. For physical devices, the firmware supported can be found here
Getting ahold of a valid support license varies in difficulty and price. If you are using the AWS Firewall Bundle, the license is included. If it’s a physical device, it can get complicated/expensive. If you buy the device new from an authorized reseller, you can activate a trial license. If you buy one through something like Ebay and it’s a “production” device, make sure the seller transfers the license to you otherwise you may have to pay a “recertification” fee. If you get really lucky and the device you buy happens to be a RMA-ed device, when you register it it will show as a spare and you get no trial license, you have to pay a fee to get it transferred to production, and then you have to buy a support license.
Once you have the appliance up and running with the version you want to test, you will need to install a global protect gateway on one of the interfaces. There’s a couple youtube videos out there that go over some of the installation steps but you basically just have to step your way through it until it works. Palo Alto provides some documentation that you can use as a reference if you get stuck. One of the blocking requirements is to install a SSL certificate for the Global Protect gateway. The easiest thing here is to just generate a self signed certificate and import it. A simple guide can be found here. Supposedly one can be generated on the device as well.
If you are setting up an AWS instance, you will need to change a key setting on the network interface or the global protect gateway will not be accessible. Goto the network interface that is being used for the Global Protect interface in AWS. Right click and select “Change Source/Dest Check”. Change the value to “Disabled” as shown below.
Alright, the vulnerable device firmware is installed and the global protect gateway is configured, we’re at the the easy part right??? Well not exactly… When you SSH into the appliance you’ll find you are in a custom restricted shell that has very limited capabilities.
In order to get those memory offsets for the exploit we need access to the sslmgr binary. This is going to be kinda hard to pull off in a restricted shell. Previous researchers found at least one way, but it appears to have been fixed. If only there was another technique that worked, then theoretically you could download each firmware version, copy it from the device, and retrieve the offsets.
What do we do if we can’t find such a jailbreak… for every version??? Well it turns out we may be able to use some of the administrative functions of the device and the nature of the vulnerability to help us. One of the features that the limited shell provides is the ability to increase the verbosity of the logs generated by key services. It also allows you to tail the log files for each service. Given that the vulnerability is a format string bug, could we leak memory into the log and then read it out? Let’s take a look at the bug(s).
And immediately following, is the exact code we were hoping for. It must be our lucky day.
So as long as we populate those four parameters, we can pass format string operators to dump memory from the process to the log. Why is this important? This means we can dump the entire binary from memory and retrieve the offsets we need for the exploit. Before we can do that, first we need to identify the offsets to the buffers on the stack for each of our parameters. I developed a script that you can grab from Github that will locate the specified parameters on the stack and print out the associated offsets.
The script should output something like the screenshot below. Each one of these offsets points to a buffer on the stack that we control. We can now select one and populate it with whatever memory address we like and then use the %s format operator to read memory at that location.
As typically goes, there are some problems we have to work around to get accurate memory dumps. Certain bad characters will cause unintended output: \x00, \x25, and \x26.
Null bytes cause a problem because sprintf and strlen recognize a null byte as the end of a string. A work around is to use a format string to point to a null byte at a known index, e.g. %10$c.
The \x25 character breaks our dump because it represents the format string character %. We can easily escape it by using two, \x25\x25.
The \x26 character is a little trickier. The reason this character is an issue is because it is the token for splitting the HTTP parameters. Since there aren’t a prevalence of ampersands on the stack at known indexes, we just write some to a known index using %n, and then reference it whenever we encounter an address with a \x26.
Putting this all together, I modified my previous script to write a user-supplied address to the stack, deference it using the %s format operator, and then output the data at that address to the log. Wrapping this logic in a loop combined with our handling of special characters allows us to dump large chunks of memory at any readable location. You can find a MIPS version of this script on our Github and executing it should give you output that looks something like the screenshot below.
Now that we have the ability to dump arbitrary memory addresses, we can finally dump the strlen GOT and system PLT addresses we need for the exploit, easy… except, where are they??? Without the sslmgr binary how do we know what memory address to start dumping to get the binary? We have a chicken or the egg situation here and the 64 bit address space is pretty big.
Luckily for us, the restricted shell provides us one more break. If a critical service like sslmgr crashes, a stack trace and crash dump can be exported using scp. At this point I’ve gotten pretty good at crashing the service so we’ll just throw some %n format operators at arbitrary indexes.
I learned something new about IDA Pro during this endeavor. You can use it to open up ELF core dumps. Opening up the segmentation view in IDA we could finally see where the binary is being loaded in memory. Another interesting detail we noticed while debugging our payloads was that it appeared ASLR was disabled on our MIPS physical device as the binary and loaded libraries were always loaded at the same addresses.
Finally, let’s start dumping the binary so we can get our offsets. We have roughly 0x40000 bytes to dump, at approximately 4 bytes/second. Ummm, that’s going to take… days. If only there was a shortcut. All we really need are the offsets to strlen and system in the GOT and PLT.
Unfortunately, even if we knew exactly were the GOT and PLT were, there’s nothing in the GOT or PLT that indicates the function name. How then does GDB or IDA Pro resolve the function names? It uses the ELF headers. We should be able to dump the ELF headers and a fraction of the binary to resolve these locations. After an hour or two, I loaded up my memory dump into Binary Ninja, (IDA Pro refused to load my malformed ELF). Binary Ninja has proven to be invaluable when analyzing and manipulating incomplete or corrupted data.
A little bit of research on ELF reveals that the PT_DYNAMIC program header will hold a table to other sections that hold pertinent information about an executable binary. Three of which are important to us, the Symbol Table (DT_SYMTAB-06), the String Table (DT_STRTAB-05), and the Global Offset Table (DT_PLTGOT-03). The Symbol Table will list the proper order of the functions in the PLT and GOT sections. It also provides an offset into the String Table to properly identify each function name.
With the offsets to the SYMBOL and STRING tables we can properly resolve the function names in the PLT & GOT. I wrote a quick and dirty script to parse through the symbol table dump and output a reordered string table that matches the PLT. This probably could have been done using Binary Ninja’s API, but I’m a n00b. With the symbol table matched to the string table, we can now overlay this with the GOT to get the offsets we need for strlen and system. We have two options for dumping the GOT. We can either use the crash dump from earlier, or manually dump the memory using our script.
I decided to go with the crash dump to save me a little time. The listing above shows entries in the GOT that point to the PLT (0x1001xxxx addresses) and several library functions that have already been resolved. Combined with our string table we can finally pull the correct offsets for strlen and system and finalize our POC. Our POC is for the MIPS based physical Palo Alto appliance but the scripts should be useful across appliance types with minor tweaking. Wow, all done, easy right?
While the sslmgr service has a watchdog monitor to restart the process when it crashes, if it crashes more than ~3 times in a short amount of time, it will not restart and will require a manual restart. Something to keep in mind if you are testing your shiny new exploit against a customer’s production appliance.
There are no more articles