Target Audience

The target audience for this blog post is individuals who have a basic understanding of cybersecurity concepts and terminology and looking to expand their knowledge on adversary emulation. This post delves into the details of adversary emulation with the Caldera framework exploring the benefits it offers. By catering to a beginner to intermediate audience, the blog post aims to strike a balance between providing fundamental information for newcomers and offering valuable insights and techniques that can benefit individuals who are already familiar with the basics of cybersecurity.

What is Adversary Emulation

Adversary emulation is a methodology used to simulate the Tactics, Techniques, and Procedures (TTPs) used by known Advanced Persistent Threats (APTs), with the goal of identifying vulnerabilities in an organization’s security defenses. By emulating real-world attacks and incident response techniques, such as exploitation of vulnerabilities and lateral movement within a network, cybersecurity teams can gain a better understanding of their security posture and identify areas for improvement.

The Need for Adversary Emulation

Adversary emulation can help organizations test their security defenses against real-world threats. Some of the benefits the emulation offers are:

• Identifying vulnerabilities: Adversary emulation assists organizations in identifying vulnerabilities, weaknesses or misconfigurations in their security defenses that might not have been detected through conventional security testing. This information can enhance the existing detection mechanisms by creating new alerts and rules that are triggered when similar activities are detected. The emulation results can also work as a guide in prioritizing mitigation and patching activities.
• Improving security controls: By identifying weaknesses in their security defenses, organizations can make informed decisions about how to improve their security controls. This can include implementing new security technologies, updating security policies, or providing additional security awareness training to employees.
• Measuring security effectiveness: Adversary emulation enables organizations to assess the effectiveness of their security defenses within a controlled environment. Through analyzing the emulation results, organizations can have a clearer understanding of how well their incidence response plan operates in real-world scenarios.If any gaps or inefficiencies are identified, the plan can be refined based on the new data.
• Staying ahead of emerging threats: Adversary emulation exercises can help organizations stay ahead of emerging threats by testing their security defenses against new and evolving attack techniques. This can help organizations prepare for future threats and ensure that their security defenses are effective in protecting against them.

Emulation VS Simulation

Emulation involves creating a replica of a specific system or environment, such as an operating system, network, or application. It provides a more realistic testing environment, which can help identify vulnerabilities and test the effectiveness of security controls in a more accurate and reliable way. However, creating an emulation environment can be time-consuming and resource-intensive, and it may not always be feasible to replicate every aspect of a real-world environment.

Simulation, on the other hand, involves creating a hypothetical scenario that models a real-world attack. It is often quicker and easier to set up, and can be used to test response plans and procedures without the need for a complex emulation environment. However, simulations may not always provide a completely accurate representation of a real-world attack scenario, and the results may be less reliable than those obtained through emulation.

The Caldera Framework

MITRE’s Caldera project is an open-source platform that allows organizations to automatically emulate the tactics, techniques, and procedures (TTPs) used by real-world APTs. The platform is designed to be modular, which means that it can be customized to fit the specific needs of an organization. More information can be found in the official documentation and on GitHub. Red team operators can benefit from this by manually executing TTPs and blue team operators can run automated incident response actions. Caldera is also highly extensible, meaning that it can be integrated with other security tools to provide a comprehensive view of an organization’s security defenses. Moreover, it is built on the MITRE ATT&CK framework which is where the platform draws all the Tactics, Techniques and Procedures (TTPs) from.

Most common use cases of this framework include but not limited to:

• Autonomous Red Team Engagements: This case is used to emulate the TTPs of known adversary profiles to discover gaps across an infrastructure, test the defenses currently in place and train operators on detection different threats.
• Manual Red Team Engagements: This case allows red team operators to replace or extend the attack capabilities of a scenario, giving them more freedom and control over the current emulation.
• Autonomous Incident Response: This case is used by blue team operators to perform automated incident response actions to aid them in identifying TTPs and threats that other security solutions may not detect and/or prevent.

Caldera consists of two main components:

• The core system, which is the framework’s code, including an asynchronous command-and-control (C2) server with a REST API and a web interface.
• Plugins which are separate repositories that expand the core framework capabilities and provide additional functionality. Examples include agents, GUI interfaces, collections of TTPs and more.

In Figure 1 below, we are greeted with when we login either as user red or blue and some basic terminology.

1. Agents: An agent is another name for Remote Access Trojan (RAT). These programs written in any language, execute an adversary’s instructions on compromised systems (victims). Often, an agent will communicate back to the adversary’s server through an internet protocol, such as HTTP, UDP or DNS. Agents also beacon into the C2 on a regular basis, asking the adversary if there are new instructions. If a beacon misses a regularly scheduled interval, there is a chance the agent itself has been discovered and compromised.
2. Abilities: An ability is a specific set of instructions to be run on a compromised host by an agent immediately after sending the first beacon in.
3. Adversaries: Adversary profiles are groups of abilities, representing the tactics, techniques, and procedures (TTPs) of known real-world APT groups. Adversary profiles are used when running an operation to determine which abilities will be executed.
4. Operations: An operation is an attack scenario which uses the TTPs of pre-configured adversary profiles. An operation can be run automatically where the agents and the C2 server run without the operator’s interference and can only run tasks in the adversary profile. On the other hand, there is the manual mode where the operator approves every command before it is tasked to an agent and executed. Additionally in manual mode the operator can add extra TTPs. In order to run an operation at least one agent must be active.
5. Plugins: They provide additional functionality over the usage of the framework.

Configuring an Agent

When we select “agents” from the figure 1 menu above, we are greeted with the figure 2 page.

If we select the “Configuration” button, a new window opens where we can configure different options for all the agents created afterwards.

• Beacon Timer(s) = This fields sets the minimum and maximum amount of seconds the agent will take to beacon back home.
• Watchdog Timer(s) = This field sets the number of seconds an agent has to wait, if the server is unreachable, before it is killed.
• Untrusted Timer(s) = This field sets the number of seconds the server has to wait before marking a missing or unresponsive agent as untrusted. Furthermore, operations will not generate new links or send new instructions to untrusted agents.
• Implant Name = This field sets the name for the newly created agents.
• Bootstrap Abilities = This is a list of abilities to be run when a new agent beacons back to the server. By default, it runs a command which clears the command history.
• Deadman Abilities = This is a list of abilities to be run immediately before an agent is killed.

To deploy an agent, we can press the “Deploy an Agent” button and we are greeted with this page. For this example, the agent Sandcat will be used.

By deploying the agent we refer to the process of installing and setting up the agent on the target system to enable it to perform specific actions or functions such as: monitoring, management, data collection, exploitation, reconnaissance and many more.

In figure 4, we can select the agent we want to deploy.

Next, in figure 5, we have to select the operating systems the agent will be deployed on.

In this example, the Linux operating system has been chosen and Caldera provides us with some options and some pre-built commands. These commands can be copied and run directly to the victim’s terminal for the agent to be deployed. There are different variations for the deployment of the selected agent such as:

• It can be deployed as a red or blue agent.
• It can be downloaded with a random name and start as a background process.
• It can be deployed as a peer-to-peer (P2P) agent with known peers included in the compiled agent.

Moreover, the settings that can be modified are:

• app.contact.http = This field is where the URL of the server’s address can be specified.
• agents.implant_name = This field represents the name of the agent binary.
• agent.extensions = This field takes a list of agent extensions to compile with the binary.

After an agent has been deployed it will be shown in the agent’s window, as illustrated in Figure 7.

If an agent is selected, a new window opens that shows some settings that can be modified along with some information about the system the agent is installed on and a kill switch, as shown in figure 8.

• Contact = This field specifies the protocol in which the agent will communicate with the server.
• Sleeper Timer = This is the same as the Beacon Timer(s).

Configuring an Adversary Profile

Caldera comes with pre-defined profiles to choose from, loaded with known TTPs. There is also the option to create a new profile with mixed TTPs, providing an operator more flexibility over the operation. An adversary profile can be created and configured in the “adversaries” window as shown below in figure 9.

After the “New profile” button is pressed, a name and a description for the new adversary profile will be asked.

A new ability can be added to the newly created profile by pressing the “add Ability” button.

Then a new window will open where the specific ability can be chosen and configured, as depicted in figure 11.

Here an already existing ability can be added by searching for it in the search bar or a new one can be configured by choosing a specific Tactic, Technique and Ability as shown above, along with all the details shown in the “Ability Details” section.

This newly created ability can be added to the TTPs of an already existing adversary profile by pressing the “Add Adversary” button. A new window will open to choose the appropriate profile.

Finally, by pressing the “Save Profile” button the new profile is created and can be added to an operation.

Configuring an Operation

An operation can be created and configured in the “operations” window.

After that a new window will open with all the modifiable settings.

• Operation Name = Specifies the name of the operation.
• Adversary = Specifies a specific adversary profile to emulate along with the pre-configured TTPs associated with this profile.
• Fact Source = In this field a fact source can be attached to the current operation. This means that the operations will start with some knowledge of the facts which can be used to fill in different variable inside some abilities. A fact is identifiable information about the target machine that can be used by some abilities, such as usernames, passwords, hostname etc.
• Group = Specifies the collection of agents to run against
• Planner = Specifies which logic library to use for the current operation. A planner is a Python module which contains logic that allows a running operation to make decisions about which abilities to use and in what order. The default planner is the “Atomic” which sends a single ability command to each agent in a group at a time. The order in which the commands are sent is the same as in the adversary’s profile.
• Obfuscators = This field specifies which obfuscator to use to encode each command before they are sent to the agents. The available options are:
  • Base64 = Encodes the commands in base64
  • Base64jumble = Encodes the commands in base64 and then adds characters
  • Base64noPadding = Encodes the commands in base64 and then removes padding
  • Caesar cipher = Obfuscates the commands with the Caesar cipher algorithm
  • Plain text = No obfuscation
  • Steganography = Obfuscates the commands with image-based steganography
• Autonomous = Specifies if the operations will run autonomously or manually. In manual mode the operator will have to approve each command.
• Parser = Parsers are Python modules that are used to extract facts from command outputs. For instance, some reconnaissance commands can output file paths, usernames, passwords, shares etc. these facts can then be fed back into future abilities. Parsers can also be used to create facts with relationships between them, such as username and password facts.
• Auto-close = This option automatically terminates the operation when there are no further actions left. Alternatively, it keeps the operation open until the operator terminates it manually.
• Run state = This option pauses the operation on start or runs immediately
• Jitter = Specifies the minimum and maximum number of seconds the agents will check in with the server while they are part of an active operations.
• Visibility = This option specifies how visible should the operation be to the defense. Abilities with higher visibility than the operation’s will be skipped.

After the “start” button is pressed the operation will start and the results will be shown on the screen whether each task fails or succeeds. There is also the option to view each command and its result, as illustrated in figure 16.

This was a red team operation, but in order to see the full picture some security solutions should also be running on the target systems to examine what was prevented and what went undetected.

Configure Automated Incident Response Plan

To form an incident response plan the “blue” user must be logged in.

The blue team’s main menu is a little different than the red team’s one. The main change is the “response plugin” which is a counterpart of the threat emulation plugins. At the time of writing this blog post, it contains 37 abilities and 4 defender profiles that focus on detection and response actions.

In the “Defenders” tab a new custom defender profile can be created and configured with the same way as the adversaries profile.

The profiles included in this plugin are:

• Incident Responder
• Elastic Hunter
• Query Sysmon
• Task Hunter

All available abilities for each defender profile can be viewed in the “abilities” section, after the specific profile has been chosen from the “response” tab, as shown in figure 17.

Defender abilities are classified by four different tactics:

• Setup: These abilities prepare information to be used by other abilities
• Detect: These abilities focus on finding suspicious behavior by continuously monitoring the ingested information and run as long as the operation is active.
• Response: These abilities act autonomously once suspicious is detected. Such actions include, killing a process, modifying firewall rules, deleting of a file and so on.
• Hunt: These abilities focus on searching for Indicators of Compromise (IOCs) via logs or file hashes.

Blue team operations are configured the same way as the red team operations. The main difference in the procedure is that the agent must be deployed as blue instead of red, in the “adversary” option a defender profile must be selected and in the Fact source section the “response” option must be selected.

The result structure is the same as the red team operation. The commands and their output are shown and whether they were successful or not.

Conclusion

In conclusion, leveraging the Caldera framework for adversary emulation presents a robust and proactive approach to enhancing cybersecurity defenses. Through the simulation of real-world attack scenarios, organizations can acquire invaluable insights into potential vulnerabilities and subsequently strengthen their incident response capabilities. The flexibility, modularity, and extensibility of Caldera establish it as an ideal tool for executing sophisticated emulation exercises.

By harnessing adversary emulation in conjunction with the Caldera framework, cybersecurity experts are equipped with the means to proactively safeguard their organizations against potential threats.

1. Glossary
2. Introduction
3. BitSight
4. Automation
   1. Operations
5. Structure
6. Installation
   1. Prerequisites
   2. Configuration
   3. Generating an API key for your BitSight account
   4. Adding the API Key to the BitSight Automation Tool
      1. Windows
      2. Linux
   5. The group_mapper.json file
   6. The guid_mapper.json file
   7. Configuring your Company’s structure
      1. The groups.conf file
      2. Letting BitSight Automation Tool handle the rest
   8. Binding into Executable
7. Execution
   1. Usage
   2. Use Cases
      1. Functional Operation: Rating
      2. Functional Operation: Historical
      3. Functional Operations: Findings
      4. Functional Operation: Assets
      5. Functional Operation: Reverse Lookup
      6. Supplementary Operation: List
      7. Supplementary Operation: Update
   3. Task Scheduler / Cron Jobs
      1. Windows – Task Scheduler
      2. Linux – Cron Jobs
8. Troubleshooting
   1. Total Risk Monitoring Subscription Required
   2. File not Found *.JSON
9. Conclusion

Glossary

Entity	A part of an organization that can be assessed as a single figure.
Subsidiary	Same as an Entity on BitSight’s side.
Group Cluster	A complex. It can contain entities/subsidiaries, or Groups, or more Group Clusters.
Group	A structure that can contain Entities.

Introduction

In this blog post you will be introduced to the BitSight Automation Tool (https://github.com/NVISOsecurity/BitSight-Automation-Tool). BitSight Automation was developed to automate certain manual procedures and extract information such as ratings, assets, findings, etc. Automating most of these tasks is crucial for simplicity and time saving. Besides that, this tool also provides the possibility to collaborate with Scheduled Tasks and cronjobs. You can configure the tool to execute in certain intervals or dates, and retrieve the results from the desired folder without needing to interact with it.

BitSight

What is BitSight? BitSight is a solution that helps organizations perform three (3) main functions.

1. Quantify their cyber risk
2. Measure the impact of their security efforts
3. Benchmark their performance against peers

It does all that by managing the company’s external facing infrastructure both automatically and manually, by allowing a company to provide updates to BitSight in order to keep their database up to date.

Other functions that are useful and provided by BitSight are:

• Performing periodic vulnerability assessments on those assets to determine the risk factors and reports back the findings.
• Identifies malicious activity such as botnet infections and much more that adds up to the risk factor.
• Provides detailed remediation tips to remediate findings.

Automation

By utilizing parts of the BitSight API Python wrapper developed by InfosecSapper, we developed an open source tool for the community to use, that fully automates some of BitSight’s operations, which we have named BitSight Automation Tool. This tool has a lot of potential to expand further with even more operations based on the needs that might arise.

Operations

You might be wondering by this point, what operations can this tool automate? Currently we have 5 operations that can be automated + 2 supplementary to assist with the tool’s maintenance.

1. Rating -> Retrieve the current score of an entity and confirm it’s above or equal to your company’s required security policies or digital mandate.
2. Findings -> Generate a filtered list of vulnerabilities for an entity to remediate.
3. Assets -> Retrieve the asset count and asset list of an entity, to validate your public IP space.
4. Reverse Lookup -> Investigate where an IP, IP Range, domain or domain wildcard is attributed to and what IPs or domains it is associated with.
5. Historical Ratings -> Sets up an overview of ratings for a given entity or group over a specified timeframe (maximum 1 year) to showcase in reports and review progress or regress.

• List ->  Review the correlation between an entity’s custom given name and BitSight’s given name in a list for all defined entities.
• Update -> Automatically update the tool and its respective JSON files.

Structure

The below image is a representation of the current state of the tool. At the time of writing the tool comes with the following structure.

• BitSightAPI: This folder contains certain vital Python files from the BitSightAPI Python wrapper.
• ArgumentsHandler.py: This file contains the instructions on how to parse the tool’s arguments.
• README.md: This file is your friend. It contains all the information on how to execute with examples, as well as troubleshooting advice.
• bitsight_automation.py: This file is the heart of the tool. You can execute this Python file and use it for your Scheduled tasks or cron jobs.
• group_mapper.json: This file is a JSON structure which represents the mapping of the groups and entities within your organization. (More on this in a dedicated section)
• guid_mapper.json: This file is a JSON structure which represents the mapping of the entities and their respective GUIDs assigned to them by BitSight. (A GUID is the unique handle used by BitSight to identify your subsidiary)
• groups.conf: This file is the main configuration to define your groups. It defines the groups which the tool will interact with.
• requirements.txt: This file indicates all the required libraries for the tool to operate.

Installation

Prerequisites

In order to use the tool, we first need to install it. Regardless of the Operating System you are using, you will need to have Python installed. The tool has been tested with Python 3.8 and 3.11 at the time of writing, so any Python 3.x.x version should work.

Note: When installing Python make sure to include it in your PATH and include the PIP package manager as well.

Next step would be to install the tool’s requirements. To do so, navigate to the tool’s directory within a command prompt / terminal or PowerShell window and execute the following command: `pip install -r requirements.txt`

All the prerequisites are installed at this point, but we still have a couple more steps to perform before we can use the tool.

Configuration

Now that you have installed the prerequisites, we are one step closer to utilizing the tool. Before we do so, we need to update a couple of files.

Generating an API key for your BitSight account

First you need to generate an API key from your account in BitSight. To do so,

1. Login to your BitSight account
2. On the top right corner of the UI, click on Settings.
3. Select Account
4. Scroll down until you see an “API Token” section
5. Select “Generate API Token”
6. Copy the newly generated token.

Adding the API Key to the BitSight Automation Tool

In order for the BitSight Automation Tool to use this API key, you need to include it as an environmental variable in the system you will be running the tool on. We’ll do so below for both Windows and Linux.

Windows

For Windows systems,

1. Open the search menu
2. Search for “Edit the System Environment Variables”
3. Select that option
4. Select “Environment Variables”
5. On the “User Variables” click “New”
6. In the “Variable Name” field, add the value “BITSIGHT_API_KEY”
7. In the “Variable Value” field, add the generated token you copied in the previous section.

Linux

For Linux systems,

1. Open a terminal
2. Replace the “{token}” section with your token, and execute the following command:

echo export BITSIGHT_API_KEY={token} >> ~/.bashrc

The group_mapper.json file

This file is the heart of the tool. It will be queried to retrieve entities for every operation.

An example of a group_mapper.json file can be found below.

{
    "Root":[
      {"Group1": "Single Entity"},
      {"Cluster Group2": ["EntityOne", "EntityTwo", "EntityThree"]},
      {"Bigger Cluster Group3": [
        {"SubCluster": ["Entity1", "Entity2"]},
        {"SubCluster2": ["EntityUno", "EntityDos"]}
      ]},
      "Random Entity that sits alone under the root"
    ]
}

The grouping subsidiaries can be organized like above. A few rules apply:

• All of your subsidiaries must be under the “Root” subsidiary. The “Root” Subsidiary is your main subscription in BitSight that contains all the others (if any).
• The Root subsidiary contains a list of other group subsidiaries or subsidiaries that are directly under the Root.
• A Group or Group Cluster subsidiary can hold one, or more subsidiaries.
• You can define bigger Cluster Group subsidiaries that contain even more group subsidiaries, which in turn can contain more subsidiaries.
• You can create your own Group subsidiaries even if they don’t exist in BitSight for better structuring.
• You can use your own naming conventions for all your subsidiaries and group subsidiaries without affecting BitSight or the retrieved information.

The guid_mapper.json file

This file is the tie between BitSight and the BitSight Automation Tool. This is where the magic happens where your naming conventions can relate back to specific subsidiaries within BitSight.

An example of a guid_mapper.json file can be found below.

{
    "Root": "463862495-ab29-32829-325829304823",
    "Group1": "463862495-ab29-32829-325829304824"
}

This structure is the most basic structure you can have. The only thing you have to do is to create a new line for each subsidiary and assign its GUID from BitSight.

Below you may find how to get the GUIDs for the subsidiaries. A few rules that apply:

• The order doesn’t matter.
• Make sure you use the exact naming convention you used on the group_mapper.json file. (It’s case sensitive)
• Do not add any lists or other structures in this file. It should be one line for every subsidiary.
• For groups you added in group_mapper.json file, that do not exist in BitSight, add a line like the following: “{your-group}”:”-“,

Configuring your Company’s structure

This step is mandatory for the tool to operate correctly. You can either use BitSight’s structure or you can create your own that suits best for your company.

The groups.conf file

Once you have completed the above steps, you need to modify one last item in the configuration of the tool.

The ‘groups.conf’ file structure should look like below (Figure 3)

You can add your groups one per line.

Note: Do not modify the first line. It should remain as is [Groups].

Letting BitSight Automation Tool handle the rest

You have completed the manual part of the configuration! Pretty simple, right?

Execute the tool with the update operation.

python bitsight_automation.py update

This will go through Bitsight and find any subsidiaries that are missing. It will then prompt you to include it into the configuration. Follow the steps, provide the required information and the tool will take care of the rest.

Example:

PS C:\Users\Konstantinos Pap\Desktop\BitSight Automation> python .\bitsight_automation.py update

Subsidiary Name – GUID not found in our configuration

Would you like to include it (Y/N)? y
What is the name of this entity? Test
Under which group should this entity fall under({your-groups}) ? myTestGroup
Adding Subsidiary Name with guid {GUID} as Test in myTestGroup

Configuration Updated

Binding into Executable

Before we dive into how to utilize this tool, let’s first dive into how we can make an executable bundle for this tool. Performing this step allows for easier sharing and makes the tool usable by anyone, from analyst to CISO, without any specific requirements. We can use Pyinstaller to create the .exe standalone file.

1. First, open a terminal
2. Now, we need to install pyinstaller. Use the `pip install pyinstaller` command to download and install pyinstaller.
3. Afterwards, navigate into the tool’s directory from within the terminal window.
4. Execute the following command: “pyinstaller bitsight_automation.py -p __pycache__ -F”

Wait for a few seconds and notice there is a new directory created named “dist”. Grab the 2 .json files and the README.md file and copy them into the dist directory. Zip the contents of that directory and distribute that bundle on any Windows system. It will execute without any need for dependencies.

Note: You still have to export the environment variables to the new machines in order for the tool to be able to connect to BitSight.

Execution

Now that we installed and fully configured BitSight, we can go ahead and use its capabilities. As we already mentioned before, the tool allows for 5 different operations + 2 supplementary to assist with the tool’s maintenance.

We’ll first have a look at the usage menu of the tool and then we’ll navigate over a breakdown of each operation and how it works with examples.

Usage

Invoke the tool with its –help attribute.

PS ~/> python .\bitsight_automation.py --help

usage: bitsight_automation.py [-h] [-g {{your-groups}}] [-e ENTITY] [-v]
                              [-s {All,Critical-High,Critical,High,Low,Medium}]
                              [-so {alphanumerically,alphabetically}] [--search SEARCH] [--months MONTHS]
                              {rating,historical,findings,assets,reverse_lookup,list,update}

BitSight Automation tool to automate certain operations like historical report generation, findings categorization, asset list retrieval, reverse lookup of IP addresses and current ratings for entites

positional arguments:
  {rating,historical,findings,assets,reverse_lookup,list,update}
                        The operation to perform.

optional arguments:
  -h, --help            show this help message and exit
  -g {{your-groups}}, --group {{your-groups}} The group of entities you want to query data for.
  -e ENTITY, --entity ENTITY A specific entity you want to query data for
  -v, --verbose         Increase output verbosity
  -s {All,Critical-High,Critical,High,Low,Medium}, --severity {All,Critical-High,Critical,High,Low,Medium}
                        Level of Severity to be captured
  -so {alphanumerically,alphabetically}, --sort {alphanumerically,alphabetically}
                        Sort rating results either alphanumerically or alphabetically.
  --search SEARCH       IP or Domain to reverse lookup for.
  --months MONTHS       Add in how many months back you want to view data for. If you want 1 year, fill in 12 months.
                        Max is 12

For any questions or feedback feel free to reach out to [email protected]

Use Cases

Now we will go through a breakdown of all different use cases within BitSight Automation Tool. We’ll go through the functional first and we’ll leave the last 2 complementary at the end.

For every operation different arguments will be required or not needed. The tool will let you know if you missed something during runtime. Example output:

[-] You need to specify one of the arguments --country or --region.

Functional Operation: Rating

Use the rating operation to retrieve the current score of an entity or group in order to confirm if it’s above or equal to your policies. If a group is supplied this operation will output all of the subsidiaries under the specified group in the order you specified them in the JSON files (You also have the option to sort them alphanumerically)

Let’s try to fetch the current rating for our “Test” subsidiary.

PS ~/> python .\bitsight_automation.py rating -e Test

Test - 790
[+] Data saved to: 2023-03-17_bitsight_rating_Test.txt

Our Test Subsidiary has a score of 790. That’s an advanced score, so we can cross-verify with the company’s policies and take further action if needed. The results are also saved as a TXT file to allow easy copy/paste if required.

We can do the same thing for our Group and retrieve all the scores from all subsidiaries under our “Test Group”.

PS ~/> python .\bitsight_automation.py rating -g “Test Group”

[*] This may take a moment. Grab a coffee
Working on Test Group...
Test Group – 660
EntityOne - 620
Test Entity 2 - 760
EntityTwo - 770
[+] Data saved to: 2023-03-17_bitsight_rating_Test Group.txt

Notice we have retrieved ratings for all subsidiaries under “Test Group” in addition with the rating of “Test Group”. Some additional notes:

• If “Test Group” didn’t have a GUID then it will not pull any data for it.
• You can change the sorting algorithm using the -so argument.
• You can retrieve the rating of “Test Group” without having to go through its subsidiaries. To do so, you can treat it like a normal entity, supplying it with the -e argument instead.
• If your group is a big cluster group containing more groups that contain more subsidiaries, the tool will recursively query Bitsight for all those groups and subsidiaries under the cluster group and its respective groups and subsidiaries. (In order words, nothing will be skipped)
• You can use -g {Root} to retrieve ratings for all the subsidiaries in your company. (Replace {Root} with the name you have given it)

Functional Operation: Historical

Use the historical operation to set up an overview of ratings for a given subsidiary or group over a specified timeframe (maximum 12 months) to showcase in reports and review progress or regress. Typically this operation is used with the -g argument but you can also utilize the -e argument for a given subsidiary only.

Let’s try to generate a report for our previous “Test Group” and its subsidiaries for the past year.

PS ~/> python .\bitsight_automation.py historical -g “Test Group” --months 12

Grab a coffee, this will take a while...
Working on Test Group...
[+] Data saved to 2023-03-17_Test Group_bitsight_historical_ratings_12_months.xlsx

Note: This command might take some time depending on the size of your organization + the number of subsidiaries it has to query data for. In any case, it is verbose enough to let you know in which group it is working on each time, so if you supplied a big cluster group you would have real time output of the progress.

The report:

There is a legend in the second sheet (tab) of the Excel file that denotes what these colors are and their scores – aligned with BitSight’s ratings and color coding.

Note: You can generate these types of reports with no limitation to a number of subsidiaries. You can even generate it for the entire organization using the Root subsidiary.

Functional Operations: Findings

Use the findings operation to generate a filtered list of vulnerabilities for a subsidiary to remediate. This operation works solely with subsidiaries and not groups! You also need to supply the severity level with the -s argument.

Note: Your subsidiaries need to have a ‘Total Risk Monitoring’ subscription for this command to work. Otherwise it will produce an error.

Let’s retrieve the findings for our ‘EntityOne’ subsidiary under ‘Test Group’ we used earlier. We will retrieve the Critical vulnerabilities only.

PS ~/> Python .\bitsight_automation.py findings -e EntityOne -s Critical

[+] Data saved to bitsight_Critical_findings_EntityOne_2023-03-17.csv

Critical findings were downloaded and saved to a file called ‘bitsight_Critical_findings_EntityOne_2023-03-17.csv’. You can now start working on remediating the findings or assign it to the proper internal team.

Functional Operation: Assets

Use the assets operation to retrieve the asset count and asset list of a subsidiary in order to validate your public IP space. This operation works solely with subsidiaries and not groups. This is a two-step process of querying. The operation first queries BitSight to retrieve the total count of public IPs in your subsidiary and then queries for the detailed asset list.

Note: This command requires a ‘Total Risk Monitoring’ subscription. If one is not available this command will produce an error.

Let’s attempt to retrieve the asset list for our ‘EntityOne’ subsidiary from the previous examples.

PS ~/ > python .\bitsight_automation.py assets -e EntityOne

EntityOne - 1410
*********** Asset List ************
[+] Asset List saved to: bitsight_asset_list_EntityOne_2023-03-17.csv

Note: This command will only fetch assets that are correctly attributed to this subsidiary. There’s a difference between correctly attributed by BitSight and internal/private Tagging.

Functional Operation: Reverse Lookup

Use this command to investigate where an IP, IP Range, domain or domain wildcard is attributed to and what IPs or domains it is associated with. This command only requires the –search argument.

Let’s attempt to find out where our test.com domain is attributed to and what public IPs it is associated with.

PS ~/> python .\bitsight_automation.py reverse_lookup --search test.com

test.com - ['<Redacted XX.XXX.XX.XXX>']: Found in: EntityOne

Supplementary Operation: List

Use this operation to review the correlation between an entity’s custom given name and BitSight’s given name in a list for all defined entities. This command does not require any arguments.

Let’s view our subsidiaries and their correlation to BitSight.

PS ~/> Python .\bitsight_automation.py list

Listing Configuration...
Root – My Test BitSight Organization
Group One – First Group Subsidiary
EntityOne- Entity1 Test
Test Entity 2- Entity 2 Test
EntityTwo – SSEntity 2

Note: The mapping is {my JSON representation – BitSight’s representation}. The two names are bound over the GUID unique value for a subsidiary.

Supplementary Operation: Update

Use this operation to automatically update the tool and its respective JSON files. We already saw how this command works in the configuration section.

Task Scheduler / Cron Jobs

As we already mentioned, we can either manually execute the BitSight Automation Tool or we can set it up to automatically execute on its own recurringly over a specified window of time. This is relatively easy to achieve in both Linux and Windows operating systems.

Windows – Task Scheduler

To achieve this in Windows we need to utilize the Task Scheduler utility provided by Microsoft itself. No need to download or install any additional software. Let’s configure it.

1. Open the Task Scheduler.
2. On the top left, select “Task Scheduler Library“. (Figure 6)

3. On the top right, select “Create Basic Task”

4. Write down a name and description like below:

5. Then click “Next“
6. Select a Monthly Trigger and click Next

7. Next, select the dates you wish to execute. I will select all months, and run on every 1^st Monday of the Month and click Next.

8. Choose “Start a Program” and click Next.

9. Browse to your bitsight_automation.exe file you created earlier for the “Program/Tool”field. For the arguments field supply “historical -g {your-group} –months XX” and replace {your-group} with the group you wish to execute for, and how many months back you want. (Remember it’s up to 12 months maximum). For the “Start in (Optional)” field add in the path to the executable. This is required here because the BitSight Automation Tool expects the JSON files in the same directory it is executing from. Finally click Next.

10. Verify all is correct and click on Finish.

Your Scheduled task is ready. You can manually invoke it once to verify it’s working correctly from the right bar, selecting ‘Run’

Note: You can follow this procedure for other tasks as well. (Update excluded as it requires manual intervention. However, the shell or prompt that will open will be interactive, so you can issue update comments on a daily basis anyway and if there are any, you can interact with the tool.)

Linux – Cron Jobs

The same process can be setup for Linux as well using the Cron Jobs it offers.

Write the following new line into the “/etc/crontab” file and replace ‘{your-tool-directory}’ with your tool’s directory (i.e. /opt/bitsight):

10 9 1 * * kali cd {your-tool-directory} && Python bitsight_automation.py historical -g Root –months 12

This will execute the tool every first of the month at 9:10 in the morning.

Troubleshooting

While executing this tool you might run into some issues here and there. This section will go over the 2 most common notifications you might encounter while using BitSight Automation.

Total Risk Monitoring Subscription Required

You may have noticed in the Execution section of a couple of operations a note saying “This operation requires a ‘Total Risk Monitoring’ subscription to work. Otherwise it will produce an error”. These types of errors are usually encountered in Findings and Assets operations. The output will look something like this.

If we remove the ‘Total Risk Monitoring’ subscription from EntityOne and execute the findings operation on it again, we will run into the following error:

PS ~/> Python .\bitsight_automation.py findings -c EntityOne -s Critical

It appears as there are no findings in EntityOne or there is something wrong with the API. Please validate the old fashioned way using your browser.
More Details: list index out of range
It might be the case you do not have a 'Total Risk Monitoring' subscription. The 'Risk Monitoring' subscription is unable to work with the API for this operation

Response: {'links': {'next': None, 'previous': None}, 'count': 0, 'results': []}

File not Found *.JSON

In case you execute the tool and it reports back with “File not found” it means that somehow the necessary files were deleted. In order to resolve this issue you need to create the files again with the text “{}”inside them.

PS ~/> Python .\bitsight_automation.py findings -c EntityOne -s Critical

File not found:  'group_mapper.json'. Please copy the  'group_mapper.json' to the same directory as this tool and try again.

Conclusion

This blogpost presented the BitSight Automation Tool as a valuable enhancement for organizations employing BitSight for performing external assessment and reducing exposure, as their solution.

Some key perks of this tool are as follow:

1. Automates a lot of operations that otherwise are time consuming.
   1. Rating -> Retrieve the current score of an entity and confirm it’s above or equal to your company’s required security policies or digital mandate.
   2. Findings -> Generate a filtered list of vulnerabilities for an entity to remediate.
   3. Assets -> Retrieve the asset count and asset list of an entity, to validate your public IP space.
   4. Reverse Lookup -> Investigate where an IP, IP Range, domain or domain wildcard is attributed to and what IPs or domains it is associated with.
   5. Historical Ratings -> Sets up an overview of ratings for a given entity or group over a specified timeframe (maximum 1 year) to showcase in reports and review progress or regress.
2. Allows the possibility to configure scheduled executions of the tool and create monthly/daily/yearly reports per your needs.
3. Provides an easy to use command interface. It can also be compiled as an executable version to avoid having to install dependencies and to make it usable by anyone. (From analysts to CISO.)