πŸ”’
❌
There are new articles available, click to refresh the page.
Before yesterdayOutflank Blog

RedELK Part 3 – Achieving operational oversight

7 April 2020 at 15:08

This is part 3 of a multipart blog series onΒ RedELK: Outflank’s open sourced tooling that acts as a red team’s SIEM and helps with overall improved oversight during red team operations.

InΒ part 1 of this blog seriesΒ I discussed the core concepts of RedELK and why you should want a tool like this. In part 2 I described a walk-through on integrating RedELK into your red teaming infrastructure. Read those blogs to get a better background understanding of RedELK.

For this blog I’ve setup and compromised a fictitious company. I use the logs from that hack to walk through various options of RedELK. It should make clear why RedELK is really helpful in gaining operational oversight during the campaign.

Intro: the Stroop lab got hacked

In this blog I continue with the offensive lab setup created in part 2. In summary this means that the offensive campaign contains two attack scenarios supported by their own offensive infrastructure, named shorthaul and longhaul. Each have their own technology for transport and each has a dedicated Cobalt Strike C2 server. The image below should give you an overview.

The offensive infrastructure as used in this demo

Now let’s discuss the target: Stroop B.V., a fictitious stroopwafel company. Although the competition is catching up, their stroopwafels are regarded as a real treat and enjoyed around the world. Stroop’s IT environment spans over a few thousand users and computers, an Active Directory domain with subdomains and locations (sites) around the world. With regards to security they are coming from a traditional β€˜coconut’-approach: hard security on the outside (perimeter), but no real segmentation or filtering on the inside. They do have a few security measures, such as proxying all internet traffic, dedicated admin-accounts and admin-workstations for AD related tasks. Finally, in order to get to the industrial controls systems (ICS) that produce the stroopwafels – and guard the recipe – it is required to go via a dedicated jump host.

In this demo, I have gained Domain Admin privileges and DCsync’ed the krbtgt account. I have not accessed the secret recipes as I do not want to give away too much details for future students of our trainings. Yes, you read that right, this is the same lab setup we use in our trainings. And yes, besides an awesome lab our students also get to enjoy delicious stroopwafels during our trainings. No digital but real stroopwafels. πŸ™‚

In preparation for this blog post I have hacked through the network already. To make it easy for you to play with the same data, I have uploaded every logfile of this demo (Cobalt Strike, HAProxy and Apache) to the RedELK github. You can use this demo data to import into your own RedELK server and to get hands-on experience with using RedELK.

The end result of the running beacons can be seen below in the two overviews from our Cobalt Strike C2 servers.

Beacon overview of the shorthaul scenario
Beacon overview of the longhaul scenario

Why RedELK?

The mere fact that I have to present to you two pictures from two different Cobalt Strike C2 servers is indicative why we started working on what later resulted in RedELK. Cobalt Strike (or any other C2 framework) is great for live hacking, but it is not great for a central overview. And you really want such a central overview for any bigger red teaming campaign. This is exactly what RedELK does: it gathers all the logs from C2 frameworks and from traffic redirectors, formats and enriches the data, presents it in a single location to give you oversight and allows for easy searching of the data. Finally, it has some logic to alarm you about suspected blue team analyses.

I’ll cover the alarming in a later blog. In this blog we focus on the central presentation and searching.

RedELK oversight

Let’s start with the oversight functionality of RedELK. We can easily see:

  • Red Team Operations: Every action from every operator on every C2 server
  • Redirector Traffic: All traffic that somewhere touched our infrastructure

Because we put everything in an Elasticsearch database, searching through the data is made easy. This way we can answer questions like β€˜Did we touch system X on a specific day?’ or β€˜What source IP has scanned our redirectors with a user agent similar to our implant but not on the correct URL?’

Besides free format searching, RedELK ships with several pre-made searches, visualisations and dashboards. These are:

  • CS Downloads: all downloaded files form every Cobalt Strike C2 server, directly accessible for download via your browser.
  • CS Keystrokes: every keystroke logged from every Cobalt Strike C2 server.
  • CS IOCs: every Indicator Of Compromise of your operation.
  • CS Screenshots: every screenshot from every Cobalt Strike C2 server, with thumbnails and full images directly visible via your browser.
  • CS Beacons: list with details of every Cobalt Strike beacon.
  • Beacon dashboard: a dashboard with all relevant info from implants.
  • Traffic dashboard: a dashboard with all relevant info on redirector traffic.

Note: at the moment of writing the stable version of RedELK is 1.0.2. In that version there is full support for Cobalt Strike, but no other C2 framework. However, future stable versions will support other C2 frameworks. The version currently in development already has support for PoshC2. Covenant is planned next. Also, the names of the pre-made views searches and of some fields are also made more generic to support other C2 frameworks.

We need to login before we can explore some of these views and show the power of search. See the example below where I log in, go to the Discover section in Kibana, change the time period and open one of the pre-made views (redirector traffic dashboard).

Note: every animated GIF in this blog is clickable to better see the details.

Opening the redirector traffic view

Red Team Operations

Let’s start with an overview of every action on every C2 server; select the Red Team Operations view. Every line represents a single log line of your C2 tool, in our case Cobalt Strike. Some events are omitted by default in this view: join-leave events, β€˜new beacon’ events from the main Event Log and everything from the weblog. Feel free to modify it to your liking – you can even click β€˜Save’ and overwrite one of RedELK’s pre-made views.

In the example below you can see the default layout contains the time, attackscenario, username, internal IP address, hostname, OS and the actual message from the C2 tool. In our case this is 640 events from multiple team servers from both attack scenarios!

Now let’s say the white team asked if, where and when we used PsExec. Easy question to answer! We search for psexec* and are presented with only the logs where psexec was mentioned. In our case only one time, where we jumped from system L-WIN224 to L-WIN227.

Searching for the execution of PsExec across all C2 servers

As you can see, RedELK does a few things for you. First of all, it indexes all the logs, parses relevant items, makes them searchable and presents them to you in an easy interface.

Cobalt Strike users will notice that there is something going on here. How can RedELK give you all the relevant metadata (username, hostname, etc) per log line while Cobalt Strike only presents that info in the very first metadata line of a new beacon? Well, RedELK has background running scripts that enrich *every* log line with the relevant data. As a result, every log line from a beacon log has the relevant info such as username, hostname, etc. Very nice!

Another simple but super useful thing we saw in the demo above is that every log line contains a clickable link to the full beacon.log. Full beacon logs directly visible in your browser! Great news for us non-Elasticsearch-heros: CTRL+F for easy searching.

As you can see, a few clicks in a web browser beat SSHing into all of your C2 servers and grepping through the logs (you shouldn’t give everybody in your red team SSH access to your C2 servers anyway, right?).

Now that was the basics. Let’s explore a few more pre-made RedELK views.

List of Indicators of Compromise

In the previous example I searched for PsExec lateral movement actions. As you probably know, PsExec uploads an executable that contains your evil code to the target system, creates a service pointing to this executable and remotely starts that service. This leaves several traces on the remote system, or Indicators of Compromise as the blue team likes to call them. Cobalt Strike does a very good job in keeping track of IOCs it generates. In RedELK I created a parser for such IOC events and pre-made a search for you. As a result you can get a full listing of every IOC of your campaign with a single click. Four clicks if you count the β€˜export to CSV’ function of Kibana as well.

IOC overview

Cobalt Strike Downloads

This is one of the functions I am personally most happy with. During our red team operations we often download files from our target to get a better understanding of the business operations (e.g. a manual on payment operations). Most C2 frameworks, and Cobalt Strike in this example, download files form the implant to the C2 server. Before you can open the file you need to sync it to your local client. The reason for this is valid: you don’t want every file to auto sync to every logged in operator. But with dozens maybe hundreds of files per C2 server, and multiple C2 servers in the campaign, I often get lost and I no longer know on which C2 which file was downloaded. Especially if it was downloaded by one of my colleagues. Within the C2 framework interface it is very hard to search for the file you are looking for. Result: time lost due to useless syncing and viewing of files.

RedELK solves this by presenting an aggregated list of every file from every C2 in the campaign. In the Kibana interface you can see and search all kinds of details (attack scenario name, username, system name, file name, etc). In the background, RedELK has already synced every file to the local RedELK server, allowing for easy one click downloads straight from Kibana.

Metadata and direct access to all downloads from all C2 servers

Logged keystrokes

Ever had difficulties trying to remember which user on what moment entered that one specific term that was logged in a keystroke? RedELK to the rescue: one click easy presenting of all logged keystrokes. And, of course searchable.

Searching logged keystrokes

I believe there is more work to be done by formatting the keystroke logs and alarming when certain keywords are found. But that is left for future versions.

Screenshots

Another thing that was bugging me was trying to recall a specific screenshot weeks/months earlier in the campaign. Most often I can’t remember the user, but when I see the picture I know what I was looking for. With hundreds of screenshots per C2 server this becomes time consuming.

To solve this, RedELK indexes every screenshot taken, and makes them ready for download and presents some sort of a thumbnail preview picture. Hooray!

Screenshots overview

I’m not entirely happy just yet with the thumbnail previews, specifically the size. I’m limited by the screen space Kibana allows. Likely something I’ll fix in a new release of RedELK.

Overview of all compromised systems

A final thing I want to discuss on the viewpoint of red team operations is the overview of every C2 beacon in the campaign. RedELK presents this with the CS Beacons overview. This is a great overview of every implant, the system it ran on, the time of start and many other details. Again you can use the Kibana export-to-CSV function to generate a list that you can share with blue and/or white.

In this example I want to highlight one thing. RedELK also keeps track if a Cobalt Strike beacon was linked to another beacon. In the example below you can see that beacon ID 455228 was linked to 22170412, which in turn was linked to 1282172642. Opening the full beacon log file and searching for β€œlink to” we circle back to the PsExec example we discussed above.

Metadata on all beacons from the operation

Redirector Traffic

The examples above all covered overview of red team operations. But RedELK also helps you with giving overview and insight into the traffic that has hit your red team infrastructure. This is done by using the pre-made view Redirector Traffic.

Redirector traffic overview

We can see that RedELK parses a lot of information. The default view shows several columns with relevant data, ordered per log line with the latest event on top. Diving into a single log line you can see there are many more information elements. The most important are:

  • The attackscenario, shorthaul in our example.
  • The beat.hostname of the redir this happened on.
  • The full log line as appeared on the system (message).
  • The IP address of the redirector traffic was received on (redir.frontendip).
  • The redirprogram name (Apache in this case) and the redir.frontname and redir.backendname where on the traffic was received and sent.
  • Several headers of the HTTP traffic, including X-Host and X-Forwarded-For.
  • The redirtraffic.sourceip of the traffic.
  • In case X-Forwarder-For was set, redirtraffic.sourceip becomes the real client’s address and redirtraffic.sourceipcnd contains the address of the CDN endpoint.

The majority of this information is not available in a default Apache or HAProxy log line. But it is information that the red team is interested in, now or in the future. This is why it is so important to modify the logging configuration of your redirector. Default log setup means limited info.

RedELK also enriches the data with information from several sources. The GeoIP was already shown in the previous example – it adds geo location info and ownership of the IP block and stores it in geoip.* fields.

RedELK als does reverse DNS lookups for the remote IP addresses and it sets a tag if the source IP address was found in a local config file like /etc/redelk/iplist_redteam.conf, /etc/redelk/torexitnodes.conf, etc.

But there is one more enrichment going on that is extremely helpful: Greynoise. Greynoise is a service that aims to identify the internet background noise from scanners, either legit ones such as Shodan and Google, and evil ones such as botnets. Most red teams are not necessarily interested in all the details that Greynoise has about an IP address. But they do want to know when an IP address is scanning their infra that is *not known* as a scanner!

Let’s see the example below. We start with almost 9000 log lines on traffic. When opening one event we can see the multitude of info that Greynoise has on this IP. In our example its likely a MIRAI botnet scanner and the greynoise.status is ok. But when we filter on greynoise.status:"unknown" and NOT redirtraffic.httpstatus:"200" we get all data from IP addresses not belonging to publicly known scanners that scan our infra. We went from almost 9000 hits to 44 hits, a number that is easily analysed by the human eye. Would any of these hits be the blue team scanning our infra?

Finding strange traffic to your redirs by filtering on Greynoise status

There are many more examples on interesting searches for traffic hitting your red team infrastructure, e.g. TOR addresses (RedELK tags those), strange user agents such as curl* and python*, etc. But I’m sure you don’t need a blog post to explore these.

Wrap-up

I hope this blog post has given you a better understanding what RedELK has to offer, and how to use it. There is more. I haven’t even touched on visualizations and on dashboards that come with RedELK, and most importantly: alarms. This is left for another blog post.

A few closing thoughts to help you have a smooth experience in getting up-and-running with RedELK:

  • Fully follow the installation including modifying of the redirector logging configuration to get the most out of your data.
  • Also perform post installation tuning by modifying the config files in /etc/redelk/ on the RedELK server.
  • RedELK’s main interface is Kibana. It is easy to get started, has a lot of features but it can be tricky to fully understand the powerful search language. Luckily, most searches can also be done via clicky-clicky. I have absolutely no shame in admitting that even after many years of experience with searching in Kibana I still regularly prefer clicks for more difficult searches.
  • The pre-made views are just that: pre-made. You can delete, modify and rename them to your liking, or create new ones from scratch.

Have fun exploring RedELK and let me know your thoughts!

Our reasoning for Outflank Security Tooling

2 April 2021 at 12:26

TLDR: We open up our internal toolkit commercially to other red teams. This post explains why.

Is blue catching your offensive actions? Are you relying on public or even commercial tools, but are these flagged by AV and EDR? Hesitant on investing deeply in offensive research and development? We’ve been there. But several years ago, we made the switch and started heavily investing in research. Our custom toolset was born.

Today we open up our toolset to other red teams in a new service called Outflank Security Tooling, abbreviated OST. We are super(!) excited about this. We truly think this commercial model is a win-win and will help other red teams and subsequently many organisations worldwide. You can find all the details at the product page. But there is more to be explained about why we do this, which is better suited in a blog post.

In this post you will find our reasoning for this service, our take on red team evolution, the relation to that other OST abbreviation and a short Q&A.

Our inhouse offensive toolset opened up for others

OST is a toolset that any red teamer would want in his arsenal. Tools that we use in our own red teaming engagements. Tools that my awesome colleagues have and continue to spend significant time researching, developing and maintaining. Proven tools that get us results.

OST is not another C2 framework. It’s an addition. A collection of tools for all stages of a red teaming operation. The following is a selection of the current toolset:

  • Office Intrusion Pack: abuse non-well-known tricks in Office to get that initial foothold.
  • Payload Generator: centralised and structured way to generate different kinds of payloads. No more heavy programming knowledge required to get payloads with awesome anti-forensics, EDR-evasion, guard-rails, transformation options, etc.
  • Lateral Pack: move lateral while staying under the radar of EDRs. A powerful collection of different ways for lateral movement.
  • Stage1 C2: OPSEC focussed C2 framework for stage 1 operations.
  • Hidden Desktop: operate interactive fat-client applications without the user experiencing anything. It’s pure interactive desktop magic.

Overall principles in a changing toolset

The toolset will change over time as we continue our R&D and as we adapt to the changing demand. But the following overall principles will stay the same:

  1. Awesome functionality that a red team would want.
  2. OPSEC safe operations that help you stay undetected.
  3. Easy to use for different skill levels within your team.
  4. Supporting documentation on concepts and details so you know what you are using.

You can find all details at the product page here. Now let’s get into our reasoning.

Public tools for red teams will not cut it anymore

Looking at our industry we have seen a strong rise in strength of blue teams the last couple of years. Both in tools and skills. This means far more effective detection and response. This is a good thing. This is what we wanted!

But this also means that public tools for red teams are becoming less and less effective against a more advanced blue team. For example, PowerShell used to be an easy choice. But nowadays any mature blue team is more than capable of stopping PowerShell based attacks. So red moves their arsenal to .NET. But proper EDRs and AMSI integration are among us. So .NET is not ideal anymore. It’s a matter of time before these attacks follow the same path as PowerShell. This pushes red into the land of direct system calls and low-level programming. But the battle has started in this area as well.

This is a good thing as it also pushes the real attackers to new territory. Hopefully shredding another layer of cybercriminals along the way.

In other words: to stay relevant, red teams need to invest heavily in their arsenal and skills.

This means more in-depth research for red teams

Doing in-depth R&D is not for the faint of hearts. It requires a distinct combination of knowledge and skills. Not only the level of detailed knowledge becomes a challenge. The broadness of knowledge as well. For example: a red team can have in-depth knowledge on low level lateral windows protocols. But without knowledge on getting your initial foothold, you miss a piece of the puzzle required for a complete operation.

It is becoming harder to have all required R&D skills in your red team. And we believe that is totally OK.

Novel R&D is not the role of a red team per se

At its core, doing novel R&D is not per se the role of the red team. Sure, it might help. But the end goal of red teams is helping their clients becoming more secure. They do this making an impact to their client via a realistic cyber-attack, and subsequently advising on how to improve. Super l33t R&D can help. But it is a means to a goal.

Take the following somewhat extreme examples:

  1. Red team A has not got the ability to do novel research and tool development. But it does have the ability to understand and use tools from others in their ops very effectively.
  2. Red team B does great detailed research and has the best custom tools. They built everything themselves. But they fail to execute this in a meaningful manner for their clients.

Red team B fails to help its client. Red team A is by far the more successful and effective red team.

This is not a new thing. We see it throughout our industry. Does a starting red team develop its own C2? No, it buys one of the available options. Even we – a pretty mature red team – still buy Cobalt Strike. Because it helps us to be more effective in our work.

This got us thinking. And eventually made us decide to start our OST service.

We founded Outflank to do red teaming right

Back in 2016, we founded Outflank because we wanted to:

  1. Help organizations battling the rising risk of targeted cyber-attacks.
  2. Push the industry with research.
  3. Have some fun along the way.

Starting with just 4 people, we were a highly specialized and high performing team. Not much has changed since then. Only the number has increased to 7. We don’t hire to grow as an objective. We grow when we find the right person on skill and personal level. It is the way we like our company to operate.

This has many benefits. Not at least a client base full of awesome companies that we are truly honoured to serve. And as we help them progress with their security, we are having fun along the way. This is what I call a win-win situation.

OST helps with heavy R&D economics

Our Outflank model does not scale well. We can’t serve every company on the planet and make it more secure. But in a way, we do want to help every company in the world. Or at least as many as we can. If we can’t serve them all, maybe we can at least have our tools serve them indirectly. Why not share these tools and in a way have them help companies worldwide?

This new model also helps with the economics of heavy R&D. As discussed earlier, modern red teaming requires tremendous research and development time. That is OK. We love doing that. But there comes a point that huge development time isn’t commercially feasible for our own engagements anymore. With OST, we have a financial incentive for heavy research which in turn helps the world to become more secure.

Or to put it boldly: OST enables us to finally take up major research areas that we were holding off due to too heavy R&D time. This then flows into the OST toolset, allowing customers and their clients to benefit.

We love sharing our novel tools and research

Our final reason is that we are techies at heart that love sharing our research on conferences, blogs and GitHub. We have done so a lot, especially if you look at the size of our little company. We would be very sad if we have to stop doing this.

But when you find your own previously shared research and tools in breach investigation reports on cyber criminals and state actors, it makes you think (example 1, example 2, example 3).

This brings me to that other OST abbreviation.

We are not blind to the public OST debate

OST is also an abbreviation for Offensive Security Tooling. You know, that heated discussion (especially on Twitter) between vocal voices on both blue and red side. A discussion where we perhaps have forgotten we are in this together. Red and blue share the same goal!

All drama aside, there is truth in the debate. Here at Outflank we highlighted the following arguments that we simply can’t ignore:

  1. Publicly available offensive tools are used in big cyber-attacks.
  2. Researchers sharing their offensive tools make other red teams (and blue teams) more effective. This in turn makes sure the defensive industry goes forward.
  3. The sharing of new research and tools is a major part of our industry’s ability to self-educate. This helps both red and blue.

The Outflank Security Tooling service contains tools built upon our research that we did not share before. We haven’t shared this because some of this research resulted in mayhem-level like tools. We don’t want these in the hands of cyber criminals and state actors. This decision was made well before the OST debate even started.

We counter the first and second arguments by not releasing our very powerful tools to the wide public, but to interested red teams.

We counter the third argument by continuing to present our research at conferences and share some PoCs of our non-mayhem-level tools. This way we can still contribute to the educational aspect that makes our industry so cool.

With our OST service we believe we make a (modest) step to a more secure world.

We are excited about OST and hope you are as well

We think OST is awesome! We believe it will allow other red teams to keep being awesome and help their clients. At the same time, OST provides an economic incentive to keep pushing for new research and tools that our customers will benefit from.

While we continue to release some of our non-dangerous research and PoCs to the public, OST allows us to share the dangerous tools only to selected customers. And have some fun while doing this. Again, a win-win situation.

We are excited about bringing OST to market. We hope you are as well!

This is not the end of the story

Instead, this is the start of an adventure. An adventure during which we already learned an awful lot about things such as the β€˜Intrusion Software’ part of the Wassenaar Agreement, export controls and how to embed this technically into a service.

We believe that sharing information will make the world a better place. So, we will make sure to share our lessons learned during this adventure in future blog posts and at conferences. Such that our industry can benefit from this.

Q&A

Does this mean you will stop publishing tools on your GitHub page?

No, sharing is at our core!

We will continue releasing proof-of-concept code and research on our GitHub page. We will keep contributing to other public offensive tools. Only our most dangerous tools will be released in a controlled manner via the OST service. Non-directly offensive tools such as RedELK will remain open source.

You can expect new public tool releases in the future.

Is OST available for everyone?

Due to the sensitivity of the tools, our ethical standards and because of export controls on intrusion software, we will be selective in which red teams we can serve with OST.

It is our obligation to prevent abuse of these tools in cybercriminal or geopolitical attacks. This will limit our clientele for sure. But so be it. We need clients that we can trust (and we take some technical measures against tool leakage of course).

Can I make my low skill pentest team be a l33t red team with OST?

Not really, and this is not the goal of OST. A toolset is an important part of a red teaming operation. But a team of skilled operators is at least as important!

We want red teams to understand what is happening under the hood when our tools are used. And OST supports them in this, for example by in-depth documentation of the techniques implemented in our tools.

Can I get a demonstration of the OST toolkit?

Yes.

  • There are no more articles
❌