🔒
There are new articles available, click to refresh the page.
Before yesterdayNVISO Labs

Going beyond traditional metrics: 3 key strategies to measuring your SOC performance

26 May 2021 at 11:59

Establishing a Security Operation Center is a great way to reduce the risk of cyber attacks damaging your organization by detecting and investigating suspicious events derived from infrastructure and network data.  In traditionally heavily regulated industries such as banking, the motivation to establish a SOC is often further complimented by a regulatory requirement. It is therefore no wonder that SOCs have been and still are on the rise. As for In-House SOCs, “only 30 percent of organizations had this capability in 2017 and 2018, that number jumps to over half (51%)” (DomainTools).

But as usual, increased security and risk reduction comes at a cost, and a SOC’s price tag can be significant. Adding up to the cost of SIEM tools are in-demand cyber security professionals whose salaries reflect their scarcity on the job market, the cost of setting up and maintaining the systems, developing processes and procedures as well as regular trainings and awareness measures.

It is only fair to expect the return on investment to reflect the large sum of money spent – that is for the SOC to run effectively and efficiently in order to secure further funding. But what does that mean?

I would like to briefly discuss a few key points when it comes to properly evaluate a SOC’s performance and capabilities. I will refrain from proposing a one size fits all-approach, but rather outline which common issues I have encountered and which approach I prefer to avoid them.

I will take into account that – like many security functions – a well-operating SOC can be perceived as a bit of a black box, as it will prevent large-scale security incidents from occurring, making it seem like the company is not at risk and is spending too much on security. Cost and budget are always important factors when it comes to risk management, the right balance between providing clear and understandable numbers and sticking to performance indicators that actually signify performance has to be found.

The limitations of security by numbers and metrics-based KPIs

To demonstrate performance, metrics and key performance indicators (KPI) are often employed. A metric is an atomic data point (e.g. the number of tickets an analyst closed in a day) while a KPI sets an expected or acceptable range for the KPI to fall into (e.g. each analyst is supposed to close from x – x+y tickets in a day).

The below table from the SANS institute’s 2019 SOC survey conveys that the top 3 metrics used to track and report a SOC’s performance are the number of incidents/cases handled, the time from detection to containment to eradication (i.e. the time from detection to full closure) and the number of incidents/cases closed per shift.

Figure 1- SANS, Common and Best Practices for Security Operations Centers: Results of the 2019 SOC Survey

Metrics are popular because they quantify complex matters into one or several simple numbers. As the report states, “It’s easy to count; it’s easy to extract this data in an automated fashion; and it’s an easy way to proclaim, ‘We’re doing something!’ Or, ‚We did more this week than last week!‘“ (SANS Institute). But busy does not equal secure.

There are 3 main issues that can arise when using metrics and KPIs to measure a SOC’s performance:

  • Picking out metrics commonly associated with a high workload or -speed does not ensure that the SOC is actually performing well. This is most apparent with the second-most used metric of the time it takes to fully resolve an incident as this will vary greatly depending on the complexity of the cases. Complex incidents may take months to actually resolve (including a full scoping, containment, communication and lessons learned). Teams should not be punished for being diligent where they should be.
    As a metric, e.g. the number of cases handled or closed are atomic pieces of information without much context and meaning to it. This data point could be made into a KPI by defining a range the metric would need to fall into to be deemed acceptable. This works well if the expected value range can be foreseen and quantified, as in ‘You answered 8 out of 10 questions correctly’. For a SOC there is no fixed number of cases supposed to reliably come up each shift.
  • Furthermore, the number of alerts processed and tickets closed can easily be influenced via the detection rules configuration. While generally the “most prominent challenge for any monitoring system—particularly IDSes—is to achieve a high true positive rate” (MITRE), a KPI based on alert volume creates an incentive to work in an opposite direction. As shown below in Figure 2, more advanced detection capabilities will likely reduce the amount of alerts generated by the SIEM, allowing analysts to spend more time to drill down on remaining key alerts and on complementary threat hunting.
Figure 2 – Mitre, Ten Strategies of a World-Class Cybersecurity Operations Center
  • Lessons learned and the respective improvement of the SOC’s capabilities are rarely rewarded with such metrics, resulting in less incentive to perform these essential activities regularly and diligently.

Especially when KPIs are used to evaluate individual people’s performance and eventually affect bonus or promotion decisions, great care must be taken to not create a conflict of interest between reaching an arbitrary target and actually improving the quality of the SOC. Bad KPIs can result in inefficiencies being rewarded and even increase risk.

Metrics and KPIs certainly have their use, but they must be chosen wisely in order to actually indicate risk reduction via the SOC as well as to avoid conflicting incentives.

Below I will highlight strategies on how to rethink KPIs and SOC performance evaluation.

Operating model-based targets

To understand how to evaluate if the SOC is doing well, it is crucial to focus on the SOCs purpose. To do so, the SOC target operating model is the golden source. A target operating model should be mandatory for each and every SOC, especially at the early stages. It details how the SOC integrates into the organization, why it was established and what it will and will not do. Clearly outlining the purpose of the SOC in the operating model, as well as establishing how the SOC plans to achieve this goal, can help to set realistic and strategically sound measures of performance and success. If you don’t know what goal the SOC is supposed to achieve, how can you measure if it got there?

One benefit of this approach is that it allows for a more holistic view on what constitutes ‘the SOC’, taking into account the maturity of the SOC as well as the people, processes and technology trinity that makes up the SOC.

A target operating model-based approach will work from the moment a SOC is being established. Which data sources are planned to be onboarded (and why)? How will detection capabilities be linked to risk, e.g. via a mapping to MITRE? Do you want to automate your response activities? These are key milestones that provide value to the SOC and reaching them can be used as indicators of performance especially in the first few years of establishing and running the SOC.

Formulating Objectives and Key Results (OKR)

From the target operating model, you can start deriving objectives and key results (OKRs) for the SOC. The idea of OKRs is to define an objective (what should be accomplished) and associate key results with it that have to be achieved to get there. KPIs can fit into this model by serving as key results, but linking them with an objective makes sure that they are meaningful and help to achieve a strategic goal (Panchadsaram).

The objectives chosen can be either project or operations-oriented. A project-oriented objective can refer to a new capability that is to be added to the SOC, e.g. the integration of SOAR capabilities for automation. The key results for this objective are then a set of milestones to complete, e.g. selecting a tool, creating an automation framework and completing a POC.

KPIs are generally well suited when it comes to daily operations. Envisioning the SOC as a service within the organization can help to define performance-oriented baselines to monitor the SOC’s health as well as to steer operational improvements.

  • While the number of cases handled is not a good measure of efficiency on its own, it would be odd if a SOC had not even a single case in a month or two, allowing this metric to act as one component to an overall health and plausibility check. If you usually get 15-25 cases each day and suddenly there is radio silence, you may want to check your systems.
  • The total number of cases handled and the number of cases closed per shift can serve to steer operational efficiency by indicating how many analysts the SOC should employ based on the current case volume.

To implement operational KPIs, metrics can be documented over a period of time to be analyzed at the end of a review cycle – e.g. once per quarter – to decide where the SOC has potential for improvement. This way, realistic targets can be defined tailored to the specific SOC.

Testing the SOC’s capabilities

While metrics and milestones can serve as a conceptional indicator of the SOC’s ability to effectively identify and act on security incidents, it is simply impossible to be sure without seeing the SOC’s capabilities applied in an actual incident. You would need to wait for an actual incident to strike, which is not something you can plan, foresee, or even want to happen. In reality, some SOCs may never face a large incident. This means that they got very lucky  – or that they missed something critical. Which of these is true, they will never know. It is very possible to be compromised without knowing.

Purple teaming is a great exercise to see how the SOC is really doing. Purple teaming refers to an activity where the SOC (the ‘blue team’) and penetration testers (the ‘red team’) work together in order to simulate a realistic attack scenario. The actual execution can vary from a complete surprise test where the red teamers act without instructions – just like a real attacker would – , to more defined approaches where specific attack steps are performed in order to confirm if and when they are being detected.

When you simulate an attack in this way, you know exactly what the SOC should have detected and what it actually found. If there is a gap, the exercise provides good visibility on where to follow up in improving the SOC’s capabilities. Areas of improvement can range from a missing data source in the SIEM to a lack of training and experience for analysts. There is rarely a better opportunity to cover people, processes and technology in one single practical assessment.

It is important that these tests are not being seen as a threat to the SOC, especially if it turns out that the SOC does not detect the red team’s activities. Red teaming may therefore be understood as “a practical response to a complex cultural problem” (DCDC), where an often valuable team-oriented culture revolving around cohesion under stress can “constrain[] thinking, discourage[] people from speaking out or exclude[] alternative perspectives” (DCDC). The whole purpose of the exercise is to identify such blind spots, which – especially when conducted for the first times – can be larger than expected. This may discourage some SOC managers from conducting these tests, fearing that they will make them look bad in front of senior management.

Management should therefore encourage such exercises from an early stage and clearly express what they expect as an outcome: That gaps are closed after a proper assessment, not that no gaps will ever show up. If “done well by the right people using appropriate techniques, red teaming can generate constructive critique of a project, inject broader thinking to a problem and provide alternative perspectives to shape plans” (DCDC).

Conducting such testing early on and on a regular basis – at least once a year – can help improve the SOCs performance as well as steering investments the right way, eventually saving money for the organization. Budget can be used effectively to close gaps and to set priorities instead of blindly adding capabilities such as tools or data sources that end up underused and eventually discarded.

Summary

Establishing and running a SOC is a complex and expensive endeavor that should yield more benefit to a company then a couple of checks on compliance checklists. Unfortunately classic SOC metrics are often insufficient to indicate actual risk reduction. Furthermore, metrics can set incentives to work inefficiently and thus waste money and provide a wrong sense of security.

A strategy focused approach on measuring whether the SOC is reaching targets as an organizational unit facilitated by a target operating model complemented by well-defined OKRs and operational KPIs can be of great benefit to lead the SOC to reduce risk more efficiently.

To really know if the SOC is capable of identifying and responding to incidents, regular tests should be conducted in a purple team manner, starting early on and making them a habit as the SOC improves its maturity.

Sarah Wisbar
Sarah Wisbar

Sarah Wisbar is a GCDA and GCFA-certified IT security expert. With several years of experience as a team lead and senior consultant in the financial services sector under her wings, she now manages the NVISO SOC. She likes implementing lean but efficient processes in operations and keeps her eyes on the ever-changing threat landscape to strengthen the SOC’s defenses.

Sources

Domaintools : https://www.domaintools.com/content/survey_security_report_card_2019.pdf

SANS Institute: https://www.sans.org/media/analyst-program/common-practices-security-operations-centers-results-2019-soc-survey-39060.pdf

MITRE : https://www.mitre.org/sites/default/files/publications/pr-13-1028-mitre-10-strategies-cyber-ops-center.pdf

DCDC: https://www.act.nato.int/images/stories/events/2011/cde/rr_ukdcdc.pdf

Panchadsaram: https://www.whatmatters.com/resources/difference-between-okr-kpi/

Cyber Security Contests – A look behind the scenes about how to expand the community

10 December 2020 at 16:12

Cyber security has long since become a strategic priority for organizations across the globe and in all sectors. Therefore, training and hiring young potential in information security has become a crucial goal.  

To raise awareness of cyber security threats and help train a generation of security aware security experts, we at NVISO organize Capture the Flag (CTF) Cyber Security Events in two countries, Belgium and Germany and reach a broad audience.  

Each year, we organize the Cyber Security Challenge Belgium and the Cyber Security Rumble Germany. After six successful editions in Belgium and two in Germany, we want to share a little information on how the events came to be, and what the main challenges are that we face.

This image has an empty alt attribute; its file name is image-7.png

The organization team of this year’s Challenge

The Capture the Flag events at a glance

Capture the Flag is most known as a game you used to play when you were kids. The field is divided into two camps, and the goal of your team is to steal the opponent’s flag and bring it to your own camp. Although that version of CTF is a lot of fun, the context in Cyber Security is slightly different. In a security CTF, flags can be stored on a vulnerable webserver, compiled into malicious executables, or encrypted using flawed cryptography. Teams then need to solve the various challenges using very broad skills to get the flag and score the points. 

CTFs have been very popular in the information security field for a long time – the DefCon CTF has been organized since 1996! – and are a great way to learn new skillsets, hang out with friends and colleagues and generally have a great time. The rush of finally getting that flag after hours (or days) or work really gets the adrenaline flowing. 😉  

CTFs are very popular as well. If you want, you can play one almost each week(end), often even multiple CTFs are running at the same time! For an overview of all CTFs, you can take a look at ctftime.org

Why do we organize ‘yet another CTF’? 

With a CTF being organized every week, why would we want to add yet another one? Well, the goal of our CTFs is quite different than a typical CTF. Most CTFs act as a competition for experienced security professionals, where incredibly skilled hackers show off their skills and take home the prizes. When we started organizing the first CTF in Belgium in 2015, there was just one goal: Get more students into the information security community. 

It’s no secret that the industry is desperately searching for more motivated people to join us, and positions often stay vacant for a long time. Universities and colleges often offer security courses, but the amount of students that actually end up joining the information security sector is rather low.  

With our CTF, we want to show students that: 

  • Hacking is fun (Who doesn’t like breaking stuff?) 
  • General computer skills and the right attitude can take you very far 
  • Even though it looks like a niche market, the cyber security field is very broad with many different aspects 

As our target audience, we chose all graduating students from local colleges and universities, as they will most likely be choosing a career after graduating and it would be nice if we can push them into our direction 😎

But this ain’t no ordinary CTF 

To reach our goal, we’ve created the Challenge in Belgium. We chose for a jeopardy-style CTF (as opposed to an attack/defense style) to keep the entry level low and give us the possibility to introduce a wide range of challenges to students.  

A participant at the Rumble 2019 life-event

While the core of both the Challenge and the Rumble is a CTF, there’s a little bit more to it to accommodate these sub goals. 

The first one is probably the easiest. Each year, we contact everyone we know in the Belgian/German infosec field and ask if they want to create a challenge. By outsourcing challenge creation, we can both shine a spotlight on talented individuals, as make sure that there is a very wide range of challenges to solve. 

Testing social skills is quite difficult for a CTF, as contestants typically sit behind their laptop screen for the entirety of the competition, and don’t really have to interact with other contestants or the organizers. To add this aspect to our event, we came up with the concept of challenges created by our sponsors. For these challenges, the qualifying teams have to face a panel of experts where they have to solve problems interactively. We’ve had live forensics investigations, incident response roll-playing, debates on the pros/cons of a cashless society, and calling up people to social engineer them into giving you valuable information.  

These challenges also automatically allow students and future employers to interact, which is a double win. 

Expanding to Germany 

After 6 years, the Cyber Security Challenge in Belgium is reaching over 700 students from more than 30 schools and the Challenge is even used as a preselection for the Belgian team for the European Cyber Security Challenge, organized by ENISA. Due to this success  and the interest of the industry, NVISO launched a sister event in Germany in 2019, called the Cyber Security Rumble. With the focus on mainly German academic students, the event was set up in cooperation with RedRocket (a famous German CTF team), the University of Bonn-Rhein-Sieg, SANS, and the German Federal Office for Information Security. The collaboration between these parties already shows that the goal remains to have the CTF driven by the community, and not by a single company.  

Even though the Challenge in Belgium had been organized successfully for quite a few years, it was still a gamble to see if Germany was as receptive to the students-only concept. Luckily, the first year managed to reach 300 participants in the qualifier rounds, from which 13 teams made it into the finals.  

The Challenge and Rumble in 2020 

The organization of the latest edition of the Cyber Security Challenge & Rumble was, as with all other events in 2020, defined by the COVID pandemic. While we love the interaction we have with the students during each edition, it was clear that we had to move to an online-only event to make sure everyone can stay safe. 

For the Challenge in Belgium, we decided to open the finals CTF to all the students that would have qualified for our computer-less CTF, and once again the top 12 teams would continue on day 2 with interactive challenges, this time in an online format. The online format took a lot more work on the day itself, as we needed to make sure everyone was joining (and leaving 😉) the correct meeting rooms. Discord allowed us to interact directly with students in case there were issues or questions, and also helped to still have a relaxed atmosphere in the general channels. The second day ended with an online prize ceremony, where all top 12 teams received their prizes, such as a trip to DefCon Las Vegas, a SANS course and much more.  

The German Rumble, in turn, was a full two-day online event organized on Halloween and welcomed more than 470 active teams, both German academic teams as well as international teams. By also communicating with the participants via a Discord chat, the players could get in contact with the sponsors that created the challenges and to interact with other participants about the challenges. Moreover, a scoreboard showed the progress and listing of the teams so that the speed and team spirit was cheered up a little more. Also the Rumble was rounded off with a prize ceremony, in which a representative of SANS announced the prizes.  

Tweet from the Rumble during it’s online prize ceremony

The challenges we still face each year 

There are various challenges and questions that pop up each year. While we don’t have a solid answer on all of them, we still want to share them, and any input in the comments is of course appreciated! 

Reaching students 

Although both the Challenge and the Rumble have grown in popularity, it’s a very large effort each time to reach all the students. We have to actively communicate with professors, schools and student unions to make sure students participate, often even visiting schools and presenting our challenge in security-focussed courses.  

Keeping the competition fair for everyone 

With such awesome prizes on the line, there’s always the possibility of teams collaborating, sharing solutions or flags. This is something that’s hard to prevent, although we do have various technical checks in place to detect weird behaviour. Additionally, we try to rely on the schools to do the right thing. Some schools even organize a small on-campus event during the qualifiers so that teams can be in the same room. However, through our good connections with the relevant professors, we can be sure that students are behaving and that we don’t have to fear dishonest collaboration. 

A participant in this year’s online Challenge 

Keeping it students only 

Another issue that regularly pops up is how we define a student. For example: Can PhD students participate? Technically they are students, with a valid student card. In practice, they would have a huge advantage over other students. Similarly, what if someone who has been in the industry for many years decides to join an online course at a registered university/college? Can they join? The hardest part here is being consistent while also being fair to everyone involved… 

NVISO as the common organizer

With our efforts to organize these great initiatives and thus to enhance the Cyber Security Communities in both countries, we are constantly supporting cross border activities. Both can learn from each other, are in constant communication and help to drive individual events to their success. We’re happy that both events can reach a substantial number of students and that we create interactivity between Belgium and Germany.  

Come join us! 

If you’re a cyber security specialist in Belgium or Germany, we’d love your help in creating challenges. It’s a great way to show your skills and connect with other challenge creators, sponsors and of course the awesome organizing team.  

And of course, if you’re still a Belgian/German student, don’t hesitate and sign up for either the Challenge or Rumble and take home some of the awesome prizes. 😊 

If you are not convinced yet, check out our after movies and catch a glimpse of the sphere of the last years: 

After movie Cyber Security Challenge Belgium

After movie Cyber Security Rumble Germany

Stay tuned for the events in 2021 and for exciting and fun challenges to crack!   

About the authors

This article was jointly written by:

  • Annika ten Velden, Operations Manager
  • Marina Hirschberger, Senior Consultant
  • Jeroen Beckers, Mobile security expert

They are all working at NVISO and are actively contributing to the organization of the events. While Annika and Jeroen are taking care of the Challenge in Belgium, Marina is part of the organization team of the Rumble in Germany. 

NVISO and QuoIntelligence Announce Strategic Cooperation

30 October 2020 at 10:51

We are pleased to announce that we have created a unique approach with QuoIntelligence GmbH in responding to the TIBER-EU testing. Using our approach, we combine both passive threat intelligence gathering and active offensive red team testing as one seamless experience while remaining independent from each other.  

The TIBER-EU Framework, More Critical Now Than Ever 

The constant evolution of the cyber threat landscape combined with the recent acceleration of the financial sector’s digital transformation, led by new global challenges such as the COVID-19 pandemic, brings new complex cyber threats using more advanced methods and techniques. Financial institutions can better face these evolving threats and aim to reach a more secure digital environment by putting in place the right cyber and operational resilience strategies early on. 

In order to test and improve the cyber resilience of financial institutions, the European Central Bank developed a framework for ‘Threat Intelligence Based Ethical Red Teaming’, commonly known as TIBER-EU framework, to carry out a controlled cyberattack based on real-life threat scenarios. TIBER-EU exercises are designed for entities which are part of the core financial infrastructure at the national or European level.

“It is the first EU-wide guide on how authorities, entities, threat intelligence and red-team providers should work together to test and improve the cyber resilience of entities by carrying out a controlled cyberattack.”  – Fiona van Echelpoel, Deputy Director General at ECB 

By conducting a TIBER-EU test, institutions can enhance their cyber and operational resilience and operational resilience by focusing on technology, monitoring and human awareness strengths & weaknesses before they are exploited by real-life threat actors. The exercise’s main objective is to test and improve protection, detection, and response capabilities against sophisticated cyber threats. Having a TIBER-EU test implemented, European organizations will then be able to reduce the impact of potential cyberattacks.

Source: Lessons Learned and Evolving Practices of the TIBER Framework

Benefits for European Organizations 

Since the TIBER-EU testing process can be quite overwhelming for the testing entities, selecting the right qualified providers is the first step towards a successful experience and resourceful outcome. The combined work and fluent integrations and communications between the Threat Intelligence and Red Teaming providers is crucial to implement optimal strategies tailored to the testing entity’s cyber strength and weaknesses. 

For this reason, we at NVISO are cooperating with QuoIntelligence GmbH, a German Threat Intelligence provider supporting decision-makers with customized and actionable intelligence reports,, to facilitate the cyber resilience testing process. Within this approach, QuoIntelligence first looks at the range of possible threats, selects the most applicable threat actors likely to target the entity, and creates a customized Targeted Threat Intelligence Report which lays the foundation for the Red Teaming’s attack scenarios. Then, NVISO, as the Red Teaming provider, carries out the simulated attack and attempts to compromise the critical functions of the entity by mimicking one of the real-life threat actors in scope.

In cooperation with QuoIntelligence, we already implemented effective joint processes and offer a seamless experience between the Threat Intelligence and Red Teaming providers. Organizations can then take the worry out of the process and be led by experienced providers. 

Conclusion

Cybersecurity risks are becoming harder to assess and interpret due to the growing complexity of the threat landscape, adversarial ecosystem, and expansion of the attack surface.

“The expansion of knowledge and expertise in cybersecurity is crucial to improve preparedness and resilience. The EU should continue building capacity through the investment in cybersecurity training programs, professional certification, exercises and awareness campaigns.”  – ENISA Threat Landscape Report 2020 

In order to test and improve the cyber resilience of the European financial sector, the European Central Bank has put in place the TIBER-EU framework involving a close collaboration between a Threat Intelligence provider and a Red Teaming provider.

QuoIntelligence and NVISO are now offering a strategic approach to simplify the TIBER-EU testing process and offer a worry-free experience to European organizations that want to take their cyber and operational resilience to the next level.

Authors and contact

In case of questions and for more information, please contact [email protected].

This article was written by Marina Hirschberger, Senior Security Consultant, in accordance with Jonas Bauters, Solution Lead for Red Teaming at NVISO and in cooperation with Iris Fernandez , Marketing Expert at QuoIntelligence GmbH.

  • There are no more articles
❌