How to read Windows serialized certificates (with code sample)

Problem description

On a Windows machine, we can find users’ certificates stored in files in C:\Users\<USER>\AppData\Roaming\Microsoft\SystemCertificates\My\Certificates (i.e. “%APPDATA%\Microsoft\SystemCertificates\My\Certificates”). These files have seemingly random names (i.e. “3B86DFC25CFB1B47EB4CBF53FD4028239D0C690E”) and no extension. What is their format? How to open them in code? With which Windows APIs? 🤔

Let me spoil you with the answers right away, including code samples, and I’ll describe after what I tried and what I learned 💡

Answer: “serialized certificates” that can be opened using the CryptQueryObject() function

These files are “serialized certificates”. Surprisingly, even with this knowledge which wasn’t easy to discover, I did not find any Windows CryptoAPI function to directly open them!

Until I found CryptQueryObject: a very handy function that can open crypto objects with different formats. We can specify with the “dwExpectedContentTypeFlags” parameter the format(s) we expect, or accept all formats, and see what it detects. It returns notably:

• pdwContentType: equal to “CERT_QUERY_CONTENT_SERIALIZED_CERT” in this case meaning that “the content is a serialized single certificate.”
• ppvContext: pointer to a CERT_CONTEXT structure, in this case of a serialized certificate, which contains in particular:
• pCertInfo: many metadata on the certificate with a CERT_INFO structure
• pbCertEncoded: the certificate itself, so what we would expect to find in a classic .crt file

Simplified example usage:

CERT_CONTEXT* certContext = NULL;
if (!CryptQueryObject(
 CERT_QUERY_OBJECT_FILE,
 L"C:\\Users\\localuser1\\AppData\\Roaming\\Microsoft\\SystemCertificates\\My\\Certificates\\3B86DFC25CFB1B47EB4CBF53FD4028239D0C690E",
 CERT_QUERY_CONTENT_FLAG_ALL,
 CERT_QUERY_FORMAT_FLAG_ALL,
 0,
 NULL,
 NULL,
 NULL,
 NULL,
 NULL,
 (const void**)&certContext
) || certContext == NULL)
{
 if (certContext) CertFreeCertificateContext(certContext);
 return false;
}

Alternative with the CertAddSerializedElementToStore() function

There’s also an alternative. By searching for CryptoAPI functions related to “serialized certificates” we can find this function: CertAddSerializedElementToStore. It can deal with such certificates but only to load them into a store… So, the idea is to:

1. create a temporary store in memory, using CertOpenStore with “CERT_STORE_PROV_MEMORY” and “CERT_STORE_CREATE_NEW_FLAG”
2. load the serialized certificate into this temp store, using CertAddSerializedElementToStore
3. this function returns the desired CERT_CONTEXT structure of the certificate (like above) in “ppvContext”

It works properly and we get the same results, but it’s longer and less efficient I think.

How did I find that they are “serialized certificates”?

I found a comment online saying that we can open them in Windows by assigning them the “.sst” extension, which then allows to open them with a double-click. We can see in the explorer that this extension corresponds to “Microsoft Serialized Certificate Store”.

Knowing this, I found the CertOpenStore CryptoAPI function that seemed capable of opening those “Microsoft Serialized Certificate Store” files, but it refused to open this file…

I didn’t understand why, so I created a certificate store in memory and used the CertSaveStore function to export it as a serialized certificate store. And indeed, its content did not have exactly the same format. There was some header at the beginning, before the content with the same format as the one I had in the files I wanted to analyze. My guess was that this header was the certificate store header, and the rest was actually just the serialized certificate saved in the store! And this guess was correct based on the results I got afterwards 😉

Of course I also tried first to load these files with other more common extensions, like .crt, .pfx, .p12, etc. but none worked.

Why not use CertEnumCertificatesInStore?

My initial need was to enumerate the certificates for all users on the machine (provided my code is running privileged of course) so I tried first to use CertOpenStore targeting the “CERT_SYSTEM_STORE_USERS” system store. But when enumerating the certificates, with CertEnumCertificatesInStore, it did not return these certificates that I knew existed since I could see them in the certificates manager (certmgr.msc) when logged in as each user.

I discovered this issue when using Benjamin @gentilkiwi Delpy’s “mimikatz” tool. Of course Benjamin loves certificates and so he included an entire “crypto” module in his famous tool. (Yeah, it’s a good reminder that it has many other usages than just dumping credentials! 😉). The “crypto::certificates” command, which uses CertEnumCertificatesInStore, could not find any certificate in the “My” certificate store of another user accessed through the “CERT_SYSTEM_STORE_USERS” system store and as admin of course:

Even though there was indeed a certificate to see:

Actually, I could find the certificates when running as each user, and targeting the “CERT_SYSTEM_STORE_CURRENT_USER” system store:

So, it confirmed that the “CERT_SYSTEM_STORE_USERS” system store has a limitation. The only online confirmation I found is an 18 years old 😯 newsgroup post from a then Microsoft employee:

CERT_SYSTEM_STORE_USERS opens the registry stroes. so you can NOT use MY store with it.

What I noticed too is that, when using “CERT_SYSTEM_STORE_USERS”, it only goes looking for certificates into the registry only, and there’s none in this case. So these certificates, that are on disk only, are missed when using “CERT_SYSTEM_STORE_USERS”:

Whereas, it looks for certificates in the registry and on disk when using “CERT_SYSTEM_STORE_CURRENT_USER”:

Alternatives for parsing these certificates without the CryptoAPI

In particular, Benjamin @gentilkiwi Delpy kindly answered my question, and told me that there is the “crypto::system” mimikatz command which allows to parse these certificates, like this:

The code shows that he actually implemented the entire parsing himself without relying on Windows APIs! This is very interesting to discover how it works, and it can also be helpful for research, but I preferred to stick to the official CryptoAPI functions, or at least Windows APIs, to open these certificates. However, this alternative is worth mentioning!

Edit: it was brought to my attention that this article “Extracting Certificates From the Windows Registry” may cover the same topic, but I did not double-check their results. I also preferred to use an official Windows API instead of a custom parsing.

Code for reading Windows serialized certificates was originally published in Tenable TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Summary

We investigated a situation where an SMB client could not connect to an SMB server. The SMB server returned an “Access Denied” during the NTLM authentication, even though the credentials were correct and there were no restrictions on both the server-side share and client-side (notably UNC Hardened Access). The only unusual thing is that the SMB server was accessed through a NAT mapping (DNAT to be precise): the client was connecting to an IP which was not the real server’s IP. This can happen in some VPN network setups. Also, we have seen this situation at some organizations (even without a VPN in the equation) where they request to connect to machines, such as domain controllers, through a unique Virtual IP (VIP) which allows load-balancing.

💡 As cybersecurity experts, this immediately made us think that this setup was in fact similar to an NTLM relay (aka SMB relay) attack, even though the intent was not malicious. And perhaps there could be a security hardening mechanism on the server side blocking this.

And indeed we were correct: the server had the “Microsoft network server: Server SPN target name validation level” policy (i.e. SmbServerNameHardeningLevel registry key) enabled which blocked this scenario. Here is the policy description from Microsoft:

This policy setting controls the level of validation that a server with shared folders or printers performs on the service principal name (SPN) that is provided by the client device when the client device establishes a session by using the Server Message Block (SMB) protocol. The level of validation can help prevent a class of attacks against SMB services (referred to as SMB relay attacks). This setting affects both SMB1 and SMB2.

➡️ This situation could also occur in your regular SMB environments, so follow along to see how to troubleshoot this, how it is configured, how it works and what we suggest to do in this case.

Observation

Here’s an example in this screenshot (sorry for the French UI machine on the right!):

The SMB client, on the left (IP 10.10.10.20), is trying to connect to the SMB server on the right (IP 10.0.0.11 and FQDN dcfr.lab.lan), except it’s through the IP of a TCP relay (created with socat on Linux), at the bottom (IP 10.0.0.100) which simulates the NAT situation seen initially in our investigation.

So, the SMB server sees an incoming authentication, where the SMB client has declared (in the “Target Name” attribute on the left) it is expecting to authenticate to the IP of the TCP relay (10.0.0.100), which is different than the real server’s IP (10.0.0.11).

💥 Therefore, it detects the mismatch considered as an attack attempt, and denies the authentication right away, as we can see with the “Access is denied” error message and “STATUS_ACCESS_DENIED” in the SMB network capture.

With the same setup and server configuration, if the client connects directly to the server’s IP (10.0.0.11) without the relay, all is matching and it works:

How to troubleshoot?

551 “SMB Session Authentication Failure” event

The first hint in identifying this issue is that it generates a 551 “SMB Session Authentication Failure” event in the SMBServer event log (as seen in the first screenshot above).

5168 “SPN check for SMB/SMB2 failed” event

There is also a 5168 Security event “SPN check for SMB/SMB2 failed”, where we clearly see the IP address that was sent (the SPN, in red) Vs. what was expected (the semicolon-separated list, in green), sorry again for the French UI:

Note that for the 5168 event to be generated, the “Audit File Share” audit policy must be enabled for Failure at least. You can check with:

auditpol.exe /get /SubCategory:"Detailed File Share"

We can also have the same 5168 event generated “because of NTLMv1 or LM protocols usage” since they don’t carry the required SPN attribute for the server to do its check.

Policy

✅ We can also check if the “Microsoft network server: Server SPN target name validation level” policy is enabled (for those following in French: “Serveur réseau Microsoft: niveau de validation du nom de la cible de serveur SPN”).

The corresponding registry key is SmbServerNameHardeningLevel found in “HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\LanManServer\Parameters”

We can query it with:

reg query HKLM\System\CurrentControlSet\Services\LanManServer\Parameters\ /v SmbServerNameHardeningLevel

Or using this dedicated PowerShell cmdlet:

Get-SmbServerConfiguration | fl *hard*

See below for the explanation of the possible values.

How to configure the policy?

⚙️ The “Microsoft network server: Server SPN target name validation level” policy has three possible values:

• 0[default] = “Off”
  “The SPN is not required or validated by the SMB server from a SMB client.”
• 1 = “Accept if provided by client”
  “The SMB server will accept and validate the SPN provided by the SMB client and allow a session to be established if it matches the SMB server’s list of SPN’s for itself. If the SPN does NOT match, the session request for that SMB client will be denied.”
• 2 = “Required from client”
  “The SMB client MUST send a SPN name in session setup, and the SPN name provided MUST match the SMB server that is being requested to establish a connection. If no SPN is provided by the client, or the SPN provided does not match, the session is denied.”

In our testing, we observed access denied errors in such a relay/NAT situation, with either the values of 1 or 2, because the Windows SMB client knows to provide the expected SPN. However, setting the registry key to 0 disables the protection and indeed it made the connection possible even through the relay.

How does this protection work?

Protocol support

Perhaps you have noticed something strange: here we can see an “SPN” in the context of an NTLM authentication… Whereas usually SPN only appears within the context of Kerberos! 🤔

The NTLM specification, [MS-NLMP] clearly uses this term:

MsvAvTargetName: The SPN of the target server.

Also, as described in the 5168 event:

It often happens because of NTLMv1 or LM protocols usage from client side when “Microsoft Network Server: Server SPN target name validation level” group policy set to “Require from client” on server side. SPN only sent to server when NTLMv2 or Kerberos protocols are used, and after that SPN can be validated.

Indeed, NTLMv1 and LM protocols don’t have the required fields to carry the SPN expected and provided by the client.

Of course, this security mechanism works with Kerberos since service tickets embed an SPN.

Protection against NTLM relaying

📄 NTLM relay attacks, sometimes called SMB relay attacks, have been well-known for many years. I recommend these great articles if you want to learn more: https://en.hackndo.com/ntlm-relay/ and https://www.thehacker.recipes/ad/movement/ntlm/relay

During such an attack, the client authenticates to the attacker’s machine, which relays it to another machine (like in a Man-in-the-Middle attack), which is the attacker’s real target. But thanks to this additional SPN attribute, the client declares the server it’s expecting to authenticate to, which would be the attacker’s IP, and when the target server receives the relayed authentication it can detect that there’s a mismatch (the declared IP isn’t its own) and denies the authentication. Of course, it works with hostnames and FQDNs instead of IPs.

This protection is also explained in this section of the same article: https://en.hackndo.com/ntlm-relay/#service-binding

Offensive security perspective

An SMB client can be modified to send a correct target name, for example, using the impacket library as described in this article. But this doesn’t make this protection useless in the context of an NTLM relay attack, as the attacker cannot modify the SMB client used by the victim.

🔒 Moreover, this SPN attribute cannot be removed nor modified during an NTLM relay attack because it belongs to the attributes list (AV_PAIR), which is protected by the MIC as described in many articles, including this recent one from Synacktiv about the NTLM EPA protection.

What do we recommend?

🛡️ Of course, as cybersecurity experts, we do not recommend to remove this hardening feature that is usually enabled for good reason! Many cybersecurity agencies encourage evaluating this policy and enabling it where possible, as described in many security standards that Tenable products allow to audit.

As described previously, we could also create our own SMB client to send a crafted, but correct, SPN value, but obviously this solution is not possible in most cases…

1. The easiest solution, when possible, is to connect to the server directly, using its real IP (i.e., without NAT).
2. Otherwise, there is a registry key which allows for declaring of a list of alternative names and IPs allowed through this mechanism. It is the SrvAllowedServerNames key, which must be created in “HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters” with type REG_MULTI_SZ. This is described in this Microsoft support article “Description of the update that implements Extended Protection for Authentication in the Server service” and in this answer on ServerFault.
   We confirm it works (with both values enabling the policy):

SMB “Access is denied” caused by anti-NTLM relay protection was originally published in Tenable TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

I often use Wireshark to analyze Windows and Active Directory network protocols, especially those juicy RPC 😉 But I’m often interrupted in my enthusiasm by the payload dissected as “encrypted stub data”:

Can we decrypt this “encrypted stub data?” 🤔

The answer is: yes, we can! 💪 We can also decrypt Kerberos exchanges, TGTs and service tickets, etc! And same for NTLM/NTLMSSP, as I will show you near the end. Read along to learn how to decrypt DCE/RPC in Wireshark.

Wait, is that magic?

Wireshark is very powerful, as we know, but how can it decrypt data? Actually there’s no magic required because we’ll just give it the keys it needs.

The key depends on the chosen algorithm (RC4, AES128, AES256…) during the Kerberos exchange, and they derive from the password (this is simplified but you didn’t come here to read the Kerberos RFC, right? 🤓).

My preferred method to get the Kerberos keys is to use mimikatz DCSync for the target user:

You’ll directly notice the AES256, AES128, and DES keys at the bottom, but what about the RC4 key? As you may have guessed, it’s simply the NT hash 😉

Just remember that modern Windows environments will likely use AES256 so that’s what we’ll target.

Keep tabs on the keys

Kerberos keys are commonly stored in “keytab” files, especially on Linux systems. By the way, if you find a keytab during a pentest, don’t forget to extract its keys because you’ll be able to create a silver ticket against the service, as I once did (see below ️⬇️️), or access other services with this identity.

Clément Notin on Twitter: "#Pentest success story:1. Steal .keytab file from a Linux server for a webapp using Kerberos authentication🕵️2. Extract Kerberos service encryption key using https://t.co/itX7S337o03. Create silver ticket using #mimikatz🥝 and pass-the-ticket4. Browse the target5. Profit!😉 pic.twitter.com/yI9yfoXDrb / Twitter"

Pentest success story:1. Steal .keytab file from a Linux server for a webapp using Kerberos authentication🕵️2. Extract Kerberos service encryption key using https://t.co/itX7S337o03. Create silver ticket using #mimikatz🥝 and pass-the-ticket4. Browse the target5. Profit!😉 pic.twitter.com/yI9yfoXDrb

So it’s no surprise that Wireshark expects its keys in a keytab too. It’s a binary format which can contain several keys, for different encryption algorithms, and potentially for different users.

Wireshark wiki describes how to create the keytab file, using various tools like ktutil. But the one I found the most convenient is keytab.py, by Dirk-jan @_dirkjan Mollema, who wrote it to decrypt Kerberos in his research on Active Directory forest trusts. I especially like that it doesn’t ask for the cleartext password, just the raw keys, contrary to most other tools.

First, download keytab.py (you don’t even need the entire repo). Additionally, install impacket if you have not already done so.

Then, open the script and edit lines 112 to 118 and add all the keys you have (in hexadecimal format) with the number corresponding to their type. For example, as we said, most of the time AES256 is used, corresponding to type 18.

The more keys you have, the better 🎉 If you are hesitant, you can even include the RC4 and AES256 keys for the same user. As Dirk-jan comments in the code, you can include the “krbtgt” key, “user” keys (belonging to the client user), “service” keys (belonging to the service user), and even “trust” keys (if you want to decrypt referral tickets in inter-realm Kerberos authentications). You can also add “computer account” keys to decrypt machines’ Kerberos communications (machine accounts in AD are users after all! Just don’t forget the dollar at the end when requesting their keys with DCSync). You don’t need to worry about the corresponding username or domain name in the keytab; it doesn’t matter for Wireshark.

Finally, run the script and pass the output filename as argument:

$ python keytab.py keytab.kt

Back to Wireshark

Configuration

Now that you have the keytab, open the Wireshark Preferences window, and under Protocols, look for “KRB5”.

Check “Try to decrypt Kerberos blobs” and Browse to the location of the keytab file you just generated.

Decrypt Kerberos

Now you can try opening some Kerberos exchanges. Everything that is properly decrypted will be highlighted in light blue. Here are a couple examples:

⚠️ If you notice parts highlighted in yellow it means that the decryption failed. Perhaps the corresponding key is missing in the keytab, or its value for the selected algorithm was not provided (check the “etype” field to see which algorithm is used). For example:

👩‍🎓 Surprise test about Kerberos theory: can you guess whose key I provided here, and whose key is missing?

Answer: We observe that Wireshark can decrypt the first part which is the TGT encrypted with the KDC key, but it cannot decrypt the second part which is encrypted with the client’s key. Therefore, here the keytab only contains the krbtgt key.

Decrypt DCE/RPC, LDAP…

Do you remember how this all began? I wanted to decrypt DCERPC payloads, not the Kerberos protocol itself!

And… it works too! 💥

Quick reminder first, the same color rule applies: blue means that decryption is ok, and yellow means errors. If you see some yellow during the authentication phase of the protocol (here the Bind step) the rest will certainly cannot be decrypted:

Here are some examples where it works, notice how the “encrypted stub data” is now replaced with “decrypted stub data” 🏆

It also works with other protocols, like LDAP:

Tip to refresh the keytab

A modified keytab file does not take effect immediately in Wireshark. Either you have to open the Preferences, disable Kerberos decryption, confirm, then re-open it to re-enable it, which is slow and annoying… Or the fastest I’ve found is to save the capture, close Wireshark and re-open the capture file.

NTLM decryption

What about NTLM? Can we do the same decryption if NTLMSSP authentication is used? The answer is yes! 🙂

In the Preferences, scroll to the “NTLMSSP” protocol, and type the cleartext password in the “NT Password” field. This is described in the Wireshark NTLMSSP wiki page where I have added some examples. Some limitations contrary to Kerberos: you need the cleartext password and it must be ASCII only (this limitation is mentioned in the source code) so it is not applicable to machine account passwords, and you can only provide one at a time, contrary to the keytab which can hold keys for several users.

Update: actually, it is possible to decrypt using NTLM hash(es)! This feature is not documented, and not possible through the UI, but by looking at the code we can see that it is indeed possible as described in this CTF writeup: Insomni’Hack Teaser 2023 — Autopsy.
How to provide the NT hash(es)? Using a keytab too! It’s a bit confusing to use a Kerberos option to decrypt NTLMSSP but it works. If you remember earlier, I said that the RC4 key to put in a keytab is identical to the NT hash. So, you have to create a keytab entry, as explained previously, using the RC4-HMAC type (etype 23) and with the NT hash. Enable it in the Wireshark KRB5 options, same as before, and your NTLM encrypted trafic will be in clear-text if the hash is correct.

Conclusion

I hope these tips will help you in your journey to examine “encrypted stub data” payloads using Wireshark. This is something that we often do at Tenable when doing research on Active Directory, and I hope it will benefit you too!

Protocols become increasingly encrypted by default, which is a very good thing… Therefore, packet capture analysis, without decryption capabilities, will become less and less useful, and I’m thankful to see those tools including such features. Do you know other protocols that Wireshark can decrypt? Or perhaps with other tools?

Decrypt Kerberos/NTLM “encrypted stub data” in Wireshark was originally published in Tenable TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.