As I wrote about last week, there are holiday shopping-related scams already popping up all over the place.  

But another aspect of security that many shoppers don’t consider this time of year is the security of the products they’re buying, even through a legitimate online marketplace. 

This is a glaring issue with home security cameras and Wi-Fi-connected doorbells, but I can’t imagine these are particularly popular holiday gifts. With virtually everything being connected to the internet somehow these days, everything is a potential security risk if you’re buying a new piece of technology. 

Take smartwatches, for example. Apple Watches and Samsung Galaxy watches are always popular on everyone’s wishlists this time of year because they’re high-priced items you normally wouldn’t buy for yourself. Many shoppers might be looking for a deal this time of year and not looking to spend hundreds on the gift, so any sort of cheaper alternative could be appealing. 

I searched for “smart watches” on Amazon, and the results page displayed four different watches from four different vendors as their “Top Results,” none of which were Samsung and Apple. Well-known vendors are certainly not immune to security issues or vulnerabilities, but at least users can be confident that any known vulnerabilities will be disclosed and patched by these companies as they pop up. 

The top result is for a $29.99 smartwatch that offers sleep tracking, blood pressure monitoring, dozens of different workout modes, step tracking, and more. However, there are a few security flags for me right up front with this deal (after all, if it seems too good to be true, it probably is). Amazon states the seller is a company called “Nerunsa,” but a quick search did not turn up any legitimate information on who this company is, where they’re based, or the sort of security bona fides you’d be hoping for. The only search results are for the company’s Amazon store page and a few eBay listings for people reselling the watch in question. 

The app that’s listed as supporting the watch is called “GloryFit” on the Google Play and Apple app stores, and its privacy policy is equally vague. It states that the app will collect all the suspected information for someone using a smartwatch — phone calls, text messages, GPS location, personal information, health information, etc. But, the policy states that, when the user accepts the privacy policy, “You hereby consent to our process and disclose personal information to our affiliated companies (which are in the communications, social media, technology and cloud businesses) and to Third Party Service Providers for the purposes of this Privacy Policy.” And it’s not particularly clear what those other companies do, exactly — Google was no help here, either. 

Apple Air Tags are also another popular tech gift every year and are usually featured in major retailers’ Black Friday sales. I personally have my own concerns about any type of tracking tag coming into my house, but that’s for another column. 

On Walmart, which is increasingly trying to compete with Amazon by offering more products online, I searched for “smart tag” and found three results that appeared ahead of Apple’s legitimate Air Tags. The second-most-popular result is for a “Bluetooth Tracker and Item Locator” that’s only $15.98, compared to $86.88 for a four-pack of Apple’s. This tracker is listed as being made by “AILIUTOP,” which also remains elusive on the internet and does not seem to have any sort of legitimate contact information available to the public. Their store page on Walmart indicates the seller offers many types of products, from clothing to home goods and more.  

 This seems like a good bargain as a gift for someone who is always losing their keys or wallet or wants to make sure their bicycle is secure when they lock it up somewhere. But purchasing these types of “smart” devices with so much uncertainty poses a few issues. 

If you do experience some sort of security failure or issue, there is no easy way to contact any of these vendors through the traditional means that the average user would go searching for. These vendors have no clear history of responsibly disclosing vulnerabilities, releasing security updates, or testing their products’ security before release. 

When these types of gifts are dealing with such high-profile information like your personal information, health data, or physical location, users should be confident that their information is being stored correctly and securely, or at least there’s a way to contact the vendor should they have any questions. 

When searching for holiday gifts online, make sure you’re buying from a trusted vendor, or if you haven’t heard of the vendor before, take a few extra minutes just to look them up, read their app’s privacy policy, or even read the reviews to make sure there’s no clear sign of bot activity like repetitive words or phrases or using the same photo for multiple reviews.  

The one big thing 

The 2023 Cisco Talos Year in Review is now available to download. Once again, the Talos team has meticulously combed through a massive amount of data to analyze the major trends that have shaped the threat landscape in 2023. Global conflict influenced a lot of these trends, altering the tactics and approaches of many threat actors. In operations ranging from espionage to cybercrime, we’ve seen geopolitical events have a significant impact on the way these are carried out. 

Why do I care? 

The Year in Review report includes new data and telemetry from Talos about attacker trends, popular malware seen in the wild, and much more. Despite the accelerated pace of many threat actor campaigns and the geopolitical events that shaped them, our report shows that the defensive community’s diligence, inventiveness and collaborative efforts are helping to push adversaries back.   

So now what? 

Download our full report here, bookmark the Year in Review landing page for future content we have planned around the report, and listen to the Beers with Talos episode that covers the details of the report. 

Top security headlines of the week 

More than six million people are reportedly victims of a large data breach at DNA and genealogy testing firm 23andMe. The breach is larger than initially expected, with more than 5.5 million users who opted into the company’s “DNA Relatives” feature, which allows customers to automatically share some of their data with other users. Another 1 million-plus users had their family tree information accessed. The attackers accessed the accounts because of password reuse from users, likely who used easy-to-guess login information or passwords they used across multiple other accounts. 23andMe was not the target of the initial breach, nor was a company account the source of the compromised credentials. Security experts are urging users to move away from traditional username-and-password login methods as these types of attacks happen more often, instead moving toward multi-factor authentication or passwordless logins. (TechCrunch, Wall Street Journal) 

Apple released emergency fixes for two zero-day vulnerabilities in its WebKit browser engine that have already been exploited in the wild. The company reported that the flaws are being exploited on devices running on iOS versions before iOS 16.7.1 (released on Oct. 10, 2023). There are new patches available, which users should install immediately, in iOS, iPadOS, macOS Sonoma and the Safari web browser. The two vulnerabilities tracked as CVE-2023-42916 and CVE-2023-42917, leave affected devices vulnerable to adversaries accessing sensitive information on targeted devices. CVE-2023-42917 could also allow an attacker to execute arbitrary code on the targeted machine. (SC Magazine, Decipher) 

Security researchers say a new threat actor known as “AeroBlade” compromised a U.S. aerospace company for more than a year. The actor reportedly started testing their malware and infection chain on the targeted network in September 2022 and executed malware on the network in July 2023. The activity sat undetected for months due to anti-analysis techniques. It is currently unknown what actions, if any, the actor carried out during that time or if they compromised any user or customer data. The initial infection began with a Microsoft Word lure document with the title, “"SOMETHING WENT WRONG Enable Content to load the document." The ensuing malicious Microsoft Word template (DOTM) file then loaded a DLL that served as a reverse shell. Researchers say the attacker’s intent was likely to steal data from the target to sell it, potentially supply it to international competitors, or use it to extort the target into paying a ransom. (Dark Reading, Bleeping Computer)  

Can’t get enough Talos? 

Security journalists from Decipher bring you the headlines, including new U.S. government sanctions on threat actor groups in our latest Threat Spotlight video.

Then, Hazel chats to Talos security researcher Joe Marshall to discuss the Talos 2023 Year in Review, and Project PowerUp, the story of how Cisco Talos worked with a multi-national, multi-company coalition of volunteers and experts to help “keep the lights on” in Ukraine, by injecting a measure of stability in Ukraine’s power transmission grid.

Upcoming events where you can find Talos 

"Power of the Platform” by Cisco (Dec. 5 & 7) 

Virtual (Please note: This presentation will only be given in German) 

The annual IT event at the end of the year where Cisco experts, including Gergana Karadzhova-Dangela from Cisco Talos Incident Response, discuss the future-oriented topics in the implementation of digitalization together with you.  

What Threats Kept Us Up in 2023: A Year in Review and a Look Ahead (Dec. 13, 11 a.m. PT) 

Virtual 

Each year brings new threats that take advantage of increasingly complex security environments. Whether it’s Volt Typhoon targeting critical infrastructure organizations across the United States or ALPHV launching an attack against casino giant MGM, threat actors are becoming bolder and more evasive. That’s why it’s never been more important to leverage broad telemetry sources, deep network insights and threat intelligence to respond effectively and recover faster from sophisticated attacks. Join Amy Henderson, Director of Strategic Planning and Communications at Cisco Talos and Briana Farro, Director of XDR Product Management at Cisco, as they discuss some of the top threat trends and threats we have seen this past year and how to leverage security technology like XDR and network insights to fight against them. 

NIS2 Directive: Why Organizations Must Act Now to Ensure Compliance and Security (Jan. 11, 2024, 10 a.m. GMT) 

Virtual 

The NIS2 Directive is a crucial step toward securing Europe’s critical infrastructure and essential services in an increasingly interconnected world. Organizations must act now to prepare for the new requirements, safeguard their operations, and maintain a robust cybersecurity posture. Gergana Karadzhova-Dangela from Cisco Talos Incident Response and other Cisco experts will talk about how organizations can best prepare for the coming regulations.  

Most prevalent malware files from Talos telemetry over the past week 

SHA 256: bea312ccbc8a912d4322b45ea64d69bb3add4d818fd1eb7723260b11d76a138a 
MD5: 200206279107f4a2bb1832e3fcd7d64c 
Typical Filename: lsgkozfm.bat 
Claimed Product: N/A 
Detection Name: Win.Dropper.Scar::tpd 

SHA 256: 4c3c7be970a08dd59e87de24590b938045f14e693a43a83b81ce8531127eb440 
MD5: ef6ff172bf3e480f1d633a6c53f7a35e 
Typical Filename: iizbpyilb.bat 
Claimed Product: N/A  
Detection Name: Trojan.Agent.DDOH 

SHA 256: 5e537dee6d7478cba56ebbcc7a695cae2609010a897d766ff578a4260c2ac9cf 
MD5: 2cfc15cb15acc1ff2b2da65c790d7551 
Typical Filename: rcx4d83.tmp 
Claimed Product: N/A   
Detection Name: Win.Dropper.Pykspa::tpd 

SHA 256: 8664e2f59077c58ac12e747da09d2810fd5ca611f56c0c900578bf750cab56b7  
MD5: 0e4c49327e3be816022a233f844a5731  
Typical Filename: aact.exe  
Claimed Product: AAct x86  
Detection Name: PUA.Win.Tool.Kmsauto::in03.talos 

SHA 256: 77c2372364b6dd56bc787fda46e6f4240aaa0353ead1e3071224d454038a545e 
MD5: 040cd888e971f2872d6d5dafd52e6194 
Typical Filename: streamer.exe 
Claimed Product: Ultra Virus Killer 
Detection Name: PUA.Win.Virus.Ultra::95.sbx.tg 

Cisco Talos has disclosed 10 vulnerabilities over the past two weeks, including nine that exist in a popular online PDF reader that offers a browser plugin. 

Attackers could exploit these vulnerabilities in the Foxit PDF Reader to carry out a variety of malicious actions, but most notably could gain the ability to execute arbitrary code on the targeted machine. Foxit aims to have feature parity with Adobe Acrobat Reader, the most popular PDF-reading software currently on the market. The company offers paid versions of its software for a variety of users, including individuals and enterprises. There are also browser plugins of Foxit that run in a variety of web browsers, including Google Chrome and Mozilla Firefox. 

Talos’ Vulnerability Research team also found an integer overflow vulnerability in the GPSd daemon, which is triggered if an attacker sends a specially crafted packet, causing the daemon to crash. 

For Snort coverage that can detect the exploitation of these vulnerabilities, download the latest rule sets from Snort.org, and our latest Vulnerability Advisories are always posted on Talos Intelligence’s website.  

Multiple vulnerabilities in Foxit PDF Reader 

Discovered by Kamlapati Choubey. 

Foxit PDF Reader contains multiple vulnerabilities that could lead to remote code execution if exploited correctly.  

TALOS-2023-1837 (CVE-2023-32616) and TALOS-2023-1839 (CVE-2023-38573) can be exploited if an attacker embeds malicious JavaScript into a PDF, and the targeted user opens that PDF in Foxit. These vulnerabilities can trigger the use of a previously freed object, which can lead to memory corruption and arbitrary code execution.  

TALOS-2023-1838 (CVE-2023-41257) works in the same way, but in this case, it is caused by a type confusion vulnerability.  

Three other vulnerabilities could allow an attacker to create arbitrary HTA files in the context application, and eventually gain the ability to execute arbitrary code on the targeted machine. TALOS-2023-1832 (CVE-2023-39542), TALOS-2023-1833 (CVE-2023-40194) and TALOS-2023-1834 (CVE-2023-35985) are all triggered if the targeted user opens a specially crafted file in the Foxit software or browser plugin. 

GPSd NTRIP Stream Parsing access violation vulnerability 

Discovered by Dimitrios Tatsis. 

An integer overflow vulnerability exists in the NTRIP Stream Parsing functionality of GPS daemon, which is used to collect and display GPS information in other software. A specially crafted network packet can lead to memory corruption. An attacker can send a malicious packet to trigger TALOS-2023-1860 (CVE-2023-43628). 

According to GPSd’s website, this service daemon powers the map service on Android mobile devices and is “ubiquitous in drones, robot submarines, and driverless cars.” 

Buildroot - embedded Linux systems builder tool 

Discovered by Claudio Bozzato and Francesco Benvenuto. 

Talos researchers recently found multiple data integrity vulnerabilities in Buildroot, a tool that automates builds of Linux environments for embedded systems. 

An adversary could carry out a man-in-the-middle attack to exploit TALOS-2023-1845 (CVE-2023-43608) and TALOS-2023-1844 (CVE-2023-45842, CVE-2023-45839, CVE-2023-45838, CVE-2023-45840 and CVE-2023-45841) to execute arbitrary code in the builder. 

As a direct consequence, an attacker could then also tamper with any file generated for Buildroot’s targets and hosts. 

Malformed Excel file could lead to arbitrary code execution in WPS Office 

Discovered by Marcin “Icewall” Noga. 

An uninitialized pointer use vulnerability (TALOS-2023-1748/CVE-2023-31275) exists in the functionality of WPS Office, a suite of software for word and data processing, that handles Data elements in an Excel file.  

A specially crafted malformed Excel file can lead to remote code execution. 

WPS Office, previously known as a Kingsoft Office, is a software suite for Microsoft Windows, macOS, Linux, iOS, Android, and HarmonyOS developed by Chinese software developer Kingsoft. It is installed by default on Amazon Fire tablet devices. 

Talos disclosed this vulnerability in November despite no official fix or patch from Kingsoft after the company did not respond to our notification attempts and failed the 90-day deadline as outlined in Cisco’s third-party vendor vulnerability disclosure policy.  

In this episode the Beers with Talos team, led by special guest Dave Liebenberg, set out to save Thanksgiving. The TurkeyLurkey man is the hero that everybody needs, but perhaps don't deserve.

For fans and opposers of Dave's Ranksgiving list, you'll be pleased to know he's back with a whole new order, and some new snackable entrants.

Oh, and if it's security content you're after, we have some! Our 2023 Year in Review is out now, and the team recaps the top malware and attacker trends from the year. We also discussed the recent CNN article and Talos blog on our work to protect Ukraine's power grid.

If you'd like to read more, download the full Talos Year in Review report here.

Subscribe to future episodes of Beers with Talos at your own peril here.

The 2023 Cisco Talos Year in Review is now available to download. 

Once again, the Talos team has meticulously combed through a massive amount of data to analyze the major trends that have shaped the threat landscape in 2023. Global conflict influenced a lot of these trends, altering the tactics and approaches of many threat actors. In operations ranging from espionage to cybercrime, we’ve seen geopolitical events have a significant impact on the way these are carried out.

Read the 2023 Cisco Talos Year in Review

Download Now

At the beginning of the Year in Review is a “Top Trends” section comprised of regional trends over time and the influence of geopolitical events, the CVEs attackers exploited most often, spam tactics, and the top MITRE ATT&CK techniques that have been used within attacks.  The report then deep dives on four topics:  

The evolution of ransomware and extortion. The concerning rate of attacks against network infrastructure devices. The activities of advanced persistent threat (APT) actors in China, Russia, and the Middle East. This section also includes the major threats our Ukraine Task Unit dealt with this year. The shifting activities and impact of commodity loaders. 

Cisco’s global presence and Talos’ world-class expertise provided a massive amount of data to research — endpoint detections, incident response engagements, network traffic, email corpus, sandboxes, honeypots and much more. Thankfully, our teammates include subject matter experts from all ends of the cybersecurity space to help us turn this intelligence into actionable information for defenders and users.  

So, what is the main story of the 2023 Year in Review? Despite the accelerated pace of many threat actor campaigns and the geopolitical events that shaped them, the defensive community’s diligence, inventiveness and collaborative efforts are helping to push adversaries back.  

Download the Cisco Talos Year in Review today, and please share it with your colleagues and communities. This report was written by defenders, for defenders, and we hope it proves a useful and insightful resource for you. 

For more Year in Review content, visit the 2023 Year in Review landing page.

• As Russia’s invasion of Ukraine entered its first winter in late 2022, nearly half of Ukraine’s energy infrastructure had been destroyed, leaving millions without power. The resulting energy deficit has exacerbated something that hasn’t had much media attention: The effects of electronic GPS jammers affecting vital electrical equipment.
• Ukraine’s high-voltage electricity substations rely on GPS for time synchronization. So, when the GPS is jammed, the stations can’t accurately report to power dispatchers on the state of the grid.
• This complicates efforts to balance loads between different parts of the system, which is necessary to avoid outages and failure — especially during peak demand and surge times. Until recently, there was no solution to this problem.
• Cisco Talos worked alongside several other teams at Cisco, along with government partners in the U.S and Ukraine, to find a technological solution.

Since the start of the Russian invasion of Ukraine, Talos has been unwavering in our commitment to protect Ukrainian critical infrastructure from cyberattacks. 

In this blog post, you won’t find any mention of malware, DDoS, or espionage campaigns. In fact, it’s not about cybersecurity at all. This is a story about electronic warfare and GPS. It’s about how one chance conversation over dinner led me on a path through Cisco to find a solution to some very tough questions, and difficult answers.  

So, who am I? My name is Joe Marshall. I work at Cisco Talos as a cyber threat researcher and security strategist. My expertise is in industrial control systems and electric grids. My colleagues and friends at Talos are on the front lines of keeping the internet safe — and from more than just cyber threats, as you’ll read.

Project PowerUp is the story of how Cisco Talos worked with a multi-national, multi-company coalition of volunteers and experts to inject a measure of stability in Ukraine’s power transmission grid.

Our ultimate goal was to “keep the lights on” in Ukraine, and help make the lives of Ukrainians who are living in an active war zone, just that little bit easier.

Chapter 1: The energy deficit

As Russia’s invasion of Ukraine entered its first winter in late 2022, Russia stepped up attacks on Ukraine’s energy sector to deprive citizens of electricity and heat during the coldest part of the year. Nearly half of Ukraine’s energy infrastructure had been destroyed, leaving millions without power. The resultant energy deficit was exacerbated by another wartime challenge that, for some reason, hasn’t had much media attention: the effects of deliberate GPS disruptions affecting vital electrical equipment.

For the past year, there have been numerous reports of Russia interfering with GPS signals, especially near and within its own borders. Use of electronic jamming devices has been linked to attempts to disrupt GPS guided munitions, protect troops, and advance the tactical and strategic goals of armed conflict. 

While electronic interference can affect the battlefield, it is also having a secondary, unintended effect on Ukraine’s energy sector. Many of Ukraine’s high-voltage electrical substations — which play a vital role in the country’s domestic transmission of power — make extensive use of the availability of precise GPS timing information to help operators anticipate, react and diagnose a complex high-voltage electric grid. This is a complicated task during normal times, much less during a war.

When GPS signals are widely disrupted, substations cannot synchronize their time reporting accurately because they cannot assign accurate timestamps. Without good synchronized data, efforts to manage loads between different parts of the system can be affected, and this management avoids outages and failure, especially during peak demand and surge times. This disruption can be widespread, causing wide areas to lose GPS service for long periods of time. 

Until now, Ukraine has not had a viable solution to this issue for electric power systems.

Chapter 2: A chance meeting

I first learned about this situation when I was in delivering a cyber security presentation in February of 2023.  The audience just so happened to include a delegation from Ukrenergo, the electricity transmission system operator in Ukraine and is solely responsible for operating the country’s high-voltage electrical lines. Talos has been working with Ukrenergo for many years. 

The night before the presentation, colleagues from Ukrenergo invited me to dinner. When we sat down, I couldn’t help but persist with a barrage of questions: “How are you? Are you safe? What’s going on?” 

They started to tell me the true extent of what had been happening. This was one year since the start of the invasion. It was still deeply cold in Ukraine, and Russia had continually bombarded critical infrastructure for the entire winter. By March, there would be word that Russia’s campaign was beginning to tail off, but we didn’t know that at the time.

Ukrenergo started to list problem after problem, specifically with regards to the power grids. The obvious problems we all knew of course – kinetic strikes against substations were knocking out the power. Energy transformers were being destroyed, and replacements were scarce. One problem mentioned was rather casual, sandwiched in-between others, “We can’t get reliable timing due to electronic GPS jamming.” 

As I mentioned earlier, Ukraine’s high-voltage electricity substations rely on GPS for time synchronization. So, when the GPS is deliberately disrupted, the stations can’t accurately report to power dispatchers on the state of the grid. 

My ill-informed, rather bombastic response to this was, “Just buy some atomic clocks! You know…the type used by NASA.” Only after the words came out of my mouth did I remember that atomic clocks might not be a financially feasible option for this war-torn country. In fact, one member of the Ukrenergo delegation wryly retorted (I'm paraphrasing here), “Sure. Show me the aisle of the grocery store where atomic clocks can be found cheaply.” 

For the rest of the night, we talked about the GPS issues, the war, and Ukraine’s response to being attacked. Despite the sober undertones, the dinner company was superb, and the fellowship top-notch. The GPS timing issue, however, wouldn’t leave my head. I tried to look at it from all different angles. 

When we said goodbye that night, I silently vowed I was going to do everything in my power to help. But at the time, I had no answers. 

Chapter 3: The time paradox

While thinking about viable solutions, I was guided by an important principle: Whatever we do, speed is key. As I was wracking my brain, Ukraine was at war and suffering. However, I soon began to learn that it wasn’t as simple as that, due to the sheer complexities of what the country was up against. 

To truly understand the layers of solving this issue, I need to talk about why GPS clock timing is so important to electric grids. Most people are familiar with GPS because we rely on it for navigation, but it has also become the dominant system for the distribution of time and frequency signals globally. The U.S. controls and operates GPS satellites that orbit the earth twice a day which broadcast signals anyone in the world can use.

These satellites send very precise time data to GPS receivers on the ground that receive and decode the signals, effectively synchronizing each receiver to the same clock. This enables users to determine the time within 100 nanoseconds without the cost of owning and operating expensive and complex equipment, such as atomic clocks. 

Because GPS time is so accurate, GPS-disciplined clocks are commonly used in industrial systems, like Ukraine’s power grid, that require extremely precise time across a vast geographic area. 

Most devices that rely on time to calculate measurements have frequency references. The frequency reference is provided by an internal crystal oscillator within the device, and that crystal tells the device how fast time is going. However, these times are never perfectly accurate due to manufacturing variations and other variables in the crystal oscillators, causing time to advance at slightly different rates across various devices. This is why the clock on your laptop might be a few seconds or minutes ahead or behind the clock in your car. 

GPS solves this challenge. Devices can use the GPS satellites’ time signal to determine how accurate its local time reference is and then adjust the time accordingly, thereby enabling all devices running GPS-enabled clocks to be aligned to the exact same time. 

These GPS time signals are crucial for making a key piece of power equipment called a phasor measurement unit (PMU) run effectively. PMUs are used in power systems around the world to augment operators’ visibility into what is happening throughout a vast power grids network. A PMU measures a quantity called a phasor, which is the magnitude and phase angle of a voltage or current at a specific location on a power line.  

PMUs are essential to providing a detailed and accurate view of power quality across a wide geographic grid. Data from PMUs allows operators to predict and detect stress and stability on the grid, identify inefficiencies, and provide information for event analysis after a disturbance occurs. 

PMU data is time-stamped — to the precision of a microsecond — using the timing signal from GPS satellites. Therefore, measurements taken by PMUs in different locations are accurately synchronized with each other and time-aligned using the same global time reference marker. This allows all PMU data to be combined to provide precise and comprehensive information about an entire power grid. 

When GPS clocks are unavailable and the corresponding time signal has an error, that error can cause false calculations of phase angle and mis-alignment of grid conditions relative to other PMUs. Without the ability to analyze the precise timing of an electrical anomaly as it propagates through a grid, grid operators have difficulty diagnosing the exact issue that requires correction. Relatedly, if GPS timing is down, grid operators will have increased difficulty balancing power during the adverse events that occur during wartime.

Chapter 4: "You don't need atomic clocks"

After that fateful dinner with Ukrenergo, I spent the next few nights in deep thought. My brain wouldn’t let go of this timing issue. I consulted with colleagues and experts from other organizations who specialize in electric grid security, and ironically, they all suggested the same thing – atomic clocks. 

I knocked on Talos Vice President Matt Watchinksi’s door. I explained the situation to him, and ended by saying, “So can Cisco make an atomic clock?” I’d got it into my head that the only possible solution was to create a version of an atomic clock, as their holdover is measured in nanoseconds of accuracy. More than enough accuracy for a power grid.

Matt responded by saying he had no idea, but he would make some phone calls. That led me to a meeting with our Cisco Internet of Things (IoT) division. I asked them the same question I asked Matt: “Can Cisco make an atomic clock to counteract the GPS jamming, like what is being reported in Ukraine? 

After some research and identifying all manners of issues with locating an atomic clock, the team said, “Actually. We don’t think you need one. We think we have an existing solution within our IoT networking equipment. We can use that to build something unique for this specific situation.”

As is the case with most things in life, you should put your faith in the experts. And I’m so glad I listened to the IoT team. Because that was how we turned the ship, and Project PowerUp was a go.

Together with Cisco’s IOT networking team, we were going to design, create and deliver custom devices to Ukraine to keep substations running and delivering power to the entire country. 

“Throughout this war, I’ve seen and heard how resilient Ukrainians are. It’s very true. Citizens are dealing with one awful situation after the other, to the extent that this mentality of everyday trauma has become normalized. However, ‘getting used’ to power outages and not being able to keep warm in the Winter shouldn’t be normal. That’s what this whole project is about.”  Eric Wenger, senior director of technology policy for Cisco Government Affairs

Chapter 5: Is it good enough?

I mentioned earlier that this initiative was guided by the principle that speed was key. Delays meant potentially disastrous consequences. But I soon came to add another principle: Perfection is the enemy of good enough. 

The IoT team’s suggestion was that a Cisco Industrial Ethernet switch would be the best starting point in identifying a potential solution to the issues caused by Ukraine’s GPS outages. Industrial Ethernet switches do not have atomic clocks for holdover accuracy – but they have a good enough clock, able to measure time accurately in microseconds, to sustain an accurate time sync. This is important – Ukraine's electric grids operate on a 50hz frequency and have timing needs in microseconds.  

An Industrial Ethernet switch is part of Cisco’s hardened suite of switches, routers and other products designed specifically for rugged deployment, and Ukraine’s warzone undoubtedly fits into that category. These devices are built to withstand harsh industrial environments and extreme temperature ranges (-40° to 75°C). 

Hardened switches also have various internal resiliency features, including a source for its internal clock. Most network hardware devices use an internal crystal oscillator to generate their clock time, but these crystals’ frequencies can oscillate widely based on local conditions. However, an Industrial Ethernet switch can avoid this problem, as its crystal is a superior and resilient design, providing better frequency stability for precise synchronization of features such as GPS reception. 

Despite an Industrial Ethernet switch’s advantages, we needed to make some software modifications that would enable the device to address the specific set of challenges facing Ukraine’s power grid. 

There were two core issues we had to address with the Industrial Ethernet switch that required us to make enhancements to the device. First, we had to ensure interoperability between an Industrial Ethernet switch and the PMUs, and second, an Industrial Ethernet switch needed to provide the necessary holdover during GPS outages for the PMUs to continue working. Holdover is the time period to keep the clocks in sync until timing signals can be restored. 

During Ukraine’s GPS outages, which can last several hours, the PMUs effectively declare that something is wrong and stop sending data to the broader power management infrastructure — which causes significant upstream effects. Our first goal was to find a way to keep the PMU transmitting data. By modifying the metadata that an Industrial Ethernet switch sends to the PMUs, the PMUs will continue operating and sending data even without that signal. 

Next, we had to enable the Industrial Ethernet switch to provide an accurate time to the PMUs when time was unavailable (aka, the “holdover” period). We modified the Industrial Ethernet switch’s code to provide trusted time. 

With an Industrial Ethernet switch deployed to Ukraine’s substations, it measures the difference between the PMU’s local time reference used by the PMU and GPS time while GPS is still active. Then, when GPS signal is lost, the PMU can revert to using the local time reference, which is now highly accurate from the earlier error measurements, thereby allowing the PMU to continue operating.   

To ensure that an Industrial Ethernet switch fully understands what the GPS signal is telling it before the signal shuts down, Cisco created new, enhanced clock recovery algorithms. We also applied some additional filtering to the device’s software to allow it to recognize that the signal is down and to provide a “best guess” of what the time was when GPS was lost. 

We now had a device that was ready for production, but the job wasn’t done until testing was completed. After successful testing, Cisco immediately prioritized production of these devices. Hardware and software engineers from across the company pooled their collective expertise and created a production line capable of supporting the unique needs of Ukraine.

From the very start of Project PowerUp all I kept thinking about was the big picture of what we were trying to achieve. I’m proud to say that Cisco did this in an incredibly fast timeline. It is no easy feat to re-prioritize production efforts, especially in a technology company as vast as Cisco. But we had that guiding principle of speed and urgency – the longer this took for us to get these devices into Ukraine, the more days Ukrainians would be threatened with grid instability. 

A special shoutout to our Cisco Critical Accounts team. This team has been relentless in helping get key deliveries to Ukraine since the start of the invasion, and they were able to help drive the urgency for Project PowerUp too.

Chapter 6: Closing thoughts

As I write this, our Industrial Ethernet switches are in Ukraine, and helping keep the lights on. This reminds me of what we do at Talos every day. We fight the good fight every day to protect others.  

It is a lamentable fact that in cybersecurity and in critical infrastructure protection, we’re often confronted with the fact that our work, while valuable, may never be realized in our lifetimes as professionals. It is the legacy we leave with others we help protect and is built upon a large community who believe in fighting that good fight for generations to come. 

Project PowerUp is a little different. We know, beyond a doubt, that our work there will help save lives and will help keep the lights on. The effects are incredibly difficult to calculate, but we know it’s going to make life better. It’s helping others stay out of harm’s way. It’s helping a hospital that may not have reliable backup power. It’s giving a child just five more minutes of their childhood watching cartoons. 

If anything can be taken away from this, it’s that acting and leading with empathy is core to our mission at Talos. This year we took a chance to make a tangible difference in the lives of others and help them have a better life. Fighting the good fight isn’t just about cybersecurity – it’s about doing the right thing and helping others in the face of adversity. 

What started as a chance presentation this year turned into a multi-national, multi-company global team of power grid security practitioners who had never worked together before. As a team, we overcame numerous challenges to make Project PowerUp work. We could not have been successful without the support of numerous experts in Cisco who helped innovate this novel solution. And, of course, we must thank our partners in Ukraine, the U.S. government, and ICS vendors and experts who lent us their time, empathy, and expertise. We are humble and grateful for their help.

Slava Ukraini!