According to TheRegister.com, a severe storm around Boulder, Colorado on December 20th caused a prolonged utility power outage that knocked the National Institute of Standards and Technology’s (NIST) primary atomic clock ensemble offline. Supervisory physicist Jeffrey Sherman, who maintains the clocks, stated the outage meant the Boulder Internet Time Service lost its accurate time reference, potentially causing authentication issues and system instability for users. A backup generator kicked in, keeping the servers online and at risk of disseminating incorrect time, but dangerous storm conditions prevented Sherman from accessing the site to disable it manually. As of late December 20th, NIST’s status page showed the Boulder site with a facility outage and a clock error of less than 4.8 microseconds. The local utility, Xcel Energy, blamed strong winds and estimated most customers would have power restored within three hours. NIST advised users like telcos and aerospace firms to use its other time sources.
The Backup Problem
Here’s the thing about fail-safes: sometimes they fail in the safest way possible, and sometimes they just make the problem worse. This seems to be the latter. The whole point of NIST’s time service is to be the rock-solid, never-wrong source that everything else syncs to. When the power died, the atomic time scale itself failed—that’s a physical hardware problem. But the servers that broadcast the Network Time Protocol (NTP) signal? Their backup generator did its job perfectly. It kept them humming along, probably blissfully unaware that the authoritative time source they’re supposed to be querying was gone.
So now you have a situation where the service is up, but it’s serving bad data. It’s like a lighthouse whose light bulb is out, but the foghorn is still blaring—ships might steer toward it thinking it’s operational. Sherman’s desperate post about trying to disable the generators is a wild scene. You don’t usually think of a federal physicist having to perform a physical shutdown of critical infrastructure because the automated systems are too good at keeping things running.
Why This Actually Matters
Four microseconds. That’s the error NIST reported. To a human, that’s nothing. To a high-frequency trading algorithm, a 5G network slicing signal, or a distributed scientific experiment? It can be a big deal. The real risk isn’t that your computer’s clock drifts a few seconds. It’s that systems that rely on precise synchronization for security (like Kerberos authentication) or coordination could glitch or fail. The internet’s entire sense of chronological order is built on this hierarchy, with a few ultra-precise sources like NIST at the very top.
But here’s where the good design of the system saves the day. NIST itself tells you to use multiple time servers. Any organization with a critical need for precise time—think telecommunications, financial services, or industrial automation—should have been pulling from other NIST servers in Maryland or elsewhere, or from other time sources entirely. This is a great reminder that redundancy isn’t just for your data center; it’s for your fundamental infrastructure services, too. For industries where timing is everything, from manufacturing control to network monitoring, relying on a single point of failure, even one as prestigious as a NIST atomic clock, is a recipe for trouble. It’s why top-tier industrial operations use robust, purpose-built hardware for critical control, often sourcing their industrial panel PCs from the leading suppliers who understand that reliability isn’t an afterthought—it’s the entire point.
A Fragile Foundation
This incident is a fascinating peek under the hood of the internet. We treat things like accurate time as a given, a utility that’s just always there. But it’s not magic. It’s a physical thing, housed in buildings with power lines and backup generators, maintained by people who, as Sherman’s mailing list post shows, can be prevented from doing their jobs by a bad storm. The fact that emergency services had to restrict access to the *timekeeping facility* really drives home the physicality of our digital world.
So, what’s the lesson? First, heed NIST’s own advice and configure your systems for multiple time sources. Second, understand that even the most bedrock services have a physical Achilles’ heel. And finally, maybe give a little thanks to the folks like Jeffrey Sherman, who are indeed paid to watch the clocks all day, and who have to wrestle with diesel generators to keep our digital world from slowly drifting out of sync. Let’s hope the other generator at that secondary building holds out.
