The NOC Is a Stage

The NOC Is a Stage

This post is part of my Medium blog.

I've always liked network operations centers.

Rows of monitors. Wall-sized displays. People in headsets looking at dashboards with the concentrated expression of somebody who knows the next five minutes could become expensive. I toured one at Microsoft that looked like a movie set — space-age room, control panels, screens everywhere. The kind of room where somebody says, "We've got a problem," and a person in a headset leans forward and saves the world.

I liked that.

I've also been in operations rooms during real outages. The room changes. People talk over each other. Somebody says, "I'm on it," which is useful even when they have no idea whether they are. The room gets loud. Everybody moves toward the problem at once. It feels like Apollo 13.

That is part of the appeal.

Spend enough time in these rooms and you notice the difference between operating a system and performing the fact that it is being operated.

A NOC can be useful. In a high-consequence environment — launching spacecraft, managing a power grid — having the right people together in the same room is not ceremony. But most technology organizations are not launching spacecraft. They are running websites, APIs, and databases. Services that can be watched remotely, operated across time zones, and increasingly monitored by software.

The room is not always doing the work. Sometimes it is telling a story about the work.

That story is easy to understand. Somebody is always watching. The system is under supervision. The organization is serious because the room looks serious. Executives like it. A wall of green panels is reassuring. It says somebody has looked at the system recently. It says there is a process, and the process has people inside it.

But a system can be watched without a person staring at a screen. An alert can trigger a response. A rollback can be tested, escalated, and audited without summoning twelve people to a conference bridge at two in the morning.

The best operators I know don't need the room to make them feel like operators. They know what they own, what the alerts mean, which failures can be handled automatically and which require a human decision. They have rehearsed the response before the incident arrives.

That is operating a system.

The ceremony is different. The ceremony is the belief that the system is safe because the right people are physically near the screens.

I've been in enough of these rooms to know what else happens. During a real incident, the people at the desks in headsets are often not the ones doing the technical work. The actual response is happening on a conference line — a bridge full of engineers who dialed in from home, from another time zone. The people in the room are managing the process. They are asserting ownership of the incident, summarizing what the bridge is saying, and making sure the right names appear on the right slides for the post-mortem.

Some of them are qualified. Some are not. But in a room full of screens, during a live incident, nobody stops to ask. The room confers authority. If you are sitting at the console, you must be the one who knows.

I have watched people perform competence in an emergency who did not have it. Not maliciously. The room pulls it out of you. The screens are red. People are watching. Somebody has to say something. So somebody says something, and if they say it with enough confidence, the room accepts it. The real engineers on the bridge know. The person at the desk may not. The gap between the performance and the work is invisible to anyone who isn't on the bridge.

That is part of what I mean by the stage.

The same thing happens in Redundant. The first executive meeting takes place in a network operations center. The room is full of screens, live traffic, and the visual language of control. The executives are not meeting in an ordinary conference room because the NOC makes the meeting feel more consequential.

Rob has been flown in to present cloud-cost data. He expects the usual ritual: show the numbers, identify waste, watch executives promise to do something, go home. But the room is part of the setting and part of the performance. Everyone is using the environment to make the situation feel controlled while the information underneath is moving somewhere else.

The NOC provides a kind of institutional gravity. Put executives in front of live dashboards and the meeting acquires a seriousness it might not have earned. It becomes a command center. The people inside it look like they are managing something important, even before they have agreed on what the thing is.

That is why these rooms are useful in fiction. They expose the difference between the appearance of control and the work of control.

The irony is that modern systems are increasingly capable of watching and responding without a room full of people. Automation can detect a failure, move traffic, restart a service, and roll back a deployment. AI systems may eventually handle parts of the investigation and response that once required a shift of people looking at monitors.

That does not mean humans disappear. It means the human work changes. People should be deciding what the system is allowed to do, testing the automated responses, and handling the failures that do not fit the playbook.

The human should be where the judgment is. Not necessarily where the screens are.

I still like NOCs. I like the screens. I like the sound of people becoming serious when a system starts failing. I like the feeling that a group of humans has gathered around something complicated and decided to pay attention to it together.

I just don't confuse that feeling with reliability anymore.

In Redundant, the meeting happens in a room designed to make the people inside it look in control. The company has built a place where the presence of screens can stand in for the presence of understanding.

The screens are real, and so is the ceremony. The question is whether anyone is watching the thing that matters.

Subscribe to The Condition Set newsletter