The Alerts Worked

Stillblueist

7 min read

·

2 days ago

--

Listen

Share

Two hundred a day

NNA Series — Part 2

The customer always knew first

The ops lead came to me.

“We need to find out about outages sooner.”

I asked. Right now, they said, the customer calls and tells them. They take the call, then dig through the logs. Thirty minutes on average, longer on weekends.

We want to know before the customer does.

A real problem. A real owner. I’d gotten it right again.

I filled it all in and handed it over

REQUIREMENT - Faster incident awareness■ REQUIREMENT                                   [Verified]  The responder must be able to know of an anomaly immediately.
■ CURRENT STATE  Path to awareness — customer call → responder checks → log lookup
  Average time to notice — 30 min. Longer on weekends.
  12 of the last 14 outages (3 months) were reported by the customer first.
■ VERIFICATION  Ops lead interview.
  "The customer calls. We almost never catch it first."
■ SCOPE  In  — outage detection, responder notification
  Out — auto-recovery, root-cause analysis

I read it back. Nothing was wrong.

A verb. An actor. [Verified]. The current state, in numbers. Verification noted, scope drawn.

I handed it over.

One question came

The dev lead asked.

“How do we send the alert? Email? Messenger?”

I hadn’t defined that. I said let’s use messenger.

That was all. It shipped two weeks later. It ran cleanly.

I must have defined it well.

A month later

The ops lead came back.

“Could you turn the alerts off?”

Me        ...You're saying it was better when they didn't come?
Ops lead  No, getting them is good. But two hundred come a day.
Ops lead  The first few days we read them all. Now nobody does.
Me        What kind of things come in?
Ops lead  CPU up. Response slow. Disk full. Retries up.
Ops lead  But last week payments were down for 40 minutes.
Ops lead  That alert came too. Somewhere in the two hundred.
Ops lead  In the end we heard it from the customer.

Back to the start.

The alerts came. The system didn’t break. The signal was just buried in the noise.

The dev team kept exactly what was written. They delivered alert immediately.

The requirement was met. What we wanted failed.

The dev team wasn’t wrong

I went to check. They had built exactly what was written.

Me        How did you decide what counts as an anomaly?
Dev lead  Well... it didn't say what counts as abnormal.
Dev lead  We couldn't risk missing one, so we put everything in.
Dev lead  CPU, memory, response time, error rate, disk, retries...
Dev lead  Tell us if we left anything out.

They put everything in so as not to miss anything. In their shoes, it was the right call.

The sentence only said “anomaly,” so what counts as abnormal was left to the builder to decide.

And the vague word wasn’t the only one.

anomaly      →   scope as wide as possible
immediately  →   speed as high as possible
                 = everything, in real time

The two multiplied. If only one had been vague, it would never have reached two hundred.

What they asked, and what they didn’t

I set them side by side.

Asked        "How do we send the alert?"    ← not defined at all
Not asked    "What is an anomaly?"          ← defined, but vaguely

A missing definition won’t let you start.

If you don’t know where to send it, you can’t start that part. So you ask.

A vague definition lets you start.

Read “anomaly” and your own idea of an anomaly comes to mind. Once it does, you can start.

And once you start, you go all the way.

A missing definition stops you. A vague one moves you along.

And nobody realizes they read it two different ways.

What I meant        payments going down, orders not getting through
What the team read  anything out of the ordinary

Both are “an anomaly.” Neither is wrong.

A vague sentence looks like agreement.

Both sides nod — and what they nodded to is different. It surfaces only after the thing is built.

I went back to ask

I returned to the ops lead.

Me        What is it you actually want to know about?
Ops lead  It's not that I want to know everything.
Ops lead  I just need to know before the customer calls.
Me        ...So only what the customer would notice?
Ops lead  Yes. What they don't notice, we can look at slowly.
Me        How long after an outage does the customer usually call?
Ops lead  About ten minutes. Payments is faster — around five.
Me        Then we should set it by payments.
Ops lead  Yes. And the on-call has to get it. That's where the weekend thirty minutes came from.

Here, three things were settled at once.

What is an anomaly   an outage the customer would notice
How fast             within 3 minutes
To whom              the on-call responder

All three came from the same one sentence — “before the customer calls.”

What my document didn’t say

I opened it again. I thought I’d said everything needed.

The words I’d written were vague. How far “anomaly” reached, how many minutes “immediately” meant — the document said neither.

The job was to clear the fog.

To make a vague spot clear, you need something to go by. That yardstick is what we want — “before the customer.”

How far to count “anomaly,” how many minutes “immediately” — what we want decides it. For clearing the fog, there is no better yardstick.

Without what we want, the builder has to decide for themselves what counts as an anomaly. So they put everything in. Better than missing one.

Leave out what we want and the scope goes vague.

The two aren’t separate. The one line — “we want to know before the customer” — set both the reach of “anomaly” and the speed of the alert.

Leave that line out and the next person sets both. And that person has never taken a customer’s call.

And that yardstick — the document never said it.

■ CURRENT STATE   how things are now        ← said
■ VERIFICATION    who confirmed it          ← said
■ SCOPE           this far                  ← said

■ INTENT          what we want              ← not said

It said everything else, and left out only the yardstick for the fog.

And not because I didn’t know it. I had already found it.

Found    we want to know before the customer
Wrote    the responder must know of an anomaly immediately

I handed over only the bottom. The top stayed in my head.

Writing the current state and writing what we want are different.

I’d written 30 minutes on average, 12 of 14, and thought I’d explained enough.

That’s how things are now. Not what I want.

Current state   the customer knows first       ← shows the present
What we want    we want to know before them    ← sets the end

Gerald Weinberg defined a problem as a difference between things as desired and things as perceived. The current state is only one of the two — the perceived.

Without the desired side — what we want — there is no difference. And with no difference, there is no telling how far is far enough.

Cut 30 minutes to 29 and it’s an improvement. And yet the customer still knows first.

Only what we want makes it possible to tell whether you’ve arrived.

I wrote it again

■ REQUIREMENT                                    [Verified]
  The on-call responder must be able to know of an outage
  the customer would notice, within 3 minutes of its onset.

■ INTENT
  We want to know before the customer does.

■ CURRENT STATE
  Outages are noticed by customer call. 30 min on average.
  The customer call comes 5 min after onset. (Payments - the fastest.)
  → To know before the customer, it must be within 3 min.

■ SCOPE OF "an outage the customer would notice"
  Rule - in if the failure shows in the customer's screen or flow; out if not.
  In   - order failure, payment failure, login failure, page not responding
  Out  - CPU, memory, disk, retries and other internal metrics
         (the customer doesn't see them; watched on a separate dashboard)

■ HOW IT'S SENT
  Direct to the on-call responder, by messenger. (not the ops team's group email)

■ VERIFICATION
  Ops lead interview. "What the customer doesn't notice, we can look at slowly."

The alerts went from two hundred a day to two or three.

And the two or three all get read.

One thing surprised me.

What I wrote      immediately
What was needed   within 3 minutes

Writing it precisely meant building less. Immediately means millisecond detection, kept running without a gap. 3 minutes, and all of that shrinks.

A vague word can make you build too little, or too much.

What this article asks

Clarity        does it read only one way
What we want   is it written

There’s a reason the two sit together.

Without what we want, you can’t write clearly.

Try to narrow “anomaly” and there’s nothing to narrow it against. The moment that one line arrives, a yardstick appears — and with a yardstick, the scope is set.

The number, the same. 3 minutes is not a number I picked. It came from the customer calls 5 minutes after a payment outage.

Self-check

When you’ve written it, read it through twice.

First — find the nouns.

anomaly · related data · when needed · key items · the responder

When you see one, ask yourself. “Who? What?”

Second — find the modifiers.

immediately · promptly · quickly · appropriately · sufficiently · reliably · easily · regularly

When you see one, ask. “How much?”

If the answer comes right away — you knew it and didn’t write it. Write it.

If it doesn’t — you haven’t found it yet. Don’t invent it. Go find it.

Next

This article made the problem definition clear.

But clear is not buildable yet. I wrote must know within 3 minutes — but

what does the alert pass through to get there? what if one step in the middle fails?

The next article is the path.

Don’t Go Too Fast. 🐢

NNA Series, Part 2.

Series:

1. I Found the Real Problem
2. The Alerts Worked ← you are here
3. The Scenario — Paths
4. The Data and Where It Comes From
5. It’s All One Thing

How many words like “related,” “appropriately,” “as needed” are in the requirements you’ve written?