The check that could never fail
Picture a school that marks its attendance register every morning. By ten o’clock, the office knows who is in and who isn’t, and the parents of the children who aren’t get a text. That is the whole point of the register: not the tick marks, but the message that follows them.
One Tuesday the teacher who marks it is off sick. Nobody else does it. No absences are recorded, so no absence messages go out. A child who never arrived is, as far as the software knows, not missing. The school finds out at two in the afternoon when a parent calls to ask why nobody told them. Or it never finds out, which is worse.
Nothing crashed. No error was logged. The software did exactly what it was told and said nothing about what it wasn’t told. This article is about that kind of silence, the bug it hid in our own system, and what we rebuilt so a person hears about it.
A rule for what didn’t happen
Raven, our messaging platform, has had a feature for a while that we call a cadence rule. A business says: “this kind of record should exist every weekday by 10:00” or “a payment record should be made at least once a week by 17:00”. If the deadline passes and the record isn’t there, Raven raises a flag.
The flag is useful because most software only knows about things that happened. A message sent, a payment recorded, a form submitted. Almost nothing watches for the absence of an event. Yet absences are where small organisations lose money and trust: the register nobody marked, the reminder batch that stopped, the payments list nobody has touched since the person who kept it left.
We tested the rule. We seeded it for every school: register by 10:00 on weekdays. The flag appeared on the rules screen when it fired. We moved on.
The bug
Three weeks later we started building the next step: instead of showing the flag on a screen that someone may or may not open, send it to named people by message. The first question you ask before messaging anyone is “how often does this fire?” You don’t want to wake a head teacher for noise.
The answer was zero. Not once, for any school, since the rule shipped.
The cause was small. The job that checked cadence rules ran once a day, early in the morning, alongside the other daily housekeeping. Each run looked at the “by 10:00” rule, saw that 10:00 hadn’t arrived yet, and correctly skipped it, because you can’t say a register is late before its deadline. The next morning’s run did the same. There was never a run that happened after the deadline on the same day.
A check that runs before the deadline it checks will never fail. And a check that never fails looks exactly like a system with no problems. Our tests passed because they ran the check at a chosen time. Production never chose that time.
What we changed
Finding the bug was the easy part. The interesting work was deciding what an alert should be, once a human is on the other end of it.
Get Yarzue Tarlee Zolia’s stories in your inbox
Join Medium for free to get updates from this writer.
1. Check at a frequency that matches the deadline. The cadence check now runs every hour, so a 10:00 rule is checked at ten-something. A separate sweep that runs every ten minutes sits beneath it as a net, so that if the hourly run is late or skipped, the alert still goes out. A rule still raises at most one flag a day; checking more often must not mean nagging more often.
2. Tell a named person. Each rule has a “who to tell” list, up to ten staff. Raven’s own problems, like an SMS sender that stops working or email access that was withdrawn, go to a separate short list of up to five people the business chooses. The message says what is missing, which rule, and which day: “Absence has not been recorded today (3 Nov). Rule: Register by 10:00.”
3. Never report a problem down the channel that is broken. Alerts go by WhatsApp first, falling back to SMS, with an email as well. But if the SMS sender is what stopped, the alert goes by WhatsApp and email only. If the problem is with WhatsApp, it goes by SMS and email. This sounds obvious written down. Most monitoring setups I have seen send “your email is down” by email.
4. Respect working hours, except for emergencies. Each business has alert hours, 07:00 to 21:00 by default. An ordinary alert that becomes due at 23:00 waits until morning. A critical one, a sender that stopped or a number that was flagged, goes at any hour, because by morning a day of messages will have failed.
5. One message per person per problem. If a rule raises several flags in one run, the people on its list get one message, not five. Nobody reads the fifth.
6. Stale news is not news. A flag that is more than a day old when the system gets to it is never sent. An alert about yesterday’s register is noise with a timestamp; the school already knows.
7. Let the fix close the alert. When a late record is finally made, it resolves the open flag quietly. Nobody gets a “resolved” message for a problem they fixed themselves.
Why this matters more for small organisations
A bank has a monitoring team. A 300-family school has a head teacher with a phone. A clinic has one person at the front desk. A shop has an owner who checks the books on Sunday evening.
For these businesses, a dashboard is not monitoring. Nobody is paid to look at it. The only alert that works is a message to a person, on a channel they already use, at an hour they are awake, saying plainly what is missing. If the software doesn’t tell them, nobody does, and the customer finds the gap first.
That is also why we didn’t build this as “log everything and let the business filter”. Choosing who hears what is the product. More alerts is not more safety; it is a faster route to everyone ignoring them.
The lesson I keep
For every check you write, find the day it should have fired and prove that it did. In a test, in staging, in production with a deliberately missed deadline. “No alerts” is a measurement only after you have seen the system alert.
A simple habit helps: keep a short list of your checks with the date you last saw each one fire. It is a boring list. It is also the list that would have caught a check that slept through every deadline for three weeks.
Silence from your software is not good news. It is just silence. Make it tell someone.
Yarzue Zolia is the founder of Utilitay in Monrovia, Liberia. Utilitay builds software that keeps working where networks, power and payments don’t. Raven is its messaging platform for schools, clinics and businesses.
