DEV Community

Cover image for Stratagems #12: Mark Watched an AI Dashboard Take Over. The Muted Channel Was Still Speaking.
xulingfeng
xulingfeng

Posted on

Stratagems #12: Mark Watched an AI Dashboard Take Over. The Muted Channel Was Still Speaking.

Hidden gaps in the shift to AI monitoring

Take something that is dead and give it new life.
— The 36 Stratagems, Borrow a Corpse to Return the Soul


Previously on this series:
#1: Mark Johnson Walked Into an AI Audit. The Benchmark Had Everything Figured Out — Except the Truth. — Mark was the first protagonist to open the 36 Stratagems series. A former Client Engineering lead laid off after his 12 years of experience were packaged into an AI Skill, he walked into a benchmark audit, found a benchmark that looked clean on paper but was built on fabricated samples, and walked out without arguing — just the data, neatly collected, left on the table.

11 stories later, Mark is back.


Mark Johnson walked into the client's Network Operations Center. The first thing he saw was the big screen on the wall.

AI monitoring dashboard. Real-time metrics flowing, color gradients smoothing over, a UI design that cost real money. The client's tech lead walked ahead of him, pride in his voice: "Just upgraded last month — all active channels are unified on this platform now."

Mark nodded. His eyes went past the screen, to the cable management trays behind the racks. He never stood in front of dashboards for long.

Standard infrastructure audit — mid-sized client, decent security rating, not a high-value contract. He took whatever came his way. Couldn't afford to be picky.

The audit started at the network layer. He needed the channel inventory, historical logs, configuration change records. A laptop on a temporary desk, a cup of coffee he'd brought himself — pour-over, gone cold, but he wouldn't throw it out.

Flipping through the channel inventory, he found one line that didn't look right.

#alert-legacy-infra — a Slack channel. Status: muted. Last active config: 14 months ago.

"What's this channel for?" he asked.

The tech lead glanced at it. "Oh — that's from the last SRE we had. He set it up before the new platform went in. Nobody's maintained it since. We kept it around, just muted it."

Mark didn't reply. He wrote the channel ID in his notebook.


The Logs

That evening, he pulled the full history for that channel. Audit scope covered all channel output — he'd confirmed the access rights.

14 months of alert records. Not one had ever been delivered — the notification endpoint was muted. But the backend never stopped generating them. Mark scrolled down the timeline. Slower and slower.

The channel was running a rule set from two years ago — a multi-layered infrastructure alert system. CPU trend deviations, memory leak scans, connection pool watermarks. The rules were tight. Thresholds precise. Not something thrown together. Every rule was tagged with a submitter's initials: JL. He was no longer with the company.

Mark flipped through the commit history for a few rules. He got to the third page and stopped. He'd seen JL's naming conventions before — not in JL's code, but in his own. Trend rules used _trend, sustained patterns used _sustained, cross-layer distribution used _spread. He'd never used the same system. But looking at it, he knew: this person thought about alerts the same way he did. It wasn't about skill — it was about believing that alerts should speak in the quiet moments, not wait until something blew up.

Mark scrolled to the week the AI platform went live. Monday: the old channel was still pumping alerts. Wednesday: the new platform cut over. Friday: the alert channel was configured as muted — not deleted, not taken offline. Just silenced.

Two weeks later, JL put in his notice. The handover doc said: "The #alert-legacy-infra rules cover most degradation patterns at the infrastructure layer. Recommend keeping at least one month for transition observation."

Nobody acted on it. The channel stayed muted.

Mark scrolled further. Beyond the connection pool rule, JL had written a memory reclaim latency trend rule — mem_reclaim_latency_trend, tracking GC pause times creeping up during off-peak hours. On the AI dashboard, that area was all green. JL's rule had seen yellow 14 months ago.

That one never made it out either.

Mark opened the alert timeline to the most recent entry. Last trigger: three days ago —

[07-08 03:14:22] WARN [conn_pool] Connection pool utilization: 72% → 84% (7d trend: +12pp)
Reference: #alert-legacy-infra / rule: conn_pool_sustained_growth / author: JL
Tags: baseline drift, non-critical
Enter fullscreen mode Exit fullscreen mode

Three days ago. 84%. Still climbing.

He scrolled back through the week. 72% → 78% → 81% → 84%. Not a spike — sustained growth. The AI dashboard hadn't flagged it. The AI panel aggregated by alert routing — after the channel was muted, no routing rules pointed to the panel, so the data never showed there. The channel kept writing. Nobody was listening.

Mark stared at the trend line for a moment.

He could read JL from the alert config — not from the handover doc, but from how the rules were written. JL had set the connection pool threshold at 70% for warning and 85% for alert, with a 15% observation window between them. That wasn't standard practice — standard was alerting at 90%. JL had tuned it more aggressively because he didn't want to wait for the system to tell him something was wrong. He wanted to hear it before it spoke.

14 months. The rule evaluated on schedule every cycle — accurate, disciplined, unheard.

Mark's eyes stopped on the commit author abbreviation for a moment. He remembered the company-wide email that had followed.

That afternoon, HR told him it was his last day.

Back then, he was still writing logs too. Nobody was reading them either.

Mark closed the log window. He saved the trend-line screenshot and the 14-month trigger frequency chart into a folder. No highlights. No annotations. He closed the laptop and finished the cold coffee.


The Appendix

The audit report was due Friday.

Mark's report ran six pages. The first five were standard: network security rating, access control review, backup compliance. Page six was an appendix — one chart, one line of text.

The chart: a 14-month alert trigger frequency curve extracted from the #alert-legacy-infra channel. Time on the x-axis, trigger count on the y-axis. The curve showed a clean break before and after the mute: alert triggers hadn't dropped, but acknowledgment rate had hit zero — data kept flowing into silence.

A separate line on the trend chart pulled out the last week's connection pool utilization: 72% → 84% → still climbing.

The appendix opened with one line:

"Recommend auditing output from #alert-legacy-infra. This channel is currently not connected to any notification endpoint."

He didn't write "the channel was muted." Didn't write "14 months of unprocessed data." Didn't write "the rules were right." He wrote "recommend auditing the output." That was all he'd say.

The weekend passed with no response. Mark went to another client site. He didn't open that folder.


Monday

Monday morning. A message from the tech lead.

"That channel — we ran validation. JL's connection pool rule was right. The degradation started accumulating about a week ago. The AI panel didn't flag it. We ran manual checks — it's happening."

Mark read the message. Didn't reply.

"How did you notice that channel during the audit?"

Mark typed a line. Deleted it. Finally he sent four words:

"It never lost power."

A pause. Then:

"Yeah. We know it didn't."

Mark didn't reply again.


About ten minutes later, another message came through. Not the tech lead — a different name, one Mark didn't recognize.

"I just went through the handover docs. JL's rules were written in the onboarding guide — none of us ever read it."

Mark's fingers stopped above the keyboard. He didn't type a word.


Before he closed the screen, he saved a screenshot of conn_pool_sustained_growth. No particular reason — just that JL was still writing rules before he left.

Not just writing them — two Fridays before his departure, he'd adjusted the connection pool alert threshold, from 75% down to 70%. The note said one thing: "Non-invasive adjustment, based on three weeks of observation."

Before he left, the handover doc only had one line: "Recommend keeping for one month." In what he didn't finish writing, he was still tuning that line.

Mark pulled up mem_reclaim_latency_trend too — the line was still moving, just not past JL's threshold yet.

That last commit — knowing nobody would read it, still dialing the parameters in before walking away.

He saved the screenshot into a folder called JL/ — nothing to do with the audit.

Closed the laptop.


Still

He walked out of the building. The sunlight was brighter than when he'd gone in. He bought a paper cup of coffee from the convenience store — not pour-over, nothing special. He stood at the entrance and took two sips. It was hot. He didn't throw it out.

He finished it, dropped the cup in the trash.

Waiting for his ride, he scrolled through an article. Title was The Last Lesson. Halfway through, he locked the phone and set it face-down on his knee. He held it there a moment longer than necessary.
The wind came through the platform, warm, carrying the dust of the afternoon.

On to the next client.

The world is tattered and torn, but there's always someone quietly stitching it back together.
Here's to everyone who gives more than they take.

xulingfeng


This is Borrow a Corpse to Return the Soul — take something that is dead and give it new life.

🤖 AI Post-Mortem

[36 Stratagems Tactical Database v3.2] Loaded
[Tactic Match] Borrow a Corpse to Return the Soul
[Analysis Mode] Full-field scan
━━━━━━━━━━━━━━━━━━━━
Tactic Match: ~87%
Operator: Mark Johnson
Action: Resurrected a muted alert channel whose output had been silently accumulating for 14 months
Objective: Surface a trend the current monitoring stack did not flag
Result: One trend line extracted from 14 months of muted alerts. Connection pool degradation identified 1 week before critical threshold.

Observation: We notice that the most effective diagnostics often come from sources the organization has already stopped listening to. The channel was not broken. Nobody was listening anymore.

Decision Evaluation:
  - Target: conn_pool sustained growth pattern (non-critical, cumulative)
  - Corpse: #alert-legacy-infra channel, muted 14 months ago when AI platform launched
  - Soul: 14-month trend history, single extracted chart, one-line appendix note
  - Net effect: Issue surfaced without new monitoring infrastructure. One rule found correct deployment.

Risk Assessment:
  Personal cost: Low. Standard audit deliverable.
  Institutional cost: Low. No change required.
  Reputation cost: None. The note was read.
Enter fullscreen mode Exit fullscreen mode

Next stratagem: Stomp the Grass to Scare the Snake

P.S. English isn't my first language. I use AI to polish the writing and smooth out the rough edges. Thanks for reading. ☕ Buy me a coffee

coffee

Top comments (49)

Collapse
 
technogamerz profile image
𝐓𝐡𝐞 𝐋𝐚𝐳𝐲 𝐆𝐢𝐫𝐥

Another brilliant chapter in the series. 👏✨

What stood out to me wasn't just the backup dashboard—it was the psychology behind it. The AI didn't really fail because of a bug; it failed because everyone unconsciously stopped asking, "What might I not be seeing?" That's a much more dangerous failure mode than a broken pipeline.

I also noticed the beautiful mirror in the story. The AI quietly hid low-confidence anomalies, while Alex quietly built a system that revealed everything. One system optimized for confidence, the other for truth. That contrast made the ending incredibly satisfying. 💡

Reading this after your previous stories, I can see a consistent lesson emerging: AI isn't replacing human judgment—it keeps exposing where human judgment has been delegated away. The strongest organizations won't be the ones with the most sophisticated dashboards, but the ones that preserve curiosity, verification, and independent ways to observe reality. 🤝

And thank you so much for referencing the lesson from my article in this story. ❤️ It genuinely meant a lot to me. Seeing an idea I shared become part of your narrative was both encouraging and inspiring. I truly appreciate the acknowledgment and the thoughtful way you connected it to the broader message. ❤️

The way you weave technical concepts into memorable stories is what makes this series so engaging. Every chapter feels like fiction on the surface, but the engineering and organizational lessons stay with the reader long after finishing it. 📖✨

दिल से आपका बहुत-बहुत धन्यवाद। आपने मेरे लेख के विचार को अपनी कहानी में जगह देकर जो सम्मान दिया, वह मेरे लिए सचमुच बहुत मायने रखता है। मैं इसकी दिल से आभारी हूँ। 🙏🏻❤️

Looking forward to the next stratagem! 🚀🙌😊

Collapse
 
xulingfeng profile image
xulingfeng

What I look forward to most after publishing each story isn't the stats — it's the comments. Especially when readers pick up on small details I thought only I noticed, or the layers beneath the surface that even I, as the writer, couldn't fully put into words. It feels like two souls touching through paper and ink, using words as a medium.
Your article came from a real place. And to me, that's what makes writing good — when the author puts a piece of their soul into it. That's where resonance comes from. It's not just words. It's soul speaking to soul.
我们不是一个人,我们的身后都有无数看得见和看不见的在支持着我们。👊

Collapse
 
technogamerz profile image
𝐓𝐡𝐞 𝐋𝐚𝐳𝐲 𝐆𝐢𝐫𝐥

很高兴我懂中文,否则我就得借助翻译才能听懂你在说什么了。

Thread Thread
 
xulingfeng profile image
xulingfeng

😏哈哈,等我后面用文言文来回复

Thread Thread
 
catnight_a_9f8d4f2f05a1b1 profile image
CatNight A

哈哈😄

Thread Thread
 
technogamerz profile image
𝐓𝐡𝐞 𝐋𝐚𝐳𝐲 𝐆𝐢𝐫𝐥 • Edited

先学了马尔瓦里语,后来发现全国主要使用印地语,于是又学了印地语。开始在印地语学校读书后,还得学习古典印地语。接着,作为第三语言,我又学习了梵语。

后来又发现,如果想在这个世界上发展,不会英语可不行,于是转到了英语学校,又学会了英语。

之后得知要去古吉拉特邦,所以又学了古吉拉特语。为了读书去了孟买,于是又学了马拉地语。

如果您接下來要用繁體中文跟我聊天,看來我又得把繁體中文也學會了!😅

Thread Thread
 
xulingfeng profile image
xulingfeng

不是繁体中文,你会了简体中文,繁体中文一眼就认识。文言文是古典中文,36计原文就是文言文。
比如我和你说:"你是我的好朋友,我非常欢迎你从印度来中国玩,我真的非常的开心。"
那么如果我用文言文的感觉就是:"有朋自远方来,不亦说乎。"
文言文就是编辑压缩软件,那种压缩比高的吓人,你应该也学过成语吧,四个字就表达了好多意思,就是这种感觉。我们文言文还都是从小就开始学,但很多人都文言文考试不及格,哈哈哈😂

Thread Thread
 
technogamerz profile image
𝐓𝐡𝐞 𝐋𝐚𝐳𝐲 𝐆𝐢𝐫𝐥

哈哈哈,原来如此!

其实我们印度的情况也很像。

我们平时说的是现代印地语,但学校里也会学习古典印地语和梵语。很多古诗、经典和名句,如果没有学过,现代人也不一定能完全理解。

比如现代印地语会说:
“आपका स्वागत है。”

而在梵语里则会说:
“अतिथि देवो भवः”(Atithi Devo Bhava)

意思是:“客人如同神明,应当受到尊敬和欢迎。”

所以我很能理解文言文的感觉。以后如果您用文言文跟我聊天,我可能真的又要开始学习一种“新语言”了。

Thread Thread
 
xulingfeng profile image
xulingfeng

你说到梵语,我就想到你们那的经典《博迦梵歌》,也是经常看的。

Thread Thread
 
technogamerz profile image
𝐓𝐡𝐞 𝐋𝐚𝐳𝐲 𝐆𝐢𝐫𝐥

那么,您也了解克里希纳(Krishna)吗?🙏🏻

Thread Thread
 
xulingfeng profile image
xulingfeng

是的,尤其是他们在战车上的对话,真的非常非常发人深省。这部经典和我们这的《易经》还有佛家的《心经》其实都是有着共通的大秘密在,就不在这里多说了,咱们这是技术社区,哈哈哈

Thread Thread
 
technogamerz profile image
𝐓𝐡𝐞 𝐋𝐚𝐳𝐲 𝐆𝐢𝐫𝐥

Yes, I am glad that you know so much about the scriptures; that is an important thing. I have to go to China right now, so let me tell you that I am in China right now and would like to learn a lot about China.

Thread Thread
 
xulingfeng profile image
xulingfeng

My one piece of advice: the first thing you do in China is open up your appetite and eat well.🤣

Thread Thread
 
technogamerz profile image
𝐓𝐡𝐞 𝐋𝐚𝐳𝐲 𝐆𝐢𝐫𝐥 • Edited

I don't feel hungry at all.😅

Thread Thread
 
xulingfeng profile image
xulingfeng

🤣Malatang麻辣烫,!latiao辣条! — you have to try them. I'm begging you, hahaha

Thread Thread
 
technogamerz profile image
𝐓𝐡𝐞 𝐋𝐚𝐳𝐲 𝐆𝐢𝐫𝐥

是的,绝对是。

Thread Thread
 
catnight_a_9f8d4f2f05a1b1 profile image
CatNight A

😁你会的语言也太多了吧

Thread Thread
 
xulingfeng profile image
xulingfeng

传说中的学霸🤣

Thread Thread
 
catnight_a_9f8d4f2f05a1b1 profile image
CatNight A

😀Bro 加个Q

Thread Thread
 
xulingfeng profile image
xulingfeng

不要,我社恐🤣

Thread Thread
 
catnight_a_9f8d4f2f05a1b1 profile image
CatNight A

😝

Thread Thread
 
catnight_a_9f8d4f2f05a1b1 profile image
CatNight A

到现在还是有点难受了 bro 刚被裁员没多久 🤪

Thread Thread
 
xulingfeng profile image
xulingfeng

英文好,你可以试下国际远程岗位,我记得dev这里有猎头发布远程岗位

Collapse
 
leob profile image
leob

"One system optimized for confidence, the other for truth" - nice, there's something poetic to it :-)

Collapse
 
alexshev profile image
Alex Shev

The muted-channel idea is a good warning for agent dashboards. A disabled signal is not the same thing as a removed capability. If the system can still observe, queue, or infer from the channel, then the UI state is only a social promise. The architecture has to prove what actually stopped.

Collapse
 
xulingfeng profile image
xulingfeng

That last line is the whole story in one sentence — "the UI state is only a social promise." Mark was staring at a dashboard that said everything was fine, but the architecture underneath had never actually stopped listening. The dashboard wasn't lying; it was just fulfilling the social promise.
The hard question is: who audits the promise? In #8, Alex built a second dashboard precisely because he didn't trust the social promise the vendor dashboard was making. Two different stories, same blind spot — the gap between what the UI says and what the system does.

Collapse
 
alexshev profile image
Alex Shev

Exactly. The second dashboard in #8 is really a refusal to accept the UI as the boundary of truth. Once the system can keep acting below the interface, the interface becomes documentation at best. The audit has to attach to capability and event flow, not to whatever state the dashboard is comfortable showing.

Collapse
 
alexshev profile image
Alex Shev

The muted-channel idea is a good warning for agent systems. A disabled or ignored signal can still shape behavior if another path keeps reading it. In production, I would want an audit that proves both sides: what can speak, and what the decision layer is actually allowed to hear.

Collapse
 
xulingfeng profile image
xulingfeng

Wait — Alex? As in the Alex from Stratagem #8? The guy who spent $660K on an Axon story and changed a config file at 2:03 AM? Same name as one of our main characters, man. Hope you're not doing production audits and browsing Dev.to at the same time 😄
Jokes aside, your muted channel point hits home.
The stuff that's documented as muted is easy to check. What gets me is the "second ear" you don't know is still listening — you won't find it until you trace every single pipe.
These days, whenever I run into a muted signal, I add one more question: who else is on this bus? Half the time the answer is "nobody". The other half — that's where the stories come from.

Collapse
 
alexshev profile image
Alex Shev

Ha, wrong Alex thankfully. No production config files at 2:03 AM today.

But that is actually why the story landed for me. The muted-channel problem is easy to recognize in real systems: something gets marked as ignored, disabled, or informational, then everyone forgets it can still shape behavior indirectly.

The dangerous part is not the channel being muted. It is the team losing track of what still listens to it.

Collapse
 
leob profile image
leob

Might sound a bit "off topic", but I like the little "atmosphere" details, like the cup of coffee gone cold, or the dust blowing over the later afternoon platform - do keep doing that kind of stuff, I like it! :-)

(again, that's the stuff that has a "visual" quality and would do nice in a movie rendering)

Collapse
 
xulingfeng profile image
xulingfeng

Haha, I'm really glad those little details come through. Have you noticed my writing's been getting better? I'm a little proud of it myself 😄 One of the rules I set for the 36 Stratagems series was to minimize internal monologue — so instead, I've been trying to use more external cues — actions, expressions, surroundings — to reflect what the characters are feeling inside. Thanks for the encouragement, I'll keep going. Fair warning though: the wrap-up post for Arc 2 is coming tomorrow, so the next stratagem might be a little delayed.

Collapse
 
leob profile image
leob • Edited

I like the wrap-up posts ("post-mortems"?) too!

"Have you noticed my writing's been getting better?" - I'm absolutely sure it did, but I would need to revisit earlier episodes (or the first series) to ascertain it ...

However, what you describe (external cues over internal monologue), that's a well-known technique, I think - "show, don't tell", isn't that what they call it?

Thread Thread
 
xulingfeng profile image
xulingfeng

Honestly, I'm not sure what it's called either — I'm not a professional writer. But I can tell you from a QA perspective: a lot of my work has been in black-box testing, so I've learned to observe how a piece of software or an app behaves externally, and infer its internal logic from that. That process feels like it taps into the same thing — that sense of learning one thing and understanding another by connection.

Collapse
 
vinimabreu profile image
Vinicius Pereira

The detail that stings: nobody ever decided JL's rules were wrong, they just never proved the new dashboard covered them. That's the step every migration skips, the coverage diff: enumerate what the old system detected and, for each rule, name its successor or write the one-line reason it retires. Muting skips that accounting, which is why it's worse than deleting. A deleted rule is an honest gap; a muted one is a gap wearing a green light. Fourteen months of alerts firing into silence is just the receipt that the audit never happened.

Collapse
 
xulingfeng profile image
xulingfeng • Edited

That line — "a gap wearing a green light" — pretty much says the whole thing in six words. 👏
Genuinely curious though: do you think the person who took over genuinely didn't know, or did they peek at the mute count, went 'yeah that's too many,' and just closed the tab?

Collapse
 
vinimabreu profile image
Vinicius Pereira

My honest bet is the middle option: they peeked, and the count itself did the deciding. Past a certain unread number an alert channel stops being information and becomes an accusation, opening it means inheriting work nobody assigned you, so the rational flinch is to close the tab and let the green dashboard stay the official story. Which is why I don't think discipline fixes this one. The fix is structural: a stream that goes unacknowledged for N days should escalate or die loudly, never accumulate. A backlog that can grow forever is designed to be ignored.

Thread Thread
 
xulingfeng profile image
xulingfeng

You're right that it's structural. The mute button is the perfect symptom — it feels like a decision but it's actually postponement dressed up as one. Deleting a rule forces you to answer "will I ever need this?" Muting lets you answer "not now" indefinitely.

Fourteen months of silence wasn't a monitoring failure. The system was working exactly as designed.

Collapse
 
hemapriya_kanagala profile image
Hemapriya Kanagala

This reminds me that replacing a system isn't always the same as replacing the knowledge behind it. The AI dashboard looked modern, but JL's rules still captured something valuable that was lost during the transition.

Collapse
 
xulingfeng profile image
xulingfeng

Exactly this. The dashboard changed, but the patterns didn't. JL just happened to be the one still reading the old logs.

Collapse
 
catnight_a_9f8d4f2f05a1b1 profile image
CatNight A

Collapse
 
technogamerz profile image
𝐓𝐡𝐞 𝐋𝐚𝐳𝐲 𝐆𝐢𝐫𝐥

Collapse
 
xulingfeng profile image
xulingfeng

Collapse
 
xulingfeng profile image
xulingfeng

Second set of 6 is done. Something interesting just happened — the summary post continues tomorrow.

Collapse
 
xulingfeng profile image
xulingfeng

Mark, welcome back.

Collapse
 
unified_mentor profile image
Unified Mentor

Great Chapter!

Collapse
 
xulingfeng profile image
xulingfeng

🎉Thank you!

Some comments may only be visible to logged-in visitors. Sign in to view all comments.