How Your Security Robot Will Eventually Shut Down the Wrong Server

Umashankara Kalaiah

3 min read

·

Aug 8, 2026

--

Listen

Share

Automate the things you can undo. Ask a human before the things you can’t.

GuardDuty spots a compromised EC2 instance. A Step Functions workflow responds. The tempting design is “flag goes up, robot terminates instance” — until the flag is wrong and you’ve taken down production faster than any attacker could have.

The fix is to sort every action by one question: can I undo this? Snapshots and network isolation run instantly. Termination waits for a human. And the part almost every tutorial skips — you decide in advance what happens when nobody answers at 3 a.m.

The 3 a.m. problem

GuardDuty raises false alarms. Your own pen test trips it. A vulnerability scanner trips it. A legitimate batch job that suddenly talks to a new region trips it.

So “alert fires → terminate instance” means you will, eventually, destroy a healthy server because a scanner did its job. You didn’t get attacked. You attacked yourself, automatically.

The standard advice is to add an approval step. Fine — but then nobody finishes the thought. The on-call engineer is asleep. The approval message sits unread. If you don’t set a time limit, Step Functions will happily wait up to a year while a real intruder keeps working.

You have to pick an answer. There’s no universally right one. There is a wrong one: not picking.

The rule that makes this work

Split your response actions into two piles.

Undoable — run immediately, no approval:

• Snapshot the disk (evidence, and it costs pennies)
• Tag the instance as suspicious
• Move it to an isolated security group

None of these break anything. The server keeps running. If it’s a false alarm, you reverse it in one command — and meanwhile the evidence got collected while everyone slept.

Not undoable — always ask:

• Terminate the instance
• Revoke the stolen credentials
• Disable the IAM user

Most people gate everything (so nothing useful happens overnight) or gate nothing (so you eventually delete the wrong thing). Sorting by reversibility fixes both.

Step 1: Catch the finding

GuardDuty scores findings 1–8.5. Only wake the machine up for serious ones:

{
  "source": ["aws.guardduty"],
  "detail-type": ["GuardDuty Finding"],
  "detail": { "severity": [{ "numeric": [">=", 7] }] }
}

That numeric block matters. "severity": [7] matches only exactly 7 and silently misses your 8.0 findings — a common and very quiet bug.

Point the rule at a Step Functions state machine as its target.

Step 2: Do the reversible things right away

Quarantine is a security group with no rules at all — nothing in, nothing out:

SG=$(aws ec2 create-security-group --group-name quarantine \
      --description "no ingress, no egress" --vpc-id vpc-abc123 \
      --query GroupId --output text)

# New SGs get an allow-all outbound rule by default. Remove it.
aws ec2 revoke-security-group-egress --group-id "$SG" \
  --protocol -1 --port -1 --cidr 0.0.0.0/0

That last command is the one everyone forgets. Skip it and your “isolated” instance can still reach the internet — including whoever it’s exfiltrating data to.

Swapping the instance onto it is one call, and reversible:

aws ec2 modify-instance-attribute --instance-id i-0abc123 --groups "$SG"

Step 3: Ask a human before the rest

This is the state that does the work. Step Functions hands you a token, pauses, and waits for someone to send it back:

"AskAHuman": {
  "Type": "Task",
  "Resource": "arn:aws:states:::sns:publish.waitForTaskToken",
  "Parameters": {
    "TopicArn": "arn:aws:sns:us-east-1:111122223333:security-oncall",
    "Message": {
      "instance.$": "$.detail.resource.instanceDetails.instanceId",
      "finding.$":  "$.detail.type",
      "approve.$":  "States.Format('https://approve.example.com/?t={}', $$.Task.Token)"
    }
  },
  "TimeoutSeconds": 900,
  "Catch": [{
    "ErrorEquals": ["States.Timeout"],
    "Next": "LeaveIsolatedAndPage"
  }],
  "Next": "TerminateInstance"
}

Read the two exits:

• Someone approves within 15 minutes → TerminateInstance.
• Nobody answers → LeaveIsolatedAndPage. The instance stays quarantined and harmless, and the alert escalates to a second on-call. Nothing gets destroyed by silence.

That TimeoutSeconds line is the entire design decision, and it's the line most examples on the internet don't have.

Two traps

1. Your approval link is a credential. That URL terminates EC2 instances. Build it the easy way — an open API Gateway endpoint that calls SendTaskSuccess — and anyone who gets the link can shut down your servers. Forwarded email, screenshot in Slack, done. Put auth in front of it and expire the token.

2. Callbacks only work on Standard workflows. .waitForTaskToken does not exist in Express. Teams build the whole state machine on Express to save money and discover this at the very end.

Automation isn’t the risky part. Automation with no undo button is. Snapshot and isolate on instinct — and make a human own every decision you can’t take back.