Terraform Drift Detection: 5 Root Causes and How to Catch Them Before Production
Why infrastructure drifts out of sync, why routine Terraform checks can miss it, and how to detect and fix changes before they become production incidents.
TL;DR
Terraform drift happens when a cloud resource’s live configuration no longer matches what your .tf files or state file describe, usually from a manual console change, a parallel automation script, or a third-party tool acting outside the IaC workflow.
Terraform plan won’t catch it by default, since it compares your configuration against the state file, not the actual cloud, which is why the fix starts with terraform refresh. This guide covers the five most common causes of drift, the practices that catch it between deployments, and where scheduled and PR-triggered detection still leaves a gap open.
Getting Started: What Terraform Drift Actually Means
Terraform works with two representations of your infrastructure. The desired state is the configuration defined in your .tf files, describing exactly which resources should exist and how they should be configured.
The actual state is what Terraform believes is running, recorded in terraform.tfstate every time you run terraform apply. Drift happens when what’s actually running in your cloud provider no longer matches either source.
A platform engineer debugging a production incident at 11 pm logs into the AWS Console, finds an EC2 autoscaling group’s launch template, and bumps the instance type from t3.medium to t3.large to stabilize things. Four days later, a routine terraform apply runs.
The state file still says t3.medium; the .tf configuration hasn’t changed, so Terraform sees no diff and reverts the launch template back to t3.medium. The scaling issue returns, and it takes two hours to trace the regression back to that apply, because every tool reading Terraform as the source of truth showed zero issues the whole time.
Why Terraform Plan Doesn’t Catch Drift by Default
terraform plan doesn’t query your cloud provider. It compares the .tf configuration against terraform.tfstate, and if nobody has touched that file since the last apply, it reflects history rather than the current cloud reality.
If an engineer logs into the AWS Console and switches an S3 bucket from private to public-read, the state file still says private, the .tf file still says private, and terraform plan returns no changes. The bucket is publicly accessible. Terraform reports nothing to fix.
This is why drift doesn’t surface through normal development workflows. A team can run an entire sprint of PRs, reviews, and applies without any indication that a production resource has sat misconfigured for a week, because the checks being run compare .tf against the state file. The actual cloud only enters that comparison once someone explicitly runs terraform refresh first.
Five Root Causes of Configuration Drift in Terraform-Managed Infrastructure
Each cause creates a different kind of gap, and knowing which kind you’re dealing with changes how you fix it:
- Out-of-band console and CLI changes. An engineer opens a security group inbound rule to debug connectivity, or a DBA bumps allocated_storage on an RDS instance during a capacity crunch. Legitimate operational decisions made under pressure, none of which touch the .tf files or update the state file.
- Parallel automation pipelines. A CI/CD pipeline sets a Lambda environment variable directly through the AWS CLI, or a separate Pulumi stack writes route table entries Terraform expects to own. These systems make real changes to real resources that Terraform never requested and never recorded.
- Third-party tooling that modifies resources post-provisioning. An Ansible playbook updates system packages on an EC2 instance, or a cloud provider’s managed patching service bumps an RDS minor version. Terraform keeps operating as if the original configuration is still in place.
- Direct terraform.tfstate edits. Teams occasionally edit the state file by hand to unstick a failed apply or force a resource re-create. If the edit doesn’t accurately reflect the actual cloud configuration, the state file becomes a false baseline for every subsequent plan.
- Cloud provider API updates. AWS, GCP, and Azure periodically change default attribute values, encryption defaults on S3 buckets, or the default http_tokens setting on EC2 instance metadata. The drift wasn’t caused by anyone on the team, but terraform plan still flags it as a change to apply.
How terraform refresh and terraform plan Work Together to Surface Drift
Terraform doesn’t continuously monitor your cloud environment. It only detects drift when you ask it to, through two commands run in sequence. terraform refresh queries the cloud provider APIs for the current configuration of every resource in the state file and rewrites terraform.tfstate to match. Without this step, the state file holds whatever Terraform last wrote, which may be weeks out of date.
Once the state file is current, terraform plan compares it against the .tf configuration and produces a diff. Take a security group that restricts inbound HTTP to 192.168.1.0/24. If someone opens it to the world through the console, the state file and .tf file both still say 192.168.1.0/24, so nothing looks wrong until a refresh runs. After that, the plan output looks like this:
~ aws_security_group.web_sg
- cidr_blocks = ["0.0.0.0/0"]
+ cidr_blocks = ["192.168.1.0/24"]The line is what’s actually running in the cloud right now. The + line is what Terraform will restore from the .tf configuration on the next apply. Until someone runs that refresh and plan, the security group stays open to the entire internet with no automated alert firing.
Why Drift Compounds Faster Than Teams Expect
A single drifted resource is manageable on its own. The problem is that drift rarely stays isolated. An engineer opens an inbound port to fix an incident; the fix works, but the rule now allows traffic from anywhere.
A week later, a second engineer deploys a new service behind that same security group, not knowing the inbound rules changed, and the new service inherits the open rule. Two services are now exposed, and the original console change that caused it is buried in CloudTrail from a week earlier, disconnected from the deployment that just inherited it.
Get Yuvraj Singh’s stories in your inbox
Join Medium for free to get updates from this writer.
The risk compounds hardest in three situations: when multiple engineers share infrastructure with no central inventory of what owns what, when console changes made during incidents never get cleaned up afterward, and when multiple tools (Terraform, CDK, Ansible, manual console access) all have write access to the same resources. In each case, Terraform’s view of the infrastructure diverges from reality faster than routine plan runs can catch it.
Five Practices That Reduce Drift Before It Reaches Production
These cover the full drift lifecycle: keeping drift out of the state file, catching it between deployment cycles, and correcting it without breaking stateful resources in production.
- Automate terraform plan on a schedule, not just before deployments. Drift that enters infrastructure on a Tuesday doesn’t get caught until the next pipeline run, which might be Thursday. A scheduled job that runs terraform plan across every environment independently of pending deployments, hourly for critical IAM resources and daily for compute and networking, catches divergence between cycles.
- Enforce GitOps as the only path for infrastructure changes. Every .tf change goes through a PR, which triggers terraform plan automatically and posts the output as a comment before any reviewer approves the merge. Git history becomes the audit trail: if a change isn’t in a commit, it didn’t go through the approved workflow.
- Ingest cloud audit logs to catch out-of-band changes in real time. AWS CloudTrail, Azure Activity Logs, and GCP Cloud Audit Logs all record manual console and CLI changes. Alerting on write operations against Terraform-managed resources catches them as they happen, not when the next plan runs.
- Enforce least privilege on Terraform-managed resources. Scoping production IAM roles to read-only for most engineers, and reserving write access for the service account running terraform apply in CI/CD, makes the IaC workflow the path of least resistance rather than something engineers have to remember to use.
- Apply automated remediation with guardrails for stateful resources. Reverting a stateless security group rule through an automated pipeline apply is safe. Reverting an RDS storage increase made during a capacity crunch could cause an outage, so stateful resources need human approval before apply and a prevent_destroy = true lifecycle rule to block accidental deletion during automated correction.
Where PR-Triggered Checks and Scheduled Scans Still Leave Gaps
Two approaches commonly run alongside Terraform for drift detection, and understanding where each stops clarifies what still has to be handled by hand.
PR-triggered plan checks run terraform plan automatically when a pull request opens and post the diff for code review, which is genuinely useful, but the trigger is the limit: these checks only fire when a PR opens. A console change made on Monday in an environment where no PR opens until Friday sits invisible for that entire window.
Scheduled state comparison tools take a different angle, comparing live cloud state against the state file to surface resources that exist in the cloud but have no state file entry at all, which fills a real gap for teams with a backlog of manually created resources.
The limit here is scope and continuity: these tools need to run on a schedule, not continuously monitor drifted managed resources; they offer no actor attribution and no remediation path.
Even with both approaches running, an engineer still has to set up and maintain the detection schedules, cross-reference audit logs to find who made a change, decide whether to revert or codify each drifted resource, write the corrective .tf change, and get it reviewed by hand.
How Firefly Closes the Gap Left by Manual Drift Investigation
The gap left by scheduled and PR-triggered detection isn’t just about speed; it’s about everything that happens after detection: tracing a diff back to a specific person and action, deciding whether to revert or codify, writing the corrective IaC, opening the PR. Firefly replaces that manual workflow directly, and it’s built around the same window-between-runs problem described above.
Firefly uses event-driven detection, so it flags a resource configuration change in the cloud provider the moment it happens, rather than at the next scheduled scan or the next opened PR. Mechanically, this runs the same checks that terraform refresh and terraform plan perform, just continuously instead of on demand, against Firefly’s live Inventory of every resource’s actual configuration.
Attribution comes from a parallel source: Firefly’s Event Center ingests the same audit trail (AWS CloudTrail, Azure Activity Logs, or Google Cloud Audit Logs) a team would otherwise cross-reference by hand, and matches the mutation event to the drift the moment both land, naming the exact IAM user or service account responsible without anyone opening a CloudTrail console.
For a drifted resource, Firefly shows a field-by-field diff between the running cloud configuration and the desired IaC state, with the cost impact of that gap calculated alongside it, so remediation gets prioritized by business impact rather than a generic severity label.
Remediation itself offers two paths instead of the one-directional revert a plain terraform apply gives you: reconcile the cloud resource back to what the IaC defines, or codify the manual change into the Terraform configuration through a pull request, if the manual change turned out to be the correct call.
Either path opens a PR in the connected repository with the diff already written, so no fix bypasses code review. Alerts fire through Slack, PagerDuty, or a webhook within minutes of the change, and the same drift data is queryable through Firefly’s MCP server from Claude Code, Cursor, or Claude Desktop, so a question like “which resources have drifted in the last 24 hours” gets answered without leaving the IDE.
Where Should You Start With Terraform Drift Detection
The core Terraform workflow — terraform refresh to sync the state file, terraform plan to surface the diff, terraform apply to correct it — is the right foundation, and running it consistently through GitOps and least-privilege access eliminates most drift before it reaches production.
The foundation breaks down in the window between pipeline runs: drift that enters infrastructure on a Tuesday doesn’t get caught until Thursday’s plan, and in active incident response, that window is exactly when the most consequential manual changes happen.
Firefly’s drift management platform closes that window by detecting drift the moment it occurs and generating both remediation paths, so nothing gets reverted or codified without a deliberate decision. Book a demo to see how much of your own Terraform-managed estate is actually still in sync with what’s running.
FAQs
What is infrastructure drift in Terraform?
Infrastructure drift occurs when a cloud resource’s actual configuration differs from what Terraform expects based on the state file or the .tf configuration. A security group that allows traffic from anywhere in AWS while the state file still records a restricted CIDR block is a drifted resource. Terraform won’t flag this automatically; it requires running terraform refresh followed by terraform plan to surface the difference.
How does terraform plan actually detect drift?
terraform plan compares the .tf configuration against the terraform.tfstate file and outputs a diff. To detect drift from manual console or CLI changes, run terraform refresh first to update the state file with the actual cloud configuration. Without that refresh, the plan only compares configuration against Terraform’s last recorded state, which may not reflect current reality.
Can Terraform automatically fix drift once it’s detected?
Running terraform apply after detecting drift reverts a drifted resource to the desired state, but this still requires a human to review the plan output and confirm the apply. Auto-applying drift corrections in production carries real risk: a resource changed during incident response may have been modified intentionally, and reverting it automatically could reintroduce the original problem.
What’s the difference between a drifted resource and an unmanaged resource?
A drifted resource exists in the Terraform state file, but its actual configuration has diverged from what the state file or .tf files describe. An unmanaged resource exists in the cloud but doesn’t appear in any state file at all, so Terraform has no awareness that it exists. terraform plan detects drifted resources; it doesn’t surface unmanaged ones.
How is event-driven drift detection different from running terraform plan on a schedule?
Scheduled terraform plan runs catch drift at fixed intervals, hourly, daily, or tied to a pipeline, and drift that happens between those intervals sits undetected until the next run. Event-driven detection surfaces drift the moment a resource configuration changes in the cloud provider, rather than waiting for the next scheduled check or opened PR.
Does multi-cloud infrastructure make drift harder to catch?
Yes. Native drift checks are typically scoped to one IaC tool’s state, so a team running Terraform alongside CloudFormation or Pulumi across multiple clouds has to run separate detection for each one, with no single view of what’s drifted across all of them. A tool watching only one cloud or one IaC framework has no visibility into changes happening anywhere else.
This story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!
