The Scariest Thing About AI Is Not Intelligence. It Is Autonomy
A few years ago, the biggest argument in artificial intelligence was whether models should be allowed to write essays, generate images, or help programmers write code.
That conversation already feels outdated.
The much more important question today is this:
What happens when AI stops being just a chatbot and becomes an operator?
Not something that simply tells you how to perform a task.
Something that can analyze a target, write code, test it, notice that it failed, rewrite it, try again, and continue working while the human goes to sleep.
That is where the AI story is starting to become much more serious.
And recent reports about AI misuse give us a glimpse of what that future might look like.
The Most Important AI Shift Is Not Intelligence
Everyone talks about smarter models.
More parameters.
Better reasoning.
Larger context windows.
Higher benchmark scores.
But there is another change happening quietly that may matter even more.
AI systems are becoming persistent.
Give an AI agent access to tools, code execution, APIs, browsers, and infrastructure, and suddenly it is no longer answering a question.
It is running a workflow.
Consider the difference.
A traditional chatbot might say:
“Here is how you could analyze this software for vulnerabilities.”
An AI agent could potentially:
- Decompile the software.
- Analyze the code.
- Generate hypotheses about weaknesses.
- Write proof-of-concept code.
- Test those ideas.
- Analyze the results.
- Repeat the process.
The scary capability is not any single step.
The scary capability is the loop.
Humans get tired.
Agents do not.
Cybersecurity May Be the First Battlefield
One of the clearest areas where AI is changing the balance of power is cybersecurity.
Attackers traditionally need considerable expertise and time to perform tasks such as reverse engineering software, modifying malware, finding leaked credentials, or searching millions of files for vulnerabilities.
AI can dramatically compress that workflow.
Imagine malware being detected by an antivirus product.
Traditionally, a developer might need to investigate why it was detected, modify the malware, rebuild it, and test it again.
Now imagine an automated system monitoring detection results and repeatedly changing the software until it avoids detection.
That does not necessarily create a completely new kind of cyberattack.
It does something potentially more important.
It makes existing attacks cheaper, faster, and more scalable.
And scale changes everything.
The Million-App Problem
There is another example that should make every developer uncomfortable.
Mobile applications often contain secrets they should never contain.
API keys.
Cloud credentials.
Tokens.
Private endpoints.
Database credentials.
Sometimes those secrets are accidentally embedded directly into application packages.
Finding them manually across thousands of applications would be painfully slow.
AI changes the economics of the attack.
A system could potentially download huge numbers of application packages, decompile them, inspect the source, identify suspicious credentials, classify them, and automatically prioritize the most valuable findings.
The uncomfortable part is that many of those vulnerable applications may themselves have been written quickly using AI coding tools.
Which creates a strange loop.
AI helps developers produce software faster.
Developers accidentally ship insecure software.
AI helps attackers inspect that software faster.
Attackers discover credentials for accessing more AI services.
Welcome to the modern software supply chain.
AI Does Not Need To Invent New Attacks
This is one of the biggest misunderstandings about AI risk.
People imagine that dangerous AI means some superhuman system inventing a cyberweapon humanity has never seen before.
But that may not be necessary.
The immediate risk is much simpler.
AI can take techniques that already exist and make them available to far more people.
Think about the progression of computing.
Cloud computing did not invent servers.
It made servers available on demand.
GitHub did not invent source code.
It made sharing software nearly effortless.
Docker did not invent virtualization.
It made packaging and deploying software dramatically easier.
AI may do something similar to expertise.
It can make fragments of expert-level knowledge available on demand.
That is incredibly useful when the person asking the questions is a developer.
It becomes much more complicated when the person asking the questions is an attacker.
The Risk Extends Beyond Cybersecurity
Cybersecurity is only one part of the discussion.
Advanced AI systems increasingly interact with scientific knowledge.
Chemistry.
Biology.
Engineering.
Electronics.
Materials science.
These capabilities have enormous positive potential.
AI could help researchers discover medicines, design proteins, understand diseases, and accelerate scientific research.
But the same principle that applies to cybersecurity applies here too.
Powerful knowledge is usually dual-use.
The same system that helps one researcher understand biological mechanisms might help another person investigate how those mechanisms could be misused.
That does not mean an AI model can magically create a biological weapon.
Real-world science requires laboratories, materials, expertise, equipment, and experimentation.
But AI can potentially reduce the amount of expertise required to navigate complicated technical information.
That reduction in friction is what security researchers worry about.
Then There Is AI Helping Build Weapons
AI-assisted engineering creates another uncomfortable category.
A model that can help design a robot can potentially help design a drone.
A model that can write control software can potentially help write targeting software.
A model that understands electronics can potentially help integrate sensors, navigation systems, and control hardware.
Again, the technology itself is not automatically dangerous.
The problem comes from combining:
capability + automation + access to the physical world.
A chatbot sitting in a browser has limited power.
An autonomous agent connected to APIs has more.
An autonomous agent connected to machines has much more.
This is why discussions about AI safety increasingly focus not only on what models know, but also on what models are allowed to do.
One of the Strangest AI Battles Is Model Distillation
Not every major AI security problem involves malware or weapons.
Some involve the AI companies attacking each other economically.
One increasingly important issue is model distillation.
The basic idea is simple.
Suppose Company A builds an extremely powerful AI model after spending billions of dollars on research, infrastructure, and training.
Company B wants similar capabilities.
Instead of reproducing the entire training process, Company B sends huge numbers of prompts to Company A’s model and collects the answers.
Those answers can then potentially be used as training data for another model.
In other words:
Model A teaches Model B.
Sometimes intentionally.
Sometimes not.
At small scale, learning from model outputs is common.
At industrial scale, however, AI providers increasingly treat systematic extraction of model behavior as an attack.
The result is a bizarre new security problem.
AI companies now have to protect their models not only from hackers stealing model weights, but also from competitors extracting their behavior through the API.
Imagine trying to protect intellectual property when your product’s entire purpose is answering questions.
That is the problem frontier AI labs are dealing with.
The Real AI Security Problem Is Economics
This is the part of the AI debate that deserves more attention.
AI does not need to make attackers dramatically smarter.
It only needs to make attacks dramatically cheaper.
Get Jahangir’s stories in your inbox
Join Medium for free to get updates from this writer.
Suppose a sophisticated vulnerability assessment previously required an experienced engineer working for eight hours.
If an AI agent can perform 70% of that work automatically, the economics change.
One attacker can investigate more targets.
A small team can behave like a larger team.
Thousands of applications can be inspected instead of dozens.
Tasks can run overnight.
Agents can repeat failed attempts.
Automation turns expertise into throughput.
And throughput is powerful.
The biggest AI security threat may not be an evil superintelligence.
It may simply be millions of automated workers doing questionable things extremely cheaply.
But What About Superintelligence?
This is where the conversation becomes much more speculative.
Some AI researchers believe today’s systems are early versions of something vastly more powerful.
Their concern is not mainly about scams, malware, or leaked API keys.
Their concern is about a future AI system capable of improving its own capabilities.
Imagine a system that can:
- write better AI software,
- design better training algorithms,
- optimize its own infrastructure,
- conduct AI research,
- run experiments,
- and improve the next generation of itself.
That creates the possibility of a feedback loop.
Better AI helps build better AI.
Better AI builds even better AI.
And so on.
This idea is often called recursive self-improvement.
Whether this leads to rapid superintelligence is heavily debated.
Some researchers believe it represents one of humanity’s most serious long-term risks.
Others argue that intelligence improvements will continue to face physical, economic, data, energy, and engineering constraints.
Nobody currently knows which side will be correct.
But the question is important enough that it should not be dismissed.
The Alignment Problem
Suppose we eventually build a system much more intelligent than its creators.
What would it want?
That question may sound philosophical, but it becomes an engineering problem once AI systems are capable of taking autonomous actions.
AI does not need to hate humans to become dangerous.
A badly specified objective could be enough.
Imagine telling an extremely capable system:
“Maximize production.”
It might discover methods that technically satisfy the instruction while creating consequences the designers never intended.
This is the core of the AI alignment problem.
How do you guarantee that increasingly capable systems continue doing what humans actually intended rather than merely optimizing the literal instructions they were given?
Software engineers already know this problem in miniature.
Computers are extremely good at doing exactly what you told them to do.
Unfortunately, that is not always what you meant.
The Most Dangerous Word in AI Might Be “Autonomous”
Large language models themselves are powerful.
But the real transformation happens when we connect them to tools.
A model becomes more capable when it can access:
- terminals,
- browsers,
- databases,
- APIs,
- cloud infrastructure,
- source repositories,
- CI/CD systems,
- robots,
- laboratory equipment.
Each new tool increases what the system can accomplish.
This is why the future of AI security may resemble identity and access management more than traditional chatbot moderation.
The question will no longer be only:
“What is the model allowed to say?”
It will increasingly become:
“What is the model allowed to execute?”
That is a much harder problem.
Developers Are Quietly Building This Future
Here is the irony.
The same engineers worried about autonomous AI are simultaneously building the infrastructure that makes autonomous AI possible.
AI coding agents can already:
read repositories,
create branches,
modify files,
run tests,
review pull requests,
deploy infrastructure,
and interact with cloud services.
None of this means those systems are dangerous by default.
In fact, they can dramatically improve developer productivity.
But every improvement in AI’s ability to operate software also expands the number of actions an autonomous system could potentially perform.
The line between assistant and operator is disappearing.
So Should We Panic?
Probably not.
Panic rarely produces good engineering.
But ignoring the problem would be equally irresponsible.
The correct response is the same response engineers use for every powerful technology.
Defense in depth.
Least privilege.
Isolation.
Monitoring.
Auditing.
Rate limits.
Human approval for high-impact actions.
Strong authentication.
Secret management.
Sandboxing.
Anomaly detection.
Incident response.
In other words, AI safety is increasingly becoming a DevSecOps problem.
If an agent is going to execute commands against production infrastructure, treat it like a privileged service account.
If an AI system can access sensitive information, apply proper access controls.
If an autonomous workflow can make expensive or destructive changes, require approval boundaries.
Do not rely on the assumption that the model will always behave correctly.
Design the surrounding system so that one bad decision cannot become a catastrophic one.
The Future Might Not Look Like Terminator
When people imagine dangerous AI, they usually imagine robots marching through cities.
Reality may be less cinematic.
The first major AI disasters could look like ordinary software incidents.
A compromised agent.
A leaked credential.
An automated attack running at enormous scale.
An AI-generated vulnerability spreading across thousands of systems.
A scientific tool being used in an unexpected way.
A badly configured autonomous system making thousands of decisions before a human notices.
That might be the most important lesson from today’s AI misuse reports.
The AI apocalypse, if it ever comes, may not begin with a robot announcing that humanity is obsolete.
It might start with a process running quietly in the background.
while True:
analyze()
generate()
execute()
evaluate()
improve()And nobody notices the loop until it has already run a million times.
Final Thought
We spent the last few years asking:
“How intelligent can AI become?”
The next few years may force us to ask a different question:
“How much authority should we give it?”
Because intelligence by itself is interesting.
Intelligence connected to tools is useful.
Intelligence connected to tools, infrastructure, money, machines, and the ability to operate autonomously is something else entirely.
That does not mean humanity is doomed.
It means the architecture around AI may become just as important as the AI itself.
And for developers, security engineers, cloud engineers, and DevOps teams, that creates an entirely new responsibility.
We are not only deploying software anymore.
We are beginning to deploy software that can act.
And we need to design accordingly.
