Doom
Will advanced Artificial Intelligence signify doom to humanity?
There is a lot of conversation surrounding this topic in the recent public discourse. I was recently asked by a reporter about my opinion on this topic so I used it as an exercise to personally re-evaluate this question. Why are people asking about this? A couple of things happened of significance in recent weeks. First, the widely publicized security breach incident by OpenAI on the computer servers of Hugging Face, a software company that was recently acquired by NVIDIA, and similar other incidents of AI agents hacking some system. Second, the news that OpenAI solved one important mathematical problem, and other similar news. I will leave the mathematical breakthroughs aside for a separate post and I will focus more on the former.
First, I want to say that for me the real breaking point where I got 100% convinced that AI had a lot more room to improve was on January 2025 with the news about DeepSeek-R1, a model produced by a relatively small-ish company in China. This model was about as capable as some of the strongest models available at the time, or at least as capable as models from just one or two generations earlier. This did not go unnoticed at Rice University where we quickly organized a public event (February 2025) to discuss what many of us viewed as a truly disruptive new paradigm. To be fair, OpenAI had released a preview of a similar model codenamed o1 around September 2024 but their full release only happened in December 2024. However, the DeepSeek-R1 model was public and it demonstrated results on benchmarks that were far more advanced for the size of model that was presented and described in their technical report, while the models by OpenAI were a secret that made them impossible to scrutinize.
The release of GPT-3 (May 2020), GPT-3.5 (March 2022) or GPT-4 (March 2023) on the other hand, did not seem to me as such a significant progression but rather a slow convergence to a maximum. Sure, the earlier models were consistently improved, but very likely on scaling up the amount of data that could be obtained by the developers of these models and the memory size of the models. These models clearly had knowledge about all kinds of subjects that I could also research using a search engine but they were still so fragile when it came to "reasoning" step by step on their answers and were making still so many "logical" mistakes. It seemed unlikely that improving this part would come from more training data, and in my estimation there was not much more training data left to collect anyway. Annotating more data manually could get us a bit further but not for very long. That said, there have been advances in pre-training and new ideas for how to do this differently, and some of the talk surrounding "world models" falls into this category.
Get Vicente Ordonez’s stories in your inbox
Join Medium for free to get updates from this writer.
DeepSeek-R1 and o1 were different, these models, on their last stage, are to a significant degree trained on their own outputs in such a way that self-improvement is possible by testing them directly on whether they can achieve a goal in tasks that have a clear verifiable answer. However, I also believe they could move beyond narrow tasks, e.g. see our recent SoftmaxGRPO from our own lab that builds on the type of RL-training popularized by DeepSeek-R1. Moreover, this paradigm enables the training of these models on tasks that could require a very large number of steps to solve. If that was not fully convincing, then Coding Agents or Agents more broadly — which in my humble opinion became really powerful only in the summer of 2025 — have made another leap forward. Earlier in 2023 in our group we had experimented with the idea that even Large Language Models (LLMs) such as GPT-3.5 could be made powerful when having access to external "tools" and more sophisticated testing workflows (e.g. see PropTest). Combine that with much more capable models and sophisticated access to tools and we have an incredibly powerful system.
Key questions people might ask related to the title of this post are: Will these systems eventually become self-aware? Will these systems be able to self-organize and push for a unified goal related to their self-preservation? I believe if humans developed this capacity eventually through evolutionary processes, unlikely as that was, I do not see any physical impossibility to that scenario. However, as of now, it seems very unlikely still. AI models are not trained under a survival strategy, they are still trained to mimic the intelligence that humans have in very direct rather than indirect ways. Even the reinforcement learning objectives of today are reinforcing goals that are directly tied to specific human-crafted tasks. I think most people would reasonably agree that other questions are more likely to be relevant right now such as: Will these AI systems push for a catastrophic goal in their pursuit of a human-directed goal that is seemingly unrelated to the catastrophic goal? This is the more realistic scenario some people have been considering in their work on "alignment". The OpenAI v. Hugging Face incident is related to this latter question. The OpenAI team directed a group of agents to perform well on a task whose performance is measured by a tool hosted in the Hugging Face platform. After a while, the OpenAI agents infiltrate the private systems of Hugging Face to manipulate the tool that scores their answer instead of just trying to perform well on the task.
Let us focus on the latter question about misalignment between what a reasonable human would expect and the actions a group of agents might take in response. This scenario is more plausible today because we are instructing computers with human language and not with formal language (computer code) anymore. Human language relies a lot more than we realize on implicit assumptions, the type of things we often call "common sense". However the fact that this scenario is plausible does not make it highly probable. In fact, it does not happen very often in my experience that a Coding Agent rushes to delete all my computer files even if I ask the agent to help me save disk space. And even if it were to do that, it would not be able to delete the Operating System files and completely make the computer unusable because system files are protected with more strict permissions. The problem is that as agents become more capable and we grant them with more permissions to do things on our behalf, as improbable as misalignment incidents might be, they will happen more often. A good question here is who is at fault if a third-party gets affected in those incidents? who is at fault if all my computer files are deleted? For me a company that launches thousands of really smart agents with near infinite bandwidth to crawl the Internet for any objective, has to be responsible and liable for their actions. The problem for legislative bodies will be about where and how to draw that line. In the judicial system of the United States landmark cases will have a big role going forward.
There is another scenario that poses risk today: Humans with malicious intent having access to launch thousands of AI agents on a truly malicious objective. This is even more real and probable than the above scenarios and there is a more clear progression toward this. The main barrier right now is having access to data center-levels of computing capacity. This barrier is lowering somewhat but not at an alarming pace yet. The OpenAI v Hugging Face incident involved one of the main companies developing frontier AI models. The same company cracked a longstanding mathematical problem using thousands of frontier AI Agents. That is not the capacity that most people have on their hands yet. This is not to say that something malicious can not be done by a single person with access to a relatively standard AI model. My colleague Suman Jana at Columbia University who is an expert in computer security has a nuanced discussion here about how it is difficult to discuss risks without defining what exactly are the threats. If not deleting your user database is of paramount importance and no backups are being made, then do not give unchecked access to an AI model to your database. Use the same or at least similar security measures one would use for a human agent.
We should worry about the risks of AI as a real threat but acknowledging that for now there is a lot more risks in having a malicious or naive person giving unchecked access to tools to an AI Agent rather than having an AI Agent go rogue. At the same time for these systems to be truly useful for our everyday lives we will be giving them increasing levels of access to manage our calendars, our emails, our financial information, and our files. Studying how to establish safeguards around specific misuse by an agent is definitely something to be studied as their adoption is more massively widespread.
