Who Pays If Anthropic’s AI Goes Too Agentic?

This essay was originally published via my newsletter.

Imagine that while you’re sleeping, your trusted AI agent is working for you. But there’s a problem. It fails.

Instead of following the script you specified, it takes irreversible actions like deleting a client database. Then it executes unauthorized financial transactions. Maybe that was other people’s money you were holding.

This is what I mean in the title when I say too agentic. An AI agent — a program that doesn’t just answer questions but actually acts, supposedly on your behalf — can cause serious trouble for yourself and others.

Don’t take my word for it. Read Anthropic’s S-1. It spends almost 100 pages going through numerous risk factors from its products (or services) causing the end of humanity to running wild with broad systems access that can potentially go unsupervised for days.

Yet through all of these risks Anthropic is reportedly targeting a $1.5 — $2 trillion valuation. It’s expected to go public in late November, just in time for Thanksgiving in America.

You can probably imagine why these risk factors are not making many feel thankful.