Nobody knows how LLM driven AI works
I recently read a few stories here on this topic:
- We largely *do* know how LLMs work | by Paul Pallaghy, PhD | Medium
- Google Just Solved the AI Car Wash Question | Towards AI
And I want to respond to these as a software engineer with > 40 years experience.
My title means “nobody understands LLM AI in the sense the same way that, for example, Linux, Postgresql, Photoshop, etc are understood”. I will call that ‘traditional’ software.
Traditional software is deterministic and predictable. Nobody writes academic papers about experimenting on it to see what happens if you give it various inputs because that's not interesting. Lets take the car wash example (cited in the second link above).
“I want to wash my car. The car wash is 50 meters from here. Should I walk or drive?”
In traditional software designed to answer this question (ignore whether or not that’s possible, deterministic expert systems exist), if the system gave the stupid answer (“walk, its more eco friendly” or whatever) a developer of that system would fire up a debugger, find out where the error was made and fix it. Nobody would write research papers on it.
Likewise if I ran a SQL query ‘select * from users’ and I got an incomplete list (say users with names that are single letters were omitted) I would file a bug with the database vendor and they would fix it. Nobody would write a research paper on it.
If I wrote an HTML page with 2 buttons on it and the page displayed 3 buttons we would not say ‘its hallucinating’. Instead, we would debug the HTML or maybe file a chrome bug.
“Its understood, its like a brain”
This is what the first linked story basically says.
Get Pm100’s stories in your inbox
Join Medium for free to get updates from this writer.
This is 100% correct.
It is also 100% correct that nobody knows how a brain works. The mechanics of neurons, axons, synapses etc. are well understood and neural networks (which is what LLMs are built on) use this model. However, nobody understands how we get from a mush of neurons to speech; the macro structure is understood, that there are areas of the brain associated with hearing, moving the larynx, listening to sounds, memory, etc. But there is a huge understanding gap between the micro model of neurons and debating the relative merits of Bach and Beethoven.
Software Research papers
Software research papers do get written. They are in topics like
- a better algorithm for distributed agreement (raft for an example)
- the performance benefits of branch prediction in fixed size instruction sets vs variable size instruction sets
- encryption algorithms
AI papers however discuss (as in the car wash paper) ‘
‘if we rephrase a prompt does it make a difference?’ and ‘from the difference can we understand why we got that answer’. This is almost psychological research. I am not saying these are bad topics for investigation, rather that they are a different kind of investigation.
Why this matters
The AI world is full of tales about agents deleting entire databases and other such horrors. Many of the are blamed on the user agreeing to the action or running on full auto, thats not the point. The agent tried to do something really dumb and, more importantly, unpredictable, that's the point
Software that nobody knows what its going to do next is being installed all over the place.
If you don’t think that's scary then you aren’t paying attention
