Between Rules & Vectors
What rebuilding 1964 AI in Elixir taught us…
What does one generally do in summer maybe go for a vacation or build a hobby but we have chosen a different path.
During our summer break, we wanted to build something from scratch instead of only reading about it. Our mentor pointed us to Chapter 7 of Peter Norvig’s classic book, Paradigms of Artificial Intelligence Programming, which explains STUDENT, a program Daniel Bobrow wrote in 1964 for his PhD at MIT, which we found super interesting!
So we rebuilt a “1964 Word-problem Solver in Elixir” during our break. Originated from one age puzzle which showed us exactly the transformation works from rules to neural networks.
Look at the two versions of the same age puzzle.
“Tom’s father’s age is four times Tom’s age. The sum of Tom’s age and his father’s age is 50. What is Tom’s age?” — Our program solved it in milliseconds.
“Tom’s father is four times his age. Together they are 50. How old is Tom?” — It was completely stuck and was confused to process further!
Let’s understand the basic for that…
How did computers first learn to read a word problem? and why did a program from 1964 get stuck on a sentence that today’s chatbots handle without effort? what stopped this ?
In both scenarios, the math is the same : Tom is 10 and his father is 40.
A ten-year-old understands both versions and so does any modern chatbot. We started digging deep on why our program could not do that and how modern language models really work.
Get Slesha_aineni’s stories in your inbox
Join Medium for free to get updates from this writer.
Bobrow’s idea was simple : Word problems are not creative writing. Words like “is”, “times” and “percent of” point straight to math operations. So you do not need a computer that truly understands language instead need good rules that turn basic English into equations.
What we worked on
Bobrow wrote STUDENT in Lisp. We rebuilt it in Elixir, whose pattern matching and pipe operator (|>) were a perfect fit. Our program passes a sentence through small steps, like an assembly line:
"If z is 3, what is twice z?"
|> Tokenizer.tokenize() # splits the sentence into words
|> Noise.strip() # removes filler words
|> Matcher.match_and_translate() # turns phrases into a symbolic tree
|> Flattener.flatten() # turns the tree into plain equations
|> Solver.solve() # finds the unknown values
# => %{z: 3, what: 6}For problems with more than one unknown, we borrowed an idea from Norvig’s Sudoku solver : once you know how two unknowns are related, pass that fact into the other equations. For “The number of boys is twice the number of girls, and their sum is 60”, our solver wrote boys = 2 * girls, got 3 * girls = 60 and found girls = 20 and boys = 40.
What it solves
The result is a “solver” that turns plain-English word problems into equations and solves them exactly. It passed 65 unit tests covering percentages, age puzzles, decimals and problems with several unknowns. It’s big strength is trust : once an equation is built, the answer is exactly correct and never made up. Its limit is that it only understands the patterns it was taught. That limit showed up sooner than we expected.
Challenges while building this…
English fought back as we added more tests :
- The trap of “of”. We removed “of” as a useless word, so “the number of students” became “students”. Then “15 percent of 200” turned into “15 percent 200”, and the percentage rule broke.
- The disappearing “a”. Our filter deleted “a” as an article, but in “If a is 5, what is twice a?”, the “a” was the unknown!
- Small traps. Splitting on periods cut “2.5” into pieces, and the minus in “-5” made our solver invent a ghost variable called “minus”.
# Before: "of" was always removed
Noise.strip(["15", "percent", "of", "200"])
# => ["15", "percent", "200"] percentage rule broke
# After: keep "of" when it follows "percent"
# => ["15", "percent", "of", "200"]
# Before: "a" was always removed
Noise.strip(["if", "a", "is", "5"])
# => ["if", "is", "5"] the unknown vanishedEach problem had a fix. But the puzzle that rules could not fix was the second one :
“Tom’s father is four times his age. Together they are 50. How old is Tom?”
- Who does “his” mean? The word “his” is not the word “Tom”. A pattern matcher only checks if words are exactly equal, so to our solver they were unrelated.
- Missing words. “Together” never says “sum”, and “is four times his age” skips the phrase “father’s age” that our patterns expected.
We could add rules for “his” and “together”. But what about “combined”, “in total” or “altogether”? Every new rule added complexity and could clash with an old one. We realised that language is held together by meaning, not just grammar.
What this taught us about modern AI
That was when our machine learning classes finally made sense. In a rule-based system, words are separate blocks with no link to each other. Neural networks changed that. In 2013, Google’s Word2Vec made word embeddings popular: each word becomes a point in a space of numbers, and words used in similar ways, like “total” and “combined”, land close together. In 2017, the Transformer (from the paper “Attention Is All You Need”) let models learn which words in a sentence matter most to each other, so “his” can learn to point back to “Tom”.
The trade-off is clear. Rule-based engines are exact but fragile. Language models understand wording very well but can slip on multi-step arithmetic.
This points to neuro-symbolic AI :
A neural model, which learns from data and is good with language, working together with a symbolic engine, which follows exact rules and logic.
In our case, a neural model would read the Tom puzzle and write father = 4 * tom and father + tom = 50. Then an exact solver like ours would work out tom = 10 and father = 40. The neural part understands the wording, and the symbolic part guarantees the maths.
Immanuel Kant is often paraphrased as saying:
“Experience without theory is blind, but theory without experience is mere intellectual play.”
Hitting the limits of our own rule engine showed us why the neural network revolution had to happen, and it made us curious enough to keep learning…
Reading about STUDENT was theory.
Rebuilding it was an experience.
Rebuilding a classic — We took an existing 1964 program and rebuilt it ourselves. To do that, we had to understand & implement it and that gave us whole experience far more than reading alone.
Learning by building — We learned Elixir while re-implementing STUDENT in it. It’s pattern matching suited the problem, so every rule we wrote was both a lesson and a small win.
Applying what we learn — We read about constraint propagation in Peter Norvig’s Sudoku solver, then applied the same idea in our STUDENT solver, so that solving for one unknown helps solve the others. Doing this in real-time gave us a rush!
What it taught us — Trying to break our own solver showed us more than stopping once the tests passed.
Rules can take us only few steps could but by learning embeddings, Transformers and neuro-symbolic AI, it makes easier to cross the walls!
