When AI Can Write the Code, What Does a Beginner Still Need to Know?
The Human Side of the Loop, Part 5: when a machine can produce the answer, the skill worth learning is recognizing which question it actually answered.
The chart is clean, the break-even point is precise, and the comparison is wrong, because one plan is priced per user and the other per team.
Picture asking an assistant to compare two subscription plans. A polished spreadsheet arrives in seconds, with a break-even point, a chart and a recommendation. One plan charges per user and the other per team, and the question is whether you would notice that the comparison mixed incompatible units.
That is a hypothetical, but the classroom evidence points the same way. In a PNAS field experiment with nearly a thousand high-school math students, a generic GPT-4 tutor improved their assisted practice and was followed by worse performance once the help was removed.
So the real question is not whether AI can do the calculation. It is what you need to understand so the calculation keeps working for you.
Part 5 turns from the tools to the learner: what stays worth knowing when execution is cheap.
What you’ll learn
- Separate doing, knowing and checking, the three skills AI discussions bundle together
- Ask for the denominator before trusting an impressive percentage
- Read what the AI-and-learning studies actually measured, and what they did not
- Match how much math and code you learn to the consequences you intend to own
Prerequisites: none. You have asked an AI for an answer you could not fully check.
Doing, knowing and checking are different skills
Execution is carrying out a procedure, like writing a formula. Understanding is knowing what the quantities represent. Judgment is deciding whether the calculation answers the question that actually matters.
Automation can take over execution without handing you the other two. Think of a GPS: it gives you every turn, but knowing the map is what tells you the route is wrong. An assistant can explain and check its own work too, yet agreement between its answer and its explanation is not independent confirmation, and NIST’s generative AI profile lists confident false output as a real risk.
In plain terms: “knowing enough to check” sounds small, but it contains most of the understanding that matters.
Why the denominator matters more than the arithmetic
Here is a hypothetical with deliberately simple numbers. Among 10,000 transactions, 100 contain a real problem, and a detector catches 90 of them. It also wrongly flags 5% of the other 9,900, which adds 495 false alarms.
Catching 90% of problems and being right about 90% of alerts are different questions, and only the second one tells you whether to investigate.
So 90 of the 585 flagged transactions, about 15.4%, are real. A detector that catches 90% of problems is right on fewer than one alert in six.
An AI can do every step of that arithmetic. What it cannot do for you is notice that the decision in front of you, whether to investigate an alert, depends on a different denominator. Asking for that denominator is mathematical competence even when nobody calculates by hand.
What the learning studies actually show
The PNAS study by Hamsa Bastani and colleagues, published in June 2025, compared ordinary resources with two GPT-4 tutors. The generic tutor improved practice and hurt independent work afterward; a tutor built with teacher-informed safeguards largely avoided that loss. It did not show a positive independent-learning effect, and it covers particular students, lessons and tests.
A January 2026 coding experiment by Anthropic researchers Judy Hanwen Shen and Alex Tamkin asked a similar question. Among 52 developers learning an unfamiliar Python library, the AI group averaged 50% on an immediate quiz against 67% for the comparison group, and the time saved was not statistically significant.
The number to remember: 17 percentage points. That was the quiz gap between developers who learned a new library with AI help and those who learned without it, in one small, short-term study.
Both experiments measured the same thing from different sides: a finished artifact can hide an unfinished learner.
Neither study proves permanent skill loss. A 2025 survey of 319 knowledge workers adds a related signal: more confidence in generative AI went with less reported critical-thinking effort, while confidence in their own ability went with more, which is an association, not a cause.
The constructive evidence is older. Karpicke and Blunt showed in 2011 that retrieving and reconstructing material improved later conceptual learning, and closing the assistant to explain an idea yourself is a reasonable application, though nobody has tested that exact workflow. The measure that matters is what you can explain, diagnose and adapt after the answer leaves the screen.
How much math and code is enough?
Someone building a family event page and someone designing a medical device carry different consequences, so one prescription for both makes no sense. The ladder below is a planning framework, not a validated standard, and the joint ACM, IEEE Computer Society and AAAI CS2023 report shows the depth a professional path keeps: mathematics and statistics beside security, software engineering and human-computer interaction.
The ladder is a proposal, not a syllabus: depth follows the consequences you intend to own, and the next step stays open.
For everyday use, start with quantitative meaning: what is the denominator, which period does the number cover, and what happens if an assumption changes? For building software, reading beats memorizing. Trace a small program, say what data it stores, and spot the action that changes something outside it.
Get Micheal Lanham’s stories in your inbox
Join Medium for free to get updates from this writer.
Grow in expanding circles. Start with a real project, learn the concepts needed to explain it, meet a controlled failure, then add depth as its responsibilities grow.
The strongest case for learning less, and its limits
The best argument for teaching less syntax is opportunity cost. Months spent on a procedure an assistant finishes in minutes could go to system behavior, problem framing or a field you care about. It gets stronger still where AI opens a door that cost, disability or a steep entry barrier used to close.
Job claims need a separate step. The ILO’s May 2025 estimate puts one in four workers in occupations with some generative AI exposure, a task measure that stresses transformation, not a timetable for unemployment.
Every card is real evidence, and every salmon half is the claim people keep attaching to it anyway.
Productivity evidence moves with context too. METR’s July 2025 experiment had 16 experienced open-source developers complete 246 tasks in familiar repositories, and they took 19% longer with the allowed AI tools. Its February 2026 update saw signs of improvement but called the newer numbers an unreliable estimate of the current effect, so neither a permanent slowdown nor a settled speedup is honest.
Where knowing enough to check pays off
The payoff is not a curriculum. It shows up in ordinary moments where an answer looks finished.
- Reviewing an AI spreadsheet. Check units, the period each number covers and the denominator before you check the formulas.
- Reading an alert or a metric. Ask what share of flags are real, not only what share of problems get caught.
- Building a personal project. Before running a generated feature, name an input that might break it, such as an empty file, an impossible date or a duplicate entry.
- Setting a knowledge budget. For each project, pick a few concepts to learn deeply, a few procedures to delegate and a few risks that need outside expertise, and revisit it when you publish, collect personal data or charge money.
- Running a team. Keep the tasks where beginners meet a whole problem, make a decision and see the consequence, or automation quietly removes the apprenticeship.
Where I land: understanding is how you stay in the loop
My premise is that AI may eventually handle most routine calculation and code better than most people. If that happens, asking everyone to practice every old procedure would be hard to defend.
I am not asking anyone to compete with AI at remembering syntax. I am asking what we need to understand so that using it makes us more capable, including when the first answer is wrong.
Two things hold at once for me. AI may change a great deal of what people need to execute themselves, and learning still pays beyond the price of anyone’s labor, because math can illuminate a political claim or a game mechanic, and code can show you why a system behaves the way it does.
Three things to carry into Part 6, where a weekend of AI-built games raises a different question: what does building this fast make possible?
Sources and further reading
- NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600–1” (July 26, 2024): https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
- Bastani et al., “Generative AI without guardrails can harm learning: Evidence from high school mathematics,” PNAS (June 25, 2025): https://pmc.ncbi.nlm.nih.gov/articles/PMC12232635/
- Anthropic, Judy Hanwen Shen and Alex Tamkin, “How AI assistance impacts the formation of coding skills” (January 29, 2026): https://www.anthropic.com/research/AI-assistance-coding-skills
- Lee et al., “The Impact of Generative AI on Critical Thinking,” CHI (April 2025): https://www.microsoft.com/en-us/research/publication/the-impact-of-generative-ai-on-critical-thinking-self-reported-reductions-in-cognitive-effort-and-confidence-effects-from-a-survey-of-knowledge-workers/
- Karpicke and Blunt, “Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping,” Science (February 11, 2011): https://learninglab.psych.purdue.edu/downloads/2011/2011_Karpicke_Blunt_Science.pdf
- ACM, IEEE Computer Society and AAAI, “Computer Science Curricula 2023” (January 2024): https://ieeecs-media.computer.org/media/education/reports/CS2023.pdf
- ILO, “Generative AI and jobs: A 2025 update” (May 20, 2025): https://www.ilo.org/publications/generative-ai-and-jobs-2025-update
- Becker, Rush, Barnes and Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” (July 2025): https://arxiv.org/abs/2507.09089
- METR, “We are Changing our Developer Productivity Experiment Design” (February 24, 2026): https://metr.org/blog/2026-02-24-uplift-update/
This is Part 5 of The Human Side of the Loop:
- Part 1: Seven AI Predictions That Stop Making Sense When You Inspect Them
- Part 2: What Would I Actually Miss If AI Went Away?
- Part 3: AI Is Helping Build AI. Which Part of the Loop Is Actually Closed?
- Part 4: We Need an Early Warning About AI, Not an Early Disaster
- Part 5: When AI Can Write the Code, What Does a Beginner Still Need to Know?
- Part 6: I Recreated Three Favorite Games in a Weekend. What Does That Make Possible?
- Part 7: AI Makes Me a Better Builder. Why Would That Make Me a Fraud?
- Part 8: Would You Accept an AI Manager if the Owner Still Held the Power?
- Part 9: I Wanted to Spend $500 Watching AI Make a Documentary. Was That the Hobby?
Check out my best-selling book AI Agents In Action, 2nd Edition by downloading your copy here. Sponsored by Anaconda Inc.
