The Model Proposes. Code Decides.

Seven lessons from building and researching customer-facing AI agents that answer with screens

Lev Kaplun

12 min read

·

10 hours ago

--

Listen

Share

Much of what gets written about AI agents is about agents that work for the person using them: coding agents, research agents, office assistants. A customer-facing agent has a harder job. The person on the other side is a stranger, the conversation touches real records and real money, and no colleague is reading along to catch a mistake before it reaches them.

We call the version we build an agentic application: a customer-facing agent that answers with screens. After more than a year of building an agent that builds them, these are the key things we’d tell anyone about to put an AI agent in front of their customers, or to buy one.

The diagram below is the architecture we think is right. It is our own synthesis of published guidance, research and what we have seen while building. It isn’t a standard. No published study tests this whole architecture, and most of the individual choices in it have no controlled test on real conversations either. Most of the performance numbers we found come from the company selling the product, so we mark those where they appear.

Press enter or click to view image in full size

Read it from the top. A customer’s message enters through a channel, passes identity and policy checks, and reaches the model. The model sits inside a harness, the code that runs it and enforces the limits around it. From there the model can read (bottom right) or propose an action (bottom left). The left column is where people step in, and the right column is how you measure and improve. At the bottom are your systems of record, which hold the truth.

1. The model proposes, code decides

A language model is very good at understanding people and writing back, and it can be talked into things. The UK’s National Cyber Security Centre (NCSC) explains why: a model has no inherent distinction between data and instruction. For a customer-facing agent, that means what a customer writes, or what sits on a web page it reads, can change what it does.

The pressure doesn’t have to be technical. Reinforce Labs, a vendor that sells testing by attack (red-teaming), attacked 14 live support bots and checked the results against each company’s own published policy. By its count, 28.5% of its 471 attack conversations ended in a policy violation, and unauthorized commitments such as refunds and price matches appeared at 11 of the 14 bots. It says what worked was social, such as emotional pressure and invented urgency, though it gives no success rate per tactic, and its advice is to prepare for an ordinary annoyed customer and not a hacker. The companies aren’t named and nobody has checked the numbers independently.

So the model shouldn’t be the gatekeeper. It proposes, and deterministic code, which gives the same result every time, decides. Identity, permissions, money and what the agent is allowed to do are enforced in code, against your records. OWASP, a non-profit security foundation, tells teams to test for exactly this: unauthorized tools, the connections an agent uses to read or change something in your systems, are denied even when the model requests them confidently. In a test by Arize, a vendor that sells monitoring tools, a simulated support agent reading order pages planted with instructions pulled another customer’s record in 28 of 100 runs. Once access was bound in code to the logged-in session, the agent still tried in 20 of 100 runs, and none returned data. That was one model, with no repeat runs.

We’d put the check at the moment the action runs, not at an earlier step. One study, on a simulated bank-support benchmark, ran the agent through a fixed workflow and found unauthorized writes no lower than without it, at 4.4% of tasks against 4.3%. The authors think the model can misjudge which step it is on, though they didn’t measure that.

Keep the record of the conversation on your server too, not in the visitor’s browser, where it can be faked. Microsoft’s agent framework documentation warns that a malicious client can inject messages to manipulate the conversation history, and recommends persisting it server-side.

People have a place here too. For high-impact actions, the customer says yes to the exact action (user confirmation), and a person from the business signs off when policy requires it (human approval).

Customers can also arrive through their own AI assistant. In Gartner’s survey they used third-party AI tools about three times as often as a company’s chatbot. The same rule applies: verify who the assistant is, then authorize what it may do.

2. The model knows nothing about your business

A model arrives with general knowledge from its training. It doesn’t know your policies, your catalog or this customer’s account. All it knows about your business is what gets brought to it at the moment of the question, and how you bring it is one of the most important decisions you’ll make.

You will hear that some new technology has replaced RAG, the common approach of finding passages in your documents and adding them to the prompt. We found no controlled test showing that. Each kind of question suits a different method:

• Live facts, such as an account or an order, come straight from your systems through tool calls, limited to this customer. The case for reading them live and not from a searchable copy rests on freshness, accuracy, permissions and one fewer copy of personal data. It is a strong argument and not a measurement: we found no controlled test comparing the two on customer records.
• Business numbers, such as what counts as revenue, come from written definitions (a semantic layer) and sometimes from code that computes them. With code computing, tests by dbt and AtScale, both vendors, show 8 to 31 more points of accuracy on the kind of questions a data analyst asks, not on customer conversations, and AtScale compares its best of three runs with a single control run. Whether the code is needed is still argued: AtScale says the math must run in code, and MotherDuck, also a vendor, scored plain-text definitions above a semantic layer in its own rough test. According to dbt, a model writing its own query fails with a plausible wrong number, and its semantic layer fails with an error message. In front of a customer, we’d take the error message.
• Written knowledge comes from classic RAG, a single search of your documents. It holds up on simple questions: an independent benchmark found basic RAG comparable to or better than systems built on a map of how facts connect, though the result depended on the model.
• Complex questions often suit agentic retrieval, also called agentic RAG or agentic search, where the agent searches, reads and decides whether to search again. Vendors selling it report large gains on complex questions and small ones on simple questions (AWS: under 5 points on one-step questions). It also costs more and takes longer: Microsoft measured 2.6 to 7.8 times the tokens, the units of text models bill by, and IBM, also a vendor, reports 25 seconds against 3 on its customer-support test. A peer-reviewed study found neither a tuned fixed pipeline nor the agentic version universally better.
• What a customer said earlier comes from long-term memory, for identified customers only. Memory has a write path, so treat each write as an action that code decides. Varonis, a security vendor, showed text hidden on a web page being written into Copilot Personal’s persistent memory. Microsoft has patched it, with no sign of exploitation.

The order we would invest in is our own reading of all this: first the content, meaning clear business definitions and policy documents without contradictions. Then the procedures and tools that apply them. Only then a better search engine.

3. One agent with written procedures

Our default for a customer conversation is one agent that holds the whole conversation and follows written procedures. We expect fewer points of failure and an easier time understanding what happened, and the customer hears one voice. The procedures are text the business can read and change, with versions, so the agent’s behavior isn’t buried in code.

Vendors don’t agree on what sits behind that voice. Fin and Decagon each described one agent in their launches this fall, and Anthropic’s guidance, drawing on its own deployments, says a single agent with instruction packs it loads when needed (skills) often beat separate helper agents (subagents). Salesforce, Sierra and Zendesk describe several agents working together, and Microsoft, in preview, builds its customer conversation product as multi-agent. All of these are vendors describing their own approach. A study across tasks found that, depending on the task, adding agents ranged from 80.8% better to 70.0% worse, with no customer-service test. So this is a default and not a settled question. A team of agents makes sense when there’s a good reason, and scripted steps make sense where the order must be exact.

4. Agents fail silently

An agent can fail with a confident, well-formed answer. There is no error message, just a wrong answer that looks like a right one. The layers underneath can do the same: Cloudflare, reporting on its own service, counted 155 responses containing other customers’ data that went out marked as successful, while its pre-launch tests looked only at response codes and engine performance, not at what the answers said. Knowing that a system responded tells you little about whether it responded correctly.

That is why we think every customer-facing agent needs an automatic eval process, often called online evaluation when it runs on live traffic. An eval is a repeatable check of whether the agent did the job right. In plain terms:

1. Sample real conversations.
2. A language model judges them (an LLM judge), and code checks what the agent actually did.
3. Compare the result with what happened in your records.
4. Turn what failed into a proposed fix.
5. A person from the business approves the fix before it goes live (human in the loop), and it reaches everyone gradually when you have enough conversations to try it on a small share first.

Each step is there for a reason. In the Arize test above, a model judge reading only the final answer caught 5 of the 28 leaks; the 28 themselves were counted by a code check on the tool calls. Models judging models can also share the same mistakes: one independent study found a strong overlap, though its authors say disagreement among human raters may explain part of it, and another found it varies by test. So have people check the judges on your own conversations. Testing against AI-simulated customers isn’t enough either: a study with 451 real people found simulators too cooperative, which inflates success. And keep your test set away from whatever builds or improves the agent: researchers at Sierra, a vendor, and Princeton warn that self-authored tests “encode their own blind spots”.

The last step is the one we hold to on conviction. Several vendors now sell an agent that improves the agent, the few results published are their own, and we found nothing independent. We still think the business should approve what changes, because it is the business’s name on the answer. And an approval only means something if someone actually reads it.

5. Guardrails that guess make mistakes in both directions

Guardrails are the checks around a model, and they come in two kinds. A deterministic guardrail is a rule in code: the same input always gives the same result. A model-based guardrail, often called probabilistic, relies on a model’s judgment, often a score that code compares with a threshold.

The second kind is a guess, so it can be wrong both ways: it blocks things that are fine and lets through things that aren’t. NCSC warns against relying on filters that block known bad content, because an attack can always be rephrased. They help, but you can’t rely on them to stop everything.

This matters more since this fall, when decision models arrived: small models such as TypeSafe’s Jev and OpenAI’s Decisions API that return probabilities over fixed answers and write no text. They are fast and cheap, which makes it tempting to put one at every gate, and they are still guesses. We found no threshold validated beyond the data it was set on for checking whether an answer is backed by its source, researchers at Carnegie Mellon conclude that a threshold must be set on the target workload, and research on language models deciding when to escalate finds they misjudge how likely they are to be right. So each team has to test its thresholds on its own labeled data.

Our rule: anything that must never happen is stopped by an exact rule in code, and that rule acts only on what is never legitimate, so it leaves real customers alone. A model’s score is an extra layer. We’d use it to raise a flag, ask a clarifying question and, if needed, send the conversation to a person, where a wrong guess costs little.

6. A conversation that stayed with the agent wasn’t necessarily resolved

The number people like to show is containment: the share of conversations that stay with the agent and never reach a person. It doesn’t tell you whether the problem was solved. Many vendors now headline a resolution rate instead, but the definitions vary, and at least one major vendor counts a customer who leaves without asking for more help as a resolution.

The freshest independent numbers we found come from customers. In J.D. Power’s 2026 wireless study, of 57,339 customers, AI assistants resolved the request in 48% of phone support cases. Satisfaction was 20 points lower on a 1,000-point scale when an assistant was involved, and rose 83 points when it resolved the request. What customers respond to is whether the problem got solved. Nubank, reporting on five of its own production agents, found satisfaction within about 1 to 10 points of expert human agents for four of them and 23.6 points below for debt management. The gap depends on the job.

So measure outcomes from your own records: was the problem solved, and did the customer come back about it? We found no audited measurement across vendors, so your own numbers are the ones to trust.

And keep a person within reach, with the full context of the conversation, and test that path like any other feature. Gartner’s analyst warns against chatbots becoming containment traps, and in its survey of 3,566 customers only 27% would try a chatbot again after a bad experience. In a survey by Five9, a vendor, trust that AI would solve the problem rose from 26% to 55% when a person was reachable. The law is moving on both disclosure and access: EU rules have required since August 2026 that chat systems tell people they are talking to AI, unless it’s obvious; a new California law gives customers of large businesses a way to reach a person from 2027; and New York City has proposed a rule of its own.

7. Text isn’t the right answer in most cases

We think chat is a good way to ask and a poor way to compare, choose, fill in details or confirm an action. A customer doing any of those doesn’t need a wall of text. They need a screen that fits their question, built at the moment they ask it. That is generative UI.

The evidence so far points this way without settling it. In a study at ACL 2026, 428 annotators preferred generated interfaces to chat by 84% to 12% against one model and 69% to 30% against another, and 76 real users on their own questions split 50.8% to 41.1%. A study from Google researchers found them overwhelmingly preferred over standard markdown answers. Those measure preference and not outcomes, on general questions and freely generated pages. We found no test with a business’s real customers, so “most cases” is our conviction from building and not a measured result.

One design point is worth knowing. The Google team behind one such format and the CEO of CopilotKit, a vendor, describe the most popular approach, the declarative one: the model assembles the page from a catalog of approved screen parts and fills them with data, and anything outside the catalog is caught before it reaches the screen. They note the cost: it is safe for the brand but limits deep customization. We’d take that trade. It is the first lesson again: the model proposes, and code decides what appears.

Check it against what you have

Save the diagram and hold it up against what you’ve built or what you’re being offered. Seven questions will get you most of the way:

1. On identity, money and what the agent may do, who decides: the model or code?
2. Where does each kind of answer come from, and what happens when the data is missing?
3. Can the business read and change how the agent behaves?
4. How would you find out about a wrong answer nobody complained about, and who approves the fix?
5. What does each guardrail do when it is wrong?
6. Beyond containment, how do you know the problem was solved, and how does a customer reach a person?
7. What does the customer see when text isn’t enough?

We build agentic applications at BranderUX, so we aren’t neutral on any of this, and we’d love ideas for improvements and refinements.