Polished Is Not the Same as Usable: What AI UI Generators Still Get Wrong About UX
An interface can look polished and still fail its users. Here is how to test the difference.
In 2026, researchers published an evaluation of 138 mobile screens generated by three AI design tools (Figma, Banani and Stitch) and scored each one against Nielsen’s ten usability heuristics. Their headline finding was a gap between how the screens looked and how well they supported actual use. The tools reproduced the visual conventions of modern mobile apps reliably. They were substantially weaker on what the authors call support-oriented heuristics, and the paper observes that generated interfaces often assume an idealized, uninterrupted user journey.
That matters for a reason that has little to do with whether AI “can design.” A generated screen that looks finished invites a finished-screen review: is the spacing clean, does the palette match the brand, does it resemble a real product? Those are questions a generator is well equipped to answer in advance. The questions that decide whether people can use the product are different, and the polish makes them easier to skip.
This article covers what the current evidence says generators get wrong, why the usual ways of reviewing their output miss it, and a five-part test for telling a polished interface from a usable one.
What the evidence shows
Several studies with different methods point at the same weak spots. None is large, and the tools change quickly, so read what follows as a pattern worth checking in your own work, not a verdict on any product.
Convention stands in for intent. The authors of the ACM study describe their finding as usability by convention rather than intent: the tools have learned what good mobile interfaces look like and reproduce it. Nielsen Norman Group (NN/g) saw the same tendency in a separate evaluation. Outputs leaned toward mainstream conventions, often built on shadcn components and Tailwind, and when the prompt gave no detailed visual direction, the screens came out generic and hard to tell apart.
Convention is not a flaw in itself. NN/g notes that people find interfaces easier to use when they follow external standards. The problem starts when the convention is wrong for the context, and NN/g documented a clear case. Its brief asked for a “profile page” for people enrolled in NN/g’s online training: a private page showing courses, exam status and certification progress. Given a broad prompt, several tools read “profile” as a social-media profile and built the large identity block familiar from networking sites, which pushed the task content lower on the page. NN/g judged this a minor violation rather than a severe one, but it put visual attention on secondary information. The tool had matched a word to a pattern without asking what the page was for.
The main path gets built; deviations get less attention. The ACM paper reports substantial weaknesses in support-oriented heuristics and notes that generated interfaces offer limited assistance when users stray from the primary task flow. The summaries accessible for this article do not list exactly which heuristics fall in that group, so the full paper is worth reading before relying on the specifics.
Hierarchy and grouping slip even with a good brief. NN/g found that a detailed prompt brought the outputs close to the human-designed version. A closer look still revealed weak hierarchy and grouping among related elements, overused color creating visual tension, poor contrast, and inconsistent margins. In one example, Bolt’s output separated a course-materials password from the link it unlocks with a large gap, although the two belong together. NN/g draws a further conclusion: writing a brief that specific, or attaching a mockup, means much of the design work has already happened before the AI is involved.
Accessible behavior is where failures are easiest to measure. AudioEye asked five AI tools (OpenAI, Anthropic, Google, xAI and Lovable) to build three sites each (a news site, an online store and a financial services site) to WCAG 2.2 AA. It then scanned and tested the 15 results, including keyboard focus, dialog handling, form error announcements and screen reader output. According to AudioEye’s account, all 15 sites failed WCAG Level A, the most basic tier, averaging 55 issues per page, and 91% of the issues were medium or high severity.
Two cautions apply. AudioEye sells accessibility products, and that account is a sponsored article written by its CEO, so the numbers should be checked against the full study before they are repeated. And the comparison point matters: the February 2026 WebAIM Million analysis found detectable WCAG 2 failures on 95.9% of the top million home pages, and AudioEye’s own index puts the typical site at 62 issues per page. The human-built web is in poor shape too. The two figures come from different methods, so the comparison is rough, but the defensible conclusion is not that AI builds less accessible pages than people do. It is that asking for accessibility in the brief did not produce accessible results, while AudioEye’s companion survey found 81% of teams using AI tools confident that their AI-generated code met accessibility guidelines.
Why standard reviews miss it
Reviews usually look at screenshots. The ACM authors list this as a limitation of their own work: expert inspection of static screens cannot capture dynamic behavior, real task performance or longitudinal experience, and they call for user studies with interactive prototypes. A preprint from Brazilian researchers shows the consequence. Ninety-two undergraduates, almost all in computer science, rated ten static prototypes for the same product scenario, five from AI tools and five made by one researcher, using the short User Experience Questionnaire. Pragmatic ratings (usability, efficiency) were positive across the board, hedonic ratings (originality, excitement) were neutral to negative, and authorship was not a determining factor. AI and human prototypes both appeared among the best and the worst, and the top scorer was AI-generated, with a human-made prototype statistically indistinguishable from it. One reasonable reading is that on a static view AI screens can look as usable as human ones, which is exactly why a static view cannot settle the question.
Preference is not performance. The ACM authors note that usability in this literature is often framed as user preference rather than measured performance. Google Research’s generative UI work illustrates the distinction. Google reports that human raters strongly preferred its own generated interfaces to standard LLM text and markdown output, with human-expert-made sites preferred most. Preference is a legitimate measure. It answers a different question from whether someone can finish a task.
Fidelity inflates credibility. NN/g warns that showing polished AI prototypes to stakeholders without framing can derail the conversation, for instance into a debate about button color when the visual hierarchy is nowhere near done. A high-fidelity surface signals a finished decision even when none has been made.
A five-part test
The protocol below is a proposal assembled from the failure patterns above. It has not been validated in a study, and none of the sources prescribe it. Each step is tied to the evidence it responds to.
1. Name the task, then look at the screen. Write one sentence stating the primary job this screen does for this person. Then look at what carries the most visual weight. If an identity block, a hero image or a decorative statistic dominates and the task sits lower, the hierarchy is serving a template. This is the profile-page failure in miniature.
Get Tahsin Hani’s stories in your inbox
Join Medium for free to get updates from this writer.
2. Break the flow on purpose. Walk each flow with invalid input, an empty state, a slow or failed load, a cancel halfway through, a back step and an undo. Check whether those states exist and whether they tell the person what happened and what to do next. In Nielsen’s terms, these touch user control and freedom, error prevention, and helping users recognize, diagnose and recover from errors. If the generator gave you only the main path, those states are unresolved. They have not stopped being requirements.
3. Audit the vocabulary. List the nouns in your prompt that can mean several patterns: profile, dashboard, account, feed, checkout. For each, see which pattern the tool chose and whether it is the one you meant. Then tighten the prompt and regenerate to see whether the output changes. If it changes a lot, the first result was the tool’s assumption, not your design.
4. Use it without a mouse and without sight. Tab through every interactive element. Is focus visible, is the order logical, can you open and leave every dialog and menu? Run a screen reader through the main task and listen for whether form errors are announced. Measure contrast on the rendered colors rather than trusting the palette in the prompt: NN/g’s detailed prompt asked for WCAG AA colors, and poor contrast was still among the problems it listed for the outputs. Automated scanners help, but AudioEye’s materials argue that behaviors such as focus handling and screen reader output need expert manual testing, and that matches how WCAG failures show up in practice.
5. Watch someone use it. Give a few people from the real audience a concrete task with no help and note where they hesitate or give up. Heuristic review is cheap and catches a lot, but it is an expert’s inspection, and the ACM authors themselves name user studies as the next step.
What this evidence cannot tell us
The studies are small and varied: 138 screens rated by three evaluators, ten static prototypes rated by 92 students, two scenarios from a single organization tested at free-account level, and 15 websites. Several rely on static screens. The tools update often. NN/g’s May 2025 status update reported that no generative AI tool effectively supported design systems, and that may no longer be true. Neither the AudioEye test nor, as far as the published summaries show, the ACM study compared outputs against human-built equivalents for the same briefs, so the evidence says little about whether a human working from the same brief would have done better. The Brazilian preprint cuts the other way on one point: it found no authorship penalty on perceived usability, which suggests some of these weaknesses may be shared with human work rather than unique to generators. And the vendor-sourced accessibility figures need confirmation against the underlying study.
What the evidence does support is narrower and still useful. Generated screens are fast to produce, which suggests review becomes the scarce step; that is an interpretation, not a finding. Spending that review on spacing and color is spending it where the tools look strongest. The studies above point to the problems sitting elsewhere: in the task hierarchy, the unhappy paths and the keyboard and screen reader experience.
Sources
- Looks Good, But Is It Usable? Evaluating Usability in AI-Generated Mobile User Interfaces (ACM Digital Library, 2026): https://dl.acm.org/doi/10.1145/3772363.3799002
- Wang and Brown, Good from Afar, But Far from Good: AI Prototyping in Real Design Contexts (NN/g, Oct. 24, 2025; last reviewed Aug. 19, 2026): https://www.nngroup.com/articles/ai-prototyping/
- Wang, Brown and Zhang, Testing AI with Real Design Scenarios: Evaluation Methodology and Prompts (NN/g, Oct. 24, 2025): https://www.nngroup.com/articles/testing-ai-methodology/
- Brown, Sponheim and Dykes, AI Design Tools Are Marginally Better: Status Update (NN/g, May 9, 2025): https://www.nngroup.com/articles/ai-design-tools-update-2/
- Romero et al., Usable but Conventional: An Empirical Study on the UX of AI-Generated Interface Prototypes (arXiv preprint, May 14, 2026): https://arxiv.org/abs/2605.15124
- Georgevich (AudioEye), We asked five AI tools to build accessible websites. All 15 sites failed. (VentureBeat, sponsored content, Sept. 30, 2026): https://venturebeat.com/technology/we-asked-five-ai-tools-to-build-accessible-websites-all-15-sites-failed
- AudioEye study press release (MarketScreener): https://www.marketscreener.com/news/audioeye-study-finds-ai-coding-tools-do-not-write-accessible-code-ce785ad2d18bf520
- The WebAIM Million: https://webaim.org/projects/million/
- Generative UI: A rich, custom, visual interactive user experience for any prompt (Google Research): https://research.google/blog/generative-ui-a-rich-custom-visual-interactive-user-experience-for-any-prompt/
- 10 Usability Heuristics for User Interface Design (NN/g): https://www.nngroup.com/articles/ten-usability-heuristics/
