Automating Unity Smoke Tests:
The Stack We Chose, and the Work We Refused to Automate
We started with a question that seemed pretty straightforward: Which test automation framework should we use for our Unity projects?
A few weeks into the research, I stopped thinking that was the useful question.
What we really needed to decide was: Which problems should be automated at which layer — and which ones are still better left to a human tester?
That changed the direction of the whole project. It affected the tools we chose, what we decided not to automate, and how we are going to judge whether any of this was worth doing.
One important caveat upfront: the rollout has not started yet.
The research stage is done. We have confirmed that the approach is feasible and decided which layer should own what. Next comes the less glamorous part: measuring the baseline and setting up the first test environments.
So this is not a victory lap. It is a record of the decisions we made before implementation — including a few things I already expect us to get wrong.
The problem we actually wanted to solve
We run several Unity projects in parallel.
Every time a new build appears, QA ends up answering the same basic question:
Is this build healthy enough to spend real testing time on?
Launch it. Get through startup. Check permissions. Open the main screens. Load an existing save. Enter gameplay. Put the app in the background and bring it back. Trigger an ad or purchase flow. Make sure nothing obviously falls apart.
None of that is especially expensive once.
The problem is doing it again and again across builds, projects, platforms and devices.
That was the part we wanted to remove.
The goal was never to reduce QA headcount. I do not want skilled testers spending their time acting as build-health monitors when they could be testing new features, exploring edge cases, looking at gameplay behavior or doing the kind of work that actually benefits from judgment.
That gave us a much clearer scope:
automate build acceptance, not QA.
That one boundary ended up shaping almost every decision that followed.
Unity makes normal mobile automation a little awkward
If you come from web or conventional mobile development, tools like Appium or Maestro are an obvious place to start.
They work well when an app exposes a useful native accessibility or UI hierarchy.
A conventional Unity game usually does not.
The buttons, panels, level objects and gameplay state you care about live inside Unity. An external mobile automation framework does not automatically know what those GameObjects are.
So something like “find the Continue button by accessibility ID” may work perfectly on a native permission dialog and tell you almost nothing about the actual game screen sitting behind it.
Unity 6 now has an Accessibility API, so it is possible to construct an accessibility hierarchy deliberately. That matters for accessibility, and in theory it could also become part of a custom automation strategy.
But you still have to build and maintain that hierarchy.
For this project, we did not want to start by building another test driver just so we could begin testing.
That pushed us toward a layered approach instead.
The four approaches we looked at
We initially had a fairly long list of tools. Comparing them one by one was not especially useful because many of them solve completely different problems.
Grouping them by what they can actually observe made the landscape much clearer.
That classification also killed the idea that we needed to find one framework that could somehow do everything.
A Unity Play Mode test and a system-level mobile automation test are not competing solutions. They are looking at different parts of the application.
There was also a useful lesson in the research process itself.
Several comparison articles we found were written by vendors whose own products appeared in the comparison. That does not automatically make the information bad, but we treated those pages as vendor material rather than neutral market research and relied more heavily on official documentation and hands-on reports.
The stack we ended up choosing
Our current plan looks like this:
Unity Test Framework sits at the bottom for a simple reason: if we can test something without launching the whole game, navigating through five screens and tapping buttons, we probably should.
UI tests are slower and more fragile. They should be used for things that genuinely need UI.
For Unity UI and game state, we chose AltTester.
The architecture fits what we need: the Unity application is instrumented, which lets the test driver interact with the Unity object hierarchy instead of trying to guess what is happening from pixels. AltTester describes that model in its documentation.
Maestro handles the other side of the boundary: the things that sit outside Unity.
So there is very little reason for the two frameworks to duplicate each other’s work.
If we need to verify that the correct level loaded, that is a Unity-side problem.
If we need to verify that an OS permission dialog appeared and that the app recovered correctly afterwards, that belongs on the device side.
Once we started thinking about ownership this way, the architecture became much easier to reason about.
iOS is still the awkward part
One constraint had more impact on the plan than I expected.
As of September 2026, Maestro’s official documentation says local iOS execution supports Simulators, while physical iOS devices are still not supported.
Maestro Cloud has the same practical limitation on iOS: Cloud runs iOS tests on Simulators and expects a Simulator-targeted `.app`, not a physical-device build.
So the iOS side of this setup is not symmetrical with Android.
Simulators are still useful. They can cover plenty of OS-level behavior. But they are not a substitute for a real iPhone. Apple makes the same point in its own documentation: Simulator does not reproduce the performance or every feature of a physical device.
Performance and hardware-dependent behavior still need real hardware, and some third-party SDK integrations may as well.
There is another practical check we want to do early: simulator architecture support across the mediation stack.
An ad mediation setup can include a surprisingly large collection of SDKs and adapters, and simulator support can vary between versions. Google’s Unity Ads mediation adapter changelog, for example, explicitly notes the version where arm64 Simulator support was added.
So one of the preflight questions for our automation build is very simple:
Does every SDK in this exact mediation stack include the simulator architecture we need?
If not, the build may fail to link, or we may end up excluding or stubbing something. And if an SDK is stubbed, a green simulator test obviously does not prove that the real integration works.
I would rather make that gap explicit than pretend the coverage is better than it is.
In the setup we are planning, real-device Unity coverage will come through the instrumented test layer. Anything we still cannot validate reliably through the automated iOS path remains a small, explicit manual check.
I would take a known five-minute manual check over a green automated test that quietly gives us false confidence.
The decisions that may matter more than the frameworks
This was probably the most useful part of the research.
Choosing AltTester or Maestro matters, of course.
But I suspect the rules around the automation will have a much bigger impact on whether people still trust it six months later.
Start with the boring crash gate
Before the pipeline starts tapping buttons, it should already know whether the app produced a fatal error.
So the first layer is intentionally uninteresting: read the logs, detect crashes and fatal conditions, and stop early.
In the current design, I expect this to be a script rather than a framework — something on the order of fifty lines.
That is exactly why it goes first.
There is not much value in spending ten minutes executing UI scenarios against a build that is already fundamentally broken.
Give tests a way to enter known states
The second thing may end up being more valuable than another automation framework: a test-only way to jump directly into a known application state.
A smoke test should not have to replay onboarding every time it wants to check something that happens later in the game.
For automation builds, we want controlled deep links or equivalent debug commands that can do things like:
open this screen, load this save, enter this level, initialize this account state.
Get Denis Chernitsyn’s stories in your inbox
Join Medium for free to get updates from this writer.
Those entry points obviously should not exist in production builds.
Inside an automation build, though, they can remove a huge amount of slow and fragile setup.
Put a hard limit on the smoke suite
Our initial plan has 14 smoke scenarios and a target runtime below 12 minutes per device.
The hard ceiling is 20 scenarios per platform.
If somebody wants to add scenario 21, the question is not simply: “Should we add this?”
The question becomes:
Which existing smoke scenario does it replace?
I like that rule because smoke suites have a natural tendency to grow into regression suites.
A few extra checks do not seem like a problem. Then there are a few more. Runtime gets longer. More dependencies creep in. Failures become harder to diagnose.
Eventually people stop trusting the gate and start running the old manual checklist anyway.
At that point the automation has not removed work. It has created another system that needs to be maintained on top of the old one.
A flaky smoke test is a broken smoke test
For a build gate, reliability is not a nice-to-have.
If a test fails intermittently, people learn very quickly to rerun it.
Then rerunning becomes normal.
Then a red test stops meaning “the build may be broken” and starts meaning “try again.”
That destroys most of the value of the gate.
Our planned rule is to quarantine intermittent tests immediately, then either fix or remove them. “Rerun until green” cannot become part of the workflow.
Coverage nobody trusts is not useful coverage.
Automation phones are infrastructure
An automation device is not somebody’s spare QA phone.
If scheduled tests depend on a device being charged, connected, on the right network, running the expected OS version and physically available, that device is infrastructure.
If somebody borrows it for manual testing, your automated test reliability is suddenly dependent on office logistics.
This sounds almost comically mundane next to a discussion about automation frameworks.
I would not be surprised if it causes more operational pain than the frameworks themselves.
Do not use the nicest phone as the reference device
Our first instinct for the primary reference device is not to grab the newest flagship.
On our projects, memory pressure and frame-rate issues tend to show up on low- and mid-range hardware much earlier than they do on a flagship.
That makes those devices more useful for this particular job.
The purpose of a smoke gate is not to demonstrate that the game runs beautifully under ideal conditions.
It is to catch unhealthy builds early.
What will actually be in the smoke suite
The suite is deliberately small.
The current plan covers cold launch, startup and permission flows, basic SDK screens, purchase-dialog reachability, rewarded ads, background and resume, deep links, onboarding completion, loading an existing save, save migration, entering and exiting gameplay, primary navigation, fatal-log checks and a basic performance threshold.
That should be enough to answer the question we actually care about:
Can QA safely start testing this build?
It is not enough to answer:
Is this release ready to ship?
Those are completely different questions.
Trying to make one suite answer both is how a smoke suite becomes an expensive regression suite.
The things we are deliberately leaving manual
Feature testing stays with QA.
Exploratory testing stays with QA.
Gameplay feel stays with QA.
Full regression is outside the scope of this project.
Maybe some of that changes later. Right now, expanding automation coverage for its own sake is not the goal.
The goal is to get repetitive build acceptance out of the critical path and give testers more time for work that benefits from curiosity, context and judgment.
For me, that boundary matters much more than the percentage of test cases we can technically automate.
Being able to automate something does not automatically make it worth automating.
How we will know if this actually helped
Before implementation starts, we will record one very basic baseline:
QA hours per week spent on build acceptance, per project.
Our primary target is a reduction of at least 40%.
I want that number recorded before the rollout because otherwise, three months from now, the discussion becomes subjective very quickly.
“QA feels faster now” is not a useful measurement.
We are publishing the target before we have the result, so the baseline needs to exist before we start changing the process.
We will also track false positives.
If the automation starts producing too much noise, the response should be to repair or shrink the suite, not keep adding more checks to it.
There is another outcome I care about just as much as the raw hours saved: what happens to the time we free up?
Ideally, more QA capacity goes into new features and exploratory testing.
We will also watch the number of feature defects QA finds, although I would treat that as a directional signal rather than a standalone KPI. The number naturally changes depending on how much work is shipping and how complex that work is.
And we are budgeting for maintenance from day one.
Our current planning assumption is roughly half a day per week per project once the system is running, plus the usual iOS signing and provisioning overhead.
That is not a measured result. It is simply the maintenance cost we are putting into the business case before deciding whether the automation actually saved anything.
What I expect to go wrong
I expect iOS provisioning to break the pipeline at some point.
I expect the second Unity project to expose assumptions we accidentally baked into the first one.
I expect QA throughput to get a little worse during adoption before it gets better.
And I would not be surprised if USB cables, hubs, signing certificates and device availability cause more interruptions than the automation frameworks themselves.
I am writing those predictions down now on purpose.
When we come back to this in a few months, I do not want to rewrite history and make every decision look obvious in hindsight.
This is the starting point
The rollout is the experiment.
The more interesting article will probably be the next one.
Once we have enough real usage, I want to come back with the actual numbers: build-acceptance hours before and after, maintenance time, false positives, scenarios we removed, assumptions that failed, and whether QA really gained more time for feature and exploratory testing.
Including the parts that did not work.
It is easy to write a tool-selection article before reality gets involved.
I would rather put the assumptions on record now and see how well they survive contact with the actual rollout.
If you already run automated tests against Unity games, I would be especially interested in one thing:
What started hurting only after the first few months?
Those were the details I found hardest to get from tooling comparison pages.
