Hybrid AI in Flutter: Routing Between On-Device and Cloud Models
A tour of the options — and how I do it with Genkit Dart
I used to say “the future belongs to local or, at the very least, hybrid agents.” I wrote that at the end of my FunctionGemma article back in December 2025. I don’t say “at the very least” anymore.
Hybrid IS the future. Not because on-device won, not because cloud lost — but because they solve different problems, and pretending one wins is naive. On-device wins on: latency, privacy, cost-per-token, offline availability. Cloud wins on: capability ceiling, model size, world knowledge. Stop arguing. Use both.
Why is this the future and not just the present? Because for years the on-device side wasn’t ready. Models that fit on a phone were toys. They couldn’t reason, couldn’t follow instructions, couldn’t handle anything beyond keyword classification. So “hybrid” meant “cloud with a tiny ML model for autocomplete.” Not a real architectural choice.
That changed in 2025. Gemma 3 1B, EmbeddingGemma 300M, FunctionGemma 270M, Qwen 0.5B — small models that genuinely work. They reason, follow tool schemas, embed text into useful vectors, generate coherent multi-turn responses. The on-device side caught up enough that you can route real work to it, not just “the easy stuff.”
And then Gemma 4 landed. E2B and E4B variants — Matryoshka-style nested models that let you pick your accuracy/speed tradeoff on the fly. Multi-Token Prediction via speculative decoding making generation 2–3x faster on the same hardware. Native multimodal — image and audio input in a model that fits on a mid-range phone. Suddenly the on-device side isn’t catching up anymore. It’s a real peer to the cloud for most use cases that don’t need world knowledge.
But “real peer” isn’t “replacement.” A phone is not a laptop is not a desktop is not a datacenter — and that hardware ladder is real: RAM, CPU/GPU/NPU, storage, and battery all climb as you move up it. A 1B model that flies on a flagship phone crawls on a three-year-old mid-ranger; a model that needs 8GB of RAM simply won’t load on most devices in the wild. So even with great small models, the device sets a ceiling — and when a request needs more reasoning, more context, or more world knowledge than that ceiling allows, the cloud is where it has to go. That’s not a failure of on-device AI; it’s the reason hybrid exists.
Once both sides are capable, the question stops being “cloud or local?” and becomes “which one, when?” That’s the hybrid question — and answering it well, in code, without a mess, is what this article is about.
What “hybrid” actually means
Hybrid AI is not just “local OR cloud” — it’s a routing policy. Every request gets evaluated, and the answer to “which model handles this” depends on the request itself.
The key shift in thinking: it’s not about where the model lives, it’s about how you decide per request. A tourist chat app — the kind we’ll build later in this article — illustrates this well. Same user, same app, but four different routing decisions in four different queries:
- “Tell me about the Eiffel Tower” → cloud handles this better, world knowledge wins. If offline, fall back to local with whatever it knows.
- “Plan me a 7-day Tokyo itinerary with restaurants, museums, and day trips” → cloud. Multi-step planning over millions of places, long structured output, real-world knowledge about opening hours and neighborhoods — this needs reasoning depth and breadth a small local model most probably can’t match.
- “Translate my passport number to Japanese” → never leaves the device. Doesn’t matter if you have 5G — PII routing is non-negotiable.
- “Find similar attractions to ones I liked” → local. Embedding the query and searching the user’s history never needs the cloud — it’s a vector lookup over private data.
Four queries, four routing decisions. That’s hybrid.
Behind each decision is a different signal: world-knowledge capability, data sensitivity, latency budget, cost. The routing policy is just code that reads those signals and picks a model. There’s nothing magical about it — except that without the right tooling, you end up writing it twice. Once for the cloud path, once for the on-device path, with two different SDKs that don’t speak the same language.
That’s the real problem hybrid AI has had in Flutter until recently. Not the inference. The plumbing.
The ways to build hybrid today
Hybrid AI isn’t a single technology — it’s a problem space, and over the last two years a handful of frameworks have grown up around it. Each one tackles the same question (how do you route between cloud and on-device models without writing the layer yourself?) and lands on a different answer. Before committing to one, it’s worth knowing what’s actually on the table today.
1. Roll your own
The default starting point. Pick an on-device runtime (onnx, llama.cpp, MLC, LiteRT-LM), pick a cloud SDK, and write the routing logic by hand in your app code.
It works. It also means: two completely different APIs, two `Message` types that don’t talk to each other, two tool-definition formats, two streaming protocols, two observability paths. Everything in your codebase that touches AI ends up duplicated — once for the cloud branch, once for the device branch. You become a translator between two ecosystems.
2. Firebase AI Logic
First, what it actually is: Firebase AI Logic is Google’s client SDK for calling Gemini straight from your app — text, images, multimodal, structured output, all with Firebase console config and App Check security. Hybrid isn’t its main job; it’s an experimental add-on layered on top, letting the SDK fall back to (or prefer) an on-device model instead of always hitting the cloud. That hybrid mode has been rolling out platform by platform — Web first (June 2025), then Android (April 2026), then iOS (May 2026) — each still experimental. So judge it as “a mature cloud SDK with early, evolving hybrid,” not as a hybrid-first tool.
Where it runs: hybrid is available — experimentally — on three native platforms now. Android (via ML Kit’s Prompt API, the `firebase-ai-ondevice` library), Chrome Desktop (via the W3C Prompt API), and as of recently iOS (via Apple’s Foundation Models framework, SDK v12.13.0+). What it does not have is a Flutter SDK for the on-device side: `firebase_ai` gives you the cloud half in Dart, but to get the hybrid on-device behavior you’re writing platform channels down into each native implementation. So for a Flutter app, “Firebase hybrid” still isn’t a Flutter-native story.
On-device model: whatever the platform ships — and you don’t get to pick. On Android and Chrome that’s Gemini Nano (Nano 2 on Android, managed by AICore / the browser); on iOS it’s Apple’s Foundation Models (a different model entirely, only on devices with Apple Intelligence enabled). So the on-device behavior isn’t even consistent across platforms — it’s two different models with two different envelopes. And it only runs where the platform provides it: on Android, Pixel 6 and newer plus a growing set of Samsung phones, not the whole fleet.
Cloud models: Gemini family. No Claude, no GPT, no Llama.
Routing: rule-based modes — `PREFER_ON_DEVICE`, `PREFER_IN_CLOUD`, `ONLY_ON_DEVICE`, `ONLY_IN_CLOUD`. Simple fallback, no custom logic.
Best for: native Android/iOS or Chrome apps already in the Firebase ecosystem, doing simple single-shot text generation that fits the on-device envelope. If you’re single-platform native and already in Firebase, that built-in hybrid is genuinely the lowest-friction option. But if you need desktop, want to choose your on-device model, or want one consistent path across platforms — this isn’t your tool.
That’s where the Flutter angle bites. Firebase’s built-in on-device hybrid is per-platform native (Gemini Nano on Android/Chrome, Apple Foundation Models on iOS), it has no Flutter SDK, and no desktop at all. “Flutter” means one codebase across iOS, Android, web, and desktop, with the same model and the same behavior everywhere — and Firebase’s on-device path can’t give you that.
So you can still use Firebase here, just for a different job. I built a workshop that does exactly this: workshop-flutter-gemma-hybrid-ai runs on every Flutter platform by using Firebase AI Logic for the cloud side only (Gemini), and `flutter_gemma` for the on-device side. The cross-platform hybrid comes from flutter_gemma, not from Firebase’s own on-device path — which is exactly what later led me to wrap flutter_gemma in Genkit, so the cloud and on-device halves finally speak one API.
3. Cactus
A relatively new entrant — YC-backed, ~5k GitHub stars (cactus-compute/cactus), v1.7 as of writing. Cactus is a from-scratch on-device inference engine optimized for ARM (NPU acceleration, INT4 quantization, zero-copy memory mapping) plus a hybrid routing layer on top.
Where it runs: cross-platform SDKs for Flutter, React Native, Swift, Kotlin, Python, C++. Mobile (iOS 12+, Android API 24+), macOS, wearables. No Windows or Linux client SDKs.
On-device models: Cactus uses a proprietary `.cact` format (migrated from GGUF in v1). Catalog includes Qwen 3, Gemma 3 270M, LFM2 (Liquid AI), DeepSeek, Whisper, Moonshine. You can request new models or port them yourself.
Cloud side: this is where the lock-in shows up. Cactus hybrid handoff routes through Cactus Cloud their own API. You can’t plug in your own Gemini or OpenAI key for the cloud branch — it goes through Cactus. For Live Transcription the handoff is fully automatic; for Language Model and Batch Transcription it’s a paid feature gated behind “contact us to enable cloud handoff”.
Routing: fully automatic, based on real-time confidence measurement during inference. `CompletionResult.needsCloudHandoff` flips to true when confidence drops below a threshold or context exceeds device limits. This is Cactus’s signature idea — the routing decision is made for you, so you ship less code. Whether that’s a feature or a limitation depends on your app: it’s a great default when you want handoff to “just work,” and a constraint when your routing policy needs to be explicit and testable (privacy rules, custom thresholds). Different philosophy from the write-it-yourself approach later in this article — neither is wrong.
Best for: voice/transcription apps where automatic confidence-based handoff matches your UX, cross-platform mobile, teams willing to trade routing control and cloud-provider choice for fast on-device inference and a single SDK.
4. Genkit
Google’s open-source AI orchestration framework (genkit.dev). Originally launched for Node.js/TypeScript as a server-side framework — define flows, plug in models from any provider, get observability and evals for free. Then the Dart SDK landed in 2025 (still officially Preview, worth knowing before you bet production on it) and made Genkit run client-side too — and as a Flutter dev, that was the moment I genuinely got excited: the same clean flow/model abstraction I’d envied from the TypeScript side, now running right inside my app. (If you want a proper from-scratch introduction to Genkit Dart — flows, schemas, the CLI — Building AI Features in Dart and Flutter with Genkit by Pavel Jbanov, a lead engineer on the Genkit team, is the best starting point; this article assumes the basics and focuses on hybrid.)
Where it runs: server (Node.js, Go, Python, Dart) for cloud-first scenarios. Dart/Flutter via the `genkit` package for client-side flows. On its own, Genkit doesn’t run anything on-device — it’s a routing and orchestration framework.
Cloud models: any model with a Genkit plugin. `genkit_google_genai` for Gemini, `genkit_openai` for GPT, `genkit_anthropic` for Claude, `genkit_vertexai`, `genkit_ollama`, plus the broader Node.js plugin ecosystem if you go server-side. The model is just a `model:` reference — swap it, the rest of your flow doesn’t notice.
Routing: plain code. You decide. There’s no automatic confidence scoring, no `PREFER_` mode — you write `if (privacy) … else if (offline) …` in normal Dart. Less magic, more control.
On-device on its own: none. Genkit by itself doesn’t have an on-device backend. That’s where the next plugin comes in.
Genkit on-device: the plugin ecosystem
Genkit on its own routes to cloud models. To bring it on-device you add a provider plugin — and this is where Genkit’s open architecture pays off. There isn’t one blessed on-device backend; there’s a growing set of them, each wrapping a different inference engine but all exposed the same way — an interchangeable `model:` reference behind the same `ai.generate()` call:
- genkit_flutter_gemma — wraps `flutter_gemma` (MediaPipe / LiteRT-LM), all six Flutter platforms.
- genkit_llamadart (by Jhin Lee) — wraps `llamadart` (llama.cpp), GGUF models, in-process.
- genkit_chrome — Gemini Nano in the browser; Web-only and generation-only, so narrower than the other two, but zero model download.
And the list keeps growing. Because a Genkit plugin is just an adapter between a model runtime and Genkit’s `Model` interface, there’s nothing stopping you from writing your own — wrap whatever inference engine you like, register it, and it becomes another `model:` reference in the same flows. That’s the whole point: the framework doesn’t lock you to one engine the way a single-vendor SDK does.
The choice between them isn’t cosmetic, though — the first two wrap genuinely different stacks with different strengths. Since this article is about building, let’s get concrete about that choice.
Genkit + `genkit_flutter_gemma`: the Flutter hybrid story
When Genkit Dart launched, a few of us picked it up early — and noticed the same gap. The cloud side had clean plugins (`googleAI` for Gemini, others for OpenAI and Anthropic), but the on-device side didn’t exist in Genkit at all. Jhin Lee and I were working in parallel, comparing notes — he shipped `genkit_llamadart` first, in mid-March 2026, wrapping his own `llamadart` llama.cpp engine; I followed a week later with `genkit_flutter_gemma`. Two different on-device runtimes, the same Genkit `Model` interface.
The two plugins are not interchangeable — they wrap fundamentally different stacks, and the choice matters. `genkit_llamadart` is built on llama.cpp through Jhin’s `llamadart` package. That means GGUF — the de-facto open format with the widest community model zoo (Llama family proper, Mistral, the open Qwen variants, original DeepSeek, Phi, anything quantized to GGUF on Hugging Face). It comes with `llamadart`’s “Pure Native Assets” build pipeline, so there’s zero CMake or platform-project setup. And because llama.cpp itself is the upstream reference for on-device LLM inference outside Google’s ecosystem, you get fast updates the moment a new architecture lands there. The natural fit if you want vendor neutrality and the broadest possible model selection.
`genkit_flutter_gemma` is the opposite choice — deeply tied to Google’s stack via MediaPipe / LiteRT-LM, and that’s the point. You get the entire Gemma family with the latest hardware optimizations: Gemma 4 with Matryoshka nested variants (E2B/E4B) and Multi-Token Prediction for 2–3x speedup, native multimodal — image and audio input running on a phone — plus the specialized models that exist only in this ecosystem: FunctionGemma for tool calling and EmbeddingGemma for vectors. And because `flutter_gemma` ships an on-device vector store (qdrant-edge on native, wa-sqlite on Web) plus a managed model installer with download progress, retries, and Android foreground service for >500MB files, you don’t have to build the surrounding infrastructure either. The fit if your app benefits from Google’s bleeding-edge model work, multimodal audio, or a ready-made RAG layer.
`genkit_flutter_gemma` registers `flutter_gemma` as a standard Genkit on-device model provider. Once installed, your local Gemma (or Qwen, or Phi, or DeepSeek, or FunctionGemma, or EmbeddingGemma) shows up in the same `Genkit` instance as your cloud Gemini, and the only difference between them is the `model:` parameter.
// cloud
ai.generateStream(model: googleAI.gemini('gemini-2.5-flash'), prompt: prompt)
// on-device
ai.generateStream(model: flutterGemma.model('gemma-3-1b-it'), prompt: prompt)The honest tradeoff is that you write more code than with Firebase AI Logic, and you don’t get Cactus’s automatic confidence handoff or its hand-tuned ARM engine — MediaPipe / LiteRT-LM is fast, but you’re writing the routing yourself. What you get in return is control: full open source, every platform, any cloud provider, any model the LiteRT ecosystem supports. For a production Flutter app that needs real hybrid — not “Android-only Gemini Nano hybrid” — this is the path I reach for first.
From here on I’ll build with `genkit_flutter_gemma` — it’s the one I wrote, so I know where the bodies are buried, and it has the widest reach. But everything that follows is about the routing, which is identical whichever on-device plugin you pick: swap the provider, the policy code doesn’t change.
Setup
dependencies:
genkit: ^0.13.2
genkit_google_genai: ^0.2.7
genkit_flutter_gemma: ^0.4.0
flutter_gemma: ^0.16.3
connectivity_plus: ^6.0.0import 'package:genkit/genkit.dart';
import 'package:genkit_google_genai/genkit_google_genai.dart';
import 'package:genkit_flutter_gemma/genkit_flutter_gemma.dart';
final ai = Genkit(
plugins: [
googleAI(apiKey: Env.geminiApiKey),
GenkitFlutterGemmaPlugin(
models: [
FlutterGemmaModelConfig(
name: 'gemma-3-1b-it',
modelType: ModelType.gemmaIt,
),
],
),
],
);Two plugins, one `ai` instance. Every model — cloud or on-device — is now available through the same `ai.generate()` call.
The types of routing
Before any code — what are you actually routing on? In practice the signal behind a routing decision is one of five. You don’t need all of them; most apps use one or two. But they’re worth understanding individually, because each answers a different product question and each has its own failure mode.
- Complexity. Can the local model handle this, or does it need the cloud’s reasoning depth? “What’s the currency in Japan?” is a lookup any 1B model nails; “compare three neighborhoods for a family with toddlers, factoring in stroller access and nap-friendly cafés” is multi-step reasoning that a small model will confidently botch. The trap here is that complexity is hard to measure before you’ve answered — you often only know a query was too hard once you see the local model’s mediocre answer. So complexity routing tends to need either a cheap up-front classifier (which has its own accuracy ceiling) or a confidence check after the fact.
- Privacy. Is this data allowed to leave the device at all? A passport number, a hotel address, a booking reference, anything a user typed into a “my account” screen — these stay on-device no matter how strong the signal or how much better the cloud answer would be. This is the one type that’s categorically different from the other four: it’s not a performance tradeoff you tune, it’s a hard constraint, often a legal one (GDPR, HIPAA, your own privacy policy). Mixing it in with the “optimize for quality” signals is how data leaks happen — privacy routing has to win every tie, and it has to be auditable.
- Connectivity. Is there a network at all? This is the most binary signal and the one users feel most viscerally — an app that goes dead in airplane mode or a foreign country with no roaming is an app they stop trusting. Offline, the cloud simply isn’t an option, so the on-device model becomes the graceful-degradation path. The subtlety is that connectivity isn’t really binary in the field: you can be “online” on a hotel captive portal that resolves DNS but blocks your API, or on a 2G cell edge where the request technically goes through but takes 40 seconds. A real connectivity policy is as much about timeouts as about a boolean.
- Cost. Every on-device token is free; every cloud token is billed. For a consumer app with a free tier, that math compounds fast — a few thousand active users each firing “is it raining in Lisbon?” at a paid endpoint is a real monthly line item, and most of those queries a local model could have answered. Cost routing keeps the high-volume, low-stakes traffic on-device and reserves the cloud for the queries that actually justify the spend. The thing to watch is that “cheap” and “good” often point the same way (simple queries are both cheaper and fine for local), so cost and complexity routing frequently collapse into the same policy.
- Capability / confidence. Let the local model try first and escalate to the cloud only when its own answer isn’t good enough. This is the most sophisticated signal because it’s reactive rather than predictive — you don’t guess in advance whether local can handle it, you let it attempt and judge the result. It gives the best UX in theory (instant local draft, silent cloud upgrade when needed) and the most ways to get it wrong in practice: a model rating its own confidence is notoriously unreliable, and “escalate on low confidence” can quietly turn into “escalate on everything” if your threshold is off.
These are policies, not techniques. The interesting question is how you express them in code — and there’s a ladder, from a one-line model swap to a full fallback service. Let’s climb it.
Implementing it: from a one-liner to a service
Here’s the thing most articles get wrong: they show you the most elaborate version first, and you walk away thinking hybrid AI needs a whole abstraction layer. It doesn’t. The right amount of structure depends entirely on how complex your routing policy is — and that ranges from a single ternary to a dedicated service.
So instead of one canonical “how to do hybrid”, think of it as four levels, each adding structure only when the previous one runs out of room:
Get Sasha Denisov’s stories in your inbox
Join Medium for free to get updates from this writer.
1. Swap the `model:` — the decision is a known flag, so a ternary inside one `generate` call is the entire routing layer.
2. A local model as the router — the signal isn’t known up front, so a cheap on-device inference decides which model answers.
3. Framework middleware — for cross-cutting concerns (retry, logging) that aren’t routing per se but live around the call.
4. A fallback service — for true graceful degradation across providers, the one piece Genkit won’t write for you.
You don’t graduate through these like a tutorial — you pick the lowest level that expresses your policy. Most apps never leave Level 1. Let’s climb anyway, because the higher levels show what’s there when you need it.
Level 1 — just swap the `model:`
The simplest hybrid is a ternary. Here’s privacy routing: a request that touches personal data goes on-device, everything else goes to the cloud.
Stream<String> answer(Query q) {
return ai.generateStream(
model: q.isSensitive
? flutterGemma.model('gemma-3-1b-it') // PII stays on the device
: googleAI.gemini('gemini-2.5-flash'),
prompt: q.text,
).map((chunk) => chunk.text);
}That’s the whole routing layer. One `generateStream`, the decision lives in the `model:` argument.
The part that matters is where `isSensitive` comes from. Don’t regex the user’s text hoping to spot a passport number — that’s a losing game (it misses “AB1234567”, trips on a phone number quoted in a history question, and knows nothing about non-US formats). Instead, let the app’s context set the flag: a query typed into the “my bookings” screen, or a form field you’ve marked sensitive, is sensitive by construction — you don’t have to guess. If you genuinely must scan free-form text, that’s a job for a real PII detector (Microsoft Presidio, Google DLP, or an on-device classifier), not a handful of regexes.
Level 2 — a local model as the router
Sometimes the signal isn’t known up front — you need a model to judge it. Complexity routing: run a tiny local classifier first (free, on-device), then route the real work based on its verdict.
Future<String> answer(String query) async {
// Fast local classifier — costs nothing, runs on device
final verdict = await ai.generate(
model: flutterGemma.model('gemma-3-1b-it'),
prompt: 'Classify as SIMPLE or COMPLEX: "$query". One word.',
);
final isComplex = verdict.text.trim().toUpperCase() == 'COMPLEX';
return (await ai.generate(
model: isComplex
? googleAI.gemini('gemini-2.5-flash')
: flutterGemma.model('gemma-3-1b-it'),
prompt: query,
)).text;
}Same `model:`-swap idea as Level 1 — the only new thing is that the swap is driven by an on-device inference instead of a flag. Honest caveat: a 1B model classifying its own complexity has limits. It nails the extremes and gets fuzzy in the middle; in production, gate it behind a cheap keyword/length heuristic so you only pay for classification when it’s ambiguous.
Level 3 — let the framework handle the cross-cutting stuff
Routing isn’t only about which model — it’s also about what happens when a call fails. This is where Genkit’s middleware earns its place: instead of hand-rolling a retry loop, you attach the built-in `retry` middleware to the cloud call.
final ai = Genkit(plugins: [googleAI(...), GenkitFlutterGemmaPlugin(...), RetryPlugin()]);
final response = await ai.generate(
model: googleAI.gemini('gemini-2.5-flash'),
prompt: query,
use: [retry(maxRetries: 3, statuses: [StatusCodes.UNAVAILABLE, StatusCodes.RESOURCE_EXHAUSTED])],
);The `use:` parameter wraps a single `generate` call with middleware — `retry` here re-issues the request with exponential backoff on transient cloud errors, no extra code from you. It’s the idiomatic Genkit answer for cross-cutting concerns (retry, logging, caching); you don’t reinvent them per call.
But notice what `retry` does not do: it retries the same model. It won’t fall back from cloud to on-device when the network is gone — that’s a different model entirely, and Genkit Dart has no built-in `fallback` for it (the JS SDK does; Dart, as of 0.13, only ships `retry`). Cross-provider fallback is the one piece you write yourself — which brings us to the last level.
Level 4 — a fallback service
When you want real graceful degradation — try the local model, fall back to the cloud (or vice versa), and stream the whole way — you wrap the two services in a small router.
enum AIStrategy { localFirst, cloudFirst, localOnly, cloudOnly }
class HybridAIService {
final LocalAIService local;
final CloudAIService cloud;
AIStrategy strategy = AIStrategy.localFirst;
HybridAIService({required this.local, required this.cloud});
Stream<String> generateResponseStream(String prompt) async* {
switch (strategy) {
case AIStrategy.localFirst:
var yielded = false;
try {
await for (final chunk in local.generateResponseStream(prompt)) {
yielded = true;
yield chunk;
}
} catch (e) {
if (yielded) rethrow; // don't silently switch mid-response
yield* cloud.generateResponseStream(prompt);
}
case AIStrategy.cloudFirst:
// ... symmetric: try cloud, fall back to local
case AIStrategy.localOnly:
yield* local.generateResponseStream(prompt);
case AIStrategy.cloudOnly:
yield* cloud.generateResponseStream(prompt);
}
}
}This is the actual hybrid layer from the workshop app (full source and codelab)
Notice the router knows nothing about Genkit — Genkit lives inside each `AIService`, and the inner generation is the same `model:` swap from Level 1. The one detail worth dwelling on is `if (yielded) rethrow`: fallback is great before the first token, but once the local model has started streaming to the user you can’t silently swap to the cloud mid-sentence — you’d get garbled output. So it only falls back if the primary failed before producing anything. That’s exactly the decision a black-box confidence router makes for you; here it’s three words you can read, test, and change.
And that’s the ladder. Most apps live on Level 1. Reach for the higher rungs only when the policy demands it — a model-judged signal, framework retry, or true cross-provider fallback. The same RAG layer in the workshop rides on top of all of it: `ai.embed(embedder: flutterGemma.embedder(‘embedding-gemma-300m’), …)` to vectorize documents and queries on-device, then a plain cosine search — swap the embedder reference for a cloud one and nothing else changes.
One more wrap: flows
I kept the routing examples as plain functions on purpose — the routing logic is the point, and `ai.generate` works perfectly well on its own. But there’s one optional layer worth knowing about, because it’s where the next section’s tooling comes from: the flow.
A flow is just your function wrapped in `ai.defineFlow(…)` with typed input/output schemas. The wrapping is what unlocks Genkit’s scaffolding — tracing, the Developer UI, schema enforcement, and the ability to expose the function as an HTTP endpoint. It’s also a hard requirement for some of Genkit’s features: observability and deployment only work on flows, not on bare `generate` calls. So if you want the tooling below, you wrap your routing function (any of the four levels) in a flow. I won’t reproduce the `defineFlow` boilerplate here — the official Genkit Dart flows guide covers it well.
What you get for free from Genkit
Once your routing lives in a flow, you get Genkit’s tooling on your on-device path too:
Observability. Genkit instruments every flow and `generate` call with spans, so each execution — including the on-device steps — produces a trace. Run your app through the Genkit CLI and the local Developer UI lets you inspect those traces step by step: which model a request was routed to, what the prompt was, where the time went. It’s a development-time debugger for your routing logic, not a remote telemetry pipe from users’ phones — but for figuring out why a hybrid decision went the way it did, it’s exactly the view you want.
Evals. Run the same eval harness against local Gemma and cloud Gemini. Compare quality side-by-side. A hybrid policy is only as good as the routing thresholds you pick — and evals are the only honest way to find out where local is “good enough” and where you actually need the cloud. Don’t guess; measure on your own data.
Dev-on-cloud, ship-on-device. Prototype with Gemini in development. Ship with Gemma in production. The flow logic doesn’t notice. This alone has cut my iteration time significantly — no need to download a model file every time I change a prompt.
Production gotchas
Cold start. flutter_gemma loads the model on first `getActiveModel()` call — for a 1B model, that’s about 1–2 seconds on a modern phone. Don’t lazy-load on the first user message. Pre-warm on app resume when the device is idle.
iOS memory. iOS aggressively kills apps exceeding ~1.5GB RAM. Llama 7B+ won’t fit. Stick to 1B–3B models on iPhone, or escalate heavy tasks to cloud.
Model file delivery. Gemma 3 1B `.task` (int4) is around 600MB. Don’t bundle it in the APK — the App Store will reject you. Download on demand, show progress. I covered the full model delivery story in [the second article in this series](https://medium.com/@denisov.shureg/the-dawn-of-offline-ai-agents-in-your-pocket-55fff80d794b).
Non-deterministic tests. Set `temperature: 0` and `randomSeed: 42` in unit tests for flows. Otherwise routing decisions become flaky.
Web. On-device inference does run in the browser — and as of `flutter_gemma 0.16.2` there are two backends: the established MediaPipe path (GPU-backend models via `-web.task` builds) and an early-preview LiteRT-LM path (`@litert-lm/core`, WebGPU/WASM, text-only, for `.litertlm` models). The catch is model format: a mobile `.task` won’t just work on Web, you need a web-specific build. So “works on Web” is real, but check that your chosen model ships a web variant before you count on the local path there.
Closing
If there’s one thing to take away, it’s that hybrid AI is no longer an exotic architecture — it’s a routing policy, and the framework support to express it cleanly finally exists. Cloud and on-device models now sit behind one interface; the `model:` reference is the whole switch. That removes the part that used to make hybrid painful and leaves you with the part that was always the real work.
So the work that’s left is the work that actually matters: deciding your policy. And that’s a product decision, not a framework one. My advice, in order: start with privacy routing — it’s non-negotiable and the easiest to get right when the signal comes from app context rather than guessing at text. Add cost routing once you have real traffic and a bill to look at. Reach for complexity or confidence routing last, and only after your evals tell you where the local model actually falls short — not where you assume it does. Don’t build all five signals because an article listed them; build the one your users feel.
And keep it as simple as the policy allows. Most apps never need to leave Level 1 — a ternary on a known flag. The fallback service is there when you genuinely need graceful degradation, not as a starting point.
I’m still deciding where this series goes next. A few candidates are on my mind:
- Hybrid RAG — the same routing idea applied to retrieval, not generation: private data searched on-device with qdrant-edge, a larger shared corpus in the cloud, and the query routed between them. The previous RAG article covered the on-device-only case; this would be the hybrid version.
- On-device agents — multi-step orchestration with tool calling in a loop, running entirely locally. What does an agent that plans, calls tools, and iterates look like when there’s no server in the picture?
- On-device multimodal — building real features on Gemma 4’s image and audio input: snap a photo of a menu and ask about it, or talk to the app, all without leaving the device.
But I’d rather write what you actually want to read — so tell me in the comments which of these (or something else entirely) you’d find most useful.
Links:
Genkit:
- - genkit.dev — official site and docs
- - Genkit Dart flows guide — defining flows, schemas, observability
- - genkit — Genkit Dart SDK
- - genkit_google_genai — Gemini cloud via Genkit
- - Building AI Features in Dart and Flutter with Genkit — from-scratch intro by Pavel Jbanov (Genkit team)
On-device plugins:
- - genkit_flutter_gemma — on-device Gemma as a Genkit model provider
- - flutter_gemma — cross-platform on-device inference engine (MediaPipe / LiteRT-LM)
- - genkit_llamadart / llamadart — GGUF / llama.cpp on-device, by Jhin Lee
Other hybrid approaches:
- - Firebase AI Logic — Google’s hybrid SDK (Gemini Nano + cloud Gemini)
- - Cactus — ARM-optimized on-device engine with confidence-based handoff
Workshops:
Previous articles in this series:
