Building a Conversational AI App in Flutter with Agora
Build a voice agent for web, mobile, and desktop with a Flutter interface you can extend.
A useful voice interface needs to do several things at once: capture speech, deliver the agent’s response, show what was said, and keep the user informed while the conversation connects or ends. Those details shape the experience as much as the model behind it.
The Agora Conversational AI Flutter Quickstart brings those pieces together in an app you can run and explore. It pairs a Flutter client with a small Node.js backend and Agora’s Conversational AI Engine.
In this tutorial, we’ll start a conversation in Chrome, trace it through the code, and customize the agent for a different use case. Then we’ll explore the web, Android, iOS, and macOS targets. The repository shares its Flutter UI and session controller across these platforms, with platform-specific services underneath.
What you’ll build
The sample starts a voice conversation with Ada, an assistant configured to help developers with Agora. You can speak into your microphone, hear the agent’s replies, read the transcript, mute yourself, and end the session.
The interface also exposes the channel, user IDs, agent ID, and connection state. Those details make it easier to understand what is happening while you develop your own experience.
You can adapt this foundation into a language practice partner, an onboarding assistant, a product guide, or another voice experience with your own prompt and interface.
One Flutter interface, several places to talk. The platform section below explains what each target currently supports.
How the pieces fit together
The app has three main parts:
- Flutter client: Renders the interface, requests microphone access, joins audio, and displays conversation updates.
- Node.js backend: Generates tokens and starts or stops agent sessions using server-side credentials.
- Agora services: Carry RTC audio, run the configured conversational AI pipeline, and deliver RTM events.
RTC carries the live audio. RTM carries messages such as transcript and agent-state updates. The backend keeps the Agora App Certificate out of the Flutter application.
That separation gives you clear places to work: change the interface in Flutter, adjust the agent configuration on the server, and keep credentials on the server throughout.
1. Get the quickstart
You’ll need Flutter, its bundled Dart SDK, Node.js 22 or newer, npm, Chrome, and an Agora project with Conversational AI and RTM enabled. You’ll also need the project’s App ID and App Certificate from the Agora Console.
This walkthrough uses the dependency-upgrade snapshot at commit d859829, which requires Flutter 3.47.7 and Dart 3.13.5 or newer. Checking out that revision keeps the code aligned with the examples here:
git clone https://github.com/AgoraIO-Conversational-AI/agent-quickstart-flutter.git
cd agent-quickstart-flutter
git checkout d859829
flutter --version
node --versionTo make your own changes on this snapshot, create a branch:
git switch -c my-voice-agent2. Configure the backend
Create a file named .env.local in the repository root, beside pubspec.yaml:
NEXT_PUBLIC_AGORA_APP_ID=your_agora_app_id
NEXT_AGORA_APP_CERTIFICATE=your_agora_app_certificate
NEXT_PUBLIC_AGENT_UID=123456
NEXT_AGENT_GREETING="Hi! I'm Ada. What would you like to build today?"The backend reads this file and exposes the public client configuration through /api/client-config. The NEXT_ prefixes are the environment-variable names used by this sample; the companion server is a Node.js application.
The App ID identifies your project. The App Certificate is a secret used by the backend for authentication and token generation. Keep the certificate in this server-side file and out of Dart code, browser JavaScript, and --dart-define values. The repository ignores .env.local in Git.
The sample configures its speech recognition, language model, and speech synthesis through Agora-managed inference, so its base configuration does not require separate provider API keys. Your Agora project must have access to the configured services.
Start the backend from the repository root:
cd backend
npm ci
npm run devIt listens on port 3001 by default. In another terminal, check that it is running:
curl http://localhost:3001/healthThe response should be:
{"ok":true}That confirms the server is reachable. The first conversation will exercise the Agora credentials and agent configuration.
3. Start the Flutter app
In a second terminal, navigate to the repository root and run:
flutter pub get
flutter run -d chromeClick Start Conversation and allow microphone access when Chrome asks. Once the agent joins, speak a short question such as “What can I build with Agora?”
Look for the agent’s audio, transcript updates, and connection information. Try Mute mic, unmute, and then choose End Conversation.
You now have a concrete flow to modify: start a session, exchange speech, observe its state, and clean it up.
4. Follow a conversation through the code
The main session logic lives in ConversationSessionController. It extends ChangeNotifier, allowing the interface to rebuild as session state changes.
When you start a conversation, the controller requests microphone access, obtains a token, and asks the backend to invite the agent. This excerpt shows the token-to-agent handoff:
final tokenData = await _backendApi.generateToken();
final agentResponse = await _backendApi.inviteAgent(
ClientStartRequest(
requesterId: tokenData.uid,
channelName: tokenData.channel,
),
);The backend routes provide a small API surface:
GET /api/client-config: Returns the public App ID, agent UID, and greeting.GET /api/generate-agora-token: Returns a token, client UID, and channel name.POST /api/invite-agent: Starts the agent for the requested channel and returns its agent ID.POST /api/stop-conversation: Stops the agent identified by that agent ID.
The controller then initializes RTM with the returned client identity and subscribes to the conversation channel. It initializes RTC and joins that same channel for audio.
The RTC service joins with these media options:
options: const ChannelMediaOptions(
clientRoleType: ClientRoleType.clientRoleBroadcaster,
publishMicrophoneTrack: true,
autoSubscribeAudio: true,
autoSubscribeVideo: false,
),This publishes microphone audio and subscribes to incoming audio. The sample does not subscribe to video.
Keep the identities consistent when extending the app: the token’s client UID is also the RTM login identity, and the agent, RTC client, and RTM subscription use the same conversation channel.
The client joining RTC and the agent joining RTC are separate events. A successful client connection is useful progress; the remote-user join event tells you that the agent’s audio participant has arrived.
Get Akshay Nandwana’s stories in your inbox
Join Medium for free to get updates from this writer.
Here is the successful session path, from the first click to cleanup:
The channel ties the audio and events together. The agent ID identifies the server-side session to stop; the client UID identifies the person joining it.
5. Understand the agent configuration
The backend configures the agent in backend/src/agent.js.
At this snapshot, the pipeline uses Deepgram nova-3 for speech recognition, OpenAI gpt-4o-mini for language responses, and MiniMax speech_2_6_turbo for speech synthesis. These are the sample’s configured choices; they are not requirements for every app you build.
A voice turn takes a trip through the pipeline. RTM separately brings transcript and state updates back to the interface.
The agent configuration contains its instructions, greeting, turn-detection settings, and RTM options. The following excerpt enables the RTM data channel and requests metrics and error messages:
advancedFeatures: { enable_rtm: true, enable_tools: true },
parameters: {
data_channel: 'rtm',
enable_error_message: true,
enable_metrics: true,
},The server creates a session for the client’s channel and starts it:
const session = agent.createSession({
channel: channel_name,
agentUid: process.env.NEXT_PUBLIC_AGENT_UID ?? String(DEFAULT_AGENT_UID),
remoteUids: [requester_id],
idleTimeout: 30,
expiresIn: ExpiresIn.hours(1),
debug: false,
});
const agentId = await session.start();Flutter stores the returned agent ID so it can ask the backend to stop that specific session later.
6. Make the agent your own
Start with a focused change to the agent’s purpose. For example, replace the ADA_PROMPT text in backend/src/agent.js with instructions for a language practice partner:
const ADA_PROMPT = `You are a patient English conversation partner.
Help the learner practice everyday situations.
Keep each response to one or two short sentences.
Ask one question at a time.
When correcting a mistake, offer one natural alternative and continue the conversation.`;Update the greeting in .env.local to match:
NEXT_AGENT_GREETING="Let's practice ordering at a café. What would you like to order?"Restart the backend and start a fresh conversation. The existing RTC, RTM, and session flow can support the new persona without changes to the transport code.
Next, adapt the Flutter interface in lib/main.dart. You might add a scenario selector, a lesson objective, or a completion screen. Transcript formatting lives under lib/utils/, while the session controller owns lifecycle state.
If you add scenarios that affect the server prompt, send an allowed scenario identifier to the backend and map it to server-owned instructions. That gives your app a controlled configuration surface as it grows.
The scenario selector is an extension you can add; the current sample starts with Ada’s configured persona. A possible extension would look like this:
7. Take the same app to web, mobile, and desktop
Flutter lets this sample share its interface and session logic across targets. The RTC and RTM implementations still depend on the platform, so the current quickstart has the following scope:
- Web / Chrome: Run
flutter run -d chromefor browser RTC audio and RTM transcripts/state through the JavaScript bridge. - Android: Run
flutter run -d <android-device-id>for native RTC audio and RTM transcripts/state. - iOS: Run
flutter run -d <ios-device-id>for native RTC audio and RTM transcripts/state. Requires Xcode, CocoaPods, and device setup. - macOS: Run
flutter run -d macosfor RTC audio and microphone capture. RTM is stubbed, so live transcripts and RTM agent-state updates are unavailable.
This repository includes host projects for these targets. Windows and Linux would require additional host setup, compatible SDKs, and validation; this article does not treat them as supported quickstart targets.
Web: share a browser experience
Keep the Node.js companion running and launch Chrome:
flutter run -d chromeThe browser target defaults to http://localhost:3001. To generate the web assets for hosting, run:
flutter build webThe output goes into build/web. A hosted version also needs a reachable backend, HTTPS for microphone access, and the public backend URL supplied when building:
flutter build web \
--dart-define=BACKEND_BASE_URL=https://your-backend.example.comThe backend address is embedded in that build. Supply the real address before deploying the web assets, and add the backend access controls described at the end of this tutorial.
Android: emulator or physical device
For an Android emulator, keep the backend running and use:
flutter devices
flutter run -d <android-device-id>The app defaults to http://10.0.2.2:3001 on Android, which lets an emulator reach the host computer. The updated project builds against Android SDK 37 and requires API 24 or newer at runtime. For a physical Android phone, use the LAN override below.
iOS: simulator or physical device
On a Mac with the required Xcode components installed, select an iOS simulator or connected device from flutter devices:
flutter run -d <ios-device-id>The iOS default is http://127.0.0.1:3001, suitable for a simulator reaching the backend on the same Mac. The iOS project requires iOS 15 or newer, Xcode with the appropriate platform components, and CocoaPods for the RTM plugin. Physical-device builds also need signing configured and a reachable backend URL.
Physical phones: use a reachable backend
For a physical Android or iOS device, point Flutter at a backend address the device can reach. For example, with the phone and computer on the same network:
flutter run -d <device-id> \
--dart-define=BACKEND_BASE_URL=http://192.168.1.10:3001Replace the example address with your computer’s LAN IP. Allow local-network access if prompted and make sure the computer’s firewall allows the backend connection. Use HTTPS for a deployed backend.
macOS: explore a desktop voice interface
With the backend running on the same Mac:
flutter run -d macosThe app defaults to http://127.0.0.1:3001. You can explore RTC audio and microphone capture in a native desktop window. The current macOS RTM service is a stub, so transcript and RTM agent-state events do not reach the UI. Adding a compatible macOS event transport is one way to extend this quickstart.
Choose a target for the experience you want to explore: Chrome offers the tutorial’s initial browser path, Android and iOS take the conversation onto phones, and macOS provides a starting point for a desktop voice interface.
Where to look when something goes wrong
If the app cannot load configuration, check the backend’s /health endpoint and the client’s backend URL. On a physical phone, localhost points to the phone itself.
If a conversation fails to start, check the backend terminal, the App ID and certificate, and whether your Agora project has access to Conversational AI and RTM. If you already use the Agora CLI, agora project doctor --deep provides additional project diagnostics.
If audio works but transcripts do not appear, inspect the RTM connection and channel identity. On web, web/index.html loads the browser SDKs, and web/rtm_bridge.js connects the RTM JavaScript SDK to Flutter.
When you change SDK versions, keep the Flutter RTC package and its documented Iris browser script compatible. The snapshot above pins a matching combination.
Build on top of the quickstart
Once your first conversation works, choose one useful behavior to add: a different agent persona, a guided practice session, a product-specific interface, or a conversation history view.
The session controller already provides places to work with microphone controls, token renewal, transcript updates, and ending a conversation. When End Conversation succeeds, it stops the backend agent, leaves RTM and RTC, disposes the services, and resets the client state.
For a deployed product, extend the companion backend with user authentication and session ownership checks before exposing its agent-control routes publicly. Its current routes are designed for a local quickstart. Add recovery for partial startup and cleanup failures as you develop the complete session experience.
Clone the Flutter quickstart, run the sample conversation, and give Ada a new purpose. A working voice session is a useful starting point for discovering what your own app should do next.
