Build a Voice RPG with Agora Conversational AI and MCP
A voice agent isn’t particularly complicated when you break it down into its parts; speech recognition, an LLM, tool calls, and text-to-speech. The real challenge is getting all of those pieces working together smoothly so it feels like a single real-time interaction.
Agora’s Voice RPG Recipe demonstrates this end-to-end flow through a small, playable RPG. You speak with an AI Dungeon Master, choose a character, enter combat, attack, cast spells, or flee, while the agent interprets your voice, triggers the appropriate game action, and responds with the result.
This tutorial walks you through the setup and helps you get to your first conversation with the agent.
What You will get
By the end, you will have:
- a browser-based voice RPG;
- an AI Dungeon Master powered by Agora ConvoAI;
- a playable character with persistent game progress;
- voice-controlled gameplay with spoken responses.
Together, these components form a complete voice-agent demo rather than a chat-only interface.
Architecture at a Glance
Before starting the setup, it helps to have a simple architecture diagram of the pieces and how they connect:
To explain it simplify, the browser runs a local NextJS app and exposes a public MCP endpoint (using a tunnel out) that allows the cloud-based Agent to use the game tools in real-time. During setup, we’ll configure each of these pieces and connect them together.
Prerequisites
Prepare the following tools and account access:
Get started a Voice RPG
Step 1: Create the project
Open the Voice RPG Recipe and locate the initialization command. You can also run it directly:
agora init my-voice-rpg --recipe voice-rpg
cd my-voice-rpgThis creates the project directory containing the frontend, Agent service, and game logic. Run the remaining commands from this directory.
Step 2: Set up the project dependencies
Run:
bun run setupThis prepares the dependencies required by the frontend and the Python service.
Step 3: Connect the project to Agora
Log in to Agora, select the project you want to use, and write its environment settings to the server configuration:
agora login
agora project use <your-project>
agora project env write server/.env.localReplace <your-project> with your Agora project.
Get June’s stories in your inbox
Join Medium for free to get updates from this writer.
The default Recipe uses Agora-managed OpenAI configuration, so you generally do not need to add an OpenAI API key for this walkthrough.
Step 4: Create a public address for the Agent service
The Agent service runs on your computer, but the cloud-based Agent needs to reach its MCP tools. Use ngrok to expose port 8000 temporarily:
ngrok http 8000Keep this terminal window open. Copy the public https:// address shown by ngrok; you will use it in the next step.
Step 5: Configure the MCP endpoint
Open server/.env.local and add:
MCP_ENDPOINT=https://<your-tunnel>/mcpReplace <your-tunnel> with the hostname provided by ngrok. For example:
MCP_ENDPOINT=https://example.ngrok-free.dev/mcpKeep the /mcp suffix. This is the endpoint the cloud-based Agent uses to access the game tools. Do not use localhosthere.
Step 6: Start the application
From the project directory, run:
bun run devWhen the services are ready, open you browser to http://localhost:3000 and load the front-end. Once the page loads you should see the Voice RPG interface.
Step 7: Start your first conversation
Follow these steps in the browser:
- Click Start Conversation.
- Allow microphone access when your browser asks for it.
- Say:
I want to be a warrior. - Wait for the agent to create your character and respond.
- Once your character is ready, try saying something like:
Check my inventory
Find a fight
Attack
Cast a spell
Run away
You don’t need to use these exact word, but they are meant to illustrate the point of being able to interact with the game using natural speech.
Happy building
That’s it. You now have a working voice RPG with an AI Dungeon Master, persistent game state, and MCP tools that turn natural speech into actions in the game.
From here, make it your own. Add new character classes, build more tools, change the game mechanics, or take the same architecture and use it for something completely different.
If you build something with the recipe, drop a link or tell us what you changed in the comments. We’d love to see where you take it.
Happy building!
