Building Snap & Solve: Connecting a Web Frontend with Kotlin, Node.js, Python and Gemini AI

suraj Kumar

7 min read

·

Sep 9, 2026

--

Listen

Share

Building a feature that looks simple from the user’s perspective can involve several different technologies behind the scenes.

For Snap & Solve, the user only sees a simple flow:

Take a picture → Ask a question → Get a step-by-step solution.

Behind that screen, however, the request moves through multiple services:

JavaScript → Kotlin/Ktor → Node.js/Express → Python/FastAPI → Gemini AI

This article explains how the complete flow works, why these layers are used, and some of the important problems that came up during implementation.

1. The Architecture

The Snap & Solve feature is divided into four main parts:

Browser
   │
   ▼
JavaScript Frontend
student-library.js
   │
   ▼
Kotlin / Ktor
App.kt
   │
   ▼
Node.js / Express
studentLibraryRoutes.ts
   │
   ▼
Python / FastAPI
app.py
   │
   ▼
Gemini Vision AI

There is also Google Cloud Storage involved when the user uploads an image.

The main components map roughly to concepts familiar from iOS development:

iOSWeb / BackendSwiftUI ViewHTML / JavaScript UIViewModelJavaScript state / logicURLSessionfetch()API ClientKtor / Express HTTP callsRepository / ServiceBackend serviceAsyncStream / publisherSSE streamingImage uploadFormDataBackend APIKtor / Express / FastAPI

The important thing is that each service has a specific responsibility.

2. What Happens When a Student Solves a Doubt?

The complete request looks like this:

Student
   │
   │ submits question + image
   ▼
student-library.js
   │
   │ POST /api/solve-doubt
   ▼
Ktor Gateway
   │
   │ POST /api/v1/studentLibrary/solve-doubt
   ▼
Node.js / Express
   │
   │ validate user + prepare request
   ▼
Google Cloud Storage
   │
   │ image URL
   ▼
Mitra AI Engine
   │
   │ Gemini Vision
   ▼
Streaming response
   │
   ▼
Node.js
   │
   ▼
Ktor
   │
   ▼
JavaScript
   │
   ▼
Step-by-step solution

This is important because the browser does not directly call the AI service.

Instead, the request passes through the backend layers.

3. Frontend: Sending the Doubt

The frontend implementation lives in:

student-library.js

When the student taps Snap & Solve, the frontend collects:

• Prompt
• Grade
• Subject
• Optional image URL
• Authentication token

The request is sent using fetch().

Conceptually, it looks like:

const formData = new FormData();

formData.append("prompt", prompt);
formData.append("grade", grade);
formData.append("subject", subject);if (uploadedGcsImageUrl) {
    formData.append("imageUrl", uploadedGcsImageUrl);
}const response = await fetch("/api/solve-doubt", {
    method: "POST",
    headers: {
        Authorization: `Bearer ${token}`
    },
    body: formData
});

The important part here is FormData.

Because the request can contain both normal fields and image-related data, multipart form data is used.

The frontend then reads the response as a stream rather than waiting for one large JSON response.

4. Why Streaming Is Used

A normal API might work like this:

Request
   ↓
Wait 10 seconds
   ↓
Complete response

For an AI solution, this can feel slow.

Instead, Snap & Solve streams the solution:

Request
   ↓
AI starts generating
   ↓
Chunk 1
   ↓
Chunk 2
   ↓
Chunk 3
   ↓
Chunk 4
   ↓
Complete

The browser uses:

const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
    const { done, value } = await reader.read();    if (done) break;    const chunk = decoder.decode(value, { stream: true });    appendSolutionChunkToUI(chunk);
}

This is somewhat similar conceptually to consuming an AsyncSequence or streaming publisher in Swift.

The UI can start displaying the answer before the complete AI response has arrived.

5. Kotlin/Ktor as the Gateway

The Kotlin layer is implemented in:

App.kt

The browser calls:

POST /api/solve-doubt

Ktor receives that request and forwards it to the Node.js backend.

The important part is that Ktor doesn’t consume the complete AI response and then send it back.

Instead, it copies the response stream directly:

call.respondBytesWriter {
    response.body<ByteReadChannel>().copyTo(this)
}

This allows the streaming response to travel through the gateway.

The request therefore becomes:

Browser
   ↓
Ktor
   ↓
Node
   ↓
AI
   ↓
Node
   ↓
Ktor
   ↓
Browser

The Ktor implementation also normalizes the API base URL so that the expected /api/v1 path is present.

6. The /api/v1 Routing Problem

One of the important issues encountered during implementation was routing.

The Express service exposes the endpoint:

/api/v1/studentLibrary/solve-doubt

But the Ktor gateway could construct a URL like:

https://paathshala-bf93b.uc.r.appspot.com/studentLibrary/solve-doubt

That endpoint does not match the Express route.

The expected URL is:

https:XXXXX/studentLibrary/solve-doubt

Because of this mismatch, the request could fall through to the Express fallback:

Hello from server!

That response looked unrelated to Snap & Solve, but the actual problem was simply the URL path.

The fix was to normalize the API base URL in Ktor:

if already ends with /api/v1
    use it

else if ends with /api
    append /v1else
    append /api/v1

This small routing detail was critical to getting the complete request chain working.

7. Node.js / Express

The Node.js layer contains:

studentLibraryRoutes.ts

The endpoint is registered under:

/api/v1/studentLibrary

and handles:

POST /solve-doubt

So the complete endpoint becomes:

/api/v3/studentLibrary/solve-doubt

The Express route extracts values such as:

prompt
grade
subject
context
imageUrl

It then creates a multipart request for the AI service.

The Node.js service also adds the API key required by the AI service:

x-api-key

Finally, instead of waiting for the AI response to finish, it pipes the response body directly back to the client.

This is another important part of maintaining streaming through the architecture.

8. Python / FastAPI AI Service

The AI service is implemented in:

app.py

using:

Python
FastAPI
Modal Cloud

The endpoint is:

POST /api/solve-doubt

It receives:

prompt
grade
subject
image_url

The service validates the API key and then prepares the prompt for the Gemini vision model.

The AI response is returned as a stream using Server-Sent Events.

The important point is that Python is responsible for the AI-specific work.

The other services mainly handle:

• Routing
• Authentication
• File handling
• Request forwarding
• Streaming

The AI service focuses on generating the actual solution.

9. Image Upload Flow

When the student selects an image, the image is uploaded before the AI request.

The flow is:

Browser
   ↓
Ktor
   ↓
Node.js
   ↓
Google Cloud Storage
   ↓
Public image URL
   ↓
Browser

The browser then sends that image URL as part of the solve-doubt request.

The storage path used for these images is:

student_doubts/

This means the AI service does not necessarily need to receive the original binary image directly from the browser. It can receive the image URL.

10. Authentication

The browser sends the user’s authentication token:

Authorization: Bearer <token>

The Node.js backend validates the Firebase user token.

After validation, the request can continue to the AI service.

The Node.js service also adds its own API key when communicating with the AI service.

So there are two different concerns:

User authentication
        ↓
Is this a valid student/user?

Service authentication
        ↓
Is this request coming from an authorized backend?

Keeping these responsibilities separate makes the architecture easier to control.

11. Gemini Vision

The final AI processing happens inside the Mitra AI Engine.

The service uses:

Gemini 2.5 Flash

with vision capabilities.

This allows the model to work with:

Text question
+
Image
+
Grade
+
Subject

The prompt is designed to produce a step-by-step educational solution rather than simply returning a short answer.

The response is then streamed back through the architecture.

12. Why There Are Multiple Backend Services

At first, having:

Kotlin
+
Node.js
+
Python

can seem unnecessarily complicated.

But each layer has a different responsibility.

Kotlin / Ktor

Acts as the website/API gateway.

Browser → Ktor

Node.js / Express

Handles application-level backend functionality.

Ktor → Node.js

Python / FastAPI

Handles the AI pipeline.

Node.js → Python → Gemini

This separation makes it possible to change one layer without rewriting everything.

For example, the AI implementation can change without requiring the JavaScript frontend to know anything about Python or Gemini.

13. The Most Important Concept: Follow the Request

When debugging this type of architecture, it is easy to look only at the UI.

But the UI may be working perfectly while the problem exists several services later.

The easiest way to debug is to follow the request:

1. Did JavaScript send the request?
        ↓
2. Did Ktor receive it?
        ↓
3. Did Ktor construct the correct URL?
        ↓
4. Did Express match the route?
        ↓
5. Did authentication succeed?
        ↓
6. Did Express call Python?
        ↓
7. Did Python authenticate the request?
        ↓
8. Did Gemini start generating?
        ↓
9. Did the stream return?
        ↓
10. Did JavaScript render the chunks?

This approach was especially useful for finding the /api/v1 routing issue.

14. What I Learned From This Implementation

Coming from an iOS background, some of these concepts initially felt unfamiliar.

The biggest shift was understanding that the application is not necessarily one API server.

Instead, a single user action can pass through several independent services.

The most important concepts to understand are:

HTTP request forwarding

One server can receive a request and make another HTTP request to a different server.

Multipart/FormData

Useful when sending files together with normal form fields.

Streaming

Instead of waiting for a complete response, data can be processed as it arrives.

SSE

Server-Sent Events provide a mechanism for streaming server-generated data to the client.

API Gateway

A gateway can provide a single entry point to multiple backend services.

Authentication vs API keys

A user’s authentication token and service-to-service API credentials serve different purposes.

Routing

A small difference such as:

/api/v1

vs

/api

can completely change which backend route receives the request.

15. Complete Snap & Solve Flow

Putting everything together:

STUDENT
                       │
                       ▼
              ┌─────────────────┐
              │ JavaScript UI   │
              │ student-library │
              └────────┬────────┘
                       │
                       │ upload image
                       ▼
              ┌─────────────────┐
              │ Kotlin / Ktor   │
              │    Gateway      │
              └────────┬────────┘
                       │
                       ▼
              ┌─────────────────┐
              │ Node.js Express │
              │     Server      │
              └────────┬────────┘
                       │
                       ▼
              ┌─────────────────┐
              │ Google Cloud    │
              │    Storage      │
              └────────┬────────┘
                       │
                       │ image URL
                       ▼
              ┌─────────────────┐
              │ Node.js Express │
              └────────┬────────┘
                       │
                       ▼
              ┌─────────────────┐
              │ Python FastAPI  │
              │  Mitra AI       │
              └────────┬────────┘
                       │
                       ▼
              ┌─────────────────┐
              │ Gemini 2.5      │
              │ Flash Vision    │
              └────────┬────────┘
                       │
                       │ SSE stream
                       ▼
              ┌─────────────────┐
              │ Node.js         │
              └────────┬────────┘
                       │
                       ▼
              ┌─────────────────┐
              │ Kotlin / Ktor   │
              └────────┬────────┘
                       │
                       ▼
              ┌─────────────────┐
              │ JavaScript UI   │
              │ Step-by-step    │
              │ Solution        │
              └─────────────────┘

Conclusion

Snap & Solve looks simple from the student’s perspective, but the implementation involves several layers working together.

The overall architecture is:

JavaScript
    ↓
Kotlin / Ktor
    ↓
Node.js / Express
    ↓
Python / FastAPI
    ↓
Gemini Vision

The most important lesson is not any single technology. It is understanding how data moves between services.

Once the request flow is clear, debugging becomes much easier. A problem that looks like an AI issue may actually be a routing issue. A UI issue may actually be a streaming issue. An API failure may actually be authentication or URL construction.

Understanding the complete request lifecycle makes it much easier to maintain and extend features like Snap & Solve.