I Needed One Python Library. I Didn’t Rewrite My Backend.

Ankur Bag

9 min read

·

Aug 19, 2026

--

Listen

Share

Why I separated Avento’s crawler from the rest of its Next.js application instead of forcing two very different workloads into one runtime.

Press enter or click to view image in full size

Avento was already built around Next.js and TypeScript when I decided to upgrade its website-ingestion system.

The product already had its own application logic: authentication, assistants, document ingestion, vector indexing, analytics, widget configuration, and the embedded support experience.

Then I found the crawler I wanted to use.

It was Python-based.

More specifically, I wanted to use Crawl4AI for Avento’s website crawling and extraction layer.

And that immediately created one of those architecture decisions that looks trivial at first:

Do I somehow run Python inside my existing Next.js application, move part of the backend to Python, or deploy crawling separately?

I chose the third option.

Not because “microservices are better.”

Not because I wanted another box in an architecture diagram.

And definitely not because rewriting a working backend sounded fun.

The decision came down to something much simpler:

Crawling behaves very differently from the rest of Avento.

The Problem Started With a Product Upgrade

The earlier version of Avento relied heavily on manually provided knowledge.

A business could upload its documents, Avento would process them, generate embeddings, store them in the vector database, and use retrieval-augmented generation to answer customer questions.

That worked.

But most businesses already have a huge portion of their knowledge sitting on their website:

• product pages
• documentation
• FAQs
• pricing
• policies
• feature pages
• support articles
• blogs

So the next version of Avento needed a simpler workflow:

Give Avento a website URL
          ↓
Discover relevant pages
          ↓
Extract useful content
          ↓
Index the knowledge
          ↓
Use it during customer conversations

That meant I needed something significantly more capable than fetching a page with a normal HTTP request.

I needed actual crawling.

Why Crawl4AI?

For Avento, crawling is not just:

fetch(url)

and then parsing whatever comes back.

Modern websites can involve:

• JavaScript-rendered content
• client-side routing
• nested internal pages
• repeated navigation structures
• redirects
• dynamic sections
• content loaded after the initial HTML
• pages that technically return successfully but contain almost nothing useful

I wanted the crawler to have one responsibility:

Give it a starting URL and return useful, structured content from the relevant parts of that website.

Crawl4AI fit that job well.

There was just one obvious mismatch.

Avento was primarily TypeScript.

Crawl4AI was Python.

Option 1: Rewrite Part of Avento in Python

Technically, I could have moved more of the backend to Python.

But that would have been solving the wrong problem.

Avento already had working systems for:

• authentication
• tenant management
• assistant configuration
• ingestion jobs
• analytics
• vector storage
• conversations
• widget integration
• API communication

None of those systems suddenly became better candidates for Python just because the crawler happened to use it.

Rewriting working product logic would have added migration work, new bugs, and another learning surface without giving users anything new.

The actual requirement was:

I need Python for crawling.

Not:

I need Avento to become a Python application.

That distinction made the first option easy to reject.

Option 2: Make Python Run Inside the Next.js Deployment

The second idea sounds attractive initially.

Keep everything together.

Maybe spawn a Python process somewhere. Install the crawler alongside the application. Let Next.js trigger it directly.

One repository.

One deployment.

One service.

Nice and simple.

Except the workloads are not actually similar.

A typical application API request might:

1. Validate a request.
2. Read from the database.
3. Perform some business logic.
4. Return JSON.

A crawl can look more like:

1. Start or connect to a browser.
2. Open a website.
3. Wait for content.
4. Parse the page.
5. Discover more URLs.
6. Open those pages.
7. Repeat extraction.
8. Handle redirects or failures.
9. Clean the results.
10. Return potentially large amounts of content.

That can involve a browser runtime such as Chromium, higher memory usage, longer execution times, multiple network requests, retries, and significantly more unpredictable execution.

Trying to hide all of that inside a normal application endpoint would technically reduce the number of deployments.

But it would increase the coupling between two very different systems.

That trade didn’t make sense to me.

The Architecture I Chose

Instead, I made the crawler its own service.

The boundary is intentionally simple.

AVENTO
          Next.js / TypeScript
                  │
                  │
        Create ingestion job
                  │
                  ▼
        Python Crawler Service
                  │
              Crawl4AI
                  │
         Discover + Extract
                  │
                  ▼
        Structured page data
                  │
                  ▼
        Avento ingestion layer
                  │
      Clean → Chunk → Embed
                  │
                  ▼
             Vector DB

The important part is what the Python service does not own.

It does not own users.

It does not own assistants.

It does not decide which tenant a vector belongs to.

It does not manage analytics.

It does not generate answers.

It does not become a second backend.

It crawls.

That’s it.

Next.js Owns the Product

Avento’s main application remains responsible for product-level decisions.

For example, when someone asks Avento to ingest a website, the application can determine:

• which user requested it
• which assistant the website belongs to
• whether the request is valid
• what URL should be crawled
• the status of the ingestion job
• where the extracted knowledge should be stored
• which tenant owns the resulting vectors

Then it sends the crawling workload to the Python service.

The crawler has no reason to understand most of that context.

It receives the information it needs to perform the crawl and returns the result.

That separation keeps the product logic where it already belongs.

Python Owns the Crawl

The crawler’s world is much smaller.

Its responsibilities revolve around things such as:

• opening the website
• discovering internal pages
• applying crawl boundaries
• extracting page content
• handling browser rendering
• collecting metadata
• preserving source URLs
• reporting crawl failures

The service can evolve around those requirements without forcing the rest of Avento to change with it.

If I replace the crawling library later, the application should not care much.

If I change how browser instances are managed, the vector retrieval system should not care.

If crawling suddenly needs more memory, the dashboard shouldn’t need to move to larger infrastructure with it.

That’s the kind of boundary I wanted.

One Crawler Service Was Enough

Once I had separated crawling, there was another temptation:

Over-engineer it.

It is very easy to imagine the final version immediately:

API
 ↓
Message Queue
 ↓
Crawler Worker Pool
 ↓
Distributed Browser Cluster
 ↓
Redis
 ↓
Load Balancer
 ↓
Autoscaling
 ↓
Monitoring stack

And at enough scale, some version of that architecture may make sense.

But Avento didn’t need that architecture yet.

So I deployed one dedicated crawler service.

One service could handle the current workload while keeping the operational model understandable.

That’s important when you’re still actively building the product.

Every new infrastructure component has a cost beyond its cloud bill.

You also inherit:

• configuration
• deployment
• monitoring
• debugging
• failure modes
• local development complexity
• documentation
• maintenance

I didn’t want infrastructure designed for hypothetical traffic to slow down development for the traffic I actually had.

So the decision became:

Start with one independently deployable crawler. Scale the boundary later if the workload proves that I need to.

“Single Service” Doesn’t Mean “Single Crawl Forever”

There is an important distinction here.

Deploying one crawler service does not mean the architecture can only process one website at a time forever.

Concurrency is a separate decision.

The crawler can eventually control:

• how many jobs run simultaneously
• how many browser contexts exist
• per-domain limits
• crawl timeouts
• memory thresholds
• retry behavior

If that service becomes saturated, the architecture already gives me a natural place to scale.

I can add:

Jobs
                  │
                  ▼
               Queue
          ┌───────┼───────┐
          ▼       ▼       ▼
       Worker   Worker   Worker

without rewriting Avento’s main application.

That is one of the benefits of creating the boundary early without building the distributed system early.

There Are Tradeoffs

Separating the crawler didn’t make the architecture magically simpler.

It introduced new concerns.

Another deployment

There is now another service that needs to be built, configured, and deployed.

Network communication

Avento and the crawler need to communicate over the network instead of calling a local function.

That means dealing with failure.

The crawler may be unavailable.

The request may time out.

The application may receive an invalid response.

Authentication

The crawling endpoint should not simply be an unrestricted public API that anyone can use to consume browser resources.

Communication between the application and crawler needs to be protected.

Observability

Logs now exist in more than one place.

When an ingestion fails, I need enough information to determine whether the problem happened:

Avento
   ↓
job creation

ornetwork communication
   ↓orcrawler
   ↓orextraction
   ↓orindexing

Those are real costs.

But they are costs associated with a clear system boundary rather than costs created by forcing incompatible workloads together.

For me, that was the better trade.

This Also Changed How I Think About Microservices

“Microservices” can sometimes become the objective instead of the solution.

Break the authentication system into a service.

Break analytics into another.

Create a notification service.

Create an ingestion service.

Create another database.

Then spend the next month making six services communicate so that you can ship a feature that could have been one function.

That wasn’t what I wanted for Avento.

Most of the product still benefits from living together.

The crawler became separate because it had a real operational reason to be separate:

Different runtime
+
Different dependencies
+
Different resource profile
+
Different execution behaviour
+
Clear responsibility

That’s a much stronger justification than:

Microservices scale better.

The Repository Boundary Doesn’t Have to Match the Runtime Boundary

Another useful lesson was that architecture doesn’t necessarily mean creating completely disconnected projects.

You can still organise a product coherently while deploying specific workloads differently.

Conceptually, Avento can remain:

avento/
│
├── web/
│   └── Next.js application
│
├── crawler/
│   └── Python + Crawl4AI
│
└── shared docs/configuration

The code belongs to the same product.

The runtime responsibilities are separated.

That distinction made the architecture feel much less complicated.

I wasn’t creating another product.

I was giving one workload the environment it needed.

What Happens When A Crawl Starts

At a high level, the workflow now looks like this:

User submits website
        ↓
Avento validates request
        ↓
Ingestion job created
        ↓
Crawler receives target URL
        ↓
Website pages discovered
        ↓
Content extracted
        ↓
Structured result returned
        ↓
Avento validates content
        ↓
Content cleaned and chunked
        ↓
Embeddings generated
        ↓
Knowledge indexed
        ↓
Job marked complete

That separation also gives me much cleaner failure states.

A failed crawl does not have to crash the user’s request.

The ingestion job can fail independently and report what went wrong.

Later, the same architecture can support retries or background execution without changing the user-facing flow dramatically.

Why This Decision Matters for Avento

The interesting part isn’t really Python.

I could replace Crawl4AI tomorrow with something written in another language and the architectural principle would remain the same.

The decision was about isolating a workload whose operational requirements differ from the application around it.

Avento’s application needs to be responsive and reliable for users interacting with:

• dashboards
• assistants
• conversations
• analytics
• configurations

The crawler needs to be good at:

• opening websites
• running browsers
• waiting
• exploring
• extracting
• occasionally failing in unpredictable ways

Those systems can communicate.

They don’t have to live inside the same process.

Could I Have Made It Work in One Deployment?

Probably.

That’s not really the question.

A surprising number of architecture decisions are not about whether something is technically possible.

They are about whether the resulting system will still make sense six months later.

I could force Python into the application deployment.

I could rewrite everything in Python.

I could build a distributed crawler cluster immediately.

All three are technically valid approaches.

But the simplest architecture for Avento today was:

Next.js for the product.

Python for the crawl.

And a small API boundary between them.

Build for the System You Have

One thing I’m trying to get better at while building Avento is avoiding architecture for imaginary scale.

It’s easy to design for:

“What if 100,000 websites get submitted simultaneously?”

It is harder to ask:

“What does the product actually need right now, and can the architecture evolve when that changes?”

Right now, one crawler deployment gives me:

• clean runtime isolation
• independent scaling
• simpler debugging
• freedom to use Crawl4AI properly
• no unnecessary rewrite of the existing application
• a clear path to queues and workers later

That’s enough.

If crawl volume eventually becomes a bottleneck, that’s a good problem to have.

And importantly, the architecture already gives me somewhere to solve it.

The Takeaway

I started this decision with what looked like a language problem:

My application is TypeScript. My crawler is Python. What do I do?

But language compatibility was not really the interesting part.

The better question was:

Does this workload have the same operational characteristics as the rest of my application?

For crawling, the answer was no.

So I stopped trying to make everything live in the same runtime.

Avento continues to own the product logic.

Crawl4AI gets the environment it needs to crawl.

And instead of rewriting a backend because I needed one Python library, I gave that library exactly one job.