-
Notifications
You must be signed in to change notification settings - Fork 12
Pay per crawl Walkthrough
A copy/paste recipe for taking a real site from "I have no idea who's crawling me" to "AI agents pay me per request" in three deploys. Total elapsed time: ~30 minutes; total written code: ~5 lines.
This is the safe rollout. Each phase is reversible by changing one flag and redeploying.
- Phase 1 (10 min): Observe. Deploy in observe mode - no enforcement, no risk. Watch the dashboard for 24h to see what's actually crawling you.
-
Phase 2 (10 min): Charge known bots. Flip to
mode: "bots". Real AI crawlers get 402. Humans and search engines pass. -
Phase 3 (10 min): Charge everyone non-human. Flip to
mode: "strict"(or"all", which charges browsers too) once your dashboard says it's safe.
observeandmodeare different options:observeis a boolean,modeisbots/all/strict.mode: "observe"is not a thing and would silently leave you on the charging default.
- A Node 18+ runtime in front of your site (Express, or a Cloudflare Worker, or a Next.js middleware - all supported).
- A wallet address to receive USDC on Base (or Solana, Polygon, Arbitrum, Monad, Celo, Avalanche, Sei, Optimism, Stellar, Algorand). (Or skip it entirely and accept proof-of-work only - no wallet needed.)
- ~30 minutes.
Install the gate and put it in front of your routes - observe mode means nothing gets blocked. Every request is classified (human / bot / paid / would-charge) and counted, so you can see who's crawling you before you ever touch enforcement.
npm install agent402-tollbooth// app.js
import express from "express";
import { createTollbooth } from "agent402-tollbooth";
import { timingSafeEqual } from "node:crypto";
const app = express();
const gate = createTollbooth({
payTo: "0xYourWalletHere", // future-proof; not used yet
price: "$0.002",
observe: true, // <- no 402s sent yet
});
// The middleware does NOT register any route. Serve your own stats endpoint
// from gate.stats() (sync, in-process) or gate.snapshot() (async, durable
// sink), and gate it yourself.
const ADMIN_TOKEN = process.env.TOLLBOOTH_ADMIN_TOKEN || "";
const presented = (req) => {
const auth = req.headers.authorization;
if (typeof auth === "string" && auth.startsWith("Bearer ")) return auth.slice(7);
return typeof req.headers["x-admin-token"] === "string" ? req.headers["x-admin-token"] : "";
};
const authed = (req) => {
const got = presented(req);
if (!ADMIN_TOKEN || got.length !== ADMIN_TOKEN.length) return false;
return timingSafeEqual(Buffer.from(got), Buffer.from(ADMIN_TOKEN));
};
app.get("/__stats", (req, res) =>
authed(req) ? res.json(gate.stats()) : res.status(401).send("Unauthorized"));
app.use(gate);
app.use(yourExistingRoutes);
app.listen(3000);TOLLBOOTH_ADMIN_TOKEN=$(node -e "console.log(require('crypto').randomBytes(24).toString('hex'))") \
node app.jsThen read your counters with the token in a header (never a query string, which lands in access logs):
curl -H "Authorization: Bearer $TOLLBOOTH_ADMIN_TOKEN" http://localhost:3000/__statsIf instead you run the package as a reverse proxy (TOLLBOOTH_UPSTREAM=https://your-site.com npx agent402-tollbooth), it mounts the dashboard for you at /__tollbooth with JSON at /__tollbooth/stats, gated by TOLLBOOTH_ADMIN_TOKEN (or TOLLBOOTH_STATS_TOKEN for the JSON only) using the same Authorization: Bearer / X-Admin-Token headers. Either way you get:
- Total requests by classification (human / would-charge / bot-paid / errors)
- Top user-agents seen
- Top paths hit
- 24h rolling counters
Wait 24 hours. Real traffic is louder than any synthetic test. You're looking for:
- Is there any AI-bot traffic? (If your dashboard says zero would-charge after a day on real traffic, pay-per-crawl isn't worth deploying yet.)
- Are humans getting misclassified as bots? (Should be near zero; investigate before flipping.)
- Are there UAs you didn't expect - internal tools, vendor probes, monitoring bots - that need to be allow-listed?
Once Phase 1 shows realistic-looking numbers, flip one flag. The 28 user-agents in AI_BOTS get 402: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, CCBot, Bytespider, Google-Extended, Amazonbot, cohere-ai, Meta-ExternalAgent, Meta-ExternalFetcher, Applebot-Extended, Diffbot, Omgilibot, ImagesiftBot, YouBot, Timpibot, DuckAssistBot, PetalBot, FriendlyCrawler, AI2Bot, plus the generic scraper signatures Scrapy and python-requests. The live list is tollbooth/bots.js (AI_BOTS); add your own with botUserAgents: [...]. Everyone else passes.
app.use(createTollbooth({
payTo: "0xYourWalletHere",
price: "$0.002",
// observe: true, // <- delete this line
mode: "bots", // <- enforce against the AI-bot list
}));Note the shape: observe is a boolean, and mode is one of bots / all / strict. They are separate options. There is no mode: "observe". That string is not a valid mode, so it falls through to the bots default and you would start charging while believing you were only counting.
Redeploy. Now:
- Humans visit free.
- Search-engine crawlers (Googlebot etc.) pass.
- AI crawlers see a 402 whose body is exactly this shape:
Two things to code against rather than assume: the gate emits no
{ "error": "Payment Required", "message": "This resource charges automated / AI clients per request. …", "accepts": [{ "scheme": "exact", "network": "base", "maxAmountRequired": "$0.002", "asset": "USDC", "payTo": "0xYourWallet", "resource": "https://yoursite.com/article" }], "proofOfWork": { "algorithm": "sha256", "challenge": "…", "difficulty": 18, "expires": 1750000000000, "token": "…", "rule": "Find an integer nonce so sha256(\"<challenge>:\" + nonce) has >= 18 leading zero bits, then resend the request with header X-Pow-Solution: <token>:<nonce>" } }x402Versionfield, andmaxAmountRequired/assetare echoed verbatim from your config (thepricestring and theassetsymbol), not converted to base units or a token address.acceptsis an empty array when nopayTois set, leaving proof-of-work as the only rail. - An agent that wants the page either signs an x402 USDC transaction (any standard x402 client does this -
agent402-client,@x402/fetch, AWS Bedrock AgentCore Payments, …), or solves the proof-of-work for free.
Verify it's working:
# Human - should be 200
curl -A "Mozilla/5.0" https://yoursite.com/article
# AI bot - should be 402
curl -A "ClaudeBot/1.0" https://yoursite.com/article
# Watch real settlement
curl -H "Authorization: Bearer $TOLLBOOTH_ADMIN_TOKEN" https://yoursite.com/__stats
# the x402Paid / powSolved counters should start incrementing within hoursUSDC settles to your payTo wallet directly on Base via the standard x402 facilitator - no Stripe, no Merchant-of-Record, no holding period. You can verify any payment on Basescan.
Leave it here for a week before considering Phase 3. The bot list catches the vast majority of revenue; charging everything is mostly upside-on-the-margins and downside-on-edge-cases.
Once Phase 2 has been stable, you can go stricter. Two modes available:
app.use(createTollbooth({
payTo: "0xYourWalletHere",
price: "$0.002",
mode: "strict", // <- charge anything that isn't a real-browser request
// mode: "all", // <- charge EVERY client, browsers included
}));Read these two carefully, because the names are less intuitive than they look. From the code (shouldCharge in tollbooth/index.js):
-
"strict"charges anything that is not a real-browser-shaped request:return !looksHuman(req), wherelooksHumanrequires both aMozilla/5.0user-agent and atext/htmlAcceptheader. So it catches undeclared bots, headless scrapers, and empty UAs, while a normal browser visit still passes free. This is the stricter-than-botsstep most content sites want. -
"all"charges every client, browsers included:return true, subject only to afree()predicate you supply. It is a paywall, not a bot filter. Useful for high-value APIs and data feeds; almost never the right call for content sites.
An explicit charge(req) / free(req) predicate wins over whichever mode is set, so client-specific allowlists do not require a mode change.
Backstop: in either mode, adaptive proof-of-work means the page is still reachable for free; the cost is just CPU time. So you're not actually locking anyone out - you're just making cheap bulk scraping economically unattractive.
The same gate runs at the edge with no Node, no servers:
-
Cloudflare Workers - see
tollbooth/deploy/cloudflare/. Onewrangler deploy, KV namespace for durable stats and replay protection. -
Next.js middleware - see
tollbooth/deploy/nextjs/. One file inmiddleware.ts. -
Docker reverse proxy - see
tollbooth/deploy/docker/. Wrap any backend regardless of language.
All three share the same Web-Crypto core and the same observe → bots → all/strict flow.
| Symptom | Likely cause | Fix |
|---|---|---|
| Dashboard says 0 would-charge after 24h on real traffic | UA matcher misses your traffic | Add custom UAs via botUserAgents: [...]
|
| Browsers being charged |
mode: "all" charges every client. strict is the one that lets real browsers through |
Use mode: "strict", or roll back to mode: "bots"
|
You set mode: "observe" and it charged anyway |
Not a valid mode, so it fell through to the bots default |
Use the separate observe: true boolean |
| 402 with no PoW or USDC option | Missing payTo and pow: false
|
Set one or both - at least one rail must be on |
| Dashboard/stats returns 401 | Token missing or wrong, or sent as ?token=
|
Send Authorization: Bearer <token> or X-Admin-Token: <token>; the query-string form is not accepted |
| Worker complains about KV | KV binding not wired | See Cloudflare README |
Hard rollback to free in 5 seconds: set observe: true (or TOLLBOOTH_OBSERVE=true), or remove the app.use(...) line entirely, and redeploy. No data is lost, and the counters keep going.
- Pay-per-crawl - full reference (modes, dashboards, deploy templates)
- AWS Bedrock AgentCore - the buy side: agents paying tollbooths over x402
-
tollbooth/demo.js-node demo.jsnarrated end-to-end demo -
examples/agentcore-tollbooth/- Strands agent paying a tollbooth
agent402.tools · synced from wiki/ in the main repo — edit there, not here.
Using it (for agents / buyers)
- Getting Started
- Paying with x402
- Robinhood Chain (USDG)
- Paying with Compute
- MCP Connector
- Adapters
- AWS Bedrock AgentCore
- Tool Catalog
- Skill Packs
- x402 Index and Router
- x402 Leaderboard
- LLM Gateway (OpenAI /v1)
- LLM Proxy Gateway
- Image Generation Gateway
- Code Execution Sandbox
- Text-to-Speech
- Speech-to-Text
- Text Embeddings
- Payments and x402
- Memory and Coordination
Tollbooth (for site owners)
- Pay-per-crawl — what it is, install, modes
- Pay-per-crawl Walkthrough — 5-min hands-on
- Tollbooth for Agencies — many-site playbook
- Try Tollbooth Cloud (managed)
Website & Developer
- Quickstart — first call in 60 seconds
- Playground — try tools in your browser
- SDK REPL — live code editor
- API Explorer — browse OpenAPI
- Adapter Docs — per-framework guides
- Workflows — chaining patterns
- Blog · Changelog
Under the hood