🐝 BeHive

Open-source research engine that extracts structured knowledge from any topic.
Feed it a question. Get back scored claims, entity graphs, and a synthesized report — not paragraphs of slop.

Quick Start • Use with AI Assistants • Drone Arsenal • Benchmarks • Architecture • API • Self-Hosting • Integrations

The Problem

You ask Claude to research a topic. It gives you a confident-sounding summary based on training data that's months old. No sources. No structure. No way to verify.

You ask Perplexity. Better — it cites sources. But the output is still unstructured text. You can't query it, cross-reference it, or build on it.

BeHive is different. It produces machine-readable intelligence: typed claims with confidence scores, entity relationship graphs, and structured JSON you can pipe into any downstream system.

Your AI assistant → BeHive → Verified, structured, scored knowledge
                              ├── 363 claims (avg quality 0.77)
                              ├── 42 entities with relationships
                              └── Synthesized report with citations

Quick Start

pip install behive

# Set your LLM API key (you use YOUR OWN subscription — BeHive costs nothing)
export ANTHROPIC_API_KEY=your-key  # or OPENAI_API_KEY, or AWS creds for Bedrock

# PostgreSQL required for storage (or use Docker below)
export BEHIVE_DB_URL=postgresql://user:***@localhost:5432/behive

# Start the server
behive serve

Fastest path — Docker Compose (PostgreSQL included):

git clone https://github.com/qa10devteam/behive && cd behive
echo "ANTHROPIC_API_KEY=your-key" > .env
docker compose up -d
# API at http://localhost:8091

Full install (stealth drones, content extraction, NLP processing):

pip install "behive[all]"

⚠️ GPU/CUDA note: behive[all] does NOT include GPU dependencies. If you need vector embeddings (Qdrant), install separately: pip install "behive[qdrant]" — this pulls PyTorch + sentence-transformers (~4GB with CUDA). For CPU-only machines, install torch CPU-only first: pip install torch --index-url https://download.pytorch.org/whl/cpu

Or pick what you need:

pip install "behive[stealth]"   # curl_cffi, primp, nodriver, patchright
pip install "behive[harvest]"   # trafilatura, newspaper4k, PyMuPDF, crawl4ai
pip install "behive[process]"   # rapidfuzz, spacy, litellm, tiktoken
pip install "behive[mcp,api]"   # MCP server + REST API

That's it. BeHive is now running:

• API → http://localhost:8091 (REST endpoints)
• MCP → http://localhost:8090/mcp (for AI assistants)
• Docs → http://localhost:8091/docs (Swagger UI)

🧠 Model Routing — Use Cheap Models to Collect, Smart Models to Analyze

BeHive doesn't force you into one model. You choose what runs each pipeline stage:

Quick Setup (one command)

# Apply a preset
behive config --preset balanced   # Haiku collects, Sonnet synthesizes (~$1.50/mission)
behive config --preset budget     # Haiku everywhere (~$0.30/mission)
behive config --preset quality    # Sonnet everywhere (~$4.00/mission)
behive config --preset local      # Your own LLM server ($0.00/mission)

Interactive Setup

behive config --quick    # Pick one model for everything
behive config --full     # Choose model per stage (interactive)

Fine-grained Control

# Set a single stage
behive config --stage synth --model claude-opus
behive config --stage scout --model ollama/deepseek-r1

# Check current config
behive config --show

Environment Variable Override (Docker/CI)

export BEHIVE_MODEL_SCOUT=ollama/llama3.1
export BEHIVE_MODEL_SYNTH=anthropic/claude-sonnet-4-20250514
behive serve

Priority: BEHIVE_MODEL_{STAGE} > BEHIVE_MODEL > config.yaml > defaults

Available Model Presets

Or pass any litellm-compatible model string directly.

🔌 Setup with Claude Desktop (30 seconds)

You bring your Claude subscription. BeHive adds research superpowers. No extra cost from us.

Step 1: Install and start BeHive:

pip install behive
export ANTHROPIC_API_KEY=*** # your own key
behive serve

Step 2: Open Claude Desktop → Settings → Developer → Edit Config → paste:

{
  "mcpServers": {
    "behive": {
      "url": "http://localhost:8090/mcp",
      "transport": "streamable-http"
    }
  }
}

Step 3: Restart Claude Desktop. Done. Now ask:

"Research the EU AI Act enforcement timeline and penalties"

Claude will call BeHive automatically, search across multiple sources, and return scored claims instead of guessing from training data.

You ask Claude a question
    ↓
Claude calls BeHive MCP tool "research_topic"
    ↓
BeHive scouts 70+ APIs, fetches 1000+ URLs via stealth drones
    ↓
Your LLM key extracts claims (Claude Haiku = ~$0.50 per mission)
    ↓
BeHive scores, deduplicates, builds knowledge graph
    ↓
Returns structured report to Claude
    ↓
Claude presents findings with confidence scores and source links

🔌 Setup with ChatGPT (Custom GPT)

Step 1: Start BeHive on a server with a public URL (or use tunneling):

pip install behive
export OPENAI_API_KEY=*** # your own key
behive serve --host 0.0.0.0

# Expose with a tunnel (for testing):
# npx cloudflared tunnel --url http://localhost:8091

Step 2: Create a Custom GPT at chat.openai.com/gpts/editor:

• Name: "Deep Researcher (BeHive)"
• Instructions: "You are a research analyst. Use the BeHive actions to research topics. Always cite claim confidence scores."
• Actions → Import URL: paste your server URL + /openapi.json

Or manually add this schema:

openapi: 3.1.0
info:
  title: BeHive Research API
  version: 0.3.0
servers:
  - url: https://*** paths:
  /research:
    post:
      operationId: startResearch
      summary: Start a deep research mission
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required: [query]
              properties:
                query:
                  type: string
                  description: Research topic or question
                depth:
                  type: integer
                  default: 3
                  description: 1=quick, 3=standard, 5=deep
      responses:
        '200':
          description: Mission started successfully
  /research/{mission_id}:
    get:
      operationId: getResearchResults
      summary: Get completed research with scored claims
      parameters:
        - name: mission_id
          in: path
          required: true
          schema:
            type: string
      responses:
        '200':
          description: Research results with claims and report
  /claims/search:
    get:
      operationId: searchKnowledge
      summary: Search across all previously researched knowledge
      parameters:
        - name: q
          in: query
          required: true
          schema:
            type: string
        - name: limit
          in: query
          schema:
            type: integer
            default: 20
      responses:
        '200':
          description: Matching claims with scores

Step 3: Use your Custom GPT. Ask: "Research quantum computing breakthroughs 2026"

🔌 Setup with Cursor / Windsurf / Any MCP Client

Any editor or tool supporting MCP works identically to Claude Desktop:

{
  "mcpServers": {
    "behive": {
      "url": "http://localhost:8090/mcp",
      "transport": "streamable-http"
    }
  }
}

Available MCP tools:

🔌 Setup with Hermes Agent / OpenClaw

Hermes Agent (automatic — skill already published):

# BeHive skill auto-loads when you ask Hermes to research anything
# Just ensure behive serve is running on the same machine
behive serve

OpenClaw:

# Install from integrations directory
cp integrations/openclaw/SKILL.md ~/.openclaw/skills/behive-research.md

Drone Arsenal

BeHive doesn't just search the web. It deploys stealth drones — multi-layered fetch agents that break through anti-bot defenses, paywalls, and rate limits.

8-Layer Evasion Stack

Every URL goes through an escalation cascade. If Layer 1 gets blocked, Layer 2 fires. All the way to Layer 8.

Layer 1 │ DIRECT          — aiohttp + full Chrome 131 headers
Layer 2 │ UA ROTATION     — 10 browser fingerprints (Chrome/Firefox/Safari/Edge)
Layer 3 │ curl_cffi       — TLS impersonation (JA3/JA4 fingerprint matching)
Layer 4 │ primp           — Rust-native TLS, newer fingerprints than curl_cffi
Layer 5 │ nodriver        — Headless Chrome via CDP, passes Cloudflare Bot Management
Layer 6 │ patchright      — Stealth Playwright (no Runtime.enable/Console.enable leak)
Layer 7 │ Jina relay      — r.jina.ai proxy (paywall + captcha bypass)
Layer 8 │ Archives        — Wayback Machine + archive.org fallback

What they bypass

Parallel fetch architecture

                    ┌─── HEAD sweep (974+ URLs, async semaphore) ───┐
                    │                                                │
                    ▼                                                ▼
          ┌─────────────────┐                            ┌────────────────┐
          │  Resource Router │                            │  Domain Recon  │
          │  (8 resource     │                            │  (tier scoring │
          │   types detected)│                            │   reputation)  │
          └────────┬────────┘                            └───────┬────────┘
                   │                                              │
        ┌──────────┼──────────┬──────────┐                       │
        ▼          ▼          ▼          ▼                       ▼
   api_bee    pdf_drone   std_drone  heavy_drone         domain_score
   (70 APIs)  (VLM parse) (Layer 1-8) (patchright)       (0.0 - 1.0)

Routing decisions per resource type:

• api_endpoint → Direct API bee (structured JSON, no parsing needed)
• pdf → PDF drone (Vision LLM extraction)
• static_html → Standard drone (Layer 1-4 usually sufficient)
• spa → Heavy drone (Layer 5-6, needs JS execution)
• paywall → Jina relay or archive fallback
• rss_feed → RSS bee (structured, fast)
• database_portal → Dedicated connector (custom scraping logic)

70+ API Sources

Scout bees don't just Google. They query specialized APIs across 37 categories:

Total: 70 APIs, 125 endpoints — each checked per-mission based on topic relevance.

Benchmarks

Real results. No cherry-picking. Scale 30 (standard depth).

Hardware: EC2 g6.24xlarge — 4× NVIDIA L4 (92 GB VRAM), 96 vCPU, 384 GB RAM
Models: Bedrock Claude Haiku (bulk extraction) + Sonnet (enrichment), SGLang/Qwen on local GPUs

Quality score meaning:

• 0.90+ — Exceptional: specific numbers, dates, sources, fully verifiable
• 0.82+ — Excellent: multi-source corroboration, publication-ready (top 25% of missions)
• 0.75+ — Good: useful intelligence with some specifics
• 0.65+ — Acceptable: general facts, entered into DB
• <0.55 — Rejected: too vague, not stored

Honest scoring, no tricks. No sigmoid rescaling, no artificial inflation. The score is a weighted average of specificity, information density, uniqueness, verifiability, and structure.

Architecture

                         ┌──────────────────────────────────┐
                         │         BeHive Pipeline           │
                         └──────────────────────────────────┘
                                        │
        ┌───────────┬───────────┬───────┴───────┬───────────┬───────────┐
        ▼           ▼           ▼               ▼           ▼           ▼
   ┌─────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐ ┌─────────┐ ┌────────┐
   │  SCOUT  │ │ HARVEST │ │ PROCESS  │ │   V4     │ │  SYNTH  │ │ GRAPH  │
   │         │ │         │ │          │ │          │ │         │ │        │
   │ Queen   │ │ Parallel│ │ BeeHive  │ │ Haiku    │ │ Claude  │ │ Neo4j  │
   │ plans   │ │ HTTP    │ │ fast     │ │ extract  │ │ report  │ │ entity │
   │ 5 axes  │ │ 1000+   │ │ extract  │ │ + Sonnet │ │ + cite  │ │ fuse   │
   │ × N     │ │ URLs    │ │ + score  │ │ enrich   │ │         │ │        │
   └─────────┘ └─────────┘ └──────────┘ └──────────┘ └─────────┘ └────────┘
       │              │            │            │            │          │
       │              │            ▼            │            │          │
       │              │    ┌──────────────┐     │            │          │
       │              │    │ Quality Gate │     │            │          │
       │              │    │  conf ≥ 0.55 │     │            │          │
       │              │    │  dedup 0.60  │     │            │          │
       │              │    └──────────────┘     │            │          │
       │              │            │            │            │          │
       └──────────────┴────────────┴────────────┴────────────┴──────────┘
                                        │
                              ┌─────────┴─────────┐
                              │   PostgreSQL       │
                              │   Claims + KG      │
                              │   25K+ records     │
                              └───────────────────┘

What makes it different from GPT-Researcher:

1. Dual-model extraction — Fast model (Haiku) for bulk extraction, powerful model (Sonnet) for enriching thin claims. Not just "summarize this page."
2. Quality scoring — Every claim gets a 0.0-1.0 score. Below threshold = rejected. No filler.
3. Knowledge graph — Entities and relationships persist across missions. Research compounds.
4. 70+ API sources — Not just web search. SEC filings, arXiv, patent databases, government APIs.
5. Deduplication — Jaccard 0.60 threshold prevents the same fact from different sources inflating counts.

API Reference

BeHive exposes a REST API (port 8091) and MCP server (port 8090).

Start Research

curl -X POST http://localhost:8091/research \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SpaceX Starship launch cadence 2026",
    "depth": 3,
    "scale": 30
  }'
# → {"job_id": "hive_1785227949_815112", "status": "started"}

Stream Progress (SSE)

curl -N http://localhost:8091/research/hive_1785227949_815112/events

event: start
data: {"topic": "SpaceX Starship...", "status": "scout"}

event: phase
data: {"phase": "process", "event": "started"}

event: claims
data: {"count": 142, "avg_quality": 0.77, "above_082": 23, "new_since_last": 18}

event: done
data: {"total_claims": 363, "avg_quality": 0.77, "sources": 64}

Get Report

curl http://localhost:8091/research/hive_1785227949_815112/report
# → {"synthesis": "## SpaceX Starship...", "claims_count": 363, ...}

Search Knowledge

# Full-text search across all missions
curl "http://localhost:8091/search?query=NVIDIA+revenue&limit=20"

# Entity intelligence
curl http://localhost:8091/intelligence/entity/NVIDIA

# Network graph (2-hop neighborhood)  
curl "http://localhost:8091/intelligence/network/OpenAI?depth=2"

All Endpoints

Full Swagger docs: http://localhost:8091/docs

MCP Integration

BeHive implements the Model Context Protocol — the emerging standard for AI tool connectivity.

{
  "mcpServers": {
    "behive": {
      "url": "http://localhost:8090/mcp",
      "transport": "streamable-http"
    }
  }
}

Compatible with:

• Claude Desktop / Claude Code
• Cursor IDE
• Windsurf
• n8n (via MCP node)
• Any MCP-compatible client

Tools exposed:

Self-Hosting

Docker (recommended)

git clone https://github.com/qa10devteam/behive.git
cd behive
cp .env.example .env     # add your LLM API key
docker compose up -d     # API ready at localhost:8091

Full stack with knowledge graph + vector search:

docker compose --profile full up -d

Manual Setup

pip install behive[all]

# PostgreSQL
createdb hive
behive db init

# Configure
export BEHIVE_DB_URL="postgresql://user:pass@localhost:5432/hive"
export BEHIVE_LLM=bedrock  # or openai, local

# Start services
behive serve              # Starts both REST API (:8091) + MCP (:8090)

Minimum Requirements

How It Works (for humans)

1. You give it a topic. "NVIDIA GPU market 2026"

2. Scout bees plan the research. The Queen decomposes it into 5 axes (market share, financials, products, competition, supply chain). Generates 12-14 search queries per axis. Checks 70+ APIs.

3. Harvest bees collect sources. Parallel HTTP fetches ~1000 URLs. HEAD sweep first (fast), then full content extraction on promising ones. Typically lands 60-90 usable documents.

4. Worker bees extract claims. This is where BeHive shines:

   • Every document gets parsed into atomic, verifiable claims
   • Each claim scored on 5 dimensions (specificity, density, uniqueness, verifiability, structure)
   • Claims below 0.55 quality → rejected
   • Thin claims (missing dates/numbers) → enriched by Sonnet
   • Duplicate claims (Jaccard >0.60) → merged

5. The Queen synthesizes. Claude weaves the verified claims into a structured report with inline citations. No hallucination — every statement maps to a scored claim.

6. Knowledge graph grows. Entities (companies, people, products, amounts) and their relationships are stored in Neo4j. Next research mission on a related topic starts with existing context.

Configuration

Search Backends (priority order)

BeHive tries search backends in priority order and falls through on failure:

No env vars set? DDG is the default. Add any key above to instantly upgrade search quality.

Comparison

Roadmap

• V4 pipeline (BYOK — bring your own LLM key, any provider)
• Quality scoring (avg 0.77, top missions reach 0.82+)
• REST API (14 endpoints)
• MCP Server (Streamable HTTP)
• SSE streaming (real-time progress)
• Knowledge graph (Neo4j)
• 64 API sources (37 free APIs confirmed working, 26 need BYOK keys)
• Browser search (Chromium/Playwright — scrapes Google/Bing, zero API keys)
• pip install behive (PyPI)
• Docker Compose one-liner
• n8n community node (npm)
• Agent skills (Hermes, OpenClaw, Claude Desktop)
• Web UI dashboard
• Multi-tenant API keys
• Webhook callbacks
• Scheduled recurring research
• PDF export with charts

Integrations

BeHive works with every major AI agent platform:

See integrations/ for detailed setup guides.

Contributing

See CONTRIBUTING.md for development setup, code style, and PR guidelines.

git clone https://github.com/qa10devteam/behive.git
cd behive
pip install -e ".[all,dev]"
pytest

License

MIT — use it, fork it, ship it, sell it.

_{Built by QA10 · Structured knowledge, not text soup.}