# Ai Web Research Agent (`devwithbobby/ai-web-research-agent`) Actor An autonomous agent that researches topics across the web, synthesizes information from multiple sources, and produces comprehensive reports. Perfect for researchers, students, content creators, and analysts who need fast, reliable web research. - **URL**: https://apify.com/devwithbobby/ai-web-research-agent.md - **Developed by:** [Dev with Bobby](https://apify.com/devwithbobby) (community) - **Categories:** AI, Agents, Developer tools - **Stats:** 19 total users, 6 monthly users, 100.0% runs succeeded, 1 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## AI Web Research Agent An **autonomous AI-powered agent** that researches any topic across the web, synthesizes information from multiple sources, and produces comprehensive reports with citations and confidence scores. Stop spending hours on manual research - let AI do the heavy lifting. ### What does AI Web Research Agent do? This Actor is a **true AI research assistant** that goes beyond simple web scraping. Give it a topic or question, and it will: 1. **Generate smart search queries** based on your research topic 2. **Crawl relevant web sources** automatically 3. **Analyze each source with AI** to extract key information 4. **Synthesize findings** into a structured research report 5. **Cite sources** and provide confidence scores for each finding **This is not a scraper - it's a research agent that thinks.** ### Why use AI Web Research Agent? #### Save 10+ hours on every research project Instead of manually searching, reading, and note-taking across dozens of sources, get a comprehensive research report in minutes. #### Perfect for multiple use cases - **Market Research** - Competitive analysis, industry trends, market sizing - **Academic Research** - Literature reviews, topic exploration, source compilation - **Content Creation** - Background research for articles, blogs, whitepapers - **Due Diligence** - Company research, technology assessment, risk analysis - **Trend Analysis** - Track emerging topics, technologies, and market shifts #### What makes this different? | Feature | Basic Scrapers | AI Web Research Agent | |---------|---------------|----------------------| | Understands context | No | Yes | | Generates search queries | No | Yes | | Analyzes relevance | No | Yes (1-10 score) | | Synthesizes findings | No | Yes | | Provides citations | No | Yes | | Identifies key points | No | Yes | | Confidence scoring | No | Yes | #### Platform advantages - **API Access** - Integrate research into your workflows programmatically - **Scheduling** - Set up recurring research tasks for trend monitoring - **Proxy Rotation** - Built-in proxy support prevents blocking - **Cloud Execution** - No need to keep your computer running - **Multiple Formats** - Export as JSON, CSV, Excel, or Markdown ### What data can AI Web Research Agent extract? #### Research Report A comprehensive markdown report with: - Executive summary - Key findings with citations - Detailed analysis - Conclusions and recommendations #### Source Analysis | Field | Description | |-------|-------------| | `title` | Page title | | `url` | Source URL | | `relevanceScore` | AI-determined relevance (1-10) | | `summary` | AI-generated 2-3 sentence summary | | `keyPoints` | Array of key points from the source | | `category` | Source type (primary, supporting, expert opinion, etc.) | #### Key Findings | Field | Description | |-------|-------------| | `finding` | The synthesized finding | | `confidence` | High, Medium, or Low | | `sources` | Which sources support this finding | | `category` | Topic category | ### How to use AI Web Research Agent #### Step 1: Get an OpenAI API Key You'll need an OpenAI API key for the AI analysis. Get one at [platform.openai.com/api-keys](https://platform.openai.com/api-keys) #### Step 2: Configure your research 1. Click **"Try for free"** to open the Actor 2. Enter your **Research Topic** - be specific for better results 3. Paste your **OpenAI API Key** 4. Choose your **Research Depth**: - Quick: 5-10 sources (fast overview) - Standard: 15-25 sources (balanced) - Comprehensive: 30-50 sources (thorough) 5. Select **Output Format**: - Summary: Key findings only - Detailed: Full analysis with sections - Academic: Citations and references #### Step 3: Run and get results Click **Start** and wait for the research to complete. You'll get: - A markdown research report (in Key-Value Store) - Analyzed sources with relevance scores (in Dataset) - Synthesized key findings with confidence levels #### Input example ```json { "researchTopic": "What are the latest trends in artificial intelligence for 2024?", "researchDepth": "standard", "outputFormat": "detailed", "openaiApiKey": "sk-...", "model": "gpt-4o-mini", "maxSources": 20 } ```` #### Output example - Source Analysis ```json { "type": "source", "title": "State of AI Report 2024", "url": "https://example.com/ai-report-2024", "relevanceScore": 9, "summary": "Comprehensive analysis of AI developments in 2024, focusing on LLM advances, multimodal models, and enterprise adoption trends.", "keyPoints": [ "GPT-4 and Claude 3 represent major capability jumps", "Enterprise AI adoption increased 45% year-over-year", "Open-source models closing gap with proprietary ones", "AI regulation becoming major focus globally" ], "category": "primary_source" } ``` #### Output example - Key Finding ```json { "type": "finding", "finding": "Enterprise AI adoption has accelerated significantly in 2024, with 45% year-over-year growth driven primarily by generative AI applications.", "confidence": "high", "sources": [1, 3, 7], "category": "Market Trends" } ``` #### Output example - Research Report (excerpt) ```markdown ## Research Report: AI Trends in 2024 ### Executive Summary This research analyzed 23 sources to identify the key trends shaping artificial intelligence in 2024... ### Key Findings #### 1. Large Language Models Continue to Advance [1][3][7] The release of GPT-4 Turbo and Claude 3 marked significant capability improvements... #### 2. Enterprise Adoption Accelerating [2][5][9] Organizations are moving beyond experimentation to production deployments... ### References [1] State of AI Report 2024 - https://example.com/ai-report [2] Enterprise AI Survey - https://example.com/survey ... ``` ### How much does it cost? AI Web Research Agent uses **pay-per-usage** pricing based on: 1. **Apify compute costs** - For crawling and processing 2. **OpenAI API costs** - For AI analysis (billed directly to your OpenAI account) #### Estimated Apify costs | Research Depth | Sources | Estimated Cost | |---------------|---------|----------------| | Quick | 5-10 | $0.01-0.03 | | Standard | 15-25 | $0.03-0.08 | | Comprehensive | 30-50 | $0.08-0.15 | #### OpenAI API costs (approximate) Using GPT-4o-mini (recommended): - \~$0.01-0.05 per research run **Free tier**: New Apify users get $5 in free credits - enough for 50+ research runs! ### Input parameters | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `researchTopic` | string | Yes | The topic or question to research | | `openaiApiKey` | string | Yes | Your OpenAI API key | | `searchQueries` | array | No | Custom search queries (auto-generated if empty) | | `startUrls` | array | No | Specific URLs to include | | `maxSources` | integer | No | Max pages to analyze (default: 20) | | `researchDepth` | string | No | quick, standard, or comprehensive | | `outputFormat` | string | No | summary, detailed, or academic | | `model` | string | No | gpt-4o-mini, gpt-4o, or gpt-4-turbo | | `includeSourceContent` | boolean | No | Include raw content in output | ### Tips for best results - **Be specific with your topic** - "AI trends in healthcare diagnostics 2024" works better than "AI trends" - **Use the right depth** - Quick for overviews, Comprehensive for thorough research - **Add specific URLs** - If you know authoritative sources, include them in startUrls - **Choose the right model** - GPT-4o-mini is fast and cheap; GPT-4o for complex topics - **Academic format** - Use this for research that needs proper citations ### Integrations Connect AI Web Research Agent to your workflow: - **Google Sheets** - Export research to spreadsheets automatically - **Slack** - Get notifications when research completes - **Zapier/Make** - Trigger workflows based on findings - **Webhooks** - Send reports to your applications - **API** - Programmatically run research and retrieve results ### FAQ #### Is this legal? This Actor only accesses publicly available information. It respects robots.txt and implements rate limiting. Always ensure your use case complies with applicable laws and website Terms of Service. #### Why do I need an OpenAI API key? The AI analysis (relevance scoring, summarization, synthesis) requires OpenAI's GPT models. You're billed directly by OpenAI for this usage, keeping costs transparent. #### Can I use other AI models? Currently, the Actor supports OpenAI models (GPT-4o-mini, GPT-4o, GPT-4-turbo). Support for other providers (Anthropic, etc.) may be added in future versions. #### How accurate are the confidence scores? Confidence scores reflect how well-supported a finding is across sources. "High" means multiple reliable sources agree; "Low" means limited or conflicting evidence. #### What if a source is behind a paywall? The crawler uses HTTP requests and can't bypass paywalls. Paywalled content will be skipped or have limited analysis. #### How do I get support? - Check the **Issues** tab for known problems - Open a new issue for bugs or feature requests - Contact for custom enterprise solutions ### Related Actors - [Website Content Crawler](https://apify.com/apify/website-content-crawler) - Extract full text from websites - [Google Search Results Scraper](https://apify.com/apify/google-search-scraper) - Get Google search results - [RAG Web Browser](https://apify.com/apify/rag-web-browser) - AI-powered web browsing ### Changelog - **v0.0** - Initial release with AI-powered research, synthesis, and reporting *** **Built with [Crawlee](https://crawlee.dev) and [OpenAI](https://openai.com)** *Transform hours of research into minutes. Let AI be your research assistant.* # Actor input Schema ## `researchTopic` (type: `string`): The topic or question you want to research. Be as specific as possible for better results. ## `searchQueries` (type: `array`): Additional search queries to explore. If empty, the agent will generate queries based on the research topic. ## `startUrls` (type: `array`): Specific URLs to include in the research. Useful for targeting specific sources. ## `maxSources` (type: `integer`): Maximum number of web pages to analyze for the research. ## `researchDepth` (type: `string`): How thorough should the research be? ## `outputFormat` (type: `string`): Format of the final research report. ## `includeSourceContent` (type: `boolean`): Include extracted content from each source in the dataset. ## `openaiApiKey` (type: `string`): Your OpenAI API key for AI-powered synthesis. Leave empty to use the built-in key, or provide your own for higher limits. ## `model` (type: `string`): Which OpenAI model to use for synthesis. ## Actor input object example ```json { "researchTopic": "What are the latest trends in artificial intelligence for 2024?", "searchQueries": [], "startUrls": [], "maxSources": 20, "researchDepth": "standard", "outputFormat": "detailed", "includeSourceContent": true, "model": "gpt-4o-mini" } ``` # Actor output Schema ## `report` (type: `string`): No description ## `sources` (type: `string`): No description ## `findings` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "researchTopic": "What are the latest trends in artificial intelligence for 2024?" }; // Run the Actor and wait for it to finish const run = await client.actor("devwithbobby/ai-web-research-agent").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "researchTopic": "What are the latest trends in artificial intelligence for 2024?" } # Run the Actor and wait for it to finish run = client.actor("devwithbobby/ai-web-research-agent").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "researchTopic": "What are the latest trends in artificial intelligence for 2024?" }' | apify call devwithbobby/ai-web-research-agent --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=devwithbobby/ai-web-research-agent", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Ai Web Research Agent", "description": "An autonomous agent that researches topics across the web, synthesizes information from multiple sources, and produces comprehensive reports. Perfect for researchers, students, content creators, and analysts who need fast, reliable web research.", "version": "0.0", "x-build-id": "a773QcxscNSaxRTar" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/devwithbobby~ai-web-research-agent/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-devwithbobby-ai-web-research-agent", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/devwithbobby~ai-web-research-agent/runs": { "post": { "operationId": "runs-sync-devwithbobby-ai-web-research-agent", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/devwithbobby~ai-web-research-agent/run-sync": { "post": { "operationId": "run-sync-devwithbobby-ai-web-research-agent", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "researchTopic" ], "properties": { "researchTopic": { "title": "Research Topic", "type": "string", "description": "The topic or question you want to research. Be as specific as possible for better results." }, "searchQueries": { "title": "Search Queries (Optional)", "type": "array", "description": "Additional search queries to explore. If empty, the agent will generate queries based on the research topic.", "default": [], "items": { "type": "string" } }, "startUrls": { "title": "Start URLs (Optional)", "type": "array", "description": "Specific URLs to include in the research. Useful for targeting specific sources.", "default": [], "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "maxSources": { "title": "Max Sources to Analyze", "minimum": 5, "maximum": 100, "type": "integer", "description": "Maximum number of web pages to analyze for the research.", "default": 20 }, "researchDepth": { "title": "Research Depth", "enum": [ "quick", "standard", "comprehensive" ], "type": "string", "description": "How thorough should the research be?", "default": "standard" }, "outputFormat": { "title": "Output Format", "enum": [ "summary", "detailed", "academic" ], "type": "string", "description": "Format of the final research report.", "default": "detailed" }, "includeSourceContent": { "title": "Include Source Content", "type": "boolean", "description": "Include extracted content from each source in the dataset.", "default": true }, "openaiApiKey": { "title": "OpenAI API Key (Optional)", "type": "string", "description": "Your OpenAI API key for AI-powered synthesis. Leave empty to use the built-in key, or provide your own for higher limits." }, "model": { "title": "AI Model", "enum": [ "gpt-4o-mini", "gpt-4o", "gpt-4-turbo" ], "type": "string", "description": "Which OpenAI model to use for synthesis.", "default": "gpt-4o-mini" } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```