# Semrush Full SEO Scraper - Traffic, Authority, Backlinks (`pro100chok/semrush-scraper`) Actor All-in-one Semrush & Moz scraper โ€” no login required. Domain authority, traffic analytics, backlinks, AI traffic (ChatGPT/Gemini/Claude), competitors, top 100 trending websites. 80+ fields per domain, 10 parallel workers, unique IP per request. Two modes: Domain Analysis & Top Websites Ranking. - **URL**: https://apify.com/pro100chok/semrush-scraper.md - **Developed by:** [Raven](https://apify.com/pro100chok) (community) - **Categories:** SEO tools, Automation, Developer tools - **Stats:** 149 total users, 86 monthly users, 98.8% runs succeeded, 7 bookmarks - **User rating**: 5.00 out of 5 stars ## Pricing from $2.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Semrush Scraper โ€” Domain Overview, Keyword Research, AI Search, Backlinks, Competitors & Top Websites | ๐Ÿฆ… More from Raven โ–บ | [Ahrefs SEO Tools](https://apify.com/pro100chok/ahrefs-seo-tools) | [Semrush Scraper](https://apify.com/pro100chok/semrush-scraper) | [Similarweb Scraper](https://apify.com/pro100chok/similarweb-scraper) | |---|---|---|---| This scraper turns Semrush into a **plug-in SEO data API**. Pick a **mode** and go: - ๐ŸŒ **Domain Overview** โ€” drop in a domain, get the **full Domain Overview dashboard** as one structured JSON record: Authority Score, organic & paid keywords with CPC and intent, top organic + paid competitors, AI Search visibility across **ChatGPT, AI Overview, AI Mode, Gemini**, the full backlinks profile with anchor breakdown, daily and monthly traffic trends, and an optional Moz Domain Analysis snapshot. - ๐Ÿ”‘ **Keyword Research** โ€” drop in a keyword, get **volume, CPC, Keyword Difficulty, competition, search intent, a 12-month trend, volume across every country, related keywords, question keywords, topic clusters, ad copies and the live SERP**. - ๐Ÿ” **SEO Audit (on-page)** โ€” drop in a page URL, get a **full on-page audit** (title, meta, H1โ€“H6, internal/external links, images & missing alt, canonical, robots/sitemap, Open Graph, hreflang) **plus Semrush organic context** (authority, traffic, top keywords, competitors). - ๐Ÿ† **Top Websites Ranking** โ€” the top 100 trending websites for any of 90+ countries ร— 80+ industries. **Everything that's behind the Semrush login โ€” without a Semrush subscription or API key.** --- ### Key features - **Four modes, one Actor** โ€” ๐ŸŒ Domain Overview, ๐Ÿ”‘ Keyword Research, ๐Ÿ” SEO Audit, ๐Ÿ† Top Websites โ€” switch with a single `mode` selector - **Keyword Research in one record** โ€” volume, CPC, KD, competition, decoded intent, 12-month trend, per-country volume, related + question keywords, clusters, ad copies, and the full SERP - **On-page SEO Audit** โ€” title, meta, H1โ€“H6, link & image checks, canonical, robots/sitemap, Open Graph & hreflang, enriched with Semrush organic keywords and competitors - **Full Domain Overview in one record** โ€” every plate from the Semrush UI, flattened into one JSON row per domain - **Authenticated Semrush data** โ€” no degraded "public preview" numbers; you get what a paying seat sees - **AI Search visibility** โ€” per-LLM mentions and cited pages across **ChatGPT / AI Overview / AI Mode / Gemini**, plus top co-cited domains - **Top 50 organic keywords** โ€” position, volume, CPC, KD, traffic, traffic share %, decoded intent, SERP features, landing URL - **Top 50 paid ads with full ad copy** โ€” title, description, visible URL, position, volume, CPC - **Organic + paid competitors** โ€” top 20 each plus total counts, with shared keywords and traffic - **Multi-year daily backlink and traffic trends** โ€” global and per-country, branded vs non-branded split - **Optional Moz Domain Analysis** โ€” DA, Spam Score, top pages, ranking distribution, 60-day linking-domain trend - **Top Websites Ranking mode** โ€” top 100 trending websites by 90+ countries ร— 80+ industries - **Up to 100 domains per run, parallel** โ€” configurable concurrency (default 5, max 50) - **Flat KPI columns up front** โ€” render as plain table columns in the Apify dataset view; no `dot.notation` paths needed - **Export-ready** โ€” CSV / JSON / Excel / API webhook out of the box --- ### How it works 1. Pick a **mode** โ€” ๐ŸŒ **Domain Overview**, ๐Ÿ”‘ **Keyword Research**, or ๐Ÿ† **Top Websites Ranking** 2. **Domain Overview:** paste 1โ€“100 domains (e.g. `amazon.com`, `github.com`); optionally enable **Include Moz Data** 3. **Keyword Research:** paste 1โ€“100 keywords (e.g. `best running shoes`, `seo tools`) 4. Choose a regional **database** (Worldwide + 100 country options; Keyword mode uses the country, Worldwide โ†’ United States) 5. Set parallelism (default 5 โ€” matches Apify's free Datacenter proxy slots; up to 50 on paid plans) 6. Click **Start** โ€” results land in the dataset as one row per domain / keyword Domain Overview triggers a single batched JSON-RPC call covering **organic + paid + backlinks + AI search + competitors**. Keyword Research fans out the keyword's volume/CPC/KD/intent, related + question ideas, clusters, ad copies and SERP โ€” all behind your authenticated session. Cost scales linearly with the number of items, not with how many sections you want back. --- ### Primary use cases | User type | Challenge | Solution | |-----------|-----------|----------| | **SEO specialist** | Need Authority Score, keyword and backlink profile for 50 clients weekly | Bulk-feed 50 domains, get full Domain Overview as one CSV โ€” without 50 Semrush seats | | **Competitive analyst** | "Who is actually beating us in organic and paid?" | Top 20 organic + 20 paid competitors per domain, with shared keywords and traffic | | **AI/LLM researcher** | Need to track which brands LLMs cite, and which sources LLMs cite alongside them | Per-LLM mentions and cited pages across ChatGPT / AIO / AIM / Gemini + top co-cited domains | | **Performance marketer** | "What ad copy is the competitor running, and on which keywords?" | Top 50 paid keywords per domain with full ad title, description, URL and CPC | | **Agency / data team** | Need daily traffic time-series to feed dashboards | Multi-year daily resolution, per-country and global, branded vs non-branded split | | **Market researcher** | Need the top 100 sites in a specific country ร— industry | Top Websites mode โ€” 90+ countries ร— 80+ industries, with MoM/YoY change and traffic-source split | --- ### Input parameters - **`mode`** โ€” `domain` (๐ŸŒ Domain Overview), `keyword` (๐Ÿ”‘ Keyword Research), `seo_audit` (๐Ÿ” SEO Audit) or `top_websites` (๐Ÿ† Top Websites Ranking) - **`domains`** โ€” 1โ€“100 domains or URLs. Domain Overview normalizes them to domains; **SEO Audit keeps the full URL** and audits that exact page (e.g. `https://stripe.com/pricing`). Legacy `urls` input is still supported. - **`keywords`** โ€” 1โ€“100 keywords/phrases for Keyword Research mode (e.g. `best running shoes`, `seo tools`) - **`database`** โ€” Semrush regional database: `worldwide` or one of 100 country codes (`us`, `de`, `uk`, `jp`, `br`, โ€ฆ). Keyword mode uses the chosen country; `worldwide` falls back to United States. - **`include_moz`** โ€” adds Moz DA / Spam Score / top pages / ranking distribution (off by default; adds 5โ€“10 s and Residential traffic per domain) - **`country_top_websites`** โ€” for Top Websites mode: 90+ countries or `global` - **`industry_top_websites`** โ€” for Top Websites mode: 80+ industries or `all` - **`concurrency`** โ€” domains/keywords processed in parallel (default 5, max 50). Each parallel slot needs its own proxy IP. --- ### Output data Each domain returns **one structured JSON record** with two layers: 1. **Flat top-level KPI fields** โ€” what Semrush UI shows as the big plates on the overview page. Render as plain table columns in the Apify dataset view. 2. **Full nested data** โ€” every detail behind those KPIs: per-keyword arrays, daily/monthly trends, per-country splits, sample backlinks, AI-search per-LLM breakdown. #### Flat KPI fields (table-friendly) | Field | Meaning | |---|---| | `domain` / `database` | Domain + selected region (`worldwide` or country code) | | `authority_score` | Semrush Authority Score 0โ€“100 | | `backlinks` / `referring_domains` | Total backlinks + total referring domains | | `follow_backlinks` / `nofollow_backlinks` | Follow vs nofollow split | | `organic_traffic` / `organic_keywords` | Estimated monthly organic visits (excluding SERP features) + organic keyword count | | `total_traffic` | Organic + SERP-feature traffic โ€” matches the big number on Semrush's "Organic Search Traffic" plate | | `organic_traffic_cost_usd` | What that organic traffic would cost via paid ads | | `paid_traffic` / `paid_keywords` / `paid_traffic_cost_usd` | Paid search KPIs | | `ai_visibility` / `ai_visibility_benchmark` | AI visibility 0โ€“100 + industry benchmark | | `ai_mentions` / `ai_cited_pages` | Total mentions across LLMs + cited pages | | `top_country` / `top_country_traffic` | Country with the most organic traffic (most useful in `worldwide` mode) | | `organic_competitors_count` / `paid_competitors_count` | Total competitor counts | | `moz_domain_authority` / `moz_spam_score` | Moz DA + Spam Score *(when Moz is enabled)* | > **Source rule for `organic_*` flat fields:** for a specific country DB (`us`, `de`, `uk`, โ€ฆ) we use that country's summary directly. For `worldwide` we use the latest point of the global daily trend. #### Nested sections (raw detail) | Section | What's inside | |---------|---------------| | **`authority`** | Authority Score (0โ€“100), backlinks total, referring domains, link power, naturalness, health, search-traffic factor | | **`backlinks_detail`** | Follow / nofollow / images / forms / frames split, link types, top anchor texts with backlink + domain counts, sample referring URLs | | **`organic.country_summary`** | Selected country: rank, organic traffic, keywords, traffic cost USD, branded vs non-branded split, SERP-feature share | | **`organic.summary_by_country_*`** | Per-country daily and monthly aggregates across **all** Semrush regional databases | | **`organic.top_keywords`** | Top 50 organic keywords with position (and previous), volume, CPC USD, KD, traffic, traffic share %, cost, **decoded intent** (informational/navigational/commercial/transactional), SERP feature codes, landing URL | | **`organic.competitors`** | Top organic competitors with competition level, common/organic/paid keywords, organic traffic, traffic cost, total traffic, SERP-feature traffic | | **`organic.competitors_total`** | Total organic competitor count | | **`organic.trend_*_daily/monthly`** | Daily and monthly time-series of organic + paid + serp-feature traffic, country and global, with branded/non-branded split | | **`paid.top_ads`** | Top 50 paid keywords with **full ad copy** (title, description, visible URL), position, volume, CPC, KD, traffic, traffic cost | | **`paid.competitors`** | Top paid competitors with shared keywords, ad traffic, ad spend | | **`paid.competitors_total`** | Total paid competitor count | | **`ai_search.ai_visibility`** | 0โ€“100 AI Visibility score plus industry benchmark | | **`ai_search.by_llm`** | Per-LLM breakdown: **ChatGPT, AI Overview, AI Mode, Gemini** โ€” mentions, self-mentions, cited pages | | **`ai_search.top_cited_sources`** | Top 20 domains cited by AI together with this domain | | **`ai_search.by_country`** | AI visibility and mention counts per country | | **`serp_features`** | Distribution of organic positions across Google SERP features | | **`moz`** | DA, Spam Score, linking root domains, ranking keywords count, top pages by PA, top linking domains by DA, search competitors with visibility, top ranking keywords with rank, ranking distribution (#1-3 โ†’ #41-50), discovered/lost linking domains over 60 days *(optional โ€” toggle in input)* | #### Keyword Research mode (`mode = keyword`) Each keyword returns **one structured JSON record**: a flat KPI block up front, full nested research data behind it. **Flat KPI fields (table-friendly)** | Field | Meaning | |---|---| | `keyword` / `database` | Keyword + country database | | `volume` | Average monthly searches in the selected country | | `cpc_usd` | Average Google Ads cost-per-click | | `competition` | Paid competition density 0โ€“1 | | `keyword_difficulty` | How hard it is to rank organically (0โ€“100) | | `referring_domains_median` | Median referring domains of current top pages | | `organic_results_count` | Number of organic results Google returns | | `intents` | Decoded intent(s): informational / navigational / commercial / transactional | | `keyword_ideas_total` / `keyword_ideas_total_volume` | Size of the related-ideas pool and its total volume | | `related_keywords_count` / `questions_count` / `serp_results_count` | Counts of what's returned in the nested arrays | **Nested sections** | Section | What's inside | |---------|---------------| | `volume_trend_12mo` | 12-point relative monthly volume trend (0โ€“100) | | `volume_by_country` | Search volume for the same keyword across every Semrush country DB (top 40 by volume) | | `related_keywords` | Broad-match keyword ideas with volume and difficulty | | `questions` | Question-form keyword ideas with volume and difficulty | | `clusters` | Topic clusters with per-cluster intent distribution | | `ads` | Google Ads creatives for the keyword โ€” title, description, display URL, landing URL, domain | | `serp` | Live SERP โ€” position, domain, URL, result kind, SERP feature codes | ```json { "keyword": "best running shoes", "database": "us", "volume": 60500, "cpc_usd": 1.1, "competition": 1, "keyword_difficulty": 49, "referring_domains_median": 11, "organic_results_count": 753000000, "intents": ["commercial"], "keyword_ideas_total": 31402, "keyword_ideas_total_volume": 1003570, "volume_trend_12mo": [44, 44, 44, 36, 44, 44, 44, 36, 44, 44, 44, 100], "volume_by_country": [ {"country": "us", "volume": 60500}, {"country": "in", "volume": 40500}, {"country": "uk", "volume": 12100} ], "related_keywords": [ {"keyword": "best running shoes for men", "volume": 33100, "keyword_difficulty": 45} ], "questions": [ {"keyword": "what are the best running shoes", "volume": 2400, "keyword_difficulty": 42} ], "clusters": [ {"name": "best shoes to run long distance", "intents": {"commercial": 1, "informational": 0, "navigational": 0, "transactional": 0}} ], "ads": [ {"title": "Shop Trail Performance Shoe", "description": "Engineered for performance.", "domain": "weekends.world"} ], "serp": [ {"position": 1, "domain": "runrepeat.com", "url": "https://runrepeat.com/catalog/running-shoes", "result_kind": 50, "serp_features_codes": []} ] } ```` #### SEO Audit mode (`mode = seo_audit`) Each URL returns **one record**: on-page SEO fetched directly from the page, plus authenticated Semrush organic context for the domain. | Group | Fields | |---|---| | **Response** | `status_code`, `final_url`, `response_time_ms`, `page_size_bytes` | | **Meta** | `title` + `title_length`, `meta_description` + `meta_description_length`, `meta_robots`, `canonical`, `lang` | | **Headings** | `h1` (array) + `h1_count`, `h2_count`, full `headings` H1โ€“H6 | | **Content & links** | `word_count`, `internal_links`, `external_links`, `images_count`, `images_missing_alt`, `structured_data_blocks` | | **Social / i18n** | `open_graph`, `twitter_card`, `hreflang` | | **Crawlability** | `has_robots_txt`, `has_sitemap`, `sitemaps` | | **Semrush context** | `authority_score`, `organic_traffic`, `organic_keywords_count`, `backlinks`, `referring_domains`, plus `semrush.top_organic_keywords` and `semrush.organic_competitors` | ```json { "url": "https://stripe.com/pricing", "final_url": "https://stripe.com/en-cz/pricing", "domain": "stripe.com", "status_code": 200, "response_time_ms": 906, "page_size_bytes": 898028, "title": "Pricing & Fees | Stripe", "title_length": 23, "meta_description": "Find Stripe fees and pricing information...", "meta_description_length": 138, "canonical": "https://stripe.com/en-cz/pricing", "lang": "en-CZ", "h1_count": 1, "word_count": 4205, "internal_links": 301, "external_links": 115, "images_count": 56, "images_missing_alt": 10, "has_robots_txt": true, "has_sitemap": true, "authority_score": 79, "organic_traffic": 9512300, "organic_keywords_count": 1640000, "backlinks": 156797276, "semrush": { "top_organic_keywords": [ {"keyword": "stripe", "position": 1, "volume": 550000, "cpc_usd": 5.18, "keyword_difficulty": 83} ], "organic_competitors": [{"domain": "paypal.com"}] } } ``` #### Top Websites Ranking mode | Section | What's inside | |---------|---------------| | **Top 100 Websites** | Domain, total visits, desktop/mobile visits, desktop/mobile %, MoM %, YoY %, primary traffic source โ€” filterable by 90+ countries and 80+ industries | *** ### Output example *(abbreviated)* ```json { "domain": "amazon.com", "database": "worldwide", "authority_score": 100, "backlinks": 5867395493, "referring_domains": 4342366, "follow_backlinks": 2839704998, "nofollow_backlinks": 2574690164, "organic_traffic": 568288700, "total_traffic": 615432100, "organic_keywords": 99114924, "organic_traffic_cost_usd": 428258681, "paid_traffic": 14773778, "paid_keywords": 58366, "paid_traffic_cost_usd": 14284968, "ai_visibility": 67, "ai_visibility_benchmark": 66, "ai_mentions": 8189191, "ai_cited_pages": 7827618, "top_country": "us", "top_country_traffic": 642837782, "organic_competitors_count": 6007717, "paid_competitors_count": 7988, "moz_domain_authority": "100", "moz_spam_score": "1%", "is_root_domain": false, "authority": { "score": 100, "backlinks": 5867395493, "referring_domains": 4342366, "link_power": 6.99, "naturalness": 6, "health": 26, "search_traffic_factor": 6.99 }, "backlinks_detail": { "total_backlinks": 5867395493, "follow": 2839704998, "nofollow": 2574690164, "domains": 4342366, "top_anchors": [ {"anchor": "", "backlinks": 849358331, "domains": 1094218}, {"anchor": "amazon.com", "backlinks": 35520664, "domains": 210383}, {"anchor": "buy from amazon","backlinks": 23179702, "domains": 10176} ], "sample_backlinks": [ {"anchor": "Apple iPad Air 11-Inch", "nofollow": true, "source_url": "https://bestreviews.com/", "target_url": "https://www.amazon.com/dp/B0DZ754XKT"} ] }, "organic": { "country_summary": null, "top_keywords": [ { "keyword": "amazon", "position": 1, "volume": 151000000, "cpc_usd": 0.03, "keyword_difficulty": 100, "traffic": 120800000, "traffic_share_pct": 18.79, "url": "https://www.amazon.com/", "intents": ["commercial"] } ], "competitors": [ {"domain": "walmart.com", "competition_level": 0.52, "common_keywords": 9651380, "organic_traffic": 117649918} ], "competitors_total": 6007717, "trend_global_daily": [ {"date": "20250507", "organic_traffic": 568288700, "organic_keywords": 99114924, "paid_traffic": 14773778, "branded_traffic": 207381624, "non_branded_traffic": 435456158} ], "summary_by_country_daily": [ {"database": "us", "rank": 6, "organicTraffic": 568288700, "organicPositions": 99114924, "traffic": 642837782}, {"database": "in", "rank": 167, "organicTraffic": 12054910, "organicPositions": 1413942, "traffic": 16008667} ] }, "paid": { "top_ads": [ { "keyword": "amazon", "title": "Amazon", "description": "Amazonยฎ Official Site โ€” Find deals and compare prices on amazon at Amazon.", "url": "https://www.amazon.com/", "position": 1, "volume": 151000000, "cpc_usd": 0.03 } ], "competitors_total": 7988 }, "ai_search": { "ai_visibility": 67, "ai_visibility_benchmark": 66, "total_cited_pages": 7827618, "total_mentions": 8189191, "by_llm": [ {"llm": "ChatGPT", "mentions": 1388071, "cited_pages": 998414}, {"llm": "AI Overview", "mentions": 1887053, "cited_pages": 2067040}, {"llm": "AI Mode", "mentions": 2851034, "cited_pages": 5640097}, {"llm": "Gemini", "mentions": 2063033, "cited_pages": 26249} ], "top_cited_sources": [ {"domain": "youtube.com", "mentions": 1511374}, {"domain": "reddit.com", "mentions": 954001} ], "by_country": [ {"country": "us", "visibility": 79, "mentions": 4872535}, {"country": "uk", "visibility": 77, "mentions": 486610} ] }, "moz": { "domain_authority": "100", "spam_score": "1%", "linking_root_domains": "5.5m", "ranking_keywords": "27.4m" } } ``` > The first 23 keys are the **flat KPI block** (what gets rendered as table columns in the Apify dataset view). Everything from `is_root_domain` downwards is the same data plus the nested detail (per-keyword arrays, history, per-country splits) for downstream pipelines. *** ### What makes this scraper different | Feature | Public-page scrapers | This scraper | |---------|---------------------|--------------| | Authority Score and full backlink profile | Limited | **Yes โ€” full Domain Overview** | | Top keywords with CPC, KD, intent, SERP features | No | **Yes โ€” top 50 per domain** | | Paid ad copy (title + description + URL) | No | **Yes โ€” top 50 paid keywords** | | Organic + paid competitors with shared keywords | Partial | **Yes โ€” 20 each + total counts** | | AI Search per-LLM breakdown (ChatGPT / AIO / AIM / Gemini) | No | **Yes** | | Top cited sources alongside the domain in AI answers | No | **Yes** | | Daily + monthly traffic trends, country + global | 3โ€“6 months | **Multi-year daily resolution** | | Top anchor breakdown + sample backlinks | No | **Yes** | | Moz Domain Analysis (DA, Spam, top pages, ranking distribution, 60-day trend) | DA only | **Yes โ€” 10+ metrics** | | No Semrush subscription or API key needed | Varies | **Yes** | | Async parallel processing (configurable, default 5) | Varies | **Yes** | | Up to 100 domains per run | Varies | **Yes** | *** ### Frequently asked questions **How does this differ from running domains through the public Semrush homepage?** The public preview shows only Authority Score and a clipped traffic estimate. This scraper returns the **full authenticated Domain Overview** โ€” top 50 organic and paid keywords, full backlink profile, AI search per-LLM breakdown, competitors with shared-keyword counts, and multi-year daily trends. **Do I need a Semrush subscription or API key?** No. The actor handles authentication and proxies internally โ€” you only need an Apify account. **What does Keyword Research mode return?** Set `mode` to `keyword` and paste keywords. Each keyword returns volume, CPC, Keyword Difficulty, competition, decoded search intent, a 12-month volume trend, the same keyword's volume across every country, related and question keywords, topic clusters, the ad copies running on it, and the live SERP (position, domain, URL). It's the full Semrush Keyword Overview as one JSON row. **Can I get Traffic Analytics (visits, bounce rate, time on site)?** Not in this Actor. Semrush's Traffic & Market and Site Audit are paid-tier products that aren't exposed to the authenticated free-account pool this Actor uses, so they're intentionally out of scope. Domain Overview already includes organic, paid and SERP-feature traffic estimates. **How do I find which competitors are stealing my organic traffic?** Put your domain in `domains`, set the appropriate `database` (e.g. `us`), and read `organic.competitors`. You get up to 20 organic competitors per domain with competition level, common keywords, organic traffic, and total traffic โ€” plus `organic.competitors_total` for the full count. **Can I see exactly what ad copy a competitor is running?** Yes. `paid.top_ads` returns the **top 50 paid keywords** with full ad title, description, and visible URL alongside position, volume, and CPC. **How is AI Search visibility measured?** `ai_search.by_llm` returns mentions and cited pages **per LLM** โ€” ChatGPT, AI Overview, AI Mode, and Gemini โ€” plus a 0โ€“100 aggregate AI Visibility score and the industry benchmark. `ai_search.top_cited_sources` lists the 20 domains most often cited *alongside* yours in AI answers, so you see who shares your AI-result shelf-space. **What's the difference between `organic_traffic` and `total_traffic`?** `organic_traffic` is the classic estimate of monthly visits from organic positions only. `total_traffic` includes SERP features (featured snippets, People Also Ask, AI Overview, etc.) and matches the big number Semrush shows on the "Organic Search Traffic" plate. **Why are some `organic_*` numbers blank for a country DB?** Some smaller Semrush regional databases don't have a full country summary; the actor falls back to the global trend point for those. Use `worldwide` if you want the aggregated view. **How many domains can I process at once?** Up to **100 domains per run**, processed in parallel up to your configured `concurrency` (default 5, max 50). Each parallel slot uses its own proxy IP โ€” keep concurrency at 5 on Apify's free tier. **Should I enable Moz data?** Only if you specifically need Moz DA, Spam Score, top pages, or ranking distribution. Moz adds 5โ€“10 seconds per domain and routes through Residential proxy. Off by default. *** ### Who is this for - **SEO specialists** โ€” Authority Score, full backlink profile, organic/paid keywords with CPC and intent โ€” without paying for a Semrush seat - **Competitive analysts** โ€” Top competitors per domain with shared keywords, traffic, and traffic cost โ€” organic and paid side by side - **AI/LLM researchers** โ€” Per-LLM mentions and cited pages across ChatGPT, AI Overview, AI Mode, Gemini โ€” including which third-party domains co-rank in AI answers - **Performance marketers** โ€” Top paid ads with full ad copy and CPC for any competitor โ€” feed into ad-research pipelines - **Agencies & data teams** โ€” Daily and monthly time-series for traffic, keywords, branded vs non-branded split, exportable as JSON / CSV / Excel from the Apify dataset *** ### Pro tips - **Pick the right `database`** โ€” country-specific DBs (`us`, `de`, `uk`, โ€ฆ) give a real country summary; `worldwide` aggregates the global trend. Don't mix them in one analysis. - **Use `summary_by_country_*`** โ€” even in a single run you get a per-country breakdown across all Semrush regional databases. Great for "which country drives this domain's traffic" without re-running. - **Read `intents` on keywords** โ€” informational vs commercial vs transactional changes how you treat a competitor's ranking. The decoded intent saves you from inferring it. - **Use `top_cited_sources` for AI co-citation research** โ€” the domains most often cited alongside yours in AI answers are the natural targets for digital PR and link campaigns. - **Keep `concurrency` โ‰ค 5 on free tier** โ€” Apify's free Datacenter pool gives you 5 slots. Higher concurrency starves itself on IP rotation. - **Combine with the Top Websites mode** โ€” find the top 100 sites in your industry, then bulk-feed them back into Domain Overview mode to score them all. *** ### Support Having issues or need a custom feature? Open an issue in the Issues tab or contact: **africanec@gmail.com** *** ### Tags Semrush Scraper, Domain Overview, Authority Score, Backlinks Profile, Organic Keywords, Paid Keywords, Ad Copy Scraper, Competitor Analysis, AI Search Visibility, ChatGPT Mentions, AI Overview Mentions, AI Mode, Gemini Mentions, Moz Domain Authority, Moz Spam Score, Top Cited Sources, Bulk SEO, Bulk Domain Analysis, Parallel Scraper, Trending Websites, Market Research Tool, Competitive Intelligence, Daily Traffic Trends, SERP Features Distribution, Branded Traffic Split # Actor input Schema ## `mode` (type: `string`): Pick what you want to scrape. Each mode has its own input fields below. ๐Ÿ™‚ ## `database` (type: `string`): Semrush regional database โ€” controls which country's data is returned. ๐ŸŒ Worldwide = aggregated global view (Domain mode). ๐Ÿ”‘ For Keyword Research a specific country is used (Worldwide falls back to United States). ## `domains` (type: `array`): ๐ŸŒ Domains or URLs to analyze (e.g. amazon.com, https://www.amazon.com/path?x=1). The Actor extracts only the domain before scraping. Each domain triggers a single batched JSON-RPC call covering organic, paid, backlinks, AI search and competitors. Up to 100 per run โ€” every request goes through a freshly rotated authenticated session. โœจ ## `include_moz` (type: `boolean`): Adds Moz Domain Authority, Spam Score, top pages, top linking domains, ranking distribution, etc. ๐Ÿ” Goes through Residential proxy (Moz is Cloudflare-protected). Off by default โ€” enable if you specifically need Moz metrics; adds 5โ€“10 s and Residential traffic per domain. โฑ๏ธ ## `keywords` (type: `array`): ๐Ÿ”‘ Keywords / phrases to research (e.g. best running shoes, seo tools). Each keyword returns volume, CPC, keyword difficulty, competition, intent, 12-month trend, volume by country, related keywords, questions, ad copies and the live SERP. Up to 100 per run. ๐Ÿš€ ## `country_top_websites` (type: `string`): ๐Ÿ† Filter top websites by country. Global = worldwide ranking. ## `industry_top_websites` (type: `string`): ๐Ÿ† Filter top websites by industry. ## `concurrency` (type: `integer`): How many items to process in parallel (Domain & Keyword modes). Default 5 โ€” matches the number of free Datacenter proxy slots Apify gives to free-tier users. Increase if you have a paid plan. ๐ŸŽ๏ธ ## Actor input object example ```json { "mode": "domain", "database": "worldwide", "domains": [ "amazon.com" ], "include_moz": false, "keywords": [ "best running shoes" ], "country_top_websites": "global", "industry_top_websites": "all", "concurrency": 5 } ``` # Actor output Schema ## `defaultDataset` (type: `string`): One item per domain with 80+ SEO fields (Domain Analysis mode) or one item with top 100 websites (Top Websites mode). Export as JSON, CSV, or Excel. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "domains": [ "amazon.com" ], "keywords": [ "best running shoes" ] }; // Run the Actor and wait for it to finish const run = await client.actor("pro100chok/semrush-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "domains": ["amazon.com"], "keywords": ["best running shoes"], } # Run the Actor and wait for it to finish run = client.actor("pro100chok/semrush-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "domains": [ "amazon.com" ], "keywords": [ "best running shoes" ] }' | apify call pro100chok/semrush-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=pro100chok/semrush-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Semrush Full SEO Scraper - Traffic, Authority, Backlinks", "description": "All-in-one Semrush & Moz scraper โ€” no login required. Domain authority, traffic analytics, backlinks, AI traffic (ChatGPT/Gemini/Claude), competitors, top 100 trending websites. 80+ fields per domain, 10\n parallel workers, unique IP per request. Two modes: Domain Analysis & Top Websites Ranking.", "version": "0.1", "x-build-id": "HlATeX0B0UO9EhaHL" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/pro100chok~semrush-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-pro100chok-semrush-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/pro100chok~semrush-scraper/runs": { "post": { "operationId": "runs-sync-pro100chok-semrush-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/pro100chok~semrush-scraper/run-sync": { "post": { "operationId": "run-sync-pro100chok-semrush-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "mode" ], "properties": { "mode": { "title": "๐ŸŽ›๏ธ Mode", "enum": [ "domain", "keyword", "seo_audit", "top_websites" ], "type": "string", "description": "Pick what you want to scrape. Each mode has its own input fields below. ๐Ÿ™‚", "default": "domain" }, "database": { "title": "๐ŸŒ Database (country)", "enum": [ "worldwide", "us", "uk", "ca", "au", "ie", "nz", "za", "de", "fr", "es", "it", "nl", "be", "ch", "at", "pt", "gr", "se", "no", "dk", "fi", "is", "lu", "cz", "sk", "pl", "hu", "ro", "bg", "hr", "si", "rs", "lt", "lv", "ee", "ua", "by", "ru", "kz", "am", "ge", "az", "tr", "il", "ae", "sa", "eg", "jo", "lb", "kw", "qa", "bh", "om", "iq", "in", "jp", "cn", "hk", "tw", "kr", "sg", "my", "ph", "id", "th", "vn", "bd", "pk", "lk", "np", "mm", "mx", "br", "ar", "cl", "co", "pe", "ec", "uy", "py", "bo", "sv", "hn", "ni", "gt", "pa", "cr", "do", "jm", "tt", "cu", "pr", "ng", "ke", "gh", "tz", "ug", "dz", "ma", "tn" ], "type": "string", "description": "Semrush regional database โ€” controls which country's data is returned. ๐ŸŒ Worldwide = aggregated global view (Domain mode). ๐Ÿ”‘ For Keyword Research a specific country is used (Worldwide falls back to United States).", "default": "worldwide" }, "domains": { "title": "๐ŸŒ Domains or URLs", "maxItems": 100, "type": "array", "description": "๐ŸŒ Domains or URLs to analyze (e.g. amazon.com, https://www.amazon.com/path?x=1). The Actor extracts only the domain before scraping. Each domain triggers a single batched JSON-RPC call covering organic, paid, backlinks, AI search and competitors. Up to 100 per run โ€” every request goes through a freshly rotated authenticated session. โœจ", "items": { "type": "string" } }, "include_moz": { "title": "โž• Include Moz Data", "type": "boolean", "description": "Adds Moz Domain Authority, Spam Score, top pages, top linking domains, ranking distribution, etc. ๐Ÿ” Goes through Residential proxy (Moz is Cloudflare-protected). Off by default โ€” enable if you specifically need Moz metrics; adds 5โ€“10 s and Residential traffic per domain. โฑ๏ธ", "default": false }, "keywords": { "title": "๐Ÿ”‘ Keywords", "maxItems": 100, "type": "array", "description": "๐Ÿ”‘ Keywords / phrases to research (e.g. best running shoes, seo tools). Each keyword returns volume, CPC, keyword difficulty, competition, intent, 12-month trend, volume by country, related keywords, questions, ad copies and the live SERP. Up to 100 per run. ๐Ÿš€", "items": { "type": "string" } }, "country_top_websites": { "title": "๐ŸŒ Country", "enum": [ "global", "us", "gb", "de", "fr", "es", "it", "br", "ca", "au", "in", "jp", "ru", "cn", "kr", "nl", "pl", "se", "no", "dk", "fi", "be", "at", "ch", "pt", "ie", "nz", "za", "mx", "ar", "co", "cl", "pe", "il", "ae", "sa", "tr", "eg", "ng", "ke", "ph", "id", "my", "sg", "th", "vn", "tw", "hk", "ua", "cz", "ro", "hu", "bg", "hr", "sk", "si", "rs", "lt", "lv", "ee", "bd", "pk", "iq", "dz", "ma", "tn", "gh", "tz", "ug", "mm", "kz", "by", "ge", "az", "am", "jo", "lb", "kw", "qa", "bh", "om", "do", "ec", "gt", "pa", "cr", "uy", "py", "bo", "sv", "hn", "ni", "jm", "tt", "cu", "pr" ], "type": "string", "description": "๐Ÿ† Filter top websites by country. Global = worldwide ranking.", "default": "global" }, "industry_top_websites": { "title": "๐Ÿญ Industry", "enum": [ "all", "accounting-and-auditing", "adult", "advertising-and-marketing", "aerospace-and-defense", "agriculture", "airlines", "apparel-and-fashion", "architecture", "automotive", "banking", "beauty-and-cosmetics", "biotech", "computer-and-video-games", "computer-hardware", "computer-software-and-development", "computers-and-electronics", "construction-and-maintenance", "consulting", "consumer-electronics", "design", "distance-learning", "e-commerce-and-retail", "ecology-and-environment", "education", "entertainment", "events-services", "finance", "food-and-beverages", "furniture", "gambling", "government", "graphic-design", "healthcare", "hospitality", "human-resources", "import-and-export", "information-technology", "insurance", "investment", "jewelry-and-luxury-products", "legal", "machinery", "manufacturing", "market-research", "mass-media", "mental-health", "metals-and-mining", "music", "nanotechnology", "newspapers", "nonprofit-organizations", "oil-and-gas", "online-services", "pharmaceuticals", "photography", "printing", "public-relations", "public-safety", "publishing", "real-estate", "recruitment-and-staffing", "religion-and-belief", "renewable-energy", "restaurants", "retail", "science", "security-products-and-services", "social-and-charitable-organizations", "sporting-goods", "sports", "telecom", "training-and-certification", "transportation-and-logistics", "travel-and-tourism", "tv-and-movies", "veterinary", "visual-art", "wellness", "wine-and-spirits" ], "type": "string", "description": "๐Ÿ† Filter top websites by industry.", "default": "all" }, "concurrency": { "title": "โšก Concurrency", "minimum": 1, "maximum": 50, "type": "integer", "description": "How many items to process in parallel (Domain & Keyword modes). Default 5 โ€” matches the number of free Datacenter proxy slots Apify gives to free-tier users. Increase if you have a paid plan. ๐ŸŽ๏ธ", "default": 5 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```