SimilarWeb Traffic Scraper

Actor ID: morph_coder/similarweb-scraper

SimilarWeb Traffic Scraper is a powerful, production-ready Apify Actor designed to extract website traffic intelligence, competitor data, domain WHOIS records, and on-page SEO keyword density in bulk.

This Actor is highly optimized to avoid detection using rotated residential proxies and custom TLS/HTTP2 fingerprints.

This Actor is not affiliated with SimilarWeb, SimilarSites, or AITDK. Users must comply with applicable terms of service and local laws.

---

❶ Key Features

• 🌍 Comprehensive Website Analytics (SimilarWeb Mode): Extract global rank, country rank, category rank, total monthly visits, 3-month historical visits, and traffic source breakdowns (direct, search, referrals, social, mail, display ads).
• 🖼️ Asset Reconstruction: Reconstruct website icons, desktop previews, and mobile previews directly from metadata, eliminating the need for heavy browser instances.
• 🚦 Geographic Breakdown: Get the top 5 traffic-driving countries with percentage shares resolved to full country names.
• 🤝 Similar Sites Discovery (Similar Sites Mode): Retrieve up to 20 similar competitor websites from Similarsites.com, including site descriptions and total visits.
• 🔍 SEO & Domain Intelligence (AITDK Mode): Perform structured domain WHOIS lookups (via RDAP) and homepage n-gram keyword density analysis (1-to-5 word phrases) with multi-language stop-word filtering.
• ⚡ Concurrency & Rotated Proxies: Supports bulk lists processed in parallel. Utilizes unique proxy sessions per request to ensure maximum bypass success rates.

---

❷ Scraper Modes

The Actor operates in three distinct search modes selected via the searchType input:

1. similarweb (Traffic Data - Default): Retrieves core website metrics, rank history, monthly visits, and traffic sources.
2. similar_sites (Competitors): Retrieves a list of related/alternative websites with similarity scores.
3. aitdk (SEO & WHOIS): Returns WHOIS registry details (creation date, expiration date, registrar, nameservers) and visible page keyword density.

---

❸ Input Parameters

The Actor accepts the following JSON properties:

Field	Type	Required	Default	Description
urls	array	No	["google.com"]	List of webpage URLs, domains, or websites to analyze.
domains	array	No	—	Alternative input list of domains (merged with urls).
websites	array	No	—	Alternative input list of website domains (merged with urls).
searchType	string	No	"similarweb"	Mode of scraping: "similarweb", "similar_sites", or "aitdk".
include_base_data	boolean	No	true	When enabled, returns base SimilarWeb traffic and ranking data.
include_similar_sites	boolean	No	false	When enabled, includes SimilarSites competitor lists.
include_indepth_data	boolean	No	false	Includes additional historical trends and company details if available.
output_mode	string	No	"aggregated"	Format for output items: "aggregated" or "individual".
proxyConfiguration	object	No	—	Proxy configuration. Residential proxy rotation is highly recommended.

Sample Input JSON

{
    "urls": ["sportsdigest.in"],
    "searchType": "similarweb",
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": ["RESIDENTIAL"]
    }
}

---

❹ Output Examples

SimilarWeb Mode (Traffic & Ranks)

{
  "url": "https://similarweb.com/website/sportsdigest.in",
  "name": "sportsdigest.in",
  "title": "Football News: Live Scores, Results & Match Updates",
  "description": "Football news, live scores, match results...",
  "category": "sports/sports",
  "icon": "https://site-images.similarcdn.com/image?url=sportsdigest.in&t=2&s=1&h=47c9aefee98a6b...",
  "previewDesktop": "https://site-images.similarcdn.com/image?url=sportsdigest.in&t=1&s=1&h=47c9aefee98a6b...",
  "previewMobile": "https://site-images.similarcdn.com/image?url=sportsdigest.in&t=4&s=1&h=47c9aefee98a6b...",
  "globalRank": { "rank": 441621 },
  "countryRank": { "countryCode": "IN", "rank": 36173 },
  "categoryRank": { "category": "Sports/Sports", "rank": 135 },
  "engagements": {
    "visits": 49385,
    "timeOnSite": 371.7,
    "pagePerVisit": 9.62,
    "bounceRate": 0.33
  },
  "trafficSources": {
    "direct": 0.58,
    "referrals": 0.08,
    "mail": 0.01,
    "social": 0.03,
    "search": 0.29,
    "paidReferrals": 0
  },
  "topKeywords": [
    { "name": "hardik pandya trade to csk", "estimatedValue": 970, "volume": 980 }
  ],
  "topCountries": [
    { "countryCode": "IN", "countryName": "India", "countryUrlCode": "india", "visitsShare": 0.85 }
  ],
  "estimatedMonthlyVisits": {
    "2026-03-01": 61051,
    "2026-04-01": 54721,
    "2026-05-01": 49385
  },
  "scrapedAt": "2026-06-21T13:38:31.361Z",
  "snapshotDate": "2026-05-01T00:00:00+00:00"
}

Similar Sites Mode (Competitors)

{
  "SiteName": "sportsdigest.in",
  "SimilarSites": [
    {
      "Site": "iplt20.com",
      "TotalVisits": 1353414.01,
      "CategoryRank": null,
      "TopCountryRank": 5418,
      "Description": "The official IPL website with live scores, match updates..."
    }
  ]
}

AITDK Mode (WHOIS + Keyword Density)

{
  "SiteName": "sportsdigest.in",
  "Whois": {
    "registrationDate": "2021-03-07T03:46:41.533Z",
    "expirationDate": "2028-03-07T03:46:41.533Z",
    "registrar": "GoDaddy",
    "registrantCountry": "IN",
    "nameservers": ["samara.ns.cloudflare.com", "porter.ns.cloudflare.com"],
    "status": ["client update prohibited", "client delete prohibited"]
  },
  "KeywordDensity": {
    "1word": [
      { "keyword": "sports", "count": 17, "total": 803, "density": 2.12 }
    ],
    "2words": [
      { "keyword": "world cup", "count": 10, "total": 1013, "density": 0.99 }
    ]
  }
}

---

❺ Pricing & Monetization

This Actor is configured with custom pay-per-event pricing on the Apify Store:

• Custom Event (processed-item): Charged once per successfully scraped domain.
• Compute & Data Transfer: Billed in standard platform usage rates.
• Apify Proxy: Residential proxy data transmission usage is billed separately per gigabyte.

---