Pricing

from $9.99 / 1,000 results

Open Directory Link Scraper

Open Directory Link Scraper is an Apify Actor that extracts all hyperlinks from web pages and open directory listings. It can recursively crawl directory trees or same-host links, filter files by extension, and returns clean, absolute URLs with metadata such as file type and source page.

Pricing

from $9.99 / 1,000 results

Rating

0.0

(0)

Developer

Jamshaid Arif

Actor stats

Bookmarked

Total users

Monthly active users

6 days ago

Last modified

Open Directory Link Scraper (Apify Actor)

Extracts every hyperlink from any page or "index of" open directory, resolved to absolute URLs. Optionally filters by file extension and recurses into directory trees or same-host links.

Input

Field	Description
`startUrls`	Pages / open-directory URLs to scrape
`extensions`	Keep only links ending in these, e.g. `["pdf","zip"]` (empty = all)
`recursive`	Follow subdirectory links in "index of" listings
`followSameHostLinks`	Follow ANY same-host link — turns it into a bounded crawler
`maxDepth`	Link-hops from a start URL (0 = start URLs only)
`maxPages`	Hard cap on pages fetched for the whole run
`dedupeLinks`	Emit each distinct URL only once
`proxyConfiguration`	Optional — most open directories need none
`requestDelaySecs`	Pause between fetches (default 0.5)

Output

{
  "text": "report-2024.pdf",
  "url": "https://files.example.com/docs/report-2024.pdf",
  "extension": "pdf",
  "is_directory": false,
  "source_url": "https://files.example.com/docs/"
}

Modes

List one page's links (default) — no recursion, just the start URLs.
Harvest an open directory — set recursive: true to walk index of subdirectories and collect every file link. Pair with extensions to grab, say, only pdf or mp3 files.
Bounded same-host crawl — set followSameHostLinks: true to follow any same-host link up to maxDepth / maxPages.

The crawler never leaves the start URL's host, skips mailto:/tel:/anchor links, and ignores the sort/parent links common in Apache/nginx directory listings.

Public APIs Directory Scraper - API Catalog Data

benthepythondev/publicapis-directory-scraper

Scrape public API directory pages and extract API names, URLs, descriptions, categories and developer integration metadata.

Ben

Public Directory Change Detector

eager_vitamin/public-directory-change-detector

Monitor public directory pages for new, unchanged, and removed listings.

Lau Kuan Ee

Dun & Bradstreet Business Directory Scraper

mscraper/dun-bradstreet-business-directory-scraper

Extract company information such as name, sales revenue, address, and URL from the DNB directory. Get an overview of the number of companies that can be scraped.

mscraper

389

ISSA Directory Scraper

songd/ISSA-Directory-Scraper

Scrapes company directory data from ISSA's member directory.

Singed

Pals Cyprus Directory Scraper - Cyprus Business Directory

crawlerbros/cypindex-scraper

Scrape Pals Cyprus Directory (palscyprus.directory), a free Cyprus business directory. Browse by category or fetch profiles by URL. Get names, phones, addresses, categories, descriptions, and source links.

Crawler Bros

Directory Vendor Data Extractor

seeb/directory-vendor-data-extractor

Run Directory Vendor Data Extractor to collect structured public web data with source URLs, clean dataset rows, and guarded execution controls.

Techionik

Open Source Actors Scraper

lukaskrivka/open-source-actors-scraper

Get all open-source Actors from Apify Store.

Lukáš Křivka

Vendor Directory Lead Extractor

glowing_glove/directory-vendor-data-extractor

Turn public member, vendor, supplier, and business directory pages into structured organization records.

Ushba Khan

Property Finder UAE Real Estate Agencies Directory Scraper

drramerets/propertyfinder-agencies

The full Property Finder agency directory: name, email, phone, office address, licence, agent headcount and listing counts by segment. Taken from the service directory that paginates properly, not the web page that repeats its first twenty.

Andrey

BBB Scraper - Business Directory Search

goat255/bbb-scraper

Search the public business directory across the US and Canada. One clean row per business with phone, full postal address split into fields and a link to its listing. No login and no API key.