# Technical SEO Audit & Website Crawler - Core Web Vitals (`groupoject/technical-seo-audit-crawler`) Actor

Crawl websites and detect technical SEO issues across metadata, indexability, headings, content, links, images, structured data, security headers, and optional PageSpeed Core Web Vitals.

- **URL**: https://apify.com/groupoject/technical-seo-audit-crawler.md
- **Developed by:** [Group Oject](https://apify.com/groupoject) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Technical SEO Audit & Website Crawler - Core Web Vitals

Crawl any public website and turn technical SEO problems into structured, prioritized data. Every crawled page receives a 0-100 score, an A-F grade, severity counts, detailed checks, and actionable recommendations.

The Actor audits metadata, crawlability, indexability, heading structure, content quality signals, internal and external links, images, JSON-LD, social tags, security headers, duplicate metadata, robots.txt, sitemaps, and optional Google PageSpeed/Core Web Vitals.

### Why use this SEO audit Actor?

- **Report-ready issue objects** - every problem has a code, severity, category, message, recommendation, and affected field.
- **Site-wide crawling** - follow internal links with depth, page, domain, subdomain, include, and exclude controls.
- **Transparent scoring** - critical, high, medium, and low issues feed a predictable 0-100 score.
- **Duplicate detection** - identify repeated titles and meta descriptions after the crawl completes.
- **Optional broken-link checks** - verify internal and external links with bounded extra requests.
- **Optional Core Web Vitals** - enrich selected pages with PageSpeed, Lighthouse, LCP, CLS, INP, TBT, and Speed Index.
- **Billing-safe limits** - the crawler respects both `maxPages` and the user's maximum-charge limit.
- **Fast HTTP crawler** - no browser is required for normal HTML audits.

### Quick start

```json
{
  "startUrls": ["https://example.com"],
  "maxPages": 25,
  "maxDepth": 3
}
```

For a focused content-section audit:

```json
{
  "startUrls": ["https://example.com"],
  "maxPages": 100,
  "maxDepth": 5,
  "includePatterns": ["https://example.com/blog/*"],
  "excludePatterns": ["*/tag/*", "*/author/*", "*?preview=*"]
}
```

### What the Actor checks

#### Metadata and indexability

- Missing, short, or long title tags
- Missing or poorly sized meta descriptions
- Missing canonical links
- `noindex` and restrictive robots directives
- Missing HTML language and mobile viewport
- HTTP error pages

#### Headings and content

- Missing or multiple H1 headings
- Skipped heading levels
- Visible word count and thin-content warnings
- Content-to-HTML ratio
- Duplicate titles and descriptions across pages

#### Links and images

- Internal, external, and unique link counts
- Optional broken and unreachable link checks
- Images missing alt text
- Images without width and height attributes
- Native lazy-loading counts

#### Structured data and social previews

- JSON-LD block count and discovered schema types
- Invalid JSON-LD
- Open Graph completeness
- Twitter/X card presence
- Hreflang count

#### Performance and security

- HTML response size
- HSTS and Content-Security-Policy presence
- Optional PageSpeed performance, SEO, accessibility, and best-practices scores
- Optional LCP, CLS, INP, Total Blocking Time, and Speed Index

### Output

Each default dataset row represents one crawled page:

```json
{
  "finalUrl": "https://example.com/pricing",
  "statusCode": 200,
  "score": 82,
  "grade": "B",
  "indexable": true,
  "title": "Pricing - Example",
  "titleLength": 17,
  "metaDescriptionLength": 148,
  "h1Count": 1,
  "wordCount": 642,
  "imagesMissingAlt": 2,
  "brokenLinksCount": 1,
  "criticalIssues": 0,
  "highIssues": 1,
  "issueCount": 6,
  "issues": [
    {
      "code": "short-title",
      "severity": "medium",
      "category": "metadata",
      "message": "Title is only 17 characters.",
      "recommendation": "Use a descriptive title that clearly communicates page intent.",
      "field": "title"
    }
  ]
}
```

The default key-value store also contains:

- `SUMMARY` - average score, grade, issue totals, indexability, failed pages, and top issue codes.
- `ISSUES` - one flat report-ready list of all issues across all pages.
- `SITE_SIGNALS` - robots.txt status, content, and discovered sitemap URLs.

### PageSpeed and Core Web Vitals

Enable `includePageSpeed` to call Google PageSpeed Insights for up to `pageSpeedMaxPages`. A Google API key is optional but recommended for reliable quota. The key is treated as secret input and never appears in output.

PageSpeed calls are slower and subject to Google's quota, so use them on key templates rather than every URL in a large crawl.

### Broken-link checking

Enable `checkBrokenLinks` to test discovered links after crawling. The Actor first sends a HEAD request and falls back to GET when a server rejects HEAD. Use `maxLinksToCheck` to bound runtime and third-party traffic.

### SEO score

The score starts at 100 and deducts:

| Severity | Deduction |
|---|---:|
| Critical | 20 |
| High | 8 |
| Medium | 4 |
| Low | 2 |

The score is a prioritization aid, not a promise of rankings. Search performance also depends on relevance, authority, content quality, competition, and off-page signals.

### Use cases

- SEO agency prospect and client audits
- Site migration and redesign QA
- Scheduled technical SEO monitoring
- Programmatic site-quality dashboards
- Content inventory cleanup
- E-commerce category and product-page audits
- Pre-launch checks for marketing teams

### FAQ

#### Does the Actor render JavaScript?

The default crawler audits server-delivered HTML. This makes it fast and inexpensive. If a site renders all SEO content only in the browser, server HTML may appear thin; such a site also deserves review because search crawlers need dependable renderable content.

#### Does this replace Google Search Console?

No. Search Console supplies Google's indexing and traffic data. This Actor supplies crawlable page-level diagnostics that can complement it.

#### Can I audit multiple sites in one run?

Yes. Add multiple `startUrls`; `maxPages` is shared across the entire run.

#### Will it crawl login, cart, or checkout pages?

Use `excludePatterns` to keep sensitive or low-value areas out of the crawl. Common exclusions are included in the default form.

#### Is a proxy required?

Usually not. Add an Apify proxy only when a public site rate-limits or blocks direct requests.

# Actor input Schema

## `startUrls` (type: `array`):

Start pages or domains. The crawler follows internal links from each URL.

## `urlsText` (type: `string`):

Paste one website URL per line.

## `maxPages` (type: `integer`):

Maximum pages to crawl and bill across all websites.

## `maxDepth` (type: `integer`):

Start URLs are depth 0. Set 0 to audit only the supplied pages.

## `includeSubdomains` (type: `boolean`):

Allow links on subdomains of each starting hostname.

## `includePatterns` (type: `array`):

Optional wildcard patterns. Only matching crawl URLs are included, for example https://example.com/blog/\*.

## `excludePatterns` (type: `array`):

Wildcard patterns to skip, such as */login*, */cart*, or *?preview=*.

## `checkBrokenLinks` (type: `boolean`):

Send extra HEAD/GET requests to test discovered links. This increases runtime.

## `maxLinksToCheck` (type: `integer`):

Maximum unique internal and external links tested when broken-link checks are enabled.

## `includePageSpeed` (type: `boolean`):

Call Google PageSpeed Insights for a limited number of crawled pages.

## `pageSpeedStrategy` (type: `string`):

Mobile is recommended for SEO audits.

## `pageSpeedApiKey` (type: `string`):

Optional. Improves PageSpeed quota and reliability. The key is never included in output.

## `pageSpeedMaxPages` (type: `integer`):

Limit expensive PageSpeed calls separately from the crawl page limit.

## `maxConcurrency` (type: `integer`):

Concurrent page requests. Lower this for small or sensitive websites.

## `requestTimeoutSecs` (type: `integer`):

Maximum time to wait for each page request before retrying or recording a crawl failure.

## `proxyConfiguration` (type: `object`):

Optional. Direct requests are usually enough for public websites.

## `userAgent` (type: `string`):

Optional HTTP user-agent override.

## `debugMode` (type: `boolean`):

Enable verbose crawler logs.

## Actor input object example

```json
{
  "startUrls": [
    "https://apify.com"
  ],
  "maxPages": 25,
  "maxDepth": 3,
  "includeSubdomains": false,
  "excludePatterns": [
    "*/login*",
    "*/cart*",
    "*/checkout*"
  ],
  "checkBrokenLinks": false,
  "maxLinksToCheck": 100,
  "includePageSpeed": false,
  "pageSpeedStrategy": "mobile",
  "pageSpeedMaxPages": 3,
  "maxConcurrency": 10,
  "requestTimeoutSecs": 30,
  "debugMode": false
}
```

# Actor output Schema

## `pages` (type: `string`):

Scored technical SEO audit rows.

## `summary` (type: `string`):

Average score, issue totals, indexability, and top issue codes.

## `issues` (type: `string`):

Flat report-ready list of every issue and recommendation.

## `siteSignals` (type: `string`):

robots.txt status and discovered sitemap URLs.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://apify.com"
    ],
    "excludePatterns": [
        "*/login*",
        "*/cart*",
        "*/checkout*"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("groupoject/technical-seo-audit-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://apify.com"],
    "excludePatterns": [
        "*/login*",
        "*/cart*",
        "*/checkout*",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("groupoject/technical-seo-audit-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://apify.com"
  ],
  "excludePatterns": [
    "*/login*",
    "*/cart*",
    "*/checkout*"
  ]
}' |
apify call groupoject/technical-seo-audit-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=groupoject/technical-seo-audit-crawler",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/RlyaAj0aqVc3Fi0Gq/builds/iTT4VagugMZRcLfe8/openapi.json