# Site Launch & Migration Technical SEO Audit (`burly_bat/site-launch-migration-audit`) Actor

Bulk technical launch/indexability audit for a list of URLs, a sitemap or a chained dataset. Checks HTTP status, redirects, robots.txt, meta robots, X-Robots-Tag, canonical, on-page tags, JSON-LD and hreflang, then returns a PASS/WARN/FAIL verdict per URL plus a go/no-go rollup.

- **URL**: https://apify.com/burly\_bat/site-launch-migration-audit.md
- **Developed by:** [Burly Bat](https://apify.com/burly_bat) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $7.00 / 1,000 audited urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Site Launch & Migration Technical SEO Audit

Run a **bulk technical launch and indexability audit** across a flat list of URLs, a sitemap,
or a chained dataset — before you flip a redesign live or after a migration. This Actor fetches
each page over plain HTTP, follows redirects, reads the server-rendered HTML and response
headers, and returns a deterministic **PASS / WARN / FAIL** verdict per URL plus a batch-level
`deploymentGate` your SEO, engineering and QA teams can act on.

The most common launch catastrophe is an accidental `noindex` or a `robots.txt Disallow` that
silently de-indexes a whole section. This audit treats that as the number-one signal.

### What it checks per URL

- **HTTP status and full redirect chain** — every hop, loops, missing `Location`, long chains,
  HTTPS→HTTP downgrades and the final status code.
- **Indexability (the core)** — `robots.txt` allow/disallow for the URL path (fetched and parsed
  per host, with Google's most-specific-rule semantics), meta robots `noindex`, the
  `X-Robots-Tag` response header, and a self / cross-domain canonical check. `noindex` is
  detected from **all three sources** and reported in `noindexSources`.
- **Core on-page tags** — `title` (presence + length), meta description (presence + length),
  `H1` count, canonical, and Open Graph (`og:title`, `og:image`). These are checked **only on
  indexable HTTP 200 HTML pages**, never on redirects or `noindex` pages, so you never get a
  false FAIL for a missing title on a page that was never meant to render one.
- **JSON-LD** — structural/syntactic validation only: is the `ld+json` valid JSON and does it
  declare an `@type`? (No schema.org scoring, no "AI-readiness".)
- **hreflang** — presence and sanity: valid language codes, a self-referencing entry and no
  conflicting language→URL mappings.

### Site-level checks (once per run)

`siteChecks` reports the primary host, whether `robots.txt` was found, any sitemap it declares,
whether a provided sitemap was parsable and how many URLs it held, and how much of the audited
sample was indexable.

### How verdicts work

- **FAIL** (blocks the deployment gate) — unreachable / DNS / timeout / redirect loop, a broken
  final status (4xx/5xx), an HTTPS downgrade, or **indexability blockers**: `noindex` (meta or
  header) and `robots.txt` disallow.
- **WARN** — quality risks that rarely block a launch: long redirect chains, missing/duplicate
  on-page tags, canonical pointing elsewhere, invalid JSON-LD, or hreflang sanity issues.
- **PASS** — indexable, reachable and free of the above.

Set `treatWarnAsFail: true` to make the batch gate stricter (any WARN fails the gate) without
changing the per-URL diagnostic verdicts. `deploymentGate` is `FAIL` when any URL fails.

### Input

Provide URLs in any combination (they merge and deduplicate):

```json
{
  "urls": ["https://www.example.com/", "https://www.example.com/pricing"],
  "urlsText": "https://www.example.com/blog\n# comments are ignored\nhttps://www.example.com/help",
  "sitemapUrl": "https://www.example.com/sitemap.xml",
  "datasetId": "<another Actor run's datasetId>",
  "datasetUrlField": "url",
  "checkRobotsTxt": true,
  "checkOnPage": true,
  "checkStructuredData": true,
  "checkHreflang": true,
  "treatWarnAsFail": false
}
```

A sitemap is expanded (a bounded `sampleSitemapUrls` sample; a sitemap index is followed one
level deep). `datasetId` reads URLs from another Actor for chaining.

> **This is not a source→target redirect-map validator.** It audits each URL on its own merits.
> If you need to validate an old-URL → new-URL migration spreadsheet (301/308 correctness, exact
> destination match, many-to-one consolidation), use the companion Actor
> **`site-migration-redirect-map-qa`** — see cross-sell below.

### Output

- **Dataset** — one PASS/WARN/FAIL record per URL (JSON/CSV/Excel), same shape family as the
  redirect-map QA Actor for easy side-by-side reporting.
- **Key-value store** — `OUTPUT` (JSON summary + siteChecks), `REPORT` (self-contained,
  CSP-protected HTML report) and `SUMMARY` (Markdown for tickets and pull requests).

Common issue codes: `NOINDEX_META`, `NOINDEX_HEADER`, `ROBOTS_TXT_DISALLOWED`, `NOT_FOUND`,
`BROKEN_STATUS`, `HTTPS_DOWNGRADE`, `REDIRECT_CHAIN_LONG`, `CANONICAL_MISSING`,
`CANONICAL_CROSS_DOMAIN`, `TITLE_MISSING`, `TITLE_TOO_LONG`, `META_DESCRIPTION_MISSING`,
`H1_MISSING`, `H1_MULTIPLE`, `OG_TITLE_MISSING`, `OG_IMAGE_MISSING`, `JSONLD_INVALID`,
`JSONLD_NO_TYPE`, `HREFLANG_INVALID_CODE`, `HREFLANG_NO_SELF`, `HREFLANG_CONFLICT`.

### Where this fits: one migration gate, three stages

1. **Redirect-map QA** — validate the old→new URL spreadsheet (301/308, exact destination,
   chains, loops): Actor `site-migration-redirect-map-qa`.
2. **Launch & indexability audit (this Actor)** — is each live URL reachable, crawlable,
   indexable and technically sound?
3. **AI-search / answerability readiness** — is the content structured for LLM answer engines?
   That is a deliberately separate scope this Actor does not cover.

### Important limitation — no JavaScript rendering

This Actor uses deterministic HTTP and **server-rendered HTML only**. It does not execute
JavaScript, log in, or bypass bot protection. Tags injected client-side (canonical, robots,
JSON-LD, hreflang) are not seen; such pages are marked `confidence: LOW` and should be verified
in a rendered browser. These limits keep audits reproducible, inexpensive and low-maintenance.

### Security and responsible use

- Audit only public URLs you own or are authorized to test.
- The Actor blocks localhost, private/link-local/reserved IP ranges, cloud metadata hosts, URL
  credentials and non-standard ports — and re-validates every redirect hop, including for
  `robots.txt` and sitemap fetches, closing DNS-rebinding gaps.
- Responses are capped, requests time out and concurrency is bounded.
- No personal data, login, proxy or external API is required.
- This is technical QA, not a guarantee of rankings or legal/SEO advice.

### Pricing (pay per event)

- **$0.007 per audited URL** that returns a resolved final HTTP response (that is $7.00 per
  1,000 URLs), plus a near-zero `$0.00005` Actor start event.
- A URL is charged whenever it resolves to a final HTTP response, regardless of PASS/WARN/FAIL —
  a detected 404 or a `noindex` is a result you are paying for.
- Blocked, invalid, DNS-failed, timed-out and unreachable URLs are **never charged**, even
  though they are still delivered to the dataset.
- Platform usage is covered by the Actor creator, so the event price is what you pay.
- Set `maxTotalChargeUsd` on the run to cap spend. The Actor trims the input before doing any
  work, so your limit is never exceeded and `skippedByChargeLimit` reports what was left out.

### Sources

- [Google: Site moves with URL changes](https://developers.google.com/search/docs/crawling-indexing/site-move-with-url-changes)
- [Google: Robots meta tag and X-Robots-Tag](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag)
- [Google: robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
- [Google: Canonical URLs](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls)
- [sitemaps.org protocol](https://www.sitemaps.org/protocol.html)

# Actor input Schema

## `urls` (type: `array`):

A flat list of absolute URLs to audit. Use this, urlsText, sitemapUrl or datasetId (they combine and deduplicate).

## `urlsText` (type: `string`):

One absolute URL per line. Lines starting with # are ignored. Paste directly from a spreadsheet or crawl export.

## `sitemapUrl` (type: `string`):

A sitemap.xml or sitemap index URL. Its <loc> entries (a bounded sample) are fetched and audited.

## `datasetId` (type: `string`):

Read URLs from another Actor run's dataset. Set datasetUrlField to the field that holds each URL.

## `datasetUrlField` (type: `string`):

Field name that holds the URL in each dataset item.

## `maxUrls` (type: `integer`):

Safety cap applied after combining and deduplicating all input sources.

## `sampleSitemapUrls` (type: `integer`):

Maximum number of URLs to take from the sitemap (a sample keeps large sitemaps fast and cheap).

## `maxRedirects` (type: `integer`):

Stop and flag a URL when its redirect chain exceeds this limit.

## `timeoutSecs` (type: `integer`):

Maximum time to wait for each redirect hop or final response.

## `maxConcurrency` (type: `integer`):

Keep this conservative when auditing one host to avoid overloading it.

## `treatWarnAsFail` (type: `boolean`):

When on, the batch deployment gate is FAIL if any URL is WARN or FAIL (per-URL verdicts are unchanged).

## `checkRobotsTxt` (type: `boolean`):

Fetch and parse each host's robots.txt and flag paths disallowed for Googlebot.

## `checkOnPage` (type: `boolean`):

Audit core on-page tags on indexable HTTP 200 pages only (never on redirects or noindex pages).

## `checkStructuredData` (type: `boolean`):

Check JSON-LD for valid JSON and a present @type. Syntactic/structural validation only, no schema.org scoring.

## `checkHreflang` (type: `boolean`):

Validate hreflang codes, self-reference and conflicting language targets.

## Actor input object example

```json
{
  "urls": [
    "https://httpbingo.org/html"
  ],
  "datasetUrlField": "url",
  "maxUrls": 1000,
  "sampleSitemapUrls": 200,
  "maxRedirects": 10,
  "timeoutSecs": 20,
  "maxConcurrency": 8,
  "treatWarnAsFail": false,
  "checkRobotsTxt": true,
  "checkOnPage": true,
  "checkStructuredData": true,
  "checkHreflang": true
}
```

# Actor output Schema

## `results` (type: `string`):

One PASS/WARN/FAIL record per audited URL.

## `summary` (type: `string`):

Counts, deployment gate, site checks and issue frequencies.

## `htmlReport` (type: `string`):

Self-contained, safely escaped report for browser review.

## `markdownSummary` (type: `string`):

Portable audit summary for tickets, docs or pull requests.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://httpbingo.org/html"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("burly_bat/site-launch-migration-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://httpbingo.org/html"] }

# Run the Actor and wait for it to finish
run = client.actor("burly_bat/site-launch-migration-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://httpbingo.org/html"
  ]
}' |
apify call burly_bat/site-launch-migration-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=burly_bat/site-launch-migration-audit",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XyW8rBxOVzSYlp3k7/builds/BNPqHZaFBfoOSKunm/openapi.json