# Sitemap, robots.txt & RSS Change Monitor (`enfex/source-change-monitor`) Actor

Monitor public robots.txt, sitemap XML, and RSS/Atom source changes for SEO and content intelligence without browser automation.

- **URL**: https://apify.com/enfex/source-change-monitor.md
- **Developed by:** [Marcel K](https://apify.com/enfex) (community)
- **Categories:** SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.60 / 1,000 checked targets

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap, robots.txt & RSS Change Monitor

Monitor public website source files that often reveal SEO, content, and publishing changes: `robots.txt`, sitemap XML, and RSS/Atom feeds. The Actor stores a normalized snapshot hash per target and outputs changed targets by default.

### What it does

- Checks public HTTPS websites without browser automation.
- Reads `robots.txt`, sitemap XML URLs, and RSS/Atom feed links.
- Normalizes volatile fields such as fetch duration and check time out of the comparison hash.
- Compares the current source snapshot to the previously stored baseline.
- Outputs changed targets with `changeTypes` such as `robots`, `sitemaps`, `feeds`, and `status`.

### Good use cases

- SEO teams monitoring sitemap or robots.txt regressions.
- Content teams watching public RSS/Atom source changes.
- Agencies checking client websites after deploys.
- RAG/data-ingestion teams tracking source-file drift.

### Input

- `targets`: domains or HTTPS URLs to check.
- `maxTargets`: safety cap, maximum 25 targets in v1.
- `maxSitemaps`: maximum sitemap URLs fetched per target, max 5.
- `maxFeeds`: maximum RSS/Atom feeds fetched per target, max 5.
- `includeUnchanged`: include targets that have not changed since the previous baseline.
- `baselineKey`: namespace for separate schedules, clients, or environments.

Example input:

```json
{
  "targets": ["example.com"],
  "maxTargets": 10,
  "maxSitemaps": 3,
  "maxFeeds": 2,
  "includeUnchanged": false,
  "baselineKey": "production"
}
```

### Output

Each dataset item represents one checked target that changed, or one included unchanged target when `includeUnchanged` is enabled.

Important fields:

- `normalizedTarget`: normalized HTTPS target URL.
- `status`: `ok`, `partial`, or `error`.
- `changed`: whether this target differs from the stored baseline.
- `changeTypes`: source groups that changed.
- `robots`, `sitemaps`, `feeds`: source-specific fetch metadata and extracted links.
- `warnings`: non-fatal warning codes or messages.
- `snapshotHash`: stable hash used for future comparisons.

### Pricing

This Actor uses Apify Pay Per Event with one custom event:

| Event | When it is charged | Current price |
| --- | --- | ---: |
| `checked-target` | Before each public target is fetched and checked, including unchanged checks that may be omitted from output | `$0.0006` |

That equals **$0.60 per 1,000 checked targets**. Pricing is target-based rather than dataset-item based because unchanged checks still consume runtime while `includeUnchanged=false` omits unchanged dataset rows.

Platform usage is included in the Actor price on the public Store pricing page.

### Limitations

- Public HTTPS targets only.
- No browser automation, screenshots, login pages, or cookie-based crawling.
- No private-network fetching and no PII enrichment.
- Sitemap and feed fetch counts are intentionally capped for predictable low-cost runs.
- A `partial` status can still be useful: it means at least one source returned a warning while other source checks completed.

### Recommended workflow

1. Run once with a stable `baselineKey` to create the first baseline.
2. Schedule repeat runs with the same `baselineKey`.
3. Keep `includeUnchanged=false` for compact change-only datasets.
4. Enable `includeUnchanged=true` only when you need a full audit trail.

# Actor input Schema

## `targets` (type: `array`):

Public HTTPS URLs or bare domains to monitor.

## `maxTargets` (type: `integer`):

Maximum number of unique targets processed in one run. Hard-capped at 25 for low-cost v1 runs.

## `maxSitemaps` (type: `integer`):

Maximum sitemap URLs fetched per target after reading robots.txt.

## `maxFeeds` (type: `integer`):

Maximum RSS or Atom feed URLs fetched per target after feed discovery.

## `includeUnchanged` (type: `boolean`):

When false, only changed targets are pushed to the dataset after the first baseline run.

## `baselineKey` (type: `string`):

Namespace for stored snapshots, e.g. production, client-a, weekly.

## Actor input object example

```json
{
  "targets": [
    "example.com"
  ],
  "maxTargets": 10,
  "maxSitemaps": 3,
  "maxFeeds": 2,
  "includeUnchanged": false
}
```

# Actor output Schema

## `results` (type: `string`):

Default dataset items. Each item contains the normalized target, change status, changed source groups, source metadata, warnings, and snapshot hash.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "targets": [
        "example.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("enfex/source-change-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "targets": ["example.com"] }

# Run the Actor and wait for it to finish
run = client.actor("enfex/source-change-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "targets": [
    "example.com"
  ]
}' |
apify call enfex/source-change-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=enfex/source-change-monitor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/xtUoP2iJuHt8gDQPb/builds/rr1180iKVsfVW42AY/openapi.json