# X/Twitter Scraper — Tweets, Profiles & Engagement Data (`oneary/twitter-scraper`) Actor

Scrape Twitter/X data at scale. Extract tweets, profiles, hashtags, trends, and engagement metrics for social media analytics.

- **URL**: https://apify.com/oneary/twitter-scraper.md
- **Developed by:** [Luan M.](https://apify.com/oneary) (community)
- **Categories:** Social media
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$10.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Twitter/X Data Scraper

[![Apify Actor](https://img.shields.io/badge/Apify-Actor-blue)](https://apify.com/)
[![Node.js](https://img.shields.io/badge/Node.js-18+-green)](https://nodejs.org/)
[![Crawlee](https://img.shields.io/badge/Crawlee-3.x-orange)](https://crawlee.dev/)

A powerful, production-ready **Twitter/X Data Scraper** built on [Apify](https://apify.com/) and [Crawlee](https://crawlee.dev/). Extracts tweets, retweets, replies, likes, user profiles, hashtags, mentions, and media URLs from Twitter/X — all with minimal configuration.

### Features

- **Multi-mode Scraping** — Search by keyword, scrape user timelines, or provide arbitrary X.com URLs
- **Rich Tweet Data** — Extracts tweet text, timestamp, engagement metrics (likes, retweets, replies, views), tweet IDs, and URLs
- **User Profile Info** — Bio, followers count, following count, location, website, join date, avatar, and banner image
- **Media Extraction** — URLs for images, videos, and GIFs embedded in tweets
- **Hashtags & Mentions** — Extracted automatically from each tweet
- **Reply & Retweet Filters** — Optionally include or exclude replies and retweets
- **Proxy Support** — Built-in Apify proxy integration with residential proxy group support
- **Configurable** — Max tweets, concurrency, retries, and more
- **Headless Browser** — Uses Playwright with Chromium for reliable JavaScript-rendered page extraction

### Input Configuration

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `startUrls` | Array | `["https://x.com/elonmusk"]` | Starting URLs (profile pages, search results) |
| `searchQuery` | String | — | Search query (overrides `startUrls`). Supports operators like `from:username`, `has:hashtags`, `lang:en` |
| `username` | String | — | Scrape a specific user's timeline (without @). Overrides `startUrls` |
| `maxTweets` | Integer | `100` | Maximum tweets to scrape (0 = unlimited) |
| `includeReplies` | Boolean | `false` | Include replies in user timeline scrape |
| `includeRetweets` | Boolean | `true` | Include retweets in user timeline scrape |
| `proxyConfiguration` | Object | Apify RESIDENTIAL | Proxy settings to avoid IP bans |
| `maxRequestRetries` | Integer | `3` | Retry limit for failed requests |
| `maxConcurrency` | Integer | `5` | Concurrent browser pages |
| `extractTweetIds` | Boolean | `true` | Include tweet IDs and URLs |
| `extractUserInfo` | Boolean | `true` | Include author profile data |
| `extractMedia` | Boolean | `true` | Extract image/video URLs |

### Output Dataset

Each tweet is stored as a dataset item with the following structure:

```json
{
  "text": "The future of AI is exciting!",
  "timestamp": "2025-05-30T12:00:00.000Z",
  "tweetId": "1234567890123456789",
  "tweetUrl": "https://x.com/username/status/1234567890123456789",
  "replyCount": 42,
  "retweetCount": 128,
  "likeCount": 1024,
  "viewCount": 50000,
  "isReply": false,
  "isRetweet": false,
  "hashtags": ["AI", "tech"],
  "mentions": ["@openai"],
  "mediaUrls": ["https://pbs.twimg.com/media/..."],
  "user": {
    "username": "elonmusk",
    "displayName": "Elon Musk",
    "avatarUrl": "https://pbs.twimg.com/profile_images/...",
    "profileUrl": "https://x.com/elonmusk"
  },
  "profile": {
    "username": "elonmusk",
    "displayName": "Elon Musk",
    "bio": "Technology entrepreneur",
    "followersCount": 180000000,
    "followingCount": 1500,
    "location": "Austin, TX",
    "website": "https://example.com",
    "avatarUrl": "https://pbs.twimg.com/profile_images/...",
    "bannerUrl": "https://pbs.twimg.com/profile_banners/..."
  },
  "sourceUrl": "https://x.com/elonmusk",
  "scrapedAt": "2025-05-30T12:05:00.000Z"
}
```

### Quick Start

```bash
## Install dependencies
npm install

## Run locally (requires Apify token)
npx apify run -p

## Or run directly
node src/main.js
```

### Deployment to Apify

1. Push this repository to GitHub
2. Go to [Apify Console](https://console.apify.com) → Create Actor → Import from GitHub
3. Set up environment variables in Apify Console as needed
4. Build and run!

### Environment Variables

| Variable | Description |
|----------|-------------|
| `APIFY_TOKEN` | Your Apify API token (required for cloud proxy) |
| `APIFY_PROXY_PASSWORD` | Apify proxy password |
| `APIFY_LOCAL_STORAGE_DIR` | Local storage directory for development |

### Technical Details

- **Runtime**: Node.js 18+
- **Browser Engine**: Chromium via Playwright
- **Crawler**: Crawlee PlaywrightCrawler
- **Data Storage**: Apify dataset
- **Proxy**: Apify Proxy (RESIDENTIAL recommended)

### Limitations & Best Practices

- **Rate Limiting**: Twitter/X aggressively rate-limits scraping. Use residential proxies and reasonable concurrency.
- **Login Walls**: Some pages may require authentication. For full access, consider adding cookie-based session management.
- **DOM Changes**: This scraper relies on Twitter's DOM structure (`data-testid` attributes). If Twitter updates their UI, selectors may need adjustment.
- **Ethical Use**: Respect Twitter's Terms of Service and robots.txt. Use responsibly and consider rate limiting.

### License

Apache 2.0

# Actor input Schema

## `startUrls` (type: `array`):

List of Twitter/X profile or search URLs to start scraping from. E.g., https://x.com/elonmusk or https://x.com/search?q=AI\&src=typed\_query

## `searchQuery` (type: `string`):

A search query to find tweets. If provided, startUrls are ignored and the search endpoint is used. Supports operators like 'from:username', 'has:hashtags', 'lang:en'.

## `username` (type: `string`):

Scrape tweets from a specific user's timeline. E.g., elonmusk (without @). Overrides startUrls if both are provided.

## `maxTweets` (type: `integer`):

Maximum number of tweets to scrape. Set to 0 for unlimited (subject to time limits).

## `includeReplies` (type: `boolean`):

Include replies in the scraped data when scraping a user timeline.

## `includeRetweets` (type: `boolean`):

Include retweets in the scraped data when scraping a user timeline.

## `proxyConfiguration` (type: `object`):

Apify proxy configuration for avoiding IP bans. Recommended to use residential proxies.

## `maxRequestRetries` (type: `integer`):

Maximum number of retries for failed requests.

## `maxConcurrency` (type: `integer`):

Maximum number of concurrent browser pages.

## `extractTweetIds` (type: `boolean`):

Include full tweet URLs and IDs in output.

## `extractUserInfo` (type: `boolean`):

Include author profile info (followers, following, bio, etc.) with each tweet.

## `extractMedia` (type: `boolean`):

Extract URLs of images, videos, and GIFs attached to tweets.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://x.com/elonmusk"
    }
  ],
  "maxTweets": 100,
  "includeReplies": false,
  "includeRetweets": true,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  },
  "maxRequestRetries": 3,
  "maxConcurrency": 5,
  "extractTweetIds": true,
  "extractUserInfo": true,
  "extractMedia": true
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://x.com/elonmusk"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": [
            "RESIDENTIAL"
        ]
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("oneary/twitter-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://x.com/elonmusk" }],
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"],
    },
}

# Run the Actor and wait for it to finish
run = client.actor("oneary/twitter-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://x.com/elonmusk"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}' |
apify call oneary/twitter-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=oneary/twitter-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/e6zDYiKaJBbhUXYGk/builds/8AtfpFnyuD87WERNs/openapi.json