# Gvt Procurement Scraper (`zcamper/gvt-procurement-scraper`) Actor Scrape tender notices & contract awards from 11 government procurement databases across 9 countries. Covers US, EU, UK, Ukraine, France, Brazil, Australia & Canada. Filter by keyword, date, and contract value. - **URL**: https://apify.com/zcamper/gvt-procurement-scraper.md - **Developed by:** [Zac](https://apify.com/zcamper) (community) - **Categories:** E-commerce, Lead generation, Other - **Stats:** 18 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / actor start This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Government & Public Procurement Data Scraper A powerful Apify actor for scraping government tender databases and contract awards from multiple sources including SAM.gov, TED EU, and UK Find a Tender. ### Features - **Multi-Source Support**: Simultaneously scrape from multiple government procurement databases - **Flexible Data Extraction**: Extract tender notices, contract awards, or both - **Advanced Filtering**: Filter by keywords, date range, contract value, and more - **Multiple Output Formats**: Export results as JSON, CSV, or JSONL - **Proxy Support**: Built-in support for Apify's residential proxies to avoid blocking - **Error Handling**: Robust error handling with detailed logging - **Rate Limiting**: Automatic rate limiting to respect source websites ### Supported Databases - **SAM.gov** (USA Federal Procurement) - **TED EU** (European Union Tenders) - **UK Find a Tender** (United Kingdom Government Contracts) ### Installation 1. Clone or create this actor in Apify Console 2. Install dependencies: ```bash npm install ```` ### Usage #### Input Parameters ```json { "sources": ["sam.gov", "ted.eu", "uk.findatender"], "dataTypes": ["tender_notices", "contract_awards"], "searchQuery": "software development", "keywords": ["IT", "cloud"], "dateFrom": "2024-01-01", "dateTo": "2024-12-31", "maxResults": 5000, "minContractValue": 50000, "maxContractValue": "5000000", "outputFormat": "json", "useProxy": true, "browserTimeout": 60, "debugMode": false } ``` #### Parameters Explained | Parameter | Type | Description | Default | |-----------|------|-------------|---------| | `sources` | array | Data sources to scrape | `["sam.gov"]` | | `dataTypes` | array | Types of data to extract | `["tender_notices", "contract_awards"]` | | `searchQuery` | string | Search keywords (e.g., "software", "consulting") | `""` | | `keywords` | array | Additional filter keywords | `[]` | | `dateFrom` | string | Start date (YYYY-MM-DD format) | `""` | | `dateTo` | string | End date (YYYY-MM-DD format) | `""` | | `maxResults` | integer | Maximum results to collect (1-10000) | `1000` | | `minContractValue` | number | Minimum contract value in USD | `0` | | `maxContractValue` | string | Maximum contract value in USD | `""` | | `outputFormat` | string | Output format: json, csv, jsonl | `"json"` | | `useProxy` | boolean | Use Apify residential proxy | `false` | | `browserTimeout` | integer | Timeout for browser operations (10-600s) | `60` | | `debugMode` | boolean | Enable verbose logging | `false` | ### Output Results are stored in Apify's Key-Value Store with the following structure: ```json { "id": "notice_id", "source": "sam.gov", "type": "tender_notice", "title": "Tender Title", "description": "Full tender description", "url": "https://...", "postedDate": "2024-01-15T10:30:00Z", "deadline": "2024-02-15", "organization": "Agency Name", "budget": "500000", "metadata": { "country": "US", "category": "Software", "reference": "SOL-2024-001" }, "scrapedAt": "2024-01-20T12:00:00Z" } ``` ### Running Locally ```bash npm start ``` Set environment variable `DEBUG=true` for verbose logging: ```bash DEBUG=true npm start ``` ### API Documentation #### SamGovScraper Scrapes tender opportunities from SAM.gov using their REST API. ```javascript const scraper = new SamGovScraper(input); const results = await scraper.scrape(); ``` #### TedEuScraper Scrapes tender notices from TED EU using browser automation. ```javascript const scraper = new TedEuScraper(input); const results = await scraper.scrape(); ``` #### UkFindTenderScraper Scrapes government contracts from UK Find a Tender using browser automation. ```javascript const scraper = new UkFindTenderScraper(input); const results = await scraper.scrape(); ``` #### DataProcessor Utility class for formatting and filtering results. ```javascript const processor = new DataProcessor('json'); const formatted = processor.format(data); // Helper methods DataProcessor.normalizeTender(tender); DataProcessor.filterByKeywords(tenders, ['IT', 'cloud']); DataProcessor.sortByDate(tenders, 'desc'); DataProcessor.removeDuplicates(tenders); ``` ### Best Practices 1. **Rate Limiting**: The scrapers include built-in rate limiting. Don't remove delays between requests. 2. **Proxy Usage**: For high-volume scraping or repeated runs, enable proxy support to avoid IP blocking. 3. **Search Strategy**: Be specific with search queries to get more relevant results and reduce processing time. 4. **Date Ranges**: Use date filters to limit scope and reduce unnecessary processing. 5. **Keywords**: Use keywords to post-filter results for higher relevance. ### Error Handling The actor captures and logs errors from each source separately. If one source fails, others continue processing. A summary of errors is provided in the output. ### Performance Tips - **Limit Results**: Set `maxResults` to a reasonable number to reduce processing time - **Narrow Date Range**: Use `dateFrom` and `dateTo` to focus on recent tenders - **Use Keywords**: Pre-filter with keywords to reduce unnecessary data collection - **Disable Debug Mode**: Keep `debugMode: false` for production runs for better performance ### Troubleshooting #### No Results Returned - Check if the source website is accessible - Verify search parameters are valid - Try with a broader search query - Check if proxy is needed due to IP blocking #### Timeout Errors - Increase `browserTimeout` parameter - Try with fewer results or narrower date range - Enable proxy support for more reliable connections #### Rate Limit Errors - Reduce `maxResults` or split into smaller date ranges - Increase delays between requests (modify scraper code) - Use Apify residential proxies ### License Apache 2.0 ### Support For issues or questions, please create an issue in the repository or contact the development team. ### Contributing Contributions are welcome! Please follow the existing code style and add tests for new features. #### Adding a New Source 1. Create a new scraper file in `src/scrapers/` (e.g., `newSourceScraper.js`) 2. Implement the scraper class extending the base pattern 3. Add the scraper to `src/main.js` in the scrapers map 4. Update input schema in `.actor/input_schema.json` 5. Add documentation in this README ### Roadmap - \[ ] Add support for additional databases (France, Germany, Canada) - \[ ] Implement contract award extraction for all sources - \[ ] Add email notification on new matching tenders - \[ ] Create web dashboard for tracking results - \[ ] Add machine learning for opportunity relevance scoring - \[ ] Implement incremental scraping with state management # Actor input Schema ## `sources` (type: `array`): Select one or more government procurement databases to scrape. Choose 'All Sources' to scrape everything. ## `samGovApiKey` (type: `string`): Only required if scraping SAM.gov. Get a free key in 30 seconds at https://api.data.gov/signup/ — all other sources work without a key. ## `dataType` (type: `string`): What type of procurement data to collect. ## `searchQuery` (type: `string`): Search terms to find specific tenders (e.g. 'software development', 'medical supplies', 'construction'). Leave empty to get all results. ## `keywords` (type: `array`): Only keep results containing at least one of these keywords in the title or description. Leave empty to keep all results. ## `dateFrom` (type: `string`): Only include tenders posted on or after this date. Format: YYYY-MM-DD (e.g. 2025-01-01). Leave empty for no start date limit. ## `dateTo` (type: `string`): Only include tenders posted on or before this date. Format: YYYY-MM-DD. Leave empty for no end date limit. ## `minContractValue` (type: `number`): Only include contracts worth at least this amount. Set to 0 for no minimum. ## `maxContractValue` (type: `string`): Only include contracts worth less than this amount. Leave empty for no maximum. ## `maxResults` (type: `integer`): Maximum number of results to collect from each source. Lower values run faster and cost less. ## `outputFormat` (type: `string`): Format for the output data file saved to Key-Value Store. ## `useProxy` (type: `boolean`): Route requests through Apify's residential proxy to avoid rate limiting. Increases reliability but costs more. Not needed for most sources. ## `browserTimeout` (type: `integer`): How long to wait for each HTTP request before timing out. Increase if you're getting timeout errors. ## `debugMode` (type: `boolean`): Log detailed information for each request. Useful for troubleshooting but produces a lot of output. ## Actor input object example ```json { "sources": [ "sam.gov" ], "dataType": "both", "searchQuery": "", "keywords": [], "dateFrom": "", "dateTo": "", "minContractValue": 0, "maxContractValue": "", "maxResults": 100, "outputFormat": "json", "useProxy": false, "browserTimeout": 60, "debugMode": false } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "maxResults": 100 }; // Run the Actor and wait for it to finish const run = await client.actor("zcamper/gvt-procurement-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "maxResults": 100 } # Run the Actor and wait for it to finish run = client.actor("zcamper/gvt-procurement-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "maxResults": 100 }' | apify call zcamper/gvt-procurement-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=zcamper/gvt-procurement-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Gvt Procurement Scraper", "description": "Scrape tender notices & contract awards from 11 government procurement databases across 9 countries. Covers US, EU, UK, Ukraine, France, Brazil, Australia & Canada. Filter by keyword, date, and contract value.", "version": "0.0", "x-build-id": "mtBxRsZsYwcXtdZQA" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/zcamper~gvt-procurement-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-zcamper-gvt-procurement-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/zcamper~gvt-procurement-scraper/runs": { "post": { "operationId": "runs-sync-zcamper-gvt-procurement-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/zcamper~gvt-procurement-scraper/run-sync": { "post": { "operationId": "run-sync-zcamper-gvt-procurement-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "sources" ], "properties": { "sources": { "title": "Data Sources", "type": "array", "description": "Select one or more government procurement databases to scrape. Choose 'All Sources' to scrape everything.", "items": { "type": "string", "enum": [ "all", "sam.gov", "ted.eu", "uk.findatender", "prozorro.gov.ua", "boamp.fr", "pncp.gov.br", "austender.gov.au", "canadabuys.canada.ca", "ny.data.gov", "eva.virginia.gov", "comptroller.texas.gov" ], "enumTitles": [ "All Sources (scrape all 11 databases)", "SAM.gov — United States (Federal)", "TED — European Union (27 member states)", "Find a Tender — United Kingdom", "ProZorro — Ukraine", "BOAMP — France", "PNCP — Brazil", "AusTender — Australia", "CanadaBuys — Canada", "NY Open Data — New York State (US)", "eVA — Virginia (US)", "Texas — DIR Contracts & TxDOT Projects (US)" ] }, "default": [ "sam.gov" ] }, "samGovApiKey": { "title": "SAM.gov API Key", "type": "string", "description": "Only required if scraping SAM.gov. Get a free key in 30 seconds at https://api.data.gov/signup/ — all other sources work without a key." }, "dataType": { "title": "Data Type", "enum": [ "both", "tender_notices", "contract_awards" ], "type": "string", "description": "What type of procurement data to collect.", "default": "both" }, "searchQuery": { "title": "Search Query", "type": "string", "description": "Search terms to find specific tenders (e.g. 'software development', 'medical supplies', 'construction'). Leave empty to get all results.", "default": "" }, "keywords": { "title": "Filter Keywords", "type": "array", "description": "Only keep results containing at least one of these keywords in the title or description. Leave empty to keep all results.", "items": { "type": "string" }, "default": [] }, "dateFrom": { "title": "Date From", "type": "string", "description": "Only include tenders posted on or after this date. Format: YYYY-MM-DD (e.g. 2025-01-01). Leave empty for no start date limit.", "default": "" }, "dateTo": { "title": "Date To", "type": "string", "description": "Only include tenders posted on or before this date. Format: YYYY-MM-DD. Leave empty for no end date limit.", "default": "" }, "minContractValue": { "title": "Min Contract Value (USD)", "type": "number", "description": "Only include contracts worth at least this amount. Set to 0 for no minimum.", "default": 0 }, "maxContractValue": { "title": "Max Contract Value (USD)", "type": "string", "description": "Only include contracts worth less than this amount. Leave empty for no maximum.", "default": "" }, "maxResults": { "title": "Max Results Per Source", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Maximum number of results to collect from each source. Lower values run faster and cost less.", "default": 100 }, "outputFormat": { "title": "Output Format", "enum": [ "json", "csv", "jsonl" ], "type": "string", "description": "Format for the output data file saved to Key-Value Store.", "default": "json" }, "useProxy": { "title": "Use Residential Proxy", "type": "boolean", "description": "Route requests through Apify's residential proxy to avoid rate limiting. Increases reliability but costs more. Not needed for most sources.", "default": false }, "browserTimeout": { "title": "Request Timeout (seconds)", "minimum": 10, "maximum": 600, "type": "integer", "description": "How long to wait for each HTTP request before timing out. Increase if you're getting timeout errors.", "default": 60 }, "debugMode": { "title": "Debug Mode", "type": "boolean", "description": "Log detailed information for each request. Useful for troubleshooting but produces a lot of output.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```