# Polite web scraper (/automation--web-scraper)

/automation--web-scraper is a Claude Code skill in the Code section. Builds a scraper that checks robots.txt and terms of service, throttles requests, handles pagination and saves results so runs can resume.

- Web version: https://skills.sgomez.dev/en/s/automation--web-scraper
- Section: [Code](https://skills.sgomez.dev/en/code.md)
- Author: Santiago Gómez de la Torre
- License: MIT
- Source: https://github.com/sgomez-dev/claude-skills/blob/main/skills/automation/web-scraper.md
- Updated 10 Jul 2026

## How to ask for it

- `/automation--web-scraper build a scraper that respects this site's robots.txt`
- `/automation--web-scraper extract prices from this site across paginated results`
- `/automation--web-scraper store the scraped data in a database`

## Install

macOS · Linux:

```
curl -fsSL https://raw.githubusercontent.com/sgomez-dev/claude-skills/main/install.sh | bash
```

Windows:

```
irm https://raw.githubusercontent.com/sgomez-dev/claude-skills/main/install.ps1 | iex
```

Claude Code plugin:

```
/plugin marketplace add sgomez-dev/claude-skills
/plugin install automation-skills@claude-skills-collection
```

## Permissions

- Reads: `package.json`, `requirements.txt`, `pyproject.toml`, `*.py`, `*.js`, `*.ts`, `.env.example`
- Writes: `scraper/**`, `scrape_*.py`, `scrape_*.js`, `scrape_*.ts`, `data/**`, `.env.example`
- Runs: `node`, `npm`, `npx`, `python`, `pip`, `uv`
- Network: Yes
- Destructive: No

## Author's description

Build a polite web scraper — robots.txt, rate limits, selectors, pagination, storage

- [How we review this](https://skills.sgomez.dev/en/methodology.md)