# Site crawl with scrape-it-now (/automation--scrape-it-now)

/automation--scrape-it-now is a Claude Code skill in the Code section. Sets up and runs a site-wide crawl with scrape-it-now to get clean markdown, locally or on Azure, with optional indexing.

- Web version: https://skills.sgomez.dev/en/s/automation--scrape-it-now
- Section: [Code](https://skills.sgomez.dev/en/code.md)
- Author: Santiago Gómez de la Torre
- License: MIT
- Source: https://github.com/sgomez-dev/claude-skills/blob/main/skills/automation/scrape-it-now.md
- Updated 16 Sept 2026

## How to ask for it

- `/automation--scrape-it-now crawl my whole site and convert it to markdown`
- `/automation--scrape-it-now configure the scrape-it-now job to run on Azure`
- `/automation--scrape-it-now index the crawled content from my docs site`

## Install

macOS · Linux:

```
curl -fsSL https://raw.githubusercontent.com/sgomez-dev/claude-skills/main/install.sh | bash
```

Windows:

```
irm https://raw.githubusercontent.com/sgomez-dev/claude-skills/main/install.ps1 | iex
```

Claude Code plugin:

```
/plugin marketplace add sgomez-dev/claude-skills
/plugin install automation-skills@claude-skills-collection
```

## Permissions

- Reads: `pyproject.toml`, `requirements.txt`, `uv.lock`, `.env`, `.env.example`, `*.py`, `Makefile`
- Writes: `.scrape-it-now/**`, `data/**`, `scrape/**`, `.env.example`, `Makefile`, `*.sh`
- Runs: `python`, `pip`, `uv`, `pipx`, `scrape-it-now`, `playwright`, `pandoc`
- Network: Yes
- Destructive: No

## Author's description

Run a site-wide markdown crawl with scrape-it-now — job config, local vs Azure, indexing

- [How we review this](https://skills.sgomez.dev/en/methodology.md)