scrapy-docs-crawler
description (verbatim frontmatter)
Keep the docs/ mirrors fresh with the Scrapy sitemap crawler (crawlers/): md-first fetching (.md variant tried per-URL, deterministic HTML distiller as fallback), bloom-filter cross-run URL skipping, and sha256 CDC so unchanged pages never rewrite. Use to refresh claude.com/anthropic.com/support/platform/code/scrapyd/scrapy mirrors, on "crawl the docs", "update mirrors", "refresh claude.com blog", or /scrapy-docs-crawler. Small-scale by design — sitemaps only, polite throttle, hard page caps.
summary
Keep the docs/ mirrors fresh with the Scrapy sitemap crawler (crawlers/): md-first fetching (.md variant tried per-URL, deterministic HTML distiller as fallback), bloom-filter cross-run URL skipping, and sha256 CDC so unchanged pages never rewrite.
source
plugins/cwc-engineering/skills/scrapy-docs-crawler/SKILL.md
provenance
created 2026-07-10 18:13:13 · JSON