SiteOne Crawler
Crawls an entire site, scores it, and can export the whole thing to browsable offline HTML.
Homebrew formula Direct download
What it does
SiteOne Crawler walks a site and runs dozens of analyses over what it finds — SEO, performance, accessibility, security headers, best practices, content problems, broken links — then produces an interactive HTML report with a 0.0–10.0 quality score. Reports can also be emitted as JSON or text for CI.
Beyond auditing it can:
- export the site to a browsable offline archive of local HTML files, with images, CSS, JS and fonts rewritten to work without a server,
- convert pages to Markdown,
- generate sitemaps,
- render each page in real Chromium via the DevTools Protocol, so JavaScript-heavy and SPA sites crawl correctly.
siteone-crawler --url=https://example.com
siteone-crawler --url=https://example.com --offline-export-dir=./archiveNotes
Available from the author’s tap (janreges/tap). There is also a cross-platform GUI build if a report UI is preferable to a terminal. Only crawl sites you own or have permission to test, and keep concurrency low against anything you do not control.
Alternative to
| Alternative | Type | Trade-off |
|---|---|---|
| Screaming Frog SEO Spider | Freemium | The commercial standard for crawl-based site audits |
| Sitebulb | Freemium | Audit-focused, with a heavier reporting layer |
| Lighthouse and Lighthouse CI | Free | Per-page scoring rather than whole-site crawling |
| wget –mirror or HTTrack | Open source | Offline mirroring only, no analysis |
Install
brew tap janreges/tap
brew install janreges/tap/siteone-crawler