SEO Web Spider is a desktop SEO crawler from London Web Factory. Both Free and Pro combine a visible in-app browser, rendered page analysis, live crawl results and local SQLite storage. Pro adds faster crawling, saved reports, CSV export and URL-list import.
What the App Does
The app lets you:
- open websites in a controlled in-app browser;
- analyse the currently loaded page;
- crawl same-host pages and assets;
- inspect internal and external link targets;
- check external link response headers;
- filter, sort and resize crawl result columns;
- open URL details for crawled rows;
- save and reopen crawl reports, export Internal Links results to CSV and import a list of URLs with Pro;
- keep crawl history and settings locally.
Choosing the Right Guide
- Start with Quick Start for the shortest path through one analysis and crawl.
- Use Crawling a Website for controls, scope and day-to-day crawl operation.
- Read How the Crawler Works when you need the discovery, rendering, normalisation, redirect, robots and storage rules.
- Read How Link Metrics Are Calculated to understand why A Tags, Outlinks, External Outlinks, Inlinks and their unique versions differ.
- Use the Result Variables and Column Glossary as the complete reference for table, Page Analysis and URL Details fields.
Important Notes
Only crawl websites that you own, manage, or have permission to test. Crawling originates from your computer and public IP address. Target websites, hosting providers, network operators or security services may rate-limit, block or blacklist requests.
SEO Web Spider stores crawl data locally in SQLite while the crawl is active. Starting another crawl replaces the current working data. In Pro, save a crawl report or export the Internal Links results to CSV before starting another crawl when you need to retain them.
Free and Pro both support a configurable limit of up to 1,000,000 URLs, including pages and assets. Free runs one crawler at a time. Pro offers 1, 6 or 12 concurrent crawlers, shared between pages and assets. See the Free and Pro feature comparison for availability.