Skip to content
SEO Web Spider
Download Now

Getting started

Quick Start

Open a website, analyse a page, run your first crawl and review the results. Save reports or export CSV with Pro.

3 min read

On this page

Open a Website

  1. Open SEO Web Spider.
  2. Click the address bar.
  3. Enter a website address, such as https://www.example.com/.
  4. Press Enter.

You can also type a search query. The browser opens Google by default.

The address bar can suggest previously crawled websites. Suggestions show the host, the full URL and a crawl count where the site has been crawled more than once.

Analyse One Page

  1. Open a page in the Browser tab.
  2. Choose Analyse page.
  3. Review the Page Analysis sidebar.

The sidebar shows title, meta description, H1, element counts, word count, path, image information, canonical URL, robots meta, first paragraph, document outline, top phrases, links, images, robots.txt and sitemap.xml. It refreshes automatically when a new browser page finishes loading while the sidebar is open.

Crawl a Website

  1. Open the website you want to crawl.
  2. Choose a crawl speed.
  3. Choose Start.
  4. Review the Crawl Results tab while the crawl runs.

Free users can use Gentle mode. Pro users can also use Balanced and Fast modes.

The crawler follows pages and assets on the same host as the crawl start URL. For example, www.example.com and shop.example.com are treated as different hosts.

Pause, Resume or Stop

Use:

  • Pause to temporarily stop scheduling more crawl work;
  • Resume to continue a paused crawl;
  • Stop to end the crawl.

The app reports the number of crawled URLs, queued URLs, errors and elapsed time.

Review Results

The Crawl Results section has two tabs:

  • Internal Links shows rendered pages and assets found on the crawled host.
  • External Links shows link targets discovered outside the crawled host.

Click a URL in the results table to open URL Details. The details drawer includes overview data, inlinks, outlinks, resources, redirects and response headers.

Check External Headers

After the crawl stops or finishes, open the External Links tab and choose Check headers to request headers for discovered external link targets. This optional check updates content type, status code and status without crawling the external pages as part of the site crawl.

Export a CSV with Pro

CSV export requires Pro. Choose Export CSV after a crawl has produced results. The app saves the visible Internal Links columns as a CSV file in the location you choose.

The default file name includes the crawled host and the current date.

Save a Report with Pro

Pro can save crawl reports and open them again later. Save a report before starting another crawl if you want to return to its results. Saved reports are separate from the address-bar history, which remembers website addresses rather than restoring results.

Pro also supports importing a list of URLs. The location and wording of report and import controls can vary by version.

Understand the Numbers

The table contains both raw observations and calculated classifications. For example, Status Code is observed from HTTP, while Indexability Status is derived from response, robots, noindex and canonical information.

Use these detailed references when reviewing a report:

Still have a question? We’re here to help.

Back to top

Technical reference adapted from the version 0.1.0 documentation; Free and Pro features reflect the current product comparison. Adapted from London Web Factory’s product help.