If you're developing a web crawler, learning how crawling works, or simply want to see how a crawler behaves on different types of websites, it helps to have websites that are specifically designed for testing.
Although publicly accessible websites can often be crawled, repeatedly testing software against a live commercial website isn't always appropriate. You may encounter restrictions in robots.txt or the site's terms, and aggressive crawling can place unnecessary load on someone else's server.
Fortunately, there are websites specifically created for developers, students and researchers to practise crawling.
Here are some useful sites to try with SEO Web Spider or another web crawler.
Site Example
Website: https://www.site-example.com/
Recommended: start here before trying the other sites below.
Site Example is a straightforward test site for basic crawling and discovery. It has only around 25 pages, so crawl results stay easy to follow rather than becoming confusing.
It has a useful mix of internal links, page structures, images, redirects, 404 pages, broken images and JavaScript files, making it a nice place to start when demonstrating how a website crawler works.
It's particularly useful for testing:
- Internal link discovery
- Page structures
- Images
- Redirects
- 404 pages
- Broken images
- JavaScript files
- Small-site crawl scope
Difficulty: Beginner
Books to Scrape
Website: https://books.toscrape.com/
Books to Scrape is one of the best websites for learning how a crawler discovers and follows links.
The site contains a fictional online bookshop with around 1,000 products arranged into categories and paginated listing pages.
It's particularly useful for testing:
- Internal link discovery
- Crawl depth
- Pagination
- Category structures
- Page titles
- Meta descriptions
- Headings
- Images
- Product pages
- HTTP responses
Once you're comfortable with a small site such as Site Example, move here to see how a crawler handles a larger, paginated structure. You should quickly see the site's structure emerge as categories, pagination and individual book pages are discovered.
Difficulty: Beginner
Quotes to Scrape
Website: https://quotes.toscrape.com/
Quotes to Scrape was created specifically for people learning how crawlers handle real website behaviours.
The standard version is deliberately simple, containing quotes, authors, tags and pagination. However, alternative versions of the site introduce more challenging technologies and behaviours.
This makes it useful for testing how crawling software responds to different website implementations.
Good for testing:
- Pagination
- Internal links
- Repeated page templates
- Tags and categories
- JavaScript
- AJAX
- Infinite scrolling
- Delayed content
Difficulty: Beginner to advanced
Scrape This Site
Website: https://www.scrapethissite.com/
Scrape This Site provides several purpose-built crawling sandboxes rather than a single test website.
These exercises include simple HTML pages as well as more complicated examples involving forms, pagination and dynamically loaded content.
It's a useful next step once you've successfully crawled a straightforward website such as Site Example or Books to Scrape.
Good for testing:
- HTML parsing
- Tables
- Pagination
- Forms
- JavaScript
- AJAX
- Navigation
- Data extraction
Difficulty: Beginner to intermediate
Web Scraper Test Sites
Website: https://webscraper.io/test-sites
Web Scraper provides a collection of test websites designed to demonstrate different website structures and technologies.
Several examples resemble small e-commerce websites, making them particularly useful when testing crawler behaviour.
You can experiment with traditional links as well as pages containing AJAX and infinite scrolling.
Good for testing:
- E-commerce structures
- Categories
- Product listings
- Product pages
- Pagination
- AJAX
- Infinite scrolling
- JavaScript-generated content
Difficulty: Intermediate
Scrapify Playground
Website: https://scrapifydatalabs.com/
Scrapify Playground provides a large collection of pages created specifically for crawler testing.
It includes dozens of different scenarios, ranging from straightforward static HTML to pages involving JavaScript, lazy loading and infinite scrolling.
This makes it particularly useful when you want to test more than basic link discovery.
Good for testing:
- Static HTML
- JavaScript rendering
- Pagination
- Lazy-loaded images
- Infinite scrolling
- APIs
- Different page structures
Difficulty: Beginner to advanced
Catalog108
Website: https://practice.scrapingcentral.com/
Catalog108 provides a considerably larger test environment than many crawling practice sites.
It contains thousands of fictional products alongside articles and other types of content, making it useful when you want to give a crawler a more substantial workload.
Good for testing:
- Larger crawls
- Product catalogues
- Pagination
- Filters
- Articles
- Structured data
- JSON-LD
- URL discovery
- Crawl performance
Difficulty: Intermediate to advanced
Boot.dev Web Scraping Sandbox
Website: https://learnwebscraping.dev/
The Boot.dev Web Scraping Sandbox provides another fictional website designed specifically for learning and testing crawler behaviour.
Its e-commerce-style structure makes it useful for understanding how crawlers move from a homepage through categories and listings to individual product pages.
Good for testing:
- Site architecture
- Category pages
- Product listings
- Product pages
- Pagination
- Structured content
- Internal linking
Difficulty: Beginner
Which Site Should You Start With?
If you're testing a crawler for the first time, Site Example is an excellent place to start.
Enter https://www.site-example.com/ as your starting URL and begin the crawl.
You can then examine how the crawler discovers the site's internal links, redirects, broken images and JavaScript files, without producing an overwhelming number of results.
Once you're comfortable with a small, straightforward website, move on to sites that introduce larger crawls, pagination, JavaScript rendering, infinite scrolling and other challenges.
A useful progression is:
Site Example → Books to Scrape → Scrape This Site → Scrapify Playground
This takes you from a small, easy-to-follow crawl to increasingly complex website structures.
Crawling Isn't the Same as Scraping
Although the terms are often used together, crawling and scraping aren't quite the same thing.
A web crawler discovers URLs by requesting pages, examining their links and following those links to other pages.
A web scraper is primarily concerned with extracting particular information from those pages, such as product names, prices or contact information.
SEO crawlers combine elements of both.
SEO Web Spider crawls a website and extracts information needed to analyse its search-engine optimisation, including page titles, meta descriptions, headings, links, indexability information and other technical SEO data.
Always Crawl Responsibly
Just because a website is publicly accessible doesn't necessarily mean you should crawl it without restriction.
Before crawling a website you don't own, consider its robots.txt directives, terms of use and the effect your crawler could have on the server.
Keep crawl rates sensible and avoid placing unnecessary load on websites.
For development, demonstrations and training, purpose-built crawler test sites such as those listed above are an excellent choice because they allow you to experiment without using an unsuspecting website as your test environment.
Test Your Website with SEO Web Spider
SEO Web Spider is a professional-grade website crawler designed for technical SEO analysis.
Use it to discover pages, analyse titles and meta descriptions, inspect headings and internal links, identify indexability issues and understand how search engines can navigate through a website.
Whether you're crawling a small business website or analysing hundreds of thousands of URLs, SEO Web Spider gives you the data you need to investigate a site's technical SEO.