Settings
Open Settings from the application menu.
The current Settings page lets you change the crawler User Agent.
| Setting | Meaning |
|---|---|
| User Agent | The HTTP User Agent sent by rendered crawler workers, robots.txt requests and direct asset requests. |
| Reset to default | Restores the default SEO Web Spider User Agent. |
The original version 0.1.0 guide records this default User Agent. Check the value in Settings for your installed release:
LWFSEOWebSpider/0.1.0 (+https://www.londonwebfactory.com/)
Persisted Preferences
The app keeps these local preferences:
- accepted licence agreement version;
- crawler User Agent;
- recent crawl URLs;
- internal result table column widths;
- hidden internal result columns;
- URL details drawer width;
- cached product messages;
- confirmed registration details.
Older saved column settings that mention removed columns are ignored safely.
Local Files
| File | Purpose |
|---|---|
settings.json |
User Agent, accepted licence version, result-column preferences, drawer width, cached messages and registration state. |
crawl-history.sqlite |
Previously started crawl URLs, first/last crawl times and crawl count. |
crawl-data/current-crawl.sqlite |
Queue, summary rows and detailed data for the current working crawl. |
crawl-data/current-crawl.sqlite-wal / -shm |
Temporary SQLite write-ahead logging files that can exist while the database is open. |
Do not edit or copy an active SQLite database and its companion files while the app is running. Close the app first if a support or backup procedure requires access to them.
Crawl History
The address bar suggestions are backed by local crawl history. The app stores recently crawled website URLs and a crawl count for each entry.
The suggestions are local to the installation. They are not a shared account history.
Crawl Data
The active crawl is stored in a local SQLite database. The database holds:
- crawl frontier;
- URL summaries;
- link and resource occurrences;
- redirect chains;
- response headers;
- external link header-check data.
URL identities and occurrence records are stored separately. This allows the app to group external targets, calculate Inlinks, and open URL Details without placing every occurrence in the main results row.
The app does not store full page source, rendered HTML or long page text extracts for normal crawl rows.
It does store URLs, query strings, titles, descriptions, short link/alt text, selected response headers and errors. These can contain confidential or personal information depending on the site. Treat the database and exports as potentially sensitive.
Starting Another Crawl
Starting a new crawl replaces the previous working crawl database. In Pro, save a crawl report or export the Internal Links results to CSV before starting another crawl if you need to retain them.
CSV Exports with Pro
CSV export requires Pro and writes Internal Links results to the file location you choose. The default filename uses:
seo-crawl;- the crawled host;
- the current date;
.csv.
The app streams the export to disk, which helps with large reports.
Only visible Internal Links columns are exported. External Links, occurrence details, redirect hops and headers are not currently part of the CSV export.
Saved Reports and URL Lists in Pro
Pro can save crawl reports and open saved reports again. This lets you retain a report separately from the current working crawl database. Save a report before starting another crawl when its results need to be kept.
Pro also supports importing a list of URLs. The wording and location of report and import controls can vary by application version.
Settings Compatibility
When settings load, the app validates column IDs and width values. Unknown IDs from an older release are discarded, invalid widths are constrained, and a saved state that would hide every column is reset. This prevents obsolete column preferences from breaking the table after an upgrade.
Crawl History Is Not a Result Archive
Address-bar history stores the crawl URL and how often it has been started. It does not retain a separate results database for each historical crawl. Selecting a history suggestion opens the website; it does not restore an old report. In Pro, use the saved-report feature to reopen a report you previously saved.
Registration Data
When you register, the app sends the registration key to the London Web Factory validation service over HTTPS. A successful response may return registration details such as name, email, licence identifier and expiry date. The confirmed registration is stored locally.
Temporary validation failures during startup should preserve the last confirmed registration.
Product Messages
The app can fetch London Web Factory managed messages for:
- splash screen;
- footer or advert status area;
- administrator message area.
The last valid message can be cached locally so the app can show it while refreshing in the background.