#Screaming frog seo spider get browser statistics how to

#Screaming frog seo spider get browser statistics how to

Please see our guide on how to crawl very large websites for more detail. We actually recommend this as the default setting for any users with an SSD (or faster drives), as it’s just as fast and uses much less RAM. This means the SEO Spider will hold as much data as possible within RAM (up to the user allocation), and store the rest to disk. Users can select to save to disk by choosing ‘database storage mode’, within the interface (via ‘Configuration > System > Storage). You don’t need to keep increasing RAM to do it either, switch to database storage instead. So if you need to crawl millions of URLs using a desktop crawler, you really can. That’s why early last year we introduced the first configurable hybrid storage engine, which enables the SEO Spider to crawl at truly unprecedented scale for any desktop application while retaining the same, familiar real-time reporting and usability. However, it’s not very scalable for crawling large websites.

The SEO Spider has traditionally used RAM to store data, which has enabled it to crawl lightning-fast and flexibly for virtually all machine specifications. This will mean new URLs discovered will appear in the interface, and orphan pages will appear under the respective filter in the Analytics and Search Console tabs (after performing crawl analysis). This is available under ‘Configuration > API Access’ and then either ‘Google Analytics’ or ‘Google Search Console’ and their respective ‘General’ tabs. If you wish to add any URLs discovered automatically to the queue, crawl them and see them in the interface, simply enable the ‘Crawl New URLs Discovered in Google Analytics/Search Console’ configuration. URLs are loaded, data is matched against URLs in the crawl, and any orphan URLs (URLs discovered only in GA or GSC) are available via the ‘ Orphan Pages‘ report export. If you connect to Google Analytics or Search Console via the API, by default any new URLs discovered are not automatically added to the queue and crawled. 2) Crawl New URLs Discovered In Google Analytics & Search Console The data in the export will be in the exact same order and include all of the exact URLs in the original upload, including duplicates, normalisation or any fix-ups performed. The standard export buttons on the dashboard will otherwise export URLs in order based upon what’s been crawled first, and how they have been normalised internally (which can appear quite random in a multi-threaded crawler that isn’t in usual breadth-first spider mode).

If you’ve uploaded a list of URLs into the SEO Spider, performed a crawl and want to export them in the same order they were uploaded, then use the ‘Export’ button which appears next to the ‘upload’ and ‘start’ buttons at the top of the user interface. 1) Export A List In The Same Order Uploaded Or at least, how they can be best utilised to help improve auditing.

This post covers some of the lesser-known and hidden-away features, that even more experienced users might not be aware exist. The Screaming Frog SEO Spider has evolved a great deal over the past 8 years since launch, with many advancements, new features and a huge variety of different ways to configure a crawl.