Ways to SEO Optimize Your Site with Screaming Frog

Screaming frog is a useful SEO spider tool that can be used to crawl your website's links, images, CSS, scripts, and apps from the SEO perspective. If you didn't know about it already, then where have you been? :). It is a small desktop software that will let you crawl an entire site on demand, and will help resolve common SEO issues with your website. But crawling isn't all it's good for. It can do much, much more. In this post, we'll show you some of the most important uses of the Screaming Frog SEO Spider tool.


The functionality that Screaming Frog provides is immense. And you can mix-and-match some features to create functionality of your own - the possibilities are countless. We will attempt to create a power tutorial on Screaming Frog soon. But for the time being, just check out these 5 great ways to SEO Optimize your website with Screaming Frog.

1. Verifying crawl errors

Google Webmaster Tools can be often slow to respond to the changes on your site. For example, it might not find all the fixes you've made to your broken links, You can, however use Screaming Frog to find out broken link problems such as 3xx (eg 301), 4xx (eg 404s) and 5xx. At the top, there's a bar where you can insert the URL of a website you need to crawl. Once crawled, you can filter the results by HTML content only, and then if you go to Advanced Export, you can export the links based on their status codes (3xx, 4xx, 5xx etc).

You can, for example export your 404s, and then compare them with the ones you get in Webmaster Tools. You can also take all links from the Webmaster Tools (export as CSV, then copy all links and paste them in a text document). Now, if you go to Mode in Screaming Frog, and select List instead of Spider, you will be able to import a text file, which you can then check for errors. This will tell you all the 404s that are actually there, and will eliminate all the fixes that Google Webmaster Tools missed out on.

2. Creating a sitemap

You can use Screaming Frog to create a sitemap. All you have to do is, go to Mode tab at the top, and click on Spider. Then, from the configuration menu, click on Exclude. Here, you can exclude certain types of URLs. For example, WordPress often adds something like /?replytocom=123 when a user comments (for eg, domain.com/my-post/>replytocom=12). You don't want such URLs to be in your sitemap. So in the exclude menu, you can add this "\?repytocom." (without quotes) to exclude such URLs. Check the Screaming Frog User Guide to find out more about excluding URLs.

Now do a full crawl of your site, and once that is complete, simply export the data by clicking on Advanced Export >> XML Sitemap.

3. Checking a sitemap for errors

Similarly, you can check an XML sitemap for errors. If you have your XML file with you, go into Mode, and select List. Then, Select the XML file, and import it into Screaming Frog. Let it crawl the links. Once done, it will show you all broken links, and any other such problems. This can really help keep your sitemaps fresh and healthy.

4. Check for successful site migration

When you change your site's domain, Google Webmaster Tools often goes berserk, giving tonnes of 404 errors. You can, however, use Screaming Frog to do it quickly for you. If you have a list of your old URLs, you can import them in SF, and then find out which of them are giving you 404s. You can do a full-site crawl as well to find some of the broken links. And you can also import data from Google Webmaster Tools and then check them against any 404 errors.

5. Building links

As we discussed in our link-building exercise, you can generate authentic backlinks to your site by helping other webmasters out. If you do a site crawl for another site, and give them a list of their broken links (source and destination), you can fall into their good graces. It is indeed a great way to increase your outreach and build connections, because it tells others that you're actually reading their blog/website, and care for it. You can use this strategy to build backlinks :)

Bonus: Check out the following post, which explores yet another way to use Screaming frog for backlinking.

Did you like this post? Got any questions? I know this might be very confusing, so don't hesitate to ask in the comments section below. We'd love to help. Cheers :)