How To Use Screaming Frog To Improve SEO Content

Screaming Frog SEO Spider is a powerful tool that crawls your website just like a search engine, gathering detailed insights that can be used to boost your site’s SEO content.

In the following guide, we’ll walk you through each step, from setting up your crawl with XML sitemaps and API connections, to analysing the data for actionable improvements. 

Whether you’re an SEO trying a new tool or a business leader trying to improve your business content’s SEO performance, this user-friendly guide will help you harness Screaming Frog’s capabilities to enhance your website’s SEO performance and overall content strategy.

 Want the results without the hassle? Grab a free SEO audit today

How Screaming Frog Can Improve SEO Content

Screaming Frog SEO Spider can provide a comprehensive overview of site health, content performance and content strategy efficiency, making it a great tool in any business leader or SEOs toolbox. 

Screaming Frog can help you:

• Crawl your entire site or specific website categories like your blog content for health and SEO performance
• Identify web content that isn’t yet indexed by Google so you can request indexing
• Identify pages with low word counts or not enough content for effective SEO
• Identify pages without internal links so you can add them to your wider site content
• Assess the number of clicks and impressions each URL is getting in Google:
  • Identify dead URLs or content with zero clicks or impressions for repurposing
  • Identify mid-performing content that needs to be refreshed or updated
  • Identify high-performing URLs to guide future content efforts
• Understand which content users engage with and which they don’t
• Review when the content on each page was last updated to prioritise actions
• Bulk review on-page SEO content like page titles and meta descriptions to identify improvements
• Track when search engines last crawled each URL to indicate if they have dropped off the radar

Honestly, it can do a whole lot more than this, too. Screaming Frog has a range of SEO Spider Tutorials and additional literature available on their site. But we’ll just stick to the content auditing functions we use here at Dandy for the purposes of this guide.

How To Use Screaming Frog To Improve SEO Content

Here’s a step-by-step breakdown of how you can use Screaming Frog to gather all of the juicy data mentioned above. Once Screaming Frog has worked its magic, you’ll have an actionable and data-driven plan for how to improve the SEO performance of each piece of content.

1: Collect XML sitemaps

To run your Screaming Frog crawl, you will need to give the tool all the URLs you want to crawl. To do this, you can usually find your sitemap (a list of the URLs live on your site) by entering your website URL in the browser and then adding one of the following:

• /sitemap.xml
• /sitemap_index.xml

These are the most common URL paths to find sitemaps, but won’t always work. You may need to do some more digging to find yours if you don’t have a sitemap listed in these locations. 

If you do find your sitemaps here, they should look like this if your on WordPress:

If you’re auditing blog content, select the “post-sitemap” link to get a full list of all your blog post URLs, which should look something like this:

⚠️ Don’t add your URLs into Screaming Frog until after you’ve configured your crawl settings. Copy and paste the URL list into a document or note to keep hold of for now. 

2: Set up your API access

Screaming Frog API Configuration Menu

Screaming Frog allows you to connect to external APIs to pull through the data you need into one handy crawl. You can connect to your accounts in Google Search Console, Google Analytics, Ahrefs and even other APIs by clicking them in this master dropdown menu to bring up a settings window.

For Google Search Console:

• Connect your GSC account via the “Account Information” tab.
• You will need to input your target date range under the “Search Analytics” tab, we usually opt for 12 months for our content audits, but you can go shorter, too.
• You’ll also need to “Enable URL inspection” under the URL inspection tab of the configuration settings menu. The crawl will not work if this is not enabled.
• Then hit “Okay” to save these settings.

Search Console API Configuration Menu

For Google Analytics:

• Connect your Google Analytics account via the “Account Information” tab.
• Input your target date range under the “Date Range” tab.
• Choose the on-page metrics you would like to track from GA in the “Metrics” tab.
• You can also edit page dimensions and filters for the data you pull through, allowing you to focus on conversion-type events, if needed.
• Click “Okay” to save these settings.

GA4 API Configuration Menu

3: Add any custom extractions

If there are any additional metrics you need such as publish date or time, you can add custom extractions to your crawl. You’ll find this in the “Configuration” dropdown menu.

Screaming Frog Custom Extraction Configuration Menu

Once you’ve opened the “Custom Extraction” menu, you can add in the data and name the custom extractions. Click “Add” to add a new custom extraction. The following two XPaths should work for extracting the publish date and modified date of URLs on any WordPress website that is using Yoast:

• WP Publish  – //meta[@property=’article:published_time’]/@content
• WP Modified – //meta[@property=’article:modified_time’]/@content

Then, click “Okay” to add these to your crawl.

4: Crawl data

Now that you’ve added all the APIs and extractions you need from your crawl, it’s time to run it. 

We recommend using the “List” mode to upload your URLs for the crawl, you will likely need to edit crawl settings from the auto “Spider” mode in the “Mode” dropdown menu.

Changing Screaming Frog to “List” Mode in Crawl Settings

Once you’re set in list mode, you’ll need that list of URLs from your sitemap that we prepared earlier. Copy the entire list of URLs, then click “Upload” and “Paste” in the top banner of Screaming Frog. 

Uploading Your Sitemap For Screaming Frog to Crawl

All copied URLs will automatically paste into the pop up box, click “Okay” to upload these target URLs and your Screaming Frog crawl will automatically begin. That’s why it’s important to configure all of your crawl settings before you add your URLs and begin the crawl.

5: Exporting your crawl

Once your crawl is complete, it will look something like this:

You can then click “Export” to export your crawl into your preferred spreadsheet format.

6: Analyse crawl data to improve content

You completed your Screaming Frog SEO crawl, congrats! Now, you need to analyse the data you’ve collected to make sense of it all. 

Once you’ve exported your crawl into a Sheet, you’ll have a wealth of data on each URL, including (but not limited to) metrics like:

• Search Console Data (Clicks, Impressions, CTR)
• Analytics Data (Events, Conversions, etc)
• URL Status
• Page Titles
• Meta Descriptions
• Headings
• Word Count
• Date Last Crawled (By Google)
• Any Custom Extractions

You can work through each column to choose the data you’re interested in reviewing, deleting any columns of data you aren’t interested in. 

Then, get to analysing! Look out for the key data points we discussed earlier in the guide to generate a tailored action for each URL, which could include:

• Updating meta titles and descriptions
• Adding more valuable content to low word count pages
• Adding internal links pointing to URLs that are currently low
• Targeting backlinks for URLs that have none
• Refreshing declining pieces of content that haven’t been updated in a long time
• Deleting or repurposing content with no clicks or impressions.

Now, all that’s left to do is delegate or schedule actions for each URL — and watch their SEO performance grow.

Closing Thoughts

If this was all a bit technical for you, or you just want to sit back and watch your site’s SEO performance grow without all the audits, why not let us do it for you?

We love content audits here at Dandy, and provide them as a central part of our SEO and content packages. We also offer a free full website SEO analysis for those interested in the SEO health of their entire site. 

Get in touch today or grab your free SEO audit now to get started

Screaming Frog For SEO Content FAQs