Ecommerce Website Developer

JournalAbout 2 min read

Reading server logs for SEO problems

Crawl tools tell you what a tool found. Logs tell you where crawl budget actually went. When they disagree, the logs are describing reality.

Written by

Javed IqbalApproved contributor

Contributed by

Nazir AbbasApproved contributor

Published

Abstract measured panel in the brand colours

Key takeaways

  • Group requests by URL pattern, then sort by count; the top is usually a surprise
  • Watch for crawl going to parameterised URLs that are not indexable anyway
  • Repeated 404 crawling means something still links to them
  • High volume 301s mean internal links were never updated
  • Verify the bot by reverse DNS before drawing conclusions
  • Review monthly, because the value is in the change not the snapshot

A crawl tool starts at your homepage and follows links. A search engine does not. It has its own history of your site, its own queue, and its own opinion about what is worth revisiting.

Server access logs are the only place you can see what it actually did.

What the logs answer that a crawl does not

  • Which URLs are being crawled most often, and whether those are the ones you care about
  • Whether crawl budget is being spent on filtered, sorted or parameterised URLs
  • Which pages have not been crawled at all in months
  • Whether a bot is receiving different status codes from the ones you see in a browser
  • Whether a redirect chain is being followed repeatedly

The pattern to look for first

Group requests by URL pattern rather than by individual URL, then sort by request count. On most ecommerce sites the top of that list is a surprise, and usually in a bad way.

Typical findings:

  1. A large share of crawl going to parameterised category URLs that are not indexable anyway
  2. Repeated crawling of redirect sources that were never updated in internal links
  3. Product pages for discontinued lines still being requested long after removal
  4. Session or tracking parameters generating effectively infinite distinct URLs

Status codes tell their own story

Group by response code and watch for:

  • 404s being crawled repeatedly, which usually means something still links to them
  • 301s with high volume, which means internal links were never updated to the destination
  • 5xx responses to bots, which is worse than it looks and often intermittent
  • 200s on URLs you believed were blocked, which means the block is not working

Verify the bot

Not everything claiming to be Googlebot is. Before drawing conclusions, confirm by reverse DNS lookup on the requesting address. A surprising share of apparent crawl traffic on a busy shop is scrapers, and attributing their behaviour to a search engine leads to the wrong fixes.

Make it routine, not a one-off

A log review is most useful as a monthly habit, because the value is in the change rather than the snapshot. A sudden rise in 404 crawling or a fall in product page crawling is a signal worth catching in the week it starts rather than the quarter it shows up in traffic.

If your crawl report and your logs disagree about what is happening, trust the logs.

Frequently asked

Questions readers ask

What do server logs show that a crawl tool does not?

Which URLs search engines actually request and how often, where crawl budget is being wasted, which pages have not been crawled in months, and whether bots receive different status codes from the ones you see in a browser.

How do I know if Googlebot traffic is genuine?

Confirm by reverse DNS lookup on the requesting address. A surprising share of apparent crawl traffic on a busy shop is scrapers, and attributing their behaviour to a search engine leads to the wrong fixes.

How often should I review server logs?

Monthly is a useful habit. A sudden rise in 404 crawling or a fall in product page crawling is worth catching in the week it starts rather than the quarter it shows up in traffic.

About the authors

Javed Iqbal

Written by

Javed IqbalApproved contributor

SEO Specialist & Co-Founder

Leads search, with more than five years in ecommerce and technical SEO for businesses in the UK, the US and the UAE.

Nazir Abbas

Contributed by

Nazir AbbasApproved contributor

Web Developer & Co-Founder

Builds and maintains ecommerce sites in React, Next.js and WordPress, and handles the technical side of search.

Editorial process: every article is written by one of the two co-founders and reviewed by the other before publication. Technical claims are checked against how the systems actually behave, and anything we cannot evidence is left out rather than asserted.

Related reading