Log file analysis that shows what crawlers really do
Your server records every visit from Googlebot and AI crawlers. We read those records and turn them into fixes.
In plain English.
Every time a crawler requests a page, your web server writes a line in its access log: the URL, the time, the status code and the user agent. Search Console gives a summary in its Crawl stats report, but the logs show the full detail, page by page and bot by bot.
Log analysis answers questions nothing else can. Which sections does Googlebot visit daily, and which has it not seen for months? How many requests go to redirects, errors or filter URLs? Are GPTBot, ClaudeBot and other AI crawlers reading your site, and what are they reading?
We collect logs from your hosting or CDN, check that bots claiming to be Google really are by reverse DNS lookup, and load the data into a log analyser alongside a crawl of your site. The result is a list of crawl waste to cut and important pages to promote.
It suits you if.
- Your site has tens of thousands of URLs or more
- New pages take a long time to be crawled
- You suspect bots are wasting time on filters, feeds or old URLs
- You want to know whether AI crawlers visit your site
- Server load spikes and you think crawlers may be the cause
What we do.
Log collection
Access logs gathered from Apache, Nginx, IIS, Plesk or your CDN, with personal data such as IP addresses handled carefully.
Bot verification
Requests claiming to be Googlebot or Bingbot checked by reverse DNS, so fake bots are not counted.
Crawl distribution
Requests broken down by section, template, status code and response time.
Orphan and missed pages
Pages bots crawl that are not linked internally, and important pages they rarely or never visit.
AI crawler activity
Visits from user agents such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot identified and reported.
How it runs.
- 01
Gather the logs
We agree a period, usually several weeks, and collect logs from your server or CDN.
- 02
Clean and verify
Bot traffic is separated out and verified, and personal data is stripped from what we keep.
- 03
Join with a crawl
Log data is matched against a fresh site crawl and your sitemaps.
- 04
Report and fix
You get a plain summary of where crawl effort goes and a list of changes to redirect it.
Questions people ask.
Can I get server logs from my host?
Most hosts keep access logs, though some only keep a few days. Managed hosting, Plesk and cPanel usually let you download them. If yours does not, we can switch on longer log retention and collect fresh data over a few weeks.
Is my site big enough for this?
Log analysis earns its keep on larger sites, typically shops, directories and publishers with many thousands of URLs. For a small business site, the Crawl stats report in Search Console is usually enough, and we will tell you so honestly.
What about GDPR?
Logs contain IP addresses, which count as personal data. We process them under an agreement with you, strip or truncate IPs where they are not needed, and delete the raw logs once the analysis is finished.
How much does log file analysis cost?
A fixed project price based on log volume and site size. Repeat analysis, for example after a migration, can be quoted as a follow-up piece. Prices are agreed before any work starts.
Your next chapter starts with a conversation.
Nottingham & Derby / UK-wideLet’s build
what’s next.
15 minutes with Garreth.
A real conversation about your business.