Log File Analysis for SEO in 2026

Every crawler that visits your site leaves a line in a log file: the URL it asked for, when, what it got back and who it said it was. Most teams never read those lines. They rely on Search Console, which is useful but summarised, and on crawling tools, which show what could be crawled rather than what was.

The logs answer different questions. Where Googlebot actually spends its time. Which important pages it rarely visits. Which junk URLs it keeps coming back to. Whether the AI crawlers can reach your pages at all, and whether people asking ChatGPT or Claude about your category are being sent to you. This guide is the method we use: what to collect, how to verify the bots, what to look for, and the short list of fixes the findings usually point to.

What a log file tells you that no other tool does

An access log is a plain list of requests. Each line holds the visitor's IP address, the time, the URL, the HTTP status returned, the size of the response, the referrer and the user agent. Filter it to search and AI crawlers and you have a complete record of how machines read your site.

Search Console's Crawl Stats report, under Settings, gives a good summary of Google's side: total requests, response codes, file types, Googlebot types, and whether requests were for discovery of new URLs or a refresh of known ones. It covers roughly the last 90 days and shows example URLs rather than all of them. It also says nothing about Bing or the AI crawlers. A crawling tool tells you what a bot could reach. Only the logs tell you what each one actually requested.

Getting the right logs

Take the logs from the layer that answers the request. If your site sits behind a CDN, the origin server only sees cache misses, so its logs undercount crawling badly. Most CDNs and hosting platforms can export request logs, often on paid plans. On a plain server, the Nginx or Apache access log is the source.

Verify the bots before you count them

Anyone can send a request that says Googlebot. Scrapers do it to get past firewalls, and some of them crawl harder than Google ever would. If you analyse user agents without verifying them, you will draw conclusions about Google from traffic that is not Google.

Two details matter for interpretation. User-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User may not follow robots.txt, because a person asked for the page. And a request from them is not crawling in the usual sense: it means someone asked an assistant a question and the assistant came to your page for the answer. That makes them the most commercially interesting lines in the file.

Six questions to answer from the logs

Group URLs by template or section first: product pages, categories, guides, filters, internal search, everything else. Questions about thousands of single URLs are hard to act on. Questions about ten groups are not.

1. 01

   Where does Googlebot spend its requests?

   Compare each group's share of verified Googlebot requests with its share of organic clicks or revenue. A section that gets 40% of crawling and 2% of traffic is usually filters, parameters or thin pages that should not be crawlable.

2. 02

   What does it get back?

   Break responses down by status code and group. Look for redirect chains on internal links, 404s on URLs your own pages still link to, 5xx spikes, and any 429s. Google lowers its crawl rate when a site slows down or returns server errors.

3. 03

   Is it crawling URLs you never meant to publish?

   Sort the crawled URLs by pattern. Sorting and tracking parameters, faceted combinations, internal search results, old staging hostnames and calendar pages are the usual offenders. Each one is a link or a sitemap entry somewhere.

4. 04

   How fast does it find new pages?

   For pages published in the period, measure the time from publishing to the first verified Googlebot request. Days rather than hours on a site that publishes often points to weak internal linking from frequently crawled pages, or to sitemaps without honest lastmod dates.

5. 05

   Which important pages does it rarely visit?

   Join the logs to a crawl of your site or your sitemap. Indexable pages with no Googlebot visit in 30 days are close to invisible. They are usually deep, orphaned or linked only from pages that are themselves rarely crawled.

6. 06

   What are the AI crawlers doing?

   Split them into training, search and user-triggered. Check that the ones you want, especially the search and user agents, receive 200s and not 403s from a firewall rule nobody remembers. Count user-triggered visits by page: they show which pages assistants are fetching to answer questions.

What the findings usually point to

A monthly routine that takes an hour

1. Export 30 days of logs from the layer that answers requests, filtered to crawler user agents.
2. Verify Googlebot, Bingbot and the AI crawlers against each provider's DNS check or IP list, and discard the rest from the analysis.
3. Update the six numbers: crawl share by section against traffic share, error rate, share of requests to unwanted URLs, median time to first crawl for new pages, important pages not crawled in 30 days, and AI user-triggered visits by page.
4. Check the Crawl Stats report in Search Console for host status problems and a sudden change in total requests.
5. Pick one fix, ship it, and watch the same number next month. Add the check to your release list if a deploy caused it, the same way you would for a stray noindex.

The bottom line

• Use the logs for what only they show: wasted crawling, important pages Google rarely visits, and how each AI crawler treats your site.
• Collect from the layer that answers requests, keep 30 to 90 days, strip ordinary visitors' IPs, and verify every bot before you count it.
• Group by section, answer the six questions, and expect the fixes to be about links, status codes, internal linking and server stability rather than anything exotic.

Frequently asked questions

What is log file analysis in SEO?

It is the review of your server or CDN access logs to see exactly which URLs search and AI crawlers requested, how often, and what they received. It shows real crawler behaviour, where tools such as site crawlers only show what could be crawled.

How do I know a request really came from Googlebot?

Run a reverse DNS lookup on the IP address. Google's documentation says the name must end in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that name and confirm it returns the same IP. You can also match the IP against the ranges Google publishes for its crawlers.

Does crawl budget matter for a small website?

Usually not. Google's own guidance aims crawl budget advice at sites with hundreds of thousands to millions of URLs, or tens of thousands that change daily. On a small site, pages that are slow to be crawled usually have a linking or quality problem, not a budget problem.

Can I see ChatGPT or Claude visits in my logs?

Yes. OpenAI's crawlers identify as OAI-SearchBot, GPTBot and ChatGPT-User, and Anthropic's as ClaudeBot, Claude-SearchBot and Claude-User. Visits from ChatGPT-User or Claude-User happen when a person asks the assistant something and it fetches your page to answer. Verify them against each provider's published IP information before you count them.

How this guide was put together

• Crawler behaviour and verification methods were read from the providers' own documentation on 2 November 2026: Google's crawl budget guide, its page on verifying Google crawlers and fetchers, the March 2026 announcement of the new IP range location, and the Crawl Stats report help page; OpenAI's overview of its crawlers; and Anthropic's help article on its crawlers.
• The six questions and the monthly routine are the method we use on client sites. The thresholds in the callouts are Google's published estimates, which Google itself describes as rough guidance rather than exact limits.

Sources

1. Google: Optimize your crawl budget
2. Google: Verify requests from Google crawlers and fetchers
3. Google Search Central Blog: New location for the Google crawlers' IP range files (March 2026)
4. Search Console Help: Crawl Stats report
5. OpenAI: Overview of OpenAI crawlers
6. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
7. Lucidens: The Technical SEO Audit Checklist for 2026
8. Lucidens: AI Crawlers: What to Allow, What to Block