Log File Analysis
Log file analysis means reading raw server logs to see exactly which URLs search engine bots requested, how often, and what status code each request returned. It is the only source that shows real crawler behaviour rather than estimates of it.
Why it matters for your rankings
Every other crawl report is inference. Server logs are the record of what actually happened, which makes them the final word when Search Console and your crawler disagree.
They answer questions nothing else can. Which sections is Googlebot ignoring entirely? How much of your crawl budget goes to parameter URLs nobody searches? Are bots hitting redirect chains or error pages repeatedly? On large sites this regularly reveals that most crawl activity is spent on URLs with no commercial value at all. The practical barrier is access rather than difficulty: logs are often held by an infrastructure team, and verifying that a request claiming to be Googlebot really is Googlebot takes a reverse DNS check.
How to check it on your site
Request 30 days of logs
Ask for raw access logs from the server or CDN. Cloudflare, Fastly and AWS can all export them.
Verify the bots are genuine
Run reverse DNS on Googlebot IPs. A significant share of self-declared Googlebot traffic is spoofed.
Group requests by directory
Compare crawl volume per section against that section's commercial value. Mismatches are where budget is leaking.
Cross-reference with your crawl
URLs in your sitemap that bots never request are discovery problems. URLs crawled heavily but absent from your sitemap are usually waste.