The best log file analysis tools for SEO in 2026

Google’s own crawl budget documentation says most sites don’t need to think about crawl budget. It reserves the worry for roughly a million or more pages that change weekly, or ten thousand or more pages that change daily. If you run a 300-page brochure site, a log file analyser is a nice-to-have at best.

But crawl budget is only one reason to read your logs. They’re also the only complete record you own of what Googlebot really requested. Errors served to bots, orphan pages, redirect chains after a migration, and sections Googlebot has quietly stopped visiting all show up there.

This list is for people who have one of those reasons: a store with faceted navigation spitting out parameter URLs, a migration next month, a client whose traffic fell while every audit crawl came back clean. The gap between what a crawl can reach and what Googlebot actually asks for is the whole point. If you want the crawler side of that comparison, what audit tools can and cannot see is the place to start.

I looked at seven tools, from a free terminal program to platforms that sell by quote. For the quote-only ones I’m working from vendor documentation, not a paid trial, so treat those entries as a prompt for a demo call and not a verdict. Prices are what the vendors listed when I compiled this. They move, so check before you pay.

How I picked

log formats: can it read what your server really writes? Apache, NGINX and IIS at minimum, plus CDN or load balancer exports if that’s where your logs live.
bot verification: a user agent that says Googlebot proves nothing. Google’s guide to verifying Googlebot uses reverse DNS or its published IP ranges, and I marked tools up when they do that for you.
crawl data join: logs alone show what was requested. Logs plus a crawl show what was never requested, which is usually the more useful list.
volume and price: a busy site writes millions of lines a day, so I looked at what each tool claims to handle at that size and what it costs when you get there.
setup burden: a one-off file upload, or an ongoing feed that needs a developer or sysadmin to wire up.
non-Google bots: Bingbot, GPTBot, ClaudeBot and PerplexityBot all leave traces in your logs, and Search Console will never show them to you.

The picks

Ordered by who I’d point at first.

Screaming Frog

This is where most people should start. The SEO Log File Analyser is a desktop app for Windows, macOS and Ubuntu from the makers of the SEO Spider. You drag log files in and get hits by bot, status code and URL. It reads Apache, NGINX, IIS and several CDN and load balancer formats, and it verifies search engine bots so spoofed Googlebot hits get separated from the real ones.

The best feature is importing a Screaming Frog crawl next to the logs. You get URLs Googlebot requested that the crawl never found, and URLs in the crawl that no bot touched in your log window. That second list is where I’d start when a section isn’t getting indexed. The vendor’s feature page has the full rundown. The free version caps you at 1,000 log events and one project, which proves it parses your format and not much else.

pro: low annual cost next to any hosted platform
pro: crawl and log comparison built in, with bot verification
pro: runs locally, so raw logs stay on your machine
con: manual imports only, with no scheduled feed or alerts, so it audits and doesn’t monitor
con: large logs are stored in a local database, and multi-gigabyte files can slow a normal laptop

Pricing: free version limited to 1,000 log events and one project. Paid licence listed at Â£99 per year.

Link: screamingfrog.co.uk/log-file-analyser

Semrush

Semrush ships a free Log File Analyzer inside its technical SEO tools. You upload an access log in the browser and it charts Googlebot activity: hits per day, status codes, file types and the most requested URLs. It’s the quickest way I know to answer whether Googlebot is hitting your 404s and redirects, and there’s nothing to install.

It’s built around Googlebot and one file at a time, so it won’t replace an ongoing feed. If you already pay for Semrush, it’s a free extra. I wouldn’t buy Semrush for this alone, and I compare the wider suite against Moz in semrush vs moz for small teams.

pro: free at last check, with a Semrush login
pro: no install, and the charts make sense to a client without explanation
pro: fast answers on status codes and crawl frequency
con: Googlebot only, so no view of Bingbot or AI crawlers
con: browser upload, so cut huge logs down to bot lines first

Pricing: Log File Analyzer was free with a Semrush account at last check. Paid Semrush plans only matter if you want the rest of the suite.

Link: semrush.com

JetOctopus

JetOctopus is a cloud crawler that also ingests server logs and Search Console data, so crawl, logs and clicks sit in one dashboard. It sells at a lower price point than Botify or Oncrawl, which makes it the one I’d look at for a mid-sized site, roughly 100,000 to a few million URLs, that has outgrown a desktop app without needing an enterprise contract.

Logs come in through an integration or file transfer rather than a drag and drop, so someone with server or CDN access sets it up once. After that you can watch bot behaviour over weeks, which a desktop tool can’t do. I’m going on vendor documentation and user reports here, so run a trial on your own logs before you commit.

pro: crawl, logs and Search Console data in one place
pro: entry price sits well under the enterprise platforms
pro: a continuous log feed shows trends, not a single snapshot
con: needs someone with server or CDN access to set up the feed
con: a smaller vendor than Botify, so ask about support and how long logs are kept

Pricing: published tiered plans on the vendor’s site. Check the current table before budgeting.

Link: jetoctopus.com

Botify

Botify is where large ecommerce and publisher teams tend to end up. Its log analysis sits inside a wider platform with a crawler, so you can group bot activity by page type (product, category, facet, article) and see where Googlebot spends its requests across templates. It’s built for very large sites with very large log volumes.

There’s no public price list. Plan for a demo and a contract, not a card form. For a site under a few hundred thousand URLs I think this is the wrong purchase, and plenty of teams buy an enterprise platform before they’ve read a single raw log line. That order is backwards. Read the logs once with a free tool, then decide what you’re missing.

pro: built for very large sites and heavy log volumes
pro: page-type grouping turns “where does Googlebot waste requests” into a chart
pro: widely used at enterprise level, so hiring people who know it is easier
con: quote-only pricing, and likely the most expensive route on this list
con: log delivery needs engineering time, so the first report can be weeks away

Pricing: custom quote, nothing published.

Link: botify.com

Oncrawl

Oncrawl is the other enterprise name. It’s a crawler first, and its log analysis is cross-referenced with crawl data and Search Console so you can segment by site section and compare like with like. If you’re weighing it for audits and not logs, I put it next to Sitebulb in sitebulb vs oncrawl for enterprise audits.

Like Botify, logs arrive as an ongoing feed you set up once, not a file you drag in. Pricing is by request. Ask for a trial on your own logs, and ask how many months of history the plan keeps, because year on year comparisons depend on it.

pro: crawl, logs and Search Console share the same segments
pro: suits ongoing monitoring around releases and migrations
pro: a natural fit if you already run its crawler
con: quote-based pricing, so there’s no quick way to compare it with the others
con: more platform than a one-person SEO operation will use

Pricing: by quote, nothing published.

Link: oncrawl.com

GoAccess

GoAccess is an open source log analyser that runs in a terminal or writes a self-contained HTML report. It isn’t an SEO tool. It reads Apache, NGINX and CloudFront formats, and its --crawlers-only flag keeps only bot traffic, which is the trick that makes it useful here.

I’d use it on a VPS where I have shell access and want a quick look without moving logs anywhere. The trade is that it matches bots by user agent alone, so a scraper calling itself Googlebot counts as Googlebot until you check the IPs against Google’s guidance. It also knows nothing about your sitemap or crawl, so you get counts and status codes and no idea what was missed.

pro: free and open source under the MIT licence
pro: live HTML dashboard, plus a fast terminal view
pro: logs can stay on your own server
con: no crawl or sitemap comparison, so no orphan page detection
con: user agent matching only, so fake bots inflate the numbers

Pricing: free.

Link: goaccess.io

Google BigQuery

This is the do-it-yourself route for teams that already live in SQL. Load access logs into a date-partitioned table (from Cloud Storage files, or a Cloud Logging sink if you sit behind a Google Cloud load balancer), query them, and chart the results in Looker Studio. Join the table to the Search Console bulk data export and you can line up Googlebot requests against impressions and clicks per URL.

You also build what the packaged tools hand you. Bot verification means joining against the crawler IP ranges Google publishes, and parsing your log format is on you. Always filter on the partition column, or every query scans the whole table and you pay for it.

pro: no practical row limits, and the data joins to anything you own
pro: the Search Console export can sit in the same warehouse
pro: cost can stay tiny for a mid-sized site
con: you write the SQL and build bot verification yourself
con: one careless query on an unpartitioned table can cost real money

Pricing: on-demand queries were $6.25 per TiB with the first 1 TiB each month free at last check, plus storage by the GB. Looker Studio is free.

Link: cloud.google.com/bigquery

Comparison table

Prices are as the vendors listed them at last check.

Tool	Price	Primary strength	Primary weakness
Screaming Frog	Free tier, Â£99 per year paid	Crawl and log comparison	Manual imports, no monitoring
Semrush	Free with an account	Zero-install Googlebot charts	Googlebot only, browser upload
JetOctopus	Published tiered plans	Crawl, logs and Search Console in one	Needs server or CDN access
Botify	Custom quote	Very large sites, page-type grouping	Price and setup time
Oncrawl	Custom quote	Logs segmented alongside crawl data	Quote only, heavy for small teams
GoAccess	Free (MIT)	Fast, live, runs on your server	User agent matching, no SEO context
Google BigQuery	Pay per query, first 1 TiB monthly free	Flexible SQL, Search Console joins	You build everything

How to choose

Start with whether you have a problem at all. Google’s crawl budget guidance says it matters mainly for very large or fast-changing sites. Under 10,000 pages, I’d do one manual analysis before a migration or after a traffic drop, using the Â£99 Screaming Frog licence, GoAccess or the Semrush tool, and stop there. For the rest of the stack, free tools that replace paid seo software has more in the same spirit.

Getting the logs is the hard part. Shopify doesn’t give merchants raw server logs, and hosted builders like Wix and Squarespace generally don’t either. On ordinary hosting the files may exist but rotate after a week or two. If a CDN sits in front, your origin logs miss anything served from cache, so you need the CDN’s logs. Ask for 30 days at the least, and 90 if you can get them. Logs also hold visitor IP addresses, which can count as personal data under GDPR and similar laws. This is not legal advice, so check with whoever handles privacy for you before sending logs to a third-party platform. Our sister site The Privacy Wire is a starting point for the wider topic. A desktop tool or your own server keeps that question simple.

Then decide between a snapshot and a feed. Screaming Frog and GoAccess tell you what happened last month. The platforms tell you what’s happening this week, which matters around releases and migrations, when a broken redirect map or a robots.txt change often shows up in the logs days before it shows up in rankings. If you rarely ship changes like that, ongoing monitoring is a luxury. If you do, it isn’t. When the logs show Googlebot barely visiting a section, the fix is often better links into it, and internal linking done properly is where I’d go next.

Last, match the tool to whoever will open it. A team with an analyst who writes SQL gets more from BigQuery than from a dashboard they can’t change. A one-person shop wants Screaming Frog. An in-house team at a big retailer with a developer to wire up the feed can justify Botify or Oncrawl, and JetOctopus sits in the middle.

Verdict / top pick

Screaming Frog’s SEO Log File Analyser is my top pick for most people reading this. It costs Â£99 a year, runs on your own machine, and it’s the only one on the list that pairs a log import with a crawl comparison at that price. Try the free version first to confirm it parses your log format, then buy.

If you need monitoring and not audits, JetOctopus is where I’d start the trial, with Botify or Oncrawl only once log volume and headcount justify a contract. GoAccess is the quick look on a server you control, the Semrush tool is a free extra if you already have an account, and BigQuery is for teams that would rather write SQL than click through a dashboard.

One gap none of them closes: AI crawler analysis is still manual. For GPTBot, ClaudeBot and PerplexityBot you’re mostly filtering user agents yourself and checking against whatever IP ranges each company publishes. Whatever you buy, read your logs raw at least once first. The rest of our tooling coverage is in the blog index.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-09-22.