EghosaBot

EghosaBot is the crawler behind the free SEO tools on this site. It only visits a website when someone asks one of the tools to check it, and it reads just enough of each page to answer that check, so it is not indexing the web or collecting your content.

How it identifies itself

Every request carries this user agent, so you can find EghosaBot in your server logs or match it in a firewall rule:

Mozilla/5.0 (compatible; EghosaBot/1.0; +https://eghosa.me/tools/bot/)

Requests come from Cloudflare's network rather than a fixed set of IP addresses, which means the user agent is the reliable way to recognise it.

How it behaves

EghosaBot reads robots.txt before anything else and follows it the way Googlebot does: a group addressed to EghosaBot by name applies if there is one, otherwise the rules for every crawler (User-agent: *) apply. It honours Crawl-delay up to 10 seconds between requests, and without one it keeps no more than four requests in flight at once. For page checks it stops reading each page once it has the head, where the robots and canonical tags live.

Blocking or allowing it

To keep it out of a section of your site, add a group for it to robots.txt:

User-agent: EghosaBot
Disallow: /private/

If your firewall or bot protection blocks it, the tools will report that most of your pages answered 403 or 503. To let it through, add an allow rule in your firewall for requests whose user agent contains EghosaBot, then run the check again.

Questions

If EghosaBot ever behaves in a way you did not expect, email fredrick@eghosa.me with your domain and the time, and I will look into it.