SiteGraderBot

SiteGraderBot is the crawler behind SiteGrader. It reads the public homepages of websites, to grade each one, compare it with similar sites and show what to fix.

How it identifies itself

Every request carries this user agent:

Mozilla/5.0 (compatible; SiteGraderBot/1.0; +https://www.sitegrader.com/bot/)

When it loads a homepage in a browser, to see it as visitors do, it adds its name to the browser's user agent:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36 SiteGraderBot/1.0 (+https://www.sitegrader.com/bot/)

It has no fixed IP addresses, so look for the name.

What it does

It reads homepages, and only now and then. It visits when someone grades the site or runs a free tool on it, weekly if someone watches it, daily if someone follows it. The homepage is the only page it requests. It does not crawl your other pages, submit forms or log in.
It follows your robots.txt. It reads the file before it requests anything else, and asks for your sitemap and llms.txt only if the file allows them.
The exception: your homepage. A rule for every crawler (User-agent: *) does not keep it off your homepage, since such rules are usually written for search engines. A rule that names SiteGraderBot always does.
It sees what a visitor sees. The browser loads the scripts, styles and images the homepage asks for and takes a screenshot for the report. It does not solve CAPTCHAs or hide what it is.
Speed tests come from Google. SiteGrader asks Google PageSpeed to test the homepage too, so those requests come from Google, not from SiteGraderBot.

How to stop it

Add this to your robots.txt. Its next visit stops there, and the site's reports are hidden:

User-agent: SiteGraderBot
Disallow: /

Or remove your site by publishing a token, which you can undo later. Both are free and need no account.

Letting it in

If a firewall turns SiteGraderBot away, the report names the service and the rule that lets it in: allow requests whose user agent contains SiteGraderBot. Anyone can copy a user agent, so you can remove the rule once your grade appears.

The rule for each service

Cloudflare

In Cloudflare, open Security rules and create a custom rule with this expression. Choose the action Skip, then tick All remaining custom rules, All managed rules, All Super Bot Fight Mode rules and, under products, Browser Integrity Check. Official instructions

(http.user_agent contains "SiteGraderBot")

No rule can skip Bot Fight Mode, the version on Cloudflare's free plan. If it is on, turn it off under Security, Settings, Bot traffic.

Akamai

Akamai's firewall and bot settings are not public, and an account team usually manages them. Ask yours to allow requests whose user agent contains this name.

SiteGraderBot

DataDome

In DataDome, go to Access Control, then Custom Rules, and create a rule with the response Allow and this query. Official instructions

useragent:*SiteGraderBot*

The query is case sensitive. Admin and Editor users can add rules.

HUMAN (PerimeterX)

In HUMAN's console, go to Bot Defender, Policies, then Policy Settings, and add a custom rule with the action Allow for requests whose User-Agent header contains this name. If your policy does not offer that condition, ask HUMAN's support. Official instructions

SiteGraderBot

Kasada

Kasada's settings are not public. Ask your Kasada administrator or its support team to allow requests whose user agent contains this name.

SiteGraderBot

Imperva

Ask whoever runs your Imperva account, or Imperva's support, to allow requests whose user agent contains this name.

SiteGraderBot

Sucuri

Sucuri documents rules for an address or a path but not for a user agent, and SiteGraderBot has no fixed address. Ask Sucuri's support to allow requests whose user agent contains this name.

SiteGraderBot

Wordfence

Wordfence documents no rule that allows a user agent. In WordPress, open Wordfence, Tools, Live Traffic and look for this name to see what blocked it, then relax that setting, such as a Rate Limiting rule for crawlers or a custom block pattern. Official instructions

SiteGraderBot

Vercel

In your Vercel project, open Firewall, then Configure, and select Add New, Rule. Set If to User Agent contains this name and Then to Bypass, place it above any rule that blocks, then publish it. Official instructions

SiteGraderBot

AWS WAF

In your AWS WAF web ACL, add a rule that inspects the single header user-agent, with the match type Contains string and this value. Set the action to Allow and place it above any Bot Control or managed rule group. Official instructions

SiteGraderBot

SiteGround

SiteGround's Anti-Bot AI has no setting for this. Open a Help Desk ticket asking SiteGround to allow requests whose user agent contains this name.

SiteGraderBot

Fastly Next-Gen WAF

In Fastly, go to Security, Next-Gen WAF, then Rules, and add a workspace rule of the type Request. Set the condition to User agent contains this name and the action to Allow. Official instructions

SiteGraderBot

ModSecurity

Add this to your ModSecurity configuration, ahead of your other rules. On shared hosting, ask your host to add it.

SecRule REQUEST_HEADERS:User-Agent "@contains SiteGraderBot" "id:1000001,phase:1,allow,nolog"

Google Cloud Armor

Add a rule to your security policy with the action allow, a lower priority number than your other rules, and this expression. Official instructions

has(request.headers['user-agent']) && request.headers['user-agent'].contains('SiteGraderBot')

Azure Front Door

Add a custom rule to your WAF policy with the match variable RequestHeader, the selector User-Agent, the operator Contains, this value and the action Allow, at a low priority number. Official instructions

SiteGraderBot

These follow each service's public documentation, and menus change. Where a service's settings are not public, the rule is what to ask its support team for.

Something SiteGraderBot did that it should not have? Tell us.