Whitelisting Search Atlas Bot

How to allow Search Atlas’s User Agent in Cloudflare

If you want to run a site audit and monitor your site health using Search Atlas, sometimes our site crawlers get blocked by CDNs.

Content Delivery Networks (CDNs) are a group of servers that help deliver website content faster. If your website relies on a CDN to increase the performance of your website, then it’s possible the CDN’s firewall will block our crawler from accessing your site.

Resolving this issue requires turning off bot-blocking features, as Search Atlas’s crawler doesn’t use static IP addresses. This post will explain how to do so on Cloudflare, as it is the most popular CDN provider for webmasters.

If you are using another CDN, the process is similar, and can still be done by following the steps outlined below.

How to allow Search Atlas’s User Agent in Cloudflare

Follow these steps:

1. Login to your Cloudflare account
2. Click Security and select the Bots submenu
3. Toggle off either Bot Fight Mode or Super Bot Fight Mode

After performing the above steps, Search Atlas will have access to your website for monitoring.

For help and a better understanding of Cloudflare’s IP Access Rules, see Cloudflare’s support documentation.

How to allow Search Atlas’s User Agent in WordFence (WordPress)

You can allow the Search Atlas site crawler in WordFence by following these steps:

1. On the WordPress menu, go to WordFence → Firewall → Firewall Options
2. Set the “Web Application Firewall Status” to “Disabled”
3. Click the “Save Changes” button.

How to allow Search Atlas’s User Agent in Stop Bad Bots (WordPress)

Follow these steps to allow the Search Atlas crawler:

1. On the WordPress menu, go to Block Bad Bots → Settings
2. Go to the Settings tab
3. Disable the “Block All Bots” option, or add our user agent Search Atlas Bot (https://www.searchatlas.com/)to the allowed list.

How to allow Search Atlas’s User Agent in Blackhole for Bad Bots (WordPress)

To allow Search Atlas’s crawler in the Blackhole plugin:

1. On the WordPress menu, go to Blackhole → Settings
2. In the Whitelist User Agents section, add the Search Atlas Bot (https://www.searchatlas.com/)user agent.
3. Save changes.

How to allow Search Atlas’s User Agent in Shield Security (WordPress)

If you use Shield Security, follow these steps to allow the crawler:

1. On the WordPress menu, go to Shield Security → IP Rules
2. Use the gear icon for additional options and select Create New User Agent Rule
3. Add the Search Atlas Bot (https://www.searchatlas.com) user agent to the bypass/whitelist list.
4. Click Create New Rule.

Need Help?

If you are still unable to successfully run a site audit after attempting the above methods, please contact our support team between 9am-6pm (PST), M-F, via the chat feature in your dashboard. Or, book a software demo with someone from our team.