How can I mitigate bots impacting my usage of Algolia?

Your search API key is visible in your site's front end, so anyone can use it. 

Algolia isn't a bot detection service, and no single setting stops every bot. What works depends on how the bot reaches your search. This article helps you work out which case you're dealing with and which measures will help.

If you haven't yet ruled out a problem in your own implementation, start with Why has there been a spike in my search usage? If the traffic comes from search engine crawlers such as Googlebot, see Why are search engine crawlers triggering searches on my site, and how do I stop it?

How bots reach your search

Most unwanted bot traffic follows one of three patterns:

To tell them apart, compare the spike in search requests with your own CDN or server logs for the same period. Use raw logs rather than web analytics, because analytics tools often filter bots out.

Any of these patterns can come from a few IP addresses or from a large pool of them. Bots increasingly use:

• pools of rotating IP addresses, so that blocking individual addresses doesn't stop them;
• pools in specific regions, so that geographic restrictions don't stop them;
• automated browsers that disguise themselves as ordinary ones.

What you can do in Algolia

Rate-limit your search API key

A rate limit caps the number of search requests each IP address (or each userToken) can make per hour.

• It works well against traffic from a small number of addresses.
• It can't stop bots spread across many addresses without also limiting your real users.

For the steps, and for how to choose a value, see Why is it important to create a rate limited API key? If your search runs from your own server, see How do I deal with bots if I use backend search?

Rotate your search API key

If a bot is calling our API directly with a copy of your key, replacing the key stops it until the new key is copied. Do it in this order:

1. Create a new key with the restrictions you want.
2. Update your site to use the new key.
3. Delete the old key. Anyone with a copy can keep using it until it's deleted.

Rotation doesn't help if the bot gets its key from your site, because it simply picks up the new one.

Use short-lived secured API keys

Secured API keys are virtual keys that your backend generates from a base key using generateSecuredApiKey.

• Automatic expiry. Set validUntil to a short time frame, for example a few hours or a day, so a copied key stops working on its own. Your front end requests a new key from your backend when the current one expires. You can check how long a key has left with getSecuredApiKeyRemainingValidity.
• Inherited restrictions. Secured keys inherit the restrictions of their base key, such as its rate limit, and can only add more.
• Revocation. You can't revoke secured keys one at a time. To revoke them, revoke the base key they were generated from, and move your site to a new base key at the same time.
• Protect the key endpoint. The backend endpoint that issues keys needs its own bot protection, so a bot can't simply request fresh keys from it.

Short-lived keys are most effective against bots calling our API directly. They help less against bots that load your site, because your site gives those bots a fresh key on every visit.

Add an HTTP referrer restriction

A referrer restriction limits your key to requests from your own domains, which stops simple scripts. Keep two limitations in mind:

• Referrers can be faked, so treat this as a light first filter rather than a fix.
• Some browsers remove the Referer and Origin headers from third-party requests. A referrer-restricted key can stop real users on those browsers from searching.

What you can do in your own infrastructure

You can't block IP addresses in Algolia. Blocking and bot detection need to happen before requests reach Algolia.

Bot protection at your CDN or firewall

Specialised bot protection, available from many web application firewall (WAF) and CDN providers, uses behaviour-based detection, challenges and fingerprinting. It doesn't rely on IP addresses alone, so it's the most effective option against bots that load your site, including those spread across many addresses. It won't see bots that call our API directly, because that traffic never reaches your infrastructure.

Your web host or infrastructure provider is a good first point of contact. If your site runs on a platform such as Magento or WordPress, look for bot-protection extensions in its marketplace.

Proxy search through your own backend

Instead of sending searches from the browser straight to Algolia, your front end can send them to your own backend, which then queries Algolia. No search API key is exposed in the browser, and you control authentication and limits on your own endpoint. That endpoint still needs bot protection. See Backend InstantSearch for how to set this up.

Which measures to start with

Further resources

• Cloudflare: protection against malicious bots and the bot management documentation
• The Algolia Academy course Mitigating Bot Traffic
• A talk on bots from Algolia DevCon
• Another video from our team on bot traffic