Skip to main content

Algolia DevCon - June 28-29, 2023

Discover the future of the AI Stack with Algolia, be the first to see new platform demos and announcements, and get the most out of Algolia through interactive workshops.

REGISTER FOR FREE DON'T SHOW AGAIN
×

Why has there been a spike in my search usage?

  • Updated

If your search usage is higher than you expect, work through two questions in order:

  1. Is my implementation sending more requests than it needs to?
  2. Is someone other than my users sending them?

1. Rule out your own implementation

These are common causes of avoidable request volume:

  • A loop in your code that triggers thousands of search requests.
  • Automatic page refreshes that trigger empty searches.
  • A misconfiguration that sends duplicate or unnecessary requests, such as a widget that searches twice per interaction.

Start with the guide Optimize search request usage, then How do I reduce my cost while using InstantSearch?

Also check that you're looking at the figure you're billed on. A single search request can bundle several queries, and each query is a search operation, so the two counts can differ several times over on a search-as-you-type implementation. See What is the difference between a Search Operation and a Search Request?

2. Find out where the requests come from

Open the Search API Logs in your Algolia dashboard. Each request shows its originating IP address and HTTP headers, including the user agent. The getLogs API method returns the same information.

When you review the logs:

  • Look at user agents as well as IP addresses. One IP sending a very high number of requests, or repeating the same query, is easy to spot. Many bots now spread their traffic across hundreds or thousands of IP addresses, each sending only a few requests. No single IP stands out, but one user agent may account for most of the traffic.
  • The logs are a recent sample. They cover the last seven days only, and only up to 1,000 API requests per server are logged. A spike that's more than a week old, or very large, may not be visible. If you need a fuller picture, contact the Support team.
  • Calling getLogs counts towards your operations quota.
  • User agents can be faked. Traffic that claims to be a well-known crawler isn't necessarily that crawler.

Search engine crawlers

A large share of unexpected requests is often legitimate crawlers, such as Googlebot, crawling your search and category pages. This is a crawling configuration issue rather than an attack. The fix is to control what crawlers can visit. See Why are search engine crawlers triggering searches on my site, and how do I stop it?

Scrapers and other bots

Bots that hide their identity, scrape your data or flood your search need a different response. See How can I mitigate bots impacting my usage of Algolia?

Fixing it

We can advise, but you'll need to make the changes on your side. Common fixes:

  1. Remove the misconfiguration or loop in your implementation.
  2. Stop automatic refreshes that re-run searches.
  3. Keep crawlers off pages that trigger searches, using robots.txt.
  4. Use a rate-limited search API key. See Why is it important to create a rate limited API key?
  5. Add bot protection in your own infrastructure. See How can I mitigate bots impacting my usage of Algolia?

Related articles

Related articles