If you notice a spike in your usage, or your usage is much higher than you expect, there may be something wrong. Here you can find resources to identify, isolate and resolve the error:
Finding the cause:
The increased usage could be caused by, but is not limited to, any of the following:
- A runaway loop triggering thousands of search requests.
- Too many empty requests, triggered by automatic refreshes.
- An attack to your API by Google Bots
- Web scraping or a denial-of-service attack
- A misconfiguration with your implementation causing duplicated and/or unnecessary requests.
First, you'll need to make sure there are no bugs in your solution. You can do this by following our guide on Optimizing InstantSearch.js search requests. This will help rule out any misconfigurations on your end and ensure your solution is only sending the requests necessary to create a great search experience.
Once you have ruled out your own potential misconfiguration, you should investigate where your search requests are coming from. This can be done by investigating the Search API Logs on the application.
You can do this by going to your Algolia dashboard in the “Indices” “Search API Logs” section. From there, you can dig into every search request and get the associated IP and see if there are any IPs making an unusually high number of requests . You may find that most requests are coming from a search bot like Googlebot (the bot Google uses to crawl the internet and build their search engine). You can read more on investigating Google Bots here.
You can do the same with the get-logs API method.
Once you have confirmed the source of your usage, you will need to implement a solution to avoid the requests from continuing or re-occurring. We can offer advice on how to best to mitigate this, but the implementation and solution will need to be actioned from your end.
Solutions:
- Remove the misconfiguration in your solution
- Stop the automatic refreshes
- Inform Googlebot to not go to your search pages at all with a well configured robots.txt.
- Cloudflare also has good measures against bot abuse.
- Generate a new Search API Key with reduced queries per IP, per hour. See rate limiting for more information. The rate of the limit is up to you, we generally recommend starting with a higher number (in order to avoid limiting real users) and reduce it gradually based on the usage/fake bots.
- Third-party tools, such as Cloudflare, AWS Shield, or Akamai can often detect and prevent high numbers of requests.
Please find the documentation our team have created to assist you in investigating and monitoring your unexplained search operations here: https://www.algolia.com/doc/faq/basics/too-many-false-unexplained-search-operations/