If you notice a spike in your operations, there may be something wrong. Here you can find resources to identify, isolate and resolve the error:
Finding the cause:
The spike could be caused by but is not limited to any of the following:
- It could be a runaway loop triggering thousands of search requests.
- You may be sending too many empty requests, triggered by automatic refreshes.
- You could be experience An attack to your API by Google Bots
- You may be experiencing web scraping or a denial-of-service attack
First, you'll need to make sure there are no bugs in your solution. You can do this by following our guide on Optimizing InstantSearch.js search requests. This will help rule out any misconfigurations on your end and ensure your solution is running at it's optimal and most efficient potential.
Once you have ruled out your own potential misconfiguration, you will need to identify the the common agent that's making the search queries to your API. This can be done by investigating the Search API Logs on the application.
You can do this by going to your Algolia dashboard in the “Indices” “Search API Logs” section. From there, you can dig into every search request and get the associated IP or see if there’s one IP that is always doing the same request. You may find that most requests are coming from a search bot like Googlebot (the bot Google uses to crawl the internet and build their search engine). You can read more on investigating Google Bots here.
Note: You can do the same with the get-logs API method.
Once you have confirmed the source of your spike, you will need to implement a solution to avoid the spike from continuing or re-occurring. We can offer advice on how to best to mitigate but the implementation and solution will need to be actioned from your end.
- Remove the misconfiguration in your solution
- Stop the automatic refreshes
- Inform Googlebot to not go to your search pages at all with a well configured robots.txt.
- implement the latest Google reCAPTCHA
- Cloudflare also has good measures against bot abuse.
- Generate a new Search API Key with reduced queries per IP, per hour. See rate limiting for more information. The rate of the limit is up to you, we generally recommend starting with a higher number (in order to avoid limiting real users) and reduce it gradually based on the usage/fake bots.
- Third-party tools, such as Cloudflare, AWS Shield, or Akamai can often detect and prevent high numbers of requests.
Please find the documentation our team have created to assist you in investigating and monitoring your unexplained search operations here: https://www.algolia.com/doc/faq/basics/too-many-false-unexplained-search-operations/