Antipatterns often emerge when implementation strategies are misaligned with best practices. These missteps can lead to inefficiencies and unintended consequences that prevent your applications from performing optimally and your users not getting the most of their search and discovery experience.
This guide explores antipatterns commonly seen across Algolia which affect overall performance and user experience and offers best-practises so you can avoid these antipatterns.
Antipattern: Excessive Number of Indices
What is an index?
An index is where the data Algolia uses is stored. It is a collection of records which are examined by the engine whenever a search is performed.
You can think of it as the equivalent of a table in a database but optimized for search and discovery operations.
Index Limit
There is a limit to the number of indices one application can have to ensure stability and performance for all our users. For premium customers, this limit is set at 1000 indices.
There are two main reasons for this limit:
- Performance: Our architecture is optimized to handle larger indices rather than a larger number of indices. Managing a high volume of indices can lead to reduced performance.
- Stability: By putting a limit on the number of indices, we can ensure a more stable and predictable environment for all users, preventing system overload and ensuring consistent performance.
The Antipattern
A common antipattern we observe is customers adding new indices as their business grows, rather than utilizing the capability of our system to scale within existing indices. Examples of this include:
- Creating one index per new user.
- One index per customer or store: For businesses with multiple customers or physical locations, a separate index is created for each entity.
These approaches can quickly lead to reaching the index limit, resulting in performance degradation.
A Solution: Filtering your dataset
A single index can grow significantly in size without impacting performance, up to a maximum size of 100GB. This limit is usually large enough to accommodate most data needs and therefore, if you find yourself wanting to add a new index, you could first consider organizing data within one of your existing indices and adding an attribute for filtering within your dataset.
For example, if your e-commerce company offers different pricing and product availability based on the customer segment (eg. regular, VIP and wholesale buyers), instead of creating separate indices for each customer segment, you can use an attribute such as customer_segment to control access.
You would add customer_segment as a ‘filterOnly’ attributeFor Faceting so the engine does not calculate facet counts for the attribute and therefore ensure optimal performance. You can then create logic to include a filter for customer_segment at query time or generate a Secured API key depending on which customer type is performing the query to restrict which records they can retrieve.
Further examples can be found here: https://www.algolia.com/doc/guides/security/api-keys/how-to/user-restricted-access-to-data/
and here: https://support.algolia.com/hc/en-us/articles/13894396253969-How-do-I-handle-an-implementation-with-a-large-numbers-of-indices
When to use Multiple Indices
On the other hand, while our system is designed to scale efficiently with larger indices, in some cases an increase in the number of indices is acceptable, such as creating different indices based on languages provided the number of languages is relatively stable.
It is crucial to find a balance between having many small indices, and creating just one or two larger ones and while it is often better to add to an existing index rather than creating a new one, we realize each use-case is different.
If you expect to exceed the 100GB limit, or are unsure of whether creating a new index or adding to an existing index is the best option, feel free to reach out to us at Support. We are happy to help guide you in finding the best solution for your needs.
Helpful links
- Data, record size and usage limits: https://www.algolia.com/doc/guides/sending-and-managing-data/prepare-your-data/in-depth/index-and-records-size-and-usage-limitations/
- Scaling to large datasets: https://www.algolia.com/doc/guides/scaling/scaling-to-larger-datasets/
- Reducing record size: https://www.algolia.com/doc/guides/sending-and-managing-data/prepare-your-data/how-to/reducing-object-size/