Algolia’s A/B testing feature automatically excludes outlier users when calculating result metrics. This is to reduce traffic that can skew results in an unhelpful way.
One of the most common source of these outliers is the misuse of userTokens at query time. Ensuring that each query is correctly attributed to an individual user is critical for producing meaningful results.
Why userToken matters
If a userToken is not explicitly set at query time, Algolia defaults to setting one based on the request's IP address. This means that all queries originating from the same IP will be viewed as coming from the same user.
This can cause issues, particularly if you have a backend implementation and are either not forwarding your users' IP addresses or failing to explicitly set a unique userToken for each user.
In such cases, the system falls back to using your server’s IP address, grouping all traffic under a single IP. As a result, multiple users may appear as a single user and their interactions may be flagged as statistical outliers and excluded from test metrics.
Additionally, assigning unique userTokens for each user ensures that the same user consistently sees the same variant which is essential for reliable A/B test results.
Best practices
To maintain the accuracy and effectiveness of your A/B tests:
-
Assign a unique
userTokento each user. Send this token with every search query. -
For backend implementations, either explicitly set a
userTokenor forward the real user’s IP address.
Note: If you wish to assign a single userToken to all anonymous users, this can also lead to skewed results and uneven search counts. For guidance on handling this scenario, check out our guide article here.