If one variant has significantly more searches, despite having roughly the same number of users, that variant will usually lose the test by a large margin.
What's happening
Algolia A/B tests use the userToken sent at query time to keep track of which variant - A or B - the user is sees, and to ensure this remains consistent.
However, in cases where you have a mix of anonymous and identified users, you may wish to assign a single token to all anonymous users until they for example, accept cookies or sign in.
To Algolia A/B tests, as these users will all share the same token, they will appear as one single user who performs many thousands of searches. Usually, these users who didn't accept cookies are less engaged and less likely to convert so they can skew the test results.
How to fix it
To avoid these users skewing results, you have a couple of choices. Firstly you can opt to assign unique a userToken to each user, something like random UUIDs in the session.
Alternatively, you can append the enableABTest:false parameter to all queries made by users who have the single anonymous token. This will exclude users who have the anonymous token from the A/B tests. More on this can be found here.
Getting Ahead
You can check if you are affected by this issue before running proper tests by running a dummy or A/A test.
The easiest way to do this is by setting up a test with a rule context added to the variant, this won't do anything (unless you create rules) so in theory the two variants should perform identically. If you see a large difference in searches as above you know the issue is present.
Note: Incorrect userToken implementation can also cause users to be flagged as outliers, further skewing test results. If your tests are reporting a high number of outliers, read our guide article here for more details.
For a full list of common userToken issues, troubleshooting topics and best practices, see our summary article here.