If your DocSearch reindexing process fails with the following error:
"The difference between the number of records: from: 29.8k to: 0 is too large (100%)"
…it means that the crawler detected a large discrepancy between the previous number of records and the latest crawl results. This safeguard is designed to prevent accidental data loss due to unexpected content changes or misconfigurations.
Why this happens
The error occurs when the crawler expects to find a similar number of records during a re-crawl but ends up retrieving far fewer (or none). This can happen due to:
Significant changes to the site structure or content
Incorrect selectors or configuration issues
Temporary unavailability of the site or blocked URLs
Crawling exclusions or permissions issues
Solutions
1. Retry the crawl
In many cases, this error is temporary. Try restarting the crawl:
Click the “Cancel” button in the DocSearch dashboard
Start a new crawl to see if the issue persists
2. Use aggregateContent: true
This configuration option helps the crawler group related content together more effectively, which can reduce the chance of record drops.
Refer to the documentation here:
Record Extractor | DocSearch
3. Adjust the maxLostRecordsPercentage
You can increase the maxLostRecordsPercentage in your crawler configuration. This changes the threshold at which the system considers the record drop too significant to continue.
See the relevant documentation for guidance on adjusting this value and understanding its impact:
Understanding Crawler Limits
The free DocSearch plan, doesn't include technical web support from the Support team. For further assistance, you can post your question on the Algolia Community Forum or Discord. For front-end issues, you can raise the issue directly on GitHub. In addition, you can review our DocSearch Documentation and FAQ.