Make sure to check the robots.txt and see if the Crawler user agent has been whitelisted. You can read more about this on the Crawler FAQ and add the following lines at the top of your file:
1 User-agent: Algolia Crawler
2 Disallow:Algolia DevCon - June 28-29, 2023
Discover the future of the AI Stack with Algolia, be the first to see new platform demos and announcements, and get the most out of Algolia through interactive workshops.
Make sure to check the robots.txt and see if the Crawler user agent has been whitelisted. You can read more about this on the Crawler FAQ and add the following lines at the top of your file:
1 User-agent: Algolia Crawler
2 Disallow: