The Crawler aims at crawling content that’s solely owned by the user, whether it’s hosted on its own infrastructure or on a SaaS. Algolia automatically restricts the Crawler’s scope to a list of allowed domains. If you need to add an other domain to your allowed list go the Domains page and verify your domain.