Skip to main content

Algolia DevCon - June 28-29, 2023

Discover the future of the AI Stack with Algolia, be the first to see new platform demos and announcements, and get the most out of Algolia through interactive workshops.

REGISTER FOR FREE DON'T SHOW AGAIN
×

When are pages skipped or ignored?

  • Updated

At addition time

Pages skipped at addition time are not added to the URL database. This can happen for two reasons:

  1. The URL doesn’t match at least one of your actions’ pathsToMatch.
  2. The URL matches one of your crawler’s exclusionPatterns.

At retrieval time

Pages skipped at retrieval time are added to the URL database, retrieved, but not processed. Those are flagged “Ignored” in the Crawler dashboard. This can happen for these reasons:

  1. The robots.txt check didn’t allow this URL to be crawled.

  2. The URL is a redirect. The crawler adds the redirect target to the URL database but skips the rest of the page.

  3. The page’s HTTP status code is not 200.

  4. The media type isn’t one of the expected ones.

  5. The page contains a canonical link. The crawler adds the canonical target as a page to crawl according to the same addition-time filters and then skip the current page.

  6. The robots meta tag in the HTML is set to noindex or none. For example, <meta name="robots" content="noindex"/>. You can override this behavior by setting the ignoreNoIndex parameter to true.

Related articles