When the ignoreNoIndex and ignoreNoFollowTo settings are enabled in Algolia's Crawler, the following behaviour is anticipated:
-
ignoreNoIndex:true
andignoreNoFollowTo:true
let the crawler ignore the directive of “NoIndex“ and “NoFollow“, so the crawler EXTRACTS records from the page -
ignoreNoIndex:false
andignoreNoFollowTo:false
let the crawler follow the directive of “NoIndex“ and “NoFollow“, so the crawler DOES NOT EXTRACT records
However, the correct way to ignore pages it to use exclusionPatterns: Crawler: ExclusionPatterns | Algolia.