The reasons, why one of your pages wasn’t crawled, include:
- Crawling a website completely can take hours depending on the size: make sure that the crawling operation has finished.
- Some pages may not be linked with each other: make sure that there exists a way to navigate from the website’s start pages to the missing page, or that the missing page is listed in the
sitemaps
. If it is inaccessible, you may want to add its URL as a start URL in your crawler’s configuration file. - The page may have been ignored if it refers to a canonical URL, or if it doesn’t match a
pathsToMatch
in any of your crawler’s actions, or if it matches anyexclusionPatterns
. For more information, checkout the question: when are pages skipped or ignored? - If the page is rendered using JavaScript, you may need to set
renderJavaScript
totrue
in your configuration (note: this makes the crawling process slower). - If the page is behind a login wall, you may need to setup the
login
property of your configuration.
If none of these solve your problem, an error may have happened while crawling the page. Please check your logs using the Monitoring and URL Inspector tabs.
You can also use the URL tester in the Editor tab of the Admin to get details on why a URL was skipped / ignored.