Pre-caching involves proactively caching resources on the CDN network so that they are readily available for users before they visit a web page or use an app. This technology streamlines the loading process, allowing users to access what they need quickly and eliminating the need for the first cache HIT.
Excluding URLs from Pre-Cache crawling
You can exclude specific paths from being crawled using a .crawlrc file. This file contains a list of paths you want to exclude from crawling, and it must be a valid JSON file formatted as follows:
{
"SKIP_REGEX": "/basket|contact|cart/"
}The only key should be SKIP_REGEX, and the value should be a combination of paths to be excluded from your site. You can use regex, where the | symbol separates multiple paths. In the example above, we excluded /basket, /contact, and /cart from being crawled.
With the .crawlrc file, you have greater control over what URLs get crawled on your site.