Configure Custom Crawl Settings for Your Site Architecture
Before running a crawl, a specialist reviews your robots.txt, sitemap, and URL structure to set crawl scope, user-agent, and rendering mode correctly. Misconfigured crawls return misleading data, so this setup step is treated as a deliverable, not a formality. You receive a saved configuration file you can reuse or hand off.