# robots.txt for job-scouts.com # T1 (14 Oct 2026): crawlers used most of the server's requests (ClaudeBot # about 48k a day, PerplexityBot 24k, ShapBot 13k, Meta 13k, Amazon 8k, # Semrush 7k), including about 7,000 fetches of the register page, once per # job, through its ?next= links. Those pages are now off limits, and the # heaviest bulk crawlers are asked to slow down. Googlebot and Bingbot ignore # or don't need Crawl-delay and are not slowed. User-agent: * Allow: / Disallow: /admin/ Disallow: /analytics/ Disallow: /api/ Disallow: /media/private/ Disallow: /jobs/*/go/ Disallow: /jobs/*/apply/ Disallow: /jobs/*/quick-apply/ Disallow: /jobs/*/report/ Disallow: /accounts/*?next= Disallow: /talent-pool/? Disallow: /*?*utm_ Disallow: /v/ Crawl-delay: 1 # Bulk crawlers (AI training, data collection): same rules, slower. User-agent: ClaudeBot User-agent: Amazonbot User-agent: meta-externalagent User-agent: ShapBot User-agent: Bytespider User-agent: CCBot User-agent: GPTBot Allow: / Disallow: /admin/ Disallow: /analytics/ Disallow: /api/ Disallow: /media/private/ Disallow: /jobs/*/go/ Disallow: /jobs/*/apply/ Disallow: /jobs/*/quick-apply/ Disallow: /jobs/*/report/ Disallow: /accounts/*?next= Disallow: /talent-pool/? Disallow: /*?*utm_ Crawl-delay: 5 # SEO tool crawlers: no value for job seekers or employers. User-agent: SemrushBot User-agent: AhrefsBot User-agent: MJ12bot User-agent: DotBot User-agent: PetalBot Disallow: / Sitemap: https://job-scouts.com/sitemap.xml