# robots.txt for https://www.cap.org/ sitemap: https://www.cap.org/sitemaps-1-sitemap.xml # live - don't allow web crawlers to index cpresources/ or vendor/ User-agent: * Disallow: /cpresources/ Disallow: /vendor/ Disallow: /.env Disallow: /cache/ # Prevent indexing and allow search via swiftype User-agent: Swiftbot Disallow: # Allow Dead Link Checker to search User-agent: www.deadlinkchecker.com Disallow: