#Google Search Engine Robot User-agent: Googlebot # Crawl-delay: 10 -- Googlebot ignores crawl-delay ftl Disallow: /*/common/* #Yahoo! Search Engine Robot User-Agent: Slurp Crawl-delay: 1 Disallow: /*/common/* #Microsoft Search Engine Robot User-Agent: msnbot Crawl-delay: 10 Disallow: /*/common/* # Every bot that might possibly read and respect this file. User-agent: * Disallow: /*/common/*