User-agent: * Disallow: /counter/ User-agent: Slurp Crawl-delay: 1000