# www.robotstxt.org/ # http://code.google.com/web/controlcrawlindex/ User-agent: *Disallow: */preview/* User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: /