User-agent:*
Disallow: /cron/
Disallow: /modules/
User-Agent: *
Crawl-delay: 10
Request-rate: 1/60