User-agent: * Crawl-delay: 5 Disallow: /cgi-bin/ User-agent: Googlebot Crawl-delay: 5 Disallow: /cgi-bin/ User-agent: 360Spider Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: Baiduspider Disallow: / User-agent: MJ12bot Disallow: / User-agent: AhrefsBot Crawl-Delay: 10