robots.txt Generator
Build a valid robots.txt file visually — configure user-agent groups, allow/disallow rules, crawl delays, and sitemap URLs in real time.
Load Preset:
Rules
Added as a
Sitemap: directive at the end of the file.Yandex-specific
Host: directive specifying the preferred domain.Understanding robots.txt Directives
Disallow
Prevents search engine bots from crawling a specified path. Use Disallow: / to block the entire site, or Disallow: /admin/ to block a directory.
Allow
Explicitly permits access to a path, even if a broader Disallow rule would otherwise block it. Useful for WordPress's /wp-admin/admin-ajax.php.
Crawl-delay
Asks bots to wait a specified number of seconds between requests. Helps reduce server load. Note: Googlebot ignores this — use Google Search Console to manage crawl rate instead.