Create a robots.txt file for a website
Draft Allow or Disallow rules for the User-agent * group and add a sitemap URL if needed. Review paths before publishing: robots.txt guides crawling; it does not secure private data or request removal from Google’s index.
Draft rules and preview the file
Rules for User-agent: *
From rules to a robots.txt file
- Start with Allow: / or choose a WordPress, shop, or blog pattern, then edit each path for your site. Patterns are starting points; choosing one replaces all current rules.
- Add Allow/Disallow rules with paths starting with /. The draft has only a User-agent: * group; if a file also has a group for a specific bot, that bot may use its specific group instead.
- If you have a sitemap, enter its full URL. Review the preview, download robots.txt, then serve it at /robots.txt on the correct host and protocol. If your site already has a file, merge the needed rules instead of overwriting existing content.
Check rules that are easy to misread
| In the file | Meaning | Review action |
|---|---|---|
Disallow: / | Blocks the entire site from crawlers that follow this group. | Use only if you truly want to block the whole site; retest after publishing. |
Disallow: /folder/ | Blocks paths beginning with /folder/. Paths are case-sensitive. | Compare the actual path; avoid blocking CSS/JS needed to render pages. |
Allow: /folder/page | Can permit a more specific path when a broader Disallow rule exists. | Test the actual rule pair; line order does not replace path matching. |
Sitemap: https://… | Declares the absolute sitemap URL; it does not create or submit a sitemap. | Open the URL to confirm the file exists and lists the intended pages. |
What does robots.txt control?
The tool builds the file and checks path/sitemap URL formats in your browser; it does not fetch your website, verify access, or prove that a crawler will follow the rules. robots.txt is public and does not hide sensitive pages. A disallowed URL can still appear in search results if Google discovers it elsewhere.
Use access control to protect private content. To prevent indexing, use noindex on a page Google can crawl; blocking crawling can keep Google from seeing noindex. After publishing, open /robots.txt on the correct host and inspect an important URL in Search Console. Google’s robots.txt guide.
XML Sitemap Generator
Crawl linked public HTML pages on one website, review discovered URLs, and download an XML sitemap to check before publishing.
Website SEO Checker
Check technical SEO signals on one public URL, with sampled internal links and a separate mobile PageSpeed report.
Google Index Checker
Check whether a URL appears in a Google site: search. Use URL Inspection in Search Console for official verification.
Meta Tag Analyzer
Inspect metadata in the public HTML of one URL, including title, description, canonical, robots, sharing tags and JSON-LD blocks.
Comments & feedback
Loading ratings…
Write a commentNo account needed