Tools / 06
Robots.txt Generator & Tester
Build crawl directives, declare a sitemap and test whether a path is allowed.
Use this tool ↓Loading the interactive surface…
How to use it
- 01
Choose a crawler, add only paths that should not be requested and test representative URLs.
- 02
Keep public CSS and JavaScript crawlable when they are required to render a page.
- 03
Use noindex or access control when content must stay out of search; robots.txt alone is not enough.
Can robots.txt remove a URL from Google?
No. A blocked URL can still be indexed from external signals. Use noindex on a crawlable page or protect private content.
Where does the file belong?
At the root of the hostname, for example https://example.com/robots.txt.