Blocking an admin area
User-agent: *, Disallow: /admin, /api
User-agent: * Disallow: /admin Disallow: /api
Robots.txt Generator
Build a robots.txt with per-agent allow/disallow rules and a sitemap reference.
robots.txt
User-agent: * Disallow: /admin Disallow: /api Sitemap: https://example.com/sitemap.xml
Was this tool useful?
About this tool
robots.txt is the first file a well-behaved crawler checks before touching anything else on a site — a plain-text set of rules saying which paths it may or may not fetch, per user-agent. It's a request, not a lock: a crawler that ignores robots.txt (some do) isn't stopped by it, but every major search engine respects it.
How to use it
- Set the user-agent for a rule group — * matches every crawler; a specific one like Googlebot only affects that crawler and needs its own group.
- List paths to disallow, one per line, and any exceptions to allow underneath a broader disallow.
- Add more rule groups for crawlers that need different treatment, and set your sitemap URL at the bottom.
- Copy the result into a file literally named robots.txt at your site's root — /robots.txt, not any subfolder.
Examples
Common problems
- I disallowed a path but it's still showing up in Google's search results.
- robots.txt only stops crawling, not indexing — a URL Google already knows about from a link elsewhere can still appear (usually with no snippet). Use a noindex meta tag on the page itself to actually keep it out of search results.