Skip to content
Utilerio
All endpoints

Robots.txt Generator

POST/api/v1/tools/robots-txt-generator

Build a robots.txt with per-agent allow/disallow rules and a sitemap reference.

Authentication

Requires an API key in the X-API-Key header.

Try it

POST/api/v1/tools/robots-txt-generator

Parameters

groupsstringrequired

Rule groups (array): [{ userAgent, disallow, allow }], where disallow/allow are newline-separated paths.

sitemapstringoptional

Sitemap URL appended as a Sitemap: directive.

Request body

{
  "groups": [
    {
      "userAgent": "*",
      "disallow": "/admin\n/api",
      "allow": ""
    }
  ],
  "sitemap": "https://example.com/sitemap.xml"
}

Response codes

  • 200Success.
  • 400Invalid request — check your body or parameters.
  • 401Missing or invalid API key.
  • 403The API key is not scoped to this endpoint.
  • 429Rate limit exceeded — retry after a short wait.

Language

cURL

Request

curl -X POST https://utilerio.com/api/v1/tools/robots-txt-generator \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"groups":[{"userAgent":"*","disallow":"/admin\n/api","allow":""}],"sitemap":"https://example.com/sitemap.xml"}'

Response

{ "robotsTxt": "User-agent: *\nDisallow: /admin\nDisallow: /api\n\nSitemap: https://example.com/sitemap.xml" }