Robots.txt & sitemap generator ready to upload.
Build a clean robots.txt and a valid XML sitemap in one place. Set your disallow rules and AI-crawler policy, paste your URLs, then copy each file straight to your site. It all runs in your browser.
robots.txt
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /search
Disallow: /api/
# AI search crawlers (help your page get cited)
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Training-only crawlers (block to withhold content for model training)
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
Host: https://example.comUpload the file to your site root (e.g. yourdomain.com/robots.txt). Runs in your browser — nothing is sent anywhere.
Two files that decide what Google can see
robots.txt controls crawling
One stray Disallow line can hide your whole site or a key folder from search engines. A clean file also states your sitemap location and sets your AI-crawler policy deliberately.
sitemap.xml aids discovery
A tidy sitemap of only indexable, self-canonical URLs helps crawlers find and weigh your pages. Include real last-modified dates and leave out redirects, noindex and parameter URLs.
Robots & sitemap questions
What is robots.txt and why does it matter?
It is a small file at your domain root that tells search crawlers which pages they can and cannot access. A wrong rule can accidentally block your whole site from being crawled, so it is worth generating carefully rather than guessing.
Do I still need an XML sitemap?
Yes. A sitemap lists the URLs you want indexed so search engines can discover and prioritise them, especially new or deeply linked pages. It is a discovery aid that helps crawling — it does not by itself improve rankings.
Should I allow or block AI crawlers?
Separate the two kinds. Search and citation bots like OAI-SearchBot and PerplexityBot can send visitors, so allow them. Training bots like Google-Extended and CCBot feed models, so block or allow on your licensing preference — it is a business choice, not a ranking one.
Which URLs belong in my sitemap?
Only live, indexable, self-canonical pages on one host — usually 200 responses you want ranked. Leave out redirects, noindex pages, filtered or parameter URLs and thank-you pages, and give each a real last-modified date.
Last updated October 2026 · Want a technical SEO pro to audit the rest? Book a free strategy call
Get the technical foundations right
Crawling, indexing, canonicals and schema — we handle the technical layer so your pages can actually rank.
A Zumper tool, built for Indian businesses
This free Robots.txt & Sitemap Generator is built and maintained by Zumper, the digital growth agency behind zumper.in. We build SEO-ready websites, tools and campaigns for businesses across India — from Lucknow and Delhi NCR to Mumbai, Bengaluru and pan-India.
It is part of the same free toolset we use on client work. If you want the output turned into a plan, a page or a campaign, the team that built it is one link away.
The short version
Free, no signup
Every tool on zumper.in is usable without an account.
Runs on zumper.in
The same domain, the same team and the same standards as our client work.
Pan-India support
Lucknow-based, working with businesses across India.
More free tools from Zumper
The rest of the set covers the checks and calculations that come up next.
Meta Tag Generator
Write clean title, meta description, Open Graph and Twitter Card tags with live character counts and a search-result preview.
Open toolFree · In-browserOn-Page SEO Checker
Paste a page's HTML and get an instant on-page audit — title, meta, H1, canonical, Open Graph, alt text and depth — scored out of 100.
Open toolFree · No signupUTM Link Builder
Turn any URL into a trackable campaign link with source, medium, campaign, term and content, then copy it clean.
Open tool