Advanced Robots.txt Generator for Blogger & Any Site | Free SEO Tool

Advanced Robots.txt Generator (Google/Bing Ready)

Generate a clean, standards‑friendly robots.txt with sitemaps, user‑agent rules, and optional bad‑bot blocks. Lightweight, no dependencies, copy or download in one click.

What this page does (SEO introduction)

Use this tool to craft a compliant robots.txt for Blogger, WordPress, Shopify, or any site. Add per‑bot rules, sitemaps, optional crawl delays (for Bing/Yandex), and common Blogger presets. Then paste it into your platform’s robots.txt settings.

Blogger preset Google/Bing friendly Copy/Download Zero dependencies JSON‑LD

How to use

1. Enter your site URL and click “Blogger recommended” (optional).
2. Add any Allow/Disallow rules and sitemaps you need.
3. Click Generate to preview the robots.txt on the right.
4. Copy or Download and add it to Blogger → Settings → Crawlers and indexing → Custom robots.txt.
5. Test in Search Console and Bing Webmaster Tools.

Caution: Google ignores Crawl-delay and Clean-param. For true noindex, use meta robots or HTTP X‑Robots‑Tag.

Robots.txt Builder

Site URL

Use your canonical root URL (protocol + host).

Host (optional, Yandex)

Sitemaps (one per line)

Clean‑param (Yandex, optional)

Format: params separated by & then a space and a path. Example above cleans UTM and gclid site‑wide.

Include helpful comments

Adds context with “#” comments. Safe to leave on.

Block Blogger ?m=1 duplicates

Adds Disallow: /*?m=1 (non‑Google bots may respect this).

Blogger recommended Open (allow all) Maintenance (block all) Toggle bad‑bot blocks

User‑agent groups

+ Add user‑agent group Default group “*” is required; others are optional.

Generate Copy Download robots.txt Reset Remember my inputs (local only)

Preview

Enter your site URL and click Generate…

Tip: Testing & validation

• Google: Search Console → Settings → Robots.txt
• Bing: Robots.txt Tester

FAQ

What is robots.txt?

A publicly accessible file at your domain root that gives crawlers instructions (e.g., which paths they can or can’t fetch). It does not guarantee noindex.

Does robots.txt prevent indexing?

No. It blocks crawling, not indexing. Use meta robots or X‑Robots‑Tag for “noindex”.

Where do I put robots.txt on Blogger?

Blogger → Settings → Crawlers and indexing → Enable custom robots.txt → Paste the generated text → Save.

Should I disallow /search on Blogger?

Often yes, to avoid duplicate label/search pages. The “Blogger recommended” preset includes Disallow: /search and Allow: /.

Does Google support Crawl-delay or Clean-param?

Google ignores both. Bing and Yandex support them. Use only if you have a specific need.

How do I block a specific bot?

Add a new user‑agent group (e.g., AhrefsBot) with Disallow: /. Some bots may ignore robots.txt.