Where should the robots.txt file be uploaded on a web server?

The robots.txt file must always be placed directly in the top-level root directory of your website domain (e.g. https://example.com/robots.txt). Search engine crawlers will not check subdirectories or alternate file paths.

Does disallowing a page in robots.txt prevent it from appearing on Google?

Not necessarily. Disallowing a URL in robots.txt prevents Googlebot from crawling the page, but if other external websites link to that URL, Google can still index the link without descriptive page content. To ensure complete removal from search results, use a 'noindex' meta robots tag instead.

How do I block AI scrapers (like GPTBot or CCBot) in robots.txt?

You can block specific AI web scrapers by adding dedicated user-agent blocks, such as: User-agent: GPTBot followed by Disallow: / or User-agent: CCBot followed by Disallow: /.

Technical SEO Utility

Free Robots.txt Generator

Control how Googlebot, Bingbot, and AI web scrapers index your website. Generate clean, valid robots.txt rules in 1 click.

Crawler Settings

Default Robot Permission (User-agent: *):

Crawl-Delay Directive:

XML Sitemap URL:

Disallow Sensitive System Folders:

/admin/ /cgi-bin/ /cart/ /checkout/ /search/ /tmp/

Block AI Web Scrapers (GPTBot, CCBot, Anthropic)

Custom Disallow Paths (1 per line):

robots.txt Preview Valid Syntax

Responsive Ad Space (300x250 / 728x90)

Configure ADSENSE_CLIENT_ID in config.php to display live monetization units.

Understanding Robots.txt Directives & Crawl Budget Optimization

A robots.txt file is a simple text file implementing the Robots Exclusion Protocol (REP). It acts as the polite front gate for web crawlers, informing search engine user-agents (Googlebot, Bingbot, YandexBot, Baiduspider) which sections of your website they are allowed or forbidden from crawling.

Preventing crawlers from wasting time on duplicate administrative scripts, internal search query facets, and temporary folders preserves your domain's Crawl Budget. This ensures search engines index your newly published blog posts and valuable landing pages significantly faster.

Core Directive Syntax

User-agent: * Applies rule to all bots
Disallow: /admin/ Blocks entire directory
Allow: /admin/login.php Explicitly allows one file
Sitemap: [URL] Directs bots to your XML index

Critical Technical SEO Warnings

Common robots.txt pitfalls that can de-index entire production websites:

Accidental Disallow / : Leaving Disallow: / in production de-indexes your entire domain from Google.
CSS/JS Files: Never disallow /wp-content/themes/ or CSS/JS folders; Google needs them to render pages.

Common Web Crawler User-Agent Directives

Crawler Entity	User-Agent Token	Owner Organization	Recommended Access
Google Search Desktop & Mobile	Googlebot	Google LLC	Allow (Essential for SEO)
Bing Search Crawler	Bingbot	Microsoft Corp	Allow (Bing & Copilot Search)
OpenAI LLM Scraper	GPTBot	OpenAI	Optional (Disallow if protecting training data)
Common Crawl Web Corpus	CCBot	Common Crawl Foundation	Optional

Frequently Asked Questions (FAQ)

Is robots.txt case-sensitive?

Yes. Directives and path names are case-sensitive. Disallow: /Admin/ will not block crawlers from visiting /admin/.

Does Google obey Crawl-delay?

Googlebot ignores the Crawl-delay directive in robots.txt (crawl rate is managed dynamically inside Google Search Console). However, Bing and Yandex respect it.

Can I add multiple sitemaps?

Yes, you can specify multiple Sitemap: https://... directives on separate lines at the bottom of your robots.txt file.