Why does robots.txt count?
An incorrectly configured robots.txt is a classic and costly mistake: if you accidentally block your entire site, you will disappear from the search results. Used correctly, you help search engines focus their crawl budget on what matters. It's part of the basics of your technical SEO.
What should you know about robots.txt?
Robots.txt is a public text file at the root of a host that tells specified crawlers which paths they may or may not request.
| Quick fact | Practical explanation |
|---|---|
| Meaning | Robots.txt is a public text file at the root of a host that tells specified crawlers which paths they may or may not request. |
| Primary use | Use robots.txt to reduce unnecessary crawling, such as internal search results or unlimited filter combinations. Do not use it to protect confidential information. |
| Practical check | Open the file at /robots.txt, inspect each user-agent group and test representative URLs. Rules apply separately to each protocol, hostname and port. |
| Common mistake | A blocked URL can sometimes still appear as a URL-only search result when other pages link to it. Use noindex for indexing exclusion and allow the crawler to fetch that instruction. |
How does robots.txt work in practice?
Use robots.txt to reduce unnecessary crawling, such as internal search results or unlimited filter combinations. Do not use it to protect confidential information.
- Define the crawl problem: Block only paths where crawling does not contribute useful search value.
- Write specific groups: Associate Allow and Disallow rules with the intended user-agent.
- Publish at host root: Place the file at the exact root location of every relevant host.
- Test critical URLs: Confirm that CSS, JavaScript, images and indexable pages remain accessible.
- Monitor changes: Review crawl errors, rendering and indexing after rules are modified.
How do you check robots.txt?
Open the file at /robots.txt, inspect each user-agent group and test representative URLs. Rules apply separately to each protocol, hostname and port.
- The file is accessible at the exact /robots.txt URL.
- Important landing pages and rendering resources are not blocked.
- Disallow rules do not contain unintentionally broad patterns.
- Every Sitemap declaration uses a complete absolute URL.
- Crawler groups for Google, OpenAI and Perplexity are deliberate.
- Sensitive information is protected through authentication, not robots.txt.
Which mistakes should you avoid with robots.txt?
A blocked URL can sometimes still appear as a URL-only search result when other pages link to it. Use noindex for indexing exclusion and allow the crawler to fetch that instruction.
| Mistake | Why it matters |
|---|---|
| Using robots.txt as noindex | A crawler cannot read a noindex tag when it is not allowed to fetch the page. |
| Only blocking a staging site | The file is public and does not provide access control for confidential environments. |
| Blocking CSS and JavaScript | Search engines may be unable to render and evaluate the page correctly. |
How do you verify robots.txt in practice?
Combine syntax checks with tests of real URLs. The goal is not to block as much as possible but to keep useful crawling predictable.
| Signal | Interpretation |
|---|---|
| Blocked important URLs | A crawler reveals whether indexable pages or required assets are excluded accidentally. |
| Crawl activity | Server logs and Search Console show which paths crawlers actually visit. |
| Indexing warnings | Reports about blocked pages identify conflicts between crawl and indexing goals. |
How does robots.txt relate to other SEO concepts?
robots.txt rarely operates in isolation. Use the related concepts below to connect this definition with the next technical, content or measurement decision for your website, audience and market.
- XML sitemap: An XML sitemap is a machine-readable file listing URLs that a website wants search engines to discover and crawl.
- crawl budget: Crawl budget combines how much crawling a website's infrastructure can support with how much Google wants to crawl its URLs at a given time.
- Google Search Console: Google Search Console is a free Google service that helps website owners monitor search performance, indexing and technical signals for their website in Google Search.
Which primary source supports this explanation?
The factual basis for this page includes Google Search Central documentation about robots.txt and robots rules. The practical recommendations combine that documentation with page-level SEO analysis.
More questions about robots.txt
Can robots.txt remove a page from Google?
Not reliably. A blocked URL can remain visible without a descriptive snippet. Use noindex or removal methods when the URL must not be indexed.
Is a robots.txt file required?
No. Without a file, compliant crawlers can generally request all accessible paths. The file is useful when explicit crawl rules or sitemap declarations are needed.
Can different crawlers receive different rules?
Yes. You can define user-agent-specific groups, but verify each system's official crawler names and documentation.