Robots.txt is a text file at the root of your website that tells search engines which parts they can and cannot crawl. It controls crawling behavior, but is not a lock: it is not guaranteed to keep pages out of the index. You use a noindex instruction for this.
An incorrectly configured robots.txt is a classic and costly mistake: if you accidentally block your entire site, you will disappear from the search results. Used correctly, you help search engines focus their crawl budget on what matters. It's part of the basics of your technical SEO.
Not guaranteed. Robots.txt controls the crawling, but to really keep a page out of the index you use a noindex instruction.
In the root folder of your domain, accessible via yourdomein.be/robots.txt.
Schedule a no-obligation discovery call. We look at where you stand in search engines and in AI, and where the biggest opportunities lie.
20 minutes | online or by phone | without obligation
