Glossary / Robots.txt
What is Robots.txt?
Robots.txt is a small text file at the root of your site that tells search engine bots which areas to skip. It is a request, not a lock.
It sits at yoursite.com/robots.txt and anyone can read it. The rules are simple: name a bot, then name the paths it should leave alone. Well-behaved crawlers follow it. Bad ones ignore it entirely.
It controls crawling, not indexing, and people mix those up constantly. A page blocked here can still appear in results if other sites link to it, because the block stops the search engine reading the page, not knowing it exists.
One stray line can hide an entire site, and it happens most often when a staging rule ships to the live server. Check yours after every launch, and use it to point at your sitemap while you are in there.
Related terms