Español
Deutsch
Robots.txt is a simple text file telling search engine crawlers which parts of your site they should or shouldn't access.
What It's Actually For
It's mainly used to keep crawlers away from admin areas, duplicate content, or resource-heavy pages that don't need to be indexed — not a security tool, since the file itself is publicly readable.
The Costly Mistake
A misconfigured robots.txt accidentally blocking your entire site (a stray Disallow: /) is a surprisingly common cause of a site vanishing from search results — always double-check after any change.
