Robots.txt
Robots.txt is a file at your domain root telling crawlers which paths they may request. It manages crawl waste effectively, but it does not remove pages from the search index and provides no security at all, since the file itself is public.
Why it matters for your rankings
Robots.txt is powerful and frequently misunderstood. Blocking a URL prevents crawling, not indexing. Google can still list a blocked page it found through links, showing the URL with no description at all.
The dangerous consequence is that combining a robots.txt block with a noindex tag guarantees the noindex is never read, so the page stays indexed indefinitely. It is also the single easiest file to break catastrophically: one stray Disallow line can deindex an entire site, and it happens most often when staging configuration reaches production. Since the file is publicly readable at a predictable URL, using it to hide sensitive directories advertises their existence rather than protecting them.
How to check it on your site
Read the live file
Open yourdomain.com/robots.txt and check every Disallow line against what you actually intend to block.
Test in Search Console
The robots.txt report shows how Google parses the file and flags URLs it blocks.
Never block pages you want deindexed
Use a noindex directive and leave the URL crawlable, because a page blocked in robots.txt can never have its noindex tag read by Google.
Check it after every deploy
Staging rules reaching production is the most common cause of sudden total traffic loss.