robots.txt vs noindex: which one actually keeps a page out of Google
robots.txt controls which URLs crawlers may fetch; noindex controls whether a fetched page may appear in search results. Mixing them up produces the classic failure: a page blocked in robots.txt can still be indexed, while a noindex hidden behind that same block is never seen. The difference, the trap, and the right tool per job - from Google Search Central's own documentation, read 21 November 2026.
Every site eventually has a page that should not show up in Google: a staging copy, a thank-you screen, an internal tool. Two mechanisms get reached for, and Google's own documentation warns they are not interchangeable - choosing the wrong one either leaves the page visible or hides far more than intended. Everything below comes from Google Search Central's robots.txt guide and its noindex guide, both last updated 10 December 2025 and read 21 November 2026.
robots.txt manages crawler traffic, not visibility
Google states it in the first paragraph of its guide: a robots.txt file tells search engine crawlers which URLs they may access on your site, it is used mainly to avoid overloading your site with requests, and "it is not a mechanism for keeping a web page out of Google." The file does have legitimate visibility uses by file type: for image, video and audio files, robots.txt can keep those media files out of Google Search results; for ordinary web pages it only manages crawl traffic. Two more limits from the same page matter in practice. First, obedience is voluntary - Googlebot and other respectable crawlers follow robots.txt, but other crawlers might not, so the file is no security boundary; private material needs password protection on the server. Second, different crawlers interpret the syntax differently. Google also warns against blocking resource files such as scripts and stylesheets that its crawler needs to understand the page.
A blocked URL can still be indexed
This is the part that surprises people. If other pages link to your robots.txt-blocked page with descriptive text, Google can still index the URL without ever visiting it: the address - and potentially the anchor text - appears in results with no description. Google's fix list for genuinely keeping a page out of results is: password-protect the files on your server, use the noindex rule, or remove the page entirely. So robots.txt alone is the wrong tool whenever the goal is "this must not appear in Search."
noindex removes the page from Google results - if Google can see it
noindex is a rule set with a meta tag (<meta name="robots" content="noindex">) or an X-Robots-Tag HTTP response header; when Googlebot crawls the page and reads the rule, Google drops the page entirely from Search results, regardless of whether other sites link to it. The header form works for non-HTML resources like PDFs, videos and images. Two implementation notes from the guide: using name="googlebot" instead of name="robots" targets only Google's crawlers, and specifying the noindex rule inside the robots.txt file itself is not supported by Google.
The trap: robots.txt can hide your noindex
Google's noindex guide carries the warning that ties the two mechanisms together: "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file." If crawling is blocked, the crawler never fetches the page, never sees the noindex, and the URL can still appear in results via inbound links - the worst of both worlds, since you have also lost the crawl. The same guide explains the common debugging case: a page that still appears after you added noindex usually has not been recrawled since the change (revisits can take months, depending on the page), or its robots.txt entry is hiding the rule. Google's suggested verification tools are the URL Inspection tool in Search Console - to see the HTML Googlebot actually received - and the Page Indexing report, which lists pages from which a noindex rule was extracted. For fast removals there is separate removals documentation.
Which tool for which job
Per Google's own framing: to keep a page out of Google Search results, use noindex or password protection - the first leaves the page public to visitors but out of the index, the second restricts access itself - robots.txt will not do it. To reduce crawler load on your server, or to stop crawling of unimportant or near-duplicate pages, use robots.txt. To keep image, video or audio files out of Google results, robots.txt is one documented option. And if your site runs on a CMS such as Wix or Blogger, both guides note you may not edit robots.txt or page HTML directly at all - the CMS's own search-visibility settings are the mechanism. Google also cautions that combining multiple crawling and indexing rules can make rules counteract each other, so changes here deserve a check in the URL Inspection tool afterward.
Sources, both read 21 November 2026 and last updated 10 December 2025: Google Search Central, Robots.txt Introduction and Guide (crawler-traffic purpose; "not a mechanism for keeping a web page out of Google"; per-file-type effects including media files; blocked URLs still indexable with anchor text; voluntary obedience and per-crawler syntax differences; resource-blocking warning; CMS note); Google Search Central, Block Search indexing with noindex (meta tag and X-Robots-Tag header forms; page dropped entirely regardless of links; robots.txt blocking hides the noindex rule; noindex in robots.txt unsupported; recrawl timing, URL Inspection and Page Indexing report guidance).
Related: Canonical vs robots.txt vs noindex: choose the right control, for when duplicate URLs and launch-wide signal checks are the job.
Frequently asked questions
Does robots.txt hide a page from Google?
No. Google states robots.txt "is not a mechanism for keeping a web page out of Google" - it manages which URLs crawlers may fetch. A blocked URL can still appear in results, without a description, if other sites link to it with descriptive text. To keep a page out, Google lists password protection (which blocks visitors too), noindex (which only removes it from results), or removing the page.
Which one removes a page from Google results?
noindex - as a meta tag or an X-Robots-Tag HTTP header. When Googlebot crawls the page and reads the rule, Google drops the page entirely from results regardless of inbound links. robots.txt only controls crawling; it does not remove anything from the index by itself.
I added noindex but the page still appears. Why?
Google's guide gives two common causes. The page may not have been recrawled since you added the rule - revisits can take months depending on the page - or your robots.txt is blocking the URL, so the crawler never fetches the page and never sees the noindex. Check with the URL Inspection tool, which shows the HTML Googlebot received, and the Page Indexing report.
Can I write noindex inside robots.txt?
No - Google's noindex guide states that specifying the noindex rule in the robots.txt file is not supported by Google. Use the meta tag or the X-Robots-Tag response header instead.
Will robots.txt keep bad bots and scrapers out?
Not reliably. Google notes the instructions cannot enforce crawler behavior: respectable crawlers obey them, others might not. For genuinely private material, Google's own recommendation is password protection on the server, not robots.txt rules.
What about PDFs, images and videos?
Two documented routes. An X-Robots-Tag: noindex HTTP header works on non-HTML resources such as PDFs, videos and images. Separately, Google's robots.txt guide says robots.txt can be used to prevent image, video and audio files from appearing in Google Search results - a media-specific exception to the general rule that robots.txt does not hide pages.