COMPARISON · UPDATED 1 OCT 2026

Noindex vs Robots.txt

Noindex and robots.txt are both used to keep pages away from Google, but they do different jobs. One controls what appears in results, the other controls what gets crawled.

THE ONE THAT REMOVES A PAGE FROM SEARCH

Noindex

An instruction on the page itself, as a meta robots tag or an X-Robots-Tag header. Crawlers may read the page, but it stays out of search results.

Best for: pages people may visit but that should not appear in search

Robots.txt

A text file at the root of your site that tells crawlers which addresses not to visit. It controls crawling, not indexing.

Best for: areas where crawling is a waste of time, such as cart pages and internal search results
SIDE BY SIDE

What each one gives you.

FeatureNoindexRobots.txt
What it isA meta robots tag or X-Robots-Tag header on the pageA text file at the root of the site
What it controlsIndexingCrawling
Crawlers can read the pageYesNo
Keeps the page out of resultsYesthe page must be crawlableNothe address can still be indexed without its content
Stops crawling of unimportant areasNoYes
Protects private contentNoNo
Typical useA thank-you page after a formCart pages and internal search results

Choose Noindex if

  • You want a page to stay out of search results, such as a thank-you page after a form.
  • The page must still work for visitors who have the link.
  • A page is already in the results and you want it to drop out after it is crawled again.
  • The file is not an HTML page, such as a PDF, and you can send an X-Robots-Tag header.

Choose Robots.txt if

  • Crawlers spend time on areas with no search value, such as cart pages or internal search results.
  • You want to keep crawlers out of a whole folder with one rule.
  • You have a very large site and want crawlers to spend their time on the pages that matter.
  • You can accept that a blocked address may still appear in results without a description.

The honest answer

These two are often treated as the same thing. They are not. Robots.txt controls crawling, which is whether a crawler may visit an address. Noindex controls indexing, which is whether a page may appear in results.

If your goal is to keep a page out of search, noindex is the one that does it. Leave the page open to crawlers and add the tag. Google reads the page, finds the instruction and keeps the page out of results.

Robots.txt cannot promise that. A blocked page can still be indexed without its content if other pages link to it. It can show up as a bare address with no description.

The worst choice is both on the same page. If robots.txt blocks the address, the crawler never reads the noindex tag, so the tag does nothing.

Robots.txt still has a job. Use it for areas where crawling is a waste of time, such as cart pages and internal search results. Neither one is a lock. Anyone with the address can open the page, so private content needs a login.

The full explanation is in crawling and indexing, explained simply. To check one of your own pages, run the free audit.

KEY TAKEAWAYTo remove a page from search, allow crawling and add noindex. Use robots.txt to save crawling, not to hide pages.
MORE COMPARISONS

Other head-to-heads.

All comparisons
ANALYTICS

GA4 vs Plausible

Depth and a free tier against simplicity and privacy.

Read comparison
PLATFORMS

WordPress vs Webflow for SEO

Full control and plugins against built-in SEO fields and less upkeep.

Read comparison
PLUGINS

Yoast SEO vs Rank Math

A guided, simple setup against more features in the free version.

Read comparison