Why you should use the X-Robots-Tag header on your website

Last updated December 5th, 2023 22:52

Do you also focus on SEO on your website, fine-tuning any details that might push you up even one notch in the search results? Then you’ve probably already placed a robots.txt file on your site, the basics for better indexing are covered. However, do you know what the X-Robots-Tag header is for and use it on your website? If you’ve just answered ‘no’ to both questions, then this article is precisely for you. We’ll explore why you should use the X-Robots-Tag header on your website. Additionally, if you use WordPress, I’ll show you a simple snippet to add this header to your site.

Why You Should Use the X-Robots-Tag Header on Your Website

X-Robots-Tag is a special HTTP header that allows webmasters to control how their web pages should be indexed. This header serves a similar function to the robots.txt file mentioned earlier in the introduction, but it has one crucial difference. While the robots.txt file is a file you place on the server (via FTP), X-Robots-Tag is a header that is part of every HTTP request to a web page.

The X-Robots-Tag header has the following values that communicate with search engines:

  • index: Index the page.
  • noindex: Do not index the page.
  • follow: Index the page and follow links from it.
  • nofollow: Do not index the page and do not follow links from it.

You can use the X-Robots-Tag header to control the indexing of different types of content on your website. For example, you can use X-Robots-Tag to prevent indexing of certain pages with personal data or to block indexing of advertisement pages.

Examples of X-Robots-Tag Header

If you want to prevent search engines from indexing your advertisement pages, you can add the following header to each request for these pages:

					X-Robots-Tag: noindex, nofollow

Do you need search engines to index your pages but not follow the links placed on them? Conversely, add the following header to each request for these pages:

					X-Robots-Tag: index, nofollow

How to use X-Robots-Tag

  • Be specific. It’s best to use the X-Robots-Tag for specific pages or groups of pages on your website. Using X-Robots-Tag for the entire website will likely lead to unintended results.
  • Be consistent. Ensure you use the same value of X-Robots-Tag for all pages with the same purpose.
  • Test your changes. If you make changes to X-Robots-Tag, ensure they have the desired effect. You can use tools like Google Search Console to check how search engines index your pages.

Why You Should Use the X-Robots-Tag Header on Your Website

Add the X-Robots-Tag Header to WordPress Using a Snippet

A snippet is a short PHP code that adds new features or modifies the behavior of your website. To insert snippets into WordPress, I highly recommend the WPCode plugin. It’s free and extremely useful. You can find a complete guide on using this plugin here: How to insert custom codes into WordPress.

Insert this snippet using the WPCode plugin on your website. Modify the snippet according to the indexing you desire for your pages. In this case, the header will have a value of index and follow.

					add_action('template_redirect', 'my_x_robots_tag');

function my_x_robots_tag() {
  header('X-Robots-Tag: index, follow');

Why You Should Use the X-Robots-Tag Header on Your Website


If you’ve correctly added the header, you’ll find out through any SEO audit. For these cases, I use a fantastic Chrome browser extension called AIOSEO. The header itself is a great addition to the existing robots.txt file.

Proč byste měli používat X-Robots-Tag hlavičku na svém webu

The website is created with care for the included information. I strive to provide high-quality and useful content that helps or inspires others. If you are satisfied with my work and would like to support me, you can do so through simple options.

Byl pro Vás tento článek užitečný?

Klikni na počet hvězd pro hlasování.

Průměrné hodnocení. 0 / 5. Počet hlasování: 0

Zatím nehodnoceno! Buďte první

Jak užitečný vidíte tento článek.

Sledujte mě na sociálních médiích.

Je mi líto, že pro Vás nebyl článek užitečný.

Jak mohu vylepšit článek?

Řekněte mi, jak jej mohu zlepšit.


Subscribe to the Newsletter

Stay informed! Join our newsletter subscription and be the first to receive the latest information directly to your email inbox. Follow updates, exclusive events, and inspiring content, all delivered straight to your email.

Upozornit na
0 Komentáře/ů
Vložené zpětné vazby.
Zobrazit všechny komentáře.

Pokud mi chcete napsat rychlou zprávu, využije, prosím, níže uvedený
kontaktní formulář. Děkuji.

Další Kontaktní údaje