Last updated December 5th, 2023 22:52
Do you also focus on SEO on your website, fine-tuning any details that might push you up even one notch in the search results? Then you’ve probably already placed a robots.txt file on your site, the basics for better indexing are covered. However, do you know what the X-Robots-Tag header is for and use it on your website? If you’ve just answered ‘no’ to both questions, then this article is precisely for you. We’ll explore why you should use the X-Robots-Tag header on your website. Additionally, if you use WordPress, I’ll show you a simple snippet to add this header to your site.
Why You Should Use the X-Robots-Tag Header on Your Website
X-Robots-Tag is a special HTTP header that allows webmasters to control how their web pages should be indexed. This header serves a similar function to the robots.txt file mentioned earlier in the introduction, but it has one crucial difference. While the robots.txt file is a file you place on the server (via FTP), X-Robots-Tag is a header that is part of every HTTP request to a web page.
The X-Robots-Tag header has the following values that communicate with search engines:
- index: Index the page.
- noindex: Do not index the page.
- follow: Index the page and follow links from it.
- nofollow: Do not index the page and do not follow links from it.
You can use the X-Robots-Tag header to control the indexing of different types of content on your website. For example, you can use X-Robots-Tag to prevent indexing of certain pages with personal data or to block indexing of advertisement pages.
Examples of X-Robots-Tag Header
If you want to prevent search engines from indexing your advertisement pages, you can add the following header to each request for these pages:
X-Robots-Tag: noindex, nofollow
Do you need search engines to index your pages but not follow the links placed on them? Conversely, add the following header to each request for these pages:
X-Robots-Tag: index, nofollow
How to use X-Robots-Tag
- Be specific. It’s best to use the X-Robots-Tag for specific pages or groups of pages on your website. Using X-Robots-Tag for the entire website will likely lead to unintended results.
- Be consistent. Ensure you use the same value of X-Robots-Tag for all pages with the same purpose.
- Test your changes. If you make changes to X-Robots-Tag, ensure they have the desired effect. You can use tools like Google Search Console to check how search engines index your pages.
Why You Should Use the X-Robots-Tag Header on Your Website
Add the X-Robots-Tag Header to WordPress Using a Snippet
A snippet is a short PHP code that adds new features or modifies the behavior of your website. To insert snippets into WordPress, I highly recommend the WPCode plugin. It’s free and extremely useful. You can find a complete guide on using this plugin here: How to insert custom codes into WordPress.
Insert this snippet using the WPCode plugin on your website. Modify the snippet according to the indexing you desire for your pages. In this case, the header will have a value of index and follow.
add_action('template_redirect', 'my_x_robots_tag');
function my_x_robots_tag() {
header('X-Robots-Tag: index, follow');
}
Why You Should Use the X-Robots-Tag Header on Your Website
Conclusion
If you’ve correctly added the header, you’ll find out through any SEO audit. For these cases, I use a fantastic Chrome browser extension called AIOSEO. The header itself is a great addition to the existing robots.txt file.
The website is created with care for the included information. I strive to provide high-quality and useful content that helps or inspires others. If you are satisfied with my work and would like to support me, you can do so through simple options.
Subscribe to the Newsletter
Stay informed! Join our newsletter subscription and be the first to receive the latest information directly to your email inbox. Follow updates, exclusive events, and inspiring content, all delivered straight to your email.
