Last updated December 6th, 2023 06:12
The robots.txt file is a text file that specifies which pages or sections of a web server can be indexed by search engines such as Google, Seznam, Bing, etc. Essentially, this file serves as an instruction for search engines to not index sensitive or unwanted content on your website. It is a simple way to tell search engines which pages can or cannot be included in their search results. Let’s take a closer look at the robots.txt file and its examples.
The robots.txt file is uploaded to the root directory of the website. For example, if you have a website “mydomain.com“, the robots.txt file should be uploaded to “http://www.mydomain.com/robots.txt“. Search engines can use this address to find the file and obtain instructions on how to index pages on your website.
The robots.txt file and its examples
The robots.txt file usually contains these commands:
- User-agent: This command specifies which search engine should be affected by the following commands. For example: “User-agent: Googlebot”.
- Disallow: In contrast, this command specifies which pages or sections of the website should not be indexed. For example: “Disallow: /private/”.
- Allow: A command that specifies which pages or sections of the website should be indexed, even if they are prohibited in other commands. For example: “Allow: /public/”.
- The “/” notation represents the root directory of the website, and the “*” represents all robots. So, for example, “Disallow: /” instructs all robots to avoid the entire website.
Examples of writing to the robots.txt file:
Prohibition of indexing the entire website for all search engines:
User-agent: *
Disallow: /
Allowing indexing of the entire website only for Googlebot:
User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /
Prohibition of indexing private pages, but allowing indexing of public pages:
User-agent: *
Disallow: /private/
Allow: /public/
Prohibition of indexing a specific file, such as a file containing personal data:
User-agent: *
Disallow: /private/personal_data.html
Allowing indexing of all pages for Googlebot and prohibiting for other search engines:
User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /
Prohibition of indexing the search form and search result pages:
User-agent: *
Disallow: /search_form.php
Disallow: /search_results.php
The Robots.txt file and Its examples – conclusion.
These examples demonstrate how commands can be written in the robots.txt file for different needs. It is important to note that the robots.txt file is not a guarantee that search engines will respect your requests. However, it is a standardized way to offer them instructions.
If you are still struggling with the rules of the robots.txt file, there are many online generators that can generate the entire file for you. For example, https://www.seoptimer.com/robots-txt-generator.
The website is created with care for the included information. I strive to provide high-quality and useful content that helps or inspires others. If you are satisfied with my work and would like to support me, you can do so through simple options.
Subscribe to the Newsletter
Stay informed! Join our newsletter subscription and be the first to receive the latest information directly to your email inbox. Follow updates, exclusive events, and inspiring content, all delivered straight to your email.
