How to disable automatic creation of the robots.txt file

By default, on shared and reseller servers, if a robots.txt file does not exist in the document root (public_html) directory, the server automatically creates a new robots.txt file at midnight.

This article describes how to disable this behavior. You may want to do this, for example, if you do not need to specify how search engines and web crawlers index your site.

Disabling automatic generation of the robots.txt file

To prevent the server from automatically generating a robots.txt file in the document root directory, follow these steps:

  1. Log in to cPanel.
    If you do not know how to log in to your cPanel account, please see this article.
  2. In the FILES section of the cPanel home screen, click the File Manager icon:

    cPanel - File Manager icon

  3. In the left sidebar, click the public_html folder.
  4. On the top menu bar, click + File:

    cPanel - File Manager - Create file

  5. In the New File dialog box, in the New File Name text box, type robots.txt.ignore:

    cPanel - File Manager - New File dialog box

  6. Confirm that the New file will be created in text box is set to /public_html.
  7. Click Create New File. Automatic generation of the robots.txt file is now disabled.

More Information

For more information about the robots.txt file, please vist http://www.robotstxt.org.

Did you find this article helpful? Then you'll love our support. Experience the A2 Hosting difference today and get a pre-secured, pre-optimized website. Check out our web hosting plans today.

We use cookies to personalize the website for you and to analyze the use of our website. You consent to this by clicking on "I consent" or by continuing your use of this website. Further information about cookies can be found in our Privacy Policy.