1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

What is a robots.txt file?

Discussion in 'Webmasters Talk' started by business orientation inc, May 26, 2016.

  1. hi friends,

    What is a robots.txt file?
     
  2. GainERP

    GainERP New Member

    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
     
  3. dam ponting

    dam ponting Member

    In a nutshell. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. ... The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
     
  4. dam ponting

    dam ponting Member

    In a nutshell. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. ... The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robotthat it should not visit any pages on the site.
     
  5. johnsimth

    johnsimth New Member

    Robots.txt is a text file that webmasters create to instruct web robots how to crawl web pages on their website. It is a used by websites to communicate with web crawlers and other web robots.
     
  6. Sinelogix_Tech

    Sinelogix_Tech New Member

    The robots.txt file is primarily used to specify which parts of your website should be crawled by spiders or web crawlers. It can specify different rules for different spiders.
     
  7. Raj Sharma

    Raj Sharma New Member

    Robots.txt is a text file located in the site’s root directory that specifies for search engines’ crawlers and spiders what website pages and files you want or don’t want them to visit.
     
  8. Sumit_Singh

    Sumit_Singh New Member

    The robots.txt file is the simple text file that is used to make a restriction to crawlers to crawl a particular page or folder or directory by place it in the root directory with a small piece of code.
     

Share This Page