Google hopes to standardize robots.txt by going open source

    Date 02 Jul 2019
    2417
    Posted By Brittany Day
    Screenshot 2019 07 02 At 09 32 55

    Google is releasing robots.txt to the open-source community in the hopes that the system will, one day, becoming a stable internet standard.

    On Monday, the tech giant outlined the move to make the Robots Exclusion Protocol (REP) -- better known as robots.txt -- open-source, alongside its matching C++ library.

    REP is a way for webmasters to establish the behavior of code attempting to visit a website. The original creator, Martijn Koster, found that his website was being overwhelmed by crawlers and so in a bid to reduce server strain, developed the initial standard in 1994.

    The link for this article located at ZDNet is no longer available.

    LinuxSecurity Poll

    Have you ever used tcpdump for network troubleshooting or debugging?

    No answer selected. Please try again.
    Please select either existing option or enter your own, however not both.
    Please select minimum 0 answer(s) and maximum 2 answer(s).
    /main-polls/36-have-you-ever-used-tcpdump-for-network-troubleshooting-or-debugging?task=poll.vote&format=json
    36
    radio
    [{"id":"125","title":"Yes","votes":"36","type":"x","order":"1","pct":80,"resources":[]},{"id":"126","title":"No ","votes":"9","type":"x","order":"2","pct":20,"resources":[]}] ["#ff5b00","#4ac0f2","#b80028","#eef66c","#60bb22","#b96a9a","#62c2cc"] ["rgba(255,91,0,0.7)","rgba(74,192,242,0.7)","rgba(184,0,40,0.7)","rgba(238,246,108,0.7)","rgba(96,187,34,0.7)","rgba(185,106,154,0.7)","rgba(98,194,204,0.7)"] 350


    VIEW MORE POLLS

    bottom 200

    Please enable / Bitte aktiviere JavaScript!
    Veuillez activer / Por favor activa el Javascript![ ? ]

    We use cookies to provide and improve our services. By using our site, you consent to our Cookie Policy.