Google hopes to standardize robots.txt by going open source

    Date02 Jul 2019
    2245
    Posted ByBrittany Day
    Screenshot 2019 07 02 At 09 32 55

    Google is releasing robots.txt to the open-source community in the hopes that the system will, one day, becoming a stable internet standard.

    On Monday, the tech giant outlined the move to make the Robots Exclusion Protocol (REP) -- better known as robots.txt -- open-source, alongside its matching C++ library.

    REP is a way for webmasters to establish the behavior of code attempting to visit a website. The original creator, Martijn Koster, found that his website was being overwhelmed by crawlers and so in a bid to reduce server strain, developed the initial standard in 1994.

    The link for this article located at ZDNet is no longer available.

    LinuxSecurity Poll

    What do you think of the LinuxSecurity Privacy news articles?

    No answer selected. Please try again.
    Please select either existing option or enter your own, however not both.
    Please select minimum 0 answer(s) and maximum 3 answer(s).
    /main-polls/25-what-do-you-think-of-the-linuxsecurity-privacy-news-articles?task=poll.vote&format=json
    25
    radio
    [{"id":"90","title":"Love them!","votes":"90","type":"x","order":"1","pct":78.95,"resources":[]},{"id":"91","title":"I'm indifferent","votes":"18","type":"x","order":"2","pct":15.79,"resources":[]},{"id":"92","title":"Not interested in this topic","votes":"6","type":"x","order":"3","pct":5.26,"resources":[]}]["#ff5b00","#4ac0f2","#b80028","#eef66c","#60bb22","#b96a9a","#62c2cc"]["rgba(255,91,0,0.7)","rgba(74,192,242,0.7)","rgba(184,0,40,0.7)","rgba(238,246,108,0.7)","rgba(96,187,34,0.7)","rgba(185,106,154,0.7)","rgba(98,194,204,0.7)"]350
    bottom200

    We use cookies to provide and improve our services. By using our site, you consent to our Cookie Policy.