Google hopes to standardize robots.txt by going open source

    Date02 Jul 2019
    1659
    Posted ByBrittany Day

    Google is releasing robots.txt to the open-source community in the hopes that the system will, one day, becoming a stable internet standard.

    On Monday, the tech giant outlined the move to make the Robots Exclusion Protocol (REP) -- better known as robots.txt -- open-source, alongside its matching C++ library.

    REP is a way for webmasters to establish the behavior of code attempting to visit a website. The original creator, Martijn Koster, found that his website was being overwhelmed by crawlers and so in a bid to reduce server strain, developed the initial standard in 1994.

    You are not authorised to post comments.

    Comments powered by CComment

    LinuxSecurity Poll

    What do you think of the articles on LinuxSecurity?

    No answer selected. Please try again.
    Please select either existing option or enter your own, however not both.
    Please select minimum 0 answer(s) and maximum 3 answer(s).
    /main-polls/24-what-do-you-think-of-the-quality-of-the-articles-on-linuxsecurity?task=poll.vote&format=json
    24
    radio
    [{"id":"87","title":"Excellent, don't change a thing!","votes":"15","type":"x","order":"1","pct":53.57,"resources":[]},{"id":"88","title":"Should be more technical","votes":"4","type":"x","order":"2","pct":14.29,"resources":[]},{"id":"89","title":"Should include more HOWTOs","votes":"9","type":"x","order":"3","pct":32.14,"resources":[]}]["#ff5b00","#4ac0f2","#b80028","#eef66c","#60bb22","#b96a9a","#62c2cc"]["rgba(255,91,0,0.7)","rgba(74,192,242,0.7)","rgba(184,0,40,0.7)","rgba(238,246,108,0.7)","rgba(96,187,34,0.7)","rgba(185,106,154,0.7)","rgba(98,194,204,0.7)"]350
    bottom200

    Advisories

    We use cookies to provide and improve our services. By using our site, you consent to our Cookie Policy.