minus-squareShitpostCentral@lemmy.worldtoTechnology@lemmy.world•AI companies are violating a basic social contract of the web and and ignoring robots.txtlinkfedilinkEnglisharrow-up16·6 months agoYou’re second point is a good one, but you absolutely can log the IP which requested robots.txt. That’s just a standard part of any http server ever, no JavaScript needed. linkfedilink
ShitpostCentral@lemmy.world to Piracy: ꜱᴀɪʟ ᴛʜᴇ ʜɪɢʜ ꜱᴇᴀꜱ@lemmy.dbzer0.comEnglish · 7 months agoHow does Usenet content not immediately get DMCA'd into oblivion?plus-squaremessage-squaremessage-square5fedilinkarrow-up10arrow-down10
arrow-up10arrow-down1message-squareHow does Usenet content not immediately get DMCA'd into oblivion?plus-squareShitpostCentral@lemmy.world to Piracy: ꜱᴀɪʟ ᴛʜᴇ ʜɪɢʜ ꜱᴇᴀꜱ@lemmy.dbzer0.comEnglish · 7 months agomessage-square5fedilink
You’re second point is a good one, but you absolutely can log the IP which requested robots.txt. That’s just a standard part of any http server ever, no JavaScript needed.