Pro@programming.devM to Technology@programming.devEnglish · edit-24 days agoCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comimagemessage-square115linkfedilinkarrow-up1703arrow-down12file-textcross-posted to: technology@lemmy.world
arrow-up1701arrow-down1imageCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comPro@programming.devM to Technology@programming.devEnglish · edit-24 days agomessage-square115linkfedilinkfile-textcross-posted to: technology@lemmy.world
minus-squareS7rauss@discuss.tchncs.delinkfedilinkEnglisharrow-up4·5 days agoI was responding to their question if scraping the site is considered harmful. I would say as long as they are not ignoring robots they shouldn’t be contributing significant amounts of traffic if they’re really only pulling data once a day.
I was responding to their question if scraping the site is considered harmful. I would say as long as they are not ignoring robots they shouldn’t be contributing significant amounts of traffic if they’re really only pulling data once a day.