| An update on residential proxies and the scraper situation(lwn.net) | |
| 231 points by chmaynard 17 hours ago | 237 comments | |
tl;dr: AI scraper traffic increasingly comes through "residential proxy" networks—millions of compromised or opt-in devices (often via shady free VPNs, SDKs, or malware-infected streaming boxes) that make each request look like a unique human visitor, defeating traditional IP-based blocking. LWN has resisted deploying tools like Anubis to avoid burdening real users, instead using undisclosed optimizations and defenses, while noting that proof-of-work is a weak deterrent when attackers command millions of hijacked machines. Recent law-enforcement takedowns of networks like IPIDEA and NetNut have offered only temporary relief in an ongoing arms race threatening the open web. | |
HN Discussion:
| |