An update on residential proxies and the scraper situation(lwn.net)
348 points by chmaynard 61 days ago | 377 comments
tl;dr: AI scraper traffic continues to overwhelm sites like LWN via "residential proxy" networks—millions of compromised phones, smart TVs, and devices running hijacked software, often installed via shady "free VPN" apps and SDKs from companies like Bright Data. Recent takedowns of networks like IPIDEA and NetNut brought temporary relief, but the arms race continues; LWN has deployed undisclosed defenses rather than adopting Anubis, arguing proof-of-work is trivial when attackers command millions of victim machines. Without industry accountability or app store scrutiny, the open web risks disappearing behind logins, paywalls, and CAPTCHAs.
HN Discussion:
  • Agrees proof-of-work defenses like Anubis are futile against botnets with millions of machines
  • ~Anti-scraping rhetoric risks harming the open web and empowering gatekeepers like Cloudflare; a better common crawl is the solution
  • App stores and mobile OSes enabling unrestricted network access are the root cause of residential proxy proliferation
  • ~Questions whether scraping volume is really from training data collection versus user-driven AI agents fetching pages
  • Residential proxy companies are essentially legalized botnets that escape punishment by operating openly