Article by Al Landes: “Trying to track down that deleted tweet or verify what a website actually said last month? The Internet Archive’s Wayback Machine has served as your digital time machine for three decades, preserving over a trillion web pages. But this critical tool—the only public archive of its scale—now faces systematic blocking from major media outlets who fear their content might train AI competitors.
Twenty-Three Major News Sites Have Gone Dark
The New York Times, USA Today’s 200+ outlets, and Reddit have all cut off the preservation tool that journalists rely on.
According to Originality AI analysis, 23 major news sites currently block ia_archiverbot, the Wayback Machine’s web crawler. USA Today Co. operates over 200 media outlets, making its blocking decision particularly devastating. The Guardian takes a more subtle approach—allowing crawling but filtering archived content from public access. When you search for historical Guardian articles, you’ll hit digital dead ends.
Publishers Cite AI Training Fears
The New York Times claims archived content violates copyright law, though details remain murky.
Publishers justify blocking with two arguments: preventing AI companies from training on their archived content, and general anti-scraping measures. The Times stated that archived content is being used “to directly compete with us,” but declined to specify whether this represents documented violations or hypothetical concerns. USA Today Co. frames its blocking as routine bot prevention, though the impact falls hardest on preservation efforts…(More)”.
