241 Websites, Including 23 News Outlets, Deny Access to Internet Archive's Web Crawler

# 23 News Organizations Block Internet Archive's Web Crawler Access to Articles Twenty-three news organizations, including USA Today and the New York Times, are among 241 sites denying the Internet Archive’s web crawler access to their articles. This action limits the archiving of content from these outlets. Fortune reported the details via Wired.

Major media outlets are blocking the Internet Archive’s Wayback Machine from saving web pages. The blocks prevent the digital archive from capturing and preserving online content from these sources. These measures affect the availability of historical web material.

companies can use the Wayback Machine as a workaround for training language models on content.

This approach allows access to archived web data for AI development purposes. The practice has prompted responses from content providers. The digital archive has controls to limit abuse of AI automation and prevent large-scale data extraction.

Mark Graham stated these measures are in place to manage usage. Fortune reported the statement.

year, Reddit barred the Wayback Machine from data scraping.

This decision restricted the archive's ability to collect content from the platform. The move followed concerns over data usage. Mark Graham is in talks to regain access to the material. These negotiations aim to restore archiving capabilities for affected sites.

Fortune noted the ongoing discussions.

than 100 media workers signed a letter supporting Wayback.

The letter endorses the Internet Archive's role in preserving digital content. This backing highlights divided views within the media sector. The blocks by 241 sites, including major outlets, underscore tensions over web archiving and AI data access.