AI search engines that don’t pay up can’t index Reddit content

AI search engines that don’t pay up can’t index Reddit content

When Reddit said last month that it would block unauthorized data scraping from its site, everyone’s (rightful) first reaction was “AI, AI, AI.” However, now that the change has taken effect, chatbot makers may not be the only ones being locked out. The widely used forum also appears to be blocking major search engines other than Brave and Google, the latter of which reportedly inked a deal earlier this year with Reddit <a data-i13n="cpos:1;pos:1" href="https://www.engadget.com/reddit-is-licensing-its-content-to-google-to-help-train-its-ai-models-200013007.html">worth $60 million annually.[/url] However, a Reddit spokesperson told Engadget that the empty search results are about Google’s rivals not agreeing to the company’s requirements for AI training. It says it’s it’s in discussions with several of them.
404 Media <a data-i13n="elm:context_link;elmt:doNotAffiliate;cpos:2;pos:1" class="no-affiliate-link" href="https://www.404media.co/google-is-the-only-search-engine-that-works-on-reddit-now-thanks-to-ai-deal/" data-original-link="https://www.404media.co/google-is-the-only-search-engine-that-works-on-reddit-now-thanks-to-ai-deal/">reported on Wednesday[/url] (and Engadget confirmed in our queries) that searching for Reddit results from the past week on rival engine Bing (using “site:reddit.com”) returns empty results. The publication reported that DuckDuckGo produced seven links without any descriptions, only providing the note, “We would like to show you a description here but the site won’t allow us.” The engine now appears to have removed even those, as our test only produced an empty page, reading, “no results found.”
When Reddit <a data-i13n="elm:context_link;elmt:doNotAffiliate;cpos:3;pos:1" class="no-affiliate-link" href="https://www.reuters.com/technology/reddit-update-web-standard-block-automated-website-scraping-2024-06-25/" data-original-link="https://www.reuters.com/technology/reddit-update-web-standard-block-automated-website-scraping-2024-06-25/">said last month[/url] that it would update its Robots Exclusion Protocol (robots.txt) to block automated data scraping, it’s now apparent that it wasn’t only meant to thwart AI companies like Perplexity and its controversial “answer engine.” Currently, Google appears to be the only search engine allowed to crawl Reddit and produce results from “the front page of the internet.”
A Reddit spokesperson told Engadget on Wednesday it isn’t accurate to say the missing search results are a result of its Google deal. “We block all crawlers that are unwilling to commit to not using crawl data for AI training, which is in line with enforcing our Public Content Policy and updated robots.txt file,” the company said. “Anyone accessing Reddit content must abide by our policies, including those in place to protect redditors. We are selective about who we work with and trust with large-scale access to Reddit content.”
Meanwhile, a source familiar with Reddit’s thinking told Engadget on Wednesday that Bing’s omission is due to Microsoft refusing to agree to Reddit’s terms regarding AI crawling. Instead, the Bing maker allegedly claimed its standard web controls were sufficient. The source claims Microsoft’s stance conflicts with Reddit’s data privacy policy, leading to the impasse and empty search results.
The ubiquitous robots.txt is the web standard that communicates which parts of a site can be crawled. Although many crawlers are known to ignore its instructions, Google’s standard procedure is to respect it. So, on the technical side, the companies in cahoots on the lucrative deal appear to have deployed some manual override.
The saga could be seen as a trickle-down effect of <a data-i13n="cpos:4;pos:1" href="https://www.engadget.com/ai-companies-are-reportedly-still-scraping-websites-despite-protocols-meant-to-block-them-132308524.html">AI chatbots scraping the live web for results[/url]. With courts slow to determine <a data-i13n="cpos:5;pos:1" href="https://www.engadget.com/openai-admits-its-impossible-to-train-generative-ai-without-copyrighted-materials-103311496.html">how much of the open web is fair use to train chatbots on[/url], companies like Reddit, whose bottom lines now depend on safeguarding their data from those who don’t pay, are building walls at the expense of the open web. (Although, given the integral role Microsoft has played in this AI era, <a data-i13n="cpos:6;pos:1" href="https://www.engadget.com/microsofts-openai-partnership-was-born-from-google-envy-202143989.html">cozying up with OpenAI[/url] early on, it seems ironic that Bing finds itself on the losing end of at least one aspect of the fallout.)
Colin Hayhurst, CEO of lesser-known “no-tracking” search engine Mojeek, told 404 Media that Reddit is “killing everything for search but Google.” In addition, the executive said his attempts to contact Reddit were ignored. “It’s never happened to us before,” he said. “Because this happens to us, we get blocked, usually because of ignorance or stupidity or whatever, and when we contact the site you certainly can get that resolved, but we’ve never had no reply from anybody before.”
Reddit has made no secret of its desire to block AI companies from scraping its treasure trove of data in this burgeoning age of AI. Last year, CEO Steve Huffman risked alienating large portions of its user base by <a data-i13n="cpos:7;pos:1" href="https://www.engadget.com/reddit-app-developer-says-the-sites-new-api-rules-will-cost-him-20-million-a-year-203911487.html">blocking third-party API requests[/url], leading to the <a data-i13n="cpos:8;pos:1" href="https://www.engadget.com/apollo-and-other-popular-third-party-reddit-apps-have-shut-down-123149140.html">demise[/url] of beloved apps like <a data-i13n="cpos:9;pos:1" href="https://www.engadget.com/reddit-app-developer-says-the-sites-new-api-rules-will-cost-him-20-million-a-year-203911487.html">Christian Selig’s Apollo[/url]. Despite <a data-i13n="cpos:10;pos:1" href="https://www.engadget.com/reddit-sees-more-than-6000-communities-go-dark-in-protest-over-api-changes-095311637.html">widespread protests among moderators and forum-goers[/url], the company only temporarily lost negligible numbers of users.
The gamble appeared to pay off, and Reddit recovered. It <a data-i13n="cpos:11;pos:1" href="https://www.engadget.com/reddit-is-now-a-publicly-traded-company-144455403.html">went public in March[/url].
Update, July 24, 2024, 5:00 PM ET: This story has been updated to add statements from Reddit and additional context from sources familiar with the company’s thinking.This article originally appeared on Engadget at https://www.engadget.com/search-engines-that-dont-pay-up-cant-index-reddit-content-172949170.html?src=rss

Source: AI search engines that don’t pay up can’t index Reddit content