As Google Expands its AI Initiatives, Publishers Face New Hurdles

By Maggie On Oct 19, 2023

Imagine a world where you never have to skim through another news article, all thanks to artificial intelligence that can absorb the vast sea of information on the web and deliver concise summaries whenever you desire.

This notion may haunt media tycoons as companies like Google delve into the realm of generative AI, which crafts fresh content by drawing from existing data.

Since May, Google has been gradually introducing a novel form of search functionality fueled by generative AI. This initiative emerged amid concerns about Google’s continued dominance in offering information to users following the emergence of OpenAI’s question-answering chatbot, ChatGPT.

This new feature, known as Search Generative Experience (SGE), harnesses AI to generate summaries in response to specific search queries, provided Google’s system deems this format beneficial. These summaries appear prominently on the Google search homepage, accompanied by links for deeper exploration, as outlined by Google’s description of SGE.

Publishers seeking to shield their content from being utilized by Google’s AI to create these summaries must use the same tool that blocks their appearance in Google’s standard search results, effectively rendering them nearly invisible on the web.

For instance, a search for “Who is Jon Fosse,” the recent Nobel Prize in Literature laureate, results in a three-paragraph summary about the author and his work. Dropdown options offer links to Fosse-related content on Wikipedia, NPR, The New York Times, and other websites, while additional links appear alongside the summary.

Google asserts that these AI-generated overviews amalgamate information from multiple web sources and serve as starting points for further exploration. It characterizes SGE as an opt-in experiment, inviting user feedback to enhance the product while accommodating input from news publishers and other stakeholders.

For publishers, this new search tool signals another challenge in a decades-long relationship where they’ve both struggled to compete with Google for online advertising while relying on the tech giant for search traffic.

The evolving product, presently available in the United States, India, and Japan, has raised concerns among publishers trying to navigate their role in a world where AI may revolutionize how users find and pay for information, according to four major publishers who shared their insights with Reuters anonymously to avoid complicating ongoing negotiations with Google.

These concerns encompass aspects like web traffic, acknowledgment of publishers as the source of information in SGE summaries, the accuracy of these summaries, and most significantly, the compensation for the content that serves as the training material for Google and other AI companies – a contentious issue surrounding AI development.

In response, a Google spokesperson emphasized their commitment to directing valuable traffic to a broad spectrum of content creators, including news publishers, to sustain a healthy and open web. On the matter of compensation, Google aims to refine its understanding of the generative AI business model through dialogue with publishers and other stakeholders.

In late September, Google introduced a new tool named Google-Extended, allowing publishers to prevent their content from being used for training Google’s AI models. This option is viewed as a positive step by industry observers and may lead to discussions about equitable value exchange.

However, the tool does not enable publishers to block their content from being used in SGE, including both the summaries and the accompanying links, without affecting their visibility in traditional Google search results. This poses a dilemma for publishers, as appearing in Google search results is pivotal for securing advertising revenue. SGE’s design pushes standard search result links further down the page, potentially reducing their click-through rates by as much as 40%, according to an executive from one of the publishers.

More concerning is the possibility that users may abstain from clicking any links if the SGE summary satisfies their information needs. For example, they may find the best time to visit Paris without needing to visit a travel publication’s website.

According to Nikhil Lai, a Senior Analyst at Forrester Research, SGE will indeed decrease publishers’ organic traffic, necessitating a different method to gauge the value of their content. Nevertheless, he believes that publishers’ reputations will remain strong due to their links appearing in SGE.

Google contends that SGE aims to showcase web content and emphasizes that specific traffic impact estimations are speculative, given the evolving nature of the feature.

While publishers and other industries have adapted their websites over the years to rank highly in standard Google search results, they currently lack sufficient information to do the same for SGE summaries.

In the words of an executive at one publishing house, “The new AI section is a black box for us.” Publishers are uncertain about how to ensure their inclusion in SGE and understand the algorithm behind it.

Google asserts that publishers don’t need to take any different actions than they have been to appear in search results. Publishers have traditionally allowed Google to “crawl” their content for indexing and appearing in search results, essentially enabling a bot to scan and index their content.

Publishers’ concerns regarding SGE boil down to a fundamental issue: they believe Google is crawling their content at no cost to create summaries that users may read instead of clicking on their links, and Google hasn’t provided clear instructions on how to block content from being crawled for SGE.

One publisher deemed Google’s new search tool “even more threatening to us and our business than a crawler that is crawling our business illegally.” Google did not comment on this assessment.

When given the option, websites are blocking their content from being used for AI if it doesn’t affect standard search, as indicated by data from AI content detector Originality.ai. Since its release in August, 27.4% of top websites have blocked ChatGPT’s bot, including The New York Times and The Washington Post, in comparison to the 6% blocking Google-Extended since its release in late September.