A New Era of Web Scraping

Artificial Intelligence platforms have long been associated with scraping data from external websites to fuel their models. However, the tables have turned: recent research from Decodo reveals that prominent AI sites, including ChatGPT and Perplexity, have become prime targets for web scraping themselves.

This development signifies a fundamental change in corporate data strategies. According to Vaidotas Juknys, CEO of Decodo, the focus of intelligence gathering has shifted dramatically:

«Two years ago, businesses scraped the web to see what people were searching for. Now they scrape AI platforms to see what people are being told. The use case for data collection is changing.»

Market Dynamics and New Leaders

The latest data indicates that the landscape of the most-scraped websites is undergoing a major transformation. Seven of the top 10 sites are newcomers, while traditional retail giants—such as Walmart, eBay, and Airbnb—have fallen out of the rankings entirely. Only TikTok, Google, and Amazon remain from previous top-tier lists.

The current top-scraped platforms include:

  • TikTok: Captured the most attention, accounting for 18% of observed scraping activity.
  • ChatGPT: Secured the third position, responsible for 14% of activity.
  • Perplexity: Landed in fifth place, representing 10% of the data collection.

Why Businesses Are Targeting AI

Organizations are increasingly monitoring AI platforms to gain competitive advantages. By scraping these tools, companies can analyze how their products are ranked against competitors, track how the AI describes their brand, measure the frequency of brand mentions, and record citations. This shift highlights a broader move in consumer behavior, where AI and social media are becoming the primary drivers of purchase patterns.

The Future of Automated Data Collection

Looking toward the future, Decodo anticipates that other AI giants, such as Google’s Gemini and Microsoft’s Copilot, will likely break into the top 10 rankings by next year. Perhaps most significantly, the industry expects a move toward automation in these data collection efforts.

Juknys noted that the next phase of this trend may involve AI agents taking over the process: «Next year the agents might be doing the deciding instead of just reading the answers.»