Earlier this week, The Wall Street Journal reported on the AI industry’s growing concerns over the scarcity of valuable training data.
Following up, The New York Times provided insights into how companies, including OpenAI and Google, navigate this issue, sometimes by venturing into legally ambiguous territories.
How OpenAI Addressed Data Scarcity
OpenAI, in its quest for valuable data to train GPT-4, reportedly transcribed over a million hours of YouTube videos for training purposes. This move, as reported by The New York Times, was a bold step to conquer the data scarcity challenge, despite the company’s awareness of the potential legal ambiguities regarding copyright infringement.
Greg Brockman, OpenAI’s president, played a direct role in selecting videos for this project. The company defended its approach by asserting a belief in the fair use of such data, though it recognized the potential for legal contention.
Lindsay Held, a spokesperson for OpenAI, conveyed to The Verge that the organization is dedicated to curating unique datasets to enhance its models’ understanding of the world. According to Held, OpenAI employs a variety of data sources, ranging from publicly available information to exclusive partnerships, and is exploring the creation of synthetic data to supplement its needs.
The New York Times article detailed how OpenAI, facing a shortage of useful data by 2021, considered expanding its data sources to include YouTube videos, podcasts, and audiobooks, having previously utilized data from GitHub, chess databases, and Quizlet.
Google’s Approach to AI Training Data
Google, mentioned in the same context, has also been implicated in using YouTube content for training its AI models. Matt Bryant, a spokesperson for Google, acknowledged the use of YouTube content under agreements with creators but emphasized the company’s opposition to unauthorized scraping, as indicated by both YouTube’s robots.txt files and its Terms of Service. Neal Mohan, YouTube’s CEO, echoed this stance, stressing the company’s measures against unauthorized data use.
Furthermore, Google’s legal and privacy teams reportedly revised policy language to extend the company’s data usage capabilities, with a strategic policy release date intended to minimize public scrutiny.
How Is Meta Competing in the AI Training Data Race?
Meta, competing in the AI space, faced its own challenges with training data scarcity. According to The Times, Meta’s AI team discussed using copyrighted works without permission as a strategy to compete with rivals like OpenAI.
Despite exhaustive searches through English-language texts available online, Meta contemplated measures such as licensing content or acquiring a publisher to secure more data, constrained by privacy considerations stemming from past controversies like the Cambridge Analytica scandal.
With AI companies potentially outpacing the creation of new content by 2028, The Wall Street Journal points to potential solutions like the creation of “synthetic” data or “curriculum learning”, a method of systematically feeding models high-quality data to encourage them to form more sophisticated connections between concepts with minimal information. However, neither approach has been definitively proven effective yet.
Related News:
Featured Image courtesy of rawpixel.com on Freepik