Meta, the parent company of Facebook and Instagram, unveiled its latest AI innovation, Movie Gen, an artificial intelligence model that generates short, realistic video and audio clips based on simple text prompts.
The company announced the breakthrough technology through a blog post and social media updates this week, highlighting its potential for content creators and users with no technical training. Meta claims that Movie Gen can rival similar tools developed by industry leaders like OpenAI and ElevenLabs.
Movie Gen, which is expected to be available via Instagram next year, was first introduced by Meta’s CEO, Mark Zuckerberg, in an Instagram post. Zuckerberg’s post, while playful, showcased the tool’s capabilities and mentioned the possibility of it becoming available to the public by 2024. The technology was further elaborated on in a post from Meta’s AI research team, where it outlined Movie Gen’s ability to create personalized videos and audio from user-defined text inputs.
Key Features of Movie Gen
The AI-powered tool allows users to generate video clips up to 16 seconds in length at a rate of 16 frames per second. These videos can be based entirely on text descriptions or a combination of text and a single image, allowing users to personalize the output. Additionally, the audio component of Movie Gen can generate background music, sound effects, and ambient noises that align with the content of the videos. Audio clips can last up to 45 seconds, expanding the creative possibilities for users.
Several demonstrations of the tool’s capabilities were showcased on Meta’s platforms, featuring videos such as:
- an ATV riding through sand dunes,
- a skateboarder navigating a skate park, and
- penguins swimming through an AI-generated landscape.
Meta also provided blind test data showing that Movie Gen performs favorably when compared to other tools from startups like Runway, OpenAI, ElevenLabs, and Kling.
Meta emphasized that Movie Gen’s design allows for both video creation and editing, offering a range of possibilities for content creators. In addition to generating video clips, the tool can modify existing videos, enhancing or transforming elements based on user input. One such example showed how Movie Gen altered a man’s skateboarding video, changing the environment to fit the requested conditions.
Despite the impressive capabilities, Meta’s Chief Product Officer, Chris Cox, clarified in a Threads post that the tool is not yet ready for public release. He acknowledged that while Movie Gen can generate highly personalized videos with realistic audio, it remains expensive to operate, and the generation time is too long for mass production. Cox emphasized that Meta is working closely with filmmakers and other creators to refine the tool, improving its editing functions and its ability to generate specific characters and images based on text prompts.
Meta’s announcement follows the growing trend of AI-driven content creation tools. Earlier this year, OpenAI introduced Sora, a text-to-video solution that aimed to generate feature film-like videos based on text prompts.
While the entertainment industry has shown significant interest in such tools, concerns remain about the ethical implications of AI-generated content. Some worry about the potential misuse of AI models that may have been trained on copyrighted materials without permission. Others have highlighted concerns about the increasing presence of AI-generated fakes, or deepfakes, being used in political campaigns around the world.
Future Plans for Movie Gen
Meta has stated that Movie Gen is unlikely to be released for open developer use, unlike the company’s Llama series of large language models. Company spokespeople explained that each model’s risks are assessed individually, and they declined to comment specifically on the plans for Movie Gen. Instead, Meta plans to collaborate directly with the entertainment industry and other content creators before integrating Movie Gen into its own platforms by next year.
In developing Movie Gen, Meta used a combination of licensed and publicly available datasets, according to a research paper released by the company. As generative AI continues to gain traction in the entertainment industry, other companies have explored partnerships with AI developers.
In September, Lions Gate Entertainment, known for film franchises like The Hunger Games and Twilight, partnered with AI startup Runway, providing access to its film and TV library to train an AI model in exchange for using it in future productions. OpenAI has also been in discussions with Hollywood executives and agents regarding its Sora AI tool, though no official partnerships have been announced.
Featured Image courtesy of Meta
Follow us for more tech news updates.