Meta Gen generates, edits and sets videos to music with AI

Contents

The AI video generator "Meta Gen" could find its way into apps such as Facebook and Instagram as early as next year. The parent company of these social networks has now presented the project, but has not yet made it publicly available. In contrast to other video AIs such as Sora from OpenAI, Meta Gen is not only intended to generate videos, but also to enable the editing of existing clips.

On its project page, Meta not only shows videos generated via a text prompt, which can currently be up to 16 seconds long. It should also be possible to swap elements in the video by simply entering text. For example, one of the clips shows a runner on the beach with blue pom-poms pressed into his hands like a cheerleader. Meta also explicitly mentions that you can upload a photo of your face to a video – - the obvious risk of abuse is enormous.

Meta Gen remains internal for the time being

As Bloomberg reports, Meta Gen is currently only accessible internally to some Meta employees and selected filmmakers. From conversations with Meta, the medium reports, among other things, that the company is currently still looking for ways to enable the safe use of AI. According to Bloomberg author Kurt Wagner, Meta has "grown up a bit".

Meta Gen apparently goes beyond the functional scope of previous video AIs, and not just when it comes to changing the content of generated videos. Among other things, it should also be possible to edit existing videos using machine learning by describing in text which content should be retained. Meta Gen can also add music and sound effects to a video. According to Meta's announcements, the style or mood of the music can be described. In the examples also published on YouTube, however, the company points out that not all of the music in the clips was created by the AI. The structure of the LLMs behind the image and sound generators is described in a research paper (PDF).

Matching sound effects in the right place – sometimes

It could also be a great relief for video producers to be able to generate sound effects based on the content of the images. Placing them at the right time and in the right length is often a challenge for the inexperienced. The example videos are intended to show this for vehicle engine noises, for example. Striking: In a video of fireworks, the sound of the rockets goes off at the same time as the lighting effects begin. The virtual camera would therefore have to be directly in front of the rocket in order to compensate for the different travel times of light and sound.

In other areas, however, Meta Gen is even more realistic than OpenAI's Sora, which caused quite a stir just six months ago. A clip of a girl running through a sandy beach, for example, shows none of the typical AI errors such as feet rolling over and equally realistic-looking tracks in the sand. However, other examples, especially of human faces, show the typical distortions and wax look of image generators in the still image. The infamous rubber fingers are of course also present.

Meta Gen also hallucinates

The familiar hallucinations of AIs also still occur in Meta Gen. For example, a DJ controller has protruding colorful levers – that you wouldn't want to have when DJing – and something that is supposed to look like the tonearm of an analog turntable. And in the clip of a baby hippo that is most frequently seen on social media and at the top of Meta's project page, the flowers floating in the water look strangely intrusive in the room, like the effects of early, purely analog 3D films.

Despite these limitations, the generation of videos still takes several dozen minutes, as Bloomberg reports. However, Meta's aim is to run the functions locally on smartphones – presumably with the support of calculations in the cloud. Meta is likely to require considerably more computing capacity. According to the company, Meta AI, which has primarily been used for a chatbot to date, has around 500 million users worldwide. The system is not yet available in the EU.

(nie)