Loose Lips Sink Ships

Of course, seeing is no longer believing — especially if you're watching anything on video. You've seen the deepfakes of celebrities and politicians and everyone recognizes the potential societal harm that is brewing here. But, this AI AV cloud has at least one silver lining.

Case in point: The movie Fall is a survivalist thriller telling the fictional story of two adventurous women who become stranded atop a 2,000-foot-tall broadcast tower. When the movie was acquired for distribution by Lionsgate, the studio was concerned that the movie's F-bombs would result in an R rating, thus restricting the movie's box office. The studio asked the filmmakers to tone down the language, for example, changing “f***ing” to “freaking.”

The filmmakers could have cut those scenes, completely neutering the movie. They could have reshot those scenes, resulting in a huge price tag. They could have dubbed over the offending words, resulting in the typically clumsy obscene lip motion and family-friendly audio. But they chose a fourth option. They commissioned Flawless, a company with a technological solution called TrueSync, AI-based software originally designed to more convincingly dub film dialogue into other languages by altering the picture to match the replaced audio.

The movie's actors recorded more family-friendly terminology and TruSync was used to alter the original images, specifically the actor's lips and faces, to match the new vocabulary. In other words, the program performs visual dubbing to match audio dubbing. This extra post-production of “neural reshoots” took about 2 weeks, and the result is pretty impressive. You can view a short demo clip here. You can see and hear the original video and audio, see the facial mapping, dialogue removal, and replacement of image and dialogue.

In addition to the F-bomb repair, the clip shows how a movie can be visually dubbed into a foreign language. This kind of image manipulation can allow more faithful foreign-language dubbing; for example, the translated script doesn't have to be changed (i.e. butchered) to match the mouth movements. The foreign-language voice actor can use an accurate translation, and the video can be modified to fit, rather than vice versa.

Interestingly, this kind of technology is already beginning to change the movie-making process. For example, instead of shooting a scene several times, perhaps with slightly different dialogue, a filmmaker can potentially save money by shooting the scene once, then tweaking it in post production. Is this the beginning of flawless dubbing, and the end of subtitles? Maybe. In any case, hopefully the days of the lips saying one thing, while you're hearing something else, are coming to an end.

And finally, the obligatory lip-sync joke: How can you tell a [insert despised profession here] is lying? Easy. Their lips are moving.

trynberg's picture

Ugh, this is inevitably going to end in an ugly place.

johnmic's picture

I really like the way that you have expressed yourself. There is a lot to be admired from this post. You might want to click on
Rocky jacket tiger

webtechno0961's picture

DevOps, a collaborative approach that bridges the gap between development and operations, has emerged as a critical strategy for achieving agility and efficiency. By enrolling in a DevOps Training Institute in Pune, you can equip yourself with the skills to become a catalyst for positive change within your organization.

Gropen40's picture

Normally I wouldn't advocate tampering with an artist's vision, but in this case, I can understand uno online's concern that an R rating would limit its audience. Having to choose between re-shooting scenes, dubbing them awkwardly, or cutting them entirely sounds like a lose-lose situation.