Voicebox by Meta

Voicebox by Meta is a generative AI model for speech that uses a new approach called Flow Matching. It can train on diverse, unstructured data without requiring carefully labeled inputs. It can produce high-quality audio clips in a variety of styles and synthesize speech across six languages. Other features include noise removal, content editing, style conversion, and diverse sample generation. Unlike existing models, it can modify any part of a given sample, not just the end, making it versatil

Flow Matching is a new approach developed by Meta which is seen as their latest advancement on non-autoregressive generative models. This technique enables highly non-deterministic mapping between text and speech. This non-deterministic mapping is beneficial as it allows Voicebox to learn from varied speech data without the necessity for those variations to be carefully labeled. This indicates that Voicebox can be trained on significantly more diverse and larger scales of data.

Voicebox can synthesize speech in six languages: English, French, Spanish, German, Polish, and Portuguese.

Voicebox outperforms the current state-of-the-art English model, VALL-E, in terms of both intelligibility and audio similarity. It achieves a 5.9 percent word error rate versus VALL-E's 1.9 percent, and an audio similarity score of 0.580 compared to VALL-E's 0.681. Furthermore, for cross-lingual style transfer, Voicebox reduces the average word error rate from 10.9 percent to 5.2 percent, and improves audio similarity from 0.335 to 0.481.

Traditional speech synthesizers require specific training for each task using carefully prepared data and they can only modify the end part of an audio clip. Conversely, Voicebox can learn from raw audio and an accompanying transcription. It is capable of modifying any part of a given sample and doesn't require carefully labeled inputs. This difference allows for greater versatility across a wider range of tasks and data sources.

Along with producing outputs from scratch, Voicebox can modify existing samples. The model can learn to predict a speech segment by analyzing the surrounding speech and the transcript of the segment. Given this learning, it can apply it to generate or modify audio in any part of a recording without having to recreate the entire input.

No, as of the provided information, Voicebox is not available to the public due to potential risks of misuse.

Potential applications of Voicebox are wide-ranging. Its in-context text-to-speech synthesis could potentially bring speech to people who are unable to speak or allow people to customize the voices of non-player characters and virtual assistants. Its ability to perform cross-lingual style transfer could help people communicate naturally in different languages. Voicebox's abilities in speech denoising and editing could ease the process of cleaning up and editing audio. In terms of diverse speech

Voicebox was trained using more than 50,000 hours of recorded speech and transcripts from public domain audiobooks in six languages including English, French, Spanish, German, Polish, and Portuguese.

Yes, Voicebox's in-context learning enables it to generate speech to seamlessly edit segments within audio recordings. It can resynthesize the portion of speech corrupted by short-duration noise or replace misspoken words without having to re-record the entire speech.