Audiobox

Audiobox is a new foundational research model for audio generation developed by Meta's FAIR (Fundamental AI Research) team. It represents a significant leap forward in creating high-quality, controllable audio from simple inputs. Using a combination of voice samples and natural language text prompts, Audiobox empowers anyone to generate custom voices, sound effects, and complete audio narratives, opening up a wide range of creative possibilities.

The Audiobox family consists of several specialized models built upon a shared self-supervised model called Audiobox SSL. This includes Audiobox for unified speech and sound generation, Audiobox Speech for specialized voice generation, and Audiobox Sound for dedicated sound effect creation. The platform is presented as an experimental research demo, designed to showcase its capabilities and encourage responsible exploration in the field of generative audio.

How to use Audiobox

The Audiobox demo provides an intuitive, interactive interface for users to experiment with its various features. The general workflow involves providing a combination of text and/or audio inputs to guide the AI model.

1. Voice Generation: To create speech, you can either record your own voice as a style reference or use a preset sample. Then, you input the text you want the model to speak. The AI generates the speech in the vocal style of the reference audio. You can also describe a voice style (e.g., "a deep, booming voice") to create entirely new vocal personas.
2. Sound Effect Generation: Simply type a description of the sound you want to create (e.g., "waves crashing on a sandy beach" or "a futuristic car speeding by"). The model will generate a corresponding sound effect.
3. Audio Editing: For editing, you can upload an audio file. To remove unwanted noise, use the 'Magic Eraser' feature. To replace a segment of audio, use 'Sound Infilling' by selecting the portion to replace and describing the new sound you want to insert.
4. Audio Story Creation: The 'Audiobox Maker' combines all these capabilities, allowing you to build a multi-layered audio story by generating and arranging different speech clips and sound effects on a timeline.

Core Features of Audiobox

• Unified Audio Generation: A single model capable of generating both complex speech and a wide variety of sound effects.
• Voice Cloning and Styling (Your Voice): Generate speech that mimics the vocal style of any provided audio sample with high fidelity.
• Descriptive Voice Generation (Described Voices): Create novel voice styles from purely textual descriptions, without needing an audio sample.
• Voice Style Transfer (Restyled Voices): Modify the style of an existing speech recording using a text prompt (e.g., make it sound more excited or whispery).
• Text-to-Sound Effect Generation: Generate realistic and imaginative sound effects from descriptive text prompts.
• Advanced Audio Editing: Includes a 'Magic Eraser' to remove unwanted sounds (like noise from a recording) and 'Sound Infilling' to seamlessly replace or add sounds within an audio clip.
• Responsible AI Guardrails: Implements safety features like audio watermarking to trace generated content and prompt filtering to prevent misuse.

Use Cases for Audiobox

Audiobox's versatile capabilities make it suitable for a wide range of applications:

• Content Creators & Podcasters: Quickly generate custom sound effects, intro music, or even clone their own voice for ad reads or corrections without re-recording.
• Game Developers: Create unique character voices, ambient soundscapes, and dynamic sound effects for immersive gaming experiences.
• Animators & Filmmakers: Produce rich audio tracks, including dialogue, foley, and background sounds, directly from a script or description.
• Educators & Storytellers: Develop engaging audio stories and educational content with distinct character voices and illustrative sounds.
• AI Researchers: Explore the frontiers of generative audio, fairness in AI, and responsible model development.

Advantages of Audiobox

Audiobox stands out due to its comprehensive and responsible approach to audio generation:

• High Controllability: The ability to combine voice and text prompts gives users precise control over the final audio output.
• All-in-One Platform: It integrates generation and editing tools, streamlining the creative workflow from idea to finished audio.
• State-of-the-Art Quality: Built on Meta's cutting-edge research, it produces highly realistic and nuanced audio.
• Commitment to Safety: Proactive measures like watermarking and content filtering demonstrate a commitment to responsible AI development and deployment.
• Accessibility: The intuitive web demo makes advanced AI audio technology accessible to a broad audience, not just technical experts.

Pricing and Plans

Audiobox is currently available as an experimental research demo for educational and non-commercial purposes only. It is not a commercial product. As such, access to the demo is free. Meta is also offering research grants for those interested in conducting safety and responsibility research with the model.