Controllable Emotional TTS

Currently investigating multi-modal prompts for expressive and controllable speech synthesis, IMPACT-TTS.

Introduction

Have you ever wished your virtual assistant could speak with real emotions, adapting its tone just like a human? Whether it’s expressive audiobook narration, immersive gaming dialogues, or lifelike dubbing, emotional expressiveness in Text-to-Speech (TTS) systems is a game-changer. However, traditional emotional TTS models struggle with limited emotion control and unnatural expression.

That’s where IMPACT-TTS comes in! This approach integrates multimodal learning and prompt-based control to bring fine-grained, user-driven emotional expressiveness to synthetic voices.

What is IMPACT-TTS?

IMPACT-TTS (Integrated Multimodal Prompt and Control for Emotional TTS) is an advanced TTS model that overcomes the rigid emotional constraints of previous models by:

• Leveraging multimodal inputs (text, vision, and audio) for richer emotional expression.
• Using natural language prompts instead of fixed emotion labels for greater flexibility.
• Allowing real-time emotional fine-tuning through an innovative emotion modulation function.
• Supporting smooth emotion blending, so voices can transition naturally between emotions.
• Providing inference-time expressiveness control, making speech sound more dynamic and lifelike.

How Does It Work?

IMPACT-TTS builds on a modified VITS architecture while introducing a powerful multimodal emotion encoder trained on MEAD-TTS and ESD datasets. Here’s how it enhances emotional synthesis:

Multimodal Emotion Integration

Instead of relying solely on text-based cues, the model fuses vision and audio embeddings using a cross-attention mechanism to refine emotional representation.

Prompt-Based Emotion Control

Rather than being limited to predefined categories (e.g., ‘happy’ or ‘sad’), users can use descriptive prompts like ‘excited but calm’ or ‘nervous with a hint of confidence’ for customized voice generation.

Real-Time Emotion Modulation

Unlike previous models that fix emotional settings at training, IMPACT-TTS lets you adjust emotions dynamically during synthesis, refining pitch, energy, and rhythm on the fly.

Spherical Interpolation for Natural Emotion Blending

The model can smoothly transition between emotions, allowing for nuanced expressiveness rather than abrupt shifts between categories.