SEAM · ICASSP 2027

Speech-conditioned affective music generation

Given a spoken utterance and an emotion-free instrument prompt, generate background music whose valence–arousal follows the speech — without putting mood words in the text.

IEMOCAP Session 5 · piano, violin string, and drum · frozen SA3 medium

Paper companion

Task 1 · emotion alignment

Play the speech, then the three music clips. All systems share the same SA3 backbone. Ours uses speech tokens and an emotion-free prompt; the others stay text-only.

SpeechIEMOCAP reference utterance — not generated.
SA3 lexicalEmotion word appended to the instrument prompt.
SA3 VAOfficial IEMOCAP valence and arousal written into the text.
OursSpeech tokens plus an emotion-free instrument prompt.

Angry · Set 1

Ses05F_impro02_M014

Reference

Speech

Generated music

SA3 lexical piano, violin string, and drum, angry
SA3 VA piano, violin string, and drum, valence -0.75, arousal +0.25.
Ours piano, violin string, and drum + speech tokens

Angry · Set 2

Ses05F_script01_1_F038

Reference

Speech

Generated music

SA3 lexical piano, violin string, and drum, angry
SA3 VA piano, violin string, and drum, valence -0.75, arousal +0.50.
Ours piano, violin string, and drum + speech tokens

Task 2

Intensity ladder

Same spoken utterance, five α values. Ours steers the speech residual on an emotion-free prompt; SA3 writes the mix into the text and never sees the speech.

SpeechIEMOCAP reference utterance — not generated.
OursSpeech tokens plus an emotion-free instrument prompt. α scales the residual.
SA3 lexical blendText-only: p with (1−α) of neutral and α of {emotion}.

Angry

Ses05F_impro08_M003

Reference

Speech

Ours

α = 0.00 piano, violin string, and drum + speech tokens
α = 0.25 piano, violin string, and drum + speech tokens
α = 0.50 piano, violin string, and drum + speech tokens
α = 0.75 piano, violin string, and drum + speech tokens
α = 1.00 piano, violin string, and drum + speech tokens

SA3 lexical blend

α = 0.00 piano, violin string, and drum with 1.00 of neutral and 0.00 of angry.
α = 0.25 piano, violin string, and drum with 0.75 of neutral and 0.25 of angry.
α = 0.50 piano, violin string, and drum with 0.50 of neutral and 0.50 of angry.
α = 0.75 piano, violin string, and drum with 0.25 of neutral and 0.75 of angry.
α = 1.00 piano, violin string, and drum with 0.00 of neutral and 1.00 of angry.