ComedicSpeech: Text To Speech For Stand-up Comedies in Low-Resource Scenarios

ArXiv: arXiv:2305.12200

Authors

• Yuyue Wang (Gaoling School of Artificial Intelligence, Renmin University of China) wangyuyue123@ruc.edu.cn
• Huan Xiao* (School of Statistics, Renmin University of China) xiaohuan001@ruc.edu.cn
• Yihan Wu (Gaoling School of Artificial Intelligence, Renmin University of China) yihanwu@ruc.edu.cn
• Ruihua Song# (Gaoling School of Artificial Intelligence, Renmin University of China) rsong@ruc.edu.cn

* Equally contributed authors.

# Corresponding authors.

Abstract

Text to Speech (TTS) models can generate natural and high-quality speech, but it is not expressive enough when synthesizing speech with dramatic expressiveness, such as stand-up comedies. Considering comedians have diverse personal speech styles, including personal prosody, rhythm, and fillers, it requires real-world datasets and strong speech style modeling capabilities, which brings challenges. In this paper, we construct a new dataset and develop ComedicSpeech, a TTS system tailored for the stand-up comedy synthesis in low-resource scenarios. First, we extract prosody representation by the prosody encoder and condition it to the TTS model in a flexible way. Second, we enhance the personal rhythm modeling by a conditional duration predictor. Third, we model the personal fillers by introducing comedian-related special tokens. Experiments show that ComedicSpeech achieves better expressiveness than baselines with only ten-minute training data for each comedian.

Contents

Archietecture of ComedicSpeech

Baseline Comparison and Audio Samples Synthesized by ComedicSpeech

1.1 Lan Hu

Demo1: 而且他这高考出问题，我看着都心慌，这字儿我都不认识，我也是高材生啊

English translation:And he had a problem with the college entrance examination, I looked at the panic, this word I don't know, I am also a top student.

Demo2: 而且热搜上面是有标签的，就是那个什么营销号，就是那个一个井字儿对吧。

English translation:And there's a hashtag on it, the marketing account, you know, a hash symbol.

Demo3: 还拉了个群，叫“抄底小队”，后来才知道，那是“自杀小队”。

English translation:And formed a group called the "bottom Hunting Squad," which, as it turned out, was the "suicide Squad."

Demo4: 聊遗憾，我真的觉得遗憾挺好的，就相比你啥都留不下来，你至少还留下了遗憾嘛。

English translation:Talking about regret, I really think regret is good, compared to you can not save anything, you at least left regret.

1.2 Li Yang

Demo1: 然后我就给那个男的打电话，我说首先是这样的，我非常敬重你的人品。

English translation:And I called the guy, and I said, first of all, I have a lot of respect for your character.

Demo2: 不是的，这是一种偏见。我们家不重男轻女，在我们家是这样。

English translation:No, it's a form of prejudice. We don't have a preference for boys, not in our family.

Demo3: 但是没有一个孩子，把同学领回家写作业写着写着说，是不是累了？啊？

English translation:But there is no child, taking his classmates home to write homework and say, is it tired?

Demo4: 我真正开始有一点自信以后，是我开始说脱口秀以后，我觉得我长得挺好的，你知道吧。

English translation:When I really started to feel a little more confident, when I started talking talk, I thought I looked good, you know.

1.3 Bo Yang

Demo1: 有一天我睡的正香，我女朋友把我叫醒。

English translation:One day when I was fast asleep, my girlfriend woke me up.

Demo2: 后来，老板把我开了，可能他觉得，工作压力太大了。

English translation:Later, my boss fired me, maybe he felt the pressure of work was too much.

Demo3: 看见了吗，你们加一起也斗不过我。

English translation:See? You can't beat me together.

Demo4: 最后我想说，我去过很多公司。

English translation:Finally, I want to say that I've been to a lot of companies.

1.4 Niao Niao

Demo1: 谁能想到这其中最煎熬的竟然是中间的那十分钟。

English translation:Who would have thought the hardest part would be the 10 minutes in between.

Demo2: 我来之前还想，我的稿子里有地狱，会不会太沉重了呢。

English translation:Before I came here, I was wondering if my script was too heavy with hell in it.

Demo3: 自打办了健身卡，身体没有变健康，灵魂得到了升华。

English translation:Since running a gym membership, the body has not become healthy, the soul has been sublimated.

Ablation Studies

2.1 Lan Hu

compare1: 而且他这高考出问题，我看着都心慌，这字儿我都不认识，我也是高材生啊。

English translation:And he had a problem with the college entrance examination, I looked at the panic, this word I don't know, I am also a top student.

compare2: 而且热搜上面是有标签的，就是那个什么营销号，就是那个一个井字儿对吧。

English translation:And there's a hashtag on it, the marketing account, you know, a hash symbol.

compare3: 还拉了个群，叫“抄底小队”，后来才知道，那是“自杀小队”。

English translation:And formed a group called the "bottom Hunting Squad," which, as it turned out, was the "suicide Squad."

compare4: 聊遗憾，我真的觉得遗憾挺好的，就相比你啥都留不下来，你至少还留下了遗憾嘛。

English translation:Talking about regret, I really think regret is good, compared to you can not save anything, you at least left regret.

2.2 Li Yang

compare1: 然后我就给那个男的打电话，我说首先是这样的，我非常敬重你的人品。

English translation:And I called the guy, and I said, first of all, I have a lot of respect for your character.

compare2: 不是的，这是一种偏见。我们家不重男轻女，在我们家是这样。

English translation:No, it's a form of prejudice. We don't have a preference for boys, not in our family.

compare3: 但是没有一个孩子，把同学领回家写作业写着写着说，是不是累了？啊？

English translation:But there is no child, taking his classmates home to write homework and say, is it tired?

compare4: 我真正开始有一点自信以后，是我开始说脱口秀以后，我觉得我长得挺好的，你知道吧。

English translation:When I really started to feel a little more confident, when I started talking talk, I thought I looked good, you know.

2.3 Bo Yang

compare1: 有一天我睡的正香，我女朋友把我叫醒。

English translation:One day when I was fast asleep, my girlfriend woke me up.

compare2: 后来，老板把我开了，可能他觉得，工作压力太大了。

English translation:Later, my boss fired me, maybe he felt the pressure of work was too much.

compare3: 看见了吗，你们加一起也斗不过我。

English translation:See? You can't beat me together.

compare4: 最后我想说，我去过很多公司。

English translation:Finally, I want to say that I've been to a lot of companies.

2.4 Niao Niao

compare1: 谁能想到这其中最煎熬的竟然是中间的那十分钟。

English translation:Who would have thought the hardest part would be the 10 minutes in between.

compare2: 我来之前还想，我的稿子里有地狱，会不会太沉重了呢。

English translation:Before I came here, I was wondering if my script was too heavy with hell in it.

compare3: 自打办了健身卡，身体没有变健康，灵魂得到了升华。

English translation:Since running a gym membership, the body has not become healthy, the soul has been sublimated.

Case Study——The diversity of styles in ComedicSpeech to the same text (contain personal filler)

使用带个性填充物(杨笠的"你知道吧")的文本:

我真正开始有一点自信以后，是我开始说脱口秀以后，我觉得我长得挺好的，你知道吧。

English translation:Using text with personality filler (Yang Li's "You know what ") :

English translation:When I really started to feel a little more confident, when I started talking talk, I thought I looked good, you know.