The VoiceMOS Challenge 2022

The VoiceMOS Challenge 2022 has ended! Material from the challenge will remain available online. Read our summary paper of the challenge here.

Human listening tests are the gold standard for evaluating synthesized speech. Objective measures of speech quality have low correlation with human ratings, and the generalization abilities of current data-driven quality prediction systems suffer significantly from domain mismatch. The VoiceMOS Challenge aims to encourage research in the area of automatic prediction of Mean Opinion Scores (MOS) for synthesized speech. This challenge has two tracks:

• Main track: We recently collected a large-scale dataset of MOS ratings for a large variety of text-to-speech and voice conversion systems spanning many years, and this challenge releases this data to the public for the first time as the main track dataset.
• Out-of-domain track: The data for this track comes from a different listening test from the main track. The purpose of this track is to study the generalization ability of proposed MOS prediction models to a different listening test context. A smaller amount of labeled data is made available to participants, and unlabeled audio samples from the same listening test are made available as well, to encourage exploration of unsupervised and semi-supervised approaches.

Participation is open to all. The main track is required for all participants, and the out-of-domain track is optional. This challenge has preliminarily been accepted as a special session for Interspeech 2022, and participants are strongly encouraged to submit papers to the special session. The focus of the special session is on understanding and comparing MOS prediction techniques using a standardized dataset.

Participate

The challenge has ended, but you can still access the CodaLab challenge page here.

You can also find the BVCC dataset that was used in the challenge here.

Schedule

The schedule for the challenge is as follows:

• Release of main track and out-of-domain training data: current
• Release of evaluation data / start of test phase: February 21, 2022
• Test phase results submission deadline: February 28, 2022
• Results sent to participant: March 7, 2022
• Interspeech Paper submission deadline: March 21, 2022

Organizers

• Wen-Chin Huang (Nagoya University, Japan)
• Erica Cooper (National Institute of Informatics, Japan)
• Yu Tsao (Academia Sinica, Taiwan)
• Hsin-Min Wang (Academia Sinica, Taiwan)
• Tomoki Toda (Nagoya University, Japan)
• Junichi Yamagishi (National Institute of Informatics, Japan)