In 2021, SIGTYP is hosting a shared task on predicting language IDs from speech. While language ID is a fundamental speech and language processing task, it remains a challenging task, especially when going beyond the small set of languages past evaluation has focused on. Further, for many low-resource and endangered languages, only single-speaker recordings may be available, demanding a need for domain and speaker-invariant language ID systems.

We selected 16 languages from across the world, some of which share phonological features, and others where these have been lost or gained due to language contact, to perform what we call robust language ID: systems will be trained on largely single-speaker speech from one domain, but evaluated on data in other domains recorded from speakers under different recording circumstances, mimicking more realistic low-resource scenarios.

For training models, we provide participants with speech data from the CMU Wilderness Dataset, which contains read speech from the Bible in 699 languages, but usually recorded from a single speaker. This training data will be released in the form of derived MFCCs---please contact the organizers if you want to use other features.

The evaluation will be conducted on data from different sources, in particular data from the Common Voice project, several OpenSLR corpora (SLR24, SLR35, SLR35, SLR36, SLR64, SLR66, SLR79), and the Paradisec collection, testing systems’ capacity to generalize to new domains, new speakers, and new recording settings. We will also use these data sources to give participants validation data in all 16 languages to test their systems.

Please see the README in our data release below for more details about the specific languages and exact data size.

Participants will be invited to describe their system in a paper for the SIGTYP workshop proceedings. The task organizers will write an overview paper that describes the task and summarizes the different approaches taken, and analyzes their results.

  ↣  Download data!    Google Drive or OneDrive

  ↣  Register for the Task!  

  Training data Release: ↣  1 February 2021
  Test data Release: ↣  15 March 2021
  Submissions Due: ↣  31 March 2021 (AoE)
  Notification: ↣  15 April 2021
  Camera-ready Due: ↣  26 April 2021
  Workshop: ↣  10 June 2021

The shared task will consist of two settings (subtasks):
1) Constrained: only provided training data can be employed.
2) Unconstrained: training data can be extended with any publicly available source of information (e.g. additional speech, typological features, etc.)
Use of Common Voice or corpora hosted on OpenSLR for training is disallowed for all submissions.

* Note: the family and genus for Hakha Chin was initially incorrectly listed. Please follow this table.

Submissions should be emailed to the organizers by end of the day 31 March, anywhere in the world.
Submissions should follow the format of the training and validation label files, with tab-separated file ids and ISO 639-3 codes.
Files should be named as {team name}_{unconstrained/constrained} to indicate the subtask.

Papers describing shared task submissions should consist of 4 to 8 pages of content plus additional pages of references, formatted according to the NAACL 2021 format guidelines. For shared task paper submission, it is not necessary to blind the team name and authors. Accepted papers will be published online in the NAACL 2021 proceedings and will be virtually presented at the SIGTYP workshop at NAACL 2021. Writeups should be submitted through softconf (link to come), and are due by 31 March 2021 11.59 pm [UTC-12h].

    Please contact sigtyp AT gmail DOT com if you have any questions