Attacker Challenge at ICASSP 2025

The First VoicePrivacy Attacker Challenge is a new kind of challenge organized as part of the VoicePrivacy initiative. It focuses on developing attacker systems against voice anonymization, which will be evaluated against state-of-the-art anonymization systems including some submitted to the VoicePrivacy 2024 Challenge. Training, development, and evaluation datasets are provided along with baseline attacker systems. To develop attacker systems, the Challenge participants can use any additional training data and models, provided that they are openly available and declared before the specified deadline. Participants should develop their attacker systems in the form of automatic speaker verification systems and submit their scores on the test and development data to the organizers. The metric for evaluation is equal error rate (EER).

  • Task: develop an attacker system against voice anonymization systems in the form of an automatic speaker verification (ASV) system.

  • Data for training, development, and evaluation are provided along with baseline attacker systems. To develop attacker systems, the Challenge participants can use any additional training data and models, provided that they are openly available and declared before the specified deadline.

  • Metric: equal error rate (EER) resulting from the ASV attacker.

  • Results and accepted challenge papers will be presented at a special session at ICASSP 2025, Hyderabad, India.

Important dates

  • September 2024 - Release of training, development and evaluation data, baselines and evaluation software
  • 5th December 2024 - Submission of results and system descriptions
  • 9th Decemeber - Initation to submit a 2-page paper to ICASSP-2025 (sent to five top-ranked teams)
  • 9th 13th December 2024 - Submission of 2-page papers to ICASSP 2025 (by invitation)
  • 25th December 2024 - Update of the system description (allows to upload a more complete version of the submitted system description)
  • 30th December 2024 4th January 2025 - Paper acceptance notification
  • 10th January 2025 - Camera-ready 2-page ICASSP papers
  • 6th-11th April 2025 - ICASSP-2025
  • June 2025 - Journal papers

All deadlines are set for 11:59 PM on the respective day in US Pacific Standard Time

Registration

Participants are requested to register for the evaluation. Registration should be performed once only for each participating entity using the Registration form.

Participants remain the owners of their code and are not required to share it. No intellectual property rights are transferred to the challenge organizers.

Rules for the results submission

  • Submission deadline: 5th December 2024, PST
  • Submission link: dropbox links were sent by email to the registered teams

There is no limit on the number of submitted systems. Each participant may submit scores/results for one or more attacker systems, each targeting all anonymization systems or only some of them. The number of submitted attacker systems for each anonymization system should correspond to the number of submitted compressed archive files for this anonymization system. If the same attacker was applied to multiple anonymization systems, then each of these applications should correspond to a different submission file.

Each single submission should include the EER and corresponding ASV scores (for the development and evaluation data) obtained with the proposed attacker system for 4 trial lists (in the same format as generated by the baseline attacker system, see an example link in Section 7 of the evaluation plan):

  • data/libri_dev_trials_f/trials (example: libri_dev_enrolls-libri_dev_trials_f scores)
  • data/libri_dev_trials_m/trials (example: libri_dev_enrolls-libri_dev_trials_m scores)
  • data/libri_test_trials_f/trials (example: libri_test_enrolls-libri_test_trials_f scores)
  • data/libri_test_trials_m/trials (example: libri_test_enrolls-libri_test_trials_m scores)

Name of the archive file should be <Team name>_<Anonymization system id>.<Attacker id>.tar.gz (or zip), where

  • Team name - team name used in the registration form.
  • Anonymization system id - id of the anonymization system as written in the evaluation plan (B3, B4, B5, T8-5, T10-2, T12-5, or T25-1) should be exactly the same in the archive file and in the system description.
  • Attacker id - id of the attacker system, should be exactly the same as in the system description.

Rules for system descriptions

  • Submission deadline: 5th December 2024, PST
  • Submission link: dropbox links were sent by email to the registered teams, same as for results submission

  • Update deadline (allows to upload a more complete version of the submitted system description): 25th December 2024, PST
  • Submission link: will be announced later

All the submitted systems must be described in detail in the system descriptions. The challenge ranking will include only those systems which are compliant with the challenge rules (described in the evaluation plan) and for which we receive before the deadlines: scores, EER results, and complete and clear system description.

System description should contain all the results for the submitted systems. System descriptions may also include additional results without data scores and results submission, but these results will not be included in the official ranking. We also recommend adding a table that provides a summary description of each submitted system (modules/components with their short descriptions, input/output features, training data and open-source models used in training/development).

  • Format: ICASSP 2025 template, 2 pages or more (there is no upper limit)

Rules for the ICASSP-2025 paper

  • Submission deadline: 9th 13th December 2024, PST
  • Submission link: ICASSP 2025 submission system: (https://cmt3.research.microsoft.com/User/Login?ReturnUrl=%2FICASSP2025), choose “GC-7: The First VoicePrivacy Attacker Challenge”. Submission to the ICASSP-2025 will be done by invitation for top-5 participants
  • Format: ICASSP 2025, 2 pages including references. All 2-page papers should be covered by an ICASSP registration and should be presented in person at the conference.

Organizers

Contact: attacker.challenge@inria.fr

Follow @Voice-Privacy-Challenge