Metrics

While the metrics for VoicePrivacy 2022 remain similar to those used for the inaugural 2020 challenge edition, updated software packages will is provided and we will be defining a set of different evaluation conditions. These stimpulate minimum targets for anonymisation performance expressed in terms of the equal error rate (EER) for a provided automatic speaker verification system. For each evaluation condition, submissions will then be ranked in terms of the lowest resulting word error rate (WER) for a provided automatic speech recognition system. As for VoicePrivacy 2020, the challenge is hence to meet or exceed the specified EER while minimising the WER.

We propose to use both objective and subjective metrics to assess speaker verification/re-identification ability. In addition to preserving privacy, the privacy-driven transformation should preserve speech intelligibility and naturalness when used in human communication scenarios, and automatic speech recognition (ASR) training and/or testing performance when used in human-machine communication scenarios.

Primary objective metrics:

  • Privacy (speaker verifiability): Equal error rate (EER)
  • Utility: Word error rate (WER)

Secondary objective utility metrics:

  • Pitch correlation between original and anonymized speech signals
  • Gain of voice distinctiveness

Subjective metrics:

  • Subjective speaker verifiability
  • Subjective speech intelligibility
  • Subjective speech naturalness