Command line tool

There are two main issues that may arise with results reported in the literature:

  • Even though the same public datasets are used, the actual evaluation protocol may differ slightly from one paper to another.

  • The implementation of the reported evaluation metrics may also differ.

The first objective of the pyannote.metrics library is to address these two problems, and provide a convenient way for researchers to evaluate their approaches in a reproducible and comparable manner.

Here is an example use of the command line interface that is provided to solve this problem.

$ pyannote-metrics diarization --subset=development Etape.SpeakerDiarization.TV hypothesis.rttm

Diarization (collar = 0 ms)               error    purity    coverage     total    correct      %    fa.      %    miss.     %    conf.      %
--------------------------------------  -------  --------  ----------  --------  ---------  -----  ------  -----  -------  ----  -------  -----
BFMTV_BFMStory_2011-03-17_175900          14.64     94.74       90.00   2582.08    2300.22  89.08   96.16   3.72    80.14  3.10   201.72   7.81
LCP_CaVousRegarde_2011-02-17_204700       17.80     89.13       86.90   3280.72    2848.42  86.82  151.78   4.63   208.29  6.35   224.01   6.83
LCP_EntreLesLignes_2011-03-18_192900      23.46     79.52       79.03   1704.97    1337.80  78.46   32.89   1.93   157.14  9.22   210.03  12.32
LCP_EntreLesLignes_2011-03-25_192900      26.75     76.97       75.86   1704.13    1292.83  75.86   44.61   2.62   158.38  9.29   252.92  14.84
LCP_PileEtFace_2011-03-17_192900          10.73     93.33       92.30   1611.49    1487.32  92.30   48.73   3.02    55.49  3.44    68.67   4.26
LCP_TopQuestions_2011-03-23_213900        18.28     98.25       94.20    727.26     668.65  91.94   74.36  10.22    16.41  2.26    42.20   5.80
LCP_TopQuestions_2011-04-05_213900        27.97     97.95       79.81    818.03     638.68  78.08   49.45   6.04    17.46  2.13   161.89  19.79
TV8_LaPlaceDuVillage_2011-03-14_172834    21.43     92.89       89.64    996.12     892.04  89.55  109.36  10.98    11.80  1.18    92.28   9.26
TV8_LaPlaceDuVillage_2011-03-21_201334    66.23     77.24       70.64   1296.86     691.76  53.34  253.80  19.57    29.16  2.25   575.95  44.41
TOTAL                                     23.27     88.18       84.55  14721.65   12157.71  82.58  861.14   5.85   734.28  4.99  1829.67  12.43

Tasks

Not only can pyannote-metrics command line tool be used to compute the diarization error rate using NIST implementation, one can also evaluate the typical four sub-modules used in most speaker diarization systems:

_images/pipeline.png

Practically, the first positional argument (e.g. diarization, above) is a flag indicating which task should be evaluated.

Apart from the diarization flag that is used for evaluating speaker diarization results, other available flags are detection (speech activity detection), segmentation (speaker change detection), and identification (supervised speaker identification). Depending on the task, a different set of evaluation metrics is computed.

Datasets and protocols

pyannote.metrics provides an easy way to ensure the same protocol (i.e. manual groundtruth and training/development/test split) is used for evaluation.

Internally, it relies on a collection of Python packages that all derive from the pyannote.database main package, that provides a convenient API to define training/development/test splits, along with groundtruth annotations. In the example above, the development set of the TV evaluation protocol of the ETAPE dataset is used.

Results are both reported for each file in the selected subset, and aggregated into one final metric value.

As of March 2017, pyannote.database packages exist for the ETAPE corpus, the REPERE corpus, and the AMI corpus. As more people contribute new pyannote.database packages, they will be added to the pyannote ecosystem.

File formats

Hypothesis files must use the [Rich Transcription Time Marked](https://web.archive.org/web/20170119114252/http://www.itl.nist.gov/iad/mig/tests/rt/2009/docs/rt09-meeting-eval-plan-v2.pdf) (RTTM) format.