Workshop on inter-annotator agreement in Paris

Iris Eskhol(LLL)  and Carole Etienne (ICAR) organised a workshop on Inter-Annotator agreement in Paris on June, 26th, 2018

Invited talks by: Karën Fort, Jean-Yves Antoine, Yann Mathet

SegCor: Inter-annotator agreement for annotations based on both categorization and boundaries

Our workshop program:

  • Iris Eshkol-Taravella & Carole Etienne:  « A short presentation of the specificities of SegCor annotations and our expectations as regard Inter-annotator agreement »
  • Karën Fort & Yann Mathet: « The good, the bad and the metrics: understanding inter-annotator agreement »
  • Jean-Yves Antoine: « An experimental approach to give a meaning to kappa and alpha inter-annotator agreement measures »
  • Thomas Schmidt (IDS Mannheim): « A crowdsourcing annotation experiment to segment oral data using pauses »
  • Discussions applied to SegCor data


As several annotators proceeded to manual segmentation tasks, we get interested in inter-annotator agreement even if most part of the time we only have two or three annotators due to the time allowed for manual annotations before segmentation