Automatic Text Evaluation through the Lens of Wasserstein Barycenters

Pierre Colombo (1, 2, 3) , Guillaume Staerman (2, 3) , Pablo Piantanida (4) , Chloé Clavel (2, 3)

1 IBM France Lab [Biot]
2 S2A - Signal, Statistique et Apprentissage
3 IDS - Département Images, Données, Signal
4 L2S - Laboratoire des signaux et systèmes

A new metric BaryScore to evaluate text generation based on deep contextualized embeddings (e.g., BERT, Roberta, ELMo) is introduced. This metric is motivated by a new framework relying on optimal transport tools, i.e., Wasserstein distance and barycenter. By modelling the layer output of deep contextualized embeddings as a probability distribution rather than by a vector embedding; this framework provides a natural way to aggregate the different outputs through the Wasserstein space topology. In addition, it provides theoretical grounds to our metric and offers an alternative to available solutions (e.g., MoverScore and BertScore). Numerical evaluation is performed on four different tasks: machine translation, summarization, data2text generation and image captioning. Our results show that BaryScore outperforms other BERT based metrics and exhibits more consistent behaviour in particular for text summarization.

Domaines: Traitement du texte et du document; Statistiques [stat]; Machine Learning [stat.ML]

Pierre Colombo, Guillaume Staerman, Pablo Piantanida, Chloé Clavel. Automatic Text Evaluation through the Lens of Wasserstein Barycenters. EMNLP 2021, Nov 2021, Punta Cana, Dominican Republic. pp.10450-10466, ⟨10.18653/v1/2021.emnlp-main.817⟩. ⟨hal-03423971⟩