I’m happy to share that our paper, Universal NER v2, was accepted to LREC 2026.

In this paper we present Universal NER (UNER) v2, a substantial extension of the dataset introduced in 2024. UNER is a collaborative resource for multilingual named-entity annotation, designed to support cross-lingual NER research.

UNER v2 adds 11 datasets covering 10 typologically diverse languages, including several aligned evaluation benchmarks, while preserving consistent annotation guidelines and high inter-annotator agreement. We provide detailed dataset statistics and benchmark performance using both encoder-based models and LLMs.

We compared human annotation with LLM-based annotation under the same guidelines. Our results show that LLMs still lag behind human annotators,and analyze the typical mistakes they make. While performance could likely be improved through more elaborate instructions or via agentic workflows, LLMs are not yet dependable annotators. That said, they show promise not only for annotation, but also for identifying inconsistencies in human labels and weaknesses in the guidelines, which we plan to explore in future work.

Terra Blevins, Stephen Mayhew, Marek Suppa, Hila Gonen, Shachar Mirkin, Vasile Pais, Kaja Dobrovoljc, Voula Giouli, Jun Kevin, Enes Yılandiloğlu, Eugene Jang, Eungseo Kim, Jeongyeon Seo, Xenophon Gialis and Yuval Pinter. Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark. LREC 2026.

Last updated: April 19, 2026