Challenging the Abilities of Large Language Models in Italian: a Community Initiative
Italian Journal of Computational Linguistics (IJCoL), vol. 12, n. 1 (June 2026), Accademia University Press, DOI 10.17454/ijcol121.02
Published online on , 3 citations on Google Scholar
In short
The paper describes CALAMITA, a benchmarking initiative coordinated by the Italian Association for Computational Linguistics in which more than 80 contributors built over 20 tasks and almost 100 subtasks to evaluate LLMs in Italian. It reports results for four open-weight LLMs and discusses lessons on fine-grained metrics, shared evaluation pipelines and community involvement.
Cite
@article{Nissim_2026,
title={Challenging the Abilities of Large Language Models in Italian: a Community Initiative},
volume={12},
ISSN={2499-4553},
url={http://dx.doi.org/10.17454/ijcol121.02},
DOI={10.17454/ijcol121.02},
number={1},
journal={Italian Journal of Computational Linguistics},
publisher={Lexis srl},
author={Nissim, Malvina and Croce, Danilo and Patti, Viviana and Basile, Pierpaolo and Attanasio, Giuseppe and Musacchio, Elio and Rinaldi, Matteo and Borazio, Federico},
year={2026},
pages={33}
}