Surprisal and Crossword Clues Difficulty: Evaluating Linguistic Processing between LLMs and Humans
Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), Cagliari, Italy, CEUR Workshop Proceedings
Presented on , 2 citations on Google Scholar
In short
The paper measures the difficulty of Italian crossword clues as the surprisal of the answer given the clue, computed with three language models (Llama-3-8B, Llama-2-7B and Ita-GPT-2-121M), and compares it with 60 human solvers on 160 clues. Surprisal correlated negatively with human accuracy (r = -0.62 for nominal clues), so it can serve as an automatic difficulty measure for puzzle generation.
Cite
@inproceedings{iaquinta-etal-2025-surprisal,
title = "Surprisal and Crossword Clues Difficulty: Evaluating Linguistic Processing between {LLM}s and Humans",
author = "Iaquinta, Tommaso and
Zanollo, Asya and
Fusco, Achille and
Zeinalipour, Kamyar and
Chesi, Cristiano",
editor = "Bosco, Cristina and
Jezek, Elisabetta and
Polignano, Marco and
Sanguinetti, Manuela",
booktitle = "Proceedings of the Eleventh {I}talian Conference on Computational Linguistics ({CL}i{C}-it 2025)",
month = sep,
year = "2025",
address = "Cagliari, Italy",
publisher = "CEUR Workshop Proceedings",
url = "https://aclanthology.org/2025.clicit-1.50/",
pages = "512--525",
ISBN = "979-12-243-0587-3"
}