Research

Five lines of work. They share one question: can a model produce something a person would actually use without fixing it first, and how would we know? Each line has a figure you can play with, built from the released data where there is some.

The work, one scene at a time

Nine short animations, one for each part of the work. They play once in order; pick any chapter to jump to it.

Open models for Europe's languages

At Villanova.AI I work on open language models trained in Italy for English, Italian, German, Spanish and French. For the first generation I built Villanova-SFT-2603, the instruction-tuning data behind Villanova-2B-2603: 1,711,114 conversations, released under Apache 2.0.

The second generation, Villanova2, came out in November 2026: a dense 2.4B and a 15.5B mixture of experts with open weights and documented data. All four models cite Multi-SFT, our paper on permissively licensed multilingual instruction data. Instruction data decides what a base model does when asked; keeping it open is what lets the models be open too.

Multi-SFT paper Villanova-2B-2603 Villanova2

Five real examples from Multi-SFT, one per language, from the paper's appendix: an instruction and the start of its answer. Pick a language.

Benchmarks for languages that have none

Most evaluation suites stop at English. I built commonsense benchmarks for Farsi and Arabic, a dialogue benchmark across Egyptian, Maghrebi and Modern Standard Arabic, a dialogue benchmark for Farsi, and instruction data that gets Sindhi models to reason in steps.

The figure is the simplest ArabicSense task: two sentences, one of which cannot be true. A model that has read a lot of Arabic text should never pick the wrong one; the paper measures how often they still do.

ArabicSense dataset ArabicSense code

Five test items from ArabicSense, unchanged. English glosses are mine.

Study material, generated and checked

A crossword clue is a small, strict test of a generator: it has to be true, it has to point at exactly one answer, and it has to be worth solving. I started there with Italian, then rebuilt the pipeline for Arabic and Turkish, where the puzzle conventions and the morphology are not the same.

The same question carried into quizzes, Persian multiple-choice items, pedagogical hints that help without giving the answer away, and syntax feedback on student essays.

Crosswords in Italian, Arabic and Turkish. Quizzes in Turkish. Questions in Persian. Hints in MultiHint-9K.

Measuring what models really do

Sixty people solved 160 Italian clues while we recorded how long each took. Difficulty tracks the surprisal a model assigns to the answer: how unexpected the answer is, given the clue. That gives a puzzle generator an objective dial instead of a setter's instinct.

Try the clues before you reveal the answers. Higher surprisal predicts that people will find a clue harder: in the paper, the correlation with accuracy reaches r = -0.62 for nominal clues.

Related work asks where models break: plural nouns with no singular, across many languages at once, and whether iterative in-context learning helps abstract reasoning on algebra.

Surprisal of the answer under Llama-3-8B for eight of the 160 clues, from the released Crossword-Surprisal-IT data. English glosses are mine.

Science and industry

The same methods, pointed at other data. To a language model a protein is a sentence in a twenty-letter alphabet, which is how I approached protein design and PharMistral, published in Computer Methods and Programs in Biomedicine Update.

On the industrial side: knowledge graphs built from plant documentation, then written back out as text to measure what survives the round trip; question answering over Italian regulatory documents; and emission forecasting for gas turbines, deployed at Baker Hughes.

A real protein, read the way a language model reads it.

The round trip on the paper's own example: graph to text, text back to graph, then compare. Simplified from the paper's figure.

Code and models

Each paper has one repository and one model collection, and the two link to each other. Most released checkpoints are LoRA adapters, so the work can be reproduced on a single GPU rather than a cluster.

17 papers with a public repository and a model collection.
Release	What is in it	Links
FarSense	Farsi commonsense benchmark, 6 fine-tuned adapters	Code Models
Shawarma Chats	Dialogue benchmark in three Arabic varieties	Code Models
Sindhi Reasoning	Culturally grounded instruction tuning for Sindhi	Code Models
Arabic crosswords	Arabic clue generation, annotation interface	Code Models
TalkFa	Farsi dialogue generation and understanding	Code Models
ArabicSense	Arabic commonsense benchmark and models	Code Models
PersianMCQ-Instruct	Persian multiple-choice question generation	Code Models
Round-trip KG	Text to knowledge graph and back, with fidelity scores	Code Models
Turkish quizzes	Quiz generation from Turkish educational text	Code Models
Turkish crosswords	Turkish clue generation models	Code Models
Italian crosswords	Italian clue generation across three papers	Code Models
Essay syntax feedback	Automated syntax feedback for student writing	Code Models
MultiHint-9K	Pedagogical hints, 9,000 of them, with five models	Code Models
Protein design	Protein sequence generation with language models	Code Models
PharmaQA.IT	Italian question answering over regulatory documents	Code Models
Crossword Surprisal	Clue difficulty predicted from token probabilities	Code Models
PharMistral	Code and models for the CMPB Update 2026 paper	Code Models