The lecture will be postponed to autumn 2026.
Ondřej Tichý, Charles University, Prague
Benchmarking Large Language Models for DH Research
Recent advances in Large Language Models (LLMs) have shown that they can increasingly replace traditional NLP techniques across a range of linguistic tasks, such as part-of-speech tagging or orthographic normalization, and can even approach or surpass human annotators in tasks such as speech-act classification or genre annotation. However, the results of previous studies are often difficult to replicate, given the wide range of factors influencing LLM output, its inherent stochasticity, and the breakneck lifecycle of the individual models. Consequently, state-of-the-art results, their systematic comparison and generalization remain challenging.
In this paper, I propose a set of guidelines and Python scripts designed to make benchmarking LLMs on linguistic tasks more accessible, reproducible, and comparable. I also conduct several benchmarking experiments using this methodology to validate it and to identify best practices for applying LLMs to linguistic research, as well as to determine which current models perform best in specific tasks.
The proposed guidelines define input and output data structures for a variety of linguistic tasks, recommend parameter settings (e.g. temperature, top_p, chain-of-thought reasoning, retrieval-augmented generation), and outline how to interpret outputs such as self-reported confidence scores and token-level probabilities. The accompanying scripts enable researchers to (re)run tests on new tasks or new models and to generate comparable reports. While model fine-tuning is outside the scope of our framework, we support both zero- and few-shot prompting, allowing users to provide ground-truth data for evaluation and, optionally, as few-shot examples.
In my own tests, I will focus on tasks that have not been largely solved by NLP (avoiding e.g. PoS tagging in English) and that are commonly performed by empirical and more specifically corpus linguists. While most tasks target Present-Day English, I also investigate how LLMs handle low-resource languages and non-standardized varieties by including Czech and earlier stages of English. The selected tasks range from morphological and syntactic classification (e.g. identifying nominal number in Old English or the syntactic role of non-finite verbs in Present-Day English) to pragmatic annotation (e.g. contextual functions of like), semantic disambiguation, and historical spelling normalization.
I benchmark both major commercial models (e.g. ChatGPT, Gemini, Claude) and leading open-source or smaller models (e.g. gpt-oss, LLaMA, DeepSeek, Mistral), including different quantizations and configurations (leveraging the resources of the e-infra.cz research infrastructure). This enables evaluation not only of their performance but also of factors such as cost, accessibility, and data security, as well as testing claims such as smaller models outperforming larger ones on simple binary classifications.
My preliminary findings suggest that some linguistic tasks, such as text normalization and basic morphological classification, are already well-suited to LLM applications. In contrast, more complex tasks requiring extensive context and elaborate hierarchical categorization, such as discourse and pragmatic annotation, still fall significantly short of human performance.
