Defesa – Dissertação de Gustavo Tamiosso

Aluno(a): Gustavo Lopes Tamiosso
Orientador(a): Dennis Giovani Balreira

Título: Evaluating Small Language Models for Machine Translation: Quantization, Benchmark Transfer, and Temporal Robustness
Linha de Pesquisa: Processamento de Linguagem Natural

Data: 03/11/2026
Hora: 10:00
Local: Esta banca ocorrerá de forma híbrida (virtual e presencial), na sala Sala 215 / Prédio 43412 do Instituto de Informática/UFRGS e pelo link https://conferenciaweb.rnp.br/sala/dgbalreira.

Banca Examinadora:
-Viviane Pereira Moreira (UFRGS)
-Renata Vieira (UÉvora)
-Marcelo Resende Thielo (UNIPAMPA)

Presidente da Banca: Dennis Giovani Balreira

Resumo: Machine translation has increasingly benefited from language models that can be deployed locally, including relatively small and quantized models. However, evaluating their translation quality and determining whether conclusions drawn from standard benchmarks generalize to newer and domain-specific text remain open challenges. Moreover, static machine translation benchmarks such as FLORES-101 and OPUS-100 are fixed at release time, limiting their ability to capture changes in model performance across publication periods and domains. This dissertation addresses these questions through two connected studies. The first study evaluates models from 135M to 20B parameters at FP16, Q8_0, and Q4_K_M precision for English-to-Portuguese translation. Under the metrics used, Q8_0 preserves translation quality while reducing memory requirements, and several mid-sized open models approach proprietary systems. The second study introduces VaticanBen ch-MT, an alignment-audited, document-dated parallel corpus of official Vatican publications from 2008–2026 in English, Portuguese, Spanish, and Italian. We evaluate 38 models across six translation directions with BLEU, chrF, COMET-Kiwi, and xCOMET. The 2025-01-01 split is a common publication boundary, not a model-specific training cutoff; only the 14 models released before 2025 form a clean post-release cohort, and release date is used as a proxy. Within this institutional domain, neither of two pre-2025 window definitions shows practically meaningful post-2025 degradation. The same conclusion holds for papal speeches, although their absolute rate of recent-year mentions remains low. Document-level non-inferiority holds in 41 of 42 BLEU/chrF/COMET-Kiwi contrasts under operational SESOIs of 1.0 BLEU/chrF and 0.01 COMET-Kiwi; these margins were not perceptually validated, and this is not a general equivalence result. xCOMET retains a small unresolved signal. A current-stack compar ison with FLORES-101 and OPUS-100 shows strong rank transfer across directions, while absolute offsets depend on the direction. For English-to-Portuguese, FLORES exceeds VaticanBench-MT by a median 4.6 BLEU and 7.8 chrF. The combined results support quantized local translation and show that benchmark rankings transfer more consistently than absolute scores within the domains, languages, and model sizes studied here.

Palavras-Chave: Machine Translation. Small Language Models. Quantization. Temporal Shift. VaticanBench-MT