How well do today’s AI models handle Guarani? A comparative evaluation of LLMs for the second-most-spoken native language in America
This article presents a study evaluating 14 state-of-the-art generative AI models on Guarani translation tasks. The results reveal a clear trade-off between translation fidelity and linguistic validity: some models achieve strong benchmark scores while failing to consistently generate valid Guarani, whereas models fine-tuned on Guarani data substantially improve validity at the cost of multilingual robustness. Among the evaluated systems, Gemma-based open-weight models emerge as the strongest performers, making them promising candidates for future adaptation and specialization in Guarani.
Guarani is an ancestral language spoken by about 6.5 million people, including 70% of Paraguay’s population¹ and over half of its indigenous communities², as well as parts of neighboring countries. It is an official language of Paraguay and the second most widely spoken indigenous language in the Americas³.
Despite its widespread use, Guarani remains a low-resource language with a limited digital presence. This article reports on a series of experiments to assess the proficiency of state-of-the-art Large Language Models (LLMs) in handling Guarani, including both commercial and open-weight models. The ultimately goal is to find promising model candidates for future fine-tuning tasks with Guarani data.
This study is part of the GuaranIA project, an IDB-funded initiative to increase Guarani's presence on digital platforms, enabling native speakers to communicate in their language while preserving the cultural heritage of this underserved language.
Experimental setup
The study examines the ability of several state-of-the-art open-weight base and commercial LLMs to communicate in Guarani. These LLMs include variants of Google Gemma 2, 3, and 4; Meta Llama; Grok; OpenAI GPT; Nvidia Minitron; Mistral; Qwen; and a custom model based on Gemma 2 and fine-tuned with Guarani content, called Gemma-2–9B-IT-SimPO-Jopara.
RTT Framework
In the experiments, I adopt a round-trip translation (RTT) strategy⁴. In this setup, sentences in a pivot language — Spanish in our case — are first translated into Guarani, and then translated back into Spanish. Model performance is assessed by comparing the original Spanish sentences with their back-translated versions, providing a measure of how well meaning is preserved through the translation cycle.
Dataset
The evaluation is conducted on a dataset of 1,250 synthetically generated Spanish sentences spanning 25 diverse domains, including Arts, Literature, Business, and Technology, with a length range of 15 to 32 words. The sentences were generated using Azure OpenAI GPT-4.1, ensuring broad topical coverage and controlled variability across domains.
Evaluation Metrics
To assess translation quality, I rely on widely used evaluation metrics, including SacreBLEU⁵ and chrF++⁶. SacreBLEU is a standardized implementation of the BLEU score that measures n-gram overlap between the generated text and a reference translation. In contrast, chrF++ operates at the character level, capturing precision and recall for character n-grams, making it particularly well-suited for morphologically rich and low-resource languages where exact word matching may be overly strict, as in our study.
In addition, I incorporate RTTScore, a metric proposed by Zamir et al.⁷, to enable domain-conditioned evaluation and provide a more nuanced understanding of model performance across different contexts. RTTScore builds on the round-trip translation framework by comparing the original source sentence with its back-translation after passing through a target language. This fidelity is computed using metrics such as SacreBLEU or chrF++, averaging results across domains to quantify how well models preserve meaning throughout the round-trip translation cycle.
Results
Before evaluating translation performance, I first checked whether the models were actually generating valid Guarani. Beyond achieving strong round-trip translation (RTT) scores, an important question is whether these systems are actually capable of producing linguistically valid Guarani text.
To do this, I used Azure OpenAI GPT-5.4 as an independent evaluator to validate the generated Guarani translations. I then computed the proportion of valid outputs produced by each model across the 1,250 translated sentences. The reliability of this automatic evaluation was assessed by a native Guarani speaker, who manually reviewed a random sample of GPT-5.4 judgments. The assessment showed an agreement accuracy of 70%, providing reasonable confidence in the automated validation.
Figure 1 shows that certain models, specifically Gemma-4–26B, GPT-4o-mini, Grok-4, and Gemma-2–9B-IT-SimPO-Jopara, demonstrated promising capabilities in generating valid Guarani content, with Gemma-2 Jopara’s performance being particularly noteworthy due to its Guarani-specific fine-tuning. Conversely, models such as Nemotron-Mini-4B, Qwen3–4B, and Mistral-7B produced notably poor-quality Guarani translations, leading to their exclusion from subsequent analyses.
Comparative analysis
Next, I compare the models' RTT performance using the chrf++ and sacrebleu metrics.
As shown in Figure 2, GPT-4o-mini achieved the highest SacreBLEU score, followed by Gemma-2–9b-it-SimPO. When focusing solely on open-weight models, Gemma-2–9b-it-SimPO clearly stands out with the highest score, while Llama-3.1–8B-it and Gemma-4–26B-A4B-it emerge as promising alternatives.
For chrf++, GPT-4o-mini again leads with a score of 33.50, closely followed by Gemma-2–9b-it-SimPO (28.57) and Gemma-4–26B-A4B-it (25.64). Similar to SacreBLEU, Gemma-2–9B-IT-SimPO is the clear winner among the open-weight models, although the difference between it and its closest competitors (Gemma-4–26B and Llama 3.1–8B) is less pronounced.
Gemma-2–9b-it-SimPO clearly leads the RTT performance as measured by both SacreBLEU and chrF++. Gemma-4–26B-A4B-it emerges as a promising alternative.
The combined analysis of chrF++ and the proportion of valid Guarani translations offers a nuanced perspective on model performance. The combined plot for all models (see Figure 3) reveals that while some models achieve high chrF++ scores, their proportion of valid Guarani translations might be lower, and vice versa.
Get Jorge Saldivar’s stories in your inbox
Join Medium for free to get updates from this writer.
For instance, Gemma-2–9b-it-SimPO-Jopara (the Guarani fine-tuned model) stands out for its very high proportion of valid Guarani text, albeit with a significantly lower chrF++ score compared to the top-performing models. This indicates that while it produces grammatically sound Guarani, the back-translation into Spanish does not closely match the original source, as reflected by the chrF++ metric. A closer inspection of its outputs reveals that the model simply reproduces the Guarani translation rather than translating it back into Spanish, suggesting that the fine-tuning process may have severely degraded its Spanish generation capabilities.
Although the fine-tuned model Gemma-2–9b-it-SimPO-Jopara produces the highest volume of valid Guarani text, its round-trip translation (RTT) performance remains limited — even though it was specifically optimized for Guarani.
Conversely, Gemma-2–9b-it-SimPO achieves a relatively high chrF++ score, indicating better round-trip fidelity, but generates a lower proportion of valid Guarani translations. Among the evaluated models, Gemma-4–26B-A4B-it appears to offer the best trade-off between accurate back-translation into Spanish and the generation of valid Guarani content.
Gemma-4–26B-A4B-it demonstrates the strongest balance between preserving semantic fidelity and producing valid intermediate Guarani, making it a promising candidate for downstream fine-tuning and specialization in this low-resource language.
To validate the trends observed in the combined analysis, I next focus on the results obtained by applying the RTTScore metric proposed by Zamir and colleagues. As I mentioned before, RTTScore is a domain-conditioned metric that enables fine-grained analysis of how models' performance generalizes across different domains by using a fidelity metric to measure the round-trip translation results. In this case, I use chrF++ as the fidelity function, as shown in Figure 4.
Upon reviewing Figure 4, GPT-4o-mini leads significantly, with high end-to-end quality in the round-trip translation across domains. Among the open-weight models, Gemma-2–9b-it-SimPO and Gemma-4–26B-A4B-it achieve the highest cross-domain RTT performance. These figures are generally in line with previous results, reinforcing the findings on the leading models’ (Gemma-2–9b-it-SimPO, Gemma-4–26B-A4B-it) ability to maintain semantic and syntactic fidelity not only throughout the translation cycle but also aggregated by domains.
Finally, I analyze round-trip translation (RTT) performance across domains using chrF++ as the evaluation metric. As shown in Figure 5, the results are largely consistent with previous findings, reinforcing the overall ranking observed in earlier analyses. In particular, Gemma-2–9b-it-SimPO maintains a clear lead, achieving the highest chrF++ scores in the majority of domains. Its performance is especially notable for its consistency, suggesting strong cross-domain generalization rather than domain-specific strengths. However, this dominance is not universal. In domains such as literature, telecommunications, and health, its advantage diminishes.
For the health domain, Gemma-4–26B-A4B-it emerges as the strongest performer, achieving the highest scores and demonstrating a promising performance in specialized domains. This indicates that while smaller models like Gemma-2–9b-it-SimPO can excel broadly, larger models may retain an edge in domains that require more nuanced or context-rich translations.
Overall, Gemma-2–9b-it-SimPO offers the most robust and consistent performance across domains, while Gemma-4–26B-A4B-it serves as a strong alternative in domains, like health, where additional capacity appears to provide an advantage.
Conclusions
The results paint a clear and consistent picture of the current capabilities of open-weight LLMs for Guarani. Across aggregate metrics, round-trip evaluation, and domain-level analysis, a couple of models repeatedly emerge at the top. Gemma-2–9b-it-SimPO stands out for its strong and remarkably stable performance, offering the best balance between translation fidelity and cross-domain robustness.
At the same time, Gemma-4–26B-A4B-it consistently ranks among the top performers and proves particularly competitive in more challenging domains, suggesting that additional model capacity can still yield meaningful gains. Also, the analyses highlight that while fine-tuning can dramatically improve the validity of Guarani outputs, if not conducted correctly, it can degrade multilingual competence during specialization.
From a practical perspective, these findings suggest that open-weight models already provide a solid foundation for stronger models for Guarani designing by designing fine-tuning strategies that enhance both linguistic validity and semantic fidelity.
Models under study
I evaluate 14 state-of-the-art models in this study. The analysis primarily focuses on open-weight LLMs — such as the Llama and Gemma families — to better understand how far openly available models can go in a low-resource setting like Guarani. I also included two commercial models as reference points, helping us see how open models compare against proprietary AI systems.
I include the latest generation — Gemma 4 — testing both the E4B and 26B-A4B instruct variants, alongside earlier models from Gemma 3 (4B and 12B instruct). I also take a closer look at Gemma 2, not through its base form but through two fine-tuned variants of the 9B instruct model. The first is optimized using the SimPO technique⁸, while the second goes a step further, being fine-tuned on Guarani and Jopara data (a Guarani variant in which Guarani words and expressions are combined with Spanish terms) and also optimized with SimPO.
The 8B instruct variant of Llama 3.1 was also part of the study, as it is known to have been trained on low-resource languages through the No Language Left Behind initiative. In addition, the Nvidia Minitron family was represented in the study by 4B and 8B instruct variants. The 7B instruct variant of Mistral AI’s model was also included in the experiments. Similarly, the multilingual models Apertus and Qwen 3 were present in the study through the 8B and 4B instruct variants, respectively.
Among the commercial alternatives, the fast-non-reasoning variant of Grok 4, as well as the OpenAI GPT-4o-mini, were included in the experiments.
Dataset and code
The dataset and code used to conduct the experiments are publicly available here.
Acknowledgments
I would like to thank Ramón Araujo for his valuable contribution to the creation and validation of the dataset used in this study.
References
¹ Instituto Nacional de Estadística. (2025). 25 de agosto, día del Idioma Guaraní: “Ñamombarete ñane ñe’ẽ guarani, ñane rembiapópe, ha ñane rekove pukukue”. https://www.ine.gov.py/noticias/2538/25-de-agosto-dia-del-idioma-guarani-namombarete-nane-ne-guarani-nane-rembiapope-ha-nane-rekove-pukukue
² Instituto Nacional de Estadística. (2024,). Población indígena en el Paraguay se encuentra en torno a los 140.000 habitantes. https://www.ine.gov.py/noticias/1953/poblacion-indigena-en-el-paraguay-se-encuentra-en-torno-a-los-140000-habita
³ Garces, S. (2024). ¿Cuáles son las lenguas indígenas más habladas en América Latina? Babbel Magazine. https://es.babbel.com/es/magazine/lenguas-indigenas-mas-habladas-en-america-latina
⁴ Zhuo, T. Y., Xu, Q., He, X., & Cohn, T. (2023). Rethinking round-trip translation for machine translation evaluation. In Findings of the Association for Computational Linguistics: ACL 2023 (pp. 319–337).
⁵ Post, M. (2018). A call for clarity in reporting BLEU scores. In Proceedings of the third conference on machine translation: Research papers (pp. 186–191).
⁶ Popović, M. (2017). chrF++: words helping character n-grams. In Proceedings of the second conference on machine translation (pp. 612–618).
⁷ Zamir, S. W., Hamidouche, W., Amor, B. B., Marotti, L., Becker-Reshef, I., & Ferres, J. L. (2026). BYOL: Bring Your Own Language Into LLMs. arXiv preprint arXiv:2601.10804.
⁸ Meng, Y., Xia, M., & Chen, D. (2024). Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37, 124198–124235.
