📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project, a European sovereign large language model trained from scratch on extensive Italian data, achieved limited performance on academic benchmarks. This challenges assumptions about the scale needed for country-specific AI models and highlights ongoing debates about European AI strategies.
Italy’s Minerva-3B, a large language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, a performance near chance. This development questions the assumption that larger investments and scale alone can produce country-specific language models capable of handling complex tasks, highlighting a critical challenge for the European sovereign-LLM movement.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research infrastructure, trained a 7-billion-parameter model from scratch using a dataset of 2.5 trillion tokens, roughly half Italian. Despite the scale and open publication of weights, data, and code, the model’s performance on the INVALSI test was only 4.9%, a result considered significantly below expectations for a model trained on such extensive Italian data. Researchers concluded that while dataset composition matters, overall size and parameter count are more critical for complex language tasks.
This empirical result complicates the narrative that simply increasing native-language data and scale guarantees high performance. It suggests that the European sovereign-LLM effort may need to confront a harsher reality regarding the scale of investment required to develop truly country-knowledgeable models. The results highlight the importance of not just data quantity but also model architecture and training strategies in achieving meaningful language understanding.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Build a Large Language Model (From Scratch)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Fine-Tuning Large Language Models: From Custom Datasets to High-Performance AI Models Using Modern Toolchains
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Thames & Kosmos Simple Machines Science Experiment & Model Building Kit, Introduction to Mechanical Physics, Build 26 Models to Investigate The 6 Classic Simple Machines
Through 26 model-building exercise, gain hands-on experience with gears and all six classic simple machines: wheels and axles,…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-LLM Strategies
The limited performance of Minerva-3B underscores a key challenge for Europe’s AI sovereignty efforts: substantial investment in native-language data and model scale may still fall short of producing models capable of complex, country-specific tasks. This raises questions about the current approach and whether future investments should focus more on model architecture, training techniques, or larger-scale data curation to achieve desired outcomes. The findings also suggest that European projects need to reassess their expectations and strategies in building truly effective country-specific AI models, which are vital for national autonomy in AI.
European Sovereign-Language Model Development Approaches
Italy’s Minerva project represents a different approach from other European efforts like Portugal’s AMÁLIA. While AMÁLIA relies on continuation pre-training of a multilingual foundation with a small proportion of European Portuguese data, Minerva was trained from scratch on a massive Italian dataset. Despite the significant scale, Minerva’s low benchmark score reveals that larger data and parameters alone may not suffice. The project is part of Italy’s broader strategy to develop independent AI infrastructure, supported by national funding, supercomputing resources, and a dedicated research team. Prior to this, European projects have debated whether to focus on multilingual models or native-language specialization, with the latter often assuming scale would lead to better performance.
“The results highlight that the European sovereign-LLM movement may need to accept a harsher scaling reality than previously thought.”
— Thorsten Meyer
Unresolved Questions About Model Scaling and Performance
It remains unclear whether further scaling, different training methodologies, or architectural adjustments could improve Minerva’s performance on complex tasks. The ongoing research aims to determine the optimal investment level and strategies needed to produce effective country-specific models. Additionally, the impact of dataset quality versus quantity continues to be a subject of debate, and whether these findings generalize to other languages or domains is still unconfirmed.
Next Steps in European Sovereign AI Research
The Minerva team plans to continue iterating on training methodologies, including ongoing experiments with continual training cases in 2025. Future research will focus on scaling models further, refining data curation, and exploring architectural innovations. Policymakers and research institutions may need to reassess investment strategies based on these findings, emphasizing not just data volume but also model design and training techniques to achieve meaningful language understanding. The broader European AI community will likely monitor these developments to inform future projects and funding decisions.
Key Questions
Why did Minerva-3B perform so poorly on the Italian exam benchmark?
Despite extensive training on a large Italian dataset, Minerva-3B’s low score suggests that scale alone does not guarantee high performance on complex, academic language tasks. Factors like model architecture and training strategies also play critical roles.
Does this mean European sovereign-LLMs are not viable?
Not necessarily. It indicates that current approaches may need to incorporate larger scale, better data quality, or different architectures to meet their goals. The findings highlight the importance of realistic expectations and strategic adjustments.
What are the implications for future AI investments in Europe?
Investors and policymakers may need to reconsider the scale and scope of native-language AI projects, emphasizing not just data volume but also innovative training and model design to achieve desired capabilities.
Will Minerva’s results affect other European language models?
Potentially. The lessons learned from Minerva could influence strategies for other languages, emphasizing the need for substantial scale and possibly new methodologies to develop effective models.
Source: ThorstenMeyerAI.com
