Vocabulary Transfer and Knowledge Distillation for Language Model Compression

Natural Language Processing (NLP) has seen tremendous improvements over the last few years. Transformer architectures achieved impressive results in almost any NLP task, such as Text Classification, Machine Translation, and Language Generation. As time went by, transformers continued to improve thanks to larger corpora and bigger networks, reaching hundreds of billions of parameters. Training and deploying such large models has become prohibitively expensive, such that only big high tech companies can afford to train those models. Therefore, a lot of research has been dedicated to reducing a model’s size. In this thesis, we investigate the effects of Vocabulary Transfer and Knowledge Distillation for compressing large Language Models. The goal is to combine these two methodologies to further compress models without significant loss of performance. In particular, we designed different combination strategies and conducted a series of experiments on different vertical domains (medical, legal, news) and downstream tasks (Text Classification and Named Entity Recognition). Four different methods involving Vocabulary Transfer (VIPI) with and without a Masked Language Modelling (MLM) step and with and without Knowledge Distillation are compared against a baseline that assigns random vectors to new elements of the vocabulary. Results indicate that VIPI effectively transfers information of the original vocabulary and that MLM is beneficial. It is also noted that both vocabulary transfer and knowledge distillation are orthogonal to one another and may be applied jointly. The application of knowledge distillation first before subsequently applying vocabulary transfer is recommended. Finally, model performance due to vocabulary transfer does not always show a consistent trend as the vocabulary size is reduced. Hence, the choice of vocabulary size should be empirically selected by evaluation on the downstream task similar to hyperparameter tuning.

Divergence patterns in Fucus seaweeds (Phaeophyceae) in the northern Baltic Sea and in the Tjongspollen area

To be able to interpret patterns of biodiversity it is important to understand the processes by which new species evolve and how closely related species remain reproductively isolated and ecologically differentiated. Divergence and differentiation can vary during speciation and it can be seen in different stages. Groups of closely related taxa constitute important case studies to understand species and new biodiversity formation. However, it is important to assess the divergence among them at different organismal levels and from an integrative perspective. For this purpose, this study used the brown seaweeds genus Fucus as a model to study speciation, as they constitute a good opportunity to study divergence at different stages. We investigated the divergence patterns in Fucus species from two marginal areas (northern Baltic Sea and the Tjongspollen area), based on phenetic, phylogenetic and biological taxonomical criteria that are respectively characterised by algal morphology, allele frequencies of five microsatellite loci and levels of secondary polyphenolic compounds called phlorotannins. The results from this study showed divergence at morphological and genetic levels to certain extent but complete lack of divergence at biochemical level (i.e. constitutive phlorotannin production) in the Baltic Sea or Norway. Morphological divergence was clearly evident in Tjongspollen (Norway) among putative taxa as they were identified in the field and this divergence corresponds with their neutral genetic divergence. In the Baltic, there are some distinguishable patterns in the morphology of the swedish and finnish individuals according to locality to certain extent but not among putative taxa within localities. Likewise, these morphological patterns have genetic correspondence among localities but not within each locality. At the biochemical level, measured by the phlorotannin contents there were neither evidence of divergence in Norway or the Baltic Sea nor any discernable aggregation pattern among or within localities. Our study have contributed with further understanding of the Baltic Sea Fucus system and its intriguingly rapid and recent divergence as well as of the Tjongspollen area systems where formally undescribed individuals have been observed for the first time; in fact they appear largely differentiated and they may well warrant a new species status. In current times, climate change threatens, peripheral ecosystems, biodiversity, and increased knowledge of processes generating and maintaining biodiversity in those ecosystems seem particularly important and needed.

Correlation between structure and electrochemistry of LiMO2 cathode materials (M = Ni, Co)

Technical diversity and various knowledge is required for the understanding of undoubtedly complex system such as a Lithium-ion battery. The peculiarity is to combine different techniques that allow a complete investigation while the battery is working. Nowadays, research on Li-ion batteries (LIBs) is experiencing an exponential growth in the development of new cathode materials. Accordingly, Li-rich and Ni-rich NMCs, which have similar layered structure of LiMO2 oxides, have been recently proposed. Despite the promising performance on them, still a lot of issues have to be resolved and the materials need a more in depth characterisation for further commercial applications. In this study LiMO2 material, in particular M = Co and Ni, will be presented. We have focused on the synthesis of pure LiCoO2 and LiNiO2 at first, followed by the mixed LiNi0.5Co0.5O2. Different ways of synthesis were investigated for LCO but the sol-gel-water method showed the best performances. An accurate and systematic structural characterization followed by the appropriate electrochemical tests were done. Moreover, the in situ techniques (in-situ XRD and in situ OEMS) allowed a deep investigation in the structural change and gas evolution upon the electrochemically driven processes.

Problems in physics and mathematics contexts with middle and high school classes: vertical and horizontal comparison

My thesis falls within the framework of physics education and teaching of mathematics. The objective of this report was made possible by using geometrical (in mathematics) and qualitative (in physics) problems. We have prepared four (resp. three) open answer exercises for mathematics (resp. physics). The test batch has been selected across two different school phases: end of the middle school (third year, 8\textsuperscript{th} grade) and beginning of high school (second and third year, 10\textsuperscript{th} and 11\textsuperscript{th} grades respectively). High school students achieved the best results in almost every problem, but 10\textsuperscript{th} grade students got the best overall results. Moreover, a clear tendency to not even try qualitative problems resolution has emerged from the first collection of graphs, regardless of subject and grade. In order to improve students' problem-solving skills, it is worth to invest on vertical learning and spiral curricula. It would make sense to establish a stronger and clearer connection between physics and mathematical knowledge through an interdisciplinary approach.