Vocabulary Transfer and Knowledge Distillation for Language Model Compression

Natural Language Processing (NLP) has seen tremendous improvements over the last few years. Transformer architectures achieved impressive results in almost any NLP task, such as Text Classification, Machine Translation, and Language Generation. As time went by, transformers continued to improve thanks to larger corpora and bigger networks, reaching hundreds of billions of parameters. Training and deploying such large models has become prohibitively expensive, such that only big high tech companies can afford to train those models. Therefore, a lot of research has been dedicated to reducing a model’s size. In this thesis, we investigate the effects of Vocabulary Transfer and Knowledge Distillation for compressing large Language Models. The goal is to combine these two methodologies to further compress models without significant loss of performance. In particular, we designed different combination strategies and conducted a series of experiments on different vertical domains (medical, legal, news) and downstream tasks (Text Classification and Named Entity Recognition). Four different methods involving Vocabulary Transfer (VIPI) with and without a Masked Language Modelling (MLM) step and with and without Knowledge Distillation are compared against a baseline that assigns random vectors to new elements of the vocabulary. Results indicate that VIPI effectively transfers information of the original vocabulary and that MLM is beneficial. It is also noted that both vocabulary transfer and knowledge distillation are orthogonal to one another and may be applied jointly. The application of knowledge distillation first before subsequently applying vocabulary transfer is recommended. Finally, model performance due to vocabulary transfer does not always show a consistent trend as the vocabulary size is reduced. Hence, the choice of vocabulary size should be empirically selected by evaluation on the downstream task similar to hyperparameter tuning.

Development of a High-Efficiency Inverter with SiC Technology for Photovoltaic Applications

I convertitori elettronici di potenza sono da anni largamente diffusi sul mercato, poiché necessari per pilotare motori elettrici a velocità variabile in ambito industriale e a interfacciare alla rete elettrica i generatori da fonti di energia rinnovabili, in particolare eolica e fotovoltaica. In tutti i casi è richiesto che i convertitori siano efficienti, ovvero che l’energia persa durante la conversione sia minima. Considerando che il cuore di un convertitore è costituito da interruttori statici, è fondamentale una stima accurata dell’efficienza di questi ultimi per definire le prestazioni di un convertitore. Il materiale più diffuso per la realizzazione di questi componenti è il silicio (Si), ma esistono oggi altri semiconduttori che permettono prestazioni migliori, quali il carburo di silicio (SiC) e il nitruro di gallio (GaN). A causa del loro elevato costo di produzione rispetto ai componenti tradizionali, è divenuto sempre più importante stimare l’efficienza dei convertitori che montano queste tecnologie innovative, per trovare una giustificazione al loro impiego. In questa tesi, in particolare, l’efficienza di un convertitore con tecnologia SiC è stata prima calcolata analiticamente e poi testata a banco. Infine, è proposta un’architettura per un inverter trifase che possa essere impiegata per un’applicazione fotovoltaica. Questo lavoro è parte delle attività assegnate a Raw Power S.r.l. nell’ambito del progetto Europeo YesVGaN, il cui scopo ultimo entro i prossimi anni, è la realizzazione di componenti GaN commercialmente appetibili, che necessiteranno di essere testati e confrontati con i dispositivi al silicio e al carburo di silicio.

Implementation of network entropy algorithms on hpc machines, with application to high-dimensional experimental data

Network Theory is a prolific and lively field, especially when it approaches Biology. New concepts from this theory find application in areas where extensive datasets are already available for analysis, without the need to invest money to collect them. The only tools that are necessary to accomplish an analysis are easily accessible: a computing machine and a good algorithm. As these two tools progress, thanks to technology advancement and human efforts, wider and wider datasets can be analysed. The aim of this paper is twofold. Firstly, to provide an overview of one of these concepts, which originates at the meeting point between Network Theory and Statistical Mechanics: the entropy of a network ensemble. This quantity has been described from different angles in the literature. Our approach tries to be a synthesis of the different points of view. The second part of the work is devoted to presenting a parallel algorithm that can evaluate this quantity over an extensive dataset. Eventually, the algorithm will also be used to analyse high-throughput data coming from biology.