Optimizing AI at the Edge: from network topology design to MCU deployment

The first topic analyzed in the thesis will be Neural Architecture Search (NAS). I will focus on two different tools that I developed, one to optimize the architecture of Temporal Convolutional Networks (TCNs), a convolutional model for time-series processing that has recently emerged, and one to optimize the data precision of tensors inside CNNs. The first NAS proposed explicitly targets the optimization of the most peculiar architectural parameters of TCNs, namely dilation, receptive field, and the number of features in each layer. Note that this is the first NAS that explicitly targets these networks. The second NAS proposed instead focuses on finding the most efficient data format for a target CNN, with the granularity of the layer filter. Note that applying these two NASes in sequence allows an "application designer" to minimize the structure of the neural network employed, minimizing the number of operations or the memory usage of the network. After that, the second topic described is the optimization of neural network deployment on edge devices. Importantly, exploiting edge platforms' scarce resources is critical for NN efficient execution on MCUs. To do so, I will introduce DORY (Deployment Oriented to memoRY) -- an automatic tool to deploy CNNs on low-cost MCUs. DORY, in different steps, can manage different levels of memory inside the MCU automatically, offload the computation workload (i.e., the different layers of a neural network) to dedicated hardware accelerators, and automatically generates ANSI C code that orchestrates off- and on-chip transfers with the computation phases. On top of this, I will introduce two optimized computation libraries that DORY can exploit to deploy TCNs and Transformers on edge efficiently. I conclude the thesis with two different applications on bio-signal analysis, i.e., heart rate tracking and sEMG-based gesture recognition.

Vocabulary Transfer and Knowledge Distillation for Language Model Compression

Natural Language Processing (NLP) has seen tremendous improvements over the last few years. Transformer architectures achieved impressive results in almost any NLP task, such as Text Classification, Machine Translation, and Language Generation. As time went by, transformers continued to improve thanks to larger corpora and bigger networks, reaching hundreds of billions of parameters. Training and deploying such large models has become prohibitively expensive, such that only big high tech companies can afford to train those models. Therefore, a lot of research has been dedicated to reducing a model’s size. In this thesis, we investigate the effects of Vocabulary Transfer and Knowledge Distillation for compressing large Language Models. The goal is to combine these two methodologies to further compress models without significant loss of performance. In particular, we designed different combination strategies and conducted a series of experiments on different vertical domains (medical, legal, news) and downstream tasks (Text Classification and Named Entity Recognition). Four different methods involving Vocabulary Transfer (VIPI) with and without a Masked Language Modelling (MLM) step and with and without Knowledge Distillation are compared against a baseline that assigns random vectors to new elements of the vocabulary. Results indicate that VIPI effectively transfers information of the original vocabulary and that MLM is beneficial. It is also noted that both vocabulary transfer and knowledge distillation are orthogonal to one another and may be applied jointly. The application of knowledge distillation first before subsequently applying vocabulary transfer is recommended. Finally, model performance due to vocabulary transfer does not always show a consistent trend as the vocabulary size is reduced. Hence, the choice of vocabulary size should be empirically selected by evaluation on the downstream task similar to hyperparameter tuning.

Intermediate linguistic task fine-tuning on Multi-lingual models

State-of-the-art NLP systems are generally based on the assumption that the underlying models are provided with vast datasets to train on. However, especially when working in multi-lingual contexts, datasets are often scarce, thus more research should be carried out in this field. This thesis investigates the benefits of introducing an additional training step when fine-tuning NLP models, named Intermediate Training, which could be exploited to augment the data used for the training phase. The Intermediate Training step is applied by training models on NLP tasks that are not strictly related to the target task, aiming to verify if the models are able to leverage the learned knowledge of such tasks. Furthermore, in order to better analyze the synergies between different categories of NLP tasks, experimentations have been extended also to Multi-Task Training, in which the model is trained on multiple tasks at the same time.

Illuminazione pubblica: caso studio del comune di Sesto San Giovanni

Il presente elaborato finale ha lo scopo di illustrare tutti i parametri necessari alla progettazione di un “buon” impianto di illuminazione stradale, dotato di tutte le nuove tecnologie, per un maggior risparmio energetico incentrato alla valorizzazione e riqualificazione urbana. L’illuminazione pubblica negli ultimi decenni ha ricoperto un’importanza sempre maggiore all’interno della città e del territorio aumentando in modo esponenziale la presenza di corpi illuminanti e formando spesso una vera e propria macchia di luce durante la notte. L’implementazione degli apparecchi di illuminazione ha portato ad una maggiore sicurezza e, a volte, un maggiore comfort nelle zone cittadine, ma anche un notevole aumento di costi energetici a carico delle amministrazioni comunali. Pertanto, alla luce degli ultimi mesi, è necessario incrementare i servizi tecnologici basati sull’efficientamento energetico degli impianti, in particolare, adottando sempre di più la sorgente a LED e sistemi di riduzione del flusso luminoso, in sostituzione alle lampade obsolete esistenti con un basso rendimento energetico. Con lo sviluppo del traffico veicolare degli ultimi anni, l’illuminazione urbana ha modificato il suo aspetto nelle città, concentrandosi sul rispetto dei valori prescritti per consentire una maggiore sicurezza e scorrevolezza del traffico motorizzato durante le ore notturne a discapito della valorizzazione urbana. Infatti, le statistiche hanno mostrato che il tasso degli incidenti notturni, ossia il rapporto tra gli incidenti e il numero di veicoli in circolazione, è più elevato rispetto agli incidenti durante le ore diurne.

Low Power instrument Transformer: technologies and applications

Instrument transformers serve an important role in the protection and isolation of AC electrical systems for measurements of different electrical parameters like voltage, current, power factor, frequency, and energy. As suggested by name these transformers are used in connection with suitable measuring instruments like an ammeter, wattmeter, voltmeter, and energy meters. We have seen how higher voltages and currents are transformed into lower magnitudes to provide isolation between power networks, relays, and other instruments. Reducing transient, suppressing electrical noises in sensitive devices, standardization of instruments and relays up to a few volts and current. Transformer performance directly affects the accuracy of power system measurements and the reliability of relay protection. We classified transformers in terms of purpose, insulating medium, Voltage ranges, temperature ranges, humidity or environmental effect, indoor and outdoor use, performance, Features, specification, efficiency, cost analysis, application, benefits, and limitations which enabled us to comprehend their correct use and selection criteria based on our desired requirements. We also discussed modern Low power instrument transformer products that are recently launched or offered by renowned companies like Schneider Electric, Siemens, ABB, ZIV, G&W etc. These new products are innovations and problem solvers in the domain of measurement, protection, digital communication, advance, and commercial energy metering. Since there is always some space for improvements to explore new advantages of Low power instrument transformers in the domain of their wide linearity, high-frequency range, miniaturization, structural and technological modification, integration, smart frequency modeling, and output prediction of low-power voltage transformers.

Tree regularized neural networks for key point analysis

This thesis contributes to the ArgMining 2021 shared task on Key Point Analysis. Key Point Analysis entails extracting and calculating the prevalence of a concise list of the most prominent talking points, from an input corpus. These talking points are usually referred to as key points. Key point analysis is divided into two subtasks: Key Point Matching, which involves assigning a matching score to each key point/argument pair, and Key Point Generation, which consists of the generation of key points. The task of Key Point Matching was approached using different models: a pretrained Sentence Transformers model and a tree-constrained Graph Neural Network were tested. The best model was the fine-tuned Sentence Transformers, which achieved a mean Average Precision score of 0.75, ranking 12 compared to other participating teams. The model was then used for the subtask of Key Point Generation using the extractive method in the selection of key point candidates and the model developed for the previous subtask to evaluate them.

Sviluppo di Metodi di Soft Labeling per la Multi-Document Summarization in Ambito Legale

In questo elaborato viene trattata l’analisi del problema di soft labeling applicato alla multi-document summarization, in particolare vengono testate varie tecniche per estrarre frasi rilevanti dai documenti presi in dettaglio, al fine di fornire al modello di summarization quelle di maggior rilievo e più informative per il riassunto da generare. Questo problema nasce per far fronte ai limiti che presentano i modelli di summarization attualmente a disposizione, che possono processare un numero limitato di frasi; sorge quindi la necessità di filtrare le informazioni più rilevanti quando il lavoro si applica a documenti lunghi. Al fine di scandire la metrica di importanza, vengono presi come riferimento metodi sintattici, semantici e basati su rappresentazione a grafi AMR. Il dataset preso come riferimento è Multi-LexSum, che include tre granularità di summarization di testi legali. L’analisi in questione si compone quindi della fase di estrazione delle frasi dai documenti, della misurazione delle metriche stabilite e del passaggio al modello stato dell’arte PRIMERA per l’elaborazione del riassunto. Il testo ottenuto viene poi confrontato con il riassunto target già fornito, considerato come ottimale; lavorando in queste condizioni l’obiettivo è di definire soglie ottimali di upper-bound per l’accuratezza delle metriche, che potrebbero ampliare il lavoro ad analisi più dettagliate qualora queste superino lo stato dell’arte attuale.

Development and testing of a virtual energy meter using the Labview Environment

The quantity of electric energy utilized by a home, a business, or an electrically powered device is measured by an electricity meter, also known as an electric meter, electrical meter, or energy meter. Electric meters located at customers' locations are used by electric providers for billing. They are usually calibrated in billing units, with the kilowatt hour being the most popular (kWh). Typically, they are read once each billing cycle. When energy savings are sought during specific times, some meters may monitor demand, or the highest amount of electricity used during a specific time. Additionally, some meters feature relays for load shedding in response to responses during periods of peak load. The amount of electrical energy consumed by users is measured by a Watt-hour meter, also known as an energy meter. To charge the electricity usage by loads like lights, fans, and other appliances, utilities put these gadgets everywhere, including in households, businesses, and organizations. Watts are a fundamental power unit. A kilowatt is equal to one thousand watts. One kilowatt is regarded as one unit of energy used if used for one hour. These meters calculate the product of the instantaneous voltage and current readings and provide instantaneous power. This power is distributed over a period and is used during that time. Depending on the supply used by home or commercial installations, these may be single or three phase meters. These can be linked directly between line and load for minor service measurements, such as home consumers. However, step-down current transformers must be installed for greater loads to handle their higher current demands.

Antibody-Antigen Interface Characterization with Contextualized Pre-trained Embeddings

Artificial Intelligence (AI) has substantially influenced numerous disciplines in recent years. Biology, chemistry, and bioinformatics are among them, with significant advances in protein structure prediction, paratope prediction, protein-protein interactions (PPIs), and antibody-antigen interactions. Understanding PPIs is critical since they are responsible for practically everything living and have several uses in vaccines, cancer, immunology, and inflammatory illnesses. Machine Learning (ML) offers enormous potential for effectively simulating antibody-antigen interactions and improving in-silico optimization of therapeutic antibodies for desired features, including binding activity, stability, and low immunogenicity. This research looks at the use of AI algorithms to better understand antibody-antigen interactions, and it further expands and explains several difficulties encountered in the field. Furthermore, we contribute by presenting a method that outperforms existing state-of-the-art strategies in paratope prediction from sequence data.