Directly training spiking neural networks for cyber-physical systems: from supervised to reinforcement learning

Spiking Neural Networks (SNNs) are bio-inspired Artificial Neural Networks (ANNs) utilizing discrete spiking signals, akin to neuron communication in the brain, making them ideal for real-time and energy-efficient Cyber-Physical Systems (CPSs). This thesis explores their potential in Structural Health Monitoring (SHM), leveraging low-cost MEMS accelerometers for early damage detection in motorway bridges. The study focuses on Long Short-Term SNNs (LSNNs), although their complex learning processes pose challenges. Comparing LSNNs with other ANN models and training algorithms for SHM, findings indicate LSNNs' effectiveness in damage identification, comparable to ANNs trained using traditional methods. Additionally, an optimized embedded LSNN implementation demonstrates a 54% reduction in execution time, but with longer pre-processing due to spike-based encoding. Furthermore, SNNs are applied in UAV obstacle avoidance, trained directly using a Reinforcement Learning (RL) algorithm with event-based input from a Dynamic Vision Sensor (DVS). Performance evaluation against Convolutional Neural Networks (CNNs) highlights SNNs' superior energy efficiency, showing a 6x decrease in energy consumption. The study also investigates embedded SNN implementations' latency and throughput in real-world deployments, emphasizing their potential for energy-efficient monitoring systems. This research contributes to advancing SHM and UAV obstacle avoidance through SNNs' efficient information processing and decision-making capabilities within CPS domains.

Adaptive and Maladaptive Implications of Reinforcement Learning Processes: Fronto-Striatal Loops and Behavioural Correlates

That humans and animals learn from interaction with the environment is a foundational idea underlying nearly all theories of learning and intelligence. Learning that certain outcomes are associated with specific actions or stimuli (both internal and external), is at the very core of the capacity to adapt behaviour to environmental changes. In the present work, appetitive and aversive reinforcement learning paradigms have been used to investigate the fronto-striatal loops and behavioural correlates of adaptive and maladaptive reinforcement learning processes, aiming to a deeper understanding of how cortical and subcortical substrates interacts between them and with other brain systems to support learning. By combining a large variety of neuroscientific approaches, including behavioral and psychophysiological methods, EEG and neuroimaging techniques, these studies aim at clarifying and advancing the knowledge of the neural bases and computational mechanisms of reinforcement learning, both in normal and neurologically impaired population.

Prevention of binge drinking in young adolescents through the promotion of a balanced level of psychological well-being: be-dry

This doctoral thesis presents a project carried out in secondary schools located in the city of Ferrara with the primary objective of demonstrating the effectiveness of an intervention based on Well-Being Therapy (Fava, 2016) in reducing alcohol use and improving lifestyles. In the first part (chapters 1-3), an introduction on risky behaviors and unhealthy lifestyle in adolescence is presented, followed by an examination of the phenomenon of binge drinking and of the concept of psychological well-being. In the second part (chapters 4-6), the experimental study is presented. A three-arm cluster randomized controlled trial including three test periods was implemented. The study involved eleven classes that were randomly assigned to receive well-being intervention (WBI), lifestyle intervention (LI) or not receive intervention (NI). Results were analyzed by linear mixed model and mixed-effects logistic regression with the aim to test the efficacy of WBI in comparison with LI and NI. AUDIT-C total score increased more in NI in comparison with WBI (p=0.008) and LI (p=0.003) at 6-month. The odds to be classified as at-risk drinker was lower in WBI (OR 0.01; 95%CI 0.01–0.14) and LI (OR 0.01; 95%CI 0.01–0.03) than NI at 6-month. The odds to use e-cigarettes at 6-month (OR 0.01; 95%CI 0.01–0.35) and cannabis at post-test (OR 0.01; 95%CI 0.01–0.18) were less in WBI than NI. Sleep hours at night decreased more in NI than in WBI (p = 0.029) and LI (p = 0.006) at 6-month. Internet addiction scores decreased more in WBI (p = 0.003) and LI (p = 0.004) at post-test in comparison with NI. Conclusions about the obtained results, limitations of the study, and future implications are discussed. In the seventh chapter, the data of the project collected during the pandemic are presented and compared with those from recent literature.

Unsupervised reinforcement learning via state entropy maximization

Reinforcement Learning (RL) provides a powerful framework to address sequential decision-making problems in which the transition dynamics is unknown or too complex to be represented. The RL approach is based on speculating what is the best decision to make given sample estimates obtained from previous interactions, a recipe that led to several breakthroughs in various domains, ranging from game playing to robotics. Despite their success, current RL methods hardly generalize from one task to another, and achieving the kind of generalization obtained through unsupervised pre-training in non-sequential problems seems unthinkable. Unsupervised RL has recently emerged as a way to improve generalization of RL methods. Just as its non-sequential counterpart, the unsupervised RL framework comprises two phases: An unsupervised pre-training phase, in which the agent interacts with the environment without external feedback, and a supervised fine-tuning phase, in which the agent aims to efficiently solve a task in the same environment by exploiting the knowledge acquired during pre-training. In this thesis, we study unsupervised RL via state entropy maximization, in which the agent makes use of the unsupervised interactions to pre-train a policy that maximizes the entropy of its induced state distribution. First, we provide a theoretical characterization of the learning problem by considering a convex RL formulation that subsumes state entropy maximization. Our analysis shows that maximizing the state entropy in finite trials is inherently harder than RL. Then, we study the state entropy maximization problem from an optimization perspective. Especially, we show that the primal formulation of the corresponding optimization problem can be (approximately) addressed through tractable linear programs. Finally, we provide the first practical methodologies for state entropy maximization in complex domains, both when the pre-training takes place in a single environment as well as multiple environments.

A glassful of graphene: graphene-based materials for drinking water remediation

Modern world suffers from an intense water crisis. Emerging contaminants represent one of the most concerning elements of this issue. Substances, molecules, ions, and microorganisms take part in this vast and variegated class of pollutants, which main characteristic is to be highly resistant to traditional water purification technologies. An intense international research effort is being carried out in order to find new and innovative solutions to this problem, and graphene-based materials are one of the most promising options. Graphene oxide (GO) is a nanostructured material where domains populated by oxygenated groups alternate with interconnected areas of sp2 hybridized carbon atoms, on the surface of a one-atom thick nanosheets. GO can adsorb a great number of molecules and ions on its surface, thanks to the variety of different interactions that it can express, such as hydrogen bonding, p-p stacking, and electrostatic and hydrophobic interaction. These characteristics, added to the high superficial area, make it an optimal material for the development of innovative materials for drinking water remediation. The main concern in the use of GO in this field is to avoid secondary contaminations (i.e. GO itself must not become a pollutant). This issue can be faced through the immobilization of GO onto polymeric substrates, thus developing composite materials. The use of micro/ultrafiltration polymeric hollow fibers as substrates allows the design of adsorptive membranes, meaning devices that can perform filtration and adsorption simultaneously. In this thesis, two strategies for the development of adsorptive membranes were investigated: a core-shell strategy, where hollow fibers are coated with GO, and a coextrusion strategy, where GO is embedded in the polymeric matrix of the fibers. The so-obtained devices were exploited for both fundamental studies (i.e. molecular and ionic behaviour in between GO nanosheets) and real applications (the coextruded material is now at TRL 9).

Contaminants of emerging concern in drinking water: assessment of the effectiveness of removal by drinking water treatments and possible risks to human health

Contaminants of emerging concern are increasingly detected in the water cycle, with endocrine-disrupting chemicals (EDCs) receiving attention due to their potential to cause adverse health effects even at low concentrations. Although the EU has recently introduced some EDCs into drinking water legislation, most drinking water treatment plants (DWTPs) are not designed to remove EDCs, making their detection and removal in DWTPs an important challenge. The aim of this doctoral project was to investigate hormones and phenolic compounds as suspected EDCs in drinking waters across the Romagna area (Italy). The main objectives were to assess the occurrence of considered contaminants in source and drinking water from three DWTPs, characterize the effectiveness of removal by different water treatment processes, and evaluate the potential biological impact on drinking water and human health. Specifically, a complementary approach of target chemical analysis and effect-based methods was adopted to explore drinking water quality, treatment efficacy, and biological potential. This study found that nonylphenol (NP) was prevalent in all samples, followed by BPA. Sporadic contamination of hormones was found only in source waters. Although the measured EDC concentrations in drinking water did not exceed threshold guideline values, the potential role of DWTPs as an additional source of EDC contamination should be considered. Significant increases in BPA and NP levels were observed during water treatment steps, which were also reflected in estrogenic and mutagenic responses in water samples after the ultrafiltration. This highlights the need to monitor water quality during various treatment processes to improve the efficiency of DWTPs. Biological assessments on finished water did not reveal any bioactivity, except for few treated water samples that exhibited estrogenic responses. Overall, the data emphasize the high quality of produced drinking water and the value of applying integrated chemical analysis and in vitro bioassays for water quality assessment.

Toxic cyanobacteria in water intended for drinking purpose: improvement of cells and toxins’ analyses and of treatments for their removal

Massive proliferations of cyanobacteria in freshwaters have recently increased, causing ecological and economic losses. Their ever-increasing presence in water sources destined to potabilization has become a major threat for public health, since several species can produce harmful toxins (cyanotoxin). Therefore, additional specific measures to improve management and treatment of drinking water(s) are required. The PhD thesis investigates toxic cyanobacteria in drinking waters with a special focus on Emilia-Romagna (Italy), throughout three separated chapters, each with different specific objectives. The first chapter aims at improving the fast monitoring of cyanobacteria in drinking water, which was investigated by testing different models of multi-wavelength spectrofluorometers. Inter-laboratories calibrations were conducted using mono-specific cultures and field samples, and both the feasibility and the technical limitations of such tools were illustrated. The second chapter evaluates the effectiveness of drinking water treatments in removing cyanobacterial cells and toxins. Two chlorinated oxidants (sodium hypochlorite and chlorine dioxide) already in use for pre-oxidation during water potabilization, were tested on cultures of the toxic cyanobacterium Microcystis aeruginosa posing a specific focus on toxin removal and revealing that pre-oxidation can cause the release of toxins and unknown metabolites. Innovative treatments based on non-thermal plasma were also tested, observing an effective and rapid inactivation of cyanobacterial cells. The third chapter presents a study on a cyanobacterium isolated from a drinking water reservoir of Emilia-Romagna and investigated by combining biological, chemical, and genomic methods. Although the strain did not produce any known cyanotoxin, high toxicity of water-extract was observed in bioassays and potential implications for drinking water were discussed. Overall, the PhD thesis offers new insights into toxic cyanobacteria management in drinking water, highlighting best practices for drinking water managers regarding their detection and removal. Additionally, the thesis provides new contributions to the understanding of the freshwater cyanobacteria community in the Emilia-Romagna region.

How to explain: from theory to practice

Today we live in an age where the internet and artificial intelligence allow us to search for information through impressive amounts of data, opening up revolutionary new ways to make sense of reality and understand our world. However, it is still an area of improvement to exploit the full potential of large amounts of explainable information by distilling it automatically in an intuitive and user-centred explanation. For instance, different people (or artificial agents) may search for and request different types of information in a different order, so it is unlikely that a short explanation can suffice for all needs in the most generic case. Moreover, dumping a large portion of explainable information in a one-size-fits-all representation may also be sub-optimal, as the needed information may be scarce and dispersed across hundreds of pages. The aim of this work is to investigate how to automatically generate (user-centred) explanations from heterogeneous and large collections of data, with a focus on the concept of explanation in a broad sense, as a critical artefact for intelligence, regardless of whether it is human or robotic. Our approach builds on and extends Achinstein’s philosophical theory of explanations, where explaining is an illocutionary (i.e., broad but relevant) act of usefully answering questions. Specifically, we provide the theoretical foundations of Explanatory Artificial Intelligence (YAI), formally defining a user-centred explanatory tool and the space of all possible explanations, or explanatory space, generated by it. We present empirical results in support of our theory, showcasing the implementation of YAI tools and strategies for assessing explainability. To justify and evaluate the proposed theories and models, we considered case studies at the intersection of artificial intelligence and law, particularly European legislation. Our tools helped produce better explanations of software documentation and legal texts for humans and complex regulations for reinforcement learning agents.

Thermal and air quality modeling of an electric vehicle cabin with low-cost sensors and reinforcement learning

The design process of any electric vehicle system has to be oriented towards the best energy efficiency, together with the constraint of maintaining comfort in the vehicle cabin. Main aim of this study is to research the best thermal management solution in terms of HVAC efficiency without compromising occupant’s comfort and internal air quality. An Arduino controlled Low Cost System of Sensors was developed and compared against reference instrumentation (average R-squared of 0.92) and then used to characterise the vehicle cabin in real parking and driving conditions trials. Data on the energy use of the HVAC was retrieved from the car On-Board Diagnostic port. Energy savings using recirculation can reach 30 %, but pollutants concentration in the cabin builds up in this operating mode. Moreover, the temperature profile appeared strongly nonuniform with air temperature differences up to 10° C. Optimisation methods often require a high number of runs to find the optimal configuration of the system. Fast models proved to be beneficial for these task, while CFD-1D model are usually slower despite the higher level of detail provided. In this work, the collected dataset was used to train a fast ML model of both cabin and HVAC using linear regression. Average scaled RMSE over all trials is 0.4 %, while computation time is 0.0077 ms for each second of simulated time on a laptop computer. Finally, a reinforcement learning environment was built in OpenAI and Stable-Baselines3 using the built-in Proximal Policy Optimisation algorithm to update the policy and seek for the best compromise between comfort, air quality and energy reward terms. The learning curves show an oscillating behaviour overall, with only 2 experiments behaving as expected even if too slow. This result leaves large room for improvement, ranging from the reward function engineering to the expansion of the ML model.