Unsupervised reinforcement learning via state entropy maximization

Reinforcement Learning (RL) provides a powerful framework to address sequential decision-making problems in which the transition dynamics is unknown or too complex to be represented. The RL approach is based on speculating what is the best decision to make given sample estimates obtained from previous interactions, a recipe that led to several breakthroughs in various domains, ranging from game playing to robotics. Despite their success, current RL methods hardly generalize from one task to another, and achieving the kind of generalization obtained through unsupervised pre-training in non-sequential problems seems unthinkable. Unsupervised RL has recently emerged as a way to improve generalization of RL methods. Just as its non-sequential counterpart, the unsupervised RL framework comprises two phases: An unsupervised pre-training phase, in which the agent interacts with the environment without external feedback, and a supervised fine-tuning phase, in which the agent aims to efficiently solve a task in the same environment by exploiting the knowledge acquired during pre-training. In this thesis, we study unsupervised RL via state entropy maximization, in which the agent makes use of the unsupervised interactions to pre-train a policy that maximizes the entropy of its induced state distribution. First, we provide a theoretical characterization of the learning problem by considering a convex RL formulation that subsumes state entropy maximization. Our analysis shows that maximizing the state entropy in finite trials is inherently harder than RL. Then, we study the state entropy maximization problem from an optimization perspective. Especially, we show that the primal formulation of the corresponding optimization problem can be (approximately) addressed through tractable linear programs. Finally, we provide the first practical methodologies for state entropy maximization in complex domains, both when the pre-training takes place in a single environment as well as multiple environments.

A glassful of graphene: graphene-based materials for drinking water remediation

Modern world suffers from an intense water crisis. Emerging contaminants represent one of the most concerning elements of this issue. Substances, molecules, ions, and microorganisms take part in this vast and variegated class of pollutants, which main characteristic is to be highly resistant to traditional water purification technologies. An intense international research effort is being carried out in order to find new and innovative solutions to this problem, and graphene-based materials are one of the most promising options. Graphene oxide (GO) is a nanostructured material where domains populated by oxygenated groups alternate with interconnected areas of sp2 hybridized carbon atoms, on the surface of a one-atom thick nanosheets. GO can adsorb a great number of molecules and ions on its surface, thanks to the variety of different interactions that it can express, such as hydrogen bonding, p-p stacking, and electrostatic and hydrophobic interaction. These characteristics, added to the high superficial area, make it an optimal material for the development of innovative materials for drinking water remediation. The main concern in the use of GO in this field is to avoid secondary contaminations (i.e. GO itself must not become a pollutant). This issue can be faced through the immobilization of GO onto polymeric substrates, thus developing composite materials. The use of micro/ultrafiltration polymeric hollow fibers as substrates allows the design of adsorptive membranes, meaning devices that can perform filtration and adsorption simultaneously. In this thesis, two strategies for the development of adsorptive membranes were investigated: a core-shell strategy, where hollow fibers are coated with GO, and a coextrusion strategy, where GO is embedded in the polymeric matrix of the fibers. The so-obtained devices were exploited for both fundamental studies (i.e. molecular and ionic behaviour in between GO nanosheets) and real applications (the coextruded material is now at TRL 9).

Contaminants of emerging concern in drinking water: assessment of the effectiveness of removal by drinking water treatments and possible risks to human health

Contaminants of emerging concern are increasingly detected in the water cycle, with endocrine-disrupting chemicals (EDCs) receiving attention due to their potential to cause adverse health effects even at low concentrations. Although the EU has recently introduced some EDCs into drinking water legislation, most drinking water treatment plants (DWTPs) are not designed to remove EDCs, making their detection and removal in DWTPs an important challenge. The aim of this doctoral project was to investigate hormones and phenolic compounds as suspected EDCs in drinking waters across the Romagna area (Italy). The main objectives were to assess the occurrence of considered contaminants in source and drinking water from three DWTPs, characterize the effectiveness of removal by different water treatment processes, and evaluate the potential biological impact on drinking water and human health. Specifically, a complementary approach of target chemical analysis and effect-based methods was adopted to explore drinking water quality, treatment efficacy, and biological potential. This study found that nonylphenol (NP) was prevalent in all samples, followed by BPA. Sporadic contamination of hormones was found only in source waters. Although the measured EDC concentrations in drinking water did not exceed threshold guideline values, the potential role of DWTPs as an additional source of EDC contamination should be considered. Significant increases in BPA and NP levels were observed during water treatment steps, which were also reflected in estrogenic and mutagenic responses in water samples after the ultrafiltration. This highlights the need to monitor water quality during various treatment processes to improve the efficiency of DWTPs. Biological assessments on finished water did not reveal any bioactivity, except for few treated water samples that exhibited estrogenic responses. Overall, the data emphasize the high quality of produced drinking water and the value of applying integrated chemical analysis and in vitro bioassays for water quality assessment.

Toxic cyanobacteria in water intended for drinking purpose: improvement of cells and toxins’ analyses and of treatments for their removal

Massive proliferations of cyanobacteria in freshwaters have recently increased, causing ecological and economic losses. Their ever-increasing presence in water sources destined to potabilization has become a major threat for public health, since several species can produce harmful toxins (cyanotoxin). Therefore, additional specific measures to improve management and treatment of drinking water(s) are required. The PhD thesis investigates toxic cyanobacteria in drinking waters with a special focus on Emilia-Romagna (Italy), throughout three separated chapters, each with different specific objectives. The first chapter aims at improving the fast monitoring of cyanobacteria in drinking water, which was investigated by testing different models of multi-wavelength spectrofluorometers. Inter-laboratories calibrations were conducted using mono-specific cultures and field samples, and both the feasibility and the technical limitations of such tools were illustrated. The second chapter evaluates the effectiveness of drinking water treatments in removing cyanobacterial cells and toxins. Two chlorinated oxidants (sodium hypochlorite and chlorine dioxide) already in use for pre-oxidation during water potabilization, were tested on cultures of the toxic cyanobacterium Microcystis aeruginosa posing a specific focus on toxin removal and revealing that pre-oxidation can cause the release of toxins and unknown metabolites. Innovative treatments based on non-thermal plasma were also tested, observing an effective and rapid inactivation of cyanobacterial cells. The third chapter presents a study on a cyanobacterium isolated from a drinking water reservoir of Emilia-Romagna and investigated by combining biological, chemical, and genomic methods. Although the strain did not produce any known cyanotoxin, high toxicity of water-extract was observed in bioassays and potential implications for drinking water were discussed. Overall, the PhD thesis offers new insights into toxic cyanobacteria management in drinking water, highlighting best practices for drinking water managers regarding their detection and removal. Additionally, the thesis provides new contributions to the understanding of the freshwater cyanobacteria community in the Emilia-Romagna region.

How to explain: from theory to practice

Today we live in an age where the internet and artificial intelligence allow us to search for information through impressive amounts of data, opening up revolutionary new ways to make sense of reality and understand our world. However, it is still an area of improvement to exploit the full potential of large amounts of explainable information by distilling it automatically in an intuitive and user-centred explanation. For instance, different people (or artificial agents) may search for and request different types of information in a different order, so it is unlikely that a short explanation can suffice for all needs in the most generic case. Moreover, dumping a large portion of explainable information in a one-size-fits-all representation may also be sub-optimal, as the needed information may be scarce and dispersed across hundreds of pages. The aim of this work is to investigate how to automatically generate (user-centred) explanations from heterogeneous and large collections of data, with a focus on the concept of explanation in a broad sense, as a critical artefact for intelligence, regardless of whether it is human or robotic. Our approach builds on and extends Achinstein’s philosophical theory of explanations, where explaining is an illocutionary (i.e., broad but relevant) act of usefully answering questions. Specifically, we provide the theoretical foundations of Explanatory Artificial Intelligence (YAI), formally defining a user-centred explanatory tool and the space of all possible explanations, or explanatory space, generated by it. We present empirical results in support of our theory, showcasing the implementation of YAI tools and strategies for assessing explainability. To justify and evaluate the proposed theories and models, we considered case studies at the intersection of artificial intelligence and law, particularly European legislation. Our tools helped produce better explanations of software documentation and legal texts for humans and complex regulations for reinforcement learning agents.

Thermal and air quality modeling of an electric vehicle cabin with low-cost sensors and reinforcement learning

The design process of any electric vehicle system has to be oriented towards the best energy efficiency, together with the constraint of maintaining comfort in the vehicle cabin. Main aim of this study is to research the best thermal management solution in terms of HVAC efficiency without compromising occupant’s comfort and internal air quality. An Arduino controlled Low Cost System of Sensors was developed and compared against reference instrumentation (average R-squared of 0.92) and then used to characterise the vehicle cabin in real parking and driving conditions trials. Data on the energy use of the HVAC was retrieved from the car On-Board Diagnostic port. Energy savings using recirculation can reach 30 %, but pollutants concentration in the cabin builds up in this operating mode. Moreover, the temperature profile appeared strongly nonuniform with air temperature differences up to 10° C. Optimisation methods often require a high number of runs to find the optimal configuration of the system. Fast models proved to be beneficial for these task, while CFD-1D model are usually slower despite the higher level of detail provided. In this work, the collected dataset was used to train a fast ML model of both cabin and HVAC using linear regression. Average scaled RMSE over all trials is 0.4 %, while computation time is 0.0077 ms for each second of simulated time on a laptop computer. Finally, a reinforcement learning environment was built in OpenAI and Stable-Baselines3 using the built-in Proximal Policy Optimisation algorithm to update the policy and seek for the best compromise between comfort, air quality and energy reward terms. The learning curves show an oscillating behaviour overall, with only 2 experiments behaving as expected even if too slow. This result leaves large room for improvement, ranging from the reward function engineering to the expansion of the ML model.

Touristic water demand: analysis and validation of drinking water consumption data and hotel characterization in Benidorm (Spain)

The work analyses the tourist water demand in Benidorm, a sun-and-sand destination ranked fourth in Spain by number of visitors, where tourism competes with local residents, nature, agriculture and industrial sectors for scarce water resources. In particular, we have studied the correlation between the water consumption of 83 hotels in Benidorm and their characteristics and services which can impact water use. For this purpose, we have examined the water consumption billed, by the water utility company HIDRAQUA in the period January 2010 - October 2022, to the tourist structures in the municipality of Benidorm, and we have explored the hotels’ features, thanks to the collaboration of the tourism and hotels association HOSBEC. To give a better understanding and contextualization of our analysis we first described explained the of the complex water supply system and the efforts that have been made to reduce the threat posed by the peculiar climate conditions of the region. We saw that the water consumption per guest has slightly decreased in the recent years and that the tourist flux has increased: the global pandemic posed a stop to travels for more than one year, but now both the tourist flux and the tourist water consumption are reaching pre-pandemic level. We found that larger hotels, and in particular the ones opened all the year, that probably tend to offer more water-demanding service with respect to the seasonal ones, have higher water consumption per bed. From the analysis of the role of the different hotel characteristics over the water demand patterns, we found that water use increases with the increase in the hotel category and in the ratio between the surface area of the swimming pool and hotel size (number of beds). Other factors impacting the consumption are the presence of an on-site laundry for washing the hotel linen, the garden, and the implementation of environmental policies for water-saving.

Dna Extraction From Filter-paper Spots Of Vaginal Samples Collected After Sexual Violence.

To detect the presence of male DNA in vaginal samples collected from survivors of sexual violence and stored on filter paper. A pilot study was conducted to evaluate 10 vaginal samples spotted on sterile filter paper: 6 collected at random in April 2009 and 4 in October 2010. Time between sexual assault and sample collection was 4-48hours. After drying at room temperature, the samples were placed in a sterile envelope and stored for 2-3years until processing. DNA extraction was confirmed by polymerase chain reaction for human β-globin, and the presence of prostate-specific antigen (PSA) was quantified. The presence of the Y chromosome was detected using primers for sequences in the TSPY (Y7/Y8 and DYS14) and SRY genes. β-Globin was detected in all 10 samples, while 2 samples were positive for PSA. Half of the samples amplified the Y7/Y8 and DYS14 sequences of the TSPY gene and 30% amplified the SRY gene sequence of the Y chromosome. Four male samples and 1 female sample served as controls. Filter-paper spots stored for periods of up to 3years proved adequate for preserving genetic material from vaginal samples collected following sexual violence.

Changes In The Bacterial Community Of Soil From A Neutral Mine Drainage Channel.

Mine drainage is an important environmental disturbance that affects the chemical and biological components in natural resources. However, little is known about the effects of neutral mine drainage on the soil bacteria community. Here, a high-throughput 16S rDNA pyrosequencing approach was used to evaluate differences in composition, structure, and diversity of bacteria communities in samples from a neutral drainage channel, and soil next to the channel, at the Sossego copper mine in Brazil. Advanced statistical analyses were used to explore the relationships between the biological and chemical data. The results showed that the neutral mine drainage caused changes in the composition and structure of the microbial community, but not in its diversity. The Deinococcus/Thermus phylum, especially the Meiothermus genus, was in large part responsible for the differences between the communities, and was positively associated with the presence of copper and other heavy metals in the environmental samples. Other important parameters that influenced the bacterial diversity and composition were the elements potassium, sodium, nickel, and zinc, as well as pH. The findings contribute to the understanding of bacterial diversity in soils impacted by neutral mine drainage, and demonstrate that heavy metals play an important role in shaping the microbial population in mine environments.

An Isoflavone From Dipteryx Alata Vogel Is Active Against The In Vitro Neuromuscular Paralysis Of Bothrops Jararacussu Snake Venom And Bothropstoxin I, And Prevents Venom-induced Myonecrosis.

Snakebite is a neglected disease and serious health problem in Brazil, with most bites being caused by snakes of the genus Bothrops. Although serum therapy is the primary treatment for systemic envenomation, it is generally ineffective in neutralizing the local effects of these venoms. In this work, we examined the ability of 7,8,3'-trihydroxy-4'-methoxyisoflavone (TM), an isoflavone from Dipteryx alata, to neutralize the neurotoxicity (in mouse phrenic nerve-diaphragm preparations) and myotoxicity (assessed by light microscopy) of Bothrops jararacussu snake venom in vitro. The toxicity of TM was assessed using the Salmonella microsome assay (Ames test). Incubation with TM alone (200 μg/mL) did not alter the muscle twitch tension whereas incubation with venom (40 μg/mL) caused irreversible paralysis. Preincubation of TM (200 μg/mL) with venom attenuated the venom-induced neuromuscular blockade by 84% ± 5% (mean ± SEM; n = 4). The neuromuscular blockade caused by bothropstoxin-I (BthTX-I), the major myotoxic PLA2 of this venom, was also attenuated by TM. Histological analysis of diaphragm muscle incubated with TM showed that most fibers were preserved (only 9.2% ± 1.7% were damaged; n = 4) compared to venom alone (50.3% ± 5.4% of fibers damaged; n = 3), and preincubation of TM with venom significantly attenuated the venom-induced damage (only 17% ± 3.4% of fibers damaged; n = 3; p < 0.05 compared to venom alone). TM showed no mutagenicity in the Ames test using Salmonella strains TA98 and TA97a with (+S9) and without (-S9) metabolic activation. These findings indicate that TM is a potentially useful compound for antagonizing the neuromuscular effects (neurotoxicity and myotoxicity) of B. jararacussu venom.

An Asp49 Phospholipase A2 From Snake Venom Induces Cyclooxygenase-2 Expression And Prostaglandin E2 Production Via Activation Of Nf-κb, P38mapk, And Pkc In Macrophages.

Phospholipases A2 (PLA2) are key enzymes for production of lipid mediators. We previously demonstrated that a snake venom sPLA2 named MT-III leads to prostaglandin (PG)E2 biosynthesis in macrophages by inducing the expression of cyclooxygenase-2 (COX-2). Herein, we explored the molecular mechanisms and signaling pathways leading to these MT-III-induced effects. Results demonstrated that MT-III induced activation of the transcription factor NF-κB in isolated macrophages. By using NF-κB selective inhibitors, the involvement of this factor in MT-III-induced COX-2 expression and PGE2 production was demonstrated. Moreover, MT-III-induced COX-2 protein expression and PGE2 release were attenuated by pretreatment of macrophages with SB202190, and Ly294002, and H-7-dihydro compounds, indicating the involvement of p38MAPK, PI3K, and PKC pathways, respectively. Consistent with this, MT-III triggered early phosphorylation of p38MAPK, PI3K, and PKC. Furthermore, SB202190, H-7-dihydro, but not Ly294002 treatment, abrogated activation of NF-κB induced by MT-III. Altogether, these results show for the first time that the induction of COX-2 protein expression and PGE2 release, which occur via NF-κB activation induced by the sPLA2-MT-III in macrophages, are modulated by p38MAPK and PKC, but not by PI3K signaling proteins.

The Intrahippocampal Infusion Of Crotamine From Crotalus Durissus Terrificus Venom Enhances Memory Persistence In Rats.

Previous research has shown that crotamine, a toxin isolated from the venom of Crotalus durissus terrificus, induces the release of acetylcholine and dopamine in the central nervous system of rats. Particularly, these neurotransmitters are important modulators of memory processes. Therefore, in this study we investigated the effects of crotamine infusion on persistence of memory in rats. We verified that the intrahippocampal infusion of crotamine (1 μg/μl; 1 μl/side) improved the persistence of object recognition and aversive memory. By other side, the intrahippocampal infusion of the toxin did not alter locomotor and exploratory activities, anxiety or pain threshold. These results demonstrate a future prospect of using crotamine as potential pharmacological tool to treat diseases involving memory impairment, although it is still necessary more researches to better elucidate the crotamine effects on hippocampus and memory.

The Burden Of Eclampsia: Results From A Multicenter Study On Surveillance Of Severe Maternal Morbidity In Brazil.

Maternal mortality (MM) is a core indicator of disparities in women's rights. The study of Near Miss cases is strategic to identifying the breakdowns in obstetrical care. In absolute numbers, both MM and occurrence of eclampsia are rare events. We aim to assess the obstetric care indicators and main predictors for severe maternal outcome from eclampsia (SMO: maternal death plus maternal near miss). Secondary analysis of a multicenter, cross-sectional study, including 27 centers from all geographic regions of Brazil, from 2009 to 2010. 426 cases of eclampsia were identified and classified according to the outcomes: SMO and non-SMO. We classified facilities as coming from low- and high-income regions and calculated the WHO's obstetric health indicators. SPSS and Stata softwares were used to calculate the prevalence ratios (PR) and respective 95% confidence interval (CI) to assess maternal characteristics, clinical and obstetrical history, and access to health services as predictors for SMO, subsequently correlating them with the corresponding perinatal outcomes, also applying multiple regression analysis (adjusted for cluster effect). Prevalence of and mortality indexes for eclampsia in higher and lower income regions were 0.2%/0.8% and 8.1%/22%, respectively. Difficulties in access to health care showed that ICU admission (adjPR 3.61; 95% CI 1.77-7.35) and inadequate monitoring (adjPR 2.31; 95% CI 1.48-3.59) were associated with SMO. Morbidity and mortality associated with eclampsia were high in Brazil, especially in lower income regions. Promoting quality maternal health care and improving the availability of obstetric emergency care are essential actions to relieve the burden of eclampsia.

Geochemistry Of Acid Mine Drainage From A Coal Mining Area And Processes Controlling Metal Attenuation In Stream Waters, Southern Brazil.

Acid drainage influence on the water and sediment quality was investigated in a coal mining area (southern Brazil). Mine drainage showed pH between 3.2 and 4.6 and elevated concentrations of sulfate, As and metals, of which, Fe, Mn and Zn exceeded the limits for the emission of effluents stated in the Brazilian legislation. Arsenic also exceeded the limit, but only slightly. Groundwater monitoring wells from active mines and tailings piles showed pH interval and chemical concentrations similar to those of mine drainage. However, the river and ground water samples of municipal public water supplies revealed a pH range from 7.2 to 7.5 and low chemical concentrations, although Cd concentration slightly exceeded the limit adopted by Brazilian legislation for groundwater. In general, surface waters showed large pH range (6 to 10.8), and changes caused by acid drainage in the chemical composition of these waters were not very significant. Locally, acid drainage seemed to have dissolved carbonate rocks present in the local stratigraphic sequence, attenuating the dispersion of metals and As. Stream sediments presented anomalies of these elements, which were strongly dependent on the proximity of tailings piles and abandoned mines. We found that precipitation processes in sediments and the dilution of dissolved phases were responsible for the attenuation of the concentrations of the metals and As in the acid drainage and river water mixing zone. In general, a larger influence of mining activities on the chemical composition of the surface waters and sediments was observed when enrichment factors in relation to regional background levels were used.

Cytogenetic And Morphologic Approaches Of Hybrids From Experimental Crosses Between Triatoma Lenti Sherlock & Serafim, 1967 And T. Sherlocki Papa Et Al., 2002 (hemiptera: Reduviidae).

The reproductive capacity between Triatoma lenti and Triatoma sherlocki was observed in order to verify the fertility and viability of the offspring. Cytogenetic, morphological and morphometric approaches were used to analyze the differences that were inherited. Experimental crosses were performed in both directions. The fertility rate of the eggs in crosses involving T. sherlocki females was 65% and 90% in F1 and F2 offspring, respectively. In reciprocal crosses, it was 7% and 25% in F1 and F2 offspring, respectively. The cytogenetic analyses of the male meiotic process of the hybrids were performed using lacto-acetic orcein, C-banding and Feulgen techniques. The male F1 offspring presented normal chromosome behavior, a finding that was similar to those reported in parental species. However, cytogenetic analysis of F2 offspring showed errors in chromosome pairing. This post-zygotic isolation, which prevents hybrids in nature, may represent the collapse of the hybrid. This phenomenon is due to a genetic dysregulation that occurs in the chromosomes of F1. The results were similar in the hybrids from both crosses. Morphological features, such as color and size of connexive and the presence of red-orange rings on the femora, were similar to T. sherlocki, while wins size was similar to T. lenti in F1 offspring. The eggshells showed characteristics that were similar to species of origin, whereas the median process of the pygophore resulted in intermediate characteristics in the F1 and a segregating pattern in F2 offspring. Geometric morphometric techniques used on the wings showed that both F1 and F2 offspring were similar to T. lenti. These studies on the reproductive capacity between T. lenti and T. sherlocki confirm that both species are evolutionarily closed; hence, they are included in the brasiliensis subcomplex. The extremely reduced fertility observed in the F2 hybrids confirmed the specific status of the species that were analyzed.