Unsupervised reinforcement learning via state entropy maximization

Reinforcement Learning (RL) provides a powerful framework to address sequential decision-making problems in which the transition dynamics is unknown or too complex to be represented. The RL approach is based on speculating what is the best decision to make given sample estimates obtained from previous interactions, a recipe that led to several breakthroughs in various domains, ranging from game playing to robotics. Despite their success, current RL methods hardly generalize from one task to another, and achieving the kind of generalization obtained through unsupervised pre-training in non-sequential problems seems unthinkable. Unsupervised RL has recently emerged as a way to improve generalization of RL methods. Just as its non-sequential counterpart, the unsupervised RL framework comprises two phases: An unsupervised pre-training phase, in which the agent interacts with the environment without external feedback, and a supervised fine-tuning phase, in which the agent aims to efficiently solve a task in the same environment by exploiting the knowledge acquired during pre-training. In this thesis, we study unsupervised RL via state entropy maximization, in which the agent makes use of the unsupervised interactions to pre-train a policy that maximizes the entropy of its induced state distribution. First, we provide a theoretical characterization of the learning problem by considering a convex RL formulation that subsumes state entropy maximization. Our analysis shows that maximizing the state entropy in finite trials is inherently harder than RL. Then, we study the state entropy maximization problem from an optimization perspective. Especially, we show that the primal formulation of the corresponding optimization problem can be (approximately) addressed through tractable linear programs. Finally, we provide the first practical methodologies for state entropy maximization in complex domains, both when the pre-training takes place in a single environment as well as multiple environments.

Integrated communication and learning towards autonomous mobile radio networks

The integration of distributed and ubiquitous intelligence has emerged over the last years as the mainspring of transformative advancements in mobile radio networks. As we approach the era of “mobile for intelligence”, next-generation wireless networks are poised to undergo significant and profound changes. Notably, the overarching challenge that lies ahead is the development and implementation of integrated communication and learning mechanisms that will enable the realization of autonomous mobile radio networks. The ultimate pursuit of eliminating human-in-the-loop constitutes an ambitious challenge, necessitating a meticulous delineation of the fundamental characteristics that artificial intelligence (AI) should possess to effectively achieve this objective. This challenge represents a paradigm shift in the design, deployment, and operation of wireless networks, where conventional, static configurations give way to dynamic, adaptive, and AI-native systems capable of self-optimization, self-sustainment, and learning. This thesis aims to provide a comprehensive exploration of the fundamental principles and practical approaches required to create autonomous mobile radio networks that seamlessly integrate communication and learning components. The first chapter of this thesis introduces the notion of Predictive Quality of Service (PQoS) and adaptive optimization and expands upon the challenge to achieve adaptable, reliable, and robust network performance in dynamic and ever-changing environments. The subsequent chapter delves into the revolutionary role of generative AI in shaping next-generation autonomous networks. This chapter emphasizes achieving trustworthy uncertainty-aware generation processes with the use of approximate Bayesian methods and aims to show how generative AI can improve generalization while reducing data communication costs. Finally, the thesis embarks on the topic of distributed learning over wireless networks. Distributed learning and its declinations, including multi-agent reinforcement learning systems and federated learning, have the potential to meet the scalability demands of modern data-driven applications, enabling efficient and collaborative model training across dynamic scenarios while ensuring data privacy and reducing communication overhead.

Work environments implementation for genomic reporting and analytics

A global italian pharmaceutical company has to provide two work environments that favor different needs. The environments will allow to develop solutions in a controlled, secure and at the same time in an independent manner on a state-of-the-art enterprise cloud platform. The need of developing two different environments is dictated by the needs of the working units. Indeed, the first environment is designed to facilitate the creation of application related to genomics, therefore, designed more for data-scientists. This environment is capable of consuming, producing, retrieving and incorporating data, furthermore, will support the most used programming languages for genomic applications (e.g., Python, R). The proposal was to obtain a pool of ready-togo Virtual Machines with different architectures to provide best performance based on the job that needs to be carried out. The second environment has more of a traditional trait, to obtain, via ETL (Extract-Transform-Load) process, a global datamodel, resembling a classical relational structure. It will provide major BI operations (e.g., analytics, performance measure, reports, etc.) that can be leveraged both for application analysis or for internal usage. Since, both architectures will maintain large amounts of data regarding not only pharmaceutical informations but also internal company informations, it would be possible to digest the data by reporting/ analytics tools and also apply data-mining, machine learning technologies to exploit intrinsic informations. The thesis work will introduce, proposals, implementations, descriptions of used technologies/platforms and future works of the above discussed environments.

Clustering Of Water Bodies In Unpolluted And Polluted Environments Based On Escherichia Coli Phylogroup Abundance Using A Simple Interaction Database.

Different types of water bodies, including lakes, streams, and coastal marine waters, are often susceptible to fecal contamination from a range of point and nonpoint sources, and have been evaluated using fecal indicator microorganisms. The most commonly used fecal indicator is Escherichia coli, but traditional cultivation methods do not allow discrimination of the source of pollution. The use of triplex PCR offers an approach that is fast and inexpensive, and here enabled the identification of phylogroups. The phylogenetic distribution of E. coli subgroups isolated from water samples revealed higher frequencies of subgroups A1 and B23 in rivers impacted by human pollution sources, while subgroups D1 and D2 were associated with pristine sites, and subgroup B1 with domesticated animal sources, suggesting their use as a first screening for pollution source identification. A simple classification is also proposed based on phylogenetic subgroup distribution using the w-clique metric, enabling differentiation of polluted and unpolluted sites.

Ecological Literacy And Beyond: Problem-based Learning For Future Professionals.

Ecological science contributes to solving a broad range of environmental problems. However, lack of ecological literacy in practice often limits application of this knowledge. In this paper, we highlight a critical but often overlooked demand on ecological literacy: to enable professionals of various careers to apply scientific knowledge when faced with environmental problems. Current university courses on ecology often fail to persuade students that ecological science provides important tools for environmental problem solving. We propose problem-based learning to improve the understanding of ecological science and its usefulness for real-world environmental issues that professionals in careers as diverse as engineering, public health, architecture, social sciences, or management will address. Courses should set clear learning objectives for cognitive skills they expect students to acquire. Thus, professionals in different fields will be enabled to improve environmental decision-making processes and to participate effectively in multidisciplinary work groups charged with tackling environmental issues.

Flicker perimetry in healthy subjects: influence of age and gender, learning effect and short-term fluctuation

PURPOSE: To determine the mean critical fusion frequency and the short-term fluctuation, to analyze the influence of age, gender, and the learning effect in healthy subjects undergoing flicker perimetry. METHODS: Study 1 - 95 healthy subjects underwent flicker perimetry once in one eye. Mean critical fusion frequency values were compared between genders, and the influence of age was evaluated using linear regression analysis. Study 2 - 20 healthy subjects underwent flicker perimetry 5 times in one eye. The first 3 sessions were separated by an interval of 1 to 30 days, whereas the last 3 sessions were performed within the same day. The first 3 sessions were used to investigate the presence of a learning effect, whereas the last 3 tests were used to calculate short-term fluctuation. RESULTS: Study 1 - Linear regression analysis demonstrated that mean global, foveal, central, and critical fusion frequency per quadrant significantly decreased with age (p<0.05).There were no statistically significant differences in mean critical fusion frequency values between males and females (p>0.05), with the exception of the central area and inferonasal quadrant (p=0.049 and p=0.011, respectively), where the values were lower in females. Study 2 - Mean global (p=0.014), central (p=0.008), and peripheral (p=0.03) critical fusion frequency were significantly lower in the first session compared to the second and third sessions. The mean global short-term fluctuation was 5.06±1.13 Hz, the mean interindividual and intraindividual variabilities were 11.2±2.8% and 6.4±1.5%, respectively. CONCLUSION: This study suggests that, in healthy subjects, critical fusion frequency decreases with age, that flicker perimetry is associated with a learning effect, and that a moderately high short-term fluctuation is expected.

Sensitivity and specificity of machine learning classifiers for glaucoma diagnosis using Spectral Domain OCT and standard automated perimetry

PURPOSE: To evaluate the sensitivity and specificity of machine learning classifiers (MLCs) for glaucoma diagnosis using Spectral Domain OCT (SD-OCT) and standard automated perimetry (SAP). METHODS: Observational cross-sectional study. Sixty two glaucoma patients and 48 healthy individuals were included. All patients underwent a complete ophthalmologic examination, achromatic standard automated perimetry (SAP) and retinal nerve fiber layer (RNFL) imaging with SD-OCT (Cirrus HD-OCT; Carl Zeiss Meditec Inc., Dublin, California). Receiver operating characteristic (ROC) curves were obtained for all SD-OCT parameters and global indices of SAP. Subsequently, the following MLCs were tested using parameters from the SD-OCT and SAP: Bagging (BAG), Naive-Bayes (NB), Multilayer Perceptron (MLP), Radial Basis Function (RBF), Random Forest (RAN), Ensemble Selection (ENS), Classification Tree (CTREE), Ada Boost M1(ADA),Support Vector Machine Linear (SVML) and Support Vector Machine Gaussian (SVMG). Areas under the receiver operating characteristic curves (aROC) obtained for isolated SAP and OCT parameters were compared with MLCs using OCT+SAP data. RESULTS: Combining OCT and SAP data, MLCs' aROCs varied from 0.777(CTREE) to 0.946 (RAN).The best OCT+SAP aROC obtained with RAN (0.946) was significantly larger the best single OCT parameter (p<0.05), but was not significantly different from the aROC obtained with the best single SAP parameter (p=0.19). CONCLUSION: Machine learning classifiers trained on OCT and SAP data can successfully discriminate between healthy and glaucomatous eyes. The combination of OCT and SAP measurements improved the diagnostic accuracy compared with OCT data alone.

Pedagogia do esporte : pressupostos para uma teoria da avaliação da aprendizagem = Sport pedagogy: presuppositions for a learning assessment theory

Universidade Estadual de Campinas . Faculdade de Educação Física

Pedagogia do esporte : o livro didatico como um mediador no processo de ensino e aprendizagem dos jogos esportivos coletivos

Universidade Estadual de Campinas . Faculdade de Educação Física

Isolation of Pseudomonas aeruginosa strains from dental office environments and units in Barretos, state of São Paulo, Brazil, and analysis of their susceptibility to antimicrobial drugs

A wide variety of opportunistic pathogens has been detected in the tubing supplying water to odontological equipment, in special in the biofilm lining of these tubes. Among these pathogens, Pseudomonas aeruginosa, one of the leading causes of nosocomial infections, is frequently found in water lines supplying dental units. In the present work, 160 samples of water, and 200 fomite samples from forty dental units were collected in the city of Barretos, State of São Paulo, Brazil and evaluated between January and July, 2005. Seventy-six P. aeruginosa strains, isolated from the dental environment (5 strains) and water system (71 strains), were tested for susceptibility to six antimicrobial drugs most frequently used against P. aeruginosa infections. Susceptibility to ciprofloxacin, followed by meropenem was the predominant profile. The need for effective means of reducing the microbial burden within dental unit water lines is emphasized, and the risk of exposure and cross-infection in dental practice, in special when caused by opportunistic pathogens like P. aeruginosa, are highlighted.

Changes in ruminal fermentation and mineral serum level in animals kept in high temperature environments

In order to evaluate the effect of environmental temperature on ruminal fermentation and on mineral levels of growing ruminants, it was used 12 male calves (initial average weight 82.9 ± 7.7 kg, 100 days of age), were employed in a randomized block design (by weight) experiment, with repeated weight measurement and two environmental temperatures: thermoneutral (24ºC) and heat-stressed (33ºC), during 38 days. The animals exposed to 33ºC presented lower dry matter ingestion, lower T3 (triiodothyronine) serum level, higher ammoniacal nitrogen (NH3-N) level in the rumen liquid, and higher rectal and body temperatures during all the experimental period when compared to the animals kept in thermoneutral environment (24ºC). The animals kept under heat stress environment (33ºC) presented higher calcium serum level, which was the highest on 31st day and the lowest on the 38th day of the experiment; phosphorus level was the lowest during all the experimental period; sodium level was lower on the 17th, 31st and 38th experimental days. Potassium and zinc levels were lower after 24 days; copper level was lower until the 24th day; magnesium level was higher until the 17th day, if compared to the ones from the animals kept in thermoneutral environment (24ºC). The heat-stressed animals presented higher levels of ammoniacal nitrogen in the ruminal liquid and a decrease in the phosphorus, sodium, potassium and zinc serum levels. These results show the necessity of changes on feed management to ruminants in temperatures over the thermal comfort limits so that performance loss is decreased.

Learning processes and the neural analysis of conditioning

Classical and operant conditioning principles, such as the behavioral discrepancy-derived assumption that reinforcement always selects antecedent stimulus and response relations, have been studied at the neural level, mainly by observing the strengthening of neuronal responses or synaptic connections. A review of the literature on the neural basis of behavior provided extensive scientific data that indicate a synthesis between the two conditioning processes based mainly on stimulus control in learning tasks. The resulting analysis revealed the following aspects. Dopamine acts as a behavioral discrepancy signal in the midbrain pathway of positive reinforcement, leading toward the nucleus accumbens. Dopamine modulates both types of conditioning in the Aplysia mollusk and in mammals. In vivo and in vitro mollusk preparations show convergence of both types of conditioning in the same motor neuron. Frontal cortical neurons are involved in behavioral discrimination in reversal and extinction procedures, and these neurons preferentially deliver glutamate through conditioned stimulus or discriminative stimulus pathways. Discriminative neural responses can reliably precede operant movements and can also be common to stimuli that share complex symbolic relations. The present article discusses convergent and divergent points between conditioning paradigms at the neural level of analysis to advance our knowledge on reinforcement.

The Process of Simulation Authoring/Creation by Chemistry Teachers: A Service Learning Experience

Two case studies are presented to describe the process of public school teachers authoring and creating chemistry simulations. They are part of the Virtual Didactic Laboratory for Chemistry, a project developed by the School of the Future of the University of Sao Paulo. the documental analysis of the material produced by two groups of teachers reflects different selection process for both themes and problem-situations when creating simulations. The study demonstrates the potential for chemistry learning with an approach that takes students' everyday lives into account and is based on collaborative work among teachers and researches. Also, from the teachers' perspectives, the possibilities of interaction that a simulation offers for classroom activities are considered.

Cooperation, interaction and learning in local production systems: Evidence in Brazilian firms

In a local production system (LPS), besides external economies, the interaction, cooperation, and learning are indicated by the literature as complementary ways of enhancing the LPS's competitiveness and gains. In Brazil, the greater part of LPSs, mostly composed by small enterprises, displays incipient relationships and low levels of interaction and cooperation among their actors. The size of the participating enterprises itself for specificities that engender organizational constraints, which, in turn, can have a considerable impact on their relationships and learning dynamics. For that reason, it is the purpose of this article to present an analysis of interaction, cooperation, and learning relationships among several types of actors pertaining to an LPS in the farming equipment and machinery sector, bearing in mind the specificities of small enterprises. To this end, the fieldwork carried out in this study aimed at: (i) investigating external and internal knowledge sources conducive to learning and (ii) identifying and analyzing motivating and inhibiting factors related to specificities of small enterprises in order to bring the LPS members closer together and increase their cooperation and interaction. Empirical evidence shows that internal aspects of the enterprises, related to management and infrastructure, can have a strong bearing on their joint actions, interaction and learning processes.

Evaluation of gastrointestinal motility in awake rats: a learning exercise for undergraduate biomedical students

Souza MA, Souza MH, Palheta RC Jr, Cruz PR, Medeiros BA, Rola FH, Magalhaes PJ, Troncon LE, Santos AA. Evaluation of gastrointestinal motility in awake rats: a learning exercise for undergraduate biomedical students. Adv Physiol Educ 33: 343-348, 2009; doi: 10.1152/advan.90176.2008.-Current medical curricula devote scarce time for practical activities on digestive physiology, despite frequent misconceptions about dyspepsia and dysmotility phenomena. Thus, we designed a hands-on activity followed by a small-group discussion on gut motility. Male awake rats were randomly submitted to insulin, control, or hypertonic protocols. Insulin and control rats were gavage fed with 5% glucose solution, whereas hypertonic-fed rats were gavage fed with 50% glucose solution. Insulin treatment was performed 30 min before a meal. All meals (1.5 ml) contained an equal mass of phenol red dye. After 10, 15, or 20 min of meal gavage, rats were euthanized. Each subset consisted of six to eight rats. Dye recovery in the stomach and proximal, middle, and distal small intestine was measured by spectrophotometry, a safe and reliable method that can be performed by minimally trained students. In a separate group of rats, we used the same protocols except that the test meal contained (99m)Tc as a marker. Compared with control, the hypertonic meal delayed gastric emptying and gastrointestinal transit, whereas insulinic hypoglycemia accelerated them. The session helped engage our undergraduate students in observing and analyzing gut motor behavior. In conclusion, the fractional dye retention test can be used as a teaching tool to strengthen the understanding of basic physiopathological features of gastrointestinal motility.