Unsupervised reinforcement learning via state entropy maximization

Reinforcement Learning (RL) provides a powerful framework to address sequential decision-making problems in which the transition dynamics is unknown or too complex to be represented. The RL approach is based on speculating what is the best decision to make given sample estimates obtained from previous interactions, a recipe that led to several breakthroughs in various domains, ranging from game playing to robotics. Despite their success, current RL methods hardly generalize from one task to another, and achieving the kind of generalization obtained through unsupervised pre-training in non-sequential problems seems unthinkable. Unsupervised RL has recently emerged as a way to improve generalization of RL methods. Just as its non-sequential counterpart, the unsupervised RL framework comprises two phases: An unsupervised pre-training phase, in which the agent interacts with the environment without external feedback, and a supervised fine-tuning phase, in which the agent aims to efficiently solve a task in the same environment by exploiting the knowledge acquired during pre-training. In this thesis, we study unsupervised RL via state entropy maximization, in which the agent makes use of the unsupervised interactions to pre-train a policy that maximizes the entropy of its induced state distribution. First, we provide a theoretical characterization of the learning problem by considering a convex RL formulation that subsumes state entropy maximization. Our analysis shows that maximizing the state entropy in finite trials is inherently harder than RL. Then, we study the state entropy maximization problem from an optimization perspective. Especially, we show that the primal formulation of the corresponding optimization problem can be (approximately) addressed through tractable linear programs. Finally, we provide the first practical methodologies for state entropy maximization in complex domains, both when the pre-training takes place in a single environment as well as multiple environments.

Novel methods of state estimation and parameter identification for high-performance motorcycles

This research activity aims at providing a reliable estimation of particular state variables or parameters concerning the dynamics and performance optimization of a MotoGP-class motorcycle, integrating the classical model-based approach with new methodologies involving artificial intelligence. The first topic of the research focuses on the estimation of the thermal behavior of the MotoGP carbon braking system. Numerical tools are developed to assess the instantaneous surface temperature distribution in the motorcycle's front brake discs. Within this application other important brake parameters are identified using Kalman filters, such as the disc convection coefficient and the power distribution in the disc-pads contact region. Subsequently, a physical model of the brake is built to estimate the instantaneous braking torque. However, the results obtained with this approach are highly limited by the knowledge of the friction coefficient (μ) between the disc rotor and the pads. Since the value of μ is a highly nonlinear function of many variables (namely temperature, pressure and angular velocity of the disc), an analytical model for the friction coefficient estimation appears impractical to establish. To overcome this challenge, an innovative hybrid solution is implemented, combining the benefit of artificial intelligence (AI) with classical model-based approach. Indeed, the disc temperature estimated through the thermal model previously implemented is processed by a machine learning algorithm that outputs the actual value of the friction coefficient thus improving the braking torque computation performed by the physical model of the brake. Finally, the last topic of this research activity regards the development of an AI algorithm to estimate the current sideslip angle of the motorcycle's front tire. While a single-track motorcycle kinematic model and IMU accelerometer signals theoretically enable sideslip calculation, the presence of accelerometer noise leads to a significant drift over time. To address this issue, a long short-term memory (LSTM) network is implemented.

Effect Of Platform Connection And Abutment Material On Stress Distribution In Single Anterior Implant-supported Restorations: A Nonlinear 3-dimensional Finite Element Analysis.

Although various abutment connections and materials have recently been introduced, insufficient data exist regarding the effect of stress distribution on their mechanical performance. The purpose of this study was to investigate the effect of different abutment materials and platform connections on stress distribution in single anterior implant-supported restorations with the finite element method. Nine experimental groups were modeled from the combination of 3 platform connections (external hexagon, internal hexagon, and Morse tapered) and 3 abutment materials (titanium, zirconia, and hybrid) as follows: external hexagon-titanium, external hexagon-zirconia, external hexagon-hybrid, internal hexagon-titanium, internal hexagon-zirconia, internal hexagon-hybrid, Morse tapered-titanium, Morse tapered-zirconia, and Morse tapered-hybrid. Finite element models consisted of a 4×13-mm implant, anatomic abutment, and lithium disilicate central incisor crown cemented over the abutment. The 49 N occlusal loading was applied in 6 steps to simulate the incisal guidance. Equivalent von Mises stress (σvM) was used for both the qualitative and quantitative evaluation of the implant and abutment in all the groups and the maximum (σmax) and minimum (σmin) principal stresses for the numerical comparison of the zirconia parts. The highest abutment σvM occurred in the Morse-tapered groups and the lowest in the external hexagon-hybrid, internal hexagon-titanium, and internal hexagon-hybrid groups. The σmax and σmin values were lower in the hybrid groups than in the zirconia groups. The stress distribution concentrated in the abutment-implant interface in all the groups, regardless of the platform connection or abutment material. The platform connection influenced the stress on abutments more than the abutment material. The stress values for implants were similar among different platform connections, but greater stress concentrations were observed in internal connections.

What Is The Best Indication For Single-incision Ophira Mini Sling? Insights From A 2-year Follow-up International Multicentric Study.

The Ophira Mini Sling System involves anchoring a midurethral, low-tension tape to the obturator internus muscles bilaterally at the level of the tendinous arc. Success rates in different subsets of patients are still to be defined. This work aims to identify which factors influence the 2-year outcomes of this treatment. Analysis was based on data from a multicenter study. Endpoints for analysis included objective measurements: 1-h pad-weight (PWT), and cough stress test (CST), and questionnaires: International Consultation on Incontinence Questionnaire-Short Form (ICIQ-SF) and Urinary Distress Inventory (UDI)-6. A logistic regression analysis evaluated possible risk factors for failure. In all, 124 female patients with stress urinary incontinence (SUI) underwent treatment with the Ophira procedure. All patients completed 1 year of follow-up, and 95 complied with the 2-year evaluation. Longitudinal analysis showed no significant differences between results at 1 and 2 years. The 2-year overall objective results were 81 (85.3%) patients dry, six (6.3%) improved, and eight (8.4%) incontinent. A multivariate analysis revealed that previous anti-incontinence surgery was the only factor that significantly influenced surgical outcomes. Two years after treatment, women with previous failed surgeries had an odds ratio (OR) for treatment failure (based on PWT) of 4.0 [95% confidence interval (CI) 1.02-15.57). The Ophira procedure is an effective option for SUI treatment, with durable good results. Previous surgeries were identified as the only significant risk factor, though previously operated patients showed an acceptable success rate.

[impact Variables On The Decline In Infant Mortality In The State Of São Paulo, Brazil: 1998-2008].

This is an ecological, analytical and retrospective study comprising the 645 municipalities in the State of São Paulo, the scope of which was to determine the relationship between socioeconomic, demographic variables and the model of care in relation to infant mortality rates in the period from 1998 to 2008. The ratio of average annual change for each indicator per stratum coverage was calculated. Infant mortality was analyzed according to the model for repeated measures over time, adjusted for the following correction variables: the city's population, proportion of Family Health Programs (PSFs) deployed, proportion of Growth Acceleration Programs (PACs) deployed, per capita GDP and SPSRI (São Paulo social responsibility index). The analysis was performed by generalized linear models, considering the gamma distribution. Multiple comparisons were performed with the likelihood ratio with chi-square approximate distribution, considering a significance level of 5%. There was a decrease in infant mortality over the years (p < 0.05), with no significant difference from 2004 to 2008 (p > 0.05). The proportion of PSFs deployed (p < 0.0001) and per capita GDP (p < 0.0001) were significant in the model. The decline of infant mortality in this period was influenced by the growth of per capita GDP and PSFs.

Nebivolol Reduces Central Blood Pressure In Stage I Hypertensive Patients: Experimental Single Cohort Study.

Assessment of central blood pressure (BP) has grown substantially over recent years because evidence has shown that central BP is more relevant to cardiovascular outcomes than peripheral BP. Thus, different classes of antihypertensive drugs have different effects on central BP despite similar reductions in brachial BP. The aim of this study was to investigate the effect of nebivolol, a β-blocker with vasodilator properties, on the biochemical and hemodynamic parameters of hypertensive patients. Experimental single cohort study conducted in the outpatient clinic of a university hospital. Twenty-six patients were recruited. All of them underwent biochemical and hemodynamic evaluation (BP, heart rate (HR), central BP and augmentation index) before and after 3 months of using nebivolol. 88.5% of the patients were male; their mean age was 49.7 ± 9.3 years and most of them were overweight (29.6 ± 3.1 kg/m2) with large abdominal waist (102.1 ± 7.2 cm). There were significant decreases in peripheral systolic BP (P = 0.0020), diastolic BP (P = 0.0049), HR (P < 0.0001) and central BP (129.9 ± 12.3 versus 122.3 ± 10.3 mmHg; P = 0.0083) after treatment, in comparison with the baseline values. There was no statistical difference in the augmentation index or in the biochemical parameters, from before to after the treatment. Nebivolol use seems to be associated with significant reduction of central BP in stage I hypertensive patients, in addition to reductions in brachial systolic and diastolic BP.

Icatibant, An Inhibitor Of Bradykinin Receptor 2, For Hereditary Angioedema Attacks: Prospective Experimental Single-cohort Study.

Hereditary angioedema (HAE) with C1 inhibitor deficiency manifests as recurrent episodes of edema involving the skin, upper respiratory tract and gastrointestinal tract. It can be lethal due to asphyxia. The aim here was to evaluate the response to therapy for these attacks using icatibant, an inhibitor of the bradykinin receptor, which was recently introduced into Brazil. Prospective experimental single-cohort study on the efficacy and safety of icatibant for HAE patients. Patients with a confirmed HAE diagnosis were enrolled according to symptoms and regardless of the time since onset of the attack. Icatibant was administered in accordance with the protocol that has been approved in Brazil. Symptom severity was assessed continuously and adverse events were monitored. 24 attacks in 20 HAE patients were treated (female/male 19:1; 19-55 years; median 29 years of age). The symptoms were: subcutaneous edema (22/24); abdominal pain (15/24) and upper airway obstruction (10/24). The time taken until onset of relief was: 5-10 minutes (5/24; 20.8%); 10-20 (5/24; 20.8%); 20-30 (8/24; 33.4%); 30-60 (5/24; 20.8%); and 2 hours (1/24; 4.3%). The time taken for complete resolution of symptoms ranged from 4.3 to 33.4 hours. Adverse effects were only reported at injection sites. Mild to moderate erythema and/or feelings of burning were reported by 15/24 patients, itching by 3 and no adverse effects in 6. HAE type I patients who received icatibant responded promptly; most achieved improved symptom severity within 30 minutes. Local adverse events occurred in 75% of the patients.

[diet Quality And Associated Factors Among The Elderly: A Population-based Study In Campinas, São Paulo State, Brazil].

The aim of this study was to assess the quality of diet among the elderly and associations with socio-demographic variables, health-related behaviors, and diseases. A population-based cross-sectional study was conducted in a representative sample of 1,509 elderly participants in a health survey in Campinas, São Paulo State, Brazil. Food quality was assessed using the Revised Diet Quality Index (DQI-R). Mean index scores were estimated and a multiple regression model was employed for the adjusted analyses. The highest diet quality scores were associated with age 80 years or older, Evangelical religion, diabetes mellitus, and physical activity, while the lowest scores were associated with home environments shared with three or more people, smoking, and consumption of soft drinks and alcoholic beverages. The findings emphasize a general need for diet quality improvements in the elderly, specifically in subgroups with unhealthy behaviors, who should be targeted with comprehensive strategies.

Effects Of Partial Inhibition Of Respiratory Complex I On H2o 2 Production By Isolated Brain Mitochondria In Different Respiratory States.

The aim of this work was to characterize the effects of partial inhibition of respiratory complex I by rotenone on H2O2 production by isolated rat brain mitochondria in different respiratory states. Flow cytometric analysis of membrane potential in isolated mitochondria indicated that rotenone leads to uniform respiratory inhibition when added to a suspension of mitochondria. When mitochondria were incubated in the presence of a low concentration of rotenone (10 nm) and NADH-linked substrates, oxygen consumption was reduced from 45.9 ± 1.0 to 26.4 ± 2.6 nmol O2 mg(-1) min(-1) and from 7.8 ± 0.3 to 6.3 ± 0.3 nmol O2 mg(-1) min(-1) in respiratory states 3 (ADP-stimulated respiration) and 4 (resting respiration), respectively. Under these conditions, mitochondrial H2O2 production was stimulated from 12.2 ± 1.1 to 21.0 ± 1.2 pmol H2O2 mg(-1) min(-1) and 56.5 ± 4.7 to 95.0 ± 11.1 pmol H2O2 mg(-1) min(-1) in respiratory states 3 and 4, respectively. Similar results were observed when comparing mitochondrial preparations enriched with synaptic or nonsynaptic mitochondria or when 1-methyl-4-phenylpyridinium ion (MPP(+)) was used as a respiratory complex I inhibitor. Rotenone-stimulated H2O2 production in respiratory states 3 and 4 was associated with a high reduction state of endogenous nicotinamide nucleotides. In succinate-supported mitochondrial respiration, where most of the mitochondrial H2O2 production relies on electron backflow from complex II to complex I, low rotenone concentrations inhibited H2O2 production. Rotenone had no effect on mitochondrial elimination of micromolar concentrations of H2O2. The present results support the conclusion that partial complex I inhibition may result in mitochondrial energy crisis and oxidative stress, the former being predominant under oxidative phosphorylation and the latter under resting respiration conditions.

Compact Ag@fe3o4 Core-shell Nanoparticles By Means Of Single-step Thermal Decomposition Reaction.

A temperature pause introduced in a simple single-step thermal decomposition of iron, with the presence of silver seeds formed in the same reaction mixture, gives rise to novel compact heterostructures: brick-like Ag@Fe3O4 core-shell nanoparticles. This novel method is relatively easy to implement, and could contribute to overcome the challenge of obtaining a multifunctional heteroparticle in which a noble metal is surrounded by magnetite. Structural analyses of the samples show 4 nm silver nanoparticles wrapped within compact cubic external structures of Fe oxide, with curious rectangular shape. The magnetic properties indicate a near superparamagnetic like behavior with a weak hysteresis at room temperature. The value of the anisotropy involved makes these particles candidates to potential applications in nanomedicine.

Rotenone Exerts Similar Stimulatory Effects On H2o2 Production By Isolated Brain Mitochondria From Young-adult And Old Rats.

Chronic and systemic treatment of rodents with rotenone, a classical inhibitor of mitochondrial respiratory complex I, results in neurochemical, behavioral, and neuropathological features of Parkinson's disease. The aim of the present study was to evaluate whether brain mitochondria from old rats (24 months old) would be more susceptible to rotenone-induced inhibition of oxygen consumption and increased generation of H2O2 than mitochondria from young-adult rats (3-4 months old). Isolated brain mitochondria were incubated in the presence of different rotenone concentrations (5, 10, and 100nM), and oxygen consumption and H2O2 production were measured during respiratory states 3 (ADP-stimulated respiration) and 4 (resting respiration). Respiratory state 3 and citrate synthase activity were significantly lower in mitochondria from old rats. Mitochondria from young-adult and old rats showed similar sensitivity to rotenone-induced inhibition of oxygen consumption. Similarly, H2O2 production rates by both types of mitochondria were dose-dependently stimulated to the same extent by increasing concentrations of rotenone. We conclude that rotenone exerts similar effects on oxygen consumption and H2O2 production by isolated brain mitochondria from young-adult and old rats. Therefore, aging does not increase the mitochondrial H2O2 generation in response to complex I inhibition.

Burnout Syndrome In Nursing Assistants Of A Public Hospital In The State Of São Paulo.

The burnout syndrome is a psychosocial phenomenon that arises as a response to chronic interpersonal stressors present at work. There are many aspects that make nursing assistants vulnerable to chronic stress situations that may lead to burnout, highlighting the low degree of autonomy in the healthcare staff and spending more in direct contact with patients. To assess the prevalence of the burnout syndrome in nursing assistants in a public hospital, as well as its association with socio-demographic and professional variables. A socio-demographic and professional questionnaire and the Maslach Burnout Inventory (MBI-SS) were applied to 534 nursing assistants. The prevalence of burnout syndrome among nursing assistants was 5.9%. High emotional exhaustion was observed in 23.6%, 21.9% showed high depersonalization, and 29.9% low professional achievement. It was found statistically significant associations between emotional exhaustion, job sector and marital status; depersonalization, having children and health problems; low professional achievement and job sector and number of jobs. There was association between job satisfaction and the three dimensions. Professionals working in the health area must pay intense and extended attention to people who are dependent upon others. The intimate contact of the nursing assistants with hard-to-handle patients, as well as being afraid to make mistakes in healthcare are additional chronic stress factors and burnout syndrome cases related in this study.

Low-level Laser Therapy To The Mouse Femur Enhances The Fungicidal Response Of Neutrophils Against Paracoccidioides Brasiliensis.

Neutrophils (PMN) play a central role in host defense against the neglected fungal infection paracoccidioidomycosis (PCM), which is caused by the dimorphic fungus Paracoccidioides brasiliensis (Pb). PCM is of major importance, especially in Latin America, and its treatment relies on the use of antifungal drugs. However, the course of treatment is lengthy, leading to side effects and even development of fungal resistance. The goal of the study was to use low-level laser therapy (LLLT) to stimulate PMN to fight Pb in vivo. Swiss mice with subcutaneous air pouches were inoculated with a virulent strain of Pb or fungal cell wall components (Zymosan), and then received LLLT (780 nm; 50 mW; 12.5 J/cm2; 30 seconds per point, giving a total energy of 0.5 J per point) on alternate days at two points on each hind leg. The aim was to reach the bone marrow in the femur with light. Non-irradiated animals were used as controls. The number and viability of the PMN that migrated to the inoculation site was assessed, as well as their ability to synthesize proteins, produce reactive oxygen species (ROS) and their fungicidal activity. The highly pure PMN populations obtained after 10 days of infection were also subsequently cultured in the presence of Pb for trials of protein production, evaluation of mitochondrial activity, ROS production and quantification of viable fungi growth. PMN from mice that received LLLT were more active metabolically, had higher fungicidal activity against Pb in vivo and also in vitro. The kinetics of neutrophil protein production also correlated with a more activated state. LLLT may be a safe and non-invasive approach to deal with PCM infection.

The Psychedelic State Induced By Ayahuasca Modulates The Activity And Connectivity Of The Default Mode Network.

The experiences induced by psychedelics share a wide variety of subjective features, related to the complex changes in perception and cognition induced by this class of drugs. A remarkable increase in introspection is at the core of these altered states of consciousness. Self-oriented mental activity has been consistently linked to the Default Mode Network (DMN), a set of brain regions more active during rest than during the execution of a goal-directed task. Here we used fMRI technique to inspect the DMN during the psychedelic state induced by Ayahuasca in ten experienced subjects. Ayahuasca is a potion traditionally used by Amazonian Amerindians composed by a mixture of compounds that increase monoaminergic transmission. In particular, we examined whether Ayahuasca changes the activity and connectivity of the DMN and the connection between the DMN and the task-positive network (TPN). Ayahuasca caused a significant decrease in activity through most parts of the DMN, including its most consistent hubs: the Posterior Cingulate Cortex (PCC)/Precuneus and the medial Prefrontal Cortex (mPFC). Functional connectivity within the PCC/Precuneus decreased after Ayahuasca intake. No significant change was observed in the DMN-TPN orthogonality. Altogether, our results support the notion that the altered state of consciousness induced by Ayahuasca, like those induced by psilocybin (another serotonergic psychedelic), meditation and sleep, is linked to the modulation of the activity and the connectivity of the DMN.

Engineered Nanoparticles And Organic Matter: A Review Of The State-of-the-art.

Growth in the development and production of engineered nanoparticles (ENPs) in recent years has increased the potential for interactions of these nanomaterials with aquatic and terrestrial environments. Carefully designed studies are therefore required in order to understand the fate, transport, stability, and toxicity of nanoparticles. Natural organic matter (NOM), such as the humic substances found in water, sediment, and soil, is one of the substances capable of interacting with ENPs. This review presents the findings of studies of the interaction of ENPs and NOM, and the possible effects on nanoparticle stability and the toxicity of these materials in the environment. In addition, ENPs and NOM are utilized for many different purposes, including the removal of metals and organic compounds from effluents, and the development of new electronic sensors and other devices for the detection of active substances. Discussion is therefore provided of some of the ways in which NOM can be used in the production of nanoparticles. Although there has been an increase in the number of studies in this area, further progress is needed to improve understanding of the dynamic interactions between ENPs and NOM.