Gradient-based reinforcement learning techniques for underwater robotics behavior learning

Darrerament, l'interès pel desenvolupament d'aplicacions amb robots submarins autònoms (AUV) ha crescut de forma considerable. Els AUVs són atractius gràcies al seu tamany i el fet que no necessiten un operador humà per pilotar-los. Tot i això, és impossible comparar, en termes d'eficiència i flexibilitat, l'habilitat d'un pilot humà amb les escasses capacitats operatives que ofereixen els AUVs actuals. L'utilització de AUVs per cobrir grans àrees implica resoldre problemes complexos, especialment si es desitja que el nostre robot reaccioni en temps real a canvis sobtats en les condicions de treball. Per aquestes raons, el desenvolupament de sistemes de control autònom amb l'objectiu de millorar aquestes capacitats ha esdevingut una prioritat. Aquesta tesi tracta sobre el problema de la presa de decisions utilizant AUVs. El treball presentat es centra en l'estudi, disseny i aplicació de comportaments per a AUVs utilitzant tècniques d'aprenentatge per reforç (RL). La contribució principal d'aquesta tesi consisteix en l'aplicació de diverses tècniques de RL per tal de millorar l'autonomia dels robots submarins, amb l'objectiu final de demostrar la viabilitat d'aquests algoritmes per aprendre tasques submarines autònomes en temps real. En RL, el robot intenta maximitzar un reforç escalar obtingut com a conseqüència de la seva interacció amb l'entorn. L'objectiu és trobar una política òptima que relaciona tots els estats possibles amb les accions a executar per a cada estat que maximitzen la suma de reforços totals. Així, aquesta tesi investiga principalment dues tipologies d'algoritmes basats en RL: mètodes basats en funcions de valor (VF) i mètodes basats en el gradient (PG). Els resultats experimentals finals mostren el robot submarí Ictineu en una tasca autònoma real de seguiment de cables submarins. Per portar-la a terme, s'ha dissenyat un algoritme anomenat mètode d'Actor i Crític (AC), fruit de la fusió de mètodes VF amb tècniques de PG.

Comparison of bottom-up protein identification methods

With the rapid development of proteomics, a number of different methods appeared for the basic task of protein identification. We made a simple comparison between a common liquid chromatography-tandem mass spectrometry (LC-MS/MS) workflow using an ion trap mass spectrometer and a combined LC-MS and LC-MS/MS method using Fourier transform ion cyclotron resonance (FTICR) mass spectrometry and accurate peptide masses. To compare the two methods for protein identification, we grew and extracted proteins from E. coli using established protocols. Cystines were reduced and alkylated, and proteins digested by trypsin. The resulting peptide mixtures were separated by reversed-phase liquid chromatography using a 4 h gradient from 0 to 50% acetonitrile over a C18 reversed-phase column. The LC separation was coupled on-line to either a Bruker Esquire HCT ion trap or a Bruker 7 tesla APEX-Qe Qh-FTICR hybrid mass spectrometer. Data-dependent Qh-FTICR-MS/MS spectra were acquired using the quadrupole mass filter and collisionally induced dissociation into the external hexapole trap. Proteins were in both schemes identified by Mascot MS/MS ion searches and the peptides identified from these proteins in the FTICR MS/MS data were used for automatic internal calibration of the FTICR-MS data, together with ambient polydimethylcyclosiloxane ions.

Application of molecular biological methods for studying probiotics and the gut flora

Increasingly, the microbiological scientific community is relying on molecular biology to define the complexity of the gut flora and to distinguish one organism from the next. This is particularly pertinent in the field of probiotics, and probiotic therapy, where identifying probiotics from the commensal flora is often warranted. Current techniques, including genetic fingerprinting, gene sequencing, oligonucleotide probes and specific primer selection, discriminate closely related bacteria with varying degrees of success. Additional molecular methods being employed to determine the constituents of complex microbiota in this area of research are community analysis, denaturing gradient gel electrophoresis (DGGE)/temperature gradient gel electrophoresis (TGGE), fluorescent in situ hybridisation (FISH) and probe grids. Certain approaches enable specific aetiological agents to be monitored, whereas others allow the effects of dietary intervention on bacterial populations to be studied. Other approaches demonstrate diversity, but may not always enable quantification of the population. At the heart of current molecular methods is sequence information gathered from culturable organisms. However, the diversity and novelty identified when applying these methods to the gut microflora demonstrates how little is known about this ecosystem. Of greater concern is the inherent bias associated with some molecular methods. As we understand more of the complexity and dynamics of this diverse microbiota we will be in a position to develop more robust molecular-based technologies to examine it. In addition to identification of the microbiota and discrimination of probiotic strains from commensal organisms, the future of molecular biology in the field of probiotics and the gut flora will, no doubt, stretch to investigations of functionality and activity of the microflora, and/or specific fractions. The quest will be to demonstrate the roles of probiotic strains in vivo and not simply their presence or absence.

On merging gradient estimation with mean-tracking techniques for cluster identification

This paper discusses how numerical gradient estimation methods may be used in order to reduce the computational demands on a class of multidimensional clustering algorithms. The study is motivated by the recognition that several current point-density based cluster identification algorithms could benefit from a reduction of computational demand if approximate a-priori estimates of the cluster centres present in a given data set could be supplied as starting conditions for these algorithms. In this particular presentation, the algorithm shown to benefit from the technique is the Mean-Tracking (M-T) cluster algorithm, but the results obtained from the gradient estimation approach may also be applied to other clustering algorithms and their related disciplines.

Detecting an impact of predation on bird populations depends on the methods used to assess the predators

Summary 1. In recent decades there have been population declines of many UK bird species, which have become the focus of intense research and debate. Recently, as the populations of potential predators have increased there is concern that increased rates of predation may be contributing to the declines. In this review, we assess the methodologies behind the current published science on the impacts of predators on avian prey in the UK. 2. We identified suitable studies, classified these according to study design (experimental ⁄observational) and assessed the quantity and quality of the data upon which any variation in predation rates was inferred. We then explored whether the underlying study methodology had implications for study outcome. 3. We reviewed 32 published studies and found that typically observational studies comprehensively monitored significantly fewer predator species than experimental studies. Data for a difference in predator abundance from targeted (i.e. bespoke) census techniques were available for less than half of the 32 predator species studied. 4. The probability of a study detecting an impact on prey abundance was strongly, positively related to the quality and quantity of data upon which the gradient in predation rates was inferred. 5. The findings suggest that if a study is based on good quality abundance data for a range of predator species then it is more likely to detect an effect than if it relies on opportunistic data for a smaller number of predators. 6. We recommend that the findings from studies which use opportunistic data, for a limited number of predator species, should be treated with caution and that future studies employ bespoke census techniques to monitor predator abundance for an appropriate suite of predators.

Iterative Methods for Equality-Constrained Least Squares Problems

We consider the linear equality-constrained least squares problem (LSE) of minimizing ${\|c - Gx\|}_2 $, subject to the constraint $Ex = p$. A preconditioned conjugate gradient method is applied to the Kuhn–Tucker equations associated with the LSE problem. We show that our method is well suited for structural optimization problems in reliability analysis and optimal design. Numerical tests are performed on an Alliant FX/8 multiprocessor and a Cray-X-MP using some practical structural analysis data.

Intercomparison of methods of coupling between convection and large-scale circulation: 1. Comparison over uniform surface conditions

As part of an international intercomparison project, a set of single column models (SCMs) and cloud-resolving models (CRMs) are run under the weak temperature gradient (WTG) method and the damped gravity wave (DGW) method. For each model, the implementation of the WTG or DGW method involves a simulated column which is coupled to a reference state defined with profiles obtained from the same model in radiative-convective equilibrium. The simulated column has the same surface conditions as the reference state and is initialized with profiles from the reference state. We performed systematic comparison of the behavior of different models under a consistent implementation of the WTG method and the DGW method and systematic comparison of the WTG and DGW methods in models with different physics and numerics. CRMs and SCMs produce a variety of behaviors under both WTG and DGW methods. Some of the models reproduce the reference state while others sustain a large-scale circulation which results in either substantially lower or higher precipitation compared to the value of the reference state. CRMs show a fairly linear relationship between precipitation and circulation strength. SCMs display a wider range of behaviors than CRMs. Some SCMs under the WTG method produce zero precipitation. Within an individual SCM, a DGW simulation and a corresponding WTG simulation can produce different signed circulation. When initialized with a dry troposphere, DGW simulations always result in a precipitating equilibrium state. The greatest sensitivities to the initial moisture conditions occur for multiple stable equilibria in some WTG simulations, corresponding to either a dry equilibrium state when initialized as dry or a precipitating equilibrium state when initialized as moist. Multiple equilibria are seen in more WTG simulations for higher SST. In some models, the existence of multiple equilibria is sensitive to some parameters in the WTG calculations.

Intercomparison of methods of coupling between convection and large-scale circulation. 2: Comparison over non-uniform surface conditions

As part of an international intercomparison project, the weak temperature gradient (WTG) and damped gravity wave (DGW) methods are used to parameterize large-scale dynamics in a set of cloud-resolving models (CRMs) and single column models (SCMs). The WTG or DGW method is implemented using a configuration that couples a model to a reference state defined with profiles obtained from the same model in radiative-convective equilibrium. We investigated the sensitivity of each model to changes in SST, given a fixed reference state. We performed a systematic comparison of the WTG and DGW methods in different models, and a systematic comparison of the behavior of those models using the WTG method and the DGW method. The sensitivity to the SST depends on both the large-scale parameterization method and the choice of the cloud model. In general, SCMs display a wider range of behaviors than CRMs. All CRMs using either the WTG or DGW method show an increase of precipitation with SST, while SCMs show sensitivities which are not always monotonic. CRMs using either the WTG or DGW method show a similar relationship between mean precipitation rate and column-relative humidity, while SCMs exhibit a much wider range of behaviors. DGW simulations produce large-scale velocity profiles which are smoother and less top-heavy compared to those produced by the WTG simulations. These large-scale parameterization methods provide a useful tool to identify the impact of parameterization differences on model behavior in the presence of two-way feedback between convection and the large-scale circulation.

The genetic effects of Late Quaternary climatic changes over a tropical latitudinal gradient: diversification of an Atlantic Forest passerine

The increase in biodiversity from high to low latitudes is a widely recognized biogeographical pattern. According to the latitudinal gradient hypothesis (LGH), this pattern was shaped by differential effects of Late Quaternary climatic changes across a latitudinal gradient. Here, we evaluate the effects of climatic changes across a tropical latitudinal gradient and its implications to diversification of an Atlantic Forest (AF) endemic passerine. We studied the intraspecific diversification and historical demography of Sclerurus scansor, based on mitochondrial (ND2, ND3 and cytb) and nuclear (FIB7) gene sequences. Phylogenetic analyses recovered three well-supported clades associated with distinct latitudinal zones. Coalescent-based methods were applied to estimate divergence times and changes in effective population sizes. Estimates of divergence times indicate that intraspecific diversification took place during Middle-Late Pleistocene. Distinct demographic scenarios were identified, with the southern lineage exhibiting a clear signature of demographic expansion, while the central one remained more stable. The northern lineage, contrasting with LGH predictions, exhibited a clear sign of a recent bottleneck. Our results suggest that different AF regions reacted distinctly, even in opposite ways, under the same climatic period, producing simultaneously favourable scenarios for isolation and contact among populations.

Partial spectral projected gradient method with active-set strategy for linearly constrained optimization

A method for linearly constrained optimization which modifies and generalizes recent box-constraint optimization algorithms is introduced. The new algorithm is based on a relaxed form of Spectral Projected Gradient iterations. Intercalated with these projected steps, internal iterations restricted to faces of the polytope are performed, which enhance the efficiency of the algorithm. Convergence proofs are given and numerical experiments are included and commented. Software supporting this paper is available through the Tango Project web page: http://www.ime.usp.br/similar to egbirgin/tango/.

Proximal methods for nonlinear programming: double regularization and inexact subproblems

This paper describes the first phase of a project attempting to construct an efficient general-purpose nonlinear optimizer using an augmented Lagrangian outer loop with a relative error criterion, and an inner loop employing a state-of-the art conjugate gradient solver. The outer loop can also employ double regularized proximal kernels, a fairly recent theoretical development that leads to fully smooth subproblems. We first enhance the existing theory to show that our approach is globally convergent in both the primal and dual spaces when applied to convex problems. We then present an extensive computational evaluation using the CUTE test set, showing that some aspects of our approach are promising, but some are not. These conclusions in turn lead to additional computational experiments suggesting where to next focus our theoretical and computational efforts.

Development of practical HPLC methods for the separation and determination of eggplant steroidal glycoalkaloids and their aglycones

A practical set of HPLC methods was developed for the separation and determination of the eggplant steroidal glycoalkaloids, solanine, chaconine, solasonine, solamargine, and their aglycones, solasodine and solanidine. A gradient method was initially developed, but proved to be neither robust nor practical. Three separate isocratic methods using acetonitrile and ammonium dihydrogen phosphate were developed and shown to be more repeatable, less subject to fluctuations in mobile phase composition, and less time consuming. The effect of adjusting buffer pH, column temperature, and buffer type (triethylammonium phosphate vs. ammonium dihydrogen phosphate) were evaluated. It was also discovered that, by addition of 10% methanol to the acetonitrile portion of the mobile phase, more control over the separations was possible. The use of methanol as a mobile phase entrainer greatly improved separations in some cases and its effectiveness was also dependent upon column temperature. Assessments of the method recovery, limit of detection, and limit of quantitation were made using extracts from S. melongena and S. linnaeanum.

Fish community structure along a conservation gradient in Bodoquena Plateau streams, central West of Brazil.

Aim: To investigate whether composition and structure of stream ichthyofauna vary along the conservation gradient in the Bodoquena Plateau; Methods: Standardized sampling was performed in nine stream stretches (three references, three in the pasture, and three in urban areas), each 80 m long, in April, May and June/08; Results: A quantity of 8,102 individuals representing 36 species was collected. There were no significant differences with respect to absolute species richness among stretches. Urban stretches were more similar to each other than the other two groups, both regarding species composition and abundance. A set of six species was considered an indicator of reference stretches, whereas two were of pasture stretches and four of urban stretches. Poecilia reticulata and Corydoras aeneus, occurring only in urban stretches, were unique in showing the maximum values in the indicator species analysis; Conclusions: Results indicate changes in the ichthyofauna with urbanized surrounding water courses. The presence and abundance of Poecilia reticulata and Corydoras aeneus may indicate the quality loss of these environments.

Disturbances, elevation, topography and spatial proximity drive vegetation patterns along an altitudinal gradient of a top biodiversity hotspot

The correlation between vegetation patterns (species distribution and richness) and altitudinal variation has been widely reported for tropical forests, thereby providing theoretical basis for biodiversity conservation. However, this relationship may have been oversimplified, as many other factors may influence vegetation patterns, such as disturbances, topography and geographic distance. Considering these other factors, our primary question was: is there a vegetation pattern associated with substantial altitudinal variation (10-1,093 m a.s.l.) in the Atlantic Rainforest-a top hotspot for biodiversity conservation-and, if so, what are the main factors driving this pattern? We addressed this question by sampling 11 1-ha plots, applying multivariate methods, correlations and variance partitioning. The Restinga (forest on sandbanks along the coastal plains of Brazil) and a lowland area that was selectively logged 40 years ago were floristically isolated from the other plots. The maximum species richness (>200 spp. per hectare) occurred at approximately 350 m a.s.l. (submontane forest). Gaps, multiple stemmed trees, average elevation and the standard deviation of the slope significantly affected the vegetation pattern. Spatial proximity also influenced the vegetation pattern as a structuring environmental variable or via dispersal constraints. Our results clarify, for the first time, the key variables that drive species distribution and richness across a large altitudinal range within the Atlantic Rainforest. © 2013 Springer Science+Business Media Dordrecht.

Continuity of Lyapunov functions and of energy level for generalized gradient semigroup

The global attractor of a gradient-like semigroup has a Morse decomposition. Associated to this Morse decomposition there is a Lyapunov function (differentiable along solutions)-defined on the whole phase space- which proves relevant information on the structure of the attractor. In this paper we prove the continuity of these Lyapunov functions under perturbation. On the other hand, the attractor of a gradient-like semigroup also has an energy level decomposition which is again a Morse decomposition but with a total order between any two components. We claim that, from a dynamical point of view, this is the optimal decomposition of a global attractor; that is, if we start from the finest Morse decomposition, the energy level decomposition is the coarsest Morse decomposition that still produces a Lyapunov function which gives the same information about the structure of the attractor. We also establish sufficient conditions which ensure the stability of this kind of decomposition under perturbation. In particular, if connections between different isolated invariant sets inside the attractor remain under perturbation, we show the continuity of the energy level Morse decomposition. The class of Morse-Smale systems illustrates our results.