Techniques for batch reinforcement learning in robotics

This thesis addresses the Batch Reinforcement Learning methods in Robotics. This sub-class of Reinforcement Learning has shown promising results and has been the focus of recent research. Three contributions are proposed that aim to extend the state-of-art methods allowing for a faster and more stable learning process, such as required for learning in Robotics. The Q-learning update-rule is widely applied, since it allows to learn without the presence of a model of the environment. However, this update-rule is transition-based and does not take advantage of the underlying episodic structure of collected batch of interactions. The Q-Batch update-rule is proposed in this thesis, to process experiencies along the trajectories collected in the interaction phase. This allows a faster propagation of obtained rewards and penalties, resulting in faster and more robust learning. Non-parametric function approximations are explored, such as Gaussian Processes. This type of approximators allows to encode prior knowledge about the latent function, in the form of kernels, providing a higher level of exibility and accuracy. The application of Gaussian Processes in Batch Reinforcement Learning presented a higher performance in learning tasks than other function approximations used in the literature. Lastly, in order to extract more information from the experiences collected by the agent, model-learning techniques are incorporated to learn the system dynamics. In this way, it is possible to augment the set of collected experiences with experiences generated through planning using the learned models. Experiments were carried out mainly in simulation, with some tests carried out in a physical robotic platform. The obtained results show that the proposed approaches are able to outperform the classical Fitted Q Iteration.

Nonparametric Evaluation of Quantitative Traits in Population-Based Association Studies when the Genetic Model is Unknown

Statistical association between a single nucleotide polymorphism (SNP) genotype and a quantitative trait in genome-wide association studies is usually assessed using a linear regression model, or, in the case of non-normally distributed trait values, using the Kruskal-Wallis test. While linear regression models assume an additive mode of inheritance via equi-distant genotype scores, Kruskal-Wallis test merely tests global differences in trait values associated with the three genotype groups. Both approaches thus exhibit suboptimal power when the underlying inheritance mode is dominant or recessive. Furthermore, these tests do not perform well in the common situations when only a few trait values are available in a rare genotype category (disbalance), or when the values associated with the three genotype categories exhibit unequal variance (variance heterogeneity). We propose a maximum test based on Marcus-type multiple contrast test for relative effect sizes. This test allows model-specific testing of either dominant, additive or recessive mode of inheritance, and it is robust against variance heterogeneity. We show how to obtain mode-specific simultaneous confidence intervals for the relative effect sizes to aid in interpreting the biological relevance of the results. Further, we discuss the use of a related all-pairwise comparisons contrast test with range preserving confidence intervals as an alternative to Kruskal-Wallis heterogeneity test. We applied the proposed maximum test to the Bogalusa Heart Study dataset, and gained a remarkable increase in the power to detect association, particularly for rare genotypes. Our simulation study also demonstrated that the proposed non-parametric tests control family-wise error rate in the presence of non-normality and variance heterogeneity contrary to the standard parametric approaches. We provide a publicly available R library nparcomp that can be used to estimate simultaneous confidence intervals or compatible multiplicity-adjusted p-values associated with the proposed maximum test.

Signs and Symptoms of Dry Eye in Keratoconus Patients: A Pilot Study

Purpose: To compare signs and symptoms of dry eye in keratoconus (KC) patients versus healthy subjects. Methods: A total of 15 KC patients (KC group, n = 15 eyes) and 16 healthy subjects (control group, 16 eyes) were enrolled in this study. The Schirmer I test with no anesthetic, tear break-up time (TBUT), corneal staining characteristics, and ocular surface disease index (OSDI) scores were evaluated for both groups. Impression cytology, combined with/scanning laser confocal microscopy (LCM), was performed to evaluate goblet cell density, mucin cloud height (MCH), and goblet cell layer thickness (CLT). Finally, tear concentrations of di-adenosine tetraphosphate (Ap4A) were assessed. Results were statistically analyzed using Shapiro–Wilk and non-parametric Wilcoxon rank sum tests. Statistical significance was set at p < 0.05. Results: KC patients had lower tear volumes and greater corneal staining than did healthy subjects (p < 0.05). OSDI scores were 44.96 ± 8.65 and 17.78 ± 6.50 for the KC and control groups, respectively (p < 0.05). We found no statistically significant differences in TBUT between groups. Impression cytology revealed lower goblet cell densities in KC group patients versus control group subjects (84.88 ± 32.98 and 128.88 ± 50.60 cells/mm,2 respectively, p < 0.05). There was a statistically significant reduction in MCH and CLT in KC group patients compared with control group subjects. Ap4A tear concentrations were higher in KC group patients than in control group subjects (2.56 ± 1.10 and 0.15 ± 0.12 µM, respectively, p < 0.05). Conclusions: The parameters evaluated in this study indicate that KC patients suffer greater symptoms of dry eye and greater tear instability, primarily due to the decreased mucin production in their tears, than do healthy patients with no KC.

Effects of acute exercise on salivary free insulin-like growth factor 1 and interleukin 10 in sportsmen.

Background: Saliva analysis is rapidly developing as a tool for the assessment of biomarkers of sports training. It remains poorly understood whether a short bout of sport training can alter some salivary immune biomarkers. Aim: To investigate the effect of acute exercise using football training session on salivary flow rate, salivary free Insulin-like Growth Factor-1 (IGF-1) and Interleukin 10 (IL-10). Methods: Saliva samples were collected before and immediately after a football session. Salivary flow rates, salivary levels of free IGF-1 and IL-10 (using ELISA) were determined. Data was analyzed and compared using Related Samples Wilcoxon Signed Rank test (non-parametric test). Relationships between salivary flow rate and levels of free IGF-1 and IL-10 were determined using Spearman correlation test. Results: There were 22 male footballers with a mean age of 20.46 years. Salivary flow rate reduced significantly (p = 0.01) after the training session while salivary levels of free IGF-1 and IL-10 did not show any significant change. Also, there were no correlations between salivary flow rates and salivary levels of free IGF-1 and IL-10 before and after exercise. Conclusion: These findings suggest that acute exercise caused significant reduction in salivary flow rate but no change in the levels of salivary free IGF-1 and IL-10.

Active Foreground Region Extraction and Tracking for Sports Video Annotation

Automatic video segmentation plays a vital role in sports videos annotation. This paper presents a fully automatic and computationally efficient algorithm for analysis of sports videos. Various methods of automatic shot boundary detection have been proposed to perform automatic video segmentation. These investigations mainly concentrate on detecting fades and dissolves for fast processing of the entire video scene without providing any additional feedback on object relativity within the shots. The goal of the proposed method is to identify regions that perform certain activities in a scene. The model uses some low-level feature video processing algorithms to extract the shot boundaries from a video scene and to identify dominant colours within these boundaries. An object classification method is used for clustering the seed distributions of the dominant colours to homogeneous regions. Using a simple tracking method a classification of these regions to active or static is performed. The efficiency of the proposed framework is demonstrated over a standard video benchmark with numerous types of sport events and the experimental results show that our algorithm can be used with high accuracy for automatic annotation of active regions for sport videos.

Utilização de inteligência competitiva, da gestão de riscos e da computação aplicada para ganhos de competitividade em instituição organizadora de concursos

Dissertação (mestrado)—Universidade de Brasília, Instituto de Ciências Exatas, Departamento de Ciência da Computação, 2015.

Comparing the performance of the EQ-5D-3L and the EQ-5D-5L in young Portuguese adults

Background Some studies have reported a ceiling effect in EQ-5D-3L, especially in healthy and/or young individuals. Recently, two further levels have been included in its measurement model (EQ-5D-5L). The purposes of this study were (1) to assess the properties of the EQ-5D-5L in comparison with the standard EQ-5D-3L in a sample of young adults, (2) to foreground the importance of collecting qualitative data to confirm, validate or refine the EQ-5D questionnaire items and (3) to raise questions pertaining to the wording in these questionnaire items. Methods The data used came from a sample of respondents aged 30 or under (n = 624). They completed both versions of the EQ-5D, which were compared in terms of feasibility, level of inconsistency and ceiling effect. Agreement between the instruments was assessed using correlation coefficients and Bland-Altman plots. Known-groups validity of the EQ-5D-5L was also assessed using non-parametric tests. The discriminative properties were compared using receiver operating characteristic curves. Finally, four interviews were conducted for retrospective reports to elicit respondents’ understanding and perceptions of the format, instructions, items, and responses. Results Quantitative results show a ceiling effect reduction of 25.3 % and a high level agreement between both indices. Known-groups validity was confirmed for the EQ-5D-5L. Explorative interviews indicated ambiguity and low degree of certainty in regards to conceptualizing differences between levels moderate-slight across three dimensions. Conclusions The EQ-5D-5L performed better than the EQ-5D-3L. However, the explorative interviews demonstrated several limitations in the EQ-5D questionnaire wording and high context-dependent answers point to lack of illnesses’ experience amongst young adults.

Avaliação do desempenho das entidades gestoras dos serviços de resíduos urbanos em Portugal

Dissertação de Mestrado, Gestão Empresarial, Faculdade de Economia, Universidade do Algarve, 2016

Plasma D-lactate Levels in Necrotizing Enterocolitis in Premature Infants

Background: D-Lactate is normally present in the blood of humans at nanomolar concentrations due to methylglyoxal metabolism; millimolar D-lactate concentrations can arise due to excess gastrointestinal microbial production. Objectives: To examine the levels of plasma D-lactate in the necrotizing enterocolitis in premature infants. Patients and Methods: 128 premature infants were divided into control (group I, n = 69), feeding intolerance (group II, n = 42) and NEC (group III, n = 27) groups. Plasma D-lactate levels were measured at the onset of feeding intolerance or NEC and at weeks 2-3 in control infants (group I) by ELISA. Data were analyzed using descriptive statistics, non-parametric tests and Student’s t-test. Results: In groups I, II, III, median birth weights were 1845.7 ± 267.5 g, 1913.1 ± 306.5 g, and 1898.4 ± 285.3 g, median gestational ages were 34.3 ± 1.7 weeks, 33.9 ± 2.2 weeks and 35.1 ± 2.6 weeks, ages of sampling were 12.3 ± 2.9 days, 14.6 ± 3.7 days and 15.1 ± 1.8 days, respectively. The differences of median birth weights, median gestational ages and ages of sampling were not statistically significant (P > 0.05). The plasma D-lactate levels in groups I, II, III were 3.6 ± 1.9 μg/mL, 12.7 ± 8.3 μg/mL, and 35.4 ± 29.1 μg/mL, respectively, group III had higher plasma D-lactate level than groups I, II, and the difference among these groups was significant (x2 = 21.6, P < 0.01). Conclusions: Plasma D-lactate significantly increased early in NEC. Plasma D-lactate levels were associated with extensive disease in NEC infants. Therefore, it could be used as a diagnosis indicator in the early stage of NEC.

A survey on multi-output regression

In recent years, a plethora of approaches have been proposed to deal with the increasingly challenging task of multi-output regression. This paper provides a survey on state-of-the-art multi-output regression methods, that are categorized as problem transformation and algorithm adaptation methods. In addition, we present the mostly used performance evaluation measures, publicly available data sets for multi-output regression real-world problems, as well as open-source software frameworks.

Évaluation de l’impact des services en téléobstétrique du RUIS McGill offerts à une population de femmes inuites avec grossesse à risque élevé habitant sur la côte de la baie d’Hudson au Nunavik

L’accessibilité à des soins de santé pour une population habitant une région éloignée au Québec représente un défi de taille pour le Ministère de la santé et des services sociaux. Des solutions, telles que la télésanté, ont été présentées afin de pallier ce problème. Le RUIS McGill a ainsi développé un programme de téléobstétrique afin de desservir une population de femmes inuites à grossesse à risque élevé (GARE) habitant le Nunavik. L’objectif de ce mémoire fut de comprendre l’impact du service de téléobstétrique du RUIS McGill sur la santé des femmes et de leur nouveau-né ainsi que sur les coûts de santé et l’utilisation des services suite à son implantation au Centre de santé et de services sociaux Inuulitsivik sur la côte de la baie d’Hudson. Les femmes inuites à grossesse à risque élevé et leurs enfants de la région de la baie d’Hudson du Nunavik, éloignés des services obstétriques spécialisés, sont visés. Le service de téléobstétrique permet un accès aux obstétriciens du RUIS McGill localisés à Montréal. Un devis quasi-expérimental est utilisé pour examiner trois hypothèses portant sur l’état de santé des mères et des enfants, sur l’utilisation des services de santé et sur leurs coûts. Le service de téléobstétrique est devenu fonctionnel en 2006, offrant la possibilité de constituer une étude avant-après à deux groupes de femmes, soit celles ayant accouché avant 2006 (prétest) et celle ayant accouché après 2012 (post-test). La collecte de donnée se fit, dans son intégralité, par l’entremise des dossiers médicaux papier des participantes permettant l’analyse de 47 dossiers pour le prétest et de 81 dossiers pour le post-test. L’exécution d’analyse de covariance, de régression logistique et du test non paramétrique de Mann-Witney permit de conclure que le prétest et le post-test ne différent que sur deux variables, soient le poids à la naissance, plus faible dans le post-test et la pression artérielle de la mère à la naissance, plus élevée dans le post-test. Pour l’ensemble des autres variables portant sur les trois hypothèses à l’étude, les résultats de ce mémoire ne démontrent aucune différence significative entre les deux groupes démontrant ainsi qu’une même qualité de soins a été conservée suite à l’implantation du programme de téléobstétrique. Sur la base des résultats, ce mémoire recommande de revoir et modifier les objectifs du programme; de partager les bornes de communication de télésanté avec d’autres spécialités; d’entreprendre une évaluation du programme axée sur les coûts; de suivre rigoureusement l’utilisation du programme pour en maximiser l’efficacité et le potentiel; d’établir un tableau de bord; et d’entreprendre une étude évaluative comparative dans un service de téléobstétrique comparable.

Évaluation de l’impact des services en téléobstétrique du RUIS McGill offerts à une population de femmes inuites avec grossesse à risque élevé habitant sur la côte de la baie d’Hudson au Nunavik

L’accessibilité à des soins de santé pour une population habitant une région éloignée au Québec représente un défi de taille pour le Ministère de la santé et des services sociaux. Des solutions, telles que la télésanté, ont été présentées afin de pallier ce problème. Le RUIS McGill a ainsi développé un programme de téléobstétrique afin de desservir une population de femmes inuites à grossesse à risque élevé (GARE) habitant le Nunavik. L’objectif de ce mémoire fut de comprendre l’impact du service de téléobstétrique du RUIS McGill sur la santé des femmes et de leur nouveau-né ainsi que sur les coûts de santé et l’utilisation des services suite à son implantation au Centre de santé et de services sociaux Inuulitsivik sur la côte de la baie d’Hudson. Les femmes inuites à grossesse à risque élevé et leurs enfants de la région de la baie d’Hudson du Nunavik, éloignés des services obstétriques spécialisés, sont visés. Le service de téléobstétrique permet un accès aux obstétriciens du RUIS McGill localisés à Montréal. Un devis quasi-expérimental est utilisé pour examiner trois hypothèses portant sur l’état de santé des mères et des enfants, sur l’utilisation des services de santé et sur leurs coûts. Le service de téléobstétrique est devenu fonctionnel en 2006, offrant la possibilité de constituer une étude avant-après à deux groupes de femmes, soit celles ayant accouché avant 2006 (prétest) et celle ayant accouché après 2012 (post-test). La collecte de donnée se fit, dans son intégralité, par l’entremise des dossiers médicaux papier des participantes permettant l’analyse de 47 dossiers pour le prétest et de 81 dossiers pour le post-test. L’exécution d’analyse de covariance, de régression logistique et du test non paramétrique de Mann-Witney permit de conclure que le prétest et le post-test ne différent que sur deux variables, soient le poids à la naissance, plus faible dans le post-test et la pression artérielle de la mère à la naissance, plus élevée dans le post-test. Pour l’ensemble des autres variables portant sur les trois hypothèses à l’étude, les résultats de ce mémoire ne démontrent aucune différence significative entre les deux groupes démontrant ainsi qu’une même qualité de soins a été conservée suite à l’implantation du programme de téléobstétrique. Sur la base des résultats, ce mémoire recommande de revoir et modifier les objectifs du programme; de partager les bornes de communication de télésanté avec d’autres spécialités; d’entreprendre une évaluation du programme axée sur les coûts; de suivre rigoureusement l’utilisation du programme pour en maximiser l’efficacité et le potentiel; d’établir un tableau de bord; et d’entreprendre une étude évaluative comparative dans un service de téléobstétrique comparable.

Nurses knowledge of health effects of dietary fibre

Introduction: Nurses accompany patients throughout their health care to prevent and treat disease, so their knowledge about diet and dietary fibre is key to successful diet therapy, which is an essential part of a patient's non-pharmacological treatment. It is known from the literature that a high proportion of nurses have limited knowledge about diet therapy and about sources of soluble fibre and other foods that can prevent or treat certain diseases. Given the position of nurses as key providers of dietary guidance, and given the health benefits of dietary fibre, we wanted to assess the level of fibre-related knowledge among nurses in Croatia. Material and Methods: Cross-sectional study based on data collected between October 2014 and March 2015 using a survey developed by the CI&DETS Polytechnic Institute in Viseu, Portugal. The survey contains questions about demographic characteristics as well as about knowledge of sources of dietary fibre, recommended daily intake and effects of fibre intake on particular diseases. The study included a total of 369 nurses from two health institutions and one nursing school from Croatia older than 18 years. Differences in knowledge were assessed for significance using the non-parametric Mann-Whitney U test. Possible associations among variables were explored using Spearman's rank correlation. For all statistical analyses, the threshold of significance was defined as P<0.05. Results: The level of knowledge among nurses in Croatia about dietary fibre varied from «undecided» to «partial knowledge». The median for level of knowledge ranged from 3 to 4 with low variability ranging from 0.11 to 0.33. Average levels of knowledge in percentages varied from 57.6% to 82.1%. Nurses with higher education levels showed significantly higher knowledge levels about the influence of dietary fiber intake on the risk of certain diseases (p = 0.007), constipation (p = 0.016), bowel cancer (p = 0.005) and breast cancer (p = 0.039). Conclusion: The level of nurses’ knowledge about dietary fiber is suboptimal. This indicates the need to strengthen nurse education in the areas of diet and diet therapy. Increase the level of knowledge of nurses about nutrition can positively influence the quality of care.

Hilbert Space Valued Signal Plus Noise Models: Analysis of Structural Breaks under High Dimensionality and Temporal Dependence

This thesis is concerned with change point analysis for time series, i.e. with detection of structural breaks in time-ordered, random data. This long-standing research field regained popularity over the last few years and is still undergoing, as statistical analysis in general, a transformation to high-dimensional problems. We focus on the fundamental »change in the mean« problem and provide extensions of the classical non-parametric Darling-Erdős-type cumulative sum (CUSUM) testing and estimation theory within highdimensional Hilbert space settings. In the first part we contribute to (long run) principal component based testing methods for Hilbert space valued time series under a rather broad (abrupt, epidemic, gradual, multiple) change setting and under dependence. For the dependence structure we consider either traditional m-dependence assumptions or more recently developed m-approximability conditions which cover, e.g., MA, AR and ARCH models. We derive Gumbel and Brownian bridge type approximations of the distribution of the test statistic under the null hypothesis of no change and consistency conditions under the alternative. A new formulation of the test statistic using projections on subspaces allows us to simplify the standard proof techniques and to weaken common assumptions on the covariance structure. Furthermore, we propose to adjust the principal components by an implicit estimation of a (possible) change direction. This approach adds flexibility to projection based methods, weakens typical technical conditions and provides better consistency properties under the alternative. In the second part we contribute to estimation methods for common changes in the means of panels of Hilbert space valued time series. We analyze weighted CUSUM estimates within a recently proposed »high-dimensional low sample size (HDLSS)« framework, where the sample size is fixed but the number of panels increases. We derive sharp conditions on »pointwise asymptotic accuracy« or »uniform asymptotic accuracy« of those estimates in terms of the weighting function. Particularly, we prove that a covariance-based correction of Darling-Erdős-type CUSUM estimates is required to guarantee uniform asymptotic accuracy under moderate dependence conditions within panels and that these conditions are fulfilled, e.g., by any MA(1) time series. As a counterexample we show that for AR(1) time series, close to the non-stationary case, the dependence is too strong and uniform asymptotic accuracy cannot be ensured. Finally, we conduct simulations to demonstrate that our results are practically applicable and that our methodological suggestions are advantageous.

Impacto del Programa de Alimentación Escolar en el Trabajo Infantil: Una aproximación desde la toma de decisiones familiares

Esta investigación analiza el impacto del Programa de Alimentación Escolar en el trabajo infantil en Colombia a través de varias técnicas de evaluación de impacto que incluyen emparejamiento simple, emparejamiento genético y emparejamiento con reducción de sesgo. En particular, se encuentra que este programa disminuye la probabilidad de que los escolares trabajen alrededor de un 4%. Además, se explora que el trabajo infantil se reduce gracias a que el programa aumenta la seguridad alimentaria, lo que consecuentemente cambia las decisiones de los hogares y anula la carga laboral en los infantes. Son numerosos los avances en primera infancia llevados a cabo por el Estado, sin embargo, estos resultados sirven de base para construir un marco conceptual en el que se deben rescatar y promover las políticas públicas alimentarias en toda la edad escolar.