Pretraining Convolutional Neural Networks for Visual Recognition

Convolutional Neural Networks (CNN) have become the state-of-the-art methods on many large scale visual recognition tasks. For a lot of practical applications, CNN architectures have a restrictive requirement: A huge amount of labeled data are needed for training. The idea of generative pretraining is to obtain initial weights of the network by training the network in a completely unsupervised way and then fine-tune the weights for the task at hand using supervised learning. In this thesis, a general introduction to Deep Neural Networks and algorithms are given and these methods are applied to classification tasks of handwritten digits and natural images for developing unsupervised feature learning. The goal of this thesis is to find out if the effect of pretraining is damped by recent practical advances in optimization and regularization of CNN. The experimental results show that pretraining is still a substantial regularizer, however, not a necessary step in training Convolutional Neural Networks with rectified activations. On handwritten digits, the proposed pretraining model achieved a classification accuracy comparable to the state-of-the-art methods.

Comparison between constant and decreasing rest intervals: influence on maximal strength and hypertrophy

de Souza Jr, TP, Fleck, SJ, Simao, R, Dubas, JP, Pereira, B, de Brito Pacheco, EM, da Silva, AC, and de Oliveira, PR. Comparison between constant and decreasing rest intervals: influence on maximal strength and hypertrophy. J Strength Cond Res 24(7): 1843-1850, 2010-Most resistance training programs use constant rest period lengths between sets and exercises, but some programs use decreasing rest period lengths as training progresses. The aim of this study was to compare the effect on strength and hypertrophy of 8 weeks of resistance training using constant rest intervals (CIs) and decreasing rest intervals (DIs) between sets and exercises. Twenty young men recreationally trained in strength training were randomly assigned to either a CI or DI training group. During the first 2 weeks of training, 3 sets of 10-12 repetition maximum (RM) with 2-minute rest intervals between sets and exercises were performed by both groups. During the next 6 weeks of training, the CI group trained using 2 minutes between sets and exercises (4 sets of 8-10RM), and the DI group trained with DIs (2 minutes decreasing to 30 seconds) as the 6 weeks of training progressed (4 sets of 8-10RM). Total training volume of the bench press and squat were significantly lower for the DI compared to the CI group (bench press 9.4%, squat 13.9%) and weekly training volume of these same exercises was lower in the DI group from weeks 6 to 8 of training. Strength (1RM) in the bench press and squat, knee extensor and flexor isokinetic measures of peak torque, and muscle cross-sectional area (CSA) using magnetic resonance imaging were assessed pretraining and posttraining. No significant differences (p <= 0.05) were shown between the CI and DI training protocols for CSA (arm 13.8 vs. 14.5%, thigh 16.6 vs. 16.3%), 1RM (bench press 28 vs. 37%, squat 34 vs. 34%), and isokinetic peak torque. In conclusion, the results indicate that a training protocol with DI is just as effective as a CI protocol over short training periods (6 weeks) for increasing maximal strength and muscle CSA; thus, either type of program can be used over a short training period to cause strength and hypertrophy.

Repeated Double-Poling Sprint Training in Hypoxia by Competitive Cross-country Skiers.

PURPOSE: Repeated-sprint training in hypoxia (RSH) was recently shown to improve repeated-sprint ability (RSA) in cycling. This phenomenon is likely to reflect fiber type-dependent, compensatory vasodilation, and therefore, our hypothesis was that RSH is even more beneficial for activities involving upper body muscles, such as double poling during cross-country skiing. METHODS: In a double-blinded fashion, 17 competitive cross-country skiers performed six sessions of repeated sprints (each consisting of four sets of five 10-s sprints, with 20-s intervals of recovery) either in normoxia (RSN, 300 m; FiO2, 20.9%; n = 8) or normobaric hypoxia (RSH, 3000 m; FiO2, 13.8 %; n = 9). Before (pre) and after (post) training, performance was evaluated with an RSA test (10-s all-out sprints-20-s recovery, until peak power output declined by 30%) and a simulated team sprint (team sprint, 3 × 3-min all-out with 3-min rest) on a double-poling ergometer. Triceps brachii oxygenation was measured by near-infrared spectroscopy. RESULTS: From pretraining to posttraining, peak power output in the RSA was increased (P < 0.01) to the same extent (29% ± 13% vs 26% ± 18%, nonsignificant) in RSH and in RSN whereas the number of sprints performed was enhanced in RSH (10.9 ± 5.2 vs 17.1 ± 6.8, P < 0.01) but not in RSN (11.6 ± 5.3 vs 11.7 ± 4.3, nonsignificant). In addition, the amplitude in total hemoglobin variations during sprints throughout RSA rose more in RSH (P < 0.01). Similarly, the average power output during all team sprints improved by 11% ± 9% in RSH and 15% ± 7% in RSN. CONCLUSIONS: Our findings reveal greater improvement in the performance of repeated double-poling sprints, together with larger variations in the perfusion of upper body muscles in RSH compared with those in RSN.

Deep generative models for facial keypoints detection

A new area of machine learning research called deep learning, has moved machine learning closer to one of its original goals: artificial intelligence and general learning algorithm. The key idea is to pretrain models in completely unsupervised way and finally they can be fine-tuned for the task at hand using supervised learning. In this thesis, a general introduction to deep learning models and algorithms are given and these methods are applied to facial keypoints detection. The task is to predict the positions of 15 keypoints on grayscale face images. Each predicted keypoint is specified by an (x,y) real-valued pair in the space of pixel indices. In experiments, we pretrained deep belief networks (DBN) and finally performed a discriminative fine-tuning. We varied the depth and size of an architecture. We tested both deterministic and sampled hidden activations and the effect of additional unlabeled data on pretraining. The experimental results show that our model provides better results than publicly available benchmarks for the dataset.

Deep learning of representations and its application to computer vision

L’objectif de cette thèse par articles est de présenter modestement quelques étapes du parcours qui mènera (on espère) à une solution générale du problème de l’intelligence artificielle. Cette thèse contient quatre articles qui présentent chacun une différente nouvelle méthode d’inférence perceptive en utilisant l’apprentissage machine et, plus particulièrement, les réseaux neuronaux profonds. Chacun de ces documents met en évidence l’utilité de sa méthode proposée dans le cadre d’une tâche de vision par ordinateur. Ces méthodes sont applicables dans un contexte plus général, et dans certains cas elles on tété appliquées ailleurs, mais ceci ne sera pas abordé dans le contexte de cette de thèse. Dans le premier article, nous présentons deux nouveaux algorithmes d’inférence variationelle pour le modèle génératif d’images appelé codage parcimonieux “spike- and-slab” (CPSS). Ces méthodes d’inférence plus rapides nous permettent d’utiliser des modèles CPSS de tailles beaucoup plus grandes qu’auparavant. Nous démontrons qu’elles sont meilleures pour extraire des détecteur de caractéristiques quand très peu d’exemples étiquetés sont disponibles pour l’entraînement. Partant d’un modèle CPSS, nous construisons ensuite une architecture profonde, la machine de Boltzmann profonde partiellement dirigée (MBP-PD). Ce modèle a été conçu de manière à simplifier d’entraînement des machines de Boltzmann profondes qui nécessitent normalement une phase de pré-entraînement glouton pour chaque couche. Ce problème est réglé dans une certaine mesure, mais le coût d’inférence dans le nouveau modèle est relativement trop élevé pour permettre de l’utiliser de manière pratique. Dans le deuxième article, nous revenons au problème d’entraînement joint de machines de Boltzmann profondes. Cette fois, au lieu de changer de famille de modèles, nous introduisons un nouveau critère d’entraînement qui donne naissance aux machines de Boltzmann profondes à multiples prédictions (MBP-MP). Les MBP-MP sont entraînables en une seule étape et ont un meilleur taux de succès en classification que les MBP classiques. Elles s’entraînent aussi avec des méthodes variationelles standard au lieu de nécessiter un classificateur discriminant pour obtenir un bon taux de succès en classification. Par contre, un des inconvénients de tels modèles est leur incapacité de générer deséchantillons, mais ceci n’est pas trop grave puisque la performance de classification des machines de Boltzmann profondes n’est plus une priorité étant donné les dernières avancées en apprentissage supervisé. Malgré cela, les MBP-MP demeurent intéressantes parce qu’elles sont capable d’accomplir certaines tâches que des modèles purement supervisés ne peuvent pas faire, telles que celle de classifier des données incomplètes ou encore celle de combler intelligemment l’information manquante dans ces données incomplètes. Le travail présenté dans cette thèse s’est déroulé au milieu d’une période de transformations importantes du domaine de l’apprentissage à réseaux neuronaux profonds qui a été déclenchée par la découverte de l’algorithme de “dropout” par Geoffrey Hinton. Dropout rend possible un entraînement purement supervisé d’architectures de propagation unidirectionnel sans être exposé au danger de sur- entraînement. Le troisième article présenté dans cette thèse introduit une nouvelle fonction d’activation spécialement con ̧cue pour aller avec l’algorithme de Dropout. Cette fonction d’activation, appelée maxout, permet l’utilisation de aggrégation multi-canal dans un contexte d’apprentissage purement supervisé. Nous démontrons comment plusieurs tâches de reconnaissance d’objets sont mieux accomplies par l’utilisation de maxout. Pour terminer, sont présentons un vrai cas d’utilisation dans l’industrie pour la transcription d’adresses de maisons à plusieurs chiffres. En combinant maxout avec une nouvelle sorte de couche de sortie pour des réseaux neuronaux de convolution, nous démontrons qu’il est possible d’atteindre un taux de succès comparable à celui des humains sur un ensemble de données coriace constitué de photos prises par les voitures de Google. Ce système a été déployé avec succès chez Google pour lire environ cent million d’adresses de maisons.

Gaussian processes autoencoder for dimensionality reduction

Learning low dimensional manifold from highly nonlinear data of high dimensionality has become increasingly important for discovering intrinsic representation that can be utilized for data visualization and preprocessing. The autoencoder is a powerful dimensionality reduction technique based on minimizing reconstruction error, and it has regained popularity because it has been efficiently used for greedy pretraining of deep neural networks. Compared to Neural Network (NN), the superiority of Gaussian Process (GP) has been shown in model inference, optimization and performance. GP has been successfully applied in nonlinear Dimensionality Reduction (DR) algorithms, such as Gaussian Process Latent Variable Model (GPLVM). In this paper we propose the Gaussian Processes Autoencoder Model (GPAM) for dimensionality reduction by extending the classic NN based autoencoder to GP based autoencoder. More interestingly, the novel model can also be viewed as back constrained GPLVM (BC-GPLVM) where the back constraint smooth function is represented by a GP. Experiments verify the performance of the newly proposed model.

Evolução da nomeação após fortalecimento de relações auditivo-visuais em crianças com deficiência auditiva e implante coclear

Pós-graduação em Psicologia do Desenvolvimento e Aprendizagem - FC

Discriminações condicionais após treino de pareamento consistente de estímulos complexos com atraso

O objetivo do presente estudo foi verificar a formação de discriminações condicionais por meio de um procedimento de treino por pareamento consistente de estímulos complexos com atraso. Na Fase I, com cinco universitários, foi realizado o Pré-treino TX e Treinos de Aquisição AE, BF e DC envolvendo estímulos modelos e de comparação simples. Na Fase II, houve o Pré-treino MN-O/P, os Treinos AB-E/F e AD-C/F, num formato de pareamento consistente. Esses treinos foram intercalados aos Testes de Transitividade e de Equivalência EF/FE, CF/FC, BD/DB, com estímulos modelos complexos e de comparação simples. Na Fase III, os Treinos AB-E/F e AD-C/F foram desmembrados em AB-E, AB-F e AD-C e reaplicados os mesmos testes. Todos os participantes formaram as relações condicionais entre estímulos modelos simples e complexos, embora com variabilidade quanto à exposição aos blocos de tentativas. Houve variabilidade em termos de formação das relações emergentes testadas. Os resultados sugerem que o procedimento utilizado produziu discriminações condicionais e levou à formação de classes de estímulos equivalentes, não obstante a variabilidade referida. A busca de fontes de controle mais eficazes se faz necessária em pesquisas futuras.

Deep learning architecture for gesture recognition

Pós-graduação em Engenharia Mecânica - FEG

Strength training combined with plyometric jumps in adults: sex differences in fat-bone axis adaptations

[EN] Leptin and osteocalcin play a role in the regulation of the fat-bone axis and may be altered by exercise. To determine whether osteocalcin reduces fat mass in humans fed ad libitum and if there is a sex dimorphism in the serum osteocalcin and leptin responses to strength training, we studied 43 male (age 23.9 2.4 yr, mean +/- SD) and 23 female physical education students (age 23.2 +/- 2.7 yr). Subjects were randomly assigned to two groups: training (TG) and control (CG). TG followed a strength combined with plyometric jumps training program during 9 wk, whereas the CG did not train. Physical fitness, body composition (dual-energy X-ray absorptiometry), and serum concentrations of hormones were determined pre- and posttraining. In the whole group of subjects (pretraining), the serum concentration of osteocalcin was positively correlated (r = 0.29-0.42, P < 0.05) with whole body and regional bone mineral content, lean mass, dynamic strength, and serum-free testosterone concentration (r = 0.32). However, osteocalcin was negatively correlated with leptin concentration (r = -0.37), fat mass (r = -0.31), and the percent body fat (r = -0.44). Both sexes experienced similar relative improvements in performance, lean mass (+4-5%), and whole body (+0.78%) and lumbar spine bone mineral content (+1.2-2%) with training. Serum osteocalcin concentration was increased after training by 45 and 27% in men and women, respectively (P < 0.05). Fat mass was not altered by training. Vastus lateralis type II MHC composition at the start of the training program predicted 25% of the osteocalcin increase after training. Serum leptin concentration was reduced with training in women. In summary, while the relative effects of strength training plus plyometric jumps in performance, muscle hypertrophy, and osteogenesis are similar in men and women, serum leptin concentration is reduced only in women. The osteocalcin response to strength training is, in part, modulated by the muscle phenotype (MHC isoform composition). Despite the increase in osteocalcin, fat mass was not reduced.

Exercise training in normobaric hypoxia in endurance runners. I. Improvement in aerobic performance capacity

This study investigates whether a 6-wk intermittent hypoxia training (IHT), designed to avoid reductions in training loads and intensities, improves the endurance performance capacity of competitive distance runners. Eighteen athletes were randomly assigned to train in normoxia [Nor group; n = 9; maximal oxygen uptake (VO2 max) = 61.5 +/- 1.1 ml x kg(-1) x min(-1)] or intermittently in hypoxia (Hyp group; n = 9; VO2 max = 64.2 +/- 1.2 ml x kg(-1) x min(-1)). Into their usual normoxic training schedule, athletes included two weekly high-intensity (second ventilatory threshold) and moderate-duration (24-40 min) training sessions, performed either in normoxia [inspired O2 fraction (FiO2) = 20.9%] or in normobaric hypoxia (FiO2) = 14.5%). Before and after training, all athletes realized 1) a normoxic and hypoxic incremental test to determine VO2 max and ventilatory thresholds (first and second ventilatory threshold), and 2) an all-out test at the pretraining minimal velocity eliciting VO2 max to determine their time to exhaustion (T(lim)) and the parameters of O2 uptake (VO2) kinetics. Only the Hyp group significantly improved VO2 max (+5% at both FiO2, P < 0.05), without changes in blood O2-carrying capacity. Moreover, T(lim) lengthened in the Hyp group only (+35%, P < 0.001), without significant modifications of VO2 kinetics. Despite similar training load, the Nor group displayed no such improvements, with unchanged VO2 max (+1%, nonsignificant), T(lim) (+10%, nonsignificant), and VO2 kinetics. In addition, T(lim) improvements in the Hyp group were not correlated with concomitant modifications of other parameters, including VO2 max or VO2 kinetics. The present IHT model, involving specific high-intensity and moderate-duration hypoxic sessions, may potentialize the metabolic stimuli of training in already trained athletes and elicit peripheral muscle adaptations, resulting in increased endurance performance capacity.

Tracking memory's trace

There is strong converging evidence that the intermediate and medial part of the hyperstriatum ventrale of the chick brain is a memory store for information acquired through the learning process of imprinting. Neurons in this memory system come, through imprinting, to respond selectively to the imprinting stimulus (IS) neurons and so possess the properties of a memory trace. Therefore, the responses of the intermediate and medial part of the hyperstriatum ventrale neurons to a visual imprinting stimulus were determined before, during, and after training. Of the total recorded population, the proportions of IS neurons shortly after each of two 1-h training sessions were significantly higher (approximately 2 times) than the pretraining proportion. However, ≈4.5 h later this proportion had fallen significantly and did not differ significantly from the pretraining proportion. Nevertheless, ≈21.5 h after the end of training, the proportion of IS neurons was at its highest (approximately 3 times the pretraining level). No significant fluctuations occurred in the proportions of neurons responding to the alternative stimulus. In addition, nonmonotonic changes were found commonly in the activity of 230 of the neurons tracked individually from before training to shortly after the end of training. Thus the pattern of change in responsiveness both at the population level and at the level of individual neurons was highly nonmonotonic. Such a pattern of change is not consistent with simple models of memory based on synaptic strengthening to asymptote. A model is proposed that accounts for the changes in the population responses to the imprinting stimulus in terms of changes in the responses of individual neurons.

Strength and endurance training lead to different post exercise glucose profiles in diabetic participants using a continuous subcutaneous glucose monitoring system

Background Although both strength training (ST) and endurance training (ET) seem to be beneficial in type 2 diabetes mellitus (T2D), little is known about post-exercise glucose profiles. The objective of the study was to report changes in blood glucose (BG) values after a 4-month ET and ST programme now that a device for continuous glucose monitoring has become available. Materials and methods Fifteen participants, comprising four men age 56.5 +/- 0.9 years and 11 women age 57.4 +/- 0.9 years with T2D, were monitored with the MiniMed (Northridge, CA, USA) continuous glucose monitoring system (CGMS) for 48 h before and after 4 months of ET or ST. The ST consisted of three sets at the beginning, increasing to six sets per week at the end of the training period, including all major muscle groups and ET performed with an intensity of maximal oxygen uptake of 60% and a volume beginning at 15 min and advancing to a maximum of 30 min three times a week. Results A total of 17 549 single BG measurements pretraining (619.7 +/- 39.8) and post-training (550.3 +/- 30.1) were recorded, correlating to an average of 585 +/- 25.3 potential measurements per participant at the beginning and at the end of the study. The change in BG-value between the beginning (132 mg dL(-1)) and the end (118 mg dL(-1)) for all participants was significant (P = 0.028). The improvement in BG-value for the ST programme was significant (P = 0.02) but for the ET no significant change was measured (P = 0.48). Glycaemic control improved in the ST group and the mean BG was reduced by 15.6% (Cl 3-25%). Conclusion In conclusion, the CGMS may be a useful tool in monitoring improvements in glycaemic control after different exercise programmes. Additionally, the CGMS may help to identify asymptomatic hypoglycaemia or hyperglycaemia after training programmes.

Implicit learning is not subject to blocking

Explicit (aware) learning has been shown to evidence certain characteristics, such as extinction, blocking, occasion setting, and reliance on context. These characteristics have not been assessed in implicit (unaware) learning. The current study investigated whether implicit learning is subject to blocking. Participants completed a cued reaction time task, where they watched rapid presentations of a random sequence of 8 pairs of shapes, and responded to two target shapes. One target was always preceded by a cue. The experimental group completed a pretraining phase where half the cue, one shape, was followed by the target. Both experimental and control groups completed a training phase where both elements of the cue, two shapes, were followed by the target. Both aware and unaware participants evidenced learning, whereby responding was faster for cued than uncued targets. Aware participants in the experimental group responded faster to targets preceded by the pretrained element than by the other element of the cue. Control and unaware experimental participants were faster to respond to targets preceded by either element of the cue. As blocking was only evident in aware participants, but implicit learning was observed in all participants, it is concluded that implicit learning is not subject to blocking.

Improving the management of behaviour that challenges associated with dementia in care homes:protocol for pharmacy-health psychology intervention feasibility study

INTRODUCTION: The inappropriate use of antipsychotics in people with dementia for behaviour that challenges is associated with an estimated 1800 deaths annually. However, solely focusing on antipsychotics may transfer prescribing to other equally dangerous psychotropics. Little is known about the role of pharmacists in the management of psychotropics used to treat behaviours that challenge. This research aims to determine whether it is feasible to implement and measure the effectiveness of a combined pharmacy-health psychology intervention incorporating a medication review and staff training package to limit the prescription of psychotropics to manage behaviour that challenges in care home residents with dementia. METHODS/ANALYSIS: 6 care homes within the West Midlands will be recruited. People with dementia receiving medication for behaviour that challenges, or their personal consultee, will be approached regarding participation. Medication used to treat behaviour that challenges will be reviewed by the pharmacist, in collaboration with the general practitioner (GP), person with dementia and carer. The behavioural intervention consists of a training package for care home staff and GPs promoting person-centred care and treating behaviours that challenge as an expression of unmet need. The primary outcome measure is the Neuropsychiatric Inventory-Nursing Home version (NPI-NH). Other outcomes include quality of life (EQ-5D and DEMQoL), cognition (sMMSE), health economic (CSRI) and prescribed medication including whether recommendations were implemented. Outcome data will be collected at 6 weeks, and 3 and 6 months. Pretraining and post-training interviews will explore stakeholders' expectations and experiences of the intervention. Data will be used to estimate the sample size for a definitive study. ETHICS/DISSEMINATION: The project has received a favourable opinion from the East Midlands REC (15/EM/3014). If potential participants lack capacity, a personal consultee will be consulted regarding participation in line with the Mental Capacity Act. Results will be published in peer-reviewed journals and presented at conferences.