Techniques for batch reinforcement learning in robotics

This thesis addresses the Batch Reinforcement Learning methods in Robotics. This sub-class of Reinforcement Learning has shown promising results and has been the focus of recent research. Three contributions are proposed that aim to extend the state-of-art methods allowing for a faster and more stable learning process, such as required for learning in Robotics. The Q-learning update-rule is widely applied, since it allows to learn without the presence of a model of the environment. However, this update-rule is transition-based and does not take advantage of the underlying episodic structure of collected batch of interactions. The Q-Batch update-rule is proposed in this thesis, to process experiencies along the trajectories collected in the interaction phase. This allows a faster propagation of obtained rewards and penalties, resulting in faster and more robust learning. Non-parametric function approximations are explored, such as Gaussian Processes. This type of approximators allows to encode prior knowledge about the latent function, in the form of kernels, providing a higher level of exibility and accuracy. The application of Gaussian Processes in Batch Reinforcement Learning presented a higher performance in learning tasks than other function approximations used in the literature. Lastly, in order to extract more information from the experiences collected by the agent, model-learning techniques are incorporated to learn the system dynamics. In this way, it is possible to augment the set of collected experiences with experiences generated through planning using the learned models. Experiments were carried out mainly in simulation, with some tests carried out in a physical robotic platform. The obtained results show that the proposed approaches are able to outperform the classical Fitted Q Iteration.

Rational Function Interpolation of Electromagnetic Transfer Functions of High-Speed Interconnect Systems from Discrete Time-Domain and Frequency-Domain Data

We present new methodologies to generate rational function approximations of broadband electromagnetic responses of linear and passive networks of high-speed interconnects, and to construct SPICE-compatible, equivalent circuit representations of the generated rational functions. These new methodologies are driven by the desire to improve the computational efficiency of the rational function fitting process, and to ensure enhanced accuracy of the generated rational function interpolation and its equivalent circuit representation. Toward this goal, we propose two new methodologies for rational function approximation of high-speed interconnect network responses. The first one relies on the use of both time-domain and frequency-domain data, obtained either through measurement or numerical simulation, to generate a rational function representation that extrapolates the input, early-time transient response data to late-time response while at the same time providing a means to both interpolate and extrapolate the used frequency-domain data. The aforementioned hybrid methodology can be considered as a generalization of the frequency-domain rational function fitting utilizing frequency-domain response data only, and the time-domain rational function fitting utilizing transient response data only. In this context, a guideline is proposed for estimating the order of the rational function approximation from transient data. The availability of such an estimate expedites the time-domain rational function fitting process. The second approach relies on the extraction of the delay associated with causal electromagnetic responses of interconnect systems to provide for a more stable rational function process utilizing a lower-order rational function interpolation. A distinctive feature of the proposed methodology is its utilization of scattering parameters. For both methodologies, the approach of fitting the electromagnetic network matrix one element at a time is applied. It is shown that, with regard to the computational cost of the rational function fitting process, such an element-by-element rational function fitting is more advantageous than full matrix fitting for systems with a large number of ports. Despite the disadvantage that different sets of poles are used in the rational function of different elements in the network matrix, such an approach provides for improved accuracy in the fitting of network matrices of systems characterized by both strongly coupled and weakly coupled ports. Finally, in order to provide a means for enforcing passivity in the adopted element-by-element rational function fitting approach, the methodology for passivity enforcement via quadratic programming is modified appropriately for this purpose and demonstrated in the context of element-by-element rational function fitting of the admittance matrix of an electromagnetic multiport.

A numerical method to solve higher-order fractional differential equations

In this paper, we present a new numerical method to solve fractional differential equations. Given a fractional derivative of arbitrary real order, we present an approximation formula for the fractional operator that involves integer-order derivatives only. With this, we can rewrite FDEs in terms of a classical one and then apply any known technique. With some examples, we show the accuracy of the method.

Improving CEGEP Students’ ESL Accuracy through L1/L2 Synchronous Text-based Screen-Sharing Tasks

Abstract : CEGEPs are now reaping the ‘first fruits’ of the last Educational Reform in Quebec and as a result, ‘English as Second Language’ (ESL) teachers are noticing an improvement in fluency and a seemingly lower level of inhibition when it comes to production skills. However, this output is accompanied by a noticeable lack of accuracy. Keeping in mind that the purpose of language is communication, we need to find a way to reduce the number of basic common errors made by CEGEP ESL students, while maintaining a natural and motivating learning environment. Thanks to recent advances in computer-mediated communication (CMC), we now have the necessary tools to access peer native speakers throughout the world. Although this technology can be used for other language courses, this study explored the potential value of collaboration with native English speakers through the use of synchronous screen-sharing technology, in order to improve CEGEP ESL students’ accuracy in writing. The instrumentation used consisted of a questionnaire, tests, plus documents of collaborative tasks, using the ‘Google for Education’ screen-sharing tool. Fourteen Intermediate/Advanced ESL CEGEP students participated in this study. Despite the positive tendencies revealed, only a prolonged use of the innovative method yielded a significant positive impact. Moreover, a mixed linear regression for the group with more L1 intervention revealed a significant correlation between the number of errors in the task documents and the number of tasks accomplished. Thus, it could be inferred that ESL accuracy improves in proportion to the number of synchronous text-based screen-sharing tasks done with L1 collaboration.

Efficient split eld FDTD analysis of third-order nonlinear materials in two-dimensionally periodic media

In this work the split-field finite-difference time-domain method (SF-FDTD) has been extended for the analysis of two-dimensionally periodic structures with third-order nonlinear media. The accuracy of the method is verified by comparisons with the nonlinear Fourier Modal Method (FMM). Once the formalism has been validated, examples of one- and two-dimensional nonlinear gratings are analysed. Regarding the 2D case, the shifting in resonant waveguides is corroborated. Here, not only the scalar Kerr effect is considered, the tensorial nature of the third-order nonlinear susceptibility is also included. The consideration of nonlinear materials in this kind of devices permits to design tunable devices such as variable band filters. However, the third-order nonlinear susceptibility is usually small and high intensities are needed in order to trigger the nonlinear effect. Here, a one-dimensional CBG is analysed in both linear and nonlinear regime and the shifting of the resonance peaks in both TE and TM are achieved numerically. The application of a numerical method based on the finite- difference time-domain method permits to analyse this issue from the time domain, thus bistability curves are also computed by means of the numerical method. These curves show how the nonlinear effect modifies the properties of the structure as a function of variable input pump field. When taking the nonlinear behaviour into account, the estimation of the electric field components becomes more challenging. In this paper, we present a set of acceleration strategies based on parallel software and hardware solutions.

Evaluation of the GPS Accuracy of Tablets

Augmented Reality (AR) applications often require knowledge of the user’s position in some global coordinate system in order to draw the augmented content to its correct position on the screen. The most common method for coarse positioning is the Global Positioning System (GPS). One of the advantages of GPS is that GPS receivers can be found in almost every modern mobile device. This research was conducted in order to determine the accuracies of different GPS receivers. The tests included seven consumer-grade tablets, three external GPS modules and one professional-grade GPS receiver. All of the devices were tested with both static and mobile measurements. It was concluded that even the cheaper external GPS receivers were notably more accurate than the GPS receivers of the tested tablets. The absolute accuracy of the tablets is difficult to determine from the test results, since the results vary by a large margin between different measurements. The accuracy of the tested tablets in static measurements were between 0.30 meters and 13.75 meters.

EXPLICIT WRITTEN CORRECTIVE FEEDBACK AND LANGUAGE APTITUDE IN SLA: IMPLICATIONS FOR IMPROVEMENT OF LINGUISTIC ACCURACY

Most second language researchers agree that there is a role for corrective feedback in second language writing classes. However, many unanswered questions remain concerning which linguistic features to target and the type and amount of feedback to offer. This study examined two new pieces of writing by 151 learners of English as a Second Language (ESL), in order to investigate the effect of direct and metalinguistic written feedback on errors with the simple past tense, the present perfect tense, dropped pronouns, and pronominal duplication. This inquiry also considered the extent to which learner differences in language-analytic ability (LAA), as measured by the LLAMA F, mediated the effects of these two types of explicit written corrective feedback. Learners in the feedback groups were provided with corrective feedback on two essays, after which learners in all three groups completed two additional writing tasks to determine whether or not the provision of corrective feedback led to greater gains in accuracy compared to no feedback. Both treatment groups, direct and metalinguistic, performed better than the comparison group on new pieces of writing immediately following the treatment sessions, yet direct feedback was more durable than metalinguistic feedback for one structure, the simple past tense. Participants with greater LAA proved more likely to achieve gains in the direct feedback group than in the metalinguistic group, whereas learners with lower LAA benefited more from metalinguistic feedback. Overall, the findings of the present study confirm the results of prior studies that have found a positive role for written corrective feedback in instructed second language acquisition.

The use of mo and cu monochromatic radiations for quantitative phase analysis: study of the accuracy

Cement hydration is a very complex process in which crystalline phases are dissolving in water and after supersaturation hydrated crystalline and amorphous phases precipitate. Great efforts are being made to develop analytical tools to accurately quantify these processes and X-ray Powder Diffraction (XRPD) combined with Rietveld methodology is a suitable tool to quantify these complex mixtures and their time evolutions. However, some problems/drawbacks should be overcome to fully apply it to cement pastes characterization in order to get accurate phase analyses. In order to tackle this issue, a comparison of the Rietveld quantitative phase analyses (RQPA) obtained using Cu-Kα1, Mo-Kα1, and synchrotron strictly monochromatic radiations of three set of mixtures with increasing amounts of a given phase (spiking-method) is presented. The main aim is to test a simple hypothesis: high energy Mo-radiation, combined with high resolution laboratory X-ray powder diffraction optics, could yield more accurate RQPA, for challenging samples, than well-established Cu-radiation procedure(s). Firstly, a series of crystalline inorganic phase mixtures with increasing amounts of an analyte was studied in order to determine if Mo-Kα1 methodology is as robust as the well-established Cu-Kα1 one. Secondly, a series of crystalline organic phase mixtures with increasing amounts of an organic compound was analyzed. This type of mixture can result in transparency problems in reflection and inhomogeneous loading in narrow capillaries for transmission studies. Finally, a third series with variable amorphous content was studied. Limit of detection in Cu-patterns, ~0.2 wt%, are slightly lower than those derived from Mo-patterns, ~0.3 wt%, for similar recording times and limit of quantification for a well crystallized inorganic phase using laboratory powder diffraction was established ~0.10 wt%. From the obtained results it is inferred that RQPA from Mo-Kα1 radiation have slightly better accuracies than those obtained from Cu-Kα1. The results obtained in the previous comparison have been taken into account to obtain accurate RQPA, including the amorphous component with internal standard methodology, of hydrating cement pastes. The final goal of this second study was understanding the early-stage hydration mechanisms of a variety of cementing systems (Ordinary Portland Cement or Belite Alite Ye’elimite cement) as a function of water content, superplasticizer additives and type and content of sulfate source. In order to do so, X-ray powder diffraction data were taken in-situ with the humidity chamber coupled to the Mo-Kα1 powder diffractometer. Some results of this ongoing investigation will be reported and discussed.

POSITIVE FILTERED P_N METHOD FOR LINEAR TRANSPORT EQUATIONS AND THE ASSOCIATED OPTIMIZATION ALGORITHM

We propose a positive, accurate moment closure for linear kinetic transport equations based on a filtered spherical harmonic (FP_N) expansion in the angular variable. The FP_N moment equations are accurate approximations to linear kinetic equations, but they are known to suffer from the occurrence of unphysical, negative particle concentrations. The new positive filtered P_N (FP_N+) closure is developed to address this issue. The FP_N+ closure approximates the kinetic distribution by a spherical harmonic expansion that is non-negative on a finite, predetermined set of quadrature points. With an appropriate numerical PDE solver, the FP_N+ closure generates particle concentrations that are guaranteed to be non-negative. Under an additional, mild regularity assumption, we prove that as the moment order tends to infinity, the FP_N+ approximation converges, in the L2 sense, at the same rate as the FP_N approximation; numerical tests suggest that this assumption may not be necessary. By numerical experiments on the challenging line source benchmark problem, we confirm that the FP_N+ method indeed produces accurate and non-negative solutions. To apply the FP_N+ closure on problems at large temporal-spatial scales, we develop a positive asymptotic preserving (AP) numerical PDE solver. We prove that the propose AP scheme maintains stability and accuracy with standard mesh sizes at large temporal-spatial scales, while, for generic numerical schemes, excessive refinements on temporal-spatial meshes are required. We also show that the proposed scheme preserves positivity of the particle concentration, under some time step restriction. Numerical results confirm that the proposed AP scheme is capable for solving linear transport equations at large temporal-spatial scales, for which a generic scheme could fail. Constrained optimization problems are involved in the formulation of the FP_N+ closure to enforce non-negativity of the FP_N+ approximation on the set of quadrature points. These optimization problems can be written as strictly convex quadratic programs (CQPs) with a large number of inequality constraints. To efficiently solve the CQPs, we propose a constraint-reduced variant of a Mehrotra-predictor-corrector algorithm, with a novel constraint selection rule. We prove that, under appropriate assumptions, the proposed optimization algorithm converges globally to the solution at a locally q-quadratic rate. We test the algorithm on randomly generated problems, and the numerical results indicate that the combination of the proposed algorithm and the constraint selection rule outperforms other compared constraint-reduced algorithms, especially for problems with many more inequality constraints than variables.

Investigating output accuracy for a discrete event simulation model and an agent based simulation model

In this paper, we investigate output accuracy for a Discrete Event Simulation (DES) model and Agent Based Simulation (ABS) model. The purpose of this investigation is to find out which of these simulation techniques is the best one for modelling human reactive behaviour in the retail sector. In order to study the output accuracy in both models, we have carried out a validation experiment in which we compared the results from our simulation models to the performance of a real system. Our experiment was carried out using a large UK department store as a case study. We had to determine an efficient implementation of management policy in the store’s fitting room using DES and ABS. Overall, we have found that both simulation models were a good representation of the real system when modelling human reactive behaviour.