Structure Discovery in Nonparametric Regression through Compositional Kernel Search

Despite its importance, choosing the structural form of the kernel in nonparametric regression remains a black art. We define a space of kernel structures which are built compositionally by adding and multiplying a small number of base kernels. We present a method for searching over this space of structures which mirrors the scientific discovery process. The learned structures can often decompose functions into interpretable components and enable long-range extrapolation on time-series datasets. Our structure search method outperforms many widely used kernels and kernel combination methods on a variety of prediction tasks.

Automatic construction and natural-language description of nonparametric regression models

Copyright © 2014, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. This paper presents the beginnings of an automatic statistician, focusing on regression problems. Our system explores an open-ended space of statistical models to discover a good explanation of a data set, and then produces a detailed report with figures and natural- language text. Our approach treats unknown regression functions non- parametrically using Gaussian processes, which has two important consequences. First, Gaussian processes can model functions in terms of high-level properties (e.g. smoothness, trends, periodicity, changepoints). Taken together with the compositional structure of our language of models this allows us to automatically describe functions in simple terms. Second, the use of flexible nonparametric models and a rich language for composing them in an open-ended manner also results in state- of-the-art extrapolation performance evaluated over 13 real time series data sets from various domains.

Two-stage sequential procedure for nonparametric regression estimation

In nonparametric statistics the functional form of the relationship between the response variable and its associated predictor variables is unspecified but it is assumed to be a smooth function. We develop a procedure for constructing a fixed width confidence interval for the predicted value at a specified point of the independent variable. The optimal sample size for constructing this interval is obtained using a two stage sequential procedure which relies on some asymptotic properties of the Nadaraya--Watson and local linear estimators. Finally, a large scale simulation study demonstrates the applicability of the developed procedure for small and moderate sample sizes. The procedure developed here should find wide applicability since many practical problems which arise in industry involve estimating an unknown function.

Improved combinations of parametric and nonparametric Regression estimators

This paper provides a systematic and unified treatment of the developments in the area of kernel estimation in econometrics and statistics. Both the estimation and hypothesis testing issues are discussed for the nonparametric and semiparametric regression models. A discussion on the choice of windowwidth is also presented.

Application of Local Rank Tests for Nonparametric Regression

Let Y_i = f(x_i) + E_i\ (1\le i\le n) with given covariates x_1\lt x_2\lt \cdots\lt x_n , an unknown regression function f and independent random errors E_i with median zero. It is shown how to apply several linear rank test statistics simultaneously in order to test monotonicity of f in various regions and to identify its local extrema.

Efficiency Bounds for Estimating Linear Functionals of Nonparametric Regression Models with Endogenous Regressors

Consider a nonparametric regression model Y=mu*(X) + e, where the explanatory variables X are endogenous and e satisfies the conditional moment restriction E[e|W]=0 w.p.1 for instrumental variables W. It is well known that in these models the structural parameter mu* is 'ill-posed' in the sense that the function mapping the data to mu* is not continuous. In this paper, we derive the efficiency bounds for estimating linear functionals E[p(X)mu*(X)] and int_{supp(X)}p(x)mu*(x)dx, where p is a known weight function and supp(X) the support of X, without assuming mu* to be well-posed or even identified.

Canonical solution groups of a homomorphism : manipulator kinematics, nonparametric regression and distributed object systems

The kinematic mapping of a rigid open-link manipulator is a homomorphism between Lie groups. The homomorphisrn has solution groups that act on an inverse kinematic solution element. A canonical representation of solution group operators that act on a solution element of three and seven degree-of-freedom (do!) dextrous manipulators is determined by geometric analysis. Seven canonical solution groups are determined for the seven do! Robotics Research K-1207 and Hollerbach arms. The solution element of a dextrous manipulator is a collection of trivial fibre bundles with solution fibres homotopic to the Torus. If fibre solutions are parameterised by a scalar, a direct inverse funct.ion that maps the scalar and Cartesian base space coordinates to solution element fibre coordinates may be defined. A direct inverse pararneterisation of a solution element may be approximated by a local linear map generated by an inverse augmented Jacobian correction of a linear interpolation. The action of canonical solution group operators on a local linear approximation of the solution element of inverse kinematics of dextrous manipulators generates cyclical solutions. The solution representation is proposed as a model of inverse kinematic transformations in primate nervous systems. Simultaneous calibration of a composition of stereo-camera and manipulator kinematic models is under-determined by equi-output parameter groups in the composition of stereo-camera and Denavit Hartenberg (DH) rnodels. An error measure for simultaneous calibration of a composition of models is derived and parameter subsets with no equi-output groups are determined by numerical experiments to simultaneously calibrate the composition of homogeneous or pan-tilt stereo-camera with DH models. For acceleration of exact Newton second-order re-calibration of DH parameters after a sequential calibration of stereo-camera and DH parameters, an optimal numerical evaluation of DH matrix first order and second order error derivatives with respect to a re-calibration error function is derived, implemented and tested. A distributed object environment for point and click image-based tele-command of manipulators and stereo-cameras is specified and implemented that supports rapid prototyping of numerical experiments in distributed system control. The environment is validated by a hierarchical k-fold cross validated calibration to Cartesian space of a radial basis function regression correction of an affine stereo model. Basic design and performance requirements are defined for scalable virtual micro-kernels that broker inter-Java-virtual-machine remote method invocations between components of secure manageable fault-tolerant open distributed agile Total Quality Managed ISO 9000+ conformant Just in Time manufacturing systems.

Application of sequential nonparametric confidence bands in finance

In a nonparametric setting, the functional form of the relationship between the response variable and the associated predictor variables is assumed to be unknown when data is fitted to the model. Non-parametric regression models can be used for the same types of applications such as estimation, prediction, calibration, and optimization that traditional regression models are used for. The main aim of nonparametric regression is to highlight an important structure in the data without any assumptions about the shape of an underlying regression function. Hence the nonparametric approach allows the data to speak for itself. Applications of sequential procedures to a nonparametric regression model at a given point are considered.

The primary goal of sequential analysis is to achieve a given accuracy by using the smallest possible sample sizes. These sequential procedures allow an experimenter to make decisions based on the smallest number of observations without compromising accuracy. In the nonparametric regression model with a random design based on independent and identically distributed pairs of observations (X ,Y ), where the regression function m(x) is given bym(x) = E(Y X = x), estimation of the Nadaraya-Watson kernel estimator (m (x)) NW and local linear kernel estimator (m (x)) LL for the curve m(x) is considered. In order to obtain asymptotic confidence intervals form(x), two stage sequential procedure is used under which some asymptotic properties of Nadaraya-Watson and local linear estimators have been obtained.

The proposed methodology is first tested with the help of simulated data from linear and nonlinear functions. Encouraged by the preliminary findings from simulation results, the proposed method is applied to estimate the nonparametric regression curve of CAPM.

Sequential fixed-width confidence bands for kernel regression estimation

We consider a random design model based on independent and identically distributed (iid) pairs of observations (Xi, Yi), where the regression function m(x) is given by m(x) = E(Yi|Xi = x) with one independent variable. In a nonparametric setting the aim is to produce a reasonable approximation to the unknown function m(x) when we have no precise information about the form of the true density, f(x) of X. We describe an estimation procedure of non-parametric regression model at a given point by some appropriately constructed fixed-width (2d) confidence interval with the confidence coefficient of at least 1−. Here, d(> 0) and 2 (0, 1) are two preassigned values. Fixed-width confidence intervals are developed using both Nadaraya-Watson and local linear kernel estimators of nonparametric regression with data-driven bandwidths.

The sample size was optimized using the purely and two-stage sequential procedure together with asymptotic properties of the Nadaraya-Watson and local linear estimators. A large scale simulation study was performed to compare their coverage accuracy. The numerical results indicate that the confidence bands based on the local linear estimator have the best performance than those constructed by using Nadaraya-Watson estimator. However both estimators are shown to have asymptotically correct coverage properties.