Improving the performance of model-order selection criteria by partial-model selection search

The traditional searching method for model-order selection in linear regression is a nested full-parameters-set searching procedure over the desired orders, which we call full-model order selection. On the other hand, a method for model-selection searches for the best sub-model within each order. In this paper, we propose using the model-selection searching method for model-order selection, which we call partial-model order selection. We show by simulations that the proposed searching method gives better accuracies than the traditional one, especially for low signal-to-noise ratios over a wide range of model-order selection criteria (both information theoretic based and bootstrap-based). Also, we show that for some models the performance of the bootstrap-based criterion improves significantly by using the proposed partial-model selection searching method. Index Terms— Model order estimation, model selection, information theoretic criteria, bootstrap 1. INTRODUCTION Several model-order selection criteria can be applied to find the optimal order. Some of the more commonly used information theoretic-based procedures include Akaike’s information criterion (AIC) [1], corrected Akaike (AICc) [2], minimum description length (MDL) [3], normalized maximum likelihood (NML) [4], Hannan-Quinn criterion (HQC) [5], conditional model-order estimation (CME) [6], and the efficient detection criterion (EDC) [7]. From a practical point of view, it is difficult to decide which model order selection criterion to use. Many of them perform reasonably well when the signal-to-noise ratio (SNR) is high. The discrepancies in their performance, however, become more evident when the SNR is low. In those situations, the performance of the given technique is not only determined by the model structure (say a polynomial trend versus a Fourier series) but, more importantly, by the relative values of the parameters within the model. This makes the comparison between the model-order selection algorithms difficult as within the same model with a given order one could find an example for which one of the methods performs favourably well or fails [6, 8]. Our aim is to improve the performance of the model order selection criteria in cases where the SNR is low by considering a model-selection searching procedure that takes into account not only the full-model order search but also a partial model order search within the given model order. Understandably, the improvement in the performance of the model order estimation is at the expense of additional computational complexity.

Model-order selection in Zernike polynomial expansion of corneal surfaces using the efficient detection criterion

Corneal-height data are typically measured with videokeratoscopes and modeled using a set of orthogonal Zernike polynomials. We address the estimation of the number of Zernike polynomials, which is formalized as a model-order selection problem in linear regression. Classical information-theoretic criteria tend to overestimate the corneal surface due to the weakness of their penalty functions, while bootstrap-based techniques tend to underestimate the surface or require extensive processing. In this paper, we propose to use the efficient detection criterion (EDC), which has the same general form of information-theoretic-based criteria, as an alternative to estimating the optimal number of Zernike polynomials. We first show, via simulations, that the EDC outperforms a large number of information-theoretic criteria and resampling-based techniques. We then illustrate that using the EDC for real corneas results in models that are in closer agreement with clinical expectations and provides means for distinguishing normal corneal surfaces from astigmatic and keratoconic surfaces.

Model selection and error estimation

We study model selection strategies based on penalized empirical loss minimization. We point out a tight relationship between error estimation and data-based complexity penalization: any good error estimate may be converted into a data-based penalty function and the performance of the estimate is governed by the quality of the error estimate. We consider several penalty functions, involving error estimates on independent test data, empirical VC dimension, empirical VC entropy, and margin-based quantities. We also consider the maximal difference between the error on the first half of the training data and the second half, and the expected maximal discrepancy, a closely related capacity estimate that can be calculated by Monte Carlo integration. Maximal discrepancy penalty functions are appealing for pattern classification problems, since their computation is equivalent to empirical risk minimization over the training data with some labels flipped.

Margin-adaptive model selection in statistical learning

A classical condition for fast learning rates is the margin condition, first introduced by Mammen and Tsybakov. We tackle in this paper the problem of adaptivity to this condition in the context of model selection, in a general learning framework. Actually, we consider a weaker version of this condition that allows one to take into account that learning within a small model can be much easier than within a large one. Requiring this “strong margin adaptivity” makes the model selection problem more challenging. We first prove, in a general framework, that some penalization procedures (including local Rademacher complexities) exhibit this adaptivity when the models are nested. Contrary to previous results, this holds with penalties that only depend on the data. Our second main result is that strong margin adaptivity is not always possible when the models are not nested: for every model selection procedure (even a randomized one), there is a problem for which it does not demonstrate strong margin adaptivity.

Relative entropy rate based model selection for linear hybrid system filters of uncertain nonlinear systems

Hybrid system representations have been exploited in a number of challenging modelling situations, including situations where the original nonlinear dynamics are too complex (or too imprecisely known) to be directly filtered. Unfortunately, the question of how to best design suitable hybrid system models has not yet been fully addressed, particularly in the situations involving model uncertainty. This paper proposes a novel joint state-measurement relative entropy rate based approach for design of hybrid system filters in the presence of (parameterised) model uncertainty. We also present a design approach suitable for suboptimal hybrid system filters. The benefits of our proposed approaches are illustrated through design examples and simulation studies.

Physical and linkage maps for Drosophila serrata, a model species for studies of clinal adaptation and sexual selection

Drosophila serrata is a member of the montium group, which contains more than 98 species and until recently was considered a subgroup within the melanogaster group. This Drosophila species is an emerging model system for evolutionary quantitative genetics and has been used in studies of species borders, clinal variation and sexual selection. Despite the importance of D. serrata as a model for evolutionary research, our poor understanding of its genome remains a significant limitation. Here, we provide a first-generation gene-based linkage map and a physical map for this species. Consistent with previous studies of other drosophilids we observed strong conservation of genes within chromosome arms homologous with D. melanogaster but major differences in within-arm synteny. These resources will be a useful complement to ongoing genome sequencing efforts and QTL mapping studies in this species

An expressed sequence tag (EST) library for Drosophila serrata, a model system for sexual selection and climatic adaptation studies

The native Australian fly Drosophila serrata belongs to the highly speciose montium subgroup of the melanogaster species group. It has recently emerged as an excellent model system with which to address a number of important questions, including the evolution of traits under sexual selection and traits involved in climatic adaptation along latitudinal gradients. Understanding the molecular genetic basis of such traits has been limited by a lack of genomic resources for this species. Here, we present the first expressed sequence tag (EST) collection for D. serrata that will enable the identification of genes underlying sexually-selected phenotypes and physiological responses to environmental change and may help resolve controversial phylogenetic relationships within the montium subgroup.

Sample selection in a model of female labour supply : an alternative approach

Suggests an alternative and computationally simpler approach of non-random sampling of labour economics and represents an observed outcome of an individual female′s choice of whether or not to participate in the labour market. Concludes that there is an alternative to the Heckman two-step estimator.

A general model for host plant selection in phytophagous insects

We develop a general theoretical framework for exploring the host plant selection behaviour of herbivorous insects. This model can be used to address a number of questions, including the evolution of specialists, generalists, preference hierarchies, and learning. We use our model to: (i) demonstrate the consequences of the extent to which the reproductive success of a foraging female is limited by the rate at which they find host plants (host limitation) or the number of eggs they carry (egg limitation); (ii) emphasize the different consequences of variation in behaviour before and after landing on (locating) a host (termed pre- and post-alighting, respectively); (iii) show that, in contrast to previous predictions, learning can be favoured in post-alighting behaviour--in particular, individuals can be selected to concentrate oviposition on an abundant low-quality host, whilst ignoring a rare higher-quality host; (iv) emphasize the importance of interactions between mechanisms in favouring specialization or learning.

Towards a model for student selection in clinical psychology

Background Selection of candidates for clinical psychology programmes is arguably the most important decision made in determining the clinical psychology workforce. However, there are few models to inform the development of selection tools to support selection procedures. The study, using a factor analytic structure, has operationalised the model predicting applicants' capabilities. Method Eighty-eight clinical applicants for entry into a postgraduate clinical psychology programme were assessed on a series of tasks measuring eight capabilities: guided reflection, communication skills, ethical decision making, writing, conceptual reasoning, empathy, and awareness of mind and self-observation. Results Factor analysis revealed three capabilities: labelled “awareness” accounting for 35.71% of variance; “reflection” accounting for 20.56%; and “reasoning” accounting for 18.24% of variance. Fourth year grade point average (GPA) did not correlate with performance on any of the selection capabilities other than a weak correlation with performance on the ethics capability. Conclusions Eight selection capabilities are identified for the selection of candidates independent of GPA. While the model is tentative, it is hoped that the findings will stimulate the development and validation of assessment procedures with good predictive validity which will benefit the training of clinical psychologists and, ultimately, effective service delivery.

A hybrid quality function deployment and cybernetic analytic network process model for project manager selection

Identifying appropriate decision criteria and making optimal decisions in a structured way is a complex process. This paper presents an approach for doing this in the form of a hybrid Quality Function Deployment (QFD) and Cybernetic Analytic Network Process (CANP) model for project manager selection. This involves the use of QFD to translate the owner's project management expectations into selection criteria and the CANP to weight the expectations and selection criteria. The supermatrix approach then prioritises the candidates with respect to the overall decision-making goal. A case study is used to demonstrate the use of the model in selecting a renovation project manager. This involves the development of 18 selection criteria in response to the owner's three main expectations of time, cost and quality.