The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network

Sample complexity results from computational learning theory, when applied to neural network learning for pattern classification problems, suggest that for good generalization performance the number of training examples should grow at least linearly with the number of adjustable parameters in the network. Results in this paper show that if a large neural network is used for a pattern classification problem and the learning algorithm finds a network with small weights that has small squared error on the training patterns, then the generalization performance depends on the size of the weights rather than the number of weights. For example, consider a two-layer feedforward network of sigmoid units, in which the sum of the magnitudes of the weights associated with each unit is bounded by A and the input dimension is n. We show that the misclassification probability is no more than a certain error estimate (that is related to squared error on the training set) plus A3 √((log n)/m) (ignoring log A and log m factors), where m is the number of training patterns. This may explain the generalization performance of neural networks, particularly when the number of training examples is considerably smaller than the number of weights. It also supports heuristics (such as weight decay and early stopping) that attempt to keep the weights small during training. The proof techniques appear to be useful for the analysis of other pattern classifiers: when the input domain is a totally bounded metric space, we use the same approach to give upper bounds on misclassification probability for classifiers with decision boundaries that are far from the training examples.

Estimation of disability weights on malignant neoplasms in Shandong province [山东省现患恶性肿瘤残疾权重的测量]

Objective To determine stage-specific and average disability weights (DWs) of malignant neoplasm and provide support and evidence for study on burden of cancer and policy development in Shandong province. Methods Health status of each cancer patient identified during the cancer prevalence survey in Shandong, 2007 was investigated. In line with the GBD methodology in estimating DWs, the disability extent of every case was classified and evaluated according to the Six-class Disability Classification version and then the stage-specific weights and average DWs with their 95 % confidence intervals were calculated, using SAS software. Results A total of 11 757 cancer cases were investigated and evaluated. DWs of specific stage of therapy, remission, metastasis and terminal of all cancers were 0.310, 0.218, 0.450 and 0.653 respectively. The average DW of all cancers was 0.317(95 % CI:0.312-0.321). Weights of different stage and different cancer varied significantly, while no significant differences were found between males and females. DWs were found higher (>0.4) for liver cancer, bone cancer, lymphoma and pancreas cancer. Lower DWs (<0.3) were found for breast cancer, cervix uteri, corpus uteri, ovarian cancer, larynx cancer, mouth and oropharynx cancer. Conclusion Stage-specific and average DWs for various cancers were estimated based on a large sample size survey. The average DWs of 0.317 for all cancers indicated that 1/3 healthy year lost for each survived life year of them. The difference of DWs between different cancer and stage provide scientific evidence for cancer prevention strategy development. Abstract in Chinese 目的 测算各种恶性肿瘤的分病程残疾权重和平均残疾权重,为山东省恶性肿瘤疾病负担研究及肿瘤防治对策制定提供参考依据. 方法 在山东省2007年恶性肿瘤现患调查中对所有恶性肿瘤患者的健康状况进行调查,参考全球疾病负担研究的方法 ,利用六级社会功能分级标准对患者残疾状况进行分级和赋值,分别计算20种恶性肿瘤的分病程残疾权重和平均残疾权重及其95%CI. 结果 共调查恶性肿瘤患者11757例,所有恶性肿瘤治疗期、恢复期、转移期和晚期的残疾权重分别为0.310、0.218、0.450和0.653,平均残疾权重为0.317(95%CI:0.312～0.321).不同恶性肿瘤和不同病程阶段的残疾权重差别显著,性别间差异无统计学意义.肝癌、骨癌、淋巴瘤和胰腺癌平均残疾权重较高(>0.4),乳腺癌、子宫体癌、子宫颈癌、卵巢癌、喉癌和口咽部癌症相对较低(<0.3). 结论 山东省恶性肿瘤平均残疾权重为0.317,即恶性肿瘤患者每存活1年平均损失近1/3个健康生命年;不同恶性肿瘤和不同病程阶段的残疾权重差别为肿瘤防治对策的制定具有重要意义.

Common values in assessing health outcomes from disease and injury: Disability weights measurement study for the Global Burden of Disease Study 2010

BACKGROUND Measurement of the global burden of disease with disability-adjusted life-years (DALYs) requires disability weights that quantify health losses for all non-fatal consequences of disease and injury. There has been extensive debate about a range of conceptual and methodological issues concerning the definition and measurement of these weights. Our primary objective was a comprehensive re-estimation of disability weights for the Global Burden of Disease Study 2010 through a large-scale empirical investigation in which judgments about health losses associated with many causes of disease and injury were elicited from the general public in diverse communities through a new, standardised approach. METHODS We surveyed respondents in two ways: household surveys of adults aged 18 years or older (face-to-face interviews in Bangladesh, Indonesia, Peru, and Tanzania; telephone interviews in the USA) between Oct 28, 2009, and June 23, 2010; and an open-access web-based survey between July 26, 2010, and May 16, 2011. The surveys used paired comparison questions, in which respondents considered two hypothetical individuals with different, randomly selected health states and indicated which person they regarded as healthier. The web survey added questions about population health equivalence, which compared the overall health benefits of different life-saving or disease-prevention programmes. We analysed paired comparison responses with probit regression analysis on all 220 unique states in the study. We used results from the population health equivalence responses to anchor the results from the paired comparisons on the disability weight scale from 0 (implying no loss of health) to 1 (implying a health loss equivalent to death). Additionally, we compared new disability weights with those used in WHO's most recent update of the Global Burden of Disease Study for 2004. FINDINGS 13,902 individuals participated in household surveys and 16,328 in the web survey. Analysis of paired comparison responses indicated a high degree of consistency across surveys: correlations between individual survey results and results from analysis of the pooled dataset were 0·9 or higher in all surveys except in Bangladesh (r=0·75). Most of the 220 disability weights were located on the mild end of the severity scale, with 58 (26%) having weights below 0·05. Five (11%) states had weights below 0·01, such as mild anaemia, mild hearing or vision loss, and secondary infertility. The health states with the highest disability weights were acute schizophrenia (0·76) and severe multiple sclerosis (0·71). We identified a broad pattern of agreement between the old and new weights (r=0·70), particularly in the moderate-to-severe range. However, in the mild range below 0·2, many states had significantly lower weights in our study than previously. INTERPRETATION This study represents the most extensive empirical effort as yet to measure disability weights. By contrast with the popular hypothesis that disability assessments vary widely across samples with different cultural environments, we have reported strong evidence of highly consistent results.

Generalizing the use of geographical weights in biodiversity modelling

Aim Determining how ecological processes vary across space is a major focus in ecology. Current methods that investigate such effects remain constrained by important limiting assumptions. Here we provide an extension to geographically weighted regression in which local regression and spatial weighting are used in combination. This method can be used to investigate non-stationarity and spatial-scale effects using any regression technique that can accommodate uneven weighting of observations, including machine learning. Innovation We extend the use of spatial weights to generalized linear models and boosted regression trees by using simulated data for which the results are known, and compare these local approaches with existing alternatives such as geographically weighted regression (GWR). The spatial weighting procedure (1) explained up to 80% deviance in simulated species richness, (2) optimized the normal distribution of model residuals when applied to generalized linear models versus GWR, and (3) detected nonlinear relationships and interactions between response variables and their predictors when applied to boosted regression trees. Predictor ranking changed with spatial scale, highlighting the scales at which different species–environment relationships need to be considered. Main conclusions GWR is useful for investigating spatially varying species–environment relationships. However, the use of local weights implemented in alternative modelling techniques can help detect nonlinear relationships and high-order interactions that were previously unassessed. Therefore, this method not only informs us how location and scale influence our perception of patterns and processes, it also offers a way to deal with different ecological interpretations that can emerge as different areas of spatial influence are considered during model fitting.

Maternal and Paternal Genomes Differentially Affect Myofibre Characteristics and Muscle Weights of Bovine Fetuses at Midgestation

Postnatal myofibre characteristics and muscle mass are largely determined during fetal development and may be significantly affected by epigenetic parent-of-origin effects. However, data on such effects in prenatal muscle development that could help understand unexplained variation in postnatal muscle traits are lacking. In a bovine model we studied effects of distinct maternal and paternal genomes, fetal sex, and non-genetic maternal effects on fetal myofibre characteristics and muscle mass. Data from 73 fetuses (Day153, 54% term) of four genetic groups with purebred and reciprocal cross Angus and Brahman genetics were analyzed using general linear models. Parental genomes explained the greatest proportion of variation in myofibre size of Musculus semitendinosus (80-96%) and in absolute and relative weights of M. supraspinatus, M. longissimus dorsi, M. quadriceps femoris and M. semimembranosus (82-89% and 56-93%, respectively). Paternal genome in interaction with maternal genome (P<0.05) explained most genetic variation in cross sectional area (CSA) of fast myotubes (68%), while maternal genome alone explained most genetic variation in CSA of fast myofibres (93%, P<0.01). Furthermore, maternal genome independently (M. semimembranosus, 88%, P<0.0001) or in combination (M. supraspinatus, 82%; M. longissimus dorsi, 93%; M. quadriceps femoris, 86%) with nested maternal weight effect (5-6%, P<0.05), was the predominant source of variation for absolute muscle weights. Effects of paternal genome on muscle mass decreased from thoracic to pelvic limb and accounted for all (M. supraspinatus, 97%, P<0.0001) or most (M. longissimus dorsi, 69%, P<0.0001; M. quadriceps femoris, 54%, P<0.001) genetic variation in relative weights. An interaction between maternal and paternal genomes (P<0.01) and effects of maternal weight (P<0.05) on expression of H19, a master regulator of an imprinted gene network, and negative correlations between H19 expression and fetal muscle mass (P<0.001), suggested imprinted genes and miRNA interference as mechanisms for differential effects of maternal and paternal genomes on fetal muscle.

Solving MDPs using two-timescale simulated annealing with multiplicative weights

We develop extensions of the Simulated Annealing with Multiplicative Weights (SAMW) algorithm that proposed a method of solution of Finite-Horizon Markov Decision Processes (FH-MDPs). The extensions developed are in three directions: a) Use of the dynamic programming principle in the policy update step of SAMW b) A two-timescale actor-critic algorithm that uses simulated transitions alone, and c) Extending the algorithm to the infinite-horizon discounted-reward scenario. In particular, a) reduces the storage required from exponential to linear in the number of actions per stage-state pair. On the faster timescale, a 'critic' recursion performs policy evaluation while on the slower timescale an 'actor' recursion performs policy improvement using SAMW. We give a proof outlining convergence w.p. 1 and show experimental results on two settings: semiconductor fabrication and flow control in communication networks.

Practical Proposals for Specifying k-Nearest Neighbours Weights Matrices

In this article we introduce and evaluate testing procedures for specifying the number k of nearest neighbours in the weights matrix of spatial econometric models. The spatial J-test is used for specification search. Two testing procedures are suggested: an increasing neighbours testing procedure and a decreasing neighbours testing procedure. Simulations show that the increasing neighbours testing procedures can be used in large samples to determine k. The decreasing neighbours testing procedure is found to have low power, and is not recommended for use in practice. An empirical example involving house price data is provided to show how to use the testing procedures with real data.