The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network

Sample complexity results from computational learning theory, when applied to neural network learning for pattern classification problems, suggest that for good generalization performance the number of training examples should grow at least linearly with the number of adjustable parameters in the network. Results in this paper show that if a large neural network is used for a pattern classification problem and the learning algorithm finds a network with small weights that has small squared error on the training patterns, then the generalization performance depends on the size of the weights rather than the number of weights. For example, consider a two-layer feedforward network of sigmoid units, in which the sum of the magnitudes of the weights associated with each unit is bounded by A and the input dimension is n. We show that the misclassification probability is no more than a certain error estimate (that is related to squared error on the training set) plus A3 √((log n)/m) (ignoring log A and log m factors), where m is the number of training patterns. This may explain the generalization performance of neural networks, particularly when the number of training examples is considerably smaller than the number of weights. It also supports heuristics (such as weight decay and early stopping) that attempt to keep the weights small during training. The proof techniques appear to be useful for the analysis of other pattern classifiers: when the input domain is a totally bounded metric space, we use the same approach to give upper bounds on misclassification probability for classifiers with decision boundaries that are far from the training examples.

A neural network enhanced generalized minimum variance self-tuning proportional, integral and derivative control algorithm for complex dynamic systems

A neural network enhanced proportional, integral and derivative (PID) controller is presented that combines the attributes of neural network learning with a generalized minimum-variance self-tuning control (STC) strategy. The neuro PID controller is structured with plant model identification and PID parameter tuning. The plants to be controlled are approximated by an equivalent model composed of a simple linear submodel to approximate plant dynamics around operating points, plus an error agent to accommodate the errors induced by linear submodel inaccuracy due to non-linearities and other complexities. A generalized recursive least-squares algorithm is used to identify the linear submodel, and a layered neural network is used to detect the error agent in which the weights are updated on the basis of the error between the plant output and the output from the linear submodel. The procedure for controller design is based on the equivalent model, and therefore the error agent is naturally functioned within the control law. In this way the controller can deal not only with a wide range of linear dynamic plants but also with those complex plants characterized by severe non-linearity, uncertainties and non-minimum phase behaviours. Two simulation studies are provided to demonstrate the effectiveness of the controller design procedure.

Bayesian network classifiers: Beyond classification accuracy

This work proposes and discusses an approach for inducing Bayesian classifiers aimed at balancing the tradeoff between the precise probability estimates produced by time consuming unrestricted Bayesian networks and the computational efficiency of Naive Bayes (NB) classifiers. The proposed approach is based on the fundamental principles of the Heuristic Search Bayesian network learning. The Markov Blanket concept, as well as a proposed ""approximate Markov Blanket"" are used to reduce the number of nodes that form the Bayesian network to be induced from data. Consequently, the usually high computational cost of the heuristic search learning algorithms can be lessened, while Bayesian network structures better than NB can be achieved. The resulting algorithms, called DMBC (Dynamic Markov Blanket Classifier) and A-DMBC (Approximate DMBC), are empirically assessed in twelve domains that illustrate scenarios of particular interest. The obtained results are compared with NB and Tree Augmented Network (TAN) classifiers, and confinn that both proposed algorithms can provide good classification accuracies and better probability estimates than NB and TAN, while being more computationally efficient than the widely used K2 Algorithm.

A hybrid neural network classifier combining ordered fuzzy ARTMAP and the dynamic decay adjustment algorithm

This paper presents a novel conflict-resolving neural network classifier that combines the ordering algorithm, fuzzy ARTMAP (FAM), and the dynamic decay adjustment (DDA) algorithm, into a unified framework. The hybrid classifier, known as Ordered FAMDDA, applies the DDA algorithm to overcome the limitations of FAM and ordered FAM in achieving a good generalization/performance. Prior to network learning, the ordering algorithm is first used to identify a fixed order of training patterns. The main aim is to reduce and/or avoid the formation of overlapping prototypes of different classes in FAM during learning. However, the effectiveness of the ordering algorithm in resolving overlapping prototypes of different classes is compromised when dealing with complex datasets. Ordered FAMDDA not only is able to determine a fixed order of training patterns for yielding good generalization, but also is able to reduce/resolve overlapping regions of different classes in the feature space for minimizing misclassification during the network learning phase. To illustrate the effectiveness of Ordered FAMDDA, a total of ten benchmark datasets are experimented. The results are analyzed and compared with those from FAM and Ordered FAM. The outcomes demonstrate that Ordered FAMDDA, in general, outperforms FAM and Ordered FAM in tackling pattern classification problems.

An enhanced fuzzy min-max neural network for pattern classification

An enhanced fuzzy min-max (EFMM) network is proposed for pattern classification in this paper. The aim is to overcome a number of limitations of the original fuzzy min-max (FMM) network and improve its classification performance. The key contributions are three heuristic rules to enhance the learning algorithm of FMM. First, a new hyperbox expansion rule to eliminate the overlapping problem during the hyperbox expansion process is suggested. Second, the existing hyperbox overlap test rule is extended to discover other possible overlapping cases. Third, a new hyperbox contraction rule to resolve possible overlapping cases is provided. Efficacy of EFMM is evaluated using benchmark data sets and a real medical diagnosis task. The results are better than those from various FMM-based models, support vector machine-based, Bayesian-based, decision tree-based, fuzzy-based, and neural-based classifiers. The empirical findings show that the newly introduced rules are able to realize EFMM as a useful model for undertaking pattern classification problems.

The role of the market sub-system and the socio-technical sub-system in innovation and firm performance : a dynamic capabilities approach

Organizational learning has been studied as a key factor in firm performance and internationalization. Moving beyond the past emphasis on market learning, we develop a more complete explanation of learning, its relationship to innovation, and their joint effect on early internationalization. We theorize that, driven by the founders’ international vision, early internationalizing firms employ a dual subsystem of dynamic capabilities: a market subsystem consisting of market-focused learning capability and marketing capability, and a socio-technical subsystem comprised of network learning capability and internally focused learning capability. We argue that innovation mediates the proposed relationship between the dynamic capability structure and early internationalization. We conduct case studies to develop the conceptual framework and test it in a field survey of early internationalizing firms from Australia and the United States. Our findings indicate a complex interplay of capabilities driving innovation and early internationalization. We provide theoretical and practical implications and offer insights for future research.

Clustering and visualization of failure modes using an evolving tree

Despite the popularity of Failure Mode and Effect Analysis (FMEA) in a wide range of industries, two well-known shortcomings are the complexity of the FMEA worksheet and its intricacy of use. To the best of our knowledge, the use of computation techniques for solving the aforementioned shortcomings is limited. As such, the idea of clustering and visualization pertaining to the failure modes in FMEA is proposed in this paper. A neural network visualization model with an incremental learning feature, i.e., the evolving tree (ETree), is adopted to allow the failure modes in FMEA to be clustered and visualized as a tree structure. In addition, the ideas of risk interval and risk ordering for different groups of failure modes are proposed to allow the failure modes to be ordered, analyzed, and evaluated in groups. The main advantages of the proposed method lie in its ability to transform failure modes in a complex FMEA worksheet to a tree structure for better visualization, while maintaining the risk evaluation and ordering features. It can be applied to the conventional FMEA methodology without requiring additional information or data. A real world case study in the edible bird nest industry in Sarawak (Borneo Island) is used to evaluate the usefulness of the proposed method. The experiments show that the failure modes in FMEA can be effectively visualized through the tree structure. A discussion with FMEA users engaged in the case study indicates that such visualization is helpful in comprehending and analyzing the respective failure modes, as compared with those in an FMEA table. The resulting tree structure, together with risk interval and risk ordering, provides a quick and easily understandable framework to elucidate important information from complex FMEA forms; therefore facilitating the decision-making tasks by FMEA users. The significance of this study is twofold, viz., the use of a computational visualization approach to tackling two well-known shortcomings of FMEA; and the use of ETree as an effective neural network learning paradigm to facilitate FMEA implementations. These findings aim to spearhead the potential adoption of FMEA as a useful and usable risk evaluation and management tool by the wider community.

Comparative study between RBF and radial-PPS neural networks

The study of function approximation is motivated by the human limitation and inability to register and manipulate with exact precision the behavior variations of the physical nature of a phenomenon. These variations are referred to as signals or signal functions. Many real world problem can be formulated as function approximation problems and from the viewpoint of artificial neural networks these can be seen as the problem of searching for a mapping that establishes a relationship from an input space to an output space through a process of network learning. Several paradigms of artificial neural networks (ANN) exist. Here we will be investigated a comparative of the ANN study of RBF with radial Polynomial Power of Sigmoids (PPS) in function approximation problems. Radial PPS are functions generated by linear combination of powers of sigmoids functions. The main objective of this paper is to show the advantages of the use of the radial PPS functions in relationship traditional RBF, through adaptive training and ridge regression techniques.

Configuração de posições em uma comunidade epistêmica e sua relação com o sentido da aprendizagem em redes interorganizacionais: estudo de caso no campo da biotecnologia

Na literatura sobre redes interorganizacionais, parte das abordagens de pesquisa afirma que tal tipo de ambiente favorece as condições para a aprendizagem por meio da combinação de capacidade de diferentes membros. Igualmente, existem correntes de estudo que apontam haver ressalvas relacionadas às próprias características dos membros. Esta aparente contradição instigou um questionamento: como a configuração de posições de membros de redes interorganizacionais afeta o sentido que a aprendizagem adquire para os próprios membros? Utilizando-se referências conceituais da área de aprendizagem em redes interorganizacionais, o estudo apresenta os resultados da análise de um caso empírico de uma comunidade epistêmica (rede de cooperação científica formalmente instituída) brasileira no campo da biotecnologia. Adotou-se como método de pesquisa a triangulação de técnicas quantitativas e qualitativas. Inicialmente, utilizou-se a análise quantitativa sociométrica evidenciando a posição do membro de acordo com a centralidade para a aprendizagem e a produção acadêmica. Após essa identificação, foram aplicadas entrevistas semiestruturadas aos membros localizados em diferentes posições, fossem elas de alto valor agregado (membros com alta centralidade) ou marginais (membros com baixa centralidade). A partir de um referencial de análise qualitativo com foco nos conceitos de práticas e gêneros discursivos, foi possível delinear diferentes sentidos sobre a aprendizagem decorrida de acordo com a posição dos membros. A partir dos resultados, evidenciou-se a relação da posição ocupada pelo membro da rede o sentido dado à aprendizagem. Apesar das limitações associadas à pesquisa sobre um caso único que dificulta a possível generalização dos resultados, o estudo abre possibilidades de aprofundar novas pesquisas sobre a relação entre as características dos membros e sua posição com a aprendizagem em redes interorganizacionais.

Shadowing experiences and the extension of communities of practice: A case study of women education managers

Recent developments in workplace learning have focused on relational and social network views of learning that introduce practitioners to the norms, values and assumptions of the workplace as well as the learning processes through which knowledge is acquired. This article reports on a qualitative study of a mentoring programme designed to assist women education managers gain promotion by broadening their networks and stimulating insights into the senior management positions for which they were being prepared. The findings are that members reflexively assess and reassess goals and values to demystify knowledge and resolved cognitive dissonance in these processes. Moreover, this article shows that women participants learn from the networks, and that the networks learn from the participant in a reciprocal and informal way. The article concludes that organizational learning programmes must focus on enabling such networks to flourish.

GABAergic excitation in the basolateral amygdala

GABA-containing interneurons are a diverse population of cells whose primary mode of action in the mature nervous system is inhibition of postsynaptic target neurons. Using paired recordings from parvalbumin-positive interneurons in the basolateral amygdala, we show that, in a subpopulation of interneurons, single action potentials in one interneuron evoke in the postsynaptic interneuron a monosynaptic inhibitory synaptic current, followed by a disynaptic excitatory glutamatergic synaptic current. Interneuron-evoked glutamatergic events were blocked by antagonists of either AMPA/kainate or GABA(A) receptors, and could be seen concurrently in both presynaptic and postsynaptic interneurons. These results show that single action potentials in a GABAergic interneuron can drive glutamatergic principal neurons to threshold, resulting in both feedforward and feedback excitation. In interneuron pairs that both receive glutamatergic inputs after an interneuron spike, electrical coupling and bidirectional GABAergic connections occur with a higher probability relative to other interneuron pairs. We propose that this form of GABAergic excitation provides a means for the reliable and specific recruitment of homogeneous interneuron networks in the basal amygdala.

Bayesian invariant measurements of generalisation for discrete distributions

Neural network learning rules can be viewed as statistical estimators. They should be studied in Bayesian framework even if they are not Bayesian estimators. Generalisation should be measured by the divergence between the true distribution and the estimated distribution. Information divergences are invariant measurements of the divergence between two distributions. The posterior average information divergence is used to measure the generalisation ability of a network. The optimal estimators for multinomial distributions with Dirichlet priors are studied in detail. This confirms that the definition is compatible with intuition. The results also show that many commonly used methods can be put under this unified framework, by assume special priors and special divergences.

Bayesian invariant measurements of generalisation for continuous distributions

A family of measurements of generalisation is proposed for estimators of continuous distributions. In particular, they apply to neural network learning rules associated with continuous neural networks. The optimal estimators (learning rules) in this sense are Bayesian decision methods with information divergence as loss function. The Bayesian framework guarantees internal coherence of such measurements, while the information geometric loss function guarantees invariance. The theoretical solution for the optimal estimator is derived by a variational method. It is applied to the family of Gaussian distributions and the implications are discussed. This is one in a series of technical reports on this topic; it generalises the results of ¸iteZhu95:prob.discrete to continuous distributions and serve as a concrete example of a larger picture ¸iteZhu95:generalisation.

Information geometric measurements of generalisation

Neural networks can be regarded as statistical models, and can be analysed in a Bayesian framework. Generalisation is measured by the performance on independent test data drawn from the same distribution as the training data. Such performance can be quantified by the posterior average of the information divergence between the true and the model distributions. Averaging over the Bayesian posterior guarantees internal coherence; Using information divergence guarantees invariance with respect to representation. The theory generalises the least mean squares theory for linear Gaussian models to general problems of statistical estimation. The main results are: (1)~the ideal optimal estimate is always given by average over the posterior; (2)~the optimal estimate within a computational model is given by the projection of the ideal estimate to the model. This incidentally shows some currently popular methods dealing with hyperpriors are in general unnecessary and misleading. The extension of information divergence to positive normalisable measures reveals a remarkable relation between the dlt dual affine geometry of statistical manifolds and the geometry of the dual pair of Banach spaces Ld and Ldd. It therefore offers conceptual simplification to information geometry. The general conclusion on the issue of evaluating neural network learning rules and other statistical inference methods is that such evaluations are only meaningful under three assumptions: The prior P(p), describing the environment of all the problems; the divergence Dd, specifying the requirement of the task; and the model Q, specifying available computing resources.