On-line learning of unrealizable tasks

The dynamics of on-line learning is investigated for structurally unrealizable tasks in the context of two-layer neural networks with an arbitrary number of hidden neurons. Within a statistical mechanics framework, a closed set of differential equations describing the learning dynamics can be derived, for the general case of unrealizable isotropic tasks. In the asymptotic regime one can solve the dynamics analytically in the limit of large number of hidden neurons, providing an analytical expression for the residual generalization error, the optimal and critical asymptotic training parameters, and the corresponding prefactor of the generalization error decay.

Noise, regularizers, and unrealizable scenarios in online learning from restricted training sets

We study the dynamics of on-line learning in multilayer neural networks where training examples are sampled with repetition and where the number of examples scales with the number of network weights. The analysis is carried out using the dynamical replica method aimed at obtaining a closed set of coupled equations for a set of macroscopic variables from which both training and generalization errors can be calculated. We focus on scenarios whereby training examples are corrupted by additive Gaussian output noise and regularizers are introduced to improve the network performance. The dependence of the dynamics on the noise level, with and without regularizers, is examined, as well as that of the asymptotic values obtained for both training and generalization errors. We also demonstrate the ability of the method to approximate the learning dynamics in structurally unrealizable scenarios. The theoretical results show good agreement with those obtained by computer simulations.

The theory of on-line learning: a statistical physics approach

In this paper we review recent theoretical approaches for analysing the dynamics of on-line learning in multilayer neural networks using methods adopted from statistical physics. The analysis is based on monitoring a set of macroscopic variables from which the generalisation error can be calculated. A closed set of dynamical equations for the macroscopic variables is derived analytically and solved numerically. The theoretical framework is then employed for defining optimal learning parameters and for analysing the incorporation of second order information into the learning process using natural gradient descent and matrix-momentum based methods. We will also briefly explain an extension of the original framework for analysing the case where training examples are sampled with repetition.

A micro industry with closed energy and water cycles for sustainable rural development

Sustainable development requires combining economic viability with energy and environment conservation and ensuring social benefits. It is conceptualized that for designing a micro industry for sustainable rural industrialization, all these aspects should be integrated right up front. The concept includes; (a) utilization of local produce for value addition in a cluster of villages and enhancing income of the target population; (b) use of renewable energy and total utilization of energy generated by co and trigeneration (combining electric power production with heat utilization for heating and cooling); (c) conservation of water and complete recycling of effluents; (d) total utilization of all wastes for achieving closure towards a zero waste system. Enhanced economic viability and sustainability is achieved by integration of appropriate technologies into the industrial complex. To prove the concept, a model Micro Industrial Complex (MIC) has been set up in a semi arid desert region in Rajasthan, India at village Malunga in Jodhpur district. A biomass powered boiler and steam turbine system is used to generate 100-200 KVA of electric power and high energy steam for heating and cooling processes downstream. The unique feature of the equipment is a 100-150 kW back-pressure steam turbine, utilizing 3-4 tph (tonnes per hour) steam, developed by M/s IB Turbo. The biomass boiler raises steam at about 20 barg 3 tph, which is passed through a turbine to yield about 150 kW of electrical power. The steam let out at a back pressure of 1-3 barg has high exergy and this is passed on as thermal energy (about 2 MW), for use in various applications depending on the local produce and resources. The biomass fuel requirement for the boiler is 0.5-0.75 tph depending on its calorific value. In the current model, the electricity produced is used for running an oil expeller to extract castor oil and the castor cake is used as fuel in the boiler. The steam is used in a Multi Effect Distillation (MED) unit for drinking water production and in a Vapour Absorption Machine (VAM) for cooling, for banana ripening application. Additional steam is available for extraction of herbs such as mint and processing local vegetables. In this paper, we discuss the financial and economic viability of the system and show how the energy, water and materials are completely recycled and how the benefits are directed to the weaker sections of the community.