Exponentiated gradient algorithms for conditional random fields and max-margin Markov Networks

Log-linear and maximum-margin models are two commonly-used methods in supervised machine learning, and are frequently used in structured prediction problems. Efficient learning of parameters in these models is therefore an important problem, and becomes a key factor when learning from very large data sets. This paper describes exponentiated gradient (EG) algorithms for training such models, where EG updates are applied to the convex dual of either the log-linear or max-margin objective function; the dual in both the log-linear and max-margin cases corresponds to minimizing a convex function with simplex constraints. We study both batch and online variants of the algorithm, and provide rates of convergence for both cases. In the max-margin case, O(1/ε) EG updates are required to reach a given accuracy ε in the dual; in contrast, for log-linear models only O(log(1/ε)) updates are required. For both the max-margin and log-linear cases, our bounds suggest that the online EG algorithm requires a factor of n less computation to reach a desired accuracy than the batch EG algorithm, where n is the number of training examples. Our experiments confirm that the online algorithms are much faster than the batch algorithms in practice. We describe how the EG updates factor in a convenient way for structured prediction problems, allowing the algorithms to be efficiently applied to problems such as sequence learning or natural language parsing. We perform extensive evaluation of the algorithms, comparing them to L-BFGS and stochastic gradient descent for log-linear models, and to SVM-Struct for max-margin models. The algorithms are applied to a multi-class problem as well as to a more complex large-scale parsing task. In all these settings, the EG algorithms presented here outperform the other methods.

Exponentiated gradient algorithms for large-margin structured classification

We consider the problem of structured classification, where the task is to predict a label y from an input x, and y has meaningful internal structure. Our framework includes supervised training of Markov random fields and weighted context-free grammars as special cases. We describe an algorithm that solves the large-margin optimization problem defined in [12], using an exponential-family (Gibbs distribution) representation of structured objects. The algorithm is efficient—even in cases where the number of labels y is exponential in size—provided that certain expectations under Gibbs distributions can be calculated efficiently. The method for structured labels relies on a more general result, specifically the application of exponentiated gradient updates [7, 8] to quadratic programs.

Adaptive online gradient descent

We study the rates of growth of the regret in online convex optimization. First, we show that a simple extension of the algorithm of Hazan et al eliminates the need for a priori knowledge of the lower bound on the second derivatives of the observed functions. We then provide an algorithm, Adaptive Online Gradient Descent, which interpolates between the results of Zinkevich for linear functions and of Hazan et al for strongly convex functions, achieving intermediate rates between [square root T] and [log T]. Furthermore, we show strong optimality of the algorithm. Finally, we provide an extension of our results to general norms.

Thomas Young’s investigations in gradient-index optics

Purpose: James Clerk Maxwell is usually recognized as being the first, in 1854, to consider using inhomogeneous media in optical systems. However, some fifty years earlier Thomas Young, stimulated by his interest in the optics of the eye and accommodation, had already modeled some applications of gradient-index optics. These applications included using an axial gradient to provide spherical aberration-free optics and a spherical gradient to describe the optics of the atmosphere and the eye lens. We evaluated Young’s contributions. Method: We attempted to derive Young’s equations for axial and spherical refractive index gradients. Raytracing was used to confirm accuracy of formula. Results: We did not confirm Young’s equation for the axial gradient to provide aberration-free optics, but derived a slightly different equation. We confirmed the correctness of his equations for deviation of rays in a spherical gradient index and for the focal length of a lens with a nucleus of fixed index surrounded by a cortex of reducing index towards the edge. Young claimed that the equation for focal length applied to a lens with part of the constant index nucleus of the sphere removed, such that the loss of focal length was a quarter of the thickness removed, but this is not strictly correct. Conclusion: Young’s theoretical work in gradient-index optics received no acknowledgement from either his contemporaries or later authors. While his model of the eye lens is not an accurate physiological description of the human lens, with the index reducing least quickly at the edge, it represented a bold attempt to approximate the characteristics of the lens. Thomas Young’s work deserves wider recognition.

Anisotropy of articular cartilage reflects the ECM gradient architecture : Hough-Radon Transform Analysis

Tissue-specific extracellular matrix (ECM) is known to be an ideal bioscaffold to inspire the future of regenerative medicine. It holds the secret of how nature has developed such an organization of molecules into a unique functional complexity. This work exploited an innovative image processing algorithm and high resolution microscopy associated with mechanical analysis to establish a correlation between the gradient organization of cartiligous ECM and its anisotropic biomechanical response. This was hypothesized to be a reliable determinant that can elucidate how microarchitecture interrelates with biomechanical properties. Hough-Radon transform of the ECM cross-section images revealed its conformational variation from tangential interface down to subchondral region. As the orientation varied layer by layer, the anisotropic mechanical response deviated relatively. Although, results were in good agreement (Kendall's tau-b > 90%), there were evidences proposing that alignment of the fibrous network, specifically in middle zone, is not as random as it was previously thought.

A note on a reaction-diffusion model describing the bone morphogen protein gradient in Drosophila embryonic patterning

In this article, we consider the Eldar model [3] from embryology in which a bone morphogenic protein, a short gastrulation protein, and their compound react and diffuse. We carry out a perturbation analysis in the limit of small diffusivity of the bone morphogenic protein. This analysis establishes conditions under which some elementary results of [3] are valid.

Adaptive online gradient descent

We study the rates of growth of the regret in online convex optimization. First, we show that a simple extension of the algorithm of Hazan et al eliminates the need for a priori knowledge of the lower bound on the second derivatives of the observed functions. We then provide an algorithm, Adaptive Online Gradient Descent, which interpolates between the results of Zinkevich for linear functions and of Hazan et al for strongly convex functions, achieving intermediate rates between [square root T] and [log T]. Furthermore, we show strong optimality of the algorithm. Finally, we provide an extension of our results to general norms.

Child health and the income gradient : evidence from Australia

The positive relationship between household income and child health is well documented in the child health literature but the precise mechanisms via which income generates better health and whether the income gradient is increasing in child age are not well understood. This paper presents new Australian evidence on the child health–income gradient. We use data from the Longitudinal Study of Australian Children (LSAC), which involved two waves of data collection for children born between March 2003 and February 2004 (B-Cohort: 0–3 years), and between March 1999 and February 2000 (K-Cohort: 4–7 years). This data set allows us to test the robustness of some of the findings of the influential studies of Case et al. [Case, A., Lubotsky, D., Paxson, C., 2002. Economic status and health in childhood: the origins of the gradient. The American Economic Review 92 (5) 1308–1344] and Currie and Stabile [Currie, J., Stabile, M., 2003. Socioeconomic status and child health: why is the relationship stronger for older children. The American Economic Review 93 (5) 1813–1823], and a recent study by Currie et al. [Currie, A., Shields, M.A., Price, S.W., 2007. The child health/family income gradient: evidence from England. Journal of Health Economics 26 (2) 213–232]. The richness of the LSAC data set also allows us to conduct further exploration of the determinants of child health. Our results reveal an increasing income gradient by child age using similar covariates to Case et al. [Case, A., Lubotsky, D., Paxson, C., 2002. Economic status and health in childhood: the origins of the gradient. The American Economic Review 92 (5) 1308–1344]. However, the income gradient disappears if we include a rich set of controls. Our results indicate that parental health and, in particular, the mother's health plays a significant role, reducing the income coefficient to zero; suggesting an underlying mechanism that can explain the observed relationship between child health and family income. Overall, our results for Australian children are similar to those produced by Propper et al. [Propper, C., Rigg, J., Burgess, S., 2007. Child health: evidence on the roles of family income and maternal mental health from a UK birth cohort. Health Economics 16 (11) 1245–1269] on their British child cohort.

What roles do contemporaneous and cumulative incomes play in the income–child health gradient for young children? Evidence from an Australian Panel

The literature to date shows that children from poorer households tend to have worse health than their peers, and the gap between them grows with age. We investigate whether and how health shocks (as measured by the onset of chronic conditions) contribute to the income–child health gradient and whether the contemporaneous or cumulative effects of income play important mitigating roles. We exploit a rich panel dataset with three panel waves called the Longitudinal Study of Australian children. Given the availability of three waves of data, we are able to apply a range of econometric techniques (e.g. fixed and random effects) to control for unobserved heterogeneity. The paper makes several contributions to the extant literature. First, it shows that an apparent income gradient becomes relatively attenuated in our dataset when the cumulative and contemporaneous effects of household income are distinguished econometrically. Second, it demonstrates that the income–child health gradient becomes statistically insignificant when controlling for parental health and health-related behaviours or unobserved heterogeneity.

Left ventricular volume and valve gradient analysis using an Apple Macintosh computer

A description of a computer program to analyse cine angiograms of the heart and pressure waveforms to calculate valve gradients.

The mechanical triggers of plaque rupture: Shear stress vs pressure gradient

The aim of this study was to evaluate the mechanical triggers that may cause plaque rupture. Wall shear stress (WSS) and pressure gradient are the direct mechanical forces acting on the plaque in a stenotic artery. Their influence on plaque stability is thought to be controversial. This study used a physiologically realistic, pulsatile flow, two-dimensional, cine phase-contrast MRI sequence in a patient with a 70% carotid stenosis. Instead of considering the full patient-specific carotid bifurcation derived from MRI, only the plaque region has been modelled by means of the idealised flow model. WSS reached a local maximum just distal to the stenosis followed by a negative local minimum. A pressure drop across the stenosis was found which varied significantly during systole and diastole. The ratio of the relative importance of WSS and pressure was assessed and was found to be less than 0.07% for all time phases, even at the throat of the stenosis. In conclusion, although the local high WSS at the stenosis may damage the endothelium and fissure plaque, the magnitude of WSS is small compared with the overall loading on plaque. Therefore, pressure may be the main mechanical trigger for plaque rupture and risk stratification using stress analysis of plaque stability may only need to consider the pressure effect.

Transitional intermittency in boundary layers subjected to pressure gradient

Results are reported from an extensive series of experiments on boundary layers in which the location of pressure gradient and transition onset could be varied almost independently, by judicious use of tunnel wall liners and transition-fixing devices. The experiments show that the transition zone is sensitive to the pressure gradient especially near onset, and can be significantly asymmetric; no universal similarity appears valid in general. Observed intermittency distributions cannot be explained on the basis of the hypothesis, often made, that the spot propagates at speeds proportional to the local free-stream velocity but is otherwise unaffected by the pressure gradient.

Evidence of a latitudinal gradient in spider diversity in Australian cotton

The most common explanation for species diversity increasing towards the tropics is the corresponding increase in habitats (spatial heterogeneity). Consequently, a monoculture (like cotton in Australia) which is grown along a latitudinal gradient, should have the same degree of species diversity throughout its range. We tested to see if diversity in a dominant cotton community (spiders) changed with latitude, and if the community was structurally identical in different parts of Australia. We sampled seven sites extending over 20 degrees of latitude. At each site we sampled 1-3 fields 3-5 times during the cotton growing season using pitfall traps and beatsheets, recording all the spiders collected to family. We found that spider communities in cotton are diverse, including a large range of foraging guilds, making them suitable for a conservation biological control programme. We also found that spider diversity increased from high to low latitudes, and the communities were different, even though the spiders were in the same monocultural habitat. Spider beatsheet communities around Australia were dominated by different families, and responded differently to seasonal changes, indicating that different pest groups would be targeted at different locations. These results show that diversity can increase from high to low latitudes, even if spatial heterogeneity is held constant, and that other factors external to the cotton crop are influencing spider species composition. Other models which may account for the latitudinal gradient, such as non-equilibrium regional processes, are discussed.

Growth and physiological response of six Australian rainforest tree species to a light gradient

An understanding of growth and photosynthetic potential of subtropical rainforest species to variations in light environment can be useful for determining the sequence of species introductions in rainforest restoration projects and mixed species plantations. We examined the growth and physiology of six Australian subtropical rainforest tree species in a greenhouse consisting of three artificial light environments (10%, 30%, and 60% full sunlight). Morphological responses followed the typical sun-shade dichotomy, with early and late secondary species (Elaeocarpus grandis, Flindersia brayleyana, Flindersia schottiana, and Gmelina leichhardtii) displaying higher relative growth rate (RGR) compared to mature stage species (Cryptocarya erythroxyion and Heritiera trifoliolatum). Growth and photosynthetic performance of most species reached a maximum in 30-60% full sunlight. Physiological responses provided limited evidence of a distinct dichotomy between early and late successional species. E. grandis and F brayleyana, provided a clear representation of early successional species, with marked increase in Am in high light and an ability to down regulate photosynthetic machinery in low light conditions. The remaining species (F. schottiana, G. leichhardtii, and H. trifoliolatum) were better represented as failing along a shade-tolerant continuum, with limited ability to adjust physiologically to an increase or decrease in light, maintaining similar A(max) across all light environments. Results show that most species belong to a shade-tolerant constituency, with an ability to grow and persist across a wide range of light environments. The species offer a wide range of potential planting scenarios and silvicultural options, with ample potential to achieve rapid canopy closure and rainforest restoration goals.