Empirical evidence for ultrametric structure in multi-layer perceptron error surfaces

Combinatorial optimization problems share an interesting property with spin glass systems in that their state spaces can exhibit ultrametric structure. We use sampling methods to analyse the error surfaces of feedforward multi-layer perceptron neural networks learning encoder problems. The third order statistics of these points of attraction are examined and found to be arranged in a highly ultrametric way. This is a unique result for a finite, continuous parameter space. The implications of this result are discussed.

Sample-to-sample fluctuations of the overlap distributions in the three-dimensional Edwards-Anderson spin glass

We study the sample-to-sample fluctuations of the overlap probability densities from large-scale equilibrium simulations of the three-dimensional Edwards-Anderson spin glass below the critical temperature. Ultrametricity, stochastic stability, and overlap equivalence impose constraints on the moments of the overlap probability densities that can be tested against numerical data. We found small deviations from the Ghirlanda Guerra predictions, which get smaller as system size increases. We also focus on the shape of the overlap distribution, comparing the numerical data to a mean-field-like prediction in which finite-size effects are taken into account by substituting delta functions with broad peaks.