A scalable strategy for the parallelization of multiphysics unstructured mesh-iterative codes on distributed-memory systems

Realizing scalable performance on high performance computing systems is not straightforward for single-phenomenon codes (such as computational fluid dynamics [CFD]). This task is magnified considerably when the target software involves the interactions of a range of phenomena that have distinctive solution procedures involving different discretization methods. The problems of addressing the key issues of retaining data integrity and the ordering of the calculation procedures are significant. A strategy for parallelizing this multiphysics family of codes is described for software exploiting finite-volume discretization methods on unstructured meshes using iterative solution procedures. A mesh partitioning-based SPMD approach is used. However, since different variables use distinct discretization schemes, this means that distinct partitions are required; techniques for addressing this issue are described using the mesh-partitioning tool, JOSTLE. In this contribution, the strategy is tested for a variety of test cases under a wide range of conditions (e.g., problem size, number of processors, asynchronous / synchronous communications, etc.) using a variety of strategies for mapping the mesh partition onto the processor topology.

An automatable generic strategy for dynamic load balancing in parallel structured mesh CFD codes

Parallel computing is now widely used in numerical simulation, particularly for application codes based on finite difference and finite element methods. A popular and successful technique employed to parallelize such codes onto large distributed memory systems is to partition the mesh into sub-domains that are then allocated to processors. The code then executes in parallel, using the SPMD methodology, with message passing for inter-processor interactions. In order to improve the parallel efficiency of an imbalanced structured mesh CFD code, a new dynamic load balancing (DLB) strategy has been developed in which the processor partition range limits of just one of the partitioned dimensions uses non-coincidental limits, as opposed to coincidental limits. The ‘local’ partition limit change allows greater flexibility in obtaining a balanced load distribution, as the workload increase, or decrease, on a processor is no longer restricted by the ‘global’ (coincidental) limit change. The automatic implementation of this generic DLB strategy within an existing parallel code is presented in this chapter, along with some preliminary results.

An integrated modeling approach to solder joint formation

The attachment of electronic components to printed circuit boards using solder material is a complex process. This paper presents a novel modeling methodology, which integrates the governing physics taking place. Multiphysics modeling technology, imbedded into the simulation tool—PHYSICA is used to simulate fluid flow, heat transfer, solidification, and stress evolution in an integrated manner. Results using this code are presented, detailing the mechanical response of two solder materials as they cool, solidify and then deform. The shape that a solder joint takes upon melting is predicted using the SURFACE EVOLVER code. Details are given on how these predictions can be used in the PHYSICA code to provide a modeling route by which the shape, solidification history, and resulting stress profiles can be predicted.

Automatic and effective multi-dimensional parallelisation of structured mesh based codes

The most common parallelisation strategy for many Computational Mechanics (CM) (typified by Computational Fluid Dynamics (CFD) applications) which use structured meshes, involves a 1D partition based upon slabs of cells. However, many CFD codes employ pipeline operations in their solution procedure. For parallelised versions of such codes to scale well they must employ two (or more) dimensional partitions. This paper describes an algorithmic approach to the multi-dimensional mesh partitioning in code parallelisation, its implementation in a toolkit for almost automatically transforming scalar codes to parallel form, and its testing on a range of ‘real-world’ FORTRAN codes. The concept of multi-dimensional partitioning is straightforward, but non-trivial to represent as a sufficiently generic algorithm so that it can be embedded in a code transformation tool. The results of the tests on fine real-world codes demonstrate clear improvements in parallel performance and scalability (over a 1D partition). This is matched by a huge reduction in the time required to develop the parallel versions when hand coded – from weeks/months down to hours/days.

Discretisation procedures for multi-physics phenomena

Procedures are described for solving the equations governing a multi-physics process. Finite volume techniques are used to discretise, using the same unstructured mesh, the equations of fluid flow, heat transfer with solidification, and solid deformation. These discretised equations are then solved in an integrated manner. The computational mechanics environment, PHYSICA, which facilitates the building of multi-physics models, is described. Comparisons between model predictions and experimental data are presented for the casting of metal components.

Micromagnetic simulation of domain walls in iron films

The micromagnetic structure and energy of 180° domain walls spanning laminar crystals of iron having (100) or (110) surfaces and ranging in thickness from 145 to 580 nm have been investigated by numerical integration of the Landau-Lifshitz-Gilbert equation. Stable equilibrium structures with two flux symmetries were obtained for both crystal orientations at all thicknesses studied.

Modeling the dynamics of Magnetic Semilevitation Melting

In semilevitation melting, a cylindrical metal ingot is melted by a coaxial a.c. induction coil. A watercooled solid base supports the ingot, while the top and side free surface is confined by the magnetic forces as the melting front progresses. The dynamic interplay between gravity, hydrodynamic stress, and the Lorentz force in the fluid determines the instantaneous free surface shape. The coupled nonstationary equations for turbulent flow, heat with phase change, and high-frequency electromagnetic field are solved numerically for the axisymmetric time-dependent domain by a continuous mesh transformation, using a pseudospectral method. Results are obtained for the two actually existing coil configurations and several validation cases.

Molecular dynamics simulation of fractures using an N-body potential

Molecular dynamics has been employed to model the fracture of a two dimensional triangular atomic lattice. The N-body Sutton-Chen potential developed for fcc metals and its extended version (Rafii-Tabar and Sutton) for fcc random binary alloys were used for the interatomic interactions. It is shown that at low temperatures cleavage fractures can occur in both an elemental metal and an alloy. At elevated temperatures the nucleation of dislocations is shown to cause a brittle-to-ductile transition. For the brittle crack propagation in the elemental metal, crack propagation speeds have been computed for different stress rates, and a crack instability found to exist as the speed reaches a critical value of about 32% of the Rayleigh wave speed. For the random alloy, we find that the dislocation movement can be affected by the distorted lattice.

Multiphysics modeling and its application to the casting process

A modeling strategy is presented to solve the governing equations of fluid flow, temperature (with solidification), and stress in an integrated manner. These equations are discretized using finite volume methods on unstructured grids, which provide the capability to represent complex domains. Both the cell-centered and vertex-based forms of the finite volume discretization procedure are explained, and the overall integrated solution procedure using these techniques with suitable solvers is detailed. Two industrial processes, based on the casting of metals, are used to demonstrate the capabilities of the resultant modeling framework. This manufacturing process requires a high degree of coupling between the governing physical equations to accurately predict potential defects. Comparisons between model predictions and experimental observations are given.

Rotating-frame nuclear magnetic relaxation of spins diffusing on a disordered lattice: a Monte Carlo model

The rotating-frame nuclear magnetic relaxation rate of spins diffusing on a disordered lattice has been calculated by Monte Carlo methods. The disorder includes not only variation in the distances between neighbouring spin sites but also variation in the hopping rate associated with each site. The presence of the disorder, particularly the hopping rate disorder, causes changes in the time-dependent spin correlation functions which translate into asymmetry in the characteristic peak in the temperature dependence of the dipolar relaxation rate. The results may be used to deduce the average hopping rate from the relaxation but the effect is not sufficiently marked to enable the distribution of the hopping rates to be evaluated. The distribution, which is a measure of the degree of disorder, is the more interesting feature and it has been possible to show from the calculation that measurements of the relaxation rate as a function of the strength of the radiofrequency spin-locking magnetic field can lead to an evaluation of its width. Some experimental data on an amorphous metal - hydrogen alloy are reported which demonstrate the feasibility of this novel approach to rotating-frame relaxation in disordered materials.

Simulation of transverse and longitudinal magnetic ripple structures induced by surface anisotropy

Micromagnetic ripple structures on the surfaces of thick specimens of ultra-soft magnetic material having strong surface anisotropy Ks favouring out-of-surface magnetization have been calculated. These ripples have wavelengths of the order of 0.1 μm and extend to a depth ∼ √A/Ms, where A is the exchange constant and Ms is the saturation magnetization. The wave-vectors of the ripple structures are either transverse or parallel to the bulk magnetization. Both structures have lower energy than the one-dimensional structure discussed by O'Handley and Woods, and they exhibit stronger normal magnetization. The transverse structure requires a surface anisotropy Ks ≥ 0.80K0, where is that required for the one-dimensional structure. The threshold for longitudinal ripples is 0.84K0. It is suggested that the transverse structure probably constitutes the ground state. The magnitudes of Ks and A should be obtainable from measurements of the ripple wavelength and amplitude, and Ms.

The implementation of an aeronautical CFD flow code onto distributed memory parallel systems

The parallelization of an industrially important in-house computational fluid dynamics (CFD) code for calculating the airflow over complex aircraft configurations using the Euler or Navier–Stokes equations is presented. The code discussed is the flow solver module of the SAUNA CFD suite. This suite uses a novel grid system that may include block-structured hexahedral or pyramidal grids, unstructured tetrahedral grids or a hybrid combination of both. To assist in the rapid convergence to a solution, a number of convergence acceleration techniques are employed including implicit residual smoothing and a multigrid full approximation storage scheme (FAS). Key features of the parallelization approach are the use of domain decomposition and encapsulated message passing to enable the execution in parallel using a single programme multiple data (SPMD) paradigm. In the case where a hybrid grid is used, a unified grid partitioning scheme is employed to define the decomposition of the mesh. The parallel code has been tested using both structured and hybrid grids on a number of different distributed memory parallel systems and is now routinely used to perform industrial scale aeronautical simulations. Copyright © 2000 John Wiley & Sons, Ltd.

Parallelisation of electromagnetic simulation codes

In this paper results obtained from the parallelisation of existing 3D electromagnetic Finite Element codes within the ESPRIT HPCN project PARTEL are presented. The parallelisation procedure, based on the Bulk Synchronous Parallel approach, is outlined and the encouraging results obtained in terms of speed-up on some industrially significant test cases are described and discussed.