A novel adaptive weight selection algorithm for multi-objective multi-agent reinforcement learning

To solve multi-objective problems, multiple reward signals are often scalarized into a single value and further processed using established single-objective problem solving techniques. While the field of multi-objective optimization has made many advances in applying scalarization techniques to obtain good solution trade-offs, the utility of applying these techniques in the multi-objective multi-agent learning domain has not yet been thoroughly investigated. Agents learn the value of their decisions by linearly scalarizing their reward signals at the local level, while acceptable system wide behaviour results. However, the non-linear relationship between weighting parameters of the scalarization function and the learned policy makes the discovery of system wide trade-offs time consuming. Our first contribution is a thorough analysis of well known scalarization schemes within the multi-objective multi-agent reinforcement learning setup. The analysed approaches intelligently explore the weight-space in order to find a wider range of system trade-offs. In our second contribution, we propose a novel adaptive weight algorithm which interacts with the underlying local multi-objective solvers and allows for a better coverage of the Pareto front. Our third contribution is the experimental validation of our approach by learning bi-objective policies in self-organising smart camera networks. We note that our algorithm (i) explores the objective space faster on many problem instances, (ii) obtained solutions that exhibit a larger hypervolume, while (iii) acquiring a greater spread in the objective space.

Multi-objective mission flight planning in civil unmanned aerial systems

Unmanned Aerial Vehicles (UAVs) are emerging as an ideal platform for a wide range of civil applications such as disaster monitoring, atmospheric observation and outback delivery. However, the operation of UAVs is currently restricted to specially segregated regions of airspace outside of the National Airspace System (NAS). Mission Flight Planning (MFP) is an integral part of UAV operation that addresses some of the requirements (such as safety and the rules of the air) of integrating UAVs in the NAS. Automated MFP is a key enabler for a number of UAV operating scenarios as it aids in increasing the level of onboard autonomy. For example, onboard MFP is required to ensure continued conformance with the NAS integration requirements when there is an outage in the communications link. MFP is a motion planning task concerned with finding a path between a designated start waypoint and goal waypoint. This path is described with a sequence of 4 Dimensional (4D) waypoints (three spatial and one time dimension) or equivalently with a sequence of trajectory segments (or tracks). It is necessary to consider the time dimension as the UAV operates in a dynamic environment. Existing methods for generic motion planning, UAV motion planning and general vehicle motion planning cannot adequately address the requirements of MFP. The flight plan needs to optimise for multiple decision objectives including mission safety objectives, the rules of the air and mission efficiency objectives. Online (in-flight) replanning capability is needed as the UAV operates in a large, dynamic and uncertain outdoor environment. This thesis derives a multi-objective 4D search algorithm entitled Multi- Step A* (MSA*) based on the seminal A* search algorithm. MSA* is proven to find the optimal (least cost) path given a variable successor operator (which enables arbitrary track angle and track velocity resolution). Furthermore, it is shown to be of comparable complexity to multi-objective, vector neighbourhood based A* (Vector A*, an extension of A*). A variable successor operator enables the imposition of a multi-resolution lattice structure on the search space (which results in fewer search nodes). Unlike cell decomposition based methods, soundness is guaranteed with multi-resolution MSA*. MSA* is demonstrated through Monte Carlo simulations to be computationally efficient. It is shown that multi-resolution, lattice based MSA* finds paths of equivalent cost (less than 0.5% difference) to Vector A* (the benchmark) in a third of the computation time (on average). This is the first contribution of the research. The second contribution is the discovery of the additive consistency property for planning with multiple decision objectives. Additive consistency ensures that the planner is not biased (which results in a suboptimal path) by ensuring that the cost of traversing a track using one step equals that of traversing the same track using multiple steps. MSA* mitigates uncertainty through online replanning, Multi-Criteria Decision Making (MCDM) and tolerance. Each trajectory segment is modeled with a cell sequence that completely encloses the trajectory segment. The tolerance, measured as the minimum distance between the track and cell boundaries, is the third major contribution. Even though MSA* is demonstrated for UAV MFP, it is extensible to other 4D vehicle motion planning applications. Finally, the research proposes a self-scheduling replanning architecture for MFP. This architecture replicates the decision strategies of human experts to meet the time constraints of online replanning. Based on a feedback loop, the proposed architecture switches between fast, near-optimal planning and optimal planning to minimise the need for hold manoeuvres. The derived MFP framework is original and shown, through extensive verification and validation, to satisfy the requirements of UAV MFP. As MFP is an enabling factor for operation of UAVs in the NAS, the presented work is both original and significant.

UAS mission path planning system (MPPS) using hybrid-game coupled to multi-objective optimiser

This paper presents the application of advanced optimization techniques to unmanned aerial system mission path planning system (MPPS) using multi-objective evolutionary algorithms (MOEAs). Two types of multi-objective optimizers are compared; the MOEA nondominated sorting genetic algorithm II and a hybrid-game strategy are implemented to produce a set of optimal collision-free trajectories in a three-dimensional environment. The resulting trajectories on a three-dimensional terrain are collision-free and are represented by using Bézier spline curves from start position to target and then target to start position or different positions with altitude constraints. The efficiency of the two optimization methods is compared in terms of computational cost and design quality. Numerical results show the benefits of adding a hybrid-game strategy to a MOEA and for a MPPS.

UAS Mission Path Planning System, (MPPS) using hybrid-game coupled to multi-objective optimizer

This paper presents the application of advanced optimization techniques to unmanned aerial system mission path planning system (MPPS) using multi-objective evolutionary algorithms (MOEAs). Two types of multi-objective optimizers are compared; the MOEA nondominated sorting genetic algorithm II and a hybrid-game strategy are implemented to produce a set of optimal collision-free trajectories in a three-dimensional environment. The resulting trajectories on a three-dimensional terrain are collision-free and are represented by using Bézier spline curves from start position to target and then target to start position or different positions with altitude constraints. The efficiency of the two optimization methods is compared in terms of computational cost and design quality. Numerical results show the benefits of adding a hybrid-game strategy to a MOEA and for a MPPS.

Multi-objective four-dimensional vehicle motion planning in large dynamic environments

This paper presents Multi-Step A* (MSA*), a search algorithm based on A* for multi-objective 4D vehicle motion planning (three spatial and one time dimension). The research is principally motivated by the need for offline and online motion planning for autonomous Unmanned Aerial Vehicles (UAVs). For UAVs operating in large, dynamic and uncertain 4D environments, the motion plan consists of a sequence of connected linear tracks (or trajectory segments). The track angle and velocity are important parameters that are often restricted by assumptions and grid geometry in conventional motion planners. Many existing planners also fail to incorporate multiple decision criteria and constraints such as wind, fuel, dynamic obstacles and the rules of the air. It is shown that MSA* finds a cost optimal solution using variable length, angle and velocity trajectory segments. These segments are approximated with a grid based cell sequence that provides an inherent tolerance to uncertainty. Computational efficiency is achieved by using variable successor operators to create a multi-resolution, memory efficient lattice sampling structure. Simulation studies on the UAV flight planning problem show that MSA* meets the time constraints of online replanning and finds paths of equivalent cost but in a quarter of the time (on average) of vector neighbourhood based A*.

Multi-objective design optimization of morphing UAV aerofoil/wing using hybridised MOGA

The paper investigates two advanced Computational Intelligence Systems (CIS) for a morphing Unmanned Aerial Vehicle (UAV) aerofoil/wing shape design optimisation. The first CIS uses Genetic Algorithm (GA) and the second CIS uses Hybridized GA (HGA) with the concept of Nash-Equilibrium to speed up the optimisation process. During the optimisation, Nash-Game will act as a pre-conditioner. Both CISs; GA and HGA, are based on Pareto optimality and they are coupled to Euler based Computational Fluid Dynamic (CFD) analyser and one type of Computer Aided Design (CAD) system during the optimisation.

Multi-objective design optimisation of GFRP sandwich beams

A novel Glass Fibre Reinforced Polymer (GFRP) sandwich panel was developed by an Australian manufacturer for civil engineering applications. This research is motivated by the new applications of GFRP sandwich structures in civil engineering such as slab, beam, girder and sleeper. An optimisation methodology is developed in this work to enhance the design of GFRP sandwich beams. The design of single and glue laminated GFRP sandwich beam were conducted by using numerical optimisation. The numerical multi-objective optimisation considered a design two objectives simultaneously. These objectives are cost and mass. The numerical optimisation uses the Adaptive Range Multi-objective Genetic Algorithm (ARMOGA) and Finite Element (FE) method. Trade-offs between objectives was found during the optimisation process. Multi-objective optimisation shows a core to skin mass ratio equal to 3.68 for the single sandwich beam cross section optimisation and it showed that the optimum core to skin thickness ratio is 11.0.

A multi-objective optimization approach for coordinating available transfer capability with risk control

In the electricity market environment, coordination of system reliability and economics of a power system is of great significance in determining the available transfer capability (ATC). In addition, the risks associated with uncertainties should be properly addressed in the ATC determination process for risk-benefit maximization. Against this background, it is necessary that the ATC be optimally allocated and utilized within relative security constraints. First of all, the non-sequential Monte Carlo stimulation is employed to derive the probability density distribution of ATC of designated areas incorporating uncertainty factors. Second, on the basis of that, a multi-objective optimization model is formulated to determine the multi-area ATC so as to maximize the risk-benefits. Then, the solution to the developed model is achieved by the fast non-dominated sorting (NSGA-II) algorithm, which could decrease the risk caused by uncertainties while coordinating the ATCs of different areas. Finally, the IEEE 118-bus test system is served for demonstrating the essential features of the developed model and employed algorithm.

Non-monetised multi-objective decision making system for road management

This paper presents a performance-based optimisation approach for conducting trade-off analysis between safety (roads) and condition (bridges and roads). Safety was based on potential for improvement (PFI). Road condition was based on surface distresses and bridge condition was based on apparent age per subcomponent. The analysis uses a non-monetised optimisation that expanded upon classical Pareto optimality by observing performance across time. It was found that achievement of good results was conditioned by the availability of early age treatments and impacted by a frontier effect preventing the optimisation algorithm from realising of the long-term benefits of deploying actions when approaching the end of the analysis period. A disaggregated bridge condition index proved capable of improving levels of service in bridge subcomponents.

Quantum behaved Particle Swarm Optimization (QPSO) for multi-objective design optimization of composite structures

We present a new, generic method/model for multi-objective design optimization of laminated composite components using a novel multi-objective optimization algorithm developed on the basis of the Quantum behaved Particle Swarm Optimization (QPSO) paradigm. QPSO is a co-variant of the popular Particle Swarm Optimization (PSO) and has been developed and implemented successfully for the multi-objective design optimization of composites. The problem is formulated with multiple objectives of minimizing weight and the total cost of the composite component to achieve a specified strength. The primary optimization variables are - the number of layers, its stacking sequence (the orientation of the layers) and thickness of each layer. The classical lamination theory is utilized to determine the stresses in the component and the design is evaluated based on three failure criteria; Failure Mechanism based Failure criteria, Maximum stress failure criteria and the Tsai-Wu Failure criteria. The optimization method is validated for a number of different loading configurations - uniaxial, biaxial and bending loads. The design optimization has been carried for both variable stacking sequences as well as fixed standard stacking schemes and a comparative study of the different design configurations evolved has been presented. Also, the performance of QPSO is compared with the conventional PSO.

A new multi-objective model to optimise rail transport scheduler

The sugarcane transport system plays a critical role in the overall performance of Australia’s sugarcane industry. An inefficient sugarcane transport system interrupts the raw sugarcane harvesting process, delays the delivery of sugarcane to the mill, deteriorates the sugar quality, increases the usage of empty bins, and leads to the additional sugarcane production costs. Due to these negative effects, there is an urgent need for an efficient sugarcane transport schedule that should be developed by the rail schedulers. In this study, a multi-objective model using mixed integer programming (MIP) is developed to produce an industry-oriented scheduling optimiser for sugarcane rail transport system. The exact MIP solver (IBM ILOG-CPLEX) is applied to minimise the makespan and the total operating time as multi-objective functions. Moreover, the so-called Siding neighbourhood search (SNS) algorithm is developed and integrated with Sidings Satisfaction Priorities (SSP) and Rail Conflict Elimination (RCE) algorithms to solve the problem in a more efficient way. In implementation, the sugarcane transport system of Kalamia Sugar Mill that is a coastal locality about 1050 km northwest of Brisbane city is investigated as a real case study. Computational experiments indicate that high-quality solutions are obtainable in industry-scale applications.

Performance evaluation of elitist-mutated multi-objective particle swarm optimization for integrated water resources management

Optimal allocation of water resources for various stakeholders often involves considerable complexity with several conflicting goals, which often leads to multi-objective optimization. In aid of effective decision-making to the water managers, apart from developing effective multi-objective mathematical models, there is a greater necessity of providing efficient Pareto optimal solutions to the real world problems. This study proposes a swarm-intelligence-based multi-objective technique, namely the elitist-mutated multi-objective particle swarm optimization technique (EM-MOPSO), for arriving at efficient Pareto optimal solutions to the multi-objective water resource management problems. The EM-MOPSO technique is applied to a case study of the multi-objective reservoir operation problem. The model performance is evaluated by comparing with results of a non-dominated sorting genetic algorithm (NSGA-II) model, and it is found that the EM-MOPSO method results in better performance. The developed method can be used as an effective aid for multi-objective decision-making in integrated water resource management.

Vector evaluated particle swarm optimization (VEPSO) for multi-objective design optimization of composite structures

We present a generic method/model for multi-objective design optimization of laminated composite components, based on vector evaluated particle swarm optimization (VEPSO) algorithm. VEPSO is a novel, co-evolutionary multi-objective variant of the popular particle swarm optimization algorithm (PSO). In the current work a modified version of VEPSO algorithm for discrete variables has been developed and implemented successfully for the, multi-objective design optimization of composites. The problem is formulated with multiple objectives of minimizing weight and the total cost of the composite component to achieve a specified strength. The primary optimization variables are - the number of layers, its stacking sequence (the orientation of the layers) and thickness of each layer. The classical lamination theory is utilized to determine the stresses in the component and the design is evaluated based on three failure criteria; failure mechanism based failure criteria, Maximum stress failure criteria and the Tsai-Wu failure criteria. The optimization method is validated for a number of different loading configurations - uniaxial, biaxial and bending loads. The design optimization has been carried for both variable stacking sequences, as well fixed standard stacking schemes and a comparative study of the different design configurations evolved has been presented. (C) 2007 Elsevier Ltd. All rights reserved.

Artificial Bee Colony (ABC) for multi-objective design optimization of composite structures

In this paper, we present a generic method/model for multi-objective design optimization of laminated composite components, based on Vector Evaluated Artificial Bee Colony (VEABC) algorithm. VEABC is a parallel vector evaluated type, swarm intelligence multi-objective variant of the Artificial Bee Colony algorithm (ABC). In the current work a modified version of VEABC algorithm for discrete variables has been developed and implemented successfully for the multi-objective design optimization of composites. The problem is formulated with multiple objectives of minimizing weight and the total cost of the composite component to achieve a specified strength. The primary optimization variables are the number of layers, its stacking sequence (the orientation of the layers) and thickness of each layer. The classical lamination theory is utilized to determine the stresses in the component and the design is evaluated based on three failure criteria: failure mechanism based failure criteria, maximum stress failure criteria and the tsai-wu failure criteria. The optimization method is validated for a number of different loading configurations-uniaxial, biaxial and bending loads. The design optimization has been carried for both variable stacking sequences, as well fixed standard stacking schemes and a comparative study of the different design configurations evolved has been presented. Finally the performance is evaluated in comparison with other nature inspired techniques which includes Particle Swarm Optimization (PSO), Artificial Immune System (AIS) and Genetic Algorithm (GA). The performance of ABC is at par with that of PSO, AIS and GA for all the loading configurations. (C) 2009 Elsevier B.V. All rights reserved.