Wind-energy based path planning for electric unmanned aerial vehicles using Markov Decision Processes

Exploiting wind-energy is one possible way to ex- tend flight duration for Unmanned Arial Vehicles. Wind-energy can also be used to minimise energy consumption for a planned path. In this paper, we consider uncertain time-varying wind fields and plan a path through them. A Gaussian distribution is used to determine uncertainty in the Time-varying wind fields. We use Markov Decision Process to plan a path based upon the uncertainty of Gaussian distribution. Simulation results that compare the direct line of flight between start and target point and our planned path for energy consumption and time of travel are presented. The result is a robust path using the most visited cell while sampling the Gaussian distribution of the wind field in each cell.

Wind-energy based path planning for unmanned aerial vehicles using Markov decision processes

Exploiting wind-energy is one possible way to extend flight duration for Unmanned Arial Vehicles. Wind-energy can also be used to minimise energy consumption for a planned path. In this paper, we consider uncertain time-varying wind fields and plan a path through them. A Gaussian distribution is used to determine uncertainty in the Time-varying wind fields. We use Markov Decision Process to plan a path based upon the uncertainty of Gaussian distribution. Simulation results that compare the direct line of flight between start and target point and our planned path for energy consumption and time of travel are presented. The result is a robust path using the most visited cell while sampling the Gaussian distribution of the wind field in each cell.

Theoretical framework for a management decision evaluation system

A theoretical framework for a construction management decision evaluation system for project selection by means of a literature review. The theory is developed by the examination of the major factors concerning the project selection decision from a deterministic viewpoint, where the decision-maker is assumed to possess 'perfect knowledge' of all the aspects involved. Four fundamental project characteristics are identified together with three meaningful outcome variables. The relationship within and between these variables are considered together with some possible solution techniques. The theory is next extended to time-related dynamic aspects of the problem leading to the implications of imperfect knowledge and a non­deterministic model. A solution technique is proposed in which Gottinger's sequential machines are utilised to model the decision process,

Decision making under pressure : a behavioural economics perspective

This research investigates the decision making process of individuals from revealed preferences in extreme environments or life-and-death situations, from a behavioral economics perspective. The empirical analysis of revealed behavioral preferences shows how the individual decision making process can deviate from the standard self-interested or “homo economicus” model in non-standard situations. The environments examined include: elite athletes in FIFA World and Euro Cups; climbing on Everest and the Himalaya; communication during 9/11 and risk seeking after the 2011 Brisbane floods. The results reveal that the interaction of culture and environment has a significant impact on the decision process, as social behaviors and institutions are intimately intertwined, which govern the processes of human behavior and interaction. Additionally, that risk attitudes are not set and that immediate environmental factors can induce a significant shift in an individuals risk seeking behaviors.

Pipeline renewal decision support : a case study using LinEAR in a water pipeline network

This paper presents a case study for the application of a Linear Engineering Asset Renewal decision support software tool (LinEAR) at a water distribution network in Australia. This case study examines how the LinEAR can assist water utilities to minimise their total pipeline management cost, to make a long-term budget based on mathematically predicted expenditure, and to present calculated evidence for supporting their expenditure requirements. The outcomes from the study on pipeline renewal decision support demonstrate that LinEAR can help water utilities to improve the decision process and save renewal costs over a long-term by providing an optimum renewal schedules. This software can help organisation to accumulate technical knowledge and prediction future impact of the decision using what-if analysis.

Causes of rigidity in SMEs' export commencement decision

Today, small-medium sized enterprises (SMEs) collectively contribute to the largest percentage of job creation in OECD countries. SMEs have become increasingly international since the turn of the century despite being smaller in size in comparison to large multinational firms, and notably, exporting is the most favoured mode of international market entry utilised by SMEs in their internationalisation strategy. Governments around the world have acknowledged the importance of export promotion and have employed policies that are targeted at increasing the export activity of SMEs. However, in many countries, the involvement of SMEs in export operations remains rather low. Within Australia, for example, only about one-third of local SMEs are exporting and this raises an important question as to why there is such a huge percentage of non-exporters. Much scholarly research that focuses on this problem has concentrated on the broad concept of 'export barriers' that act as obstacles to a firm's export development. This paper takes a different approach to previous studies and proposes that a firm's resistance to commence exporting can be better understood through an analysis of the behavioural decision process during its pre-export state. Using a sample of Australian SMEs, the factors that are important in preventing a firm’s initial export commencement decision are categorised and discussed through the use of factor analysis.

Competitive Smart Cities through Healthy Decision-making

This chapter examines the challenges and opportunities associated with planning for competitive, smart and healthy cities. The chapter is based on the assumptions that a healthy city is an important prerequisite for a competitive city and a fundamental outcome of smart cities. Thus, it is preeminent to understand the planning decision support system based on local determinants of health, economic and social factors. One of the major decision support systems is e-health and this chapter will focus on the role of e-health planning, by utilising web-based geographic decision support systems. The proposed novel decision support system would provide a powerful and effective platform for stakeholders to access online information for a better decision-making while empowering community participation. The chapter also highlights the need for a comprehensive conceptual framework to guide the decision process of planning for healthy cities in association with opportunities and limitations. In summary, this chapter provides the critical insights of using information science-based framework and suggest online decision support methods, as part of a broader e-health approach for creating a healthy, competitive and smart city.

Linear programming for large-scale Markov decision problems

We consider the problem of controlling a Markov decision process (MDP) with a large state space, so as to minimize average cost. Since it is intractable to compete with the optimal policy for large scale problems, we pursue the more modest goal of competing with a low-dimensional family of policies. We use the dual linear programming formulation of the MDP average cost problem, in which the variable is a stationary distribution over state-action pairs, and we consider a neighborhood of a low-dimensional subset of the set of stationary distributions (defined in terms of state-action features) as the comparison class. We propose a technique based on stochastic convex optimization and give bounds that show that the performance of our algorithm approaches the best achievable by any policy in the comparison class. Most importantly, this result depends on the size of the comparison class, but not on the size of the state space. Preliminary experiments show the effectiveness of the proposed algorithm in a queuing application.

Modeling multi-stakeholder multi-objective decisions during public participation in major infrastructure and construction projects: A decision rule approach

Major infrastructure and construction (MIC) projects are those with significant traffic or environmental impact, of strategic and regional significance and high sensitivity. The decision making process of schemes of this type is becoming ever more complicated, especially with the increasing number of stakeholders involved and their growing tendency to defend their own varied interests. Failing to address and meet the concerns and expectations of stakeholders may result in project failures. To avoid this necessitates a systematic participatory approach to facilitate decision-making. Though numerous decision models have been established in previous studies (e.g. ELECTRE methods, the analytic hierarchy process and analytic network process) their applicability in the decision process during stakeholder participation in contemporary MIC projects is still uncertain. To resolve this, the decision rule approach is employed for modeling multi-stakeholder multi-objective project decisions. Through this, the result is obtained naturally according to the “rules” accepted by any stakeholder involved. In this sense, consensus is more likely to be achieved since the process is more convincing and the result is easier to be accepted by all concerned. Appropriate “rules”, comprehensive enough to address multiple objectives while straightforward enough to be understood by multiple stakeholders, are set for resolving conflict and facilitating consensus during the project decision process. The West Kowloon Cultural District (WKCD) project is used as a demonstration case and a focus group meeting is conducted in order to confirm the validity of the model established. The results indicate that the model is objective, reliable and practical enough to cope with real world problems. Finally, a suggested future research agenda is provided.

An Online Actor-Critic Algorithm with Function Approximation for Constrained Markov Decision Processes

We develop an online actor-critic reinforcement learning algorithm with function approximation for a problem of control under inequality constraints. We consider the long-run average cost Markov decision process (MDP) framework in which both the objective and the constraint functions are suitable policy-dependent long-run averages of certain sample path functions. The Lagrange multiplier method is used to handle the inequality constraints. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal solution. We also provide the results of numerical experiments on a problem of routing in a multi-stage queueing network with constraints on long-run average queue lengths. We observe that our algorithm exhibits good performance on this setting and converges to a feasible point.

Non-Stationary Semi-Markov Decision Processes on a Finite Horizon

We introduce and study a class of non-stationary semi-Markov decision processes on a finite horizon. By constructing an equivalent Markov decision process, we establish the existence of a piecewise open loop relaxed control which is optimal for the finite horizon problem.

A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.

A Markov Decision Theoretic Approach to Pilot Allocation and Receive Antenna Selection

This paper considers antenna selection (AS) at a receiver equipped with multiple antenna elements but only a single radio frequency chain for packet reception. As information about the channel state is acquired using training symbols (pilots), the receiver makes its AS decisions based on noisy channel estimates. Additional information that can be exploited for AS includes the time-correlation of the wireless channel and the results of the link-layer error checks upon receiving the data packets. In this scenario, the task of the receiver is to sequentially select (a) the pilot symbol allocation, i.e., how to distribute the available pilot symbols among the antenna elements, for channel estimation on each of the receive antennas; and (b) the antenna to be used for data packet reception. The goal is to maximize the expected throughput, based on the past history of allocation and selection decisions, and the corresponding noisy channel estimates and error check results. Since the channel state is only partially observed through the noisy pilots and the error checks, the joint problem of pilot allocation and AS is modeled as a partially observed Markov decision process (POMDP). The solution to the POMDP yields the policy that maximizes the long-term expected throughput. Using the Finite State Markov Chain (FSMC) model for the wireless channel, the performance of the POMDP solution is compared with that of other existing schemes, and it is illustrated through numerical evaluation that the POMDP solution significantly outperforms them.

Transmit power Control with ARQ in energy harvesting sensors: a decision-theoretic approach

This paper addresses the problem of finding optimal power control policies for wireless energy harvesting sensor (EHS) nodes with automatic repeat request (ARQ)-based packet transmissions. The EHS harvests energy from the environment according to a Bernoulli process; and it is required to operate within the constraint of energy neutrality. The EHS obtains partial channel state information (CSI) at the transmitter through the link-layer ARQ protocol, via the ACK/NACK feedback messages, and uses it to adapt the transmission power for the packet (re)transmission attempts. The underlying wireless fading channel is modeled as a finite state Markov chain with known transition probabilities. Thus, the goal of the power management policy is to determine the best power setting for the current packet transmission attempt, so as to maximize a long-run expected reward such as the expected outage probability. The problem is addressed in a decision-theoretic framework by casting it as a partially observable Markov decision process (POMDP). Due to the large size of the state-space, the exact solution to the POMDP is computationally expensive. Hence, two popular approximate solutions are considered, which yield good power management policies for the transmission attempts. Monte Carlo simulation results illustrate the efficacy of the approach and show that the approximate solutions significantly outperform conventional approaches.

Make or buy decision in the context of manufacturing strategy

This paper describes an approach to structuring the make or buy decision process, basing it firmly in the context of an overall manufacturing strategy. The work has been carried out jointly by the University of Cambridge Manufacturing Engineering Group and Lucas Industries. A review of the current state of ideas surrounding the linked issues of vertical integration and make or buy decisions is presented. Important features of the approach include identification of core manufacturing capabilities, assessment of the role of technology in manufacturing, the development of a cost model to support make or buy decisions and a review of the strategic implications of varying degrees of vertical integration.