Developing grounded representations for robots through the principles of sensorimotor coordination

Robots currently recognise and use objects through algorithms that are hand-coded or specifically trained. Such robots can operate in known, structured environments but cannot learn to recognise or use novel objects as they appear. This thesis demonstrates that a robot can develop meaningful object representations by learning the fundamental relationship between action and change in sensory state; the robot learns sensorimotor coordination. Methods based on Markov Decision Processes are experimentally validated on a mobile robot capable of gripping objects, and it is found that object recognition and manipulation can be learnt as an emergent property of sensorimotor coordination.

Insight into the pain customer : how to improve the value proposition for community pharmacy

1. Shoppers Profiles- Are Grocery Shoppers any different to Pharmacy Shoppers? 2. Awaking the Sleeping Giants- Supermarkets push into OTC Medicines 3. Price versus Value for Money- What’s the Difference? Value and Perceived Quality 4. Consumer Decision Making Process- The Pharmacists ‘Trump’ Card

Emotion as a boost to metacognition : how worry enhances the quality of confidence

Emotion and cognition are known to interact during human decision processes. In this study we focus on a specific kind of cognition, namely metacognition. Our experiment induces a negative emotion, worry, during a perceptual task. In a numerosity task subjects have to make a two alternative forced choice and then reveal their confidence in this decision. We measure metacognition in terms of discrimination and calibration abilities. Our results show that metacognition, but not choice, is affected by the level of worry anticipatedbefore the decision. Under worry individuals tend to have better metacognition in terms of the two measures. Furthermore understanding the formation of confidence is better explained with taking into account the level of worry in the model. This study shows the importance of an emotional component in the formation and the quality of the subjective probabilities.

Enhancing information about sustainability features for sustainable housing delivery

Sustainability practices in government regulations and within the society influence the delivery of sustainable housing. The actual delivery rate of Australian sustain-able housing is not as high as other countries. There is an absence of engagement by stakeholders in adopting sustainable housing practices. This may be due, in the current Australian property market, to confusion as to what sustainability features should be considered, given the large range of environmental, economic and social sustainability options possible. One of the main problems appears to be that information demanders, especially real estate agents, valuers, insurance agents and mortgage lenders do not include sustainability perspectives in their advice or in their decision processes. Information distribution in the Australian property market is flawed, resulting in a lack of return-on-investment value of ‘green’ features implemented by some stakeholders. This paper reviewed the global sustainable development concept and Australian sustainable assessment methods. This review identified the possibility of a research project which aimed at identifying and integrating different perceptions and priority needs of the information demanders, for developing a model for the potential implementation of sustainability features distribution in the property industry. This research will reduce confusion on the sustainability-related information which can influence the decision making of stakeholders in the supply and demand of sustainable housing.

An optimal design for screening trials

Yao, Begg, and Livingston (1996, Biometrics 52, 992-1001) considered the optimal group size for testing a series of potentially therapeutic agents to identify a promising one as soon as possible for given error rates. The number of patients to be tested with each agent was fixed as the group size. We consider a sequential design that allows early acceptance and rejection, and we provide an optimal strategy to minimize the sample sizes (patients) required using Markov decision processes. The minimization is under the constraints of the two types (false positive and false negative) of error probabilities, with the Lagrangian multipliers corresponding to the cost parameters for the two types of errors. Numerical studies indicate that there can be a substantial reduction in the number of patients required.

Understanding consumers' decisions to adopt technology-enabled transformative services

This study extends understanding of consumers' decisions to adopt transformative services delivered via technology. It incorporates competitive effects into the model of goal-directed behavior which, in keeping with the majority of consumer decision making models, neglects to explicitly account for competition. A goal-level operationalization of competition, incorporating both direct and indirect competition, is proposed. A national web-based survey collected data from 431 respondents about their decisions to adopt mental health services delivered via mobile phone. The findings show that the extent to which consumers perceived using these transformative services to be more instrumental to achieving their goals than competition had the greatest impact on their adoption decisions. This finding builds on the limited empirical evidence for the inclusion of competitive effects to more fully explain consumers' decisions to adopt technology-based and other services. It also provides support for a broader operationalization of competition with respect to consumers' personal goals.

Error bounds for calculation of the Gittins indices

For a wide class of semi-Markov decision processes the optimal policies are expressible in terms of the Gittins indices, which have been found useful in sequential clinical trials and pharmaceutical research planning. In general, the indices can be approximated via calibration based on dynamic programming of finite horizon. This paper provides some results on the accuracy of such approximations, and, in particular, gives the error bounds for some well known processes (Bernoulli reward processes, normal reward processes and exponential target processes).

Uncertainty based online planning for UAV target finding in cluttered and GPS-denied environments

There are some scenarios in which Unmmaned Aerial Vehicle (UAV) navigation becomes a challenge due to the occlusion of GPS systems signal, the presence of obstacles and constraints in the space in which a UAV operates. An additional challenge is presented when a target whose location is unknown must be found within a confined space. In this paper we present a UAV navigation and target finding mission, modelled as a Partially Observable Markov Decision Process (POMDP) using a state-of-the-art online solver in a real scenario using a low cost commercial multi rotor UAV and a modular system architecture running under the Robotic Operative System (ROS). Using POMDP has several advantages to conventional approaches as they take into account uncertainties in sensor information. We present a framework for testing the mission with simulation tests and real flight tests in which we model the system dynamics and motion and perception uncertainties. The system uses a quad-copter aircraft with an board downwards looking camera without the need of GPS systems while avoiding obstacles within a confined area. Results indicate that the system has 100% success rate in simulation and 80% rate during flight test for finding targets located at different locations.

Parametrized actor-critic algorithms for finite-horizon MDPs

Due to their non-stationarity, finite-horizon Markov decision processes (FH-MDPs) have one probability transition matrix per stage. Thus the curse of dimensionality affects FH-MDPs more severely than infinite-horizon MDPs. We propose two parametrized 'actor-critic' algorithms to compute optimal policies for FH-MDPs. Both algorithms use the two-timescale stochastic approximation technique, thus simultaneously performing gradient search in the parametrized policy space (the 'actor') on a slower timescale and learning the policy gradient (the 'critic') via a faster recursion. This is in contrast to methods where critic recursions learn the cost-to-go proper. We show w.p 1 convergence to a set with the necessary condition for constrained optima. The proposed parameterization is for FHMDPs with compact action sets, although certain exceptions can be handled. Further, a third algorithm for stochastic control of stopping time processes is presented. We explain why current policy evaluation methods do not work as critic to the proposed actor recursion. Simulation results from flow-control in communication networks attest to the performance advantages of all three algorithms.

Solving MDPs using two-timescale simulated annealing with multiplicative weights

We develop extensions of the Simulated Annealing with Multiplicative Weights (SAMW) algorithm that proposed a method of solution of Finite-Horizon Markov Decision Processes (FH-MDPs). The extensions developed are in three directions: a) Use of the dynamic programming principle in the policy update step of SAMW b) A two-timescale actor-critic algorithm that uses simulated transitions alone, and c) Extending the algorithm to the infinite-horizon discounted-reward scenario. In particular, a) reduces the storage required from exponential to linear in the number of actions per stage-state pair. On the faster timescale, a 'critic' recursion performs policy evaluation while on the slower timescale an 'actor' recursion performs policy improvement using SAMW. We give a proof outlining convergence w.p. 1 and show experimental results on two settings: semiconductor fabrication and flow control in communication networks.

Solution of MDPS using simulation-based value iteration

This article proposes a three-timescale simulation based algorithm for solution of infinite horizon Markov Decision Processes (MDPs). We assume a finite state space and discounted cost criterion and adopt the value iteration approach. An approximation of the Dynamic Programming operator T is applied to the value function iterates. This 'approximate' operator is implemented using three timescales, the slowest of which updates the value function iterates. On the middle timescale we perform a gradient search over the feasible action set of each state using Simultaneous Perturbation Stochastic Approximation (SPSA) gradient estimates, thus finding the minimizing action in T. On the fastest timescale, the 'critic' estimates, over which the gradient search is performed, are obtained. A sketch of convergence explaining the dynamics of the algorithm using associated ODEs is also presented. Numerical experiments on rate based flow control on a bottleneck node using a continuous-time queueing model are performed using the proposed algorithm. The results obtained are verified against classical value iteration where the feasible set is suitably discretized. Over such a discretized setting, a variant of the algorithm of [12] is compared and the proposed algorithm is found to converge faster.

The actor-critic algorithm as multi-time-scale stochastic approximation

The actor-critic algorithm of Barto and others for simulation-based optimization of Markov decision processes is cast as a two time Scale stochastic approximation. Convergence analysis, approximation issues and an example are studied.

Exploring use of climate information for optimal design of water resources infrastructure

We conducted surveys of fire and fuels managers at local, regional, and national levels to gain insights into decision processes and information flows in wildfire management. Survey results in the form of fire managers’ decision calendars show how climate information needs vary seasonally, over space, and through the organizational network, and help determine optimal points for introducing climate information and forecasts into decision processes. We identified opportunities to use climate information in fire management, including seasonal to interannual climate forecasts at all organizational levels, to improve the targeting of fuels treatments and prescribed burns, the positioning and movement of initial attack resources, and staffing and budgeting decisions. Longer-term (5–10 years) outlooks also could be useful at the national level in setting budget and research priorities. We discuss these opportunities and examine the kinds of organizational changes that could facilitate effective use of existing climate information and climate forecast capabilities.

A Simultaneous Deterministic Perturbation Actor-Critic Algorithm with an Application to Optimal Mortgage Refinancing

We develop a simulation-based, two-timescale actor-critic algorithm for infinite horizon Markov decision processes with finite state and action spaces, with a discounted reward criterion. The algorithm is of the gradient ascent type and performs a search in the space of stationary randomized policies. The algorithm uses certain simultaneous deterministic perturbation stochastic approximation (SDPSA) gradient estimates for enhanced performance. We show an application of our algorithm on a problem of mortgage refinancing. Our algorithm obtains the optimal refinancing strategies in a computationally efficient manner

Energy Sharing for Multiple Sensor Nodes With Finite Buffers

We consider the problem of finding optimal energy sharing policies that maximize the network performance of a system comprising of multiple sensor nodes and a single energy harvesting (EH) source. Sensor nodes periodically sense the random field and generate data, which is stored in the corresponding data queues. The EH source harnesses energy from ambient energy sources and the generated energy is stored in an energy buffer. Sensor nodes receive energy for data transmission from the EH source. The EH source has to efficiently share the stored energy among the nodes to minimize the long-run average delay in data transmission. We formulate the problem of energy sharing between the nodes in the framework of average cost infinite-horizon Markov decision processes (MDPs). We develop efficient energy sharing algorithms, namely Q-learning algorithm with exploration mechanisms based on the epsilon-greedy method as well as upper confidence bound (UCB). We extend these algorithms by incorporating state and action space aggregation to tackle state-action space explosion in the MDP. We also develop a cross entropy based method that incorporates policy parameterization to find near optimal energy sharing policies. Through simulations, we show that our algorithms yield energy sharing policies that outperform the heuristic greedy method.