Learning dynamic prices in multiseller electronic retail markets with price sensitive customers, stochastic demands, and inventory replenishments

In this paper, we use reinforcement learning (RL) as a tool to study price dynamics in an electronic retail market consisting of two competing sellers, and price sensitive and lead time sensitive customers. Sellers, offering identical products, compete on price to satisfy stochastically arriving demands (customers), and follow standard inventory control and replenishment policies to manage their inventories. In such a generalized setting, RL techniques have not previously been applied. We consider two representative cases: 1) no information case, were none of the sellers has any information about customer queue levels, inventory levels, or prices at the competitors; and 2) partial information case, where every seller has information about the customer queue levels and inventory levels of the competitors. Sellers employ automated pricing agents, or pricebots, which use RL-based pricing algorithms to reset the prices at random intervals based on factors such as number of back orders, inventory levels, and replenishment lead times, with the objective of maximizing discounted cumulative profit. In the no information case, we show that a seller who uses Q-learning outperforms a seller who uses derivative following (DF). In the partial information case, we model the problem as a Markovian game and use actor-critic based RL to learn dynamic prices. We believe our approach to solving these problems is a new and promising way of setting dynamic prices in multiseller environments with stochastic demands, price sensitive customers, and inventory replenishments.

A modell of socially optimal hydro reservoir usage - yelding socially optimal outcomes in nordic electricity markets

An extensive electricity transmission network facilitates electricity trading between Finland, Sweden, Norway and Denmark. Currently most of the area's power generation is traded at NordPool, where the trading volumes have steadily increased since the early 1990's, when the exchange was founded. The Nordic electricity is expected to follow the current trend and further integrate with the other European electricity markets. Hydro power is the source for roughly a half of the supply in the Nordic electricity market and most of the hydro is generated in Norway. The dominating role of hydro power distinguishes the Nordic electricity market from most of the other market places. Production of hydro power varies mainly due to hydro reservoirs and demand for electricity. Hydro reservoirs are affected by water inflows that differ each year. The hydro reservoirs explain remarkably the behaviour of the Nordic electricity markets. Therefore among others, Kauppi and Liski (2008) have developed a model that analyzes the behaviour of the markets using hydro reservoirs as explanatory factors. Their model includes, for example, welfare loss due to socially suboptimal hydro reservoir usage, socially optimal electricity price, hydro reservoir storage and thermal reservoir storage; that are referred as outcomes. However, the model does not explain the real market condition but rather an ideal situation. In the model the market is controlled by one agent, i.e. one agent controls all the power generation reserves; it is referred to as a socially optimal strategy. Article by Kauppi and Liski (2008) includes an assumption where an individual agent has a certain fraction of market power, e.g. 20 % or 30 %. In order to maintain the focus of this thesis, this part of their paper is omitted. The goal of this thesis is two-fold. Firstly we expand the results from the socially optimal strategy for years 2006-08, as the earlier study finishes in 2005. The second objective is to improve on the methods from the previous study. This thesis results several outcomes (SPOT-price and welfare loss, etc.) due to socially optimal actions. Welfare loss is interesting as it describes the inefficiency of the market. SPOT-price is an important output for the market participants as it often has an effect on end users' electricity bills. Another function is to modify and try to improve the model by means of using more accurate input data, e.g. by considering pollution trade rights effect on input data. After modifications to the model, new welfare losses are calculated and compared with the same results before the modifications. The hydro reservoir has the higher explanatory significance in the model followed by thermal power. In Nordic markets, thermal power reserves are mostly nuclear power and other thermal sources (coal, natural gas, oil, peat). It can be argued that hydro and thermal reservoirs determine electricity supply. Roughly speaking, the model takes into account electricity demand and supply, and several parameters related to them (water inflow, oil price, etc.), yielding finally the socially optimal outcomes. The author of this thesis is not aware of any similar model being tested before. There have been some other studies that are close to the Kauppi and Liski (2008) model, but those have a somewhat different focus. For example, a specific feature in the model is the focus on long-run capacity usage that differs from the previous studies on short-run market power. The closest study to the model is from California's wholesale electricity markets that, however, uses different methodology. Work is constructed as follows.

Reinforcement learning applications in dynamic pricing of retail markets

In this paper, we investigate the use of reinforcement learning (RL) techniques to the problem of determining dynamic prices in an electronic retail market. As representative models, we consider a single seller market and a two seller market, and formulate the dynamic pricing problem in a setting that easily generalizes to markets with more than two sellers. We first formulate the single seller dynamic pricing problem in the RL framework and solve the problem using the Q-learning algorithm through simulation. Next we model the two seller dynamic pricing problem as a Markovian game and formulate the problem in the RL framework. We solve this problem using actor-critic algorithms through simulation. We believe our approach to solving these problems is a promising way of setting dynamic prices in multi-agent environments. We illustrate the methodology with two illustrative examples of typical retail markets.

Electrochemical capacitors: Technical challenges and prognosis for future markets

Electrochemical capacitors are potential devices that could help bringing about major advances in future energy storage. They are lightweight and their manufacture and disposal has no detrimental effects on the environment. A comprehensive description of fundamental science of electrochemical capacitors is presented. Similarities and differences between electrochemical capacitors and secondary batteries for electrical energy storage are highlighted and various types of electrochemical capacitors are discussed with special reference to lead-carbon hybrid ultracapacitors. Some envisaged applications of electrochemical capacitors are described along with the technical challenges and prognosis for future markets. (C) 2012 Published by Elsevier Ltd.

Dynamic mechanism design for markets with strategic resources

The assignment of tasks to multiple resources becomes an interesting game theoretic problem, when both the task owner and the resources are strategic. In the classical, nonstrategic setting, where the states of the tasks and resources are observable by the controller, this problem is that of finding an optimal policy for a Markov decision process (MDP). When the states are held by strategic agents, the problem of an efficient task allocation extends beyond that of solving an MDP and becomes that of designing a mechanism. Motivated by this fact, we propose a general mechanism which decides on an allocation rule for the tasks and resources and a payment rule to incentivize agents' participation and truthful reports. In contrast to related dynamic strategic control problems studied in recent literature, the problem studied here has interdependent values: the benefit of an allocation to the task owner is not simply a function of the characteristics of the task itself and the allocation, but also of the state of the resources. We introduce a dynamic extension of Mezzetti's two phase mechanism for interdependent valuations. In this changed setting, the proposed dynamic mechanism is efficient, within period ex-post incentive compatible, and within period ex-post individually rational.

Analysis of volatility transmissions in integrated and interconnected markets: The case of the Iberian and French markets

This paper models the mean and volatility spillovers of prices within the integrated Iberian and the interconnected Spanish and French electricity markets. Using the constant (CCC) and dynamic conditional correlation (DCC) bivariate models with three different specifications of the univariate variance processes, we study the extent to which increasing interconnection and harmonization in regulation have favoured price convergence. The data consist of daily prices calculated as the arithmetic mean of the hourly prices over a span from July 1st 2007 until February 29th 2012. The DCC model in which the variances of the univariate processes are specified with a VARMA(1,1) fits the data best for the integrated MIBEL whereas a CCC model with a GARCH(1,1) specification for the univariate variance processes is selected to model the price series in Spain and France. Results show that there are significant mean and volatility spillovers in the MIBEL, indicating strong interdependence between the two markets, while there is a weaker evidence of integration between the Spanish and French markets. We provide new evidence that the EU target of achieving a single electricity market largely depends on increasing trade between countries and homogeneous rules of market functioning.

Long memory in return structures from developed markets

[En]The present study aimed at investigating the existence of long memory properties in ten developed stock markets across the globe. When return series exhibit long memory, the series realizations are not independent over time and past returns can help predict future returns, thus violating the market efficiency hypothesis. It poses a serious challenge to the supporters of random walk behavior of the stock returns indicating a potentially predictable component in the series dynamics. We computed Hurst-Mandelbrot’s Classical R/S statistic, Lo’s statistic and semi parametric GPH statistic using spectral regression. The findings suggest existence of long memory in volatility and random walk for logarithmic return series in general for all the selected stock market indices. Findings are in line with the stylized facts of financial time series.

Los determinantes de la rentabilidad primaria de los bonos de titulización hipotecaria: la influencia de las estructuras multitramo

[ES] Este artículo analiza los determinantes de la rentabilidad primaria de los bonos de titulización hipotecaria (conocidos en la literatura como mortgage backed securities, o MBS) emitidos en España durante el periodo 1993-2011, periodo en el que el mercado español llegó a convertirse en el más importante de Europa continental. Los resultados obtenidos sobre el análisis de la población completa de MBS emitidos (262 tramos configurados sobre 94 fondos de titulización) indican que la estructuración multitramo de los MBS ha ayudado a reducir el riesgo percibido global de las emisiones de MBS, mediante la generación de mercados más completos y la reducción de los problemas derivados de la existencia de asimetrías informativas implícitas en el proceso de selección de los activos transmitidos por parte de la entidad cedente. Esta reducción del riesgo percibido ha tenido un efecto directo sobre la rentabilidad ofrecida por los bonos de titulización emitidos. Además, no se encuentran evidencias de que la emisión de MBS persiga la transmisión efectiva de riesgos, más bien al contrario. Las Entidades de crédito, por lo general, han retenido los tramos de primeras pérdidas, lo que ha contribuido a mantener en niveles muy bajos (por debajo de la rentabilidad de la deuda soberana) la rentabilidad ofrecida por los MBS. Precisamente, el escaso diferencial ofrecido por los bonos de titulización se debe a que los tramos retenidos no han ofrecido primas de rentabilidad ajustadas al riesgo inherente.