Modeling Stock Order Flows and Learning Market-Making from Data

Stock markets employ specialized traders, market-makers, designed to provide liquidity and volume to the market by constantly supplying both supply and demand. In this paper, we demonstrate a novel method for modeling the market as a dynamic system and a reinforcement learning algorithm that learns profitable market-making strategies when run on this model. The sequence of buys and sells for a particular stock, the order flow, we model as an Input-Output Hidden Markov Model fit to historical data. When combined with the dynamics of the order book, this creates a highly non-linear and difficult dynamic system. Our reinforcement learning algorithm, based on likelihood ratios, is run on this partially-observable environment. We demonstrate learning results for two separate real stocks.

The future returns for property required by UK pension funds: the BHS data

Previous studies of the place of Property in the multi-asset portfolio have generally relied on historical data, and have been concerned with the supposed risk reduction effects that Property would have on such portfolios. In this paper a different approach has been taken. Not only are expectations data used, but we have also concentrated upon the required return that Property would have to offer to achieve a holding of 15% in typical UK pension fund portfolios. Using two benchmark portfolios for pension funds, we have shown that Property's required return is less than that expected, and therefore it could justify a 15% holding.

Investigating The Sources Of Fresh Water Affecting The Hydrological Balance Of Lakes Enriquillo And Azuei (Hispaniola) – Data Analysis

The Enriquillo and Azuei are saltwater lakes located in a closed water basin in the southwestern region of the island of La Hispaniola, these have been experiencing dramatic changes in total lake-surface area coverage during the period 1980-2012. The size of Lake Enriquillo presented a surface area of approximately 276 km2 in 1984, gradually decreasing to 172 km2 in 1996. The surface area of the lake reached its lowest point in the satellite observation record in 2004, at 165 km2. Then the recent growth of the lake began reaching its 1984 size by 2006. Based on surface area measurement for June and July 2013, Lake Enriquillo has a surface area of ~358 km2. Sumatra sizes at both ends of the record are 116 km2 in 1984 and 134 km2in 2013, an overall 15.8% increase in 30 years. Determining the causes of lake surface area changes is of extreme importance due to its environmental, social, and economic impacts. The overall goal of this study is to quantify the changing water balance in these lakes and their catchment area using satellite and ground observations and a regional atmospheric-hydrologic modeling approach. Data analyses of environmental variables in the region reflect a hydrological unbalance of the lakes due to changing regional hydro-climatic conditions. Historical data show precipitation, land surface temperature and humidity, and sea surface temperature (SST), increasing over region during the past decades. Salinity levels have also been decreasing by more than 30% from previously reported baseline levels. Here we present a summary of the historical data obtained, new sensors deployed in the sourrounding sierras and the lakes, and the integrated modeling exercises. As well as the challenges of gathering, storing, sharing, and analyzing this large volumen of data in a remote location from such a diverse number of sources.

Detecting an abnormality in a recovery boiler using dynamic multivariate data analysis with parallel coordinate plots

Data visualization techniques are powerful in the handling and analysis of multivariate systems. One such technique known as parallel coordinates was used to support the diagnosis of an event, detected by a neural network-based monitoring system, in a boiler at a Brazilian Kraft pulp mill. Its attractiveness is the possibility of the visualization of several variables simultaneously. The diagnostic procedure was carried out step-by-step going through exploratory, explanatory, confirmatory, and communicative goals. This tool allowed the visualization of the boiler dynamics in an easier way, compared to commonly used univariate trend plots. In addition it facilitated analysis of other aspects, namely relationships among process variables, distinct modes of operation and discrepant data. The whole analysis revealed firstly that the period involving the detected event was associated with a transition between two distinct normal modes of operation, and secondly the presence of unusual changes in process variables at this time.

Volcanic forcing for climate modeling: a new microphysics-based data set covering years 1600–present

As the understanding and representation of the impacts of volcanic eruptions on climate have improved in the last decades, uncertainties in the stratospheric aerosol forcing from large eruptions are now linked not only to visible optical depth estimates on a global scale but also to details on the size, latitude and altitude distributions of the stratospheric aerosols. Based on our understanding of these uncertainties, we propose a new model-based approach to generating a volcanic forcing for general circulation model (GCM) and chemistry–climate model (CCM) simulations. This new volcanic forcing, covering the 1600–present period, uses an aerosol microphysical model to provide a realistic, physically consistent treatment of the stratospheric sulfate aerosols. Twenty-six eruptions were modeled individually using the latest available ice cores aerosol mass estimates and historical data on the latitude and date of eruptions. The evolution of aerosol spatial and size distribution after the sulfur dioxide discharge are hence characterized for each volcanic eruption. Large variations are seen in hemispheric partitioning and size distributions in relation to location/date of eruptions and injected SO2 masses. Results for recent eruptions show reasonable agreement with observations. By providing these new estimates of spatial distributions of shortwave and long-wave radiative perturbations, this volcanic forcing may help to better constrain the climate model responses to volcanic eruptions in the 1600–present period. The final data set consists of 3-D values (with constant longitude) of spectrally resolved extinction coefficients, single scattering albedos and asymmetry factors calculated for different wavelength bands upon request. Surface area densities for heterogeneous chemistry are also provided.

Robust meta-analytic-predictive priors in clinical trials with historical control information.

Historical information is always relevant for clinical trial design. Additionally, if incorporated in the analysis of a new trial, historical data allow to reduce the number of subjects. This decreases costs and trial duration, facilitates recruitment, and may be more ethical. Yet, under prior-data conflict, a too optimistic use of historical data may be inappropriate. We address this challenge by deriving a Bayesian meta-analytic-predictive prior from historical data, which is then combined with the new data. This prospective approach is equivalent to a meta-analytic-combined analysis of historical and new data if parameters are exchangeable across trials. The prospective Bayesian version requires a good approximation of the meta-analytic-predictive prior, which is not available analytically. We propose two- or three-component mixtures of standard priors, which allow for good approximations and, for the one-parameter exponential family, straightforward posterior calculations. Moreover, since one of the mixture components is usually vague, mixture priors will often be heavy-tailed and therefore robust. Further robustness and a more rapid reaction to prior-data conflicts can be achieved by adding an extra weakly-informative mixture component. Use of historical prior information is particularly attractive for adaptive trials, as the randomization ratio can then be changed in case of prior-data conflict. Both frequentist operating characteristics and posterior summaries for various data scenarios show that these designs have desirable properties. We illustrate the methodology for a phase II proof-of-concept trial with historical controls from four studies. Robust meta-analytic-predictive priors alleviate prior-data conflicts ' they should encourage better and more frequent use of historical data in clinical trials.

Reversal of gender differences in Educational Attainment – historical analysis of the West German Case

Background information: During the late 1970s and the early 1980s, West Germany witnessed a reversal of gender differences in educational attainment, as females began to outperform males. Purpose: The main objective was to analyse which processes were behind the reversal of gender differences in educational attainment after 1945. The theoretical reflections and empirical evidence presented for the US context by DiPrete and Buchmann (Gender-specific trends in the value of education and the emerging gender gap in college completion, Demography 43: 1–24, 2006) and Buchmann, DiPrete, and McDaniel (Gender inequalities in education, Annual Review of Sociology 34: 319–37, 2008) are considered and applied to the West German context. It is suggested that the reversal of gender differences is a consequence of the change in female educational decisions, which are mainly related to labour market opportunities and not, as sometimes assumed, a consequence of a ‘boy’s crisis’. Sample: Several databases, such as the German General Social Survey, the German Socio-economic Panel and the German Life History Study, are employed for the longitudinal analysis of the educational and occupational careers of birth cohorts born in the twentieth century. Design and methods: Changing patterns of eligibility for university studies are analysed for successive birth cohorts and gender. Binary logistic regressions are employed for the statistical modelling of the individuals’ achievement, educational decision and likelihood for social mobility – reporting average marginal effects (AME). Results: The empirical results suggest that women’s better school achievement being constant across cohorts does not contribute to the explanation of the reversal of gender differences in higher education attainment, but the increase of benefits for higher education explains the changing educational decisions of women regarding their transition to higher education. Conclusions: The outperformance of females compared with males in higher education might have been initialised by several social changes, including the expansion of public employment, the growing demand for highly qualified female workers in welfare and service areas, the increasing returns of women’s increased education and training, and the improved opportunities for combining family and work outside the home. The historical data show that, in terms of (married) women’s increased labour market opportunities and female life-cycle labour force participation, the raising rates of women’s enrolment in higher education were – among other reasons – partly explained by their rising access to service class positions across birth cohorts, and the rise of their educational returns in terms of wages and long-term employment.

A simulation study on the statistical monitoring of condemnation rates from slaughterhouses for syndromic surveillance: an evaluation based on Swiss data

Syndromic surveillance (SyS) systems currently exploit various sources of health-related data, most of which are collected for purposes other than surveillance (e.g. economic). Several European SyS systems use data collected during meat inspection for syndromic surveillance of animal health, as some diseases may be more easily detected post-mortem than at their point of origin or during the ante-mortem inspection upon arrival at the slaughterhouse. In this paper we use simulation to evaluate the performance of a quasi-Poisson regression (also known as an improved Farrington) algorithm for the detection of disease outbreaks during post-mortem inspection of slaughtered animals. When parameterizing the algorithm based on the retrospective analyses of 6 years of historic data, the probability of detection was satisfactory for large (range 83-445 cases) outbreaks but poor for small (range 20-177 cases) outbreaks. Varying the amount of historical data used to fit the algorithm can help increasing the probability of detection for small outbreaks. However, while the use of a 0·975 quantile generated a low false-positive rate, in most cases, more than 50% of outbreak cases had already occurred at the time of detection. High variance observed in the whole carcass condemnations time-series, and lack of flexibility in terms of the temporal distribution of simulated outbreaks resulting from low reporting frequency (monthly), constitute major challenges for early detection of outbreaks in the livestock population based on meat inspection data. Reporting frequency should be increased in the future to improve timeliness of the SyS system while increased sensitivity may be achieved by integrating meat inspection data into a multivariate system simultaneously evaluating multiple sources of data on livestock health.

Event-based time series data preprocessing: application to traffic flow time series

Traffic flow time series data are usually high dimensional and very complex. Also they are sometimes imprecise and distorted due to data collection sensor malfunction. Additionally, events like congestion caused by traffic accidents add more uncertainty to real-time traffic conditions, making traffic flow forecasting a complicated task. This article presents a new data preprocessing method targeting multidimensional time series with a very high number of dimensions and shows its application to real traffic flow time series from the California Department of Transportation (PEMS web site). The proposed method consists of three main steps. First, based on a language for defining events in multidimensional time series, mTESL, we identify a number of types of events in time series that corresponding to either incorrect data or data with interference. Second, each event type is restored utilizing an original method that combines real observations, local forecasted values and historical data. Third, an exponential smoothing procedure is applied globally to eliminate noise interference and other random errors so as to provide good quality source data for future work.

Una aproximación a la predicción del valor de acciones en la bolsa de valores aplicando técnicas de Data Mining

La predicción del valor de las acciones en la bolsa de valores ha sido un tema importante en el campo de inversiones, que por varios años ha atraído tanto a académicos como a inversionistas. Esto supone que la información disponible en el pasado de la compañía que cotiza en bolsa tiene alguna implicación en el futuro del valor de la misma. Este trabajo está enfocado en ayudar a un persona u organismo que decida invertir en la bolsa de valores a través de gestión de compra o venta de acciones de una compañía a tomar decisiones respecto al tiempo de comprar o vender basado en el conocimiento obtenido de los valores históricos de las acciones de una compañía en la bolsa de valores. Esta decisión será inferida a partir de un modelo de regresión múltiple que es una de las técnicas de datamining. Para llevar conseguir esto se emplea una metodología conocida como CRISP-DM aplicada a los datos históricos de la compañía con mayor valor actual del NASDAQ.---ABSTRACT---The prediction of the value of shares in the stock market has been a major issue in the field of investments, which for several years has attracted both academics and investors. This means that the information available in the company last traded have any involvement in the future of the value of it. This work is focused on helping an investor decides to invest in the stock market through management buy or sell shares of a company to make decisions with respect to time to buy or sell based on the knowledge gained from the historic values of the shares of a company in the stock market. This decision will be inferred from a multiple regression model which is one of the techniques of data mining. To get this out a methodology known as CRISP-DM applied to historical data of the company with the highest current value of NASDAQ is used.

Design and implementation of an adaptive dashboard and visualization interface for the dynamic optimization and control of Data Centers

Over the last few years, the Data Center market has increased exponentially and this tendency continues today. As a direct consequence of this trend, the industry is pushing the development and implementation of different new technologies that would improve the energy consumption efficiency of data centers. An adaptive dashboard would allow the user to monitor the most important parameters of a data center in real time. For that reason, monitoring companies work with IoT big data filtering tools and cloud computing systems to handle the amounts of data obtained from the sensors placed in a data center.Analyzing the market trends in this field we can affirm that the study of predictive algorithms has become an essential area for competitive IT companies. Complex algorithms are used to forecast risk situations based on historical data and warn the user in case of danger. Considering that several different users will interact with this dashboard from IT experts or maintenance staff to accounting managers, it is vital to personalize it automatically. Following that line of though, the dashboard should only show relevant metrics to the user in different formats like overlapped maps or representative graphs among others. These maps will show all the information needed in a visual and easy-to-evaluate way. To sum up, this dashboard will allow the user to visualize and control a wide range of variables. Monitoring essential factors such as average temperature, gradients or hotspots as well as energy and power consumption and savings by rack or building would allow the client to understand how his equipment is behaving, helping him to optimize the energy consumption and efficiency of the racks. It also would help him to prevent possible damages in the equipment with predictive high-tech algorithms.

Historical sea ice of the Baltic Sea (1960/61-78/79)

The Baltic Sea is a seasonally ice-covered, marginal sea in central northern Europe. It is an essential waterway connecting highly industrialised countries. Because ship traffic is intermittently hindered by sea ice, the local weather services have been monitoring sea ice conditions for decades. In the present study we revisit a historical monitoring data set, covering the winters 1960/1961 to 1978/1979. This data set, dubbed Data Bank for Baltic Sea Ice and Sea Surface Temperatures (BASIS) ice, is based on hand-drawn maps that were collected and then digitised in 1981 in a joint project of the Finnish Institute of Marine Research (today the Finnish Meteorological Institute (FMI)) and the Swedish Meteorological and Hydrological Institute (SMHI). BASIS ice was designed for storage on punch cards and all ice information is encoded by five digits. This makes the data hard to access. Here we present a post-processed product based on the original five-digit code. Specifically, we convert to standard ice quantities (including information on ice types), which we distribute in the current and free Network Common Data Format (NetCDF). Our post-processed data set will help to assess numerical ice models and provide easy-to-access unique historical reference material for sea ice in the Baltic Sea. In addition we provide statistics showcasing the data quality. The website http://www.baltic-ocean.org hosts the post-processed data and the conversion code.

Analyzing the Data in OLAP Data Cubes

* Supported partially by the Bulgarian National Science Fund under Grant MM-1405/2004

Historical comparisons reveal multiple drivers of decadal change of an ecosystem engineer at the range edge

Biogenic reefs are important for habitat provision and coastal protection. Long-term datasets on the distribution and abundance of Sabellaria alveolata (L.) are available from Britain. The aim of this study was to combine historical records and contemporary data to (1) describe spatiotemporal variation in winter temperatures, (2) document short-term and long-term changes in the distribution and abundance of S. alveolata and discuss these changes in relation to extreme weather events and recent warming, and (3) assess the potential for artificial coastal defense structures to function as habitat for S. alveolata. A semi-quantitative abundance scale (ACFOR) was used to compare broadscale, long-term and interannual abundance of S. alveolata near its range edge in NW Britain. S. alveolata disappeared from the North Wales and Wirral coastlines where it had been abundant prior to the cold winter of 1962/1963. Population declines were also observed following the recent cold winters of 2009/2010 and 2010/2011. Extensive surveys in 2004 and 2012 revealed that S. alveolata had recolonized locations from which it had previously disappeared. Furthermore, it had increased in abundance at many locations, possibly in response to recent warming. S. alveolata was recorded on the majority of artificial coastal defense structures surveyed, suggesting that the proliferation of artificial coastal defense structures along this stretch of coastline may have enabled S. alveolata to spread across stretches of unsuitable natural habitat. Long-term and broadscale contextual monitoring is essential for monitoring responses of organisms to climate change. Historical data and gray literature can be invaluable sources of information. Our results support the theory that Lusitanian species are responding positively to climate warming but also that short-term extreme weather events can have potentially devastating widespread and lasting effects on organisms. Furthermore, the proliferation of coastal defense structures has implications for phylogeography, population genetics, and connectivity of coastal populations.

Historical comparisons reveal multiple drivers of decadal change of an ecosystem engineer at the range edge

Biogenic reefs are important for habitat provision and coastal protection. Long-term datasets on the distribution and abundance of Sabellaria alveolata (L.) are available from Britain. The aim of this study was to combine historical records and contemporary data to (1) describe spatiotemporal variation in winter temperatures, (2) document short-term and long-term changes in the distribution and abundance of S. alveolata and discuss these changes in relation to extreme weather events and recent warming, and (3) assess the potential for artificial coastal defense structures to function as habitat for S. alveolata. A semi-quantitative abundance scale (ACFOR) was used to compare broadscale, long-term and interannual abundance of S. alveolata near its range edge in NW Britain. S. alveolata disappeared from the North Wales and Wirral coastlines where it had been abundant prior to the cold winter of 1962/1963. Population declines were also observed following the recent cold winters of 2009/2010 and 2010/2011. Extensive surveys in 2004 and 2012 revealed that S. alveolata had recolonized locations from which it had previously disappeared. Furthermore, it had increased in abundance at many locations, possibly in response to recent warming. S. alveolata was recorded on the majority of artificial coastal defense structures surveyed, suggesting that the proliferation of artificial coastal defense structures along this stretch of coastline may have enabled S. alveolata to spread across stretches of unsuitable natural habitat. Long-term and broadscale contextual monitoring is essential for monitoring responses of organisms to climate change. Historical data and gray literature can be invaluable sources of information. Our results support the theory that Lusitanian species are responding positively to climate warming but also that short-term extreme weather events can have potentially devastating widespread and lasting effects on organisms. Furthermore, the proliferation of coastal defense structures has implications for phylogeography, population genetics, and connectivity of coastal populations.