Dynamic equilibrium through reinforcement learning

Reinforcement Learning is an area of Machine Learning that deals with how an agent should take actions in an environment such as to maximize the notion of accumulated reward. This type of learning is inspired by the way humans learn and has led to the creation of various algorithms for reinforcement learning. These algorithms focus on the way in which an agent’s behaviour can be improved, assuming independence as to their surroundings. The current work studies the application of reinforcement learning methods to solve the inverted pendulum problem. The importance of the variability of the environment (factors that are external to the agent) on the execution of reinforcement learning agents is studied by using a model that seeks to obtain equilibrium (stability) through dynamism – a Cart-Pole system or inverted pendulum. We sought to improve the behaviour of the autonomous agents by changing the information passed to them, while maintaining the agent’s internal parameters constant (learning rate, discount factors, decay rate, etc.), instead of the classical approach of tuning the agent’s internal parameters. The influence of changes on the state set and the action set on an agent’s capability to solve the Cart-pole problem was studied. We have studied typical behaviour of reinforcement learning agents applied to the classic BOXES model and a new form of characterizing the environment was proposed using the notion of convergence towards a reference value. We demonstrate the gain in performance of this new method applied to a Q-Learning agent.

Basic dyestuffs removal from textile effluents using feathers: equilibrium, kinetic and column studies

This research work has been focused in the study of gallinaceous feathers, a waste that may be valorised as sorbent, to remove the Dark Blue Astrazon 2RN (DBA) from Dystar. This study was focused on the following aspects: optimization of experimental conditions through factorial design methodology, kinetic studies into a continuous stirred tank adsorber (at pH 7 and 20ºC), equilibrium isotherms (at pH 5, 7 and 9 at 20 and 45ºC) and column studies (at 20ºC, at pH 5, 7 and 9). In order to evaluate the influence of the presence of other components in the sorption of the dyestuff, all experiments were performed both for the dyestuff in aqueous solution and in real textile effluent. The pseudo-first and pseudo-second order kinetic models were fitted to the experimental data, being the latter the best fit for the aqueous solution of dyestuff. For the real effluent both models fit the experimental results and there is no statistical difference between them. The Central Composite Design (CCD) was used to evaluate the effects of temperature (15 - 45ºC) and pH (5 - 9) over the sorption in aqueous solution. The influence of pH was more significant than temperature. The optimal conditions selected were 45ºC and pH 9. Both Langmuir and Freundlich models could fit the equilibrium data. In the concentration range studied, the highest sorbent capacity was obtained for the optimal conditions in aqueous solution, which corresponds to a maximum capacity of 47± 4 mg g-1. The Yoon-Nelson, Thomas and Yan’s models fitted well the column experimental data. The highest breakthrough time for 50% removal, 170 min, was obtained at pH 9 in aqueous solution. The presence of the dyeing agents in the real wastewater decreased the sorption of the dyestuff mostly for pH 9, which is the optimal pH. The effect of pH is less pronounced in the real effluent than in aqueous solution. This work shows that feathers can be used as sorbent in the treatment of textile wastewaters containing DBA.

Copper and lead removal by peanut hulls: equilibrium and kinetic studies

This research work aims to study the use of peanut hulls, an agricultural and food industry waste, for copper and lead removal through equilibrium and kinetic parameters evaluation. Equilibrium batch studies were performed in a batch adsorber. The influence of initial pH was evaluated (3–5) and it was selected between 4.0 and 4.5. The maximum sorption capacities obtained for the Langmuir model were 0.21 ± 0.03 and 0.18 ± 0.02 mmol/g, respectively for copper and lead. In bi-component systems, competitive sorption of copper and lead was verified, the total amount adsorbed being around 0.21 mmol of metal per gram of material in both mono and bi-component systems. In the kinetic studies equilibrium was reached after 200 min contact time using a 400 rpm stirring rate, achieving 78% and 58% removal, in mono-component system, for copper and lead respectively. Their removal follows a pseudo-second-order kinetics. These studies show that most of the metals removal occurred in the first 20 min of contact, which shows a good uptake rate in all systems.

Stochastic short-term maintenance scheduling of GENCOs in an oligopolistic electricity market

In the proposed model, the independent system operator (ISO) provides the opportunity for maintenance outage rescheduling of generating units before each short-term (ST) time interval. Long-term (LT) scheduling for 1 or 2 years in advance is essential for the ISO and the generation companies (GENCOs) to decide their LT strategies; however, it is not possible to be exactly followed and requires slight adjustments. The Cournot-Nash equilibrium is used to characterize the decision-making procedure of an individual GENCO for ST intervals considering the effective coordination with LT plans. Random inputs, such as parameters of the demand function of loads, hourly demand during the following ST time interval and the expected generation pattern of the rivals, are included as scenarios in the stochastic mixed integer program defined to model the payoff-maximizing objective of a GENCO. Scenario reduction algorithms are used to deal with the computational burden. Two reliability test systems were chosen to illustrate the effectiveness of the proposed model for the ST decision-making process for future planned outages from the point of view of a GENCO.

Equilibrium distributions of discrete non-autonomous graphs

We introduce the notions of equilibrium distribution and time of convergence in discrete non-autonomous graphs. Under some conditions we give an estimate to the convergence time to the equilibrium distribution using the second largest eigenvalue of some matrices associated with the system.

Signalling in an international cournot model

We consider a trade policy model, where the costs of the home firm are private information but can be signaled through the output levels of the firm to a foreign competitor and a home policymaker. We study the influences of the non-homogeneity of the goods and of the uncertainty on the production costs of the home firm in the signalling strategies by the home firm. We show that some results obtained for homogeneous goods are not robust under non-homogeneity.

Two-production-period in a duopoly with nonprofit and for-profit firms

We investigate endogenous roles in a competition between a nonprofit firm and a for-profit firm in a homogeneous goods market, by allowing two production periods. We find that the Cournot-type equilibrium and one Stackelberg-type equilibrium where the nonprofit firm becomes the follower exist; however, another tackelberg-type equilibrium where the nonprofit firm becomes the leader does not exist.

Uncertainty in a mixed duopoly with quadratic costs

In this paper, we consider a mixed market with uncertain demand, involving one private firm and one public firm with quadratic costs. The model is a two-stage game in which players choose to make their output decisions either in stage 1 or stage 2. We assume that the demand is unknown until the end of the first stage. We compute the output levels at equilibrium in each possible role. We also determine ex-ante and ex-post firms’ payoff functions.

Privatization and government preferences in a mixed duopoly: Stackelberg versus Cournot

We analyse the relationship between the privatization of a public firm and government preferences for tax revenue, by considering a (sequential) Stackelberg duopoly with the public firm as the leader. We assume that the government payoff is given by a weighted sum of tax revenue and the sum of consumer and producer surplus. We get that if the government puts a sufficiently larger weight on tax revenue than on the sum of both surpluses, it will not privatize the public firm. In contrast, if the government puts a moderately larger weight on tax revenue than on the sum of both surpluses, it will privatize the public firm. Furthermore, we compare our results with the ones previously published by an other author obtained in a (simultaneous) Cournot duopoly.

A physical packing sequence algorithm for the container loading problem with static mechanical equilibrium conditions

The container loading problem (CLP) is a combinatorial optimization problem for the spatial arrangement of cargo inside containers so as to maximize the usage of space. The algorithms for this problem are of limited practical applicability if real-world constraints are not considered, one of the most important of which is deemed to be stability. This paper addresses static stability, as opposed to dynamic stability, looking at the stability of the cargo during container loading. This paper proposes two algorithms. The first is a static stability algorithm based on static mechanical equilibrium conditions that can be used as a stability evaluation function embedded in CLP algorithms (e.g. constructive heuristics, metaheuristics). The second proposed algorithm is a physical packing sequence algorithm that, given a container loading arrangement, generates the actual sequence by which each box is placed inside the container, considering static stability and loading operation efficiency constraints.

An economical model for dumping by dumping in a Cournot model

We consider an international trade economical model where two firms of different countries compete in quantities and can use three different strategies: (i) repeated collusion, (ii) deviation from the foreigner firm followed by punishment by the home country and then followed by repeated Cournot, or (iii) repeated deviation followed by punishment. In some cases (ii) and (iii) can be interpreted as dumping.We compute the profits of both firms for each strategy and we characterize the economical parameters where each strategy is adopted by the firms.

Asymmetric taxation in a competition between nonprofit and for-profit firms

This paper considers a Cournot competition between a nonprofit firm and a for-profit firm in a homogeneous goods market, with uncertain demand. Given an asymmetric tax schedule, we compute explicitly the Bayesian-Nash equilibrium. Furthermore, we analize the effects of the tax rate and the degree of altruistic preference on market equilibrium outcomes.

Cournot duopoly with competition in the R&D expenditures

We present a new R&D investment in a Cournot Duopoly model and we analyze the different possible types of Nash R&D investments. We observe that the new production costs region can be decomposed in three economical regions, depending on the Nash R&D investment, showing the relevance of the use of patents in new technologies.

Collateral, default penalties and infinite horizon equilibrium

Pascoa and Seghir (2009) noticed that when collateralized promises become subject to utility penalties on default, Ponzi schemes may occur. However, equilibrium exists in some interesting cases. Under low penalties, equilibrium exists if the collateral does not yield utility (for example, when it is a productive asset or a security). Equilibrium exists also under more severe penalties and collateral utility gains, when the promise or the collateral are nominal assets and the margin requirements are endogenous: relative inflation rates and margin coefficients can make the income effects dominate the penalty effects. An equilibrium refinement avoids no-trade equilibria with unduly repayment beliefs. Our refinement differs from the one used by Dubey, Geanakoplos and Shubik (2005) as it does not eliminate no trade equilibria whose low delivery rates are consistent with the propensity to default of agents that are on the verge of selling.

Reciprocity, inequity aversion, and oligopolistic competition

This paper studies how reciprocity and inequity aversion influence the behavior of firms in imperfectly competitive markets. The paper shows that if reciprocal firms compete à la Cournot, then they are able to sustain “collusive” outcomes under a positive reciprocity equilibrium. By contrast, Stackelberg warfare outcomes may emerge under a negative reciprocity equilibrium. The results for inequity aversion are similar. Cournot competition between inequity averse firms can be harmful to consumers if it leads to equilibria where firms feel compassion toward each other. However, in equilibria where inequity averse firms are envious of each other consumers are better off than if firms were selfish. The paper also shows that only under very restrictive conditions does reciprocity or inequity aversion have an impact on Bertrand competition. Finally, the paper shows that non-selfish preferences have a greater impact on equilibrium outcomes in markets with a small number of firms.