Aktionenlernen mit Selbstorganisierenden Karten und Reinforcement Learning

Die vorliegende Arbeit beschäftigt sich mit der Entwicklung eines Funktionsapproximators und dessen Verwendung in Verfahren zum Lernen von diskreten und kontinuierlichen Aktionen: 1. Ein allgemeiner Funktionsapproximator – Locally Weighted Interpolating Growing Neural Gas (LWIGNG) – wird auf Basis eines Wachsenden Neuralen Gases (GNG) entwickelt. Die topologische Nachbarschaft in der Neuronenstruktur wird verwendet, um zwischen benachbarten Neuronen zu interpolieren und durch lokale Gewichtung die Approximation zu berechnen. Die Leistungsfähigkeit des Ansatzes, insbesondere in Hinsicht auf sich verändernde Zielfunktionen und sich verändernde Eingabeverteilungen, wird in verschiedenen Experimenten unter Beweis gestellt. 2. Zum Lernen diskreter Aktionen wird das LWIGNG-Verfahren mit Q-Learning zur Q-LWIGNG-Methode verbunden. Dafür muss der zugrunde liegende GNG-Algorithmus abgeändert werden, da die Eingabedaten beim Aktionenlernen eine bestimmte Reihenfolge haben. Q-LWIGNG erzielt sehr gute Ergebnisse beim Stabbalance- und beim Mountain-Car-Problem und gute Ergebnisse beim Acrobot-Problem. 3. Zum Lernen kontinuierlicher Aktionen wird ein REINFORCE-Algorithmus mit LWIGNG zur ReinforceGNG-Methode verbunden. Dabei wird eine Actor-Critic-Architektur eingesetzt, um aus zeitverzögerten Belohnungen zu lernen. LWIGNG approximiert sowohl die Zustands-Wertefunktion als auch die Politik, die in Form von situationsabhängigen Parametern einer Normalverteilung repräsentiert wird. ReinforceGNG wird erfolgreich zum Lernen von Bewegungen für einen simulierten 2-rädrigen Roboter eingesetzt, der einen rollenden Ball unter bestimmten Bedingungen abfangen soll.

Functional and neural mechanisms of intertemporal choice

People are daily faced with intertemporal choice, i.e., choices differing in the timing of their consequences, frequently preferring smaller-sooner rewards over larger-delayed ones, reflecting temporal discounting of the value of future outcomes. This dissertation addresses two main goals. New evidence about the neural bases of intertemporal choice is provided. Following the disruption of either the medial orbitofrontal cortex or the insula, the willingness to wait for larger-delayed outcomes is affected in odd directions, suggesting the causal involvement of these areas in regulating the value computation of rewards available with different timings. These findings were also supported by a reported imaging study. Moreover, this dissertation provides new evidence about how temporal discounting can be modulated at a behavioral level through different manipulations, e.g., allowing individuals to think about the distant time, pairing rewards with aversive events, or changing their perceived spatial position. A relationship between intertemporal choice, moral judgements and aging is also discussed. All these findings link together to support a unitary neural model of temporal discounting according to which signals coming from several cortical (i.e., medial orbitofrontal cortex, insula) and subcortical regions (i.e., amygdala, ventral striatum) are integrated to represent the subjective value of both earlier and later rewards, under the top-down regulation of dorsolateral prefrontal cortex. The present findings also support the idea that the process of outcome evaluation is strictly related to the ability to pre-experience and envision future events through self-projection, the anticipation of visceral feelings associated with receiving rewards, and the psychological distance from rewards. Furthermore, taking into account the emotions and the state of arousal at the time of decision seems necessary to understand impulsivity associated with preferring smaller-sooner goods in place of larger-later goods.

Obsessed With Work: A multi-causal and multi-rater approach to workaholism

Workaholism is defined as the combination of two underlying dimensions: working excessively and working compulsively. The present thesis aims at achieving the following purposes: 1) to test whether the interaction between environmental and personal antecedents may enhance workaholism; 2) to develop a questionnaire aimed to assess overwork climate in the workplace; 3) to contrast focal employees’ and coworkers’ perceptions of employees’ workaholism and engagement. Concerning the first purpose, the interaction between overwork climate and person characteristics (achievement motivation, perfectionism, conscientiousness, self-efficacy) was explored on a sample of 333 Dutch employees. The results of moderated regression analyses showed that the interaction between overwork climate and person characteristics is related to workaholism. The second purpose was pursued with two interrelated studies. In Study 1 the Overwork Climate Scale (OWCS) was developed and tested using a principal component analysis (N = 395) and a confirmatory factor analysis (N = 396). Two overwork climate dimensions were distinguished, overwork endorsement and lacking overwork rewards. In Study 2 the total sample (N = 791) was used to explore the association of overwork climate with two types of working hard: work engagement and workaholism. Lacking overwork rewards was negatively associated with engagement, whereas overwork endorsement showed a positive association with workaholism. Concerning the third purpose, using a sample of 73 dyads composed by focal employees and their coworkers, a multitrait-multimethod matrix and a correlated trait-correlated method model, i.e. the CT-C(M–1) model, were examined. Our results showed a considerable agreement between raters on focal employees' engagement and workaholism. In contrast, we observed a significant difference concerning the cognitive dimension of workaholism, working compulsively. Moreover, we provided further evidence for the discriminant validity between engagement and workaholism. Overall, workaholism appears as a negative work-related state that could be better explained by assuming a multi-causal and multi-rater approach.

Experiments on Pro-Sociality and Environmental Issues

The thesis comprises three essays that use experimental methods, one about other-regarding motivations in economic behavior and the others on pro-social behavior in two environmental economics problems. The first chapter studies how the expectations of the others and the concern to maintain a balance between effort exerted and rewards obtained interact in shaping the behavior in a modified dictator game. We find that dictators condition their choices on recipients' expectations only when there is a high probability that the the recipient will not be compensated for her effort. Otherwise, dictators tend to balance the efforts and rewards of the recipients, irrespective of the recipients' expectations. In the second chapter, I investigate the problem of local opposition to large public projects (e.g. landfills, incinerators, etc.). In particular, the experiment shows how the uncertainty about the project's quality makes the community living in the host site skeptical about the project. I also test whether side-transfers and costly information disclosure can help to increase the efficiency. Both tools succesfully make the host more willing to accept the project, but they lead to the realization of different types of projects. The last chapter is an experiment on climate negotiations. To avoid the global warming, countries are called to cooperate in the abatement of their emissions. We study whether the dynamic aspect of the climate change makes cooperation across countries behaviorally more difficult. We also consider inequality across countries as a possible factor that hinders international cooperation.

IMPULSE CONTROL AND COOPERATION IN CAPUCHIN MONKEYS (Cebus apella)

The benefits animals derive from living in social groups have produced the evolution of many forms of cooperative behavior. To cooperate, two or more individuals coordinate their actions to accomplish a common goal. One cognitive process that has the potential to influence cooperation is self control. Individuals delaying their impulsive choice for an immediate reward may potentially receive a larger reward later by cooperating with others. In this study, I measured whether brown capuchin monkeys (Cebus apella) were capable of impulse control and whether impulse control was related to cooperation. Impulse control and cooperation were measured using a lazy susan-like apparatus, on which animals could turn a wheel to receive food rewards. The capuchins went through two training phases that taught them how to turn the wheel efficiently to obtain rewards and how to turn the wheel to obtain the larger of two rewards. After training, I tested impulse control by giving the capuchins a choice between a smaller and a larger reward placed at shorter or more distant locations on the wheel. The capuchins demonstrated impulse control in that they tended to inhibit the impulse to select the smaller reward when it was closer and easier to reach and instead selected the larger reward when it was farther away. Cooperation was tested in all possible dyads of seven individuals, a total of 21 dyads, by allowing each dyad 10 trials to work together with effort on the lazy-susan so that each would obtain a reward. Seventeen out of 21 dyads cooperated by simultaneously moving the wheel in the same direction. The correlation between how often a particular dyad cooperated and their average impulse control score was not statistically significant, r(21) = -.125, p = .591. Capuchins demonstrated impulse control and cooperation using this novel apparatus but the two abilities were not related. Other factors such as the unique social relationship between two individuals may play a more prominent role in the motivation to cooperate rather than the cognitive capacity to inhibit behavior.

Electroencephalographic topography measures of experienced utility

Economic theory distinguishes two concepts of utility: decision utility, objectively quantifiable by choices, and experienced utility, referring to the satisfaction by an obtainment. To date, experienced utility is typically measured with subjective ratings. This study intended to quantify experienced utility by global levels of neuronal activity. Neuronal activity was measured by means of electroencephalographic (EEG) responses to gain and omission of graded monetary rewards at the level of the EEG topography in human subjects. A novel analysis approach allowed approximating psychophysiological value functions for the experienced utility of monetary rewards. In addition, we identified the time windows of the event-related potentials (ERP) and the respective intracortical sources, in which variations in neuronal activity were significantly related to the value or valence of outcomes. Results indicate that value functions of experienced utility and regret disproportionally increase with monetary value, and thus contradict the compressing value functions of decision utility. The temporal pattern of outcome evaluation suggests an initial (∼250 ms) coarse evaluation regarding the valence, concurrent with a finer-grained evaluation of the value of gained rewards, whereas the evaluation of the value of omitted rewards emerges later. We hypothesize that this temporal double dissociation is explained by reward prediction errors. Finally, a late, yet unreported, reward-sensitive ERP topography (∼500 ms) was identified. The sources of these topographical covariations are estimated in the ventromedial prefrontal cortex, the medial frontal gyrus, the anterior and posterior cingulate cortex and the hippocampus/amygdala. The results provide important new evidence regarding “how,” “when,” and “where” the brain evaluates outcomes with different hedonic impact.

Nature, Poetry, and Liminality in the Poetry of Robert Frost

Explores how Frost examines and configures the divide between life’s imperfections and its rewards. I am particularly interested in how Frost positions both the non-human natural world and poetry itself as intermediary (or liminal) realms that might help us live simultaneously in the worlds of reality and the imagination, or truth and beauty, or heaven and earth.

A Test of Absolute Numerousness Judgments in Lion-Tailed Macaques (Macaca Silenus)

Numerous studies have shown that animals have a sense of quantity and can distinguish between relative amounts. The concepts of relative numerousness, estimation, and subitizing are well established in species as diverse as chimpanzees and salamanders. Mobile animals have practical use for an understanding of number in common situations such as predation, mating, and competition. However, the ability to identify discrete quantities has only been firmly established in humans. The purpose of this study was to test for such “absolute numerousness” judgments in three lion-tailed macaques (Macaca silenus), a non-human primate. The three macaques tested had previously been trained on a computerized matchto- sample (MTS) task using geometric shapes. In this study, they were introduced to a MTS task containing a numerical cue, which required the monkeys to match stimuli containing either one or two items for rewards. If monkeys were successful at the initial matching task, they were tested with stimuli in which the position of the items and then the surface area of the items was controlled. If the monkeys could match successfully without using these non-numerical cues, they would demonstrate the capability to make absolute numerousness judgments. None of the monkeys matched successfully using the numerical cue, so no evidence of absolute numerosity was found. Each macaque progressed through the experiment in an individualized manner, attempting a variety of strategies to obtain rewards. These included side preferences and an alternating-side strategy that were unrelated to the numerical cues in the stimuli. When it became clear that the monkeys were not matching based on a stimulus-based cue, they were tested again on matching geometric shapes. All three macaques stopped using their alternate strategies and were able to match shapes successfully, demonstrating that they were still capable of completing the matching task. The data suggest that the monkeys could not transfer this ability to the numerical stimuli. This indicates that the macaques lack a sense of exact quantity, or that they could not recognize the numerical cues in the stimuli as being relevant to the task.

Use Of Conspecific Gestural Cues To Solve An Object-Choice Task In Tufted Capuchin Monkeys (Cebus Apella)

Capuchin monkeys, Cebus sp., utilize a wide array of gestural displays in the wild, including facial displays such as lip-smacking and bare-teeth displays. In captivity, they have been shown to respond to the head orientation of humans, show sensitivity to human attentional states, as well as follow human gazes behind barriers. In this study, I investigated whether tufted capuchin monkeys (Cebus apella) would attend to and utilize the gestural cues of a conspecific to obtain a hidden reward. Two capuchins faced each other in separate compartments of an apparatus with an open field in between. The open field contained two cups with holes on one side such that only one monkey, a so-called cuing monkey, could see the reward inside one of the cups. I then moved the cups toward the other signal-receiving monkey and assessed whether it would utilize untrained cues provided by the cuing monkey to select the cup containing the reward. Two of four female capuchin monkeys learned to select the cup containing the reward significantly more often than chance. Neither of these two monkeys performed over chance spontaneously, however, and the other two monkeys never performed above chance despite many blocks of trials. Successful choices by two monkeys to obtain hidden rewards provided experimental evidence that capuchin monkeys attend to and utilize the gestural cues of conspecifics.

The Returns to Migration: The Influence of Education and Migration Type

We show the impact of migration type on real wages over time. We create a migration and earnings history from the National Longitudinal Survey of Youth over the period 1979-2002. We estimate the effects of primary, onward, and two types of return migration on real wages using a panel data model with individual, location, and time fixed effects. Panel data are well suited for the study of the returns to U.S. internal migration because the influence of migration on wages has been found to occur years after the event. We differentiate return migration into two types: return to a location with ties that form a geographical anchor (home) and return to a prior place of work. We find that real wage growth varies by migration type. Education attainment is a significant factor in real wage growth. Our results show that onward migration is an important channel by which the monetary rewards to a college education are manifested.

The Use of Human and Conspecific Gestures By Capuchin Monkeys to Solve an Object-Choice Task

Most primates live in highly complex social systems, and therefore have evolved similarly complex methods of communicating with each other. One type of communication is the use of manual gestures, which are only found in primates. No substantial evidence exists indicating that monkeys use communicative gestures in the wild. However, monkeys may demonstrate the ability to learn and/or use gestures in certain experimental paradigms since they¿ve been shown to use other visual cues such as gaze. The purpose of this study was to investigate if ten brown capuchin monkeys (Cebus apella) were able to use gestural cues from monkeys and a pointing cue from a human to obtain a hidden reward. They were then tested to determine if they could transfer this skill from monkeys to humans and from humans to monkeys. One group of monkeys was trained and tested using a conspecific as the cue giver, and was then tested with a human cue-giver. The second group of monkeys began training and testing with a human cue giver, and was then tested with a monkey cue giver. I found that two monkeys were able to use gestural cues from conspecifics (e.g., reaching) to obtain a hidden reward and then transfer this ability to a pointing cue from a human. Four monkeys learned to use the human pointing cue first, and then transferred this ability to use the gestural cues from conspecifics to obtain a hidden reward. However, the number of trials it took for each monkey to transfer the ability varied considerably. Some subjects spontaneously transferred in the minimum number of trials needed to reach my criteria for successfully obtaining hidden rewards (N = 40 trials), while others needed a large number of trials to do so (e.g. N = 190 trials). Two subjects did not perform successfully in any of the conditions in which they were tested. One subject successfully used the human pointing cue and a human pointing plus vocalization cue, but did not learn the conspecific cue. One subject learned to use the conspecific cue but not the human pointing cue. This was the first study to test if brown capuchin monkeys could use gestural cues from conspecifics to solve an object choice task. The study was also the first to test if capuchins could transfer this skill from monkeys to humans and from humans to monkeys. Results showed that capuchin monkeys were able to flexibly use communicative gestures when they were both unintentionally given by a conspecific and intentionally given by a human to indicate a source of food.

Capuchin Monkeys (Cebus apella) Fail to Be Equitable in an Ultimatum Game

The ultimatum game is a commonly used economics game testing humans' sense of fairness. In the game, a "proposer" is given a sum of money and is told they can split it however they want with another human partner. The partner can then either accept the division and both proposer and responder receive the proposed amounts, or the responder can reject the offer and neither player will get anything. Human subjects from most western cultures typically share almost half of an allotted amount, but it remains unknown whether our close primate relatives share this generosity. Recent attempts to present chimpanzees with the ultimatum game have provided inconclusive results, with some studies finding the animals share humans' disposition to behave 'fairly' and others concluding that chimpanzees act selfishly to maximize their own rewards. Capuchin monkeys are known to share many human and chimpanzee social and cooperative behaviors, and this study was the first to present capuchin monkeys with a version of the ultimatum game. Subjects were presented with two differently colored tokens representing different qualitative reward contingencies, one equitable and the other inequitable in favor of the subject proposer. Subjects could select and place one of the tokens in a transfer container. The capuchins were first tested with a "dictator game" where, after the subject monkey selected a token, the rewards (equitable or inequitable) were distributed to the subject and a nearby partner monkey that was not an active participant. The capuchins were then tested on an ultimatum game in which after the subject selected and placed a token in the container, the container was moved to the partner. The partner needed to remove the token and transfer it back to the experimenter for the rewards to be distributed. As such, the partner could reject the subject's offer by refusing to participate and neither would receive a reward. The experiment was conducted to determine if the subject monkey would select the equitable reward option rather than the selfish option in order to maintain the partner's cooperation in the task. Capuchin subjects behaved selfishly and selected the inequitable token significantly more often than the equitable token in both the dictator and ultimatum game with no significant difference in preference between the two games. Interestingly, despite the occasional occurrence of rejection by the partner monkeys (resulting in no reward for the subject), subjects never altered their strategy, continuing to prefer the selfish token. The study may indicate that capuchin monkeys have an inability to judge the effect of their behavior on a conspecific's reward outcome, or an indifference to the outcome if there is an individual cost associated with behaving prosocially.

Full Cups, Full Coffers: Tax Strategies and Consumer Culture in the Early Modern German Cities

During the sixteenth and seventeenth centuries, the excise taxes (Ungeld) paid by town residents on the consumption of beer, wine, mead and brandy represented the single most important source of civic revenue for many German cities. In a crisis, these taxes could spike to 70-80% of civic income. This paper examines civic budgets and 'behind-the-scenes' deliberations in a sample of towns in southern Germany in order to illuminate how decisions affecting consumer taxes were made. Even during the sobriety movements of the Reformation and post-Reformation period, tax income from drinkers remained attractive to city leaders because the bulk of the excise tax burden could easily be shifted away from privileged members of society and placed on the population at large. At the same time, governments had to maintain a careful balance between what they needed in order to govern and what the consumer market could bear, for high taxes on drinks were also targeted in many popular revolts. This led to nimble politicking by those responsible for tax decisions. Drink taxes were introduced, raised, lowered and otherwise manipulated based not only on shifting fashions and tastes but also on the degree of economic stress faced by the community. Where civic rulers were successful in striking the right balance, the rewards were considerable. The income from drink sales was a major factor in how the cities of the Empire survived the wars and other crises of the early modern period without going into so much debt that they lost their independence.

Full Cups, Full Coffers: Tax Strategies and Consumer Culture in the Early Modern German Cities

During the sixteenth and seventeenth centuries, the excise taxes (Ungeld) paid by town residents on the consumption of beer, wine, mead and brandy represented the single most important source of civic revenue for many German cities. In a crisis, these taxes could spike to 70–80% of civic income. This paper examines civic budgets and ‘behind-the-scenes’ deliberations in a sample of towns in southern Germany in order to illuminate how decisions affecting consumer taxes were made. Even during the sobriety movements of the Reformation and post-Reformation period, tax income from drinkers remained attractive to city leaders because the bulk of the excise tax burden could easily be shifted away from privileged members of society and placed on the population at large. At the same time, governments had to maintain a careful balance between what they needed in order to govern and what the consumer market could bear, for high taxes on drinks were also targeted in many popular revolts. This led to nimble politicking by those responsible for tax decisions. Drink taxes were introduced, raised, lowered and otherwise manipulated based not only on shifting fashions and tastes but also on the degree of economic stress faced by the community. Where civic rulers were successful in striking the right balance, the rewards were considerable. The income from drink sales was a major factor in how the cities of the Empire survived the wars and other crises of the early modern period without going into so much debt that they lost their independence.

Compensation, Incentives and Risk: Economic Theory and Russian Practise

Mr. Pechersky set out to examine a specific feature of the employer-employee relationship in Russian business organisations. He wanted to study to what extent the so-called "moral hazard" is being solved (if it is being solved at all), whether there is a relationship between pay and performance, and whether there is a correlation between economic theory and Russian reality. Finally, he set out to construct a model of the Russian economy that better reflects the way it actually functions than do certain other well-known models (for example models of incentive compensation, the Shapiro-Stiglitz model etc.). His report was presented to the RSS in the form of a series of manuscripts in English and Russian, and on disc, with many tables and graphs. He begins by pointing out the different examples of randomness that exist in the relationship between employee and employer. Firstly, results are frequently affected by circumstances outside the employee's control that have nothing to do with how intelligently, honestly, and diligently the employee has worked. When rewards are based on results, uncontrollable randomness in the employee's output induces randomness in their incomes. A second source of randomness involves the outside events that are beyond the control of the employee that may affect his or her ability to perform as contracted. A third source of randomness arises when the performance itself (rather than the result) is measured, and the performance evaluation procedures include random or subjective elements. Mr. Pechersky's study shows that in Russia the third source of randomness plays an important role. Moreover, he points out that employer-employee relationships in Russia are sometimes opposite to those in the West. Drawing on game theory, he characterises the Western system as follows. The two players are the principal and the agent, who are usually representative individuals. The principal hires an agent to perform a task, and the agent acquires an information advantage concerning his actions or the outside world at some point in the game, i.e. it is assumed that the employee is better informed. In Russia, on the other hand, incentive contracts are typically negotiated in situations in which the employer has the information advantage concerning outcome. Mr. Pechersky schematises it thus. Compensation (the wage) is W and consists of a base amount, plus a portion that varies with the outcome, x. So W = a + bx, where b is used to measure the intensity of the incentives provided to the employee. This means that one contract will be said to provide stronger incentives than another if it specifies a higher value for b. This is the incentive contract as it operates in the West. The key feature distinguishing the Russian example is that x is observed by the employer but is not observed by the employee. So the employer promises to pay in accordance with an incentive scheme, but since the outcome is not observable by the employee the contract cannot be enforced, and the question arises: is there any incentive for the employer to fulfil his or her promises? Mr. Pechersky considers two simple models of employer-employee relationships displaying the above type of information symmetry. In a static framework the obtained result is somewhat surprising: at the Nash equilibrium the employer pays nothing, even though his objective function contains a quadratic term reflecting negative consequences for the employer if the actual level of compensation deviates from the expectations of the employee. This can lead, for example, to labour turnover, or the expenses resulting from a bad reputation. In a dynamic framework, the conclusion can be formulated as follows: the higher the discount factor, the higher the incentive for the employer to be honest in his/her relationships with the employee. If the discount factor is taken to be a parameter reflecting the degree of (un)certainty (the higher the degree of uncertainty is, the lower is the discount factor), we can conclude that the answer to the formulated question depends on the stability of the political, social and economic situation in a country. Mr. Pechersky believes that the strength of a market system with private property lies not just in its providing the information needed to compute an efficient allocation of resources in an efficient manner. At least equally important is the manner in which it accepts individually self-interested behaviour, but then channels this behaviour in desired directions. People do not have to be cajoled, artificially induced, or forced to do their parts in a well-functioning market system. Instead, they are simply left to pursue their own objectives as they see fit. Under the right circumstances, people are led by Adam Smith's "invisible hand" of impersonal market forces to take the actions needed to achieve an efficient, co-ordinated pattern of choices. The problem is that, as Mr. Pechersky sees it, there is no reason to believe that the circumstances in Russia are right, and the invisible hand is doing its work properly. Political instability, social tension and other circumstances prevent it from doing so. Mr. Pechersky believes that the discount factor plays a crucial role in employer-employee relationships. Such relationships can be considered satisfactory from a normative point of view, only in those cases where the discount factor is sufficiently large. Unfortunately, in modern Russia the evidence points to the typical discount factor being relatively small. This fact can be explained as a manifestation of aversion to risk of economic agents. Mr. Pechersky hopes that when political stabilisation occurs, the discount factors of economic agents will increase, and the agent's behaviour will be explicable in terms of more traditional models.