An investigation of two real time machine learning techniques that could enhance the adaptability of game AI agents

Developers strive to create innovative Artificial Intelligence (AI) behaviour in their games as a key selling point. Machine Learning is an area of AI that looks at how applications and agents can be programmed to learn their own behaviour without the need to manually design and implement each aspect of it. Machine learning methods have been utilised infrequently within games and are usually trained to learn offline before the game is released to the players. In order to investigate new ways AI could be applied innovatively to games it is wise to explore how machine learning methods could be utilised in real-time as the game is played, so as to allow AI agents to learn directly from the player or their environment. Two machine learning methods were implemented into a simple 2D Fighter test game to allow the agents to fully showcase their learned behaviour as the game is played. The methods chosen were: Q-Learning and an NGram based system. It was found that N-Grams and QLearning could significantly benefit game developers as they facilitate fast, realistic learning at run-time.

Adaptive Game Design Study of a fully adaptive framework

Adventure! The Paladin Order foi um projecto ambicioso que começou por ser desenvolvido como um video jogo completo. Tinha como objéctivo implementar uma ferramenta diferente que permitisse tornar o jogo completamente adaptativo às decisões do jogador tanto nas interacções e no diálogo com outras personagens, assim como no combate contra os variados inímigos do jogo. Devido à inexperiência do autor uma grande parte do tempo foi passado a estudar e a pesquisar várias possíveis soluções que permitissem criar um ambiente que fosse adaptativo de uma forma simples e interessante, não só para os programadores mas também para qualquer pessoa que fosse responsável por editar o diálogo e a história do jogo. Os resultados foram bastante interessantes, revelando um sistema que depende simultaneamente dos ficheiros de onde é retirado o diálogo, e de um sistema de personalidades que permite definir qual será o comportamento de qualquer objecto do jogo ou, pelo menos, como as outras personagens irão reagir. O produto final é uma ferramenta de bases sólidas que permite uma implementação relativamente simples de um sistema abrangente e adaptativo, com poucas falhas e apenas algumas questões de simplicidade de código.

Integrated communication and learning towards autonomous mobile radio networks

The integration of distributed and ubiquitous intelligence has emerged over the last years as the mainspring of transformative advancements in mobile radio networks. As we approach the era of “mobile for intelligence”, next-generation wireless networks are poised to undergo significant and profound changes. Notably, the overarching challenge that lies ahead is the development and implementation of integrated communication and learning mechanisms that will enable the realization of autonomous mobile radio networks. The ultimate pursuit of eliminating human-in-the-loop constitutes an ambitious challenge, necessitating a meticulous delineation of the fundamental characteristics that artificial intelligence (AI) should possess to effectively achieve this objective. This challenge represents a paradigm shift in the design, deployment, and operation of wireless networks, where conventional, static configurations give way to dynamic, adaptive, and AI-native systems capable of self-optimization, self-sustainment, and learning. This thesis aims to provide a comprehensive exploration of the fundamental principles and practical approaches required to create autonomous mobile radio networks that seamlessly integrate communication and learning components. The first chapter of this thesis introduces the notion of Predictive Quality of Service (PQoS) and adaptive optimization and expands upon the challenge to achieve adaptable, reliable, and robust network performance in dynamic and ever-changing environments. The subsequent chapter delves into the revolutionary role of generative AI in shaping next-generation autonomous networks. This chapter emphasizes achieving trustworthy uncertainty-aware generation processes with the use of approximate Bayesian methods and aims to show how generative AI can improve generalization while reducing data communication costs. Finally, the thesis embarks on the topic of distributed learning over wireless networks. Distributed learning and its declinations, including multi-agent reinforcement learning systems and federated learning, have the potential to meet the scalability demands of modern data-driven applications, enabling efficient and collaborative model training across dynamic scenarios while ensuring data privacy and reducing communication overhead.

Decision Support for Energy Contracts Negotiation with Game Theory and Adaptive Learning

This paper presents a decision support methodology for electricity market players’ bilateral contract negotiations. The proposed model is based on the application of game theory, using artificial intelligence to enhance decision support method’s adaptive features. This model is integrated in AiD-EM (Adaptive Decision Support for Electricity Markets Negotiations), a multi-agent system that provides electricity market players with strategic behavior capabilities to improve their outcomes from energy contracts’ negotiations. Although a diversity of tools that enable the study and simulation of electricity markets has emerged during the past few years, these are mostly directed to the analysis of market models and power systems’ technical constraints, making them suitable tools to support decisions of market operators and regulators. However, the equally important support of market negotiating players’ decisions is being highly neglected. The proposed model contributes to overcome the existing gap concerning effective and realistic decision support for electricity market negotiating entities. The proposed method is validated by realistic electricity market simulations using real data from the Iberian market operator—MIBEL. Results show that the proposed adaptive decision support features enable electricity market players to improve their outcomes from bilateral contracts’ negotiations.

Coevolution of adaptive technology, maladaptive culture and population size in a producer-scrounger game.

Technology (i.e. tools, methods of cultivation and domestication, systems of construction and appropriation, machines) has increased the vital rates of humans, and is one of the defining features of the transition from Malthusian ecological stagnation to a potentially perpetual rising population growth. Maladaptations, on the other hand, encompass behaviours, customs and practices that decrease the vital rates of individuals. Technology and maladaptations are part of the total stock of culture carried by the individuals in a population. Here, we develop a quantitative model for the coevolution of cumulative adaptive technology and maladaptive culture in a 'producer-scrounger' game, which can also usefully be interpreted as an 'individual-social' learner interaction. Producers (individual learners) are assumed to invent new adaptations and maladaptations by trial-and-error learning, insight or deduction, and they pay the cost of innovation. Scroungers (social learners) are assumed to copy or imitate (cultural transmission) both the adaptations and maladaptations generated by producers. We show that the coevolutionary dynamics of producers and scroungers in the presence of cultural transmission can have a variety of effects on population carrying capacity. From stable polymorphism, where scroungers bring an advantage to the population (increase in carrying capacity), to periodic cycling, where scroungers decrease carrying capacity, we find that selection-driven cultural innovation and transmission may send a population on the path of indefinite growth or to extinction.

An experimental study of adaptive behavior in an oligopolistic market game

We consider an oligopolistic market game, in which the players are competing firm in the same market of a homogeneous consumption good. The consumer side is represented by a fixed demand function. The firms decide how much to produce of a perishable consumption good, and they decide upon a number of information signals to be sent into the population in order to attract customers. Due to the minimal information provided, the players do not have a well--specified model of their environment. Our main objective is to characterize the adaptive behavior of the players in such a situation.

Approaches to Developing Educational Resources Using AI Embedded in Computer Game Technology. A Case Study: Primary School Literacy

In this extended abstract, we discuss recent research at Worcester into the inclusion of AI into ‘Serious Games’. Serious Games research intends to harness the power of computer game technology to produce educational and training materials. We prefer the name ‘Immersive Environments’ (IEs) since this emphasises the human psychological dimension. Creation of compelling and convincing learning software requires a rich engagement of the learner, and a convincing learning experience. We believe that various aspects of the AI tradition can inform the production of such learning.

Adaptive Learning in Games: Defining Profiles of Competitor Players

Artificial Intelligence has been applied to dynamic games for many years. The ultimate goal is creating responses in virtual entities that display human-like reasoning in the definition of their behaviors. However, virtual entities that can be mistaken for real persons are yet very far from being fully achieved. This paper presents an adaptive learning based methodology for the definition of players’ profiles, with the purpose of supporting decisions of virtual entities. The proposed methodology is based on reinforcement learning algorithms, which are responsible for choosing, along the time, with the gathering of experience, the most appropriate from a set of different learning approaches. These learning approaches have very distinct natures, from mathematical to artificial intelligence and data analysis methodologies, so that the methodology is prepared for very distinct situations. This way it is equipped with a variety of tools that individually can be useful for each encountered situation. The proposed methodology is tested firstly on two simpler computer versus human player games: the rock-paper-scissors game, and a penalty-shootout simulation. Finally, the methodology is applied to the definition of action profiles of electricity market players; players that compete in a dynamic game-wise environment, in which the main goal is the achievement of the highest possible profits in the market.

Dynamic stackelberg game with risk-averse players: optimal risk-sharing under asymmetric information

The objective of this paper is to clarify the interactive nature of the leader-follower relationship when both players are endogenously risk-averse. The analysis is placed in the context of a dynamic closed-loop Stackelberg game with private information. The case of a risk-neutral leader, very often discussed in the literature, is only a borderline possibility in the present study. Each player in the game is characterized by a risk-averse type which is unknown to his opponent. The goal of the leader is to implement an optimal incentive compatible risk-sharing contract. The proposed approach provides a qualitative analysis of adaptive risk behavior profiles for asymmetrically informed players in the context of dynamic strategic interactions modelled as incentive Stackelberg games.

Monitoring of innate and adaptive immune responses in the context of human immunotherapy

Summary1 SummaryCancer patients have a better clinical outcome when their tumours display marked infiltration by memory Τ cells. Moreover, the overrepresentation of Th1 gene signatures in primary tumours correlates with favourable prognosis. Thus, vaccination to induce Τ cells capable of infiltrating and eradicating the tumour seems a promising strategy for the treatment of cancer. Here, I monitored CD4 Τ cell responses in melanoma patients vaccinated with the long synthetic peptides Melan- A16-35(A27L) and NY-ESO-179.108. Most of the patients developed strong and diverse peptide antigen specific CD4 Τ cell responses. Analysis of the fine specificity of CD4 Τ cell antigen recognition led to the identification of two new epitopes. The peptide Melan-A16_35(A27L) was delivered by virus-like particles (VLPs) derived from bacteriophage Οβ, which themselves displayed strong immunogenicity. I show evidence for induction of Οβ- and Melan-A specific CD4 Τ cell responses that developed a Th1 functional profile after repeated vaccination cycles. They also specifically released the chemokines CCL-3 and CCL-4, which play important roles in attracting CD8 Τ cells to the APC surface for priming and formation of Τ cell memory. We further found induction of robust humoral IgG responses upon VLP vaccination, and the lgG1-lgG4 isotype composition depended on the adjuvant used. Since heavy chain class switching largely dépends on the presence of CD4 Τ cell help, this result suggests that the adjuvant can influence the differentiation of elicited CD4 Τ cells, thereby contributing to the quality and function of both Β cells and CD8 Τ cells. The nature of the inflammatory processes in the tumour microenvironment can modulate CD8 Τ cell function. A collaboration was established for the investigation regulation of inflammasome activation in human primary monocytes. We identified IL- 4 and TGF-β as strong inhibitors of IL-1 β secretion, Indicating some level of regulation from effector Th2 and Treg responses. We further found a potent inhibition of inflammasome activation by type I interferon, and demonstrated in vivo inhibition of IL-1 β responses in monocytes from active multiple sclerosis patients under IFN-β therapy. This finding further offers a possible explanation for its success, which mechanism of action is still largely unclear. Interestingly, type I interferon is also being used as adjuvant treatment for tumour free metastatic cutaneous melanoma patients. While its clinical benefit has remained controversial, recent data suggest that the subset of patients with ulcerated primary melanoma lesions can benefit from this therapy. Future investigations will shed light on the implication of the inflammasome in this context, and may offer new strategies for improved adjuvant treatments of melanoma.2 RésuméLes patients atteints de cancer ont une meilleure chance de survie si leurs tumeurs s'avèrent être largement infiltrées par des cellules Τ mémoires. De plus, la surreprésentation d'une signature génique Th1 est en corrélation avec un pronostic favorable. Ainsi, la vaccination visant à induire des cellules Τ capables d'infiltrer et de détruire la tumeur parait être une stratégie prometteuse pour le traitement du cancer. Dans ce travail, j'ai procédé au monitoring de la réponse des cellules Τ CD4 dans des patients atteints de mélanome vaccinés avec les longs peptides synthétiques Melan-A16_35(A27L) et NY-ESO-179_108. Ces peptides représentent des antigènes tumoraux reconnus par des lymphocytes T. La majorité des patients a développé une réponse forte et diversifiée des cellules Τ CD4 spécifiques contre les peptides. L'analyse de la spécificité fine de la reconnaissance antigénique des cellules Τ CD4 nous a conduits à l'identification de deux nouveaux épitopes. Le peptide Melan-Aie. 35(A27L) a été délivré par des particules de type viral (VLPs) dérivés de bactériophages Qβ, qui ont eux-mêmes démontré une forte immunogénicité. Mon travail montre les preuves d'une induction de réponses spécifiques des cellules Τ CD4 contre les Qβ et Melan-A développant un profil fonctionnel Th1 après plusieurs cycles de vaccination. Elles secrètent aussi spécifiquement les chimiokines CCL-3 et CCL-4, qui jouent un rôle important dans l'attraction des cellules Τ CD8 à la surface des cellules présentatrices d'antigènes et contribuent ainsi à induire et former la mémoire cellulaire Τ CD8. Nous avons également remarqué une induction de fortes réponses humorales IgG après vaccination avec les VLPs, et que la composition des isotypes lgG1-lgG4 dépendait de l'adjuvant utilisé. Etant donné qu'une commutation de classe de la chaîne lourde dépend largement ùie l'aide des cellules Τ CD4, ce résultat suggère que l'adjuvant puisse influencer la différeritiation de cellules Τ CD4 en différent types, contribuant ainsi à la qualité et à la fonction des cellules Β et des cellules Τ CD8.La nature des processus d'inflammation dans le microenvironnement tumoral peut moduler la fonction des cellules Τ CD8. Une collaboration a été établie pour investiguer la régulation de l'activation de l'inflammasome dans des monocytes primaires humains. Nous avons identifié l'IL-4 et le TGF-β comme étant de puissants inhibiteurs de la sécrétion de IL-Ιβ, indiquant une certaine régulation de la réponse inflammatoire induite par les cellules Th2 et Τ régulatrices. Nous avons également trouvé une forte inhibition de l'activation de l'inflammasome par l'interféron type I, et nous avons démontré une inhibition in vivo de la réponse IL-1 β dans des monocytes de patients atteints d'une sclérose en plaque active sous traitement IFN-β. Ce résultat nous offre une possible explication du succès de cette thérapie, dont le mécanisme reste à ce jour encore largement obscur. Il est intéressant de noter que l'interféron de type I est également utilisé pour le traitement de patients atteints de mélanome cutané métastasique sans tumeurs. Bien que le bénéfice clinique de ce traitement reste controversé, des études récentes montrent qu'une partie des patients atteints de mélanome primaire ulcéré peut tirer bénéfice de cette thérapie. De futures investigations pourront mieux nous renseigner sur l'implication de l'inflammasome dans ce contexte et offrir de nouvelles stratégies pour améliorer les traitements adjuvants du mélanome.

Intellectual Property Rights and Crop-Improving R&D under Adaptive Destruction, June 2007

This paper studies how the strength of intellectual property rights (IPRs) affects investments in biological innovations when the value of an innovation is stochastically reduced to zero because of the evolution of pest resistance. We frame the problem as a research and development (R&D) investment game in a duopoly model of sequential innovation. We characterize the incentives to invest in R&D under two competing IPR regimes, which differ in their treatment of the follow-on innovations that become necessary because of pest adaptation. Depending on the magnitude of the R&D cost, ex ante firms might prefer an intellectual property regime with or without a “research exemption” provision. The study of the welfare function that also accounts for benefit spillovers to consumers—which is possible analytically under some parametric conditions, and numerically otherwise—shows that the ranking of the two IPR regimes depends critically on the extent of the R&D cost.

A simple adaptive procedure leading to correlated equilibrium

We propose a simple adaptive procedure for playing a game. In thisprocedure, players depart from their current play with probabilities thatare proportional to measures of regret for not having used other strategies(these measures are updated every period). It is shown that our adaptiveprocedure guaranties that with probability one, the sample distributionsof play converge to the set of correlated equilibria of the game. Tocompute these regret measures, a player needs to know his payoff functionand the history of play. We also offer a variation where every playerknows only his own realized payoff history (but not his payoff function).

Rational reasoning or adaptive behavior? Evidence from two-person beauty contest games

Many experiments have shown that human subjects do not necessarily behave in line with game theoretic assumptions and solution concepts. The reasons for this non-conformity are multiple. In this paper we study the argument whether a deviation from game theory is because subjects are rational, but doubt that others are rational as well, compared to the argument that subjects, in general, are boundedly rational themselves. To distinguish these two hypotheses, we study behavior in repeated 2-person and many-person Beauty-Contest-Games which are strategically different from one another. We analyze four different treatments and observe that convergence toward equilibrium is driven by learning through the information about the other player s choice and adaptation rather than self-initiated rational reasoning.

Mechanism of HFR1 function in adaptive growth responses in "Arabidopsis thaliana"

AbstractPlants are sessile organisms, which have evolved an astonishing ability to sense changes in their environment. Depending on the surrounding conditions, such as changes in light and temperature, plants modulate the activity of important transcriptional regulators. The shade avoidance syndrome (SAS) is one important mechanism for shade-intolerant plants to adapt their growth in high vegetative density. In shaded conditions plants sense a diminished red/far-red ratio via the phytochrome system and respond with morphological changes such as elongation growth of stems and petioles. The Phytochrome Interacting Factors 4 and 5 (PIF4 and PIF5) are positive regulators of the SAS and required for a full response (Lorrain et al, 2008). They regulate the SAS by inducing the expression of shade avoidance marker genes such as PIL1, ATHB2, XTR7 and HFR1 (Hornitschek et al, 2009; Lorrain et al, 2008).I investigated the molecular mechanism underlying the regulation of the SAS by HFR1 (long Hypocotyl in FR light). Although HFR1 is a PIF-related bHLH transcription factor, we discovered that HFR1 is a non-DNA binding protein. Moreover, we revealed that HFR1 inhibits an exaggerated SAS by forming non-DNA binding heterodimers with PIF4 and PIF5 (Hornitschek et al, 2009). This negative feedback loop is an important mechanism to limit elongation growth also in elevated temperatures. HFR1 accumulation and activity are highly temperature-dependent and the increased activity of HFR1 at warmer temperatures also provides an important restraint on PIF4-driven elongation growth (Foreman et al, 2011).Finally we performed a genome-wide analysis to determine how PIF4 and PIF5 regulate growth in response to shade. We identified potential PIF5- target genes, which represent many well-known shade-responsive genes. Our analysis of gene expression also revealed a role of PIF4 and PIF5 in simulated sun possibly via the regulation of auxin sensitivity.RésuméLes plantes sont des organismes sessiles ayant développé une capacité surprenante à détecter des changements dans leur environnement. En fonction des conditions extérieures, telles que les variations de lumière ou de température, elles adaptent l'activité d'importants régulateurs transcriptionnels. Le syndrome d'évitement de l'ombre (SAS), est un mécanisme important pour les plantes intolérantes à l'ombre leur permettant d'adapter leur croissance lorsqu'elles se développent dans des conditions de végétations très denses. Dans ces conditions, les plantes détectent une réduction de la quantité relative de lumière rouge par rapport à la lumière rouge-lointain (rapport R/FR). Ce changement, perçu via le système des phytochromes, induit des modifications morphologiques telle qu'une élongation des tiges et des pétioles. Les protéines PIF4 et PIF5 (Phytochrome Interacting Factors) sont des régulateurs positifs du SAS et sont nécessaires pour une réponse complète (Lorrain et al, 2008). Ces facteurs de transcription régulent le SAS en induisant l'expression de gènes marqueurs de cette réponse tels que PIL1, ATHB2, XTR7 et HFR1 (Hornitschek et al, 2009; Lorrain et al, 2008).J'ai étudié les mécanismes moléculaires sous-jacents à la régulation du SAS par HFR1 (long Hypocotyl in FR light). HFR1 est un facteur de transcription type bHLH de la famille des PIF, quoique nous ayons découvert que HFR1 est une protéine ne se liant pas à Γ ADN. Nous avons montré que HFR1 inhibe un SAS exagéré en formant des heterodimères avec PIF4 et PIF5 (Hornitschek et al, 2009). Nous avons également montré que cette boucle de régulation négative est également un mécanisme important pour limiter la croissance de l'élongation dans des conditions de fortes températures. De plus l'accumulation et l'activité de HFR1 augmentent avec la température ce qui permet d'inhiber plus fortement l'effet activateur de PIF4 sur la croissance.Enfin, nous avons effectué une analyse génomique à large échelle afin de déterminer comment PIF4 et PIF5 régulent la croissance en réponse à l'ombre. Nous avons identifié les gènes cibles potentiels de PIF5, correspondant en partie à des gènes connus dans la réponse de l'évitement de l'ombre. Notre analyse de l'expression des gènes a également révélé un rôle important de PIF4 et PIF5 dans des conditions de croissance en plein soleil, probablement via la régulation de la sensibilité à l'auxine.

The underestimation of compliance to direct help requests explained by adaptive sampling

Helping behavior is any intentional behavior that benefits another living being or group (Hogg & Vaughan, 2010). People tend to underestimate the probability that others will comply with their direct requests for help (Flynn & Lake, 2008). This implies that when they need help, they will assess the probability of getting it (De Paulo, 1982, cited in Flynn & Lake, 2008) and then they will tend to estimate one that is actually lower than the real chance, so they may not even consider worth asking for it. Existing explanations for this phenomenon attribute it to a mistaken cost computation by the help seeker, who will emphasize the instrumental cost of “saying yes”, ignoring that the potential helper also needs to take into account the social cost of saying “no”. And the truth is that, especially in face-to-face interactions, the discomfort caused by refusing to help can be very high. In short, help seekers tend to fail to realize that it might be more costly to refuse to comply with a help request rather than accepting. A similar effect has been observed when estimating trustworthiness of people. Fetchenhauer and Dunning (2010) showed that people also tend to underestimate it. This bias is reduced when, instead of asymmetric feedback (getting feedback only when deciding to trust the other person), symmetric feedback (always given) was provided. This cause could as well be applicable to help seeking as people only receive feedback when they actually make their request but not otherwise. Fazio, Shook, and Eiser (2004) studied something that could be reinforcing these outcomes: Learning asymmetries. By means of a computer game called BeanFest, they showed that people learn better about negatively valenced objects (beans in this case) than about positively valenced ones. This learning asymmetry esteemed from “information gain being contingent on approach behavior” (p. 293), which could be identified with what Fetchenhauer and Dunning mention as ‘asymmetric feedback’, and hence also with help requests. Fazio et al. also found a generalization asymmetry in favor of negative attitudes versus positive ones. They attributed it to a negativity bias that “weights resemblance to a known negative more heavily than resemblance to a positive” (p. 300). Applied to help seeking scenarios, this would mean that when facing an unknown situation, people would tend to generalize and infer that is more likely that they get a negative rather than a positive outcome from it, so, along with what it was said before, people will be more inclined to think that they will get a “no” when requesting help. Denrell and Le Mens (2011) present a different perspective when trying to explain judgment biases in general. They deviate from the classical inappropriate information processing (depicted among other by Fiske & Taylor, 2007, and Tversky & Kahneman, 1974) and explain this in terms of ‘adaptive sampling’. Adaptive sampling is a sampling mechanism in which the selection of sample items is conditioned by the values of the variable of interest previously observed (Thompson, 2011). Sampling adaptively allows individuals to safeguard themselves from experiences they went through once and turned out to lay negative outcomes. However, it also prevents them from giving a second chance to those experiences to get an updated outcome that could maybe turn into a positive one, a more positive one, or just one that regresses to the mean, whatever direction that implies. That, as Denrell and Le Mens (2011) explained, makes sense: If you go to a restaurant, and you did not like the food, you do not choose that restaurant again. This is what we think could be happening when asking for help: When we get a “no”, we stop asking. And here, we want to provide a complementary explanation for the underestimation of the probability that others comply with our direct help requests based on adaptive sampling. First, we will develop and explain a model that represents the theory. Later on, we will test it empirically by means of experiments, and will elaborate on the analysis of its results.