Continual learning for computer vision applications

One of the most visionary goals of Artificial Intelligence is to create a system able to mimic and eventually surpass the intelligence observed in biological systems including, ambitiously, the one observed in humans. The main distinctive strength of humans is their ability to build a deep understanding of the world by learning continuously and drawing from their experiences. This ability, which is found in various degrees in all intelligent biological beings, allows them to adapt and properly react to changes by incrementally expanding and refining their knowledge. Arguably, achieving this ability is one of the main goals of Artificial Intelligence and a cornerstone towards the creation of intelligent artificial agents. Modern Deep Learning approaches allowed researchers and industries to achieve great advancements towards the resolution of many long-standing problems in areas like Computer Vision and Natural Language Processing. However, while this current age of renewed interest in AI allowed for the creation of extremely useful applications, a concerningly limited effort is being directed towards the design of systems able to learn continuously. The biggest problem that hinders an AI system from learning incrementally is the catastrophic forgetting phenomenon. This phenomenon, which was discovered in the 90s, naturally occurs in Deep Learning architectures where classic learning paradigms are applied when learning incrementally from a stream of experiences. This dissertation revolves around the Continual Learning field, a sub-field of Machine Learning research that has recently made a comeback following the renewed interest in Deep Learning approaches. This work will focus on a comprehensive view of continual learning by considering algorithmic, benchmarking, and applicative aspects of this field. This dissertation will also touch on community aspects such as the design and creation of research tools aimed at supporting Continual Learning research, and the theoretical and practical aspects concerning public competitions in this field.

Constraint Programming-based Job Dispatching for Modern HPC Applications

A High-Performance Computing job dispatcher is a critical software that assigns the finite computing resources to submitted jobs. This resource assignment over time is known as the on-line job dispatching problem in HPC systems. The fact the problem is on-line means that solutions must be computed in real-time, and their required time cannot exceed some threshold to do not affect the normal system functioning. In addition, a job dispatcher must deal with a lot of uncertainty: submission times, the number of requested resources, and duration of jobs. Heuristic-based techniques have been broadly used in HPC systems, at the cost of achieving (sub-)optimal solutions in a short time. However, the scheduling and resource allocation components are separated, thus generates a decoupled decision that may cause a performance loss. Optimization-based techniques are less used for this problem, although they can significantly improve the performance of HPC systems at the expense of higher computation time. Nowadays, HPC systems are being used for modern applications, such as big data analytics and predictive model building, that employ, in general, many short jobs. However, this information is unknown at dispatching time, and job dispatchers need to process large numbers of them quickly while ensuring high Quality-of-Service (QoS) levels. Constraint Programming (CP) has been shown to be an effective approach to tackle job dispatching problems. However, state-of-the-art CP-based job dispatchers are unable to satisfy the challenges of on-line dispatching, such as generate dispatching decisions in a brief period and integrate current and past information of the housing system. Given the previous reasons, we propose CP-based dispatchers that are more suitable for HPC systems running modern applications, generating on-line dispatching decisions in a proper time and are able to make effective use of job duration predictions to improve QoS levels, especially for workloads dominated by short jobs.

Towards Unstructured Knowledge Integration in Natural Language Processing

In the last decades, Artificial Intelligence has witnessed multiple breakthroughs in deep learning. In particular, purely data-driven approaches have opened to a wide variety of successful applications due to the large availability of data. Nonetheless, the integration of prior knowledge is still required to compensate for specific issues like lack of generalization from limited data, fairness, robustness, and biases. In this thesis, we analyze the methodology of integrating knowledge into deep learning models in the field of Natural Language Processing (NLP). We start by remarking on the importance of knowledge integration. We highlight the possible shortcomings of these approaches and investigate the implications of integrating unstructured textual knowledge. We introduce Unstructured Knowledge Integration (UKI) as the process of integrating unstructured knowledge into machine learning models. We discuss UKI in the field of NLP, where knowledge is represented in a natural language format. We identify UKI as a complex process comprised of multiple sub-processes, different knowledge types, and knowledge integration properties to guarantee. We remark on the challenges of integrating unstructured textual knowledge and bridge connections with well-known research areas in NLP. We provide a unified vision of structured knowledge extraction (KE) and UKI by identifying KE as a sub-process of UKI. We investigate some challenging scenarios where structured knowledge is not a feasible prior assumption and formulate each task from the point of view of UKI. We adopt simple yet effective neural architectures and discuss the challenges of such an approach. Finally, we identify KE as a form of symbolic representation. From this perspective, we remark on the need of defining sophisticated UKI processes to verify the validity of knowledge integration. To this end, we foresee frameworks capable of combining symbolic and sub-symbolic representations for learning as a solution.

Big Code Applications and Approaches

The availability of a huge amount of source code from code archives and open-source projects opens up the possibility to merge machine learning, programming languages, and software engineering research fields. This area is often referred to as Big Code where programming languages are treated instead of natural languages while different features and patterns of code can be exploited to perform many useful tasks and build supportive tools. Among all the possible applications which can be developed within the area of Big Code, the work presented in this research thesis mainly focuses on two particular tasks: the Programming Language Identification (PLI) and the Software Defect Prediction (SDP) for source codes. Programming language identification is commonly needed in program comprehension and it is usually performed directly by developers. However, when it comes at big scales, such as in widely used archives (GitHub, Software Heritage), automation of this task is desirable. To accomplish this aim, the problem is analyzed from different points of view (text and image-based learning approaches) and different models are created paying particular attention to their scalability. Software defect prediction is a fundamental step in software development for improving quality and assuring the reliability of software products. In the past, defects were searched by manual inspection or using automatic static and dynamic analyzers. Now, the automation of this task can be tackled using learning approaches that can speed up and improve related procedures. Here, two models have been built and analyzed to detect some of the commonest bugs and errors at different code granularity levels (file and method levels). Exploited data and models’ architectures are analyzed and described in detail. Quantitative and qualitative results are reported for both PLI and SDP tasks while differences and similarities concerning other related works are discussed.

Concepts and methods for efficient decentralized learning in federated settings

Deep Neural Networks (DNNs) have revolutionized a wide range of applications beyond traditional machine learning and artificial intelligence fields, e.g., computer vision, healthcare, natural language processing and others. At the same time, edge devices have become central in our society, generating an unprecedented amount of data which could be used to train data-hungry models such as DNNs. However, the potentially sensitive or confidential nature of gathered data poses privacy concerns when storing and processing them in centralized locations. To this purpose, decentralized learning decouples model training from the need of directly accessing raw data, by alternating on-device training and periodic communications. The ability of distilling knowledge from decentralized data, however, comes at the cost of facing more challenging learning settings, such as coping with heterogeneous hardware and network connectivity, statistical diversity of data, and ensuring verifiable privacy guarantees. This Thesis proposes an extensive overview of decentralized learning literature, including a novel taxonomy and a detailed description of the most relevant system-level contributions in the related literature for privacy, communication efficiency, data and system heterogeneity, and poisoning defense. Next, this Thesis presents the design of an original solution to tackle communication efficiency and system heterogeneity, and empirically evaluates it on federated settings. For communication efficiency, an original method, specifically designed for Convolutional Neural Networks, is also described and evaluated against the state-of-the-art. Furthermore, this Thesis provides an in-depth review of recently proposed methods to tackle the performance degradation introduced by data heterogeneity, followed by empirical evaluations on challenging data distributions, highlighting strengths and possible weaknesses of the considered solutions. Finally, this Thesis presents a novel perspective on the usage of Knowledge Distillation as a mean for optimizing decentralized learning systems in settings characterized by data heterogeneity or system heterogeneity. Our vision on relevant future research directions close the manuscript.

An architecture for network acceleration as a service in the cloud continuum

The pervasive availability of connected devices in any industrial and societal sector is pushing for an evolution of the well-established cloud computing model. The emerging paradigm of the cloud continuum embraces this decentralization trend and envisions virtualized computing resources physically located between traditional datacenters and data sources. By totally or partially executing closer to the network edge, applications can have quicker reactions to events, thus enabling advanced forms of automation and intelligence. However, these applications also induce new data-intensive workloads with low-latency constraints that require the adoption of specialized resources, such as high-performance communication options (e.g., RDMA, DPDK, XDP, etc.). Unfortunately, cloud providers still struggle to integrate these options into their infrastructures. That risks undermining the principle of generality that underlies the cloud computing scale economy by forcing developers to tailor their code to low-level APIs, non-standard programming models, and static execution environments. This thesis proposes a novel system architecture to empower cloud platforms across the whole cloud continuum with Network Acceleration as a Service (NAaaS). To provide commodity yet efficient access to acceleration, this architecture defines a layer of agnostic high-performance I/O APIs, exposed to applications and clearly separated from the heterogeneous protocols, interfaces, and hardware devices that implement it. A novel system component embodies this decoupling by offering a set of agnostic OS features to applications: memory management for zero-copy transfers, asynchronous I/O processing, and efficient packet scheduling. This thesis also explores the design space of the possible implementations of this architecture by proposing two reference middleware systems and by adopting them to support interactive use cases in the cloud continuum: a serverless platform and an Industry 4.0 scenario. A detailed discussion and a thorough performance evaluation demonstrate that the proposed architecture is suitable to enable the easy-to-use, flexible integration of modern network acceleration into next-generation cloud platforms.

Clustering of variables around latent components: an application in consumer science

The present work proposes a method based on CLV (Clustering around Latent Variables) for identifying groups of consumers in L-shape data. This kind of datastructure is very common in consumer studies where a panel of consumers is asked to assess the global liking of a certain number of products and then, preference scores are arranged in a two-way table Y. External information on both products (physicalchemical description or sensory attributes) and consumers (socio-demographic background, purchase behaviours or consumption habits) may be available in a row descriptor matrix X and in a column descriptor matrix Z respectively. The aim of this method is to automatically provide a consumer segmentation where all the three matrices play an active role in the classification, getting homogeneous groups from all points of view: preference, products and consumer characteristics. The proposed clustering method is illustrated on data from preference studies on food products: juices based on berry fruits and traditional cheeses from Trentino. The hedonic ratings given by the consumer panel on the products under study were explained with respect to the product chemical compounds, sensory evaluation and consumer socio-demographic information, purchase behaviour and consumption habits.

Bayesian Analysis of Linear Inverse Problems with Applications in Economics and Finance

In my PhD thesis I propose a Bayesian nonparametric estimation method for structural econometric models where the functional parameter of interest describes the economic agent's behavior. The structural parameter is characterized as the solution of a functional equation, or by using more technical words, as the solution of an inverse problem that can be either ill-posed or well-posed. From a Bayesian point of view, the parameter of interest is a random function and the solution to the inference problem is the posterior distribution of this parameter. A regular version of the posterior distribution in functional spaces is characterized. However, the infinite dimension of the considered spaces causes a problem of non continuity of the solution and then a problem of inconsistency, from a frequentist point of view, of the posterior distribution (i.e. problem of ill-posedness). The contribution of this essay is to propose new methods to deal with this problem of ill-posedness. The first one consists in adopting a Tikhonov regularization scheme in the construction of the posterior distribution so that I end up with a new object that I call regularized posterior distribution and that I guess it is solution of the inverse problem. The second approach consists in specifying a prior distribution on the parameter of interest of the g-prior type. Then, I detect a class of models for which the prior distribution is able to correct for the ill-posedness also in infinite dimensional problems. I study asymptotic properties of these proposed solutions and I prove that, under some regularity condition satisfied by the true value of the parameter of interest, they are consistent in a "frequentist" sense. Once I have set the general theory, I apply my bayesian nonparametric methodology to different estimation problems. First, I apply this estimator to deconvolution and to hazard rate, density and regression estimation. Then, I consider the estimation of an Instrumental Regression that is useful in micro-econometrics when we have to deal with problems of endogeneity. Finally, I develop an application in finance: I get the bayesian estimator for the equilibrium asset pricing functional by using the Euler equation defined in the Lucas'(1978) tree-type models.

Self-Organizing Mechanisms for Task Allocation in a Knowledge-Based Economy

A prevalent claim is that we are in knowledge economy. When we talk about knowledge economy, we generally mean the concept of “Knowledge-based economy” indicating the use of knowledge and technologies to produce economic benefits. Hence knowledge is both tool and raw material (people’s skill) for producing some kind of product or service. In this kind of environment economic organization is undergoing several changes. For example authority relations are less important, legal and ownership-based definitions of the boundaries of the firm are becoming irrelevant and there are only few constraints on the set of coordination mechanisms. Hence what characterises a knowledge economy is the growing importance of human capital in productive processes (Foss, 2005) and the increasing knowledge intensity of jobs (Hodgson, 1999). Economic processes are also highly intertwined with social processes: they are likely to be informal and reciprocal rather than formal and negotiated. Another important point is also the problem of the division of labor: as economic activity becomes mainly intellectual and requires the integration of specific and idiosyncratic skills, the task of dividing the job and assigning it to the most appropriate individuals becomes arduous, a “supervisory problem” (Hogdson, 1999) emerges and traditional hierarchical control may result increasingly ineffective. Not only specificity of know how makes it awkward to monitor the execution of tasks, more importantly, top-down integration of skills may be difficult because ‘the nominal supervisors will not know the best way of doing the job – or even the precise purpose of the specialist job itself – and the worker will know better’ (Hogdson,1999). We, therefore, expect that the organization of the economic activity of specialists should be, at least partially, self-organized. The aim of this thesis is to bridge studies from computer science and in particular from Peer-to-Peer Networks (P2P) to organization theories. We think that the P2P paradigm well fits with organization problems related to all those situation in which a central authority is not possible. We believe that P2P Networks show a number of characteristics similar to firms working in a knowledge-based economy and hence that the methodology used for studying P2P Networks can be applied to organization studies. Three are the main characteristics we think P2P have in common with firms involved in knowledge economy: - Decentralization: in a pure P2P system every peer is an equal participant, there is no central authority governing the actions of the single peers; - Cost of ownership: P2P computing implies shared ownership reducing the cost of owing the systems and the content, and the cost of maintaining them; - Self-Organization: it refers to the process in a system leading to the emergence of global order within the system without the presence of another system dictating this order. These characteristics are present also in the kind of firm that we try to address and that’ why we have shifted the techniques we adopted for studies in computer science (Marcozzi et al., 2005; Hales et al., 2007 [39]) to management science.

Copper(I) phenanthroline complexes and supramolecular systems containing fullerenes: Photophysics, photochemistry and potential applications in sustainable energy technologies.

Chemistry can contribute, in many different ways to solve the challenges we are facing to modify our inefficient and fossil-fuel based energy system. The present work was motivated by the search for efficient photoactive materials to be employed in the context of the energy problem: materials to be utilized in energy efficient devices and in the production of renewable electricity and fuels. We presented a new class of copper complexes, that could find application in lighting techhnologies, by serving as luminescent materials in LEC, OLED, WOLED devices. These technologies may provide substantial energy savings in the lighting sector. Moreover, recently, copper complexes have been used as light harvesting compounds in dye sensitized photoelectrochemical solar cells, which offer a viable alternative to silicon-based photovoltaic technologies. We presented also a few supramolecular systems containing fullerene, e.g. dendrimers, dyads and triads.The most complex among these arrays, which contain porphyrin moieties, are presented in the final chapter. They undergo photoinduced energy- and electron transfer processes also with long-lived charge separated states, i.e. the fundamental processes to power artificial photosynthetic systems.

Use of solvents and environmental friendly materials for applications in Green Chemistry

The present Thesis studies three alternative solvent groups as sustainable replacement of traditional organic solvents. Some aspects of fluorinated solvents, supercritical fluids and ionic liquids, have been analysed with a critical approach and their effective “greenness” has been evaluated from the points of view of the synthesis, the properties and the applications. In particular, the attention has been put on the environmental and human health issues, evaluating the eco-toxicity, the toxicity and the persistence, to underline that applicability and sustainability are subjects with equal importance. The “green” features of fluorous solvents and supercritical fluids are almost well-established; in particular supercritical carbon dioxide (scCO2) is probably the “greenest” solvent among the alternative solvent systems developed in the last years, enabling to combine numerous advantages both from the point of view of industrial/technological applications and eco-compatibility. In the Thesis the analysis of these two classes of alternative solvents has been mainly focused on their applicability, rather than the evaluation of their environmental impact. Specifically they have been evaluated as alternative media for non-aqueous biocatalysis. For this purpose, the hydrophobic ion pairing (HIP), which allows solubilising enzymes in apolar solvents by an ion pairing between the protein and a surfactant, has been investigated as effective enzymatic derivatisation technique to improve the catalytic activity under homogeneous conditions in non conventional media. The results showed that the complex enzyme-surfactant was much more active both in fluorous solvents and in supercritical carbon dioxide than the native form of the enzyme. Ionic liquids, especially imidazolium salts, have been proposed some years ago as “fully green” alternative solvents; however this epithet does not take into account several “brown” aspects such as their synthesis from petro-chemical starting materials, their considerable eco-toxicity, toxicity and resistance to biodegradation, and the difficulty of clearly outline applications in which ionic liquids are really more advantageous than traditional solvents. For all of these reasons in this Thesis a critical analysis of ionic liquids has been focused on three main topics: i) alternative synthesis by introducing structural moieties which could reduce the toxicity of the most known liquid salts, and by using starting materials from renewable resources; ii) on the evaluation of their environmental impact through eco-toxicological tests (Daphnia magna and Vibrio fischeri acute toxicity tests, and algal growth inhibition), toxicity tests (MTT test, AChE inhibition and LDH release tests) and fate and rate of aerobic biodegradation in soil and water; iii) and on the demonstration of their effectiveness as reaction media in organo-catalysis and as extractive solvents in the recovery of vegetable oil from terrestrial and aquatic biomass. The results about eco-toxicity tests with Daphnia magna, Vibrio fischeri and algae, and toxicity assay using cultured cell lines, clearly indicate that the difference in toxicity between alkyl and oxygenated cations relies in differences of polarity, according to the general trend of decreasing toxicity by decreasing the lipophilicity. Independently by the biological approach in fact, all the results are in agreement, showing a lower toxicity for compounds with oxygenated lateral chains than for those having purely alkyl lateral chains. These findings indicate that an appropriate choice of cation and anion structures is important not only to design the IL with improved and suitable chemico-physical properties but also to obtain safer and eco-friendly ILs. Moreover there is a clear indication that the composition of the abiotic environment has to be taken into account when the toxicity of ILs in various biological test systems is analysed, because, for example, the data reported in the Thesis indicate a significant influence of salinity variations on algal toxicity. Aerobic biodegradation of four imidazolium ionic liquids, two alkylated and two oxygenated, in soil was evaluated for the first time. Alkyl ionic liquids were shown to be biodegradable over the 6 months test period, and in contrast no significant mineralisation was observed with oxygenated derivatives. A different result was observed in the aerobic biodegradation of alkylated and oxygenated pyridinium ionic liquids in water because all the ionic liquids were almost completely degraded after 10 days, independently by the number of oxygen in the lateral chain of the cation. The synthesis of new ionic liquids by using renewable feedstock as starting materials, has been developed through the synthesis of furan-based ion pairs from furfural. The new ammonium salts were synthesised in very good yields, good purity of the products and wide versatility, combining low melting points with high decomposition temperatures and reduced viscosities. Regarding the possible applications as surfactants and biocides, furan-based salts could be a valuable alternative to benzyltributylammonium salts and benzalkonium chloride that are produced from non-renewable resources. A new procedure for the allylation of ketones and aldehydes with tetraallyltin in ionic liquids was developed. The reaction afforded high yields both in sulfonate-containing ILs and in ILs without sulfonate upon addition of a small amount of sulfonic acid. The checked reaction resulted in peculiar chemoselectivity favouring aliphatic substrates towards aromatic ketones and good stereoselectivity in the allylation of levoglucosenone. Finally ILs-based systems could be easily and successfully recycled, making the described procedure environmentally benign. The potential role of switchable polarity solvents as a green technology for the extraction of vegetable oil from terrestrial and aquatic biomass has been investigated. The extraction efficiency of terrestrial biomass rich in triacylglycerols, as soy bean flakes and sunflower seeds, was comparable to those of traditional organic solvents, being the yield of vegetable oils recovery very similar. Switchable polarity solvents as been also exploited for the first time in the extraction of hydrocarbons from the microalga Botryococcus braunii, demonstrating the efficiency of the process for the extraction of both dried microalgal biomass and directly of the aqueous growth medium. The switchable polarity solvents exhibited better extraction efficiency than conventional solvents, both with dried and liquid samples. This is an important issue considering that the harvest and the dewatering of algal biomass have a large impact on overall costs and energy balance.

Sustainable inorganic Binders and Their Applications in Building Engineering: A Green Alternative to Ordinary Portland Cement

In the last decades, the building materials and construction industry has been contributing to a great extent to generate a high impact on our environment. As it has been considered one of the key areas in which to operate to significantly reduce our footprint on environment, there has been widespread belief that particular attention now has to be paid and specific measures have to be taken to limit the use of non-renewable resources.The aim of this thesis is therefore to study and evaluate sustainable alternatives to commonly used building materials, mainly based on ordinary Portland Cement, and find a supportable path to reduce CO2 emissions and promote the re-use of waste materials. More specifically, this research explores different solutions for replacing cementitious binders in distinct application fields, particularly where special and more restricting requirements are needed, such as restoration and conservation of architectural heritage. Emphasis was thus placed on aspects and implications more closely related to the concept of non-invasivity and environmental sustainability. A first part of the research was addressed to the study and development of sustainable inorganic matrices, based on lime putty, for the pre-impregnation and on-site binding of continuous carbon fiber fabrics for structural rehabilitation and heritage restoration. Moreover, with the aim to further limit the exploitation of non-renewable resources, the synthesis of chemically activated silico-aluminate materials, as metakaolin, ladle slag or fly ash, was thus successfully achieved. New sustainable binders were hence proposed as novel building materials, suitable to be used as primary component for construction and repair mortars, as bulk materials in high-temperature applications or as matrices for high-toughness fiber reinforced composites.

When the Practice of Theorizing Meets the Theorizing of Practice. Social Knowledge Making in Organization Science Academia and Managerial Communities

The aim of the present work is to contribute to a better understanding of the relation between organization theory and management practice. It is organized as a collection of two papers, a theoretical and conceptual contribution and an ethnographic study. The first paper is concerned with systematizing different literatures inside and outside the field of organization studies that deal with the theory-practice relation. After identifying a series of positions to the theory-practice debate and unfolding some of their implicit assumptions and limitations, a new position called entwinement is developed in order to overcome status quo through reconciliation and integration. Accordingly, the paper proposes to reconceptualize theory and practice as a circular iterative process of action and cognition, science and common-sense enacted in the real world both by organization scholars and practitioners according to purposes at hand. The second paper is the ethnographic study of an encounter between two groups of expert academics and practitioners occasioned by a one-year executive business master in an international business school. The research articulates a process view of the knowledge exchange between management academics and practitioners in particular and between individuals belonging to different communities of practice, in general, and emphasizes its dynamic, relational and transformative mechanisms. Findings show that when they are given the chance to interact, academics and practitioners set up local provisional relations that enable them to act as change intermediaries vis-a-vis each other’s worlds, without tying themselves irremediably to each other and to the scenarios they conjointly projected during the master’s experience. Finally, the study shows that provisional relations were accompanied by a recursive shift in knowledge modes. While interacting, academics passed from theory to practical theorizing, practitioners passed from an involved practical mode to a reflexive and quasi-theoretical one, and then, as exchanges proceeded, the other way around.

Automatic Deployment of Applications in the Cloud

In distributed systems like clouds or service oriented frameworks, applications are typically assembled by deploying and connecting a large number of heterogeneous software components, spanning from fine-grained packages to coarse-grained complex services. The complexity of such systems requires a rich set of techniques and tools to support the automation of their deployment process. By relying on a formal model of components, a technique is devised for computing the sequence of actions allowing the deployment of a desired configuration. An efficient algorithm, working in polynomial time, is described and proven to be sound and complete. Finally, a prototype tool implementing the proposed algorithm has been developed. Experimental results support the adoption of this novel approach in real life scenarios.