Concepts and methods for efficient decentralized learning in federated settings

Deep Neural Networks (DNNs) have revolutionized a wide range of applications beyond traditional machine learning and artificial intelligence fields, e.g., computer vision, healthcare, natural language processing and others. At the same time, edge devices have become central in our society, generating an unprecedented amount of data which could be used to train data-hungry models such as DNNs. However, the potentially sensitive or confidential nature of gathered data poses privacy concerns when storing and processing them in centralized locations. To this purpose, decentralized learning decouples model training from the need of directly accessing raw data, by alternating on-device training and periodic communications. The ability of distilling knowledge from decentralized data, however, comes at the cost of facing more challenging learning settings, such as coping with heterogeneous hardware and network connectivity, statistical diversity of data, and ensuring verifiable privacy guarantees. This Thesis proposes an extensive overview of decentralized learning literature, including a novel taxonomy and a detailed description of the most relevant system-level contributions in the related literature for privacy, communication efficiency, data and system heterogeneity, and poisoning defense. Next, this Thesis presents the design of an original solution to tackle communication efficiency and system heterogeneity, and empirically evaluates it on federated settings. For communication efficiency, an original method, specifically designed for Convolutional Neural Networks, is also described and evaluated against the state-of-the-art. Furthermore, this Thesis provides an in-depth review of recently proposed methods to tackle the performance degradation introduced by data heterogeneity, followed by empirical evaluations on challenging data distributions, highlighting strengths and possible weaknesses of the considered solutions. Finally, this Thesis presents a novel perspective on the usage of Knowledge Distillation as a mean for optimizing decentralized learning systems in settings characterized by data heterogeneity or system heterogeneity. Our vision on relevant future research directions close the manuscript.

Computational methods applied to undeciphered scripts from the Aegean and Cyprus: Linear A and Cypro-Minoan

The study of ancient, undeciphered scripts presents unique challenges, that depend both on the nature of the problem and on the peculiarities of each writing system. In this thesis, I present two computational approaches that are tailored to two different tasks and writing systems. The first of these methods is aimed at the decipherment of the Linear A afraction signs, in order to discover their numerical values. This is achieved with a combination of constraint programming, ad-hoc metrics and paleographic considerations. The second main contribution of this thesis regards the creation of an unsupervised deep learning model which uses drawings of signs from ancient writing system to learn to distinguish different graphemes in the vector space. This system, which is based on techniques used in the field of computer vision, is adapted to the study of ancient writing systems by incorporating information about sequences in the model, mirroring what is often done in natural language processing. In order to develop this model, the Cypriot Greek Syllabary is used as a target, since this is a deciphered writing system. Finally, this unsupervised model is adapted to the undeciphered Cypro-Minoan and it is used to answer open questions about this script. In particular, by reconstructing multiple allographs that are not agreed upon by paleographers, it supports the idea that Cypro-Minoan is a single script and not a collection of three script like it was proposed in the literature. These results on two different tasks shows that computational methods can be applied to undeciphered scripts, despite the relatively low amount of available data, paving the way for further advancement in paleography using these methods.

Integrated communication and learning towards autonomous mobile radio networks

The integration of distributed and ubiquitous intelligence has emerged over the last years as the mainspring of transformative advancements in mobile radio networks. As we approach the era of “mobile for intelligence”, next-generation wireless networks are poised to undergo significant and profound changes. Notably, the overarching challenge that lies ahead is the development and implementation of integrated communication and learning mechanisms that will enable the realization of autonomous mobile radio networks. The ultimate pursuit of eliminating human-in-the-loop constitutes an ambitious challenge, necessitating a meticulous delineation of the fundamental characteristics that artificial intelligence (AI) should possess to effectively achieve this objective. This challenge represents a paradigm shift in the design, deployment, and operation of wireless networks, where conventional, static configurations give way to dynamic, adaptive, and AI-native systems capable of self-optimization, self-sustainment, and learning. This thesis aims to provide a comprehensive exploration of the fundamental principles and practical approaches required to create autonomous mobile radio networks that seamlessly integrate communication and learning components. The first chapter of this thesis introduces the notion of Predictive Quality of Service (PQoS) and adaptive optimization and expands upon the challenge to achieve adaptable, reliable, and robust network performance in dynamic and ever-changing environments. The subsequent chapter delves into the revolutionary role of generative AI in shaping next-generation autonomous networks. This chapter emphasizes achieving trustworthy uncertainty-aware generation processes with the use of approximate Bayesian methods and aims to show how generative AI can improve generalization while reducing data communication costs. Finally, the thesis embarks on the topic of distributed learning over wireless networks. Distributed learning and its declinations, including multi-agent reinforcement learning systems and federated learning, have the potential to meet the scalability demands of modern data-driven applications, enabling efficient and collaborative model training across dynamic scenarios while ensuring data privacy and reducing communication overhead.

An Embedded In-Cabin Lightweight High-Performance 3D Gaze Estimation System

Gaze estimation has gained interest in recent years for being an important cue to obtain information about the internal cognitive state of humans. Regardless of whether it is the 3D gaze vector or the point of gaze (PoG), gaze estimation has been applied in various fields, such as: human robot interaction, augmented reality, medicine, aviation and automotive. In the latter field, as part of Advanced Driver-Assistance Systems (ADAS), it allows the development of cutting-edge systems capable of mitigating road accidents by monitoring driver distraction. Gaze estimation can be also used to enhance the driving experience, for instance, autonomous driving. It also can improve comfort with augmented reality components capable of being commanded by the driver's eyes. Although, several high-performance real-time inference works already exist, just a few are capable of working with only a RGB camera on computationally constrained devices, such as a microcontroller. This work aims to develop a low-cost, efficient and high-performance embedded system capable of estimating the driver's gaze using deep learning and a RGB camera. The proposed system has achieved near-SOTA performances with about 90% less memory footprint. The capabilities to generalize in unseen environments have been evaluated through a live demonstration, where high performance and near real-time inference were obtained using a webcam and a Raspberry Pi4.

Optical And Nonoptical Aids For Reading And Writing In Individuals With Acquired Low Vision.

To evaluate the use of optical and nonoptical aids during reading and writing activities in individuals with acquired low vision. This study was performed using descriptive and cross-sectional surveys. The data collection instrument was created with structured questions that were developed from an exploratory study and a previous test based on interviews, and it evaluated the following variables: personal characteristics, use of optical and nonoptical aids, and activities that required the use of optical and nonoptical aids. The study population included 30 subjects with acquired low vision and visual acuities of 20/200-20/400. Most subjects reported the use of some optical aids (60.0%). Of these 60.0%, the majority (83.3%) cited spectacles as the most widely used optical aid. The majority (63.3%) of subjects also reported the use of nonoptical aids, the most frequent ones being letter magnification (68.4%), followed by bringing the objects closer to the eyes (57.8%). Subjects often used more than one nonoptical aid. The majority of participants reported the use of optical and nonoptical aids during reading activities, highlighting the use of spectacles, magnifying glasses, and letter magnification; however, even after the use of these aids, we found that the subjects often needed to read the text more than once to understand it. During writing activities, all subjects reported the use of optical aids, while most stated that they did not use nonoptical aids for such activities.

Trajectory Sensitivity Method and Master-Slave Synchronization to Estimate Parameters of Nonlinear Systems

A combination of trajectory sensitivity method and master-slave synchronization was proposed to parameter estimation of nonlinear systems. It was shown that master-slave coupling increases the robustness of the trajectory sensitivity algorithm with respect to the initial guess of parameters. Since synchronization is not a guarantee that the estimation process converges to the correct parameters, a conditional test that guarantees that the new combined methodology estimates the true values of parameters was proposed. This conditional test was successfully applied to Lorenz's and Chua's systems, and the proposed parameter estimation algorithm has shown to be very robust with respect to parameter initial guesses and measurement noise for these examples. Copyright (C) 2009 Elmer P. T. Cari et al.

Physical controls on carbon dioxide transfer velocity and flux in low-gradient river systems and implications for regional carbon budgets

Outgassing of carbon dioxide (CO(2)) from rivers and streams to the atmosphere is a major loss term in the coupled terrestrial-aquatic carbon cycle of major low-gradient river systems (the term ""river system"" encompasses the rivers and streams of all sizes that compose the drainage network in a river basin). However, the magnitude and controls on this important carbon flux are not well quantified. We measured carbon dioxide flux rates (F(CO2)), gas transfer velocity (k), and partial pressures (p(CO2)) in rivers and streams of the Amazon and Mekong river systems in South America and Southeast Asia, respectively. F(CO2) and k values were significantly higher in small rivers and streams (channels <100 m wide) than in large rivers (channels >100 m wide). Small rivers and streams also had substantially higher variability in k values than large rivers. Observed F(CO2) and k values suggest that previous estimates of basinwide CO(2) evasion from tropical rivers and wetlands have been conservative and are likely to be revised upward substantially in the future. Data from the present study combined with data compiled from the literature collectively suggest that the physical control of gas exchange velocities and fluxes in low-gradient river systems makes a transition from the dominance of wind control at the largest spatial scales (in estuaries and river mainstems) toward increasing importance of water current velocity and depth at progressively smaller channel dimensions upstream. These results highlight the importance of incorporating scale-appropriate k values into basinwide models of whole ecosystem carbon balance.

Improvements on ICA mixture models for image pre-processing and segmentation

Today several different unsupervised classification algorithms are commonly used to cluster similar patterns in a data set based only on its statistical properties. Specially in image data applications, self-organizing methods for unsupervised classification have been successfully applied for clustering pixels or group of pixels in order to perform segmentation tasks. The first important contribution of this paper refers to the development of a self-organizing method for data classification, named Enhanced Independent Component Analysis Mixture Model (EICAMM), which was built by proposing some modifications in the Independent Component Analysis Mixture Model (ICAMM). Such improvements were proposed by considering some of the model limitations as well as by analyzing how it should be improved in order to become more efficient. Moreover, a pre-processing methodology was also proposed, which is based on combining the Sparse Code Shrinkage (SCS) for image denoising and the Sobel edge detector. In the experiments of this work, the EICAMM and other self-organizing models were applied for segmenting images in their original and pre-processed versions. A comparative analysis showed satisfactory and competitive image segmentation results obtained by the proposals presented herein. (C) 2008 Published by Elsevier B.V.

Energy systems contributions in 2,000 m race simulation: a comparison among rowing ergometers and water

This study investigated the energy system contributions of rowers in three different conditions: rowing on an ergometer without and with the slide and rowing in the water. For this purpose, eight rowers were submitted to 2,000 m race simulations in each of the situations defined above. The fractions of the aerobic (W(AER)), anaerobic alactic (W(PCR)) and anaerobic lactic (W([La-])) systems were calculated based on the oxygen uptake, the fast component of excess post-exercise oxygen uptake and changes in net blood lactate, respectively. In the water, the metabolic work was significantly higher [(851 (82) kJ] than during both ergometer [674 (60) kJ] and ergometer with slide [663 (65) kJ] (P <= 0.05). The time in the water [515 (11) s] was higher (P < 0.001) than in the ergometers with [398 (10) s] and without the slide [402 (15) s], resulting in no difference when relative energy expenditure was considered: in the water [99 (9) kJ min(-1)], ergometer without the slide [99.6 (9) kJ min(-1)] and ergometer with the slide [100.2 (9.6) kJ min(-1)]. The respective contributions of the WAER, WPCR and W[La-] systems were water = 87 (2), 7 (2) and 6 (2)%, ergometer = 84 (2), 7 (2) and 9 (2)%, and ergometer with the slide = 84 (2), 7 (2) and 9 (1)%. (V) over dotO(2), HR and lactate were not different among conditions. These results seem to indicate that the ergometer braking system simulates conditions of a bigger and faster boat and not a single scull. Probably, a 2,500 m test should be used to properly simulate in the water single-scull race.

Dynamic models for computer viruses

Computer viruses are an important risk to computational systems endangering either corporations of all sizes or personal computers used for domestic applications. Here, classical epidemiological models for disease propagation are adapted to computer networks and, by using simple systems identification techniques a model called SAIC (Susceptible, Antidotal, Infectious, Contaminated) is developed. Real data about computer viruses are used to validate the model. (c) 2008 Elsevier Ltd. All rights reserved.

Prey capture and accommodation in the sandlance, Limnichthyes fasciatus (Creediidae; Teleostei)

The eyes of the sandlance, Limnichthyes fasciatus (Creediidae. Teleostei) move independently and possess a refractive cornea, a convexiclivate fovea and a non-spherical lens giving rise to a wide separation of the nodal point from the axis of rotation of the eye much like that of a chameleon. To investigate this apparent convergence of the visual optics in these phylogenetically disparate species, we examine feeding behaviour and accommodation in the sandlance with special reference to the possibility that sandlances use accommodation as a depth cue to judge strike length. Frame-by-frame analysis of over 2000 strikes show a 100% success rate. Explosive strikes are completed in 50 ms over prey distances of four body lengths. Close-up video confirms that successful strikes can be initiated monocularly (both normally and after monocular occlusion) showing that binocular cues are not necessary to judge the length of a strike. Additional means of judging prey distance may also be derived from parallax information generated by rotation of the eye as suggested for chameleons. Using photorefraction on anaesthetised sandlances, accommodative changes were induced with acetylcholine and found to range between 120 D and 180 D at a speed of 600-720 D s(-1). The large range of accommodation (25% of the total power) is also thought to be mediated by corneal accommodation where the contraction of a unique cornealis muscle acts to change the corneal curvatures.

Spectral tuning and the visual ecology of mantis shrimps

The compound eyes of mantis shrimps (stomatopod crustaceans) include an unparalleled diversity of visual pigments and spectral receptor classes in retinas of each species. We compared the visual pigment and spectral receptor classes of 12 species of gonodactyloid stomatopods from a variety of photo environments, from intertidal to deep water ( > 50 m), to learn how spectral tuning in the different photoreceptor types is modified within different photic environments. Results show that receptors of the peripheral photoreceptors, those outside the midband which are responsible for standard visual tasks such as spatial vision and motion detection, reveal the well-known pattern of decreasing lambda(max) with increasing depth. Receptors of midband rows 5 and 6, which are specialized for polarization vision, are similar in all species, having visual lambda(max)-values near 500 nm, independent of depth. Finally the spectral receptors of midband rows 1 to 4 are tuned for maximum coverage of the spectrum of irradiance available in the habitat of each species. The quality of the visual worlds experienced by each species we studied must vary considerably, but all appear to exploit the full capabilities offered by their complex visual systems.

Eye design and color signaling in a stomatopod crustacean Gonodactylus smithii

Many species of stomatopod crustaceans have multiple spectral classes of photoreceptors in their retinas. Behavioral evidence also indicates that stomatopods are capable of discriminating objects by their spectral differences alone, Most animals use only two to four different types of photoreceptors in their color vision systems, typically with broad sensitivity functions, but the stomatopods apparently include eight or more narrowband photoreceptor classes for color recognition. It is also known that stomatopods use several colored body regions in social interactions. To examine why stomatopods may be so 'concerned' with color, we measured the absorption spectra of visual pigments and intrarhabdomal filters, and the reflectance spectra from different parts of the bodies of several individuals of the gonodactyloid stomatopod species, Gonodactylus smithii. We then applied a model of multiple dichromatic channels for color encoding to examine whether the finely tuned color vision was specifically co-evolved with their complex color signals. Although the eye design of stomatopods seems suitable for detecting color signals of their own, the detection of color signals from other animals, such as reef fishes, can be enhanced as well. Color vision in G. smithii is therefore not exclusively adapted to detect its own color signals, but the spectral tuning of some photoreceptors (e.g. midband Rows 2 and 3) enhances the contrast of certain color signals to a large enough degree to make co-evolution between color vision and these rather specific color signals likely. Copyright (C) 2000 S. Karger AG, Basel.

C(alpha)-HOLDER CLASSICAL SOLUTIONS FOR NON-AUTONOMOUS NEUTRAL DIFFERENTIAL EQUATIONS

In this paper we discuss the existence of alpha-Holder classical solutions for non-autonomous abstract partial neutral functional differential equations. An application is considered.