A general communication cost optimization framework for big data stream processing in geo-distributed data centers

With the explosion of big data, processing large numbers of continuous data streams, i.e., big data stream processing (BDSP), has become a crucial requirement for many scientific and industrial applications in recent years. By offering a pool of computation, communication and storage resources, public clouds, like Amazon's EC2, are undoubtedly the most efficient platforms to meet the ever-growing needs of BDSP. Public cloud service providers usually operate a number of geo-distributed datacenters across the globe. Different datacenter pairs are with different inter-datacenter network costs charged by Internet Service Providers (ISPs). While, inter-datacenter traffic in BDSP constitutes a large portion of a cloud provider's traffic demand over the Internet and incurs substantial communication cost, which may even become the dominant operational expenditure factor. As the datacenter resources are provided in a virtualized way, the virtual machines (VMs) for stream processing tasks can be freely deployed onto any datacenters, provided that the Service Level Agreement (SLA, e.g., quality-of-information) is obeyed. This raises the opportunity, but also a challenge, to explore the inter-datacenter network cost diversities to optimize both VM placement and load balancing towards network cost minimization with guaranteed SLA. In this paper, we first propose a general modeling framework that describes all representative inter-task relationship semantics in BDSP. Based on our novel framework, we then formulate the communication cost minimization problem for BDSP into a mixed-integer linear programming (MILP) problem and prove it to be NP-hard. We then propose a computation-efficient solution based on MILP. The high efficiency of our proposal is validated by extensive simulation based studies.

Design and Analysis of Optical Interconnection Networks for a Dataflow Parallel Computer

This work shows the design, simulation, and analysis of two optical interconnection networks for a Dataflow parallel computer architecture. To verify the optical interconnection network performance on the Dataflow architecture, we have analyzed the load balancing among the processors during the parallel programs executions. The load balancing is a very important parameter because it is directly associated to the dataflow parallelism degree. This article proves that optical interconnection networks designed with simple optical devices can provide efficiently the dataflow requirements of a high performance communication system.

Simulation of free surface flows in a distributed memory environment

A parallel technique, for a distributed memory machine, based on domain decomposition for solving the Navier-Stokes equations in cartesian and cylindrical coordinates in two dimensions with free surfaces is described. It is based on the code by Tome and McKee (J. Comp. Phys. 110 (1994) 171-186) and Tome (Ph.D. Thesis, University of Strathclyde, Glasgow, 1993) which in turn is based on the SMAC method by Amsden and Harlow (Report LA-4370, Los Alamos Scientific Laboratory, 1971), which solves the Navier-Stokes equations in three steps: the momentum and Poisson equations and particle movement, These equations are discretized by explicit and 5-point finite differences. The parallelization is performed by splitting the computation domain into vertical panels and assigning each of these panels to a processor. All the computation can then be performed using nearest neighbour communication. Test runs comparing the performance of the parallel with the serial code, and a discussion of the load balancing question are presented. PVM is used for communication between processes. (C) 1999 Elsevier B.V. B.V. All rights reserved.

Reconfiguracao de sistemas de distribuicao radiais utilizando o criterio de queda de tensao

Network reconfiguration in distribution systems can be carried out by changing the status of sectionalizing switches and it is usually done for loss minimization and load balancing. In this paper it is presented an heuristic algorithm that accomplishes network reconfiguration for operation planning in order to obtain a configuration set whose configurations have the smallest active losses on its feeders. To obtain the configurations, it is used an approached radial load flow method and an heuristic proceeding based on maximum limit of voltage drop on feeders. Results are presented for three hypothetical systems largely known whose data are available in literature and a real system with 135 busses. In addition, it is used a fast and robust load flow which decreases the computational effort.

Planning of secondary distribution circuits through evolutionary algorithms

In this work, the planning of secondary distribution circuits is approached as a mixed integer nonlinear programming problem (MINLP). In order to solve this problem, a dedicated evolutionary algorithm (EA) is proposed. This algorithm uses a codification scheme, genetic operators, and control parameters, projected and managed to consider the specific characteristics of the secondary network planning. The codification scheme maps the possible solutions that satisfy the requirements in order to obtain an effective and low-cost projected system-the conductors' adequate dimensioning, load balancing among phases, and the transformer placed at the center of the secondary system loads. An effective algorithm for three-phase power flow is used as an auxiliary methodology of the EA for the calculation of the fitness function proposed for solutions of each topology. Results for two secondary distribution circuits are presented, whereas one presents radial topology and the other a weakly meshed topology. © 2005 IEEE.

Modelo de virtualização distribuída aplicado ao gerenciamento e replicação de cluster multiuso

Pós-graduação em Ciência da Computação - IBILCE

Planejamento de redes de distribuição de energia elétrica de média e baixa tensão

Pós-graduação em Engenharia Elétrica - FEIS

Roteamento multicamada baseado em eficiência energética e qualidade de enlace para redes de sensores sem fio

A pesquisa apresentada nesta dissertação descreve a elaboração de um protocolo de roteamento para aplicações de Redes de Sensores Sem Fio (RSSF) em cidade inteligentes com forte restrição de energia e alta densidade de nodos. Através do estudo dos principais objetivos da comunicação de dados e do levantamento do estado-da-arte sobre os protocolos de roteamento e tecnologias para RSSF, a proposta contempla requisitos como: vazão de dados, confiabilidade de entrega e eficiência energética. A pesquisa apresenta em detalhes o protocolo AODV (Ad hoc On Demand Distance Vector), bem como sua relevância no contexto de RSSF devido a sua popularidade entre as plataformas de dispositivos comercializados. Além disso, são apresentados protocolos derivados do AODV, e a ausência de uma proposta robusta capaz de contemplar os requisitos levantados. O protocolo REL (Routing by Energy and Link Quality) é o resultado da pesquisa levantada e a proposta de solução para roteamento plano sob demanda baseado em eficiência energética e qualidade de enlace para prover um roteamento escalável, capaz de realizar balanceamento de carga e prolongar o tempo de vida da rede. O protocolo REL foi avaliado através de simulação e tesbed, a fim de garantir validação da proposta em ambiente real de escala reduzida e simulado de alta densidade. Os resultados mostraram que o protocolo REL apresenta considerável melhoria de entrega de dados através da escolha de enlaces confiáveis de transmissão e menos suscetíveis a erro, além de moderado consumo de energia capaz de prolongar o tempo de vida da rede, evitando a saturação prematura de nodos.

Utilização do protocolo anycast em ambientes virtuais distribuídos visando à escalabilidade do sistema

Distributed Virtual Environments (DVE) allow multiple users interact with a virtual world by sharing, manipulating objects. Each user has his/her own vision of this world and every changes of state of the environment are distributed among other users. Due to technological constraints, most systems for distributed virtual environments support a limited number of users. The major limitation for development of DVE scale are imposed by the communication system. As the number of users grows, the amount of bandwidth required to exchange information in real time among stations to upgrade the environment and keep it in a consistent state for all connected users. Based on this situation is proposed a framework that uses the protocol anycast in the application layer that provides a load balancing among servers in the region and aims at reducing the latency of message delivery between participants selecting the DVE server best suited for the region participant to connect to the environment

Reconfiguração de sistemas de distribuição operando em vários níveis de demanda através de uma meta-heurística de busca em vizinhança variável

Pós-graduação em Engenharia Elétrica - FEIS

Secure gossiping techniques and components

Gossip protocols have proved to be a viable solution to set-up and manage largescale P2P services or applications in a fully decentralised scenario. The gossip or epidemic communication scheme is heavily based on stochastic behaviors and it is the fundamental idea behind many large-scale P2P protocols. It provides many remarkable features, such as scalability, robustness to failures, emergent load balancing capabilities, fast spreading, and redundancy of information. In some sense, these services or protocols mimic natural system behaviors in order to achieve their goals. The key idea of this work is that the remarkable properties of gossip hold when all the participants follow the rules dictated by the actual protocols. If one or more malicious nodes join the network and start cheating according to some strategy, the result can be catastrophic. In order to study how serious the threat posed by malicious nodes can be and what can be done to prevent attackers from cheating, we focused on a general attack model aimed to defeat a key service in gossip overlay networks (the Peer Sampling Service [JGKvS04]). We also focused on the problem of protecting against forged information exchanged in gossip services. We propose a solution technique for each problem; both techniques are general enough to be applied to distinct service implementations. As gossip protocols, our solutions are based on stochastic behavior and are fully decentralized. In addition, each technique’s behaviour is abstracted by a general primitive function extending the basic gossip scheme; this approach allows the adoptions of our solutions with minimal changes in different scenarios. We provide an extensive experimental evaluation to support the effectiveness of our techniques. Basically, these techniques aim to be building blocks or P2P architecture guidelines in building more resilient and more secure P2P services.

Diseño e instalación de un servicio web de tienda virtual en alta disponibilidad

[ES]Diseño e instalación de un servicio web de tienda virtual en un entorno de computación en clúster para proporcionar alta disponibilidad y balanceo de carga del mencionado servicio. La infraestructura requerida para este despliegue será virtual, utilizándose como plataforma de virtualización KVM. Las tareas se organizan en : 1.Tareas para la creación de un clúster de balanceo de carga donde una máquina será la encargada de recibir las peticiones de los clientes y de redirigirlas a los servidores web según la carga de éstos. 2. Tareas para la creación de un clúster de alta disponibilidad para que en caso de fallo siempre haya un servidor de base de datos que pueda atender a las peticiones de los servidores web.

Cross-layer optimizations in multi-hop ad hoc networks

Unlike traditional wireless networks, characterized by the presence of last-mile, static and reliable infrastructures, Mobile ad Hoc Networks (MANETs) are dynamically formed by collections of mobile and static terminals that exchange data by enabling each other's communication. Supporting multi-hop communication in a MANET is a challenging research area because it requires cooperation between different protocol layers (MAC, routing, transport). In particular, MAC and routing protocols could be considered mutually cooperative protocol layers. When a route is established, the exposed and hidden terminal problems at MAC layer may decrease the end-to-end performance proportionally with the length of each route. Conversely, the contention at MAC layer may cause a routing protocol to respond by initiating new routes queries and routing table updates. Multi-hop communication may also benefit the presence of pseudo-centralized virtual infrastructures obtained by grouping nodes into clusters. Clustering structures may facilitate the spatial reuse of resources by increasing the system capacity: at the same time, the clustering hierarchy may be used to coordinate transmissions events inside the network and to support intra-cluster routing schemes. Again, MAC and clustering protocols could be considered mutually cooperative protocol layers: the clustering scheme could support MAC layer coordination among nodes, by shifting the distributed MAC paradigm towards a pseudo-centralized MAC paradigm. On the other hand, the system benefits of the clustering scheme could be emphasized by the pseudo-centralized MAC layer with the support for differentiated access priorities and controlled contention. In this thesis, we propose cross-layer solutions involving joint design of MAC, clustering and routing protocols in MANETs. As main contribution, we study and analyze the integration of MAC and clustering schemes to support multi-hop communication in large-scale ad hoc networks. A novel clustering protocol, named Availability Clustering (AC), is defined under general nodes' heterogeneity assumptions in terms of connectivity, available energy and relative mobility. On this basis, we design and analyze a distributed and adaptive MAC protocol, named Differentiated Distributed Coordination Function (DDCF), whose focus is to implement adaptive access differentiation based on the node roles, which have been assigned by the upper-layer's clustering scheme. We extensively simulate the proposed clustering scheme by showing its effectiveness in dominating the network dynamics, under some stressing mobility models and different mobility rates. Based on these results, we propose a possible application of the cross-layer MAC+Clustering scheme to support the fast propagation of alert messages in a vehicular environment. At the same time, we investigate the integration of MAC and routing protocols in large scale multi-hop ad-hoc networks. A novel multipath routing scheme is proposed, by extending the AOMDV protocol with a novel load-balancing approach to concurrently distribute the traffic among the multiple paths. We also study the composition effect of a IEEE 802.11-based enhanced MAC forwarding mechanism called Fast Forward (FF), used to reduce the effects of self-contention among frames at the MAC layer. The protocol framework is modelled and extensively simulated for a large set of metrics and scenarios. For both the schemes, the simulation results reveal the benefits of the cross-layer MAC+routing and MAC+clustering approaches over single-layer solutions.

Performance evaluation of a parallel algorithm for simultaneous untangling and smoothing of tetrahedral meshes

[EN]A new parallel algorithm for simultaneous untangling and smoothing of tetrahedral meshes is proposed in this paper. We provide a detailed analysis of its performance on shared-memory many-core computer architectures. This performance analysis includes the evaluation of execution time, parallel scalability, load balancing, and parallelism bottlenecks. Additionally, we compare the impact of three previously published graph coloring procedures on the performance of our parallel algorithm. We use six benchmark meshes with a wide range of sizes. Using these experimental data sets, we describe the behavior of the parallel algorithm for different data sizes. We demonstrate that this algorithm is highly scalable when it runs on two different high-performance many-core computers with up to 128 processors...