Real-time analytics on large dynamic graphs

In today's fast-paced and interconnected digital world, the data generated by an increasing number of applications is being modeled as dynamic graphs. The graph structure encodes relationships among data items, while the structural changes to the graphs as well as the continuous stream of information produced by the entities in these graphs make them dynamic in nature. Examples include social networks where users post status updates, images, videos, etc.; phone call networks where nodes may send text messages or place phone calls; road traffic networks where the traffic behavior of the road segments changes constantly, and so on. There is a tremendous value in storing, managing, and analyzing such dynamic graphs and deriving meaningful insights in real-time. However, a majority of the work in graph analytics assumes a static setting, and there is a lack of systematic study of the various dynamic scenarios, the complexity they impose on the analysis tasks, and the challenges in building efficient systems that can support such tasks at a large scale. In this dissertation, I design a unified streaming graph data management framework, and develop prototype systems to support increasingly complex tasks on dynamic graphs. In the first part, I focus on the management and querying of distributed graph data. I develop a hybrid replication policy that monitors the read-write frequencies of the nodes to decide dynamically what data to replicate, and whether to do eager or lazy replication in order to minimize network communication and support low-latency querying. In the second part, I study parallel execution of continuous neighborhood-driven aggregates, where each node aggregates the information generated in its neighborhoods. I build my system around the notion of an aggregation overlay graph, a pre-compiled data structure that enables sharing of partial aggregates across different queries, and also allows partial pre-computation of the aggregates to minimize the query latencies and increase throughput. Finally, I extend the framework to support continuous detection and analysis of activity-based subgraphs, where subgraphs could be specified using both graph structure as well as activity conditions on the nodes. The query specification tasks in my system are expressed using a set of active structural primitives, which allows the query evaluator to use a set of novel optimization techniques, thereby achieving high throughput. Overall, in this dissertation, I define and investigate a set of novel tasks on dynamic graphs, design scalable optimization techniques, build prototype systems, and show the effectiveness of the proposed techniques through extensive evaluation using large-scale real and synthetic datasets.

Real-time decision aid display

Gemstone Team Future Firefighting Advancements

Effects of Information and Time of Use Pricing on Irish Electricity Demand and Supply

In this dissertation I quantify residential behavior response to interventions designed to reduce electricity demand at different periods of the day. In the first chapter, I examine the effect of information provision coupled with bimonthly billing, monthly billing, and in-home displays, as well as a time-of-use (TOU) pricing scheme to measure consumption over each month of the Irish Consumer Behavior Trial. I find that time-of-use pricing with real time usage information reduces electricity usage up to 8.7 percent during peak times at the start of the trial but the effect decays over the first three months and after three months the in-home display group is indistinguishable from the monthly treatment group. Monthly and bi-monthly billing treatments are not found to be statistically different from another. These findings suggest that increasing billing reports to the monthly level may be more cost effective for electricity generators who wish to decrease expenses and consumption, rather than providing in-home displays. In the following chapter, I examine the response of residential households after exposure to time of use tariffs at different hours of the day. I find that these treatments reduce electricity consumption during peak hours by almost four percent, significantly lowering demand. Within the model, I find evidence of overall conservation in electricity used. In addition, weekday peak reductions appear to carry over to the weekend when peak pricing is not present, suggesting changes in consumer habit. The final chapter of my dissertation imposes a system wide time of use plan to analyze the potential reduction in carbon emissions from load shifting based on the Ireland and Northern Single Electricity Market. I find that CO2 emissions savings are highest during the winter months when load demand is highest and dirtier power plants are scheduled to meet peak demand. TOU pricing allows for shifting in usage from peak usage to off peak usage and this shift in load can be met with cleaner and cheaper generated electricity from imports, high efficiency gas units, and hydro units.