Large scale data processing using MapReduce

As massive data sets become increasingly available, people are facing the problem of how to effectively process and understand these data. Traditional sequential computing models are giving way to parallel and distributed computing models, such as MapReduce, both due to the large size of the data sets and their high dimensionality. This dissertation, as in the same direction of other researches that are based on MapReduce, tries to develop effective techniques and applications using MapReduce that can help people solve large-scale problems. Three different problems are tackled in the dissertation. The first one deals with processing terabytes of raster data in a spatial data management system. Aerial imagery files are broken into tiles to enable data parallel computation. The second and third problems deal with dimension reduction techniques that can be used to handle data sets of high dimensionality. Three variants of the nonnegative matrix factorization technique are scaled up to factorize matrices of dimensions in the order of millions in MapReduce based on different matrix multiplication implementations. Two algorithms, which compute CANDECOMP/PARAFAC and Tucker tensor decompositions respectively, are parallelized in MapReduce based on carefully partitioning the data and arranging the computation to maximize data locality and parallelism.

A youth vision of the city: The socio-spatial lives and exclusion of street girls in Bogota, Colombia

This dissertation documents the everyday lives and spaces of a population of youth typically constructed as out of place, and the broader urban context in which they are rendered as such. Thirty-three female and transgender street youth participated in the development of this youth-based participatory action research (YPAR) project utilizing geo-ethnographic methods, auto-photography, and archival research throughout a six-phase, eighteen-month research process in Bogotá, Colombia. ^ This dissertation details the participatory writing process that enabled the YPAR research team to destabilize dominant representations of both street girls and urban space and the participatory mapping process that enabled the development of a youth vision of the city through cartographic images. The maps display individual and aggregate spatial data indicating trends within and making comparisons between three subgroups of the research population according to nine spatial variables. These spatial data, coupled with photographic and ethnographic data, substantiate that street girls’ mobilities and activity spaces intersect with and are altered by state-sponsored urban renewal projects and paramilitary-led social cleansing killings, both efforts to clean up Bogotá by purging the city center of deviant populations and places. ^ Advancing an ethical approach to conducting research with excluded populations, this dissertation argues for the enactment of critical field praxis and care ethics within a YPAR framework to incorporate young people as principal research actors rather than merely voices represented in adultist academic discourse. Interjection of considerations of space, gender, and participation into the study of street youth produce new ways of envisioning the city and the role of young people in research. Instead of seeing the city from a panoptic view, Bogotá is revealed through the eyes of street youth who participated in the construction and feminist visualization of a new cartography and counter-map of the city grounded in embodied, situated praxis. This dissertation presents a socially responsible approach to conducting action-research with high-risk youth by documenting how street girls reclaim their right to the city on paper and in practice; through maps of their everyday exclusion in Bogotá followed by activism to fight against it.^

Scaling geospatial searches in large spatial databases

Modern geographical databases, which are at the core of geographic information systems (GIS), store a rich set of aspatial attributes in addition to geographic data. Typically, aspatial information comes in textual and numeric format. Retrieving information constrained on spatial and aspatial data from geodatabases provides GIS users the ability to perform more interesting spatial analyses, and for applications to support composite location-aware searches; for example, in a real estate database: “Find the nearest homes for sale to my current location that have backyard and whose prices are between $50,000 and $80,000”. Efficient processing of such queries require combined indexing strategies of multiple types of data. Existing spatial query engines commonly apply a two-filter approach (spatial filter followed by nonspatial filter, or viceversa), which can incur large performance overheads. On the other hand, more recently, the amount of geolocation data has grown rapidly in databases due in part to advances in geolocation technologies (e.g., GPS-enabled smartphones) that allow users to associate location data to objects or events. The latter poses potential data ingestion challenges of large data volumes for practical GIS databases. In this dissertation, we first show how indexing spatial data with R-trees (a typical data pre-processing task) can be scaled in MapReduce—a widely-adopted parallel programming model for data intensive problems. The evaluation of our algorithms in a Hadoop cluster showed close to linear scalability in building R-tree indexes. Subsequently, we develop efficient algorithms for processing spatial queries with aspatial conditions. Novel techniques for simultaneously indexing spatial with textual and numeric data are developed to that end. Experimental evaluations with real-world, large spatial datasets measured query response times within the sub-second range for most cases, and up to a few seconds for a small number of cases, which is reasonable for interactive applications. Overall, the previous results show that the MapReduce parallel model is suitable for indexing tasks in spatial databases, and the adequate combination of spatial and aspatial attribute indexes can attain acceptable response times for interactive spatial queries with constraints on aspatial data.

DESIGN OF EFFICIENT IN-NETWORK DATA PROCESSING AND DISSEMINATION FOR VANETS

By providing vehicle-to-vehicle and vehicle-to-infrastructure wireless communications, vehicular ad hoc networks (VANETs), also known as the “networks on wheels”, can greatly enhance traffic safety, traffic efficiency and driving experience for intelligent transportation system (ITS). However, the unique features of VANETs, such as high mobility and uneven distribution of vehicular nodes, impose critical challenges of high efficiency and reliability for the implementation of VANETs. This dissertation is motivated by the great application potentials of VANETs in the design of efficient in-network data processing and dissemination. Considering the significance of message aggregation, data dissemination and data collection, this dissertation research targets at enhancing the traffic safety and traffic efficiency, as well as developing novel commercial applications, based on VANETs, following four aspects: 1) accurate and efficient message aggregation to detect on-road safety relevant events, 2) reliable data dissemination to reliably notify remote vehicles, 3) efficient and reliable spatial data collection from vehicular sensors, and 4) novel promising applications to exploit the commercial potentials of VANETs. Specifically, to enable cooperative detection of safety relevant events on the roads, the structure-less message aggregation (SLMA) scheme is proposed to improve communication efficiency and message accuracy. The scheme of relative position based message dissemination (RPB-MD) is proposed to reliably and efficiently disseminate messages to all intended vehicles in the zone-of-relevance in varying traffic density. Due to numerous vehicular sensor data available based on VANETs, the scheme of compressive sampling based data collection (CS-DC) is proposed to efficiently collect the spatial relevance data in a large scale, especially in the dense traffic. In addition, with novel and efficient solutions proposed for the application specific issues of data dissemination and data collection, several appealing value-added applications for VANETs are developed to exploit the commercial potentials of VANETs, namely general purpose automatic survey (GPAS), VANET-based ambient ad dissemination (VAAD) and VANET based vehicle performance monitoring and analysis (VehicleView). Thus, by improving the efficiency and reliability in in-network data processing and dissemination, including message aggregation, data dissemination and data collection, together with the development of novel promising applications, this dissertation will help push VANETs further to the stage of massive deployment.

Geospatial Data Indexing Analysis and Visualization via Web Services with Autonomic Resource Management

With the exponential growth of the usage of web-based map services, the web GIS application has become more and more popular. Spatial data index, search, analysis, visualization and the resource management of such services are becoming increasingly important to deliver user-desired Quality of Service. First, spatial indexing is typically time-consuming and is not available to end-users. To address this, we introduce TerraFly sksOpen, an open-sourced an Online Indexing and Querying System for Big Geospatial Data. Integrated with the TerraFly Geospatial database [1-9], sksOpen is an efficient indexing and query engine for processing Top-k Spatial Boolean Queries. Further, we provide ergonomic visualization of query results on interactive maps to facilitate the user’s data analysis. Second, due to the highly complex and dynamic nature of GIS systems, it is quite challenging for the end users to quickly understand and analyze the spatial data, and to efficiently share their own data and analysis results with others. Built on the TerraFly Geo spatial database, TerraFly GeoCloud is an extra layer running upon the TerraFly map and can efficiently support many different visualization functions and spatial data analysis models. Furthermore, users can create unique URLs to visualize and share the analysis results. TerraFly GeoCloud also enables the MapQL technology to customize map visualization using SQL-like statements [10]. Third, map systems often serve dynamic web workloads and involve multiple CPU and I/O intensive tiers, which make it challenging to meet the response time targets of map requests while using the resources efficiently. Virtualization facilitates the deployment of web map services and improves their resource utilization through encapsulation and consolidation. Autonomic resource management allows resources to be automatically provisioned to a map service and its internal tiers on demand. v-TerraFly are techniques to predict the demand of map workloads online and optimize resource allocations, considering both response time and data freshness as the QoS target. The proposed v-TerraFly system is prototyped on TerraFly, a production web map service, and evaluated using real TerraFly workloads. The results show that v-TerraFly can accurately predict the workload demands: 18.91% more accurate; and efficiently allocate resources to meet the QoS target: improves the QoS by 26.19% and saves resource usages by 20.83% compared to traditional peak load-based resource allocation.

Public-oriented interactive environmental decision support system

With the application of GIS methodologies to spatial data, researchers can now identify patterns of occurrence for many social problems including health-issues and crime. Further more, since this type of data also contains clues as to the underlying causes of social problems, it can be used to make well-educated and consequently, more effective policy decisions.

Web-based public participatory GIS

Decision support systems (DSS) have evolved rapidly during the last decade from stand alone or limited networked solutions to online participatory solutions. One of the major enablers of this change is the fastest growing areas of geographical information system (GIS) technology development that relates to the use of the Internet as a means to access, display, and analyze geospatial data remotely. World-wide many federal, state, and particularly local governments are designing to facilitate data sharing using interactive Internet map servers. This new generation DSS or planning support systems (PSS), interactive Internet map server, is the solution for delivering dynamic maps and GIS data and services via the world-wide Web, and providing public participatory GIS (PPGIS) opportunities to a wider community (Carver, 2001; Jankowski & Nyerges, 2001). It provides a highly scalable framework for GIS Web publishing, Web-based public participatory GIS (WPPGIS), which meets the needs of corporate intranets and demands of worldwide Internet access (Craig, 2002). The establishment of WPPGIS provides spatial data access through a support centre or a GIS portal to facilitate efficient access to and sharing of related geospatial data (Yigitcanlar, Baum, & Stimson, 2003). As more and more public and private entities adopt WPPGIS technology, the importance and complexity of facilitating geospatial data sharing is growing rapidly (Carver, 2003). Therefore, this article focuses on the online public participation dimension of the GIS technology. The article provides an overview of recent literature on GIS and WPPGIS, and includes a discussion on the potential use of these technologies in providing a democratic platform for the public in decision-making.

A decision support system for sustainable urban development : the integrated land use and transportation indexing model

The broad definition of sustainable development at the early stage of its introduction has caused confusion and hesitation among local authorities and planning professionals. The main difficulties are experience in employing loosely-defined principles of sustainable development in setting policies and goals. The question of how this theory/rhetoric-practice gap could be filled will be the theme of this study. One of the widely employed sustainability accounting approaches by governmental organisations, triple bottom line, and applicability of this approach to sustainable urban development policies will be examined. When incorporating triple bottom line considerations with the environmental impact assessment techniques, the framework of GIS-based decision support system that helps decision-makers in selecting policy option according to the economic, environmental and social impacts will be introduced. In order to embrace sustainable urban development policy considerations, the relationship between urban form, travel pattern and socio-economic attributes should be clarified. This clarification associated with other input decision support systems will picture the holistic state of the urban settings in terms of sustainability. In this study, grid-based indexing methodology will be employed to visualise the degree of compatibility of selected scenarios with the designated sustainable urban future. In addition, this tool will provide valuable knowledge about the spatial dimension of the sustainable development. It will also give fine details about the possible impacts of urban development proposals by employing disaggregated spatial data analysis (e.g. land-use, transportation, urban services, population density, pollution, etc.). The visualisation capacity of this tool will help decision makers and other stakeholders compare and select alternative of future urban developments.

An integrated information management model for ecologically sustainable development

Ecologically sustainable development has become a major feature of legal systems at the international, national and local levels throughout the world. In Australia, governments have responded to environmental crises by enacting legislation imposing obligations and restrictions over privately-owned land. Whilst these obligations and restrictions may well be necessary to achieve sustainability, the approach to management of information concerning these instruments is problematic. For example, management of information concerning obligations and restrictions in Queensland is fragmented, with some instruments registered or recorded on the land title register, some on external registers, and some information only available in the legislation itself. This approach is used in most Australian jurisdictions. This fragmented approach has led to two separate but interconnected problems. First, the Torrens system is no longer meeting its goal of providing a complete and accurate picture of title. Second, this uncoordinated approach to the management of land titles, and obligations and restrictions on land use, has created a barrier to sustainable management of natural resources. This is because compliance with environmental laws is impaired in the absence of easily accessible and accurate information. These problems demonstrate a clear need for reform in this area. To determine how information concerning these obligations and restrictions may be most effectively managed, this thesis will apply a comparative methodology and consider three case studies, which each utilise different models for management of this information. These jurisdictions will be assessed according to a set of guidelines for comparison to identify which features of their systems provide for effective management of information concerning obligations and restrictions on title and use. Based on this comparison, this thesis will devise a series of recommendations for an effective system for the management of information concerning obligations and restrictions on land title and use, taking into account any potential legal issues and barriers to implementation. This series of recommendations for reform will be supplemented by suggested draft legislative provisions.

Spatially stratified sampling using auxiliary information for geostatistical mapping

This paper presents a method of spatial sampling based on stratification by Local Moran’s I i calculated using auxiliary information. The sampling technique is compared to other design-based approaches including simple random sampling, systematic sampling on a regular grid, conditional Latin Hypercube sampling and stratified sampling based on auxiliary information, and is illustrated using two different spatial data sets. Each of the samples for the two data sets is interpolated using regression kriging to form a geostatistical map for their respective areas. The proposed technique is shown to be competitive in reproducing specific areas of interest with high accuracy.

Development of the rural statistical sustainability framework tool

It is important to promote a sustainable development approach to ensure that economic, environmental and social developments are maintained in balance. Sustainable development and its implications are not just a global concern, it also affects Australia. In particular, rural Australian communities are facing various economic, environmental and social challenges. Thus, the need for sustainable development in rural regions is becoming increasingly important. To promote sustainable development, proper frameworks along with the associated tools optimised for the specific regions, need to be developed. This will ensure that the decisions made for sustainable development are evidence based, instead of subjective opinions. To address these issues, Queensland University of Technology (QUT), through an Australian Research Council (ARC) linkage grant, has initiated research into the development of a Rural Statistical Sustainability Framework (RSSF) to aid sustainable decision making in rural Queensland. This particular branch of the research developed a decision support tool that will become the integrating component of the RSSF. This tool is developed on the web-based platform to allow easy dissemination, quick maintenance and to minimise compatibility issues. The tool is developed based on MapGuide Open Source and it follows the three-tier architecture: Client tier, Web tier and the Server tier. The developed tool is interactive and behaves similar to a familiar desktop-based application. It has the capability to handle and display vector-based spatial data and can give further visual outputs using charts and tables. The data used in this tool is obtained from the QUT research team. Overall the tool implements four tasks to help in the decision-making process. These are the Locality Classification, Trend Display, Impact Assessment and Data Entry and Update. The developed tool utilises open source and freely available software and accounts for easy extensibility and long-term sustainability.

Variational Bayes and the reduced dependence approximation for the autologistic model on an irregular grid with applications

Discrete Markov random field models provide a natural framework for representing images or spatial datasets. They model the spatial association present while providing a convenient Markovian dependency structure and strong edge-preservation properties. However, parameter estimation for discrete Markov random field models is difficult due to the complex form of the associated normalizing constant for the likelihood function. For large lattices, the reduced dependence approximation to the normalizing constant is based on the concept of performing computationally efficient and feasible forward recursions on smaller sublattices which are then suitably combined to estimate the constant for the whole lattice. We present an efficient computational extension of the forward recursion approach for the autologistic model to lattices that have an irregularly shaped boundary and which may contain regions with no data; these lattices are typical in applications. Consequently, we also extend the reduced dependence approximation to these scenarios enabling us to implement a practical and efficient non-simulation based approach for spatial data analysis within the variational Bayesian framework. The methodology is illustrated through application to simulated data and example images. The supplemental materials include our C++ source code for computing the approximate normalizing constant and simulation studies.

Measuring neighbourhood sustainability : a comparative analysis of residential types in Malaysia

The expansion of city-regions, the increase in the standard of living and changing lifestyles have collectively led to an increase in housing demand. New residential areas are encroaching onto the city fringes including suburban and green field areas. Large and small developers are actively building houses ranging from a few blocks to master-planned style projects. These residential developments, particularly in major urban areas, represent a large portion of urban land use in Malaysia, and, thus, have become a major contributor to overall urban sustainability. There are three main types that comprise the mainstream, and form integral parts to contemporary urban residential developments, namely, subdivision developments, piecemeal developments, and master-planned developments. Many new master-planned developments market themselves as environmentally friendly, and provide layouts that encompass sustainable design and development. To date, however, there have been limited studies conducted to examine such claims or to ascertain which of these three residential development layouts is more sustainable. To fill this gap, this research was undertaken to develop a framework for assessing the level of sustainability of residential developments, focusing on their layouts at the neighbourhood level. The development of this framework adopted a mixed method research strategy and embedded research design to achieve the study aim and objectives. Data were collected from two main sources, where quantitative data were gathered from a three-round Delphi survey and spatial data from a layout plan. Sample respondents for surveys were selected from among experts in the field of the built environment, both from Malaysia and internationally. As for spatial data, three case studies – master-planned, piecemeal and subdivision developments representing different types of neighbourhood developments in Malaysia have been selected. Prior to application on the case studies, the appropriate framework was subjected to validation to ascertain its robustness for application in Malaysia. Following the application of the framework on the three case studies the results revealed that master-planned development scored a better level of sustainability compared to piecemeal and subdivision developments. The results generated from this framework are expected to provide evidence to the policy makers and development agencies as well as provide an awareness of the level of sustainability and the necessary collective efforts required for developing sustainable neighbourhoods. Continuous assessment can facilitate a comparison of sustainability over time for neighbourhoods as a means to monitor changes in the level of sustainability. In addition, the framework is able to identify any particular indicator (issue) that causes a significant impact on sustainability.

‘The health-location connection’: using GIS and social media on smartphones to track engagement with the Local urban environment for work, recreation, and physical activity

A technologically innovative study was undertaken across two suburbs in Brisbane, Australia, to assess socioeconomic differences in women's use of the local environment for work, recreation, and physical activity. Mothers from high and low socioeconomic suburbs were instructed to continue with usual daily routines, and to use mobile phone applications (Facebook Places, Twitter, and Foursquare) on their mobile phones to ‘check-in’ at each location and destination they reached during a one-week period. These smartphone applications are able to track travel logistics via built-in geographical information systems (GIS), which record participants’ points of latitude and longitude at each destination they reach. Location data were downloaded to Google Earth and excel for analysis. Women provided additional qualitative data via text regarding the reasons and social contexts of their travel. We analysed 2183 ‘check-ins’ for 54 women in this pilot study to gain quantitative, qualitative, and spatial data on human-environment interactions. Data was gathered on distances travelled, mode of transport, reason for travel, social context of travel, and categorised in terms of physical activity type – walking, running, sports, gym, cycling, or playing in the park. We found that the women in both suburbs had similar daily routines with the exception of physical activity. We identified 15% of ‘check-ins’ in the lower socioeconomic group as qualifying for the physical activity category, compared with 23% in the higher socioeconomic group. This was explained by more daily walking for transport (1.7kms to 0.2kms) and less car travel each week (28.km to 48.4kms) in the higher socioeconomic suburb. We ascertained insights regarding the socio-cultural influences on these differences via additional qualitative data. We discuss the benefits and limitations of using new technologies and Google Earth with implications for informing future physical and social aspects of urban design, and health promotion in socioeconomically diverse cities.

Environments for Healthy Living (EFHL) Griffith Birth Cohort Study : background and methods

The health of an individual is determined by the interaction of genetic and individual factors with wider social and environmental elements. Public health approaches to improving the health of disadvantaged populations will be most effective if they optimise influences at each of these levels, particularly in the early part of the life course. In order to better ascertain the relative contribution of these multi-level determinants there is a need for robust studies, longitudinal and prospective in nature, that examine individual, familial, social and environmental exposures. This paper describes the study background and methods, as it has been implemented in an Australian birth cohort study, Environments for Healthy Living (EFHL): The Griffith Study of Population Health. EFHL is a prospective, multi-level, multi-year longitudinal birth cohort study, designed to collect information from before birth through to adulthood across a spectrum of eco-epidemiological factors, including genetic material from cord-blood samples at birth, individual and familial factors, to spatial data on the living environment. EFHL commenced the pilot phase of recruitment in 2006 and open recruitment in 2007, with a target sample size of 4000 mother/infant dyads. Detailed information on each participant is obtained at birth, 12-months, 3-years, 5-years and subsequent three to five yearly intervals. The findings of this research will provide detailed evidence on the relative contribution of multi-level determinants of health, which can be used to inform social policy and intervention strategies that will facilitate healthy behaviours and choices across sub-populations.