A design pattern for efficient retrieval of large data sets from remote data sources

Retrieving large amounts of information over wide area networks, including the Internet, is problematic due to issues arising from latency of response, lack of direct memory access to data serving resources, and fault tolerance. This paper describes a design pattern for solving the issues of handling results from queries that return large amounts of data. Typically these queries would be made by a client process across a wide area network (or Internet), with one or more middle-tiers, to a relational database residing on a remote server. The solution involves implementing a combination of data retrieval strategies, including the use of iterators for traversing data sets and providing an appropriate level of abstraction to the client, double-buffering of data subsets, multi-threaded data retrieval, and query slicing. This design has recently been implemented and incorporated into the framework of a commercial software product developed at Oracle Corporation.

Health services research using linked records: who consents and what is the gain?

Objective: To assess consent to record linkage, describe the characteristics of consenters and compare self-report versus Medicare records of general practitioner use. Method. Almost 40,000 women in the Australian Longitudinal Study on Women's Health were sent a request by mail for permission to link their Medicare records and survey data. Results: 19,700 women consented: 37% of young (18-23 years), 59% of mid-age (4550 years) and 53% of older women (70-75 years). Consenters tended to have higher levels of education and, among the older cohort, were in better health than nonconsenters. Women tended to under-report the number of visits to general practitioners. Conclusions: Record linkage of survey and Medicare data on a large scale is feasible. The linked data provide information on health and socio-economic status which are valuable for understanding health service utilisation. Implications: Linked records provide a powerful tool for health care research, particularly in longitudinal studies.

A multiresolution terrain model for efficient visualization query processing

Multiresolution Triangular Mesh (MTM) models are widely used to improve the performance of large terrain visualization by replacing the original model with a simplified one. MTM models, which consist of both original and simplified data, are commonly stored in spatial database systems due to their size. The relatively slow access speed of disks makes data retrieval the bottleneck of such terrain visualization systems. Existing spatial access methods proposed to address this problem rely on main-memory MTM models, which leads to significant overhead during query processing. In this paper, we approach the problem from a new perspective and propose a novel MTM called direct mesh that is designed specifically for secondary storage. It supports available indexing methods natively and requires no modification to MTM structure. Experiment results, which are based on two real-world data sets, show an average performance improvement of 5-10 times over the existing methods.

Direct mesh: A multiresolution approach to terrain visualization

Terrain can be approximated by a triangular mesh consisting millions of 3D points. Multiresolution triangular mesh (MTM) structures are designed to support applications that use terrain data at variable levels of detail (LOD). Typically, an MTM adopts a tree structure where a parent node represents a lower-resolution approximation of its descendants. Given a region of interest (ROI) and a LOD, the process of retrieving the required terrain data from the database is to traverse the MTM tree from the root to reach all the nodes satisfying the ROI and LOD conditions. This process, while being commonly used for multiresolution terrain visualization, is inefficient as either a large number of sequential I/O operations or fetching a large amount of extraneous data is incurred. Various spatial indexes have been proposed in the past to address this problem, however level-by-level tree traversal remains a common practice in order to obtain topological information among the retrieved terrain data. A new MTM data structure called direct mesh is proposed. We demonstrate that with direct mesh the amount of data retrieval can be substantially reduced. Comparing with existing MTM indexing methods, a significant performance improvement has been observed for real-life terrain data.

Ex ante evaluations of alternate data structures for end user queries: Theory and experimental test

The data structure of an information system can significantly impact the ability of end users to efficiently and effectively retrieve the information they need. This research develops a methodology for evaluating, ex ante, the relative desirability of alternative data structures for end user queries. This research theorizes that the data structure that yields the lowest weighted average complexity for a representative sample of information requests is the most desirable data structure for end user queries. The theory was tested in an experiment that compared queries from two different relational database schemas. As theorized, end users querying the data structure associated with the less complex queries performed better Complexity was measured using three different Halstead metrics. Each of the three metrics provided excellent predictions of end user performance. This research supplies strong evidence that organizations can use complexity metrics to evaluate, ex ante, the desirability of alternate data structures. Organizations can use these evaluations to enhance the efficient and effective retrieval of information by creating data structures that minimize end user query complexity.

Assessing longwall support-roof interaction from shield leg pressure data

A concept has been developed where characteristic load cycles of longwall shields can describe most of the interaction between a longwall support and the roof. A characteristic load cycle is the change in support pressure with time from setting the support against the roof to the next release and movement of the support. The concept has been validated through the back-analysis of more than 500 000 individual load cycles in five longwall panels at four mines and seven geotechnical domains. The validation process depended upon the development of new software capable of both handling the large quantity of data emanating from a modern longwall and accurately delineating load cycles. Existing software was found not to be capable of delineating load cycles to a sufficient accuracy. Load-cycle analysis can now be used quantitatively to assess the adequacy of support capacity and the appropriateness of set pressure for the conditions under which a longwall is being operated. When linked to a description of geotechnical conditions, this has allowed the development of a database for support selection for greenfield sites. For existing sites, the load-cycle characteristic concept allows for a diagnosis of strata-support problem areas, enabling changes to be made to set pressure and mining strategies to manage better, or avoid, strata control problems. With further development of the software, there is the prospect of developing a system that is able to respond to changes in strata-support interaction in real time.

Migrating eprints.org data to a Fez repository

This document records the process of migrating eprints.org data to a Fez repository. Fez is a Web-based digital repository and workflow management system based on Fedora (http://www.fedora.info/). At the time of migration, the University of Queensland Library was using EPrints 2.2.1 [pepper] for its ePrintsUQ repository. Once we began to develop Fez, we did not upgrade to later versions of eprints.org software since we knew we would be migrating data from ePrintsUQ to the Fez-based UQ eSpace. Since this document records our experiences of migration from an earlier version of eprints.org, anyone seeking to migrate eprints.org data into a Fez repository might encounter some small differences. Moving UQ publication data from an eprints.org repository into a Fez repository (hereafter called UQ eSpace (http://espace.uq.edu.au/) was part of a plan to integrate metadata (and, in some cases, full texts) about all UQ research outputs, including theses, images, multimedia and datasets, in a single repository. This tied in with the plan to identify and capture the research output of a single institution, the main task of the eScholarshipUQ testbed for the Australian Partnership for Sustainable Repositories project (http://www.apsr.edu.au/). The migration could not occur at UQ until the functionality in Fez was at least equal to that of the existing ePrintsUQ repository. Accordingly, as Fez development occurred throughout 2006, a list of eprints.org functionality not currently supported in Fez was created so that programming of such development could be planned for and implemented.

Using household survey data to inform policy decisions regarding the delivery of evidence-based parenting interventions.

Background: This study used household survey data on the prevalence of child, parent and family variables to establish potential targets for a population-level intervention to strengthen parenting skills in the community. The goals of the intervention include decreasing child conduct problems, increasing parental self-efficacy, use of positive parenting strategies, decreasing coercive parenting and increasing help-seeking, social support and participation in positive parenting programmes. Methods: A total of 4010 parents with a child under the age of 12 years completed a statewide telephone survey on parenting. Results: One in three parents reported that their child had a behavioural or emotional problem in the previous 6 months. Furthermore, 9% of children aged 2–12 years meet criteria for oppositional defiant disorder. Parents who reported their child's behaviour to be difficult were more likely to perceive parenting as a negative experience (i.e. demanding, stressful and depressing). Parents with greatest difficulties were mothers without partners and who had low levels of confidence in their parenting roles. About 20% of parents reported being stressed and 5% reported being depressed in the 2 weeks prior to the survey. Parents with personal adjustment problems had lower levels of parenting confidence and their child was more difficult to manage. Only one in four parents had participated in a parent education programme. Conclusions: Implications for the setting of population-level goals and targets for strengthening parenting skills are discussed.

Knowledge discovery and data mining in biological databases

The new technologies for Knowledge Discovery from Databases (KDD) and data mining promise to bring new insights into a voluminous growing amount of biological data. KDD technology is complementary to laboratory experimentation and helps speed up biological research. This article contains an introduction to KDD, a review of data mining tools, and their biological applications. We discuss the domain concepts related to biological data and databases, as well as current KDD and data mining developments in biology.

Gender and age-at-first-admission for schizophrenia: Data from Australia and Brazil

In order to examine whether different populations show the same pattern of onset in the Southern Hemisphere, we examined the age-at-first-admission distribution for schizophrenia based on mental health registers from Australia and Brazil. Data on age-at-first-admission for individuals with schizophrenia were extracted from two names-linked registers, (1) the Queensland Mental Health Statistics System, Australia (N=7651, F= 3293, M=4358), and (2) a psychiatric hospital register in Pelotas, Brazil (N=4428, F=2220, M=2208). Age distributions were derived for males and females for both datasets. The general population structure tbr both countries was also obtained. There were significantly more males in the Queensland dataset (gz = 56.9, df3, p < 0.0001 ). Both dataset distributions were skewed to the right. Onset rose steeply after puberty to reach a modal age group of 20-29 for men and women, with a more gradual tail toward the older age groups. In Queensland 68% of women with schizophrenia had their first admissions after age 30, while the proportion from Brazil was 58%. Compared to the Australian dataset, the Brazilian dataset had a slightly greater proportion of first admissions under the age 30 and a slightly smaller proportion over the age of 60 years. This reflects the underlying age distributions of the two populations. This study confirms the wide age range and gender differences in age-at-first-admission distributions for schizophrenia and identified a significant difference in the gender ratio between the two datasets. Given widely differing health services, cultural practices, ethic variability, and the different underlying population distributions, the age-at-first-admission in Queensland and Brazil showed more similarities than differences. Acknowledgments: The Stanley Foundation supported this project.

Is there change over time in the season-of-birth effect for schizophrenia? Data from the southern hemisphere

The aim of this paper is to examine distributions of schizophrenia and general population births over time in order to determine whether (a) the pattern has changed over time, (b) any pattern was similar for both males and females, and (c) whether there is any indication that there is any relationship between the changes in pattern between schizophrenia and general population births. Birth month and year for 7807 individuals with ICD8/9 schizophrenia were gained from the Queensland Mental Health Statistical System for 1914-1975. Monthly births for the general population in Queensland for the same period were obtained from the Australian Bureau of Statistics. For each decade we obtained two comparisons, (1) between two 'seasons' (summer-autumn/winter-spring), and (2) between the third (coldest) quarter and the remaining quarters. Based on expected contrasts from general population proportions, odds ratios and their confidence intervals were used to analyse these comparisons for all subjects, and for males and females separately. The seasonality found in our previous studies was again evident (OR 1.09; 95% CI= 1.01-1.17). However there was no significant change in its pattern over time either for the total group or for males and females separately. When the general population births alone were examined using the same contrasts, seasonality was also observed, but here there were fluctuations over time. These results suggest that exposures linked to changes in general population births over time should be examined in disorders such as schizophrenia which demonstrate seasonality in births. The Stanley Foundation supported this project.

Baseline laboratory bioassay data for spinosad against populations of Australian sheep blowfly, Lucilia cuprina (Wiedemann) (Diptera : Calliphoridae)

Spinosad was an effective larvicide against the Australian sheep blowfly, Lucilia cuprina. A survey of 41 field populations indicated no cross-resistance to spinosad from existing organophosphate resistance. The data presented serve as baseline data for future resistance surveys.

The effects of normalisation on the ability of business end-users to detect data anomalies: An experimental evaluation

With the proliferation of relational database programs for PC's and other platforms, many business end-users are creating, maintaining, and querying their own databases. More importantly, business end-users use the output of these queries as the basis for operational, tactical, and strategic decisions. Inaccurate data reduce the expected quality of these decisions. Implementing various input validation controls, including higher levels of normalisation, can reduce the number of data anomalies entering the databases. Even in well-maintained databases, however, data anomalies will still accumulate. To improve the quality of data, databases can be queried periodically to locate and correct anomalies. This paper reports the results of two experiments that investigated the effects of different data structures on business end-users' abilities to detect data anomalies in a relational database. The results demonstrate that both unnormalised and higher levels of normalisation lower the effectiveness and efficiency of queries relative to the first normal form. First normal form databases appear to provide the most effective and efficient data structure for business end-users formulating queries to detect data anomalies.