Statistical Analysis of Text Summarization Evaluation

This dissertation applies statistical methods to the evaluation of automatic summarization using data from the Text Analysis Conferences in 2008-2011. Several aspects of the evaluation framework itself are studied, including the statistical testing used to determine significant differences, the assessors, and the design of the experiment. In addition, a family of evaluation metrics is developed to predict the score an automatically generated summary would receive from a human judge and its results are demonstrated at the Text Analysis Conference. Finally, variations on the evaluation framework are studied and their relative merits considered. An over-arching theme of this dissertation is the application of standard statistical methods to data that does not conform to the usual testing assumptions.

Application of a policy framework for the public funding of drugs for rare diseases

BACKGROUND: In many countries, decisions about the public funding of drugs are preferentially based on the results of randomized trials. For truly rare diseases, such trials are not typically available, and approaches by public payers are highly variable. In view of this, a policy framework intended to fairly evaluate these drugs was developed by the Drugs for Rare Diseases Working Group (DRDWG) at the request of the Ontario Public Drug Programs. OBJECTIVE: To report the initial experience of applying a novel evaluation framework to funding applications for drugs for rare diseases. METHODS: Retrospective observational cohort study. MEASURES: Clinical effectiveness, costs, funding recommendations, funding approval. KEY RESULTS: Between March 2008 and February 2013, eight drugs were evaluated using the DRDWG framework. The estimated average annual drug cost per patient ranged from 28,000 to 1,200,000 Canadian dollars (CAD). For five drugs, full evaluations were completed, specific funding recommendations were made by the DRDWG, and funding was approved after risk-sharing agreements with the manufacturers were negotiated. For two drugs, the disease indications were determined to be ineligible for consideration. For one drug, there was insufficient natural history data for the disease to provide a basis for recommendation. For the five drugs fully evaluated, 32 patients met the predefined eligibility criteria for funding, and five were denied based on predefined exclusion criteria. CONCLUSIONS: The framework improved transparency and consistency for evaluation and public funding of drugs for rare diseases in Ontario. The evaluation process will continue to be iteratively refined as feedback on actual versus expected clinical and economic outcomes is incorporated. © 2014 Society of General Internal Medicine.

Realist evaluation of a remote hospital programme in the context of Australian health care reforms

Background: Since 2008, Australia has embarked on major healthcare reforms extending across all states and territories. Only limited evidence linking national healthcare reforms to improvement in public hospital performance exists. This evidence gap is even more pronounced in the case of remote hospital performance. This article describes a protocol retrospectively assessing a remote hospital programme to implement emergency department performance indicators, in the context of national reforms, over a period of 7 years (2008–2014). Challenges to implementing these reforms are explored.Method: Assessing the complex scenario of reform implementation requires an in-depth analysis, offered by a Realist Evaluation framework. Within this framework, a case study design is adopted to enable descriptive analysis. Interviews with key hospital stakeholders were followed by a literature review to identify a programme theory. The programme theory was articulated in the form of a preliminary context-mechanism-outcome configuration (CMOC). This theory will underlie further data collection, analysis, and interpretation. Both Realist Evaluation and case study allow flexibility in a choice of methods; both quantitative and qualitative methods will be incorporated. The thematic analysis will be employed to identify causal relationships and linkages in collected data.Discussion: Assembled data will be used to develop final CMOC patterns. The final CMOC will help in understanding the theory and mechanisms in use in the hospital.

Developing a participatory impact assessment approach and action research culture within a communication for social change organisation in Nepal

Developing an effective impact evaluation framework, managing and conducting rigorous impact evaluations, and developing a strong research and evaluation culture within development communication organisations presents many challenges. This is especially so when both the community and organisational context is continually changing and the outcomes of programs are complex and difficult to clearly identify.----- This paper presents a case study from a research project being conducted from 2007-2010 that aims to address these challenges and issues, entitled Assessing Communication for Social Change: A New Agenda in Impact Assessment. Building on previous development communication projects which used ethnographic action research, this project is developing, trailing and rigorously evaluating a participatory impact assessment methodology for assessing the social change impacts of community radio programs in Nepal. This project is a collaboration between Equal Access – Nepal (EAN), Equal Access – International, local stakeholders and listeners, a network of trained community researchers, and a research team from two Australian universities. A key element of the project is the establishment of an organisational culture within EAN that values and supports the impact assessment process being developed, which is based on continuous action learning and improvement. The paper describes the situation related to monitoring and evaluation (M&E) and impact assessment before the project began, in which EAN was often reliant on time-bound studies and ‘success stories’ derived from listener letters and feedback. We then outline the various strategies used in an effort to develop stronger and more effective impact assessment and M&E systems, and the gradual changes that have occurred to date. These changes include a greater understanding of the value of adopting a participatory, holistic, evidence-based approach to impact assessment. We also critically review the many challenges experienced in this process, including:----- • Tension between the pressure from donors to ‘prove’ impacts and the adoption of a bottom-up, participatory approach based on ‘improving’ programs in ways that meet community needs and aspirations.----- • Resistance from the content teams to changing their existing M&E practices and to the perceived complexity of the approach.----- • Lack of meaningful connection between the M&E and content teams.----- • Human resource problems and lack of capacity in analysing qualitative data and reporting results.----- • The contextual challenges, including extreme poverty, wide cultural and linguistic diversity, poor transport and communications infrastructure, and political instability.----- • A general lack of acceptance of the importance of evaluation within Nepal due to accepting everything as fate or ‘natural’ rather than requiring investigation into a problem.

An example of developing a business model for information and communication technologies (ICT) adoption on construction projects – the National Museum of Australia project

Effective use of information and communication technologies (ICT) is necessary for delivering efficiency and improved project delivery in the construction industry. Convincing clients or contracting organisations to embrace ICT is a difficult task, there are few templates of an ICT business model for the industry to use. ICT application in the construction industry is relatively low compared to automotive and aerospace industries. The National Museum of Australia project provides a unique opportunity for investigating and reporting on this deficiency in publicly available knowledge. Concentrates on the business model content and objectives, briefly indicates the evaluation framework that was used to evaluate ICT effectiveness.

Overview of the NTCIR-9 crosslink task : cross-lingual link discovery

This paper presents an overview of NTCIR-9 Cross-lingual Link Discovery (Crosslink) task. The overview includes: the motivation of cross-lingual link discovery; the Crosslink task definition; the run submission specification; the assessment and evaluation framework; the evaluation metrics; and the evaluation results of submitted runs. Cross-lingual link discovery (CLLD) is a way of automatically finding potential links between documents in different languages. The goal of this task is to create a reusable resource for evaluating automated CLLD approaches. The results of this research can be used in building and refining systems for automated link discovery. The task is focused on linking between English source documents and Chinese, Korean, and Japanese target documents.

The effectiveness of cross-lingual link discovery

This paper describes the evaluation in benchmarking the effectiveness of cross-lingual link discovery (CLLD). Cross lingual link discovery is a way of automatically finding prospective links between documents in different languages, which is particularly helpful for knowledge discovery of different language domains. A CLLD evaluation framework is proposed for system performance benchmarking. The framework includes standard document collections, evaluation metrics, and link assessment and evaluation tools. The evaluation methods described in this paper have been utilised to quantify the system performance at NTCIR-9 Crosslink task. It is shown that using the manual assessment for generating gold standard can deliver a more reliable evaluation result.

Restrung new chamber festival : an exploration of contemporary string practice

The Restrung New Chamber Festival was a practice-led research project which explored the intricacies of musical relationships. Specifically, it investigated the nature of contemporary string practice, with "New music" at its core. For the purposes of this project, "New music" will be defined in terms of representing a "global sonorous space" (Nancy 2007:12), which Hulse describes as "a spectacular comingling of styles and an unprecedented explosion of creative possibilities" (Hulse n.d.:2). Approaches to staging such an event are contextualised through a comparative analysis with relevant Australian and European festivals. The Restrung model derived inspiration from both art music and popular music festival models, in several aspects. One strategy was to engage audiences through combinations of musical, visual and spatial features. Another strategy was to encourage interaction by audiences with installations, workshops and forums. Restrung represents a new and distinctive model which presented art music within an immersive environment. This exegesis presents an evaluation framework which investigates the relationship between curatorial input and the experiential qualities of the festival. The context of an overview of trends in arts festival curation informs the discussion, as well as approaches to identifying new and receptive audiences. It is expected that the evaluation framework will provide a useful and practical guide for curators working in contemporary string practice, hybrid arts, experimental and cross-art form festival design.

Supporting and impeding factors for partnering in construction – a China study

Partnering has drawn attention from both academics and practitioners in the construction industry in the context of construction and facilities management. The past decades have seen a number of articles reporting the application of partnering in construction. The Chinese construction industry is one of the largest industries in the world; however, to the authors’ best knowledge, no project in mainland China has adopted this procurement approach in a formal and systematic manner as yet. This study employed a qualitative approach to investigate the factors that support or impede the implementation of partnering in mainland China. The findings indicate that the partnering practice is feasible in the construction industry of China due to the large demand brought about by China’s strong economic growth and government support. However, the implementation of partnering in the Chinese construction industry is being impeded by the restrictions of the current Chinese regulatory framework and tender evaluation framework, the incompatible features of Chinese culture and the general lack of trust. Six strategies that help to facilitate the implementation of partnering in China have been developed. This study offers a useful reference to implement collaborative contracting models such as partnering in developing countries.

The integrated land use and transportation indexing model : assessing the sustainability of the Gold Coast, Australia

Urban sustainability and sustainable urban development concepts have been identified as the ultimate goal of many contemporary planning endeavours and have become central concepts on which the urban development policies are formulated. In the confinement of these concepts, land use and transport integration has been highlighted as one of the most important policy objectives considering the interrelationship between them and available intervention means of planning. While its interpretation varies, in Australia, it has been embraced as integration of land use and transport planning/policies and been an integral part of regional and local plans. Accordingly, a number of principles have been defined to guide its implementation, to name a few, planning for compact and connected urban development, encouraging active transport modes, creation of mixed-use activity centres and public transport precincts, provision of high quality public transport services, and enhancing character and amenity of urban areas. However, there is lack of an evaluation framework to measure the extent of achievement of implementation of these principles. In pursuit of filling this gap, this study aims to devise an evaluation framework to measure the performance of urban settings according to the integration principles in South East Queensland, Australia context and to demarcate problematic areas which can be intervened by planning tools...

Process based synthesis to evaluate design flexibility in airport terminal layout

Flexible design concept is a relatively new trend in airport terminal design which is believed to facilitate the ever changing needs of a terminal. Current architectural design processes become more complex every day because of the introduction of new building technologies where the concept of flexible airport terminal would apparently make the design process even more complex. Previous studies have demonstrated that ever growing aviation industry requires airport terminals to be planned, designed and constructed in such a way that should allow flexibility in design process. In order to adopt the philosophy of ‘design for flexibility’ architects need to address a wide range of differing needs. An appropriate integration of the process models, prior to the airport terminal design process, is expected to uncover the relationships that exist between spatial layout and their corresponding functions. The current paper seeks to develop a way of sharing space adjacency related information obtained from the Business Process Models (BPM) to assist in defining flexible airport terminal layouts. Critical design parameters are briefly investigated at this stage of research whilst reviewing the available design alternatives and an evaluation framework is proposed in the current paper. Information obtained from various design layouts should assist in identifying and defining flexible design matrices allowing architects to interpret and to apply those throughout the lifecycle of the terminal building.

Iterative development of MobileMums : a physical activity intervention for women with young children.

Background To describe the iterative development process and final version of ‘MobileMums’: a physical activity intervention for women with young children (<5 years) delivered primarily via mobile telephone (mHealth) short messaging service (SMS). Methods MobileMums development followed the five steps outlined in the mHealth development and evaluation framework: 1) conceptualization (critique of literature and theory); 2) formative research (focus groups, n= 48); 3) pre-testing (qualitative pilot of intervention components, n= 12); 4) pilot testing (pilot RCT, n= 88); and, 5) qualitative evaluation of the refined intervention (n= 6). Results Key findings identified throughout the development process that shaped the MobileMums program were the need for: behaviour change techniques to be grounded in Social Cognitive Theory; tailored SMS content; two-way SMS interaction; rapport between SMS sender and recipient; an automated software platform to generate and send SMS; and, flexibility in location of a face-to-face delivered component. Conclusions The final version of MobileMums is flexible and adaptive to individual participant’s physical activity goals, expectations and environment. MobileMums is being evaluated in a community-based randomised controlled efficacy trial (ACTRN12611000481976).

Link discovery for Chinese/English cross-language web information retrieval

Nowadays people heavily rely on the Internet for information and knowledge. Wikipedia is an online multilingual encyclopaedia that contains a very large number of detailed articles covering most written languages. It is often considered to be a treasury of human knowledge. It includes extensive hypertext links between documents of the same language for easy navigation. However, the pages in different languages are rarely cross-linked except for direct equivalent pages on the same subject in different languages. This could pose serious difficulties to users seeking information or knowledge from different lingual sources, or where there is no equivalent page in one language or another. In this thesis, a new information retrieval task—cross-lingual link discovery (CLLD) is proposed to tackle the problem of the lack of cross-lingual anchored links in a knowledge base such as Wikipedia. In contrast to traditional information retrieval tasks, cross language link discovery algorithms actively recommend a set of meaningful anchors in a source document and establish links to documents in an alternative language. In other words, cross-lingual link discovery is a way of automatically finding hypertext links between documents in different languages, which is particularly helpful for knowledge discovery in different language domains. This study is specifically focused on Chinese / English link discovery (C/ELD). Chinese / English link discovery is a special case of cross-lingual link discovery task. It involves tasks including natural language processing (NLP), cross-lingual information retrieval (CLIR) and cross-lingual link discovery. To justify the effectiveness of CLLD, a standard evaluation framework is also proposed. The evaluation framework includes topics, document collections, a gold standard dataset, evaluation metrics, and toolkits for run pooling, link assessment and system evaluation. With the evaluation framework, performance of CLLD approaches and systems can be quantified. This thesis contributes to the research on natural language processing and cross-lingual information retrieval in CLLD: 1) a new simple, but effective Chinese segmentation method, n-gram mutual information, is presented for determining the boundaries of Chinese text; 2) a voting mechanism of name entity translation is demonstrated for achieving a high precision of English / Chinese machine translation; 3) a link mining approach that mines the existing link structure for anchor probabilities achieves encouraging results in suggesting cross-lingual Chinese / English links in Wikipedia. This approach was examined in the experiments for better, automatic generation of cross-lingual links that were carried out as part of the study. The overall major contribution of this thesis is the provision of a standard evaluation framework for cross-lingual link discovery research. It is important in CLLD evaluation to have this framework which helps in benchmarking the performance of various CLLD systems and in identifying good CLLD realisation approaches. The evaluation methods and the evaluation framework described in this thesis have been utilised to quantify the system performance in the NTCIR-9 Crosslink task which is the first information retrieval track of this kind.

Waste and resource management practices, legislation and policy encouraging and influencing ‘regeneration reuse’ of property assets

Carbon taxation governance is becoming increasingly popular, further evolving the polluter pays concept already well established in the built environment as a mechanism to controlling and licensing waste generation. This paper presents an explanation of property asset ‘regeneration reuse’ principles following deconstruction, which reduce waste generation associated with the process of demolition, construction and operation. An analysis is made of strategies in Australia and the United Kingdom, comparing jurisdiction targets pertaining to construction and demolition waste that encourage ‘regeneration reuse’. From examination of applicable Australian and United Kingdom legislation, strategic, fiscal and policy that influence on the 'regeneration reuse' of property assets, an evaluation to the variety of issues relevant to waste and resource management practices is reached. The paper concludes that a systematic evaluation framework to selecting building components and structures suitable for reuse after deconstruction must be considered in legislation.

Overview of the NTCIR-10 cross-lingual link discovery task

This paper presents an overview of NTCIR-10 Cross-lingual Link Discovery (CrossLink-2) task. For the task, we continued using the evaluation framework developed for the NTCIR-9 CrossLink-1 task. Overall, recommended links were evaluated at two levels (file-to-file and anchor-to-file); and system performance was evaluated with metrics: LMAP, R-Prec and P@N.