A comprehensive analysis of parameter settings for novelty-biased cumulative gain

In the TREC Web Diversity track, novelty-biased cumulative gain (α-NDCG) is one of the official measures to assess retrieval performance of IR systems. The measure is characterised by a parameter, α, the effect of which has not been thoroughly investigated. We find that common settings of α, i.e. α=0.5, may prevent the measure from behaving as desired when evaluating result diversification. This is because it excessively penalises systems that cover many intents while it rewards those that redundantly cover only few intents. This issue is crucial since it highly influences systems at top ranks. We revisit our previously proposed threshold, suggesting α be set on a query-basis. The intuitiveness of the measure is then studied by examining actual rankings from TREC 09-10 Web track submissions. By varying α according to our query-based threshold, the discriminative power of α-NDCG is not harmed and in fact, our approach improves α-NDCG's robustness. Experimental results show that the threshold for α can turn the measure to be more intuitive than using its common settings.

A query-basis approach to parametrizing novelty-biased cumulative gain

Novelty-biased cumulative gain (α-NDCG) has become the de facto measure within the information retrieval (IR) community for evaluating retrieval systems in the context of sub-topic retrieval. Setting the incorrect value of parameter α in α-NDCG prevents the measure from behaving as desired in particular circumstances. In fact, when α is set according to common practice (i.e. α = 0.5), the measure favours systems that promote redundant relevant sub-topics rather than provide novel relevant ones. Recognising this characteristic of the measure is important because it affects the comparison and the ranking of retrieval systems. We propose an approach to overcome this problem by defining a safe threshold for the value of α on a query basis. Moreover, we study its impact on system rankings through a comprehensive simulation.

Investigating subspace distances in semantic spaces

Semantic space models of word meaning derived from co-occurrence statistics within a corpus of documents, such as the Hyperspace Analogous to Language (HAL) model, have been proposed in the past. While word similarity can be computed using these models, it is not clear how semantic spaces derived from different sets of documents can be compared. In this paper, we focus on this problem, and we revisit the proposal of using semantic subspace distance measurements [1]. In particular, we outline the research questions that still need to be addressed to investigate and validate these distance measures. Then, we describe our plans for future research.

An analysis of ranking principles and retrieval strategies

The assumptions underlying the Probability Ranking Principle (PRP) have led to a number of alternative approaches that cater or compensate for the PRP’s limitations. All alternatives deviate from the PRP by incorporating dependencies. This results in a re-ranking that promotes or demotes documents depending upon their relationship with the documents that have been already ranked. In this paper, we compare and contrast the behaviour of state-of-the-art ranking strategies and principles. To do so, we tease out analytical relationships between the ranking approaches and we investigate the document kinematics to visualise the effects of the different approaches on document ranking.

When two is better than one : a study of ranking paradigms and their integrations for subtopic retrieval

In this paper, we consider the problem of document ranking in a non-traditional retrieval task, called subtopic retrieval. This task involves promoting relevant documents that cover many subtopics of a query at early ranks, providing thus diversity within the ranking. In the past years, several approaches have been proposed to diversify retrieval results. These approaches can be classified into two main paradigms, depending upon how the ranks of documents are revised for promoting diversity. In the first approach subtopic diversification is achieved implicitly, by choosing documents that are different from each other, while in the second approach this is done explicitly, by estimating the subtopics covered by documents. Within this context, we compare methods belonging to the two paradigms. Furthermore, we investigate possible strategies for integrating the two paradigms with the aim of formulating a new ranking method for subtopic retrieval. We conduct a number of experiments to empirically validate and contrast the state-of-the-art approaches as well as instantiations of our integration approach. The results show that the integration approach outperforms state-of-the-art strategies with respect to a number of measures.

University of Glasgow at ImageCLEFPhoto 2009 : optimising similarity and diversity in image retrieval

In this paper we describe the approaches adopted to generate the runs submitted to ImageCLEFPhoto 2009 with an aim to promote document diversity in the rankings. Four of our runs are text based approaches that employ textual statistics extracted from the captions of images, i.e. MMR [1] as a state of the art method for result diversification, two approaches that combine relevance information and clustering techniques, and an instantiation of Quantum Probability Ranking Principle. The fifth run exploits visual features of the provided images to re-rank the initial results by means of Factor Analysis. The results reveal that our methods based on only text captions consistently improve the performance of the respective baselines, while the approach that combines visual features with textual statistics shows lower levels of improvements.

The Quantum Probability Ranking Principle for Information Retrieval

While the Probability Ranking Principle for Information Retrieval provides the basis for formal models, it makes a very strong assumption regarding the dependence between documents. However, it has been observed that in real situations this assumption does not always hold. In this paper we propose a reformulation of the Probability Ranking Principle based on quantum theory. Quantum probability theory naturally includes interference effects between events. We posit that this interference captures the dependency between the judgement of document relevance. The outcome is a more sophisticated principle, the Quantum Probability Ranking Principle, that provides a more sensitive ranking which caters for interference/dependence between documents’ relevance.

Semantic spaces : measuring the distance between different subspaces

Semantic Space models, which provide a numerical representation of words’ meaning extracted from corpus of documents, have been formalized in terms of Hermitian operators over real valued Hilbert spaces by Bruza et al. [1]. The collapse of a word into a particular meaning has been investigated applying the notion of quantum collapse of superpositional states [2]. While the semantic association between words in a Semantic Space can be computed by means of the Minkowski distance [3] or the cosine of the angle between the vector representation of each pair of words, a new procedure is needed in order to establish relations between two or more Semantic Spaces. We address the question: how can the distance between different Semantic Spaces be computed? By representing each Semantic Space as a subspace of a more general Hilbert space, the relationship between Semantic Spaces can be computed by means of the subspace distance. Such distance needs to take into account the difference in the dimensions between subspaces. The availability of a distance for comparing different Semantic Subspaces would enable to achieve a deeper understanding about the geometry of Semantic Spaces which would possibly translate into better effectiveness in Information Retrieval tasks.

Failure of discontinuous railhead edges due to plastic strain accumulation

Railhead is perhaps the highest stressed civil infrastructure due to the passage of heavily loaded wheels through a very small contact patch. The stresses at the contact patch cause yielding of the railhead material and wear. Many theories exist for the prediction of these mechanisms of continuous rails; this process in the discontinuous rails is relatively sparingly researched. Discontinuous railhead edges fail due to accumulating excessive plastic strains. Significant safety concern is widely reported as these edges form part of Insulated Rail Joints (IRJs) in the signalling track circuitry. Since Hertzian contact is not valid at a discontinuous edge, 3D finite element (3DFE) models of wheel contact at a railhead edge have been used in this research. Elastic–plastic material properties of the head hardened rail steel have been experimentally determined through uniaxial monotonic tension tests and incorporated into a FE model of a cylindrical specimen subject to cyclic tension load- ing. The parameters required for the Chaboche kinematic hardening model have been determined from the stabilised hysteresis loops of the cyclic load simulation and imple- mented into the 3DFE model. The 3DFE predictions of the plastic strain accumulation in the vicinity of the wheel contact at discontinuous railhead edges are shown to be affected by the contact due to passage of wheels rather than the magnitude of the loads the wheels carry. Therefore to eliminate this failure mechanism, modification to the contact patch is essential; reduction in wheel load cannot solve this problem.

Why assessing relevance in medical IR is demanding

This study investigates if and why assessing relevance of clinical records for a clinical retrieval task is cognitively demanding. Previous research has highlighted the challenges and issues information retrieval systems are faced with when determining the relevance of documents in this domain, e.g., the vocabulary mismatch problem. Determining if this assessment imposes cognitive load on human assessors, and why this is the case, may shed lights on what are the (cognitive) processes that assessors use for determining document relevance (in this domain). High cognitive load may impair the ability of the user to make accurate relevance judgements and hence the design of IR mechanisms may need to take this into account in order to reduce the load.

Integrating understandability in the evaluation of consumer health search engines

In this paper we propose a method that integrates the no- tion of understandability, as a factor of document relevance, into the evaluation of information retrieval systems for con- sumer health search. We consider the gain-discount evaluation framework (RBP, nDCG, ERR) and propose two understandability-based variants (uRBP) of rank biased precision, characterised by an estimation of understandability based on document readability and by different models of how readability influences user understanding of document content. The proposed uRBP measures are empirically contrasted to RBP by comparing system rankings obtained with each measure. The findings suggest that considering understandability along with topicality in the evaluation of in- formation retrieval systems lead to different claims about systems effectiveness than considering topicality alone.

Monitoring spatiotemporal dynamics of human movement based on MAC address data

This thesis was a step forward in extracting valuable features from human's movement behaviour in terms of space utilisation based on Media-Access-Control data. This research offered a low-cost and less computational complexity approach compared to existing human's movement tracking methods. This research was successfully applied in QUT's Gardens Point campus and can be scaled to bigger environments and societies. Extractable information from human's movement by this approach can add a significant value to studying human's movement behaviour, enhancing future urban and interior design, improving crowd safety and evacuation plans.

An attack-localizing watermarking scheme for natural language documents

We present a text watermarking scheme that embeds a bitstream watermark Wi in a text document P preserving the meaning, context, and flow of the document. The document is viewed as a set of paragraphs, each paragraph being a set of sentences. The sequence of paragraphs and sentences used to embed watermark bits is permuted using a secret key. Then, English language sentence transformations are used to modify sentence lengths, thus embedding watermarking bits in the Least Significant Bits (LSB) of the sentences’ cardinalities. The embedding and extracting algorithms are public, while the secrecy and security of the watermark depends on a secret key K. The probability of False Positives is extremely small, hence avoiding incidental occurrences of our watermark in random text documents. Majority voting provides security against text addition, deletion, and swapping attacks, further reducing the probability of False Positives. The scheme is secure against the general attacks on text watermarks such as reproduction (photocopying, FAX), reformatting, synonym substitution, text addition, text deletion, text swapping, paragraph shuffling and collusion attacks.

Overview of the ShARe/CLEF eHealth Evaluation Lab 2014

This paper reports on the 2nd ShARe/CLEFeHealth evaluation lab which continues our evaluation resource building activities for the medical domain. In this lab we focus on patients' information needs as opposed to the more common campaign focus of the specialised information needs of physicians and other healthcare workers. The usage scenario of the lab is to ease patients and next-of-kins' ease in understanding eHealth information, in particular clinical reports. The 1st ShARe/CLEFeHealth evaluation lab was held in 2013. This lab consisted of three tasks. Task 1 focused on named entity recognition and normalization of disorders; Task 2 on normalization of acronyms/abbreviations; and Task 3 on information retrieval to address questions patients may have when reading clinical reports. This year's lab introduces a new challenge in Task 1 on visual-interactive search and exploration of eHealth data. Its aim is to help patients (or their next-of-kin) in readability issues related to their hospital discharge documents and related information search on the Internet. Task 2 then continues the information extraction work of the 2013 lab, specifically focusing on disorder attribute identification and normalization from clinical text. Finally, this year's Task 3 further extends the 2013 information retrieval task, by cleaning the 2013 document collection and introducing a new query generation method and multilingual queries. De-identified clinical reports used by the three tasks were from US intensive care and originated from the MIMIC II database. Other text documents for Tasks 1 and 3 were from the Internet and originated from the Khresmoi project. Task 2 annotations originated from the ShARe annotations. For Tasks 1 and 3, new annotations, queries, and relevance assessments were created. 50, 79, and 91 people registered their interest in Tasks 1, 2, and 3, respectively. 24 unique teams participated with 1, 10, and 14 teams in Tasks 1, 2 and 3, respectively. The teams were from Africa, Asia, Canada, Europe, and North America. The Task 1 submission, reviewed by 5 expert peers, related to the task evaluation category of Effective use of interaction and targeted the needs of both expert and novice users. The best system had an Accuracy of 0.868 in Task 2a, an F1-score of 0.576 in Task 2b, and Precision at 10 (P@10) of 0.756 in Task 3. The results demonstrate the substantial community interest and capabilities of these systems in making clinical reports easier to understand for patients. The organisers have made data and tools available for future research and development.

UNICEF, Jakarta, Indonesia : Review of Satap Schools

A. Background and context 1. Education, particularly basic education (grades1-9), has been considered critical to promoting national economic growth and social well being1. Three factors that con-tribute to the above are: (i) Education increases human capital inherent in a labor force and thus increases productivity. It also increases capacity for working with others and builds community consensus to support national development. (ii) Education can in-crease the innovative capacity of a community to support social and economic growth—use of new technologies, products and services to promote growth and wellbeing. (iii) Education can facilitate knowledge transfer needed to understand the social and eco-nomic innovations and new processes, practices and values. Cognizant of the above benefits of education, the Millennium Development Goals (MDG) and the Education for All (EFA) declarations advocating universal basic education were formulated and ratified by UN member countries. 2. Achieving universal primary education (grade 6) may not be sufficient to maxim-ize the above noted socio-economic and cultural benefits. There is general consensus that basic literacy and numeracy up to grade 9 are essential foundational blocks for any good education system to support national development. Basic Education provides an educational achievement threshold that ensures the learning is retained. To achieve this, the donor partner led interventions and the UN declarations such as the MDG goals have sought universal access to basic education (grades 1-9). As many countries progress towards achieving the universal access targets, recent research evidence suggests that we need more than just access to basic education to impact on the na-tional development. Measuring basic education completion cycle, gross enrolment rate (GER) and participation rate etc., has to now include a focus on quality and relevance of the education2. 3. While the above research finding is generally accepted by the Government of In-donesia (GoI), unlike many other developing countries, Indonesia is geographically and linguistically complex and has the fourth largest education sector in the world. It has over 3000 islands, 17,000 ethnic groups and it takes as long as 7 hours to travel from east to west of the country and has multiple time differences. The education system has six years of primary education (grades 1-6), 3 years of junior secondary education (grades 7-9) and three years of senior secondary education (grades 10-12). Therefore, applying the findings of the above cited research in a country like Indonesia is a chal-lenge. Nevertheless, since the adoption of the National Education Law (2003)3 the GoI has made significant progress in improving access to and quality of basic education (grades 1-9). The 2011/12 national education statistics show the primary education (grades 1-6) completion rate was 99.3%, the net enrolment rate (NER) was 95.4% and the GER was 115.4%. This is a significant achievement considering the complexities faced within Indonesia. This increase in the primary education sub-sector, however, has not flowed onto the Junior Secondary School (JSS) education. The transition from pri-mary to JSS is still short of the GoI targets. In 2012, there were 146,826 primary schools feeding into 33,668 junior secondary schools. The transition rate from primary to secondary in 2011/12 was 78%. When considering district or sub-district level data the transition in poor districts could be less than the aggregated national rate. Poverty and lack of parents’ education, confounded by opportunity cost, are major obstacles to transitioning to JSS4. 4. Table 1 presents a summary of GoI initiatives to accelerate the transition to JSS. GoI, with assistance from the donor community, has built 2465 new regular JSS, mak-ing the total number of regular JSS 33,668. In addition, 57,825 new classrooms have been added to existing regular JSS. Also, in rural and remote areas 4136 Satu-Atap5 (SATAP) schools were built to increase access to JSS. These SATAP schools are the focus of this study as they provide education opportunities to the most marginalized, ru-ral, remote children who otherwise would not have access to JSS and consequently not complete basic education.