Using interaction data for improving the offline and online evaluation of search engines

This thesis investigates how web search evaluation can be improved using historical interaction data. Modern search engines combine offline and online evaluation approaches in a sequence of steps that a tested change needs to pass through to be accepted as an improvement and subsequently deployed. We refer to such a sequence of steps as an evaluation pipeline. In this thesis, we consider the evaluation pipeline to contain three sequential steps: an offline evaluation step, an online evaluation scheduling step, and an online evaluation step. In this thesis we show that historical user interaction data can aid in improving the accuracy or efficiency of each of the steps of the web search evaluation pipeline. As a result of these improvements, the overall efficiency of the entire evaluation pipeline is increased. Firstly, we investigate how user interaction data can be used to build accurate offline evaluation methods for query auto-completion mechanisms. We propose a family of offline evaluation metrics for query auto-completion that represents the effort the user has to spend in order to submit their query. The parameters of our proposed metrics are trained against a set of user interactions recorded in the search engine’s query logs. From our experimental study, we observe that our proposed metrics are significantly more correlated with an online user satisfaction indicator than the metrics proposed in the existing literature. Hence, fewer changes will pass the offline evaluation step to be rejected after the online evaluation step. As a result, this would allow us to achieve a higher efficiency of the entire evaluation pipeline. Secondly, we state the problem of the optimised scheduling of online experiments. We tackle this problem by considering a greedy scheduler that prioritises the evaluation queue according to the predicted likelihood of success of a particular experiment. This predictor is trained on a set of online experiments, and uses a diverse set of features to represent an online experiment. Our study demonstrates that a higher number of successful experiments per unit of time can be achieved by deploying such a scheduler on the second step of the evaluation pipeline. Consequently, we argue that the efficiency of the evaluation pipeline can be increased. Next, to improve the efficiency of the online evaluation step, we propose the Generalised Team Draft interleaving framework. Generalised Team Draft considers both the interleaving policy (how often a particular combination of results is shown) and click scoring (how important each click is) as parameters in a data-driven optimisation of the interleaving sensitivity. Further, Generalised Team Draft is applicable beyond domains with a list-based representation of results, i.e. in domains with a grid-based representation, such as image search. Our study using datasets of interleaving experiments performed both in document and image search domains demonstrates that Generalised Team Draft achieves the highest sensitivity. A higher sensitivity indicates that the interleaving experiments can be deployed for a shorter period of time or use a smaller sample of users. Importantly, Generalised Team Draft optimises the interleaving parameters w.r.t. historical interaction data recorded in the interleaving experiments. Finally, we propose to apply the sequential testing methods to reduce the mean deployment time for the interleaving experiments. We adapt two sequential tests for the interleaving experimentation. We demonstrate that one can achieve a significant decrease in experiment duration by using such sequential testing methods. The highest efficiency is achieved by the sequential tests that adjust their stopping thresholds using historical interaction data recorded in diagnostic experiments. Our further experimental study demonstrates that cumulative gains in the online experimentation efficiency can be achieved by combining the interleaving sensitivity optimisation approaches, including Generalised Team Draft, and the sequential testing approaches. Overall, the central contributions of this thesis are the proposed approaches to improve the accuracy or efficiency of the steps of the evaluation pipeline: the offline evaluation frameworks for the query auto-completion, an approach for the optimised scheduling of online experiments, a general framework for the efficient online interleaving evaluation, and a sequential testing approach for the online search evaluation. The experiments in this thesis are based on massive real-life datasets obtained from Yandex, a leading commercial search engine. These experiments demonstrate the potential of the proposed approaches to improve the efficiency of the evaluation pipeline.

Heart failure in young adults

Heart failure (HF) is a major health concern affecting 15 million people in Europe and around 900 000 people in the U.K. HF predominantly affects the elderly, with the mean age of patients with a diagnosis of HF between 70 and 80 years. Most previous HF studies have accordingly focused on older patients. Although HF is less common in younger adults (<65 years), 15% to 20% of patients hospitalised with HF are younger than 60 years of age. Very few studies have described the characteristics of younger adults with HF and its outcome. The aims of this thesis are to describe the clinical characteristics of younger adults with HF, explore the epidemiology of HF in younger adults and determine their short- and long-term outcomes. This was made possible by access multiple databases consisting of large patient cohorts with HF. The first chapter is a systematic literature review of younger adults with HF. Gaps in the current literature were identified and the thesis focused on some of these. The CHARM study allows detail characterisations of younger adults with HF. It recorded characteristics of patients with HF, including symptoms and signs of HF, electrocardiographic changes, chest radiographic findings, and also left ventricular ejection fraction. HF hospitalisations and its precipitating factors were also recorded systematically. Younger adults were more likely to have a third heart sound and hepatomegaly, but less likely to have pulmonary crackles and peripheral oedema. Similarly, radiological findings in younger adults were less likely to show interstitial pulmonary oedema or pleural effusion. Interestingly, younger adults aged <40 years not only have similar HF hospitalisation rate to older patients, however during their presentation with decompensated HF, they were less likely to have clinical pulmonary oedema and radiological signs of HF. Physicians managing younger adults with HF need to be aware of this. Younger adults were also less compliant with medications and lifestyle restriction resulting in hospitalisation with decompensated HF. Fortunately, despite these challenges, mortality rates in younger adults with HF were lower compared to older patients. To further substantiate the findings from the CHARM study, the MAGGIC study, a meta-analysis consists of over 40 000 patients with HF from large observational studies and randomised controlled trials, was examined. In both databases, the commonest aetiology of HF in younger adults was dilated cardiomyopathy. The ejection fraction was the lowest in younger adults. Similar to the CHARM study, mortality rates in younger adults were lower compared to older patients. However, in the MAGGIC study, by stratifying mortality into patients with preserved ejection fraction and with reduced ejection fraction, younger patients with preserved ejection fraction have a much lower mortality rate compared to patients with reduced ejection fraction. Findings from clinical trials are not always reflective of the real life clinical practice. The U.K. Clinical Practice Research Datalink (CPRD), a large and well-validated primary care database with 654 practices contributing information into the database representing approximated 8% of the U.K. population, is a rich dataset offering a unique opportunity to examine the characteristics, treatments, and outcomes of younger adults with HF in the community. In contrast to the CHARM and MAGGIC studies, younger adults aged <40 years were stratified into 20-29 and 30-39 years in the CPRD analysis. This is possible due to the larger number of younger adults with HF. Further stratifying the younger age groups demonstrated heterogeneity among younger adults with HF. In contrast to previous data showing younger adults have lower co-morbidities, the proportions of depression, chronic kidney disease, asthma, and any connective tissue disease were high among patients aged 20-29 years in the analysis from the CPRD. Surprisingly, the treatment rates for angiotensin converting enzyme (ACE) inhibitor, and aldosterone antagonist were the lowest in patients aged 20-29 years. With the exception of patients aged ≥80 years, treatment rate with beta-blocker was also the lowest in patients aged 20-29 years. With over two decades of follow up, long-term mortality rates in younger adults with HF can be determined. The mortality rates continued to decline from 1988 to 2011. Physicians managing younger adults with HF can now use this contemporary data to provide prognostic information to patients and their family. A hospital administrative database is the logical next platform to explore younger adults with HF. The Alberta Ministry of Health database links an outpatient database to a hospitalisation database providing ample data to examine the relationship between outpatient clinic visits and hospital admissions in younger adults with HF. Following a diagnosis of HF in the outpatient setting, younger adults were admitted to the hospital with decompensated HF much sooner than older patients. Younger adults also presented to emergency department more frequently following their first hospitalisation for HF. In conclusion, this thesis presented the characteristics and outcomes of younger adults with HF, and helped to extend our current understanding on this important topic. I hope the data presented here will benefit not only physicians looking after younger adults with HF, but also patients and their family.