The role of context in image annotation and recommendation

With the rise of smart phones, lifelogging devices (e.g. Google Glass) and popularity of image sharing websites (e.g. Flickr), users are capturing and sharing every aspect of their life online producing a wealth of visual content. Of these uploaded images, the majority are poorly annotated or exist in complete semantic isolation making the process of building retrieval systems difficult as one must firstly understand the meaning of an image in order to retrieve it. To alleviate this problem, many image sharing websites offer manual annotation tools which allow the user to “tag” their photos, however, these techniques are laborious and as a result have been poorly adopted; Sigurbjörnsson and van Zwol (2008) showed that 64% of images uploaded to Flickr are annotated with < 4 tags. Due to this, an entire body of research has focused on the automatic annotation of images (Hanbury, 2008; Smeulders et al., 2000; Zhang et al., 2012a) where one attempts to bridge the semantic gap between an image’s appearance and meaning e.g. the objects present. Despite two decades of research the semantic gap still largely exists and as a result automatic annotation models often offer unsatisfactory performance for industrial implementation. Further, these techniques can only annotate what they see, thus ignoring the “bigger picture” surrounding an image (e.g. its location, the event, the people present etc). Much work has therefore focused on building photo tag recommendation (PTR) methods which aid the user in the annotation process by suggesting tags related to those already present. These works have mainly focused on computing relationships between tags based on historical images e.g. that NY and timessquare co-exist in many images and are therefore highly correlated. However, tags are inherently noisy, sparse and ill-defined often resulting in poor PTR accuracy e.g. does NY refer to New York or New Year? This thesis proposes the exploitation of an image’s context which, unlike textual evidences, is always present, in order to alleviate this ambiguity in the tag recommendation process. Specifically we exploit the “what, who, where, when and how” of the image capture process in order to complement textual evidences in various photo tag recommendation and retrieval scenarios. In part II, we combine text, content-based (e.g. # of faces present) and contextual (e.g. day-of-the-week taken) signals for tag recommendation purposes, achieving up to a 75% improvement to precision@5 in comparison to a text-only TF-IDF baseline. We then consider external knowledge sources (i.e. Wikipedia & Twitter) as an alternative to (slower moving) Flickr in order to build recommendation models on, showing that similar accuracy could be achieved on these faster moving, yet entirely textual, datasets. In part II, we also highlight the merits of diversifying tag recommendation lists before discussing at length various problems with existing automatic image annotation and photo tag recommendation evaluation collections. In part III, we propose three new image retrieval scenarios, namely “visual event summarisation”, “image popularity prediction” and “lifelog summarisation”. In the first scenario, we attempt to produce a rank of relevant and diverse images for various news events by (i) removing irrelevant images such memes and visual duplicates (ii) before semantically clustering images based on the tweets in which they were originally posted. Using this approach, we were able to achieve over 50% precision for images in the top 5 ranks. In the second retrieval scenario, we show that by combining contextual and content-based features from images, we are able to predict if it will become “popular” (or not) with 74% accuracy, using an SVM classifier. Finally, in chapter 9 we employ blur detection and perceptual-hash clustering in order to remove noisy images from lifelogs, before combining visual and geo-temporal signals in order to capture a user’s “key moments” within their day. We believe that the results of this thesis show an important step towards building effective image retrieval models when there lacks sufficient textual content (i.e. a cold start).

The role of cancer related inflammation, Src family kinases and matrix metalloproteinase 9 in colorectal cancer

Colorectal cancer (CRC) is the third most common cancer in the UK with 41,000 new cases diagnosed in 2011. Despite undergoing potentially curative resection, a significant amount of patients develop recurrence. Biomarkers that aid prognostication or identify patients who are suitable for adjuvant treatments are needed. The TNM staging system does a reasonably good job at offering prognostic information to the treating clinician, but it could be better and identifying methods of improving its accuracy are needed. Tumour progression is based on a complex relationship between tumour behaviour and the hosts’ inflammatory responses. Sustained tumour cell proliferation, evading growth suppressors, resisting apoptosis, replicative immortality, sustained angiogenesis, invasion & metastasis, avoiding immune destruction, deregulated cellular energetics, tumour promoting inflammation and genomic instability & mutation have been identified as hallmarks. These hallmarks are malignant behaviors are what makes the cell cancerous and the more extreme the behaviour the more aggressive the cancer the more likely the risk of a poor outcome. There are two primary genomic instability pathways: Microsatellite Instability (MSI) and Chromosomal Instability (CI) also referred to as Microsatellite Stability (MSS). Tumours arising by these pathways have a predilection for specific anatomical, histological and molecular biological features. It is possible that aberrant molecular expression of genes/proteins that promote malignant behaviors may also act as prognostic and predictive biomarkers, which may offer superior prognostic information to classical prognostic features. Cancer related inflammation has been described as a 7th hallmark of cancer. Despite the systemic inflammatory response (SIR) being associated with more aggressive malignant disease, infiltration by immune cells, particularly CD8+ lymphocytes, at the advancing edge of the tumour have been associated with improved outcome and tumour MSI. It remains unknown if the SIR is associated with tumour MSI and this requires further study. The mechanisms by which colorectal cancer cells locally invade through the bowel remain uncertain, but connective tissue degradation by matrix metalloproteinases (MMPs) such as MMP-9 have been implicated. MMP-9 has been found in the cancer cells, stromal cells and patient circulation. Although tumoural MMP-9 has been associated with poor survival, reports are conflicting and contain relatively small sample sizes. Furthermore, the influence of high serum MMP-9 on survival remains unknown. Src family kinases (SFKs) have been implicated in many adverse cancer cell behaviors. SFKs comprise 9 family members BLK, C-SRC, FGR, FYN, HCK, LCK, LYN, YES, YRK. C-SRC has been the most investigated of all SFKs, but the role of other SFKs in cellular behaviors and their prognostic value remains largely unknown. The development of Src inhibitors, such as Dasatinib, has identified SFKs as a potential therapeutic target for patients at higher risk of poor survival. Unfortunately, clinical trials so far have not been promising but this may reflect inadequate patient selection and SFKs may act as useful prognostic and predictive biomarkers. In chapter 3, the association between cancer related inflammation, tumour MSI, clinicopathological factors and survival was tested in two independent cohorts. A training cohort consisting of n=182 patients and a validation cohort of n=677 patients. MSI tumours were associated with a raised CRP (p=0.003). Hypoalbuminaemia was independently associated with poor overall survival in TNM stage II cancer (HR 3.04 (95% CI 1.44 – 6.43);p=0.004), poor recurrence free survival in TNM stage III cancer (HR 1.86 (95% 1.03 – 3.36);p=0.040) and poor overall survival in CI colorectal cancer (HR 1.49 (95% CI 1.06 – 2.10);p=0.022). Interestingly, MSI tumours were associated with poor overall survival in TNM stage III cancer (HR 2.20 (95% CI 1.10 – 4.37);p=0.025). In chapter 4, the role of MMP-9 in colorectal cancer progression and survival was examined. MMP-9 in the tissue was assessed using IHC and serum expression quantified using ELISA. Serum MMP-9 was associated with cancer cell expression (Spearman’s Correlation Coefficient (SCC) 0.393, p<0.001)) and stromal expression (SCC 0.319, p=0.002). Serum MMP-9 was associated with poor recurrence-free (HR 3.37 (95% CI 1.20 – 9.48);p=0.021) and overall survival (HR 3.16 (95% CI 1.22 – 8.15);p=0.018), but tumour MMP-9 was not survival or MSI status. In chapter 5, the role of SFK expression and activation in colorectal cancer progression and survival was studied. On PCR analysis, although LYN, C-SRC and YES were the most highly expressed, FGR and HCK had higher expression profiles as tumours progressed. Using IHC, raised cytoplasmic FAK (tyr 861) was independently associated with poor recurrence free survival in all cancers (HR 1.48 (95% CI 1.02 – 2.16);p=0.040) and CI cancers (HR 1.50 (95% CI 1.02 – 2.21);p=0.040). However, raised cytoplasmic HCK (HR 2.04 (95% CI 1.11 – 3.76);p=0.022) was independently associated with poor recurrence-free survival in TNM stage II cancers. T84 and HT29 cell lines were used to examine the cellular effects of Dasatinib. Cell viability was assessed using WST-1 assay and apoptosis assessed using an ELISA cell death detection assay. Dasatinib increased T84 tumour cell apoptosis in a dose dependent manner and resulted in reduced expression of nuclear (p=0.008) and cytoplasmic (p=0.016) FAK (tyr 861) expression and increased nuclear FGR expression (p=0.004). The results of this thesis confirm that colorectal cancer is a complex disease that represents several subtypes of cancer based on molecular biological behaviors. This thesis concentrated on features of the disease related to inflammation in terms of genetic and molecular characterisation. MSI cancers are closely associated with systemic inflammation but despite this observation, they retain their relatively improved survival. MMP-9 is a feature of tissue remodeling during inflammation and is also associated with degradation of connective tissue, advanced T-stage and poor outcome when measured in the serum. The lack of stromal quantification due to TMA use rather than full sections makes the value of tumoural MMP-9 immunoreactivity in the prognostication and its association with MSI unknown and requires further study. Finally, SFK activation was also associated with SIR, however, only cytoplasmic HCK was independently associated with poor survival in patients with TNM stage II disease, the group of patients where identifying a novel biomarker is most needed. There is still some way to go before these biomarkers are translated into clinical practice and future work needs to focus on obtaining a reliable and robust scientific technique with validation in an adequately powered independent cohort.

The role of inflammation in vascular function and pregnancy outcome in the pregnant stroke prone spontaneously hypertensive rat

During pregnancy, the maternal cardiovascular system undergoes major adaptation. One of these changes is a 40-50 % increase in circulating blood volume which requires a systemic remodelling of the vasculature in order to regulate maternal blood pressure and maximise blood supply to the developing placenta and fetus. These changes are broadly conserved between humans and rats making them an appropriate pre-clinical model in which to study the underlying mechanisms of pregnancy-dependent cardiovascular remodelling. Whilst women are normally protected against cardiovascular disease; pregnancy marks a period of time where women are susceptible to cardiovascular complications. Cardiovascular disease is the leading cause of maternal mortality in the United Kingdom; in particular hypertensive conditions are among the most common complications of pregnancy. One of the main underlying pathologies of these pregnancy complications is thought to be a failure of the maternal cardiovascular system to adapt. The remodelling of the uterine arteries, which directly supply the maternal-fetal interface, is paramount to a healthy pregnancy. Failure of the uterine arteries to remodel sufficiently can result in a number of obstetric complications such as preeclampsia, fetal growth restriction and spontaneous pregnancy loss. At present, it is poorly understood whether this deficient vascular response is due to a predisposition from existing maternal cardiovascular risk factors, the physiological changes that occur during pregnancy or a combination of both. Previous work in our group employed the stroke prone spontaneously hypertensive rat (SHRSP) as a model to investigate pregnancy-dependent remodelling of the uterine arteries. The SHRSP develops hypertension from 6 weeks of age and can be contrasted with the control strain, the Wistar Kyoto (WKY) rat. The phenotype of the SHRSP is therefore reflective of the clinical situation of maternal chronic hypertension during pregnancy. We showed that the SHRSP exhibited a deficient uterine artery remodelling response with respect to both structure and function accompanied by a reduction in litter size relative to the WKY at gestational day (GD) 18. A previous intervention study using nifedipine in the SHRSP achieved successful blood pressure reduction from 6 weeks of age and throughout pregnancy; however uterine artery remodelling and litter size at GD18 was not improved. We concluded that the abnormal uterine artery remodelling present in the SHRSP was independent of chronic hypertension. From these findings, we hypothesised that the SHRSP could be a novel model of spontaneously deficient uterine artery remodelling in response to pregnancy which was underpinned by other as yet unidentified cardiovascular risk factors. In Chapter 1 of this thesis, I have characterised the maternal, placental and fetal phenotype in pregnant (GD18) SHRSP and WKY. The pregnant SHRSP exhibit features of left ventricular hypertrophy in response to pregnancy and altered expression of maternal plasma biomarkers which have been previously associated with hypertension in human pregnancy. I developed a protocol for accurate dissection of the rat uteroplacental unit using qPCR probes specific for each layer. This allowed me to make an accurate and specific statement about gene expression in the SHRSP GD18 placenta; where oxidative stress related gene markers were increased in the vascular compartments. The majority of SHRSP placenta presented at GD18 with a blackened ring which encircled the tissue. Further investigation of the placenta using western blot for caspase 3 cleavage determined that this was likely due to increased cell death in the SHRSP placenta. The SHRSP also presented with a loss of one particular placental cell type at GD18: the glycogen cells. These cells could have been the target of cell death in the SHRSP placenta or were utilised early in pregnancy as a source of energy due to the deficient uterine artery blood supply. Blastocyst implantation was not altered but resorption rate was increased between SHRSP and WKY; indicating that the reduction in litter size in the SHRSP was primarily due to late (>GD14) pregnancy loss. Fetal growth was not restricted in SHRSP which led to the conclusion that SHRSP sacrifice part of their litter to deliver a smaller number of healthier pups. Activation of the immune system is a common pathway that has been implicated in the development of both hypertension and adverse pregnancy outcome. In Chapter 2, I proposed that this may be a mechanism of interest in SHRSP pregnancy and measured the pro-inflammatory cytokine, TNFα, as a marker of inflammation in pregnant SHRSP and WKY and in the placentas from these animals. TNFα was up-regulated in maternal plasma and urine from the GD18 SHRSP. In addition, TNFα release was increased from the GD18 SHRSP placenta as was the expression of the pro-inflammatory TNFα receptor 1 (Tnfr1). In order to investigate whether this excess TNFα was detrimental to SHRSP pregnancy, a vehicle-controlled intervention study using etanercept (a monoclonal antibody which works as a TNFα antagonist) was carried out. Etanercept treatment at GD0, 6, 12 and 18 resulted in an improvement in pregnancy outcome in the SHRSP with an increased litter size and reduced resorption rate. Furthermore, there was an improved uterine artery function in GD18 SHRSP treated with etanercept which was associated with an improved uterine artery blood flow over the course of gestation. In Chapter 3, I sought to identify the source of this detrimental excess of TNFα by designing a panel for maternal leukocytes in the blood and placenta at GD18. A population of CD3- CD161+ cells, which are defined as rat natural killer (NK) cells, were increased in number in the SHRSP. Intracellular flow cytometry also identified this cell type as a source of excess TNFα in blood and placenta from pregnant SHRSP. I then went on to evaluate the effects of etanercept treatment on these CD3- CD161+ cells and showed that etanercept reduced the expression of CD161 and the cytotoxic molecule, granzyme B, in the NK cells. Thus, etanercept limits the cytotoxicity and potential damaging effect of these NK cells in the SHRSP placenta. Analysing the urinary peptidome has clinical potential to identify novel pathways involved with disease and/or to develop biomarker panels to aid and stratify diagnosis. In Chapter 4, I utilised the SHRSP as a pre-clinical model to identify novel urinary peptides associated with hypertensive pregnancy. Firstly, a characterisation study was carried out in the kidney of the WKY and SHRSP. Urine samples from WKY and SHRSP taken at pre-pregnancy, mid-pregnancy (GD12) and late pregnancy (GD18) were used in the peptidomic screen. In order to capture peptides which were markers of hypertensive pregnancy from the urinary peptidomic data, I focussed on those that were only changed in a strain dependent manner at GD12 and 18 and not pre-pregnancy. Peptide fragments from the uromodulin protein were identified from this analysis to be increased in pregnant SHRSP relative to pregnant WKY. This increase in uromodulin was validated at the SHRSP kidney level using qPCR. Uromodulin has previously been identified to be a candidate molecule involved in systemic arterial hypertension but not in hypertensive pregnancy thus is a promising target for further study. In summary, we have characterised the SHRSP as the first model of maternal chronic hypertension during pregnancy and identified that inflammation mediated by TNFα and NK cells plays a key role in the pathology. The evidence presented in this thesis establishes the SHRSP as a pre-clinical model for pregnancy research and can be continued into clinical studies in pregnant women with chronic hypertension which remains an area of unmet research need.