Machine learning techniques for implicit interaction using mobile sensors

Interactions in mobile devices normally happen in an explicit manner, which means that they are initiated by the users. Yet, users are typically unaware that they also interact implicitly with their devices. For instance, our hand pose changes naturally when we type text messages. Whilst the touchscreen captures finger touches, hand movements during this interaction however are unused. If this implicit hand movement is observed, it can be used as additional information to support or to enhance the users’ text entry experience. This thesis investigates how implicit sensing can be used to improve existing, standard interaction technique qualities. In particular, this thesis looks into enhancing front-of-device interaction through back-of-device and hand movement implicit sensing. We propose the investigation through machine learning techniques. We look into problems on how sensor data via implicit sensing can be used to predict a certain aspect of an interaction. For instance, one of the questions that this thesis attempts to answer is whether hand movement during a touch targeting task correlates with the touch position. This is a complex relationship to understand but can be best explained through machine learning. Using machine learning as a tool, such correlation can be measured, quantified, understood and used to make predictions on future touch position. Furthermore, this thesis also evaluates the predictive power of the sensor data. We show this through a number of studies. In Chapter 5 we show that probabilistic modelling of sensor inputs and recorded touch locations can be used to predict the general area of future touches on touchscreen. In Chapter 7, using SVM classifiers, we show that data from implicit sensing from general mobile interactions is user-specific. This can be used to identify users implicitly. In Chapter 6, we also show that touch interaction errors can be detected from sensor data. In our experiment, we show that there are sufficient distinguishable patterns between normal interaction signals and signals that are strongly correlated with interaction error. In all studies, we show that performance gain can be achieved by combining sensor inputs.

Query routing in cooperative semi-structured peer-to-peer information retrieval networks

Conventional web search engines are centralised in that a single entity crawls and indexes the documents selected for future retrieval, and the relevance models used to determine which documents are relevant to a given user query. As a result, these search engines suffer from several technical drawbacks such as handling scale, timeliness and reliability, in addition to ethical concerns such as commercial manipulation and information censorship. Alleviating the need to rely entirely on a single entity, Peer-to-Peer (P2P) Information Retrieval (IR) has been proposed as a solution, as it distributes the functional components of a web search engine – from crawling and indexing documents, to query processing – across the network of users (or, peers) who use the search engine. This strategy for constructing an IR system poses several efficiency and effectiveness challenges which have been identified in past work. Accordingly, this thesis makes several contributions towards advancing the state of the art in P2P-IR effectiveness by improving the query processing and relevance scoring aspects of a P2P web search. Federated search systems are a form of distributed information retrieval model that route the user’s information need, formulated as a query, to distributed resources and merge the retrieved result lists into a final list. P2P-IR networks are one form of federated search in routing queries and merging result among participating peers. The query is propagated through disseminated nodes to hit the peers that are most likely to contain relevant documents, then the retrieved result lists are merged at different points along the path from the relevant peers to the query initializer (or namely, customer). However, query routing in P2P-IR networks is considered as one of the major challenges and critical part in P2P-IR networks; as the relevant peers might be lost in low-quality peer selection while executing the query routing, and inevitably lead to less effective retrieval results. This motivates this thesis to study and propose query routing techniques to improve retrieval quality in such networks. Cluster-based semi-structured P2P-IR networks exploit the cluster hypothesis to organise the peers into similar semantic clusters where each such semantic cluster is managed by super-peers. In this thesis, I construct three semi-structured P2P-IR models and examine their retrieval effectiveness. I also leverage the cluster centroids at the super-peer level as content representations gathered from cooperative peers to propose a query routing approach called Inverted PeerCluster Index (IPI) that simulates the conventional inverted index of the centralised corpus to organise the statistics of peers’ terms. The results show a competitive retrieval quality in comparison to baseline approaches. Furthermore, I study the applicability of using the conventional Information Retrieval models as peer selection approaches where each peer can be considered as a big document of documents. The experimental evaluation shows comparative and significant results and explains that document retrieval methods are very effective for peer selection that brings back the analogy between documents and peers. Additionally, Learning to Rank (LtR) algorithms are exploited to build a learned classifier for peer ranking at the super-peer level. The experiments show significant results with state-of-the-art resource selection methods and competitive results to corresponding classification-based approaches. Finally, I propose reputation-based query routing approaches that exploit the idea of providing feedback on a specific item in the social community networks and manage it for future decision-making. The system monitors users’ behaviours when they click or download documents from the final ranked list as implicit feedback and mines the given information to build a reputation-based data structure. The data structure is used to score peers and then rank them for query routing. I conduct a set of experiments to cover various scenarios including noisy feedback information (i.e, providing positive feedback on non-relevant documents) to examine the robustness of reputation-based approaches. The empirical evaluation shows significant results in almost all measurement metrics with approximate improvement more than 56% compared to baseline approaches. Thus, based on the results, if one were to choose one technique, reputation-based approaches are clearly the natural choices which also can be deployed on any P2P network.

Effects of modality, urgency and situation on responses to multimodal warnings for drivers

Signifying road-related events with warnings can be highly beneficial, especially when imminent attention is needed. This thesis describes how modality, urgency and situation can influence driver responses to multimodal displays used as warnings. These displays utilise all combinations of audio, visual and tactile modalities, reflecting different urgency levels. In this way, a new rich set of cues is designed, conveying information multimodally, to enhance reactions during driving, which is a highly visual task. The importance of the signified events to driving is reflected in the warnings, and safety-critical or non-critical situations are communicated through the cues. Novel warning designs are considered, using both abstract displays, with no semantic association to the signified event, and language-based ones, using speech. These two cue designs are compared, to discover their strengths and weaknesses as car alerts. The situations in which the new cues are delivered are varied, by simulating both critical and non-critical events and both manual and autonomous car scenarios. A novel set of guidelines for using multimodal driver displays is finally provided, considering the modalities utilised, the urgency signified, and the situation simulated.

Trapping phosphorus (III) into polyoxometalates: synthesis, characterisation and electronic behaviour study

The introduction of electronically-active heteroanions into polyoxometalates (POMs) is one of the emerging topics in this field. The novel clusters have shown unprecedented intramolecular electron-transfer features that can be directly mediated by the incorporated heteroanions. In this thesis, we will focus on the study of phosphite (HPO32-) as new non-traditional heteroanions, discover HPO32- templated nanostructures, investigate their electronic behaviours as well as understand the self-assembly process of HPO32--templated species. The thesis starts with incorporating HPO32- into POM cages. The feasibility of this work was illustrated by the successful trapping of HPO32- into a “Trojan Horse” type {W18O56} nanocage. The reactivity of embedded {HPO3} was fully studied, showing the cluster undergoes a structural rearrangement in solution whereby the {HPO3} moieties dimerise to form a weakly interacting (O3PH···HPO3) moiety. In the crystalline state a temperature-dependent intramolecular redox reaction and structural rearrangement occurs. This rearrangement appears to proceed via an intermediate containing two different templates, a pyramidal {HPO3} and a tetrahedral {PO4} moiety. {HPO3} templated POM cages were then vigorously expanded and led to the isolation of five either fully oxidised or mixed-valence clusters trapped with mono-, di-, or tri- {HPO3}. Interestingly, an intriguing 3D honeycomb-like host-guest structure was also synthesised. The porous framework was self-aggregated by a tri-phopshite anion templated {W21} cluster with a {VO4} templated Wells-Dawson type {W18} acting as a guest species within the hexagonal channels. Based on this work, we further extended the templating anions to two different redox-active heteroanions, and discovered a unique mixed-heteroatom templated system built by pairing redox-active {HPIIIO3} with {TeO3}, {SeO3} or {AsO3}. Two molecular systems were developed, ie. “Trojan Horse” type [W18O56(HPO3)0.8(SeO3)1.2(H2O)2]8- and cross-shaped [H4P4X4W64O224]32-/36-, where X=TeIV, SeIV, AsIII. In the case of {W18(HPO3)0.8(SeO3)1.2}, the compound is found to be a mixture of heteroleptic {W18(HPO3)(SeO3)} and homoleptic {W18(SeO3)2} and {W18(HPO3)2}, identified by single crystal x-ray diffraction, NMR as well as high resolution mass spectrometry. The cluster exhibited similar temperature-dependent electronic features to “Trojan Horse” type {W18(HPO3)2O56}. However, due to the intrinsic reactivity difference between {HPO3} and {SeO3}, the thermal treatment leads to the formation of an unusual species [W18O55(PO4)(SeO3)]5-, in which {HPO3} was fully oxidised to {PO4} within the cage, whereas and lone-pair-containing {SeO3} heteroanions were kept intact inside the shell. This finding is extremely interesting, as it demonstrated that multiple and independent intramolecular electronic performance can be achieved by the coexistence of distinct heteroatoms within a single molecule. On the other hand, the cross-shaped [H4P4X4W64O224]32-/36- were constructed by four {W15(HPO3)(XO3)} building units linked by four {WO6} octahedra. Each building unit traps two different heteroatoms. It is interesting to note that the mixed heteroatom species show self-sorting, with a highly selective positional preference. Smaller ionic sized {HPO3} are self-organised into the uncapped side of {W15} cavity, whereas closed side are occupied by larger heteroatoms, which is surprisingly opposed to steric hindrance. Density functional theory (DFT) calculations are currently underway to have a full understanding of the preference of heteroatom substitutions. This series of clusters is of great interest in terms of achieving single molecule-based heteroatom-dependent multiple levels of electron transfer. It has opened a new way to design and synthesise POMs with higher diversity of electrical states, which may lead to a new type of Q-bits for quantum computing. The third chapter is focused on developing polyoxotungstate building blocks templated by {HPO3}. A series of building blocks, {W15O48(HPO3)2}, {W9O30(HPO3)} {W12O40(HPO3)2} and hexagonal {W6O18(HPO3)} have been obtained. The first four building blocks have been reported with {SeO3} and/or {TeO3} heteroanions. This result demonstrates {HPO3} has a similar reactivity as {SeO3} and {TeO3}, therefore studying the self-assembly of {HPO3}-based building blocks would be helpful to have a general understanding of pyramidal heteroatom-based molecular systems. The hexagonal {W6O18(HPO3)} is observed for the first time in polyoxotungstates, showing some of reactivity difference between {HPO3} and {SeO3} and {TeO3}. Furthermore, inorganic salts and pH values have some directing influence on the formation and transformation of various building blocks, resulting in the discovery of a family of {HPO3}-based clusters with nuclearity ranging from {W29} to {W106}. High resolution mass spectrometry was also carried out to investigate the cluster solution behaviour and also gain information of building block speciation. It is found that some clusters experienced decomposition, which gives rise to potential building blocks accountable for the self-assembly.