Distinctive Features of Mobile Messages Processing

World’s mobile market pushes past 2 billion lines in 2005. Success in these competitive markets requires operational excellence with product and service innovation to improve the mobile performance. Mobile users very often prefer to send a mobile instant message or text messages rather than talking on a mobile. Well developed “written speech analysis” does not work not only with “verbal speech” but also with “mobile text messages”. The main purpose of our paper is, firstly, to highlight the problems of mobile text messages processing and, secondly, to show the possible ways of solving these problems.

Information Processing in a Cognitive Model of NLP

A model of the cognitive process of natural language processing has been developed using the formalism of generalized nets. Following this stage-simulating model, the treatment of information inevitably includes phases, which require joint operations in two knowledge spaces – language and semantics. In order to examine and formalize the relations between the language and the semantic levels of treatment, the language is presented as an information system, conceived on the bases of human cognitive resources, semantic primitives, semantic operators and language rules and data. This approach is applied for modeling a specific grammatical rule – the secondary predication in Russian. Grammatical rules of the language space are expressed as operators in the semantic space. Examples from the linguistics domain are treated and several conclusions for the semantics of the modeled rule are made. The results of applying the information system approach to the language turn up to be consistent with the stages of treatment modeled with the generalized net.

Applied Problems of Functional Homonymy Resolution for Russian Language

Applied problems of functional homonymy resolution for Russian language are investigated in the work. The results obtained while using the method of functional homonymy resolution based on contextual rules are presented. Structural characteristics of minimal contextual rules for different types of functional homonymy are researched. Particular attention is paid to studying the control structure of the rules, which allows for the homonymy resolution accuracy not less than 95%. The contextual rules constructed have been realized in the system of technical text analysis.

Hierarchical Three-level Ontology for Text Processing

The principal feature of ontology, which is developed for a text processing, is wider knowledge representation of an external world due to introduction of three-level hierarchy. It allows to improve semantic interpretation of natural language texts.

Vocabulary and language model adaptation using information retrieval

International audience

Structured Approaches for Exploring Interpersonal Relationships in Natural Language Text

Human relationships have long been studied by scientists from domains like sociology, psychology, literature, etc. for understanding people's desires, goals, actions and expected behaviors. In this dissertation we study inter-personal relationships as expressed in natural language text. Modeling inter-personal relationships from text finds application in general natural language understanding, as well as real-world domains such as social networks, discussion forums, intelligent virtual agents, etc. We propose that the study of relationships should incorporate not only linguistic cues in text, but also the contexts in which these cues appear. Our investigations, backed by empirical evaluation, support this thesis, and demonstrate that the task benefits from using structured models that incorporate both types of information. We present such structured models to address the task of modeling the nature of relationships between any two given characters from a narrative. To begin with, we assume that relationships are of two types: cooperative and non-cooperative. We first describe an approach to jointly infer relationships between all characters in the narrative, and demonstrate how the task of characterizing the relationship between two characters can benefit from including information about their relationships with other characters in the narrative. We next formulate the relationship-modeling problem as a sequence prediction task to acknowledge the evolving nature of human relationships, and demonstrate the need to model the history of a relationship in predicting its evolution. Thereafter, we present a data-driven method to automatically discover various types of relationships such as familial, romantic, hostile, etc. Like before, we address the task of modeling evolving relationships but don't restrict ourselves to two types of relationships. We also demonstrate the need to incorporate not only local historical but also global context while solving this problem. Lastly, we demonstrate a practical application of modeling inter-personal relationships in the domain of online educational discussion forums. Such forums offer opportunities for its users to interact and form deeper relationships. With this view, we address the task of identifying initiation of such deeper relationships between a student and the instructor. Specifically, we analyze contents of the forums to automatically suggest threads to the instructors that require their intervention. By highlighting scenarios that need direct instructor-student interactions, we alleviate the need for the instructor to manually peruse all threads of the forum and also assist students who have limited avenues for communicating with instructors. We do this by incorporating the discourse structure of the thread through latent variables that abstractly represent contents of individual posts and model the flow of information in the thread. Such latent structured models that incorporate the linguistic cues without losing their context can be helpful in other related natural language understanding tasks as well. We demonstrate this by using the model for a very different task: identifying if a stated desire has been fulfilled by the end of a story.

Aging and semantic integration in sentence processing: Testing the cognitive workload of wrap-up

The current study investigated the cognitive workload of sentence and clause wrap-up in younger and older readers. A large number of studies have demonstrated the presence of wrap-up effects, peaks in processing time at clause and sentence boundaries that some argue reflect attention to organizational and integrative semantic processes. However, the exact nature of these wrap-up effects is still not entirely clear, with some arguing that wrap-up is not related to processing difficulty, but rather is triggered by a low-level oculomotor response or the implicit monitoring of intonational contour. The notion that wrap-up effects are resource-demanding was directly tested by examining the degree to which sentence and clause wrap-up affects the parafoveal preview benefit. Older and younger adults read passages in which a target word N occurred in a sentence-internal, clause-final, or sentence-final position. A gaze-contingent boundary change paradigm was used in which, on some trials, a non-word preview of word N+1 was replaced by a target word once the eyes crossed an invisible boundary located between words N and N+1. All measures of reading time on word N were longer at clause and sentence boundaries than in the sentence-internal position. In the earliest measures of reading time, sentence and clause wrap-up showed evidence of reducing the magnitude of the preview benefit similarly for younger and older adults. However, this effect was moderated by age in gaze duration, such that older adults showed a complete reduction in the preview benefit in the sentence-final condition. Additionally, sentence and clause wrap-up were negatively associated with the preview benefit. Collectively, the findings from the current study suggest that wrap-up is cognitively demanding and may be less efficient with age, thus, resulting in a reduction of the parafoveal preview during normal reading.

The Processing of Preposition-Stranding Constructions in English

One of the prominent questions in modern psycholinguistics is the relationship between the grammar and the parser. Within the approach of Generative Grammar, this issue has been investigated in terms of the role that Principles of Universal Grammar may play in language processing. The aim of this research experiment is to investigate this topic. Specifically, this experiment aims to test whether the Minimal Structure Principle (MSP) plays a role in the processing of Preposition-Stranding versus Pied-Piped Constructions. This investigation is made with a self-paced reading task, an on-line processing test that measures participants’ unconscious reaction to language stimuli. Monolingual English speakers’ reading times of sentences with Preposition-Stranding and Pied-Piped Constructions are compared. Results indicate that neither construction has greater processing costs, suggesting that factors other than the MSP are active during language processing.

Predire la borsa valori con language model neurali

Nonostante lo scetticismo di molti studiosi circa la possibilità di prevedere l'andamento della borsa valori, esistono svariate teorie ipotizzanti la possibilità di utilizzare le informazioni conosciute per predirne i movimenti futuri. L’avvento dell’intelligenza artificiale nella seconda parte dello scorso secolo ha permesso di ottenere risultati rivoluzionari in svariati ambiti, tanto che oggi tale disciplina trova ampio impiego nella nostra vita quotidiana in molteplici forme. In particolare, grazie al machine learning, è stato possibile sviluppare sistemi intelligenti che apprendono grazie ai dati, riuscendo a modellare problemi complessi. Visto il successo di questi sistemi, essi sono stati applicati anche all’arduo compito di predire la borsa valori, dapprima utilizzando i dati storici finanziari della borsa come fonte di conoscenza, e poi, con la messa a punto di tecniche di elaborazione del linguaggio naturale umano (NLP), anche utilizzando dati in linguaggio naturale, come il testo di notizie finanziarie o l’opinione degli investitori. Questo elaborato ha l’obiettivo di fornire una panoramica sull’utilizzo delle tecniche di machine learning nel campo della predizione del mercato azionario, partendo dalle tecniche più elementari per arrivare ai complessi modelli neurali che oggi rappresentano lo stato dell’arte. Vengono inoltre formalizzati il funzionamento e le tecniche che si utilizzano per addestrare e valutare i modelli di machine learning, per poi effettuare un esperimento in cui a partire da dati finanziari e soprattutto testuali si tenterà di predire correttamente la variazione del valore dell’indice di borsa S&P 500 utilizzando un language model basato su una rete neurale.

Vocabulary Transfer and Knowledge Distillation for Language Model Compression

Natural Language Processing (NLP) has seen tremendous improvements over the last few years. Transformer architectures achieved impressive results in almost any NLP task, such as Text Classification, Machine Translation, and Language Generation. As time went by, transformers continued to improve thanks to larger corpora and bigger networks, reaching hundreds of billions of parameters. Training and deploying such large models has become prohibitively expensive, such that only big high tech companies can afford to train those models. Therefore, a lot of research has been dedicated to reducing a model’s size. In this thesis, we investigate the effects of Vocabulary Transfer and Knowledge Distillation for compressing large Language Models. The goal is to combine these two methodologies to further compress models without significant loss of performance. In particular, we designed different combination strategies and conducted a series of experiments on different vertical domains (medical, legal, news) and downstream tasks (Text Classification and Named Entity Recognition). Four different methods involving Vocabulary Transfer (VIPI) with and without a Masked Language Modelling (MLM) step and with and without Knowledge Distillation are compared against a baseline that assigns random vectors to new elements of the vocabulary. Results indicate that VIPI effectively transfers information of the original vocabulary and that MLM is beneficial. It is also noted that both vocabulary transfer and knowledge distillation are orthogonal to one another and may be applied jointly. The application of knowledge distillation first before subsequently applying vocabulary transfer is recommended. Finally, model performance due to vocabulary transfer does not always show a consistent trend as the vocabulary size is reduced. Hence, the choice of vocabulary size should be empirically selected by evaluation on the downstream task similar to hyperparameter tuning.

NLG-Metricverse: An End-to-End Library for Evaluating Natural Language Generation

Driven by recent deep learning breakthroughs, natural language generation (NLG) models have been at the center of steady progress in the last few years. However, since our ability to generate human-indistinguishable artificial text lags behind our capacity to assess it, it is paramount to develop and apply even better automatic evaluation metrics. To facilitate researchers to judge the effectiveness of their models broadly, we suggest NLG-Metricverse—an end-to-end open-source library for NLG evaluation based on Python. This framework provides a living collection of NLG metrics in a unified and easy- to-use environment, supplying tools to efficiently apply, analyze, compare, and visualize them. This includes (i) the extensive support of heterogeneous automatic metrics with n-arity management, (ii) the meta-evaluation upon individual performance, metric-metric and metric-human correlations, (iii) graphical interpretations for helping humans better gain score intuitions, (iv) formal categorization and convenient documentation to accelerate metrics understanding. NLG-Metricverse aims to increase the comparability and replicability of NLG research, hopefully stimulating new contributions in the area.

Bio-QA-GNN: Reasoning with Language Models and Knowledge Graphs for Interpretable Biomedical Question Answering

Nowadays the idea of injecting world or domain-specific structured knowledge into pre-trained language models (PLMs) is becoming an increasingly popular approach for solving problems such as biases, hallucinations, huge architectural sizes, and explainability lack—critical for real-world natural language processing applications in sensitive fields like bioinformatics. One recent work that has garnered much attention in Neuro-symbolic AI is QA-GNN, an end-to-end model for multiple-choice open-domain question answering (MCOQA) tasks via interpretable text-graph reasoning. Unlike previous publications, QA-GNN mutually informs PLMs and graph neural networks (GNNs) on top of relevant facts retrieved from knowledge graphs (KGs). However, taking a more holistic view, existing PLM+KG contributions mainly consider commonsense benchmarks and ignore or shallowly analyze performances on biomedical datasets. This thesis start from a propose of a deep investigation of QA-GNN for biomedicine, comparing existing or brand-new PLMs, KGs, edge-aware GNNs, preprocessing techniques, and initialization strategies. By combining the insights emerged in DISI's research, we introduce Bio-QA-GNN that include a KG. Working with this part has led to an improvement in state-of-the-art of MCOQA model on biomedical/clinical text, largely outperforming the original one (+3.63\% accuracy on MedQA). Our findings also contribute to a better understanding of the explanation degree allowed by joint text-graph reasoning architectures and their effectiveness on different medical subjects and reasoning types. Codes, models, datasets, and demos to reproduce the results are freely available at: \url{https://github.com/disi-unibo-nlp/bio-qagnn}.

SAB(IO): A BIOLOGICALLY PLAUSIBLE CONNECTIONIST APPROACH TO AUTOMATIC TEXT SUMMARIZATION

An implementation of a computational tool to generate new summaries from new source texts is presented, by means of the connectionist approach (artificial neural networks). Among other contributions that this work intends to bring to natural language processing research, the use of a more biologically plausible connectionist architecture and training for automatic summarization is emphasized. The choice relies on the expectation that it may bring an increase in computational efficiency when compared to the sa-called biologically implausible algorithms.

Frequency Domain Connectivity Identification: An Application of Partial Directed Coherence in fMRI

Functional magnetic resonance imaging (fMRI) has become an important tool in Neuroscience due to its noninvasive and high spatial resolution properties compared to other methods like PET or EEG. Characterization of the neural connectivity has been the aim of several cognitive researches, as the interactions among cortical areas lie at the heart of many brain dysfunctions and mental disorders. Several methods like correlation analysis, structural equation modeling, and dynamic causal models have been proposed to quantify connectivity strength. An important concept related to connectivity modeling is Granger causality, which is one of the most popular definitions for the measure of directional dependence between time series. In this article, we propose the application of the partial directed coherence (PDC) for the connectivity analysis of multisubject fMRI data using multivariate bootstrap. PDC is a frequency domain counterpart of Granger causality and has become a very prominent tool in EEG studies. The achieved frequency decomposition of connectivity is useful in separating interactions from neural modules from those originating in scanner noise, breath, and heart beating. Real fMRI dataset of six subjects executing a language processing protocol was used for the analysis of connectivity. Hum Brain Mapp 30:452-461, 2009. (C) 2007 Wiley-Liss, Inc.