Vocabulary Transfer and Knowledge Distillation for Language Model Compression

Natural Language Processing (NLP) has seen tremendous improvements over the last few years. Transformer architectures achieved impressive results in almost any NLP task, such as Text Classification, Machine Translation, and Language Generation. As time went by, transformers continued to improve thanks to larger corpora and bigger networks, reaching hundreds of billions of parameters. Training and deploying such large models has become prohibitively expensive, such that only big high tech companies can afford to train those models. Therefore, a lot of research has been dedicated to reducing a model’s size. In this thesis, we investigate the effects of Vocabulary Transfer and Knowledge Distillation for compressing large Language Models. The goal is to combine these two methodologies to further compress models without significant loss of performance. In particular, we designed different combination strategies and conducted a series of experiments on different vertical domains (medical, legal, news) and downstream tasks (Text Classification and Named Entity Recognition). Four different methods involving Vocabulary Transfer (VIPI) with and without a Masked Language Modelling (MLM) step and with and without Knowledge Distillation are compared against a baseline that assigns random vectors to new elements of the vocabulary. Results indicate that VIPI effectively transfers information of the original vocabulary and that MLM is beneficial. It is also noted that both vocabulary transfer and knowledge distillation are orthogonal to one another and may be applied jointly. The application of knowledge distillation first before subsequently applying vocabulary transfer is recommended. Finally, model performance due to vocabulary transfer does not always show a consistent trend as the vocabulary size is reduced. Hence, the choice of vocabulary size should be empirically selected by evaluation on the downstream task similar to hyperparameter tuning.

Bio-QA-GNN: Reasoning with Language Models and Knowledge Graphs for Interpretable Biomedical Question Answering

Nowadays the idea of injecting world or domain-specific structured knowledge into pre-trained language models (PLMs) is becoming an increasingly popular approach for solving problems such as biases, hallucinations, huge architectural sizes, and explainability lack—critical for real-world natural language processing applications in sensitive fields like bioinformatics. One recent work that has garnered much attention in Neuro-symbolic AI is QA-GNN, an end-to-end model for multiple-choice open-domain question answering (MCOQA) tasks via interpretable text-graph reasoning. Unlike previous publications, QA-GNN mutually informs PLMs and graph neural networks (GNNs) on top of relevant facts retrieved from knowledge graphs (KGs). However, taking a more holistic view, existing PLM+KG contributions mainly consider commonsense benchmarks and ignore or shallowly analyze performances on biomedical datasets. This thesis start from a propose of a deep investigation of QA-GNN for biomedicine, comparing existing or brand-new PLMs, KGs, edge-aware GNNs, preprocessing techniques, and initialization strategies. By combining the insights emerged in DISI's research, we introduce Bio-QA-GNN that include a KG. Working with this part has led to an improvement in state-of-the-art of MCOQA model on biomedical/clinical text, largely outperforming the original one (+3.63\% accuracy on MedQA). Our findings also contribute to a better understanding of the explanation degree allowed by joint text-graph reasoning architectures and their effectiveness on different medical subjects and reasoning types. Codes, models, datasets, and demos to reproduce the results are freely available at: \url{https://github.com/disi-unibo-nlp/bio-qagnn}.

[strategies For Integration Of Health Care Practices And Sanitary Surveillance In The Context Of The Implementation Of Rede Cegonha - A Brazilian Mother And Infant Health Care Network].

Mother and infant mortality has been the scope of analysis throughout the history of public health in Brazil and various strategies to tackle the issue have been proposed to date. The Ministry of Health has been working on this and the Rede Cegonha strategy is the most recent policy in this context. Given the principle of comprehensive health care and the structure of the Unified Health System in care networks, it is necessary to ensure the integration of health care practices, among which are the sanitary surveillance actions (SSA). Considering that the integration of health care practices and SSA can contribute to reduce mother and infant mortality rates, this article is a result of qualitative research that analyzed the integration of these actions in four cities in the State of São Paulo/Brazil: Campinas, Indaiatuba, Jaguariúna and Santa Bárbara D'Oeste. The research was conducted through interviews with SSA and maternal health managers, and the data were evaluated using thematic analysis. The results converge with other studies, identifying the isolation of health care practices and SSA. The insertion of SSA in collectively-managed areas appears to be a potential strategy for health planning and implementation of actions in the context under scrutiny.

[between The Country And The Laboratory: The Dynamics Of The Production Of Knowledge At The Menopause Outpatients Clinic At Caism, Unicamp.]

This study investigates the practices involved in the production of knowledge about menopause at Caism, Unicamp, a reference center for public policies for women's health. Gynecological appointments and psychological support meetings were observed, and women and doctors were interviewed in order to identify what discourse circulates there and how different actors are brought in to ensure that the knowledge produced attains credibility and travels beyond the boundaries of the teaching hospital to become universal. The analysis is based on localized studies aligned with social studies of science and technology.

Factors Associated With Reporting Of Abuse Against Children And Adolescents By Nurses Within Primary Health Care.

to analyze the factors associated with the underreporting on the part of nurses within Primary Health Care of abuse against children and adolescents. cross-sectional study with 616 nurses. A questionnaire addressed socio-demographic data, profession, instrumentation and knowledge on the topic, identification and reporting of abuse cases. Bivariate and multivariate logistic regression was used. female nurses, aged between 21 and 32 years old, not married, with five or more years since graduation, with graduate studies, and working for five or more years in PHC predominated. The final regression model showed that factors such as working for five or more years, having a reporting form within the PHC unit, and believing that reporting within Primary Health Care is an advantage, facilitate reporting. the study's results may, in addition to sensitizing nurses, support management professionals in establishing strategies intended to produce compliance with reporting as a legal device that ensures the rights of children and adolescents.

[between The Country And The Laboratory: The Dynamics Of The Production Of Knowledge At The Menopause Outpatients Clinic At Caism, Unicamp].

This study investigates the practices involved in the production of knowledge about menopause at Caism, Unicamp, a reference center for public policies for women's health. Gynecological appointments and psychological support meetings were observed, and women and doctors were interviewed in order to identify what discourse circulates there and how different actors are brought in to ensure that the knowledge produced attains credibility and travels beyond the boundaries of the teaching hospital to become universal. The analysis is based on localized studies aligned with social studies of science and technology.

Matrix Support And Institutional Support: Analyzing Their Construction.

This is an analysis of the theoretical and practical construction of the methodology of Matrix Support by means of studies on Paideia Support (Institutional and Matrix Support), which is an inter-professional work of joint care in recent literature and official documents of the Unified Health System (SUS). An attempt was made to describe methodological concepts and strategies. A comparative analysis of Institutional Support and Matrix Support was also conducted using the epistemological framework of Field and Core Knowledge and Practices.

[strategies For Tackling Absenteeism In Dental Appointments In The Family Health Units Of A Large Municipality: Action Research].

The aim of this study was to analyze the reasons for missed appointments in dental Family Health Units (FHU) and implement strategies to reduce same through action research. This is a study conducted in 12 FHUs in Piracicaba in the State of São Paulo from January, 1 to December, 31 2010. The sample was composed of 385 users of these health units who were interviewed over the phone and asked about the reasons for missing dental appointments, as well as 12 dentists and 12 nurses. Two workshops were staged with professionals: the first to assess the data collected in interviews and develop strategy, and the second for evaluation after 4 months. The primary cause for missed appointments was the opening hours of the units coinciding with the work schedule of the users. Among the strategies suggested were lectures on oral health, ongoing education in team meetings, training of Community Health Agents, participation in therapeutic groups and partnerships between Oral Health Teams and the social infrastructure of the community. The adoption of the single medical record was the strategy proposed by professionals. The strategies implemented led to a 66.6% reduction in missed appointments by the units and the motivating nature of the workshops elicited critical reflection to redirect health practices.

Quality Of Life Of Coronary Artery Disease Patients After The Implementation Of Planning Strategies For Medication Adherence.

to compare the general and specific health-related quality of life (HRQoL) between the Intervention (IG) and Control (CG) groups of coronary artery disease patients after the implementation of Action Planning and Coping Planning strategies for medication adherence and to verify the relationship between adherence and HRQoL. this was a controlled and randomized study. the sample (n=115) was randomized into two groups, IG (n=59) and CG (n=56). Measures of medication adherence and general and specific HRQoL were obtained in the baseline and after two months of monitoring. the findings showed that the combination of intervention strategies - Action Planning and Coping Planning for medication adherence did not affect the HRQoL of coronary artery disease patients in outpatient monitoring.

Trends and strategies of ozone application in environmental problems

Efforts presented by the scientific community in recent years towards the development of numerous green chemical processes and wastewater treatment technologies are presented and discussed. In the light of these approaches, environmentally friendly technologies, as well as the key role played by the well-known advanced oxidation processes, are discussed, giving special attention to the ones comprising ozone applications. Fundamentals and applied aspects dealing with ozone technology and its application are also presented.

A construção de uma escala de estratégias de aprendizagem para alunos do ensino fundamental

Although learning strategies are important tools for schooling process, there is a lack of national instruments to evaluate their knowledge by brazilian students. Therefore, the objectives of this paper are to describe the steps necessary for the construction of a scale to evaluate the learning strategies for basic education students and to present the preliminary study of its psychometric properties. It is also discussed the utility of this instrument for diagnosis, intervention and prevention in school and educational psychology.

Educação Fisica no ensino medio-periodo noturno : um estudo participante

Universidade Estadual de Campinas. Faculdade de Educação Física

Pedagogia do treinamento : metodo, procedimentos pedagogicos e as multiplas competencias do tecnico nos jogos desportivos coletivos

Universidade Estadual de Campinas. Faculdade de Educação Física

A constituição da intervenção profissional em Educação Fisica : interações entre o conhecimento formalizado e a noção de competencia

Universidade Estadual de Campinas. Faculdade de Educação Física

O corpo e seus textos : o estetico, o politico e o pedagogico na dança

Universidade Estadual de Campinas. Faculdade de Educação Física