Vocabulary Transfer and Knowledge Distillation for Language Model Compression

Natural Language Processing (NLP) has seen tremendous improvements over the last few years. Transformer architectures achieved impressive results in almost any NLP task, such as Text Classification, Machine Translation, and Language Generation. As time went by, transformers continued to improve thanks to larger corpora and bigger networks, reaching hundreds of billions of parameters. Training and deploying such large models has become prohibitively expensive, such that only big high tech companies can afford to train those models. Therefore, a lot of research has been dedicated to reducing a model’s size. In this thesis, we investigate the effects of Vocabulary Transfer and Knowledge Distillation for compressing large Language Models. The goal is to combine these two methodologies to further compress models without significant loss of performance. In particular, we designed different combination strategies and conducted a series of experiments on different vertical domains (medical, legal, news) and downstream tasks (Text Classification and Named Entity Recognition). Four different methods involving Vocabulary Transfer (VIPI) with and without a Masked Language Modelling (MLM) step and with and without Knowledge Distillation are compared against a baseline that assigns random vectors to new elements of the vocabulary. Results indicate that VIPI effectively transfers information of the original vocabulary and that MLM is beneficial. It is also noted that both vocabulary transfer and knowledge distillation are orthogonal to one another and may be applied jointly. The application of knowledge distillation first before subsequently applying vocabulary transfer is recommended. Finally, model performance due to vocabulary transfer does not always show a consistent trend as the vocabulary size is reduced. Hence, the choice of vocabulary size should be empirically selected by evaluation on the downstream task similar to hyperparameter tuning.

Activation of endogenous p53 by combined p19Arf gene transfer and nutlin-3 drug treatment modalities in the murine cell lines B16 and C6

Background: Reactivation of p53 by either gene transfer or pharmacologic approaches may compensate for loss of p19Arf or excess mdm2 expression, common events in melanoma and glioma. In our previous work, we constructed the pCLPG retroviral vector where transgene expression is controlled by p53 through a p53-responsive promoter. The use of this vector to introduce p19Arf into tumor cells that harbor p53wt should yield viral expression of p19Arf which, in turn, would activate the endogenous p53 and result in enhanced vector expression and tumor suppression. Since nutlin-3 can activate p53 by blocking its interaction with mdm2, we explored the possibility that the combination of p19Arf gene transfer and nutlin-3 drug treatment may provide an additive benefit in stimulating p53 function. Methods: B16 (mouse melanoma) and C6 (rat glioma) cell lines, which harbor p53wt, were transduced with pCLPGp19 and these were additionally treated with nutlin-3 or the DNA damaging agent, doxorubicin. Viral expression was confirmed by Western, Northern and immunofluorescence assays. p53 function was assessed by reporter gene activity provided by a p53-responsive construct. Alterations in proliferation and viability were measured by colony formation, growth curve, cell cycle and MTT assays. In an animal model, B16 cells were treated with the pCLPGp19 virus and/or drugs before subcutaneous injection in C57BL/6 mice, observation of tumor progression and histopathologic analyses. Results: Here we show that the functional activation of endogenous p53wt in B16 was particularly challenging, but accomplished when combined gene transfer and drug treatments were applied, resulting in increased transactivation by p53, marked cell cycle alteration and reduced viability in culture. In an animal model, B16 cells treated with both p19Arf and nutlin-3 yielded increased necrosis and decreased BrdU marking. In comparison, C6 cells were quite susceptible to either treatment, yet p53 was further activated by the combination of p19Arf and nutlin-3. Conclusions: To the best of our knowledge, this is the first study to apply both p19Arf and nutlin-3 for the stimulation of p53 activity. These results support the notion that a p53 responsive vector may prove to be an interesting gene transfer tool, especially when combined with p53- activating agents, for the treatment of tumors that retain wild-type p53.

Relative frequency of knowledge of results and task complexity in the motor skill acqusition

The aim of this study was to investigate the effects of knowledge of results (KR) frequency and task complexity on motor skill acquisition. The task consisted of throwing a bocha ball to place it as close as possible to the target ball. 120 students ages 11 to 73 years were assigned to one of eight experimental groups according to knowledge of results frequency (25, 50, 75, and 100%) and task complexity (simple and complex). Subjects performed 90 trials in the acquisition phase and 10 trials in the transfer test. The results showed that knowledge of results given at a frequency of 25% resulted in an inferior absolute error than 50% and inferior variable error than 50, 75, and 100 I frequencies, but no effect of task complexity was found.

Roadmapping for technology push and partnership: A contribution for open innovation environments

There are several tools in the literature that support innovation in organizations. Some of the most cited are the so-called technology roadmapping methods, also known as TRM. However, these methods are designed primarily for organizations that adopt the market pull strategy of technology-product integration. Organizations that adopt the technology push integration strategy are neglected in the literature. Furthermore, with the advent of open innovation, it is possible to note the need to consider the adoption of partnerships in the innovation process. Thus, this study proposes a method of technology roadmapping, identified as method for technology push (MTP), applicable to organizations that adopt the technology push integration strategy, such as SMEs and independent research centers in an open-innovation environment. The method was developed through action-research and was assessed from two analytical standpoints: externally, via a specific literature review on its theoretical contributions, and internally, through the analysis of potential users` perceptions on the feasibility of applying MTP. The results indicate both the unique character of the method and its perceived implementation feasibility. Future research is suggested in order to validate the method in different types of organizations (C) 2011 Elsevier Ltd. All rights reserved.

Comparison of oxygen mass transfer coefficient in simple and extractive fermentation systems

Aeration and agitation are important variables to ensure effective oxygen transfer rate during aerobic bioprocesses: therefore, the knowledge of the volumetric mass transfer coefficient (k(L)a) is required. In view of selecting the optimum oxygen requirements for extractive fermentation in aqueous two-phase system (ATPS), the k(L)a values in a typical ATPS medium were compared in this work with those in distilled water and in a simple fermentation medium. in the absence of biomass. Aeration and agitation were selected as the independent variables using a 2(2) full factorial design. Both variables showed statistically significant effects on k(L)a, and the highest values of this parameter in both media for simple fermentation (241 s(-1)) and extractive fermentation with ATPS (70.3 s(-1)) were observed at the highest levels of aeration (5 vvm) and agitation (1200 rpm). The k(L)a values were then used to establish mathematical correlations of this response as a function of the process variables. The exponents of the power number (N(3)D(2)) and superficial gas velocity (V(s)) determined in distilled water (alpha = 0.39 and beta = 0.47, respectively) were in reasonable agreement with the ones reported in the literature for several aqueous systems and close to those determined for a simple fermentation medium (alpha=0.38 and beta=0.41). On the other hand, as expected by the increased viscosity in the presence of polyethylene glycol, their values were remarkably higher in a typical medium for extractive fermentation (alpha=0.50 and beta=1.0). A reasonable agreement was found between the experimental data of k(L)a for the three selected systems and the values predicted by the theoretical models, under a wide range of operational conditions. (C) 2009 Elsevier B.V. All rights reserved.

Development of disabled, replication-defective gene transfer vectors from the Jembrana disease virus, a new infectious agent of cattle

Jembrana disease virus (JDV) is a newly isolated and characterised bovine lentivirus. It causes an acute disease in Ball cattle (Bos javanicus). which can be readily transmitted to susceptible cattle with 17% mortality. There is as yet no treatment or preventive vaccine. We have developed a gene transfer vector system based on JDV that has three components. The first of the components is a bicistronic transfer vector plasmid that was constructed to contain cis-sequences from the JDV genome, including 5 '- and 3 ' -long terminal repeats (LTRs), 0.4 kb of truncated gag and 1.1 kb of 3 ' -env, a multiple cloning site to accommodate the gene(s) of interest for transfer, and an internal ribosome entry site plus the neomycin phosphotransferase (Neo) gene cassette for antibiotic selection. The second element is a packaging plasmid that contains trans-sequences. including gag, pol. vif, tar and rev: but without the env and packaging signals. The third is a plasmid encoding the G glycoprotein of vesicular stomatitis virus (VSV-G) to supply the vector an envelope for pseudotyping. Cotransfection of 293T cells with these three plasmid components produced VSV-G pseudotyped. disabled, replication defective, bicistronic JDV vectors encoding the green fluorescent protein (EGFP) and the Neo resistance selection maker simultaneously with a titre range of (0.4-1.2) x 10(6) CFU/ml. Transduction of several replicating primary and transformed cells from cattle, primate and human sources and importantly growth-arrested cells with the JDV vectors showed high efficiency of EGFP gene transfer at 35-75%, which was stable and the expression of EGFP was long term. Furthermore, these JDV vectors were designed to suit the inclusion and expression of genes corresponding to JDV specific proteins, such as gag or env, for the development of vaccines for Jembrana disease. This strategy should also be applicable to other bovine diseases as wall. The design and construction of the JDV vector system should facilitate the study of the lentivirology and pathogenesis of the diseases associated with JDV or other bovine virus infections. To our knowledge, this is the first such vector system developed from a cattle virus. (C) 2001 Elsevier Science B.V. All rights reserved.

Using the World Wide Web to revolutionise technology transfer and training in the water and wastewater industries

Industry professionals of the near future will be supported by an IT infrastructure that enables them to complete a task by drawing on resources and people with expertise anywhere in the world, and access to knowledge through specific training programs that address the task requirements. The increasing uptake of new technologies enables information to reach a diverse population and to provide flexible learning environments 24 hours a day, 7 days a week. This paper examines one of the key areas where the World Wide Web will impact on the water and wastewater industries, namely technology transfer and training. The authors will present their experiences of developing online training courses for wastewater industry professionals over the last two years. The perspective is that of two people working at the coalface.

What is the value of firms' prior knowledge? Building organisational knowledge capabilities

A firm's prior knowledge facilitates the absorption of new knowledge, thereby renewing a firm's systematic search, transfer and absorption capabilities. The rapidly expanding field of biotechnology is characterised by the convergence of disparate sciences and technologies. This paper, the shift from proteinbased to DNA-based diagnostic technologies, quantifies the value of a firm's prior knowledge and its relation to future knowledge development. Four dimensions of diagnostic and four dimensions of knowledge in biotechnology firms are analysed. A simple scaled matrix method is developed to quantify the positive and negative heuristic values of prior scientific and technological knowledge that is useful for the acquisition and absorption of new knowledge.

Valuing of firm's prior knowledge: A measure of knowledge distance

Knowledge, especially scientific and technological knowledge, grows according to knowledge trajectories and guideposts that make up the prior knowledge of an organization. We argue that these knowledge structures and their specific components lead to successful innovation. A firm's prior knowledge facilitates the absorption of new knowledge, thereby renewing a firm's systematic search, transfer and acquisition of knowledge and capabilities. In particular, the exponential growth in biotechnology is characterized by the convergence of disparate scientific and technological knowledge resources. This paper examines the shift from protein-based to DNA-based diagnostic technologies as an example, to quantify the value of a firm's prior knowledge using relative values of knowledge distance. The distance between core prior knowledge and the rate of transition from one knowledge system to another has been identified as a proxy for the value a firm's prior knowledge. The overall difficulty of transition from one technology paradigm to another is discussed. We argue this transition is possible when the knowledge distance is minimal and the transition process has a correspondingly high value of absorptive capacities. Our findings show knowledge distance is a determinant of the feasibility, continuity and capture of scientific and technological knowledge. Copyright © 2003 John Wiley & Sons, Ltd.

Plano de implementa????o do projeto de desenvolvimento de capacidade de governan??a

Desde a d??cada de 1990, o Governo Federal brasileiro vem implementando uma agenda ambiciosa de reformas do Estado, centradas na redu????o da pobreza e na melhoria da efici??ncia dos servi??os p??blicos. As principais prioridades, conforme previstas no Plano Plurianual (PPA) para o per??odo 2003-2007, s??o as seguintes: inclus??o social e redu????o da desigualdade; crescimento econ??mico com gera????o de emprego; distribui????o de renda e respeito ao meio ambiente; promo????o e amplia????o dos direitos de cidadania; e fortalecimento da democracia. No in??cio de 2006, o Governo criou a Pol??tica Nacional de Desenvolvimento de Pessoal (Decreto 5.707), com o objetivo de melhorar e aumentar a efici??ncia e a efic??cia na presta????o de servi??os p??blicos. No marco dessa pol??tica recente, as escolas de administra????o p??blica desempenham um papel fundamental na identifica????o das compet??ncias que precisam ser desenvolvidas nas institui????es do governo, bem como na implementa????o de pol??ticas de capacita????o para os servidores p??blicos, diretamente e/ou em parceria com escolas de governo nos n??veis federal, estadual ou local. O Canad?? tamb??m est?? criando uma estrutura para levantar as compet??ncias necess??rias para os servidores p??blicos e desenvolv??-las como um componente da Renova????o do Servi??o P??blico em todo o governo. Como institui????es l??deres no desenvolvimento de compet??ncias de servidores p??blicos, a Canada School of Public Service (CSPS) e a Escola Nacional de Administra????o P??blica (ENAP) firmaram uma parceria para implementar o Projeto de Desenvolvimento de Capacidade de Governan??a no Brasil. A finalidade do Projeto ?? melhorar a capacidade de servidores p??blicos federais, estaduais e municipais do Brasil para desenvolver e implementar programas de capacita????o e gerenciar pol??ticas p??blicas descentralizadas. Espera-se que essa parceria e o resultante compartilhamento de experi??ncias em capacita????o para governan??a efetiva contribuam para a redu????o da pobreza e das desigualdades no Brasil, por meio do desenvolvimento de compet??ncias de servidores na presta????o de servi??os p??blicos eficazes e eficientes, voltados para o cidad??o. O Projeto re??ne, al??m das duas principais Escolas de Governo no Canad?? e no Brasil, seis Escolas Brasileiras de Administra????o P??blica regionais e duas renomadas Institui????es Acad??micas Canadenses ??? a Queen???s University e a Western Ontario University. O Minist??rio do Desenvolvimento Social e Combate ?? Fome (MDS) e tr??s Secretarias Especiais do Governo Federal ??? Ra??a (SEPPIR), Direitos Humanos (SEDH) e Pol??ticas para as Mulheres (SPM) ??? tamb??m se envolver??o nas atividades de compartilhamento de conhecimentos com o Human Resources and Skills Development Canada (HRSDC) e a Canada Public Service Agency (CPSA). A CIDA fornecer?? CND$1.700.000 por meio do Programa Brasil-Canad?? de Interc??mbio de Conhecimentos para a Promo????o da Equidade (PIPE). A contribui????o da ENAP ser?? de CND$1.069.707 em esp??cie. A CSPS contribuir?? com cursos, al??m de conhecimentos e suporte t??cnicos, avaliados em CND$1.000.000. Aproveitando a parceria entre a CSPS e a ENAP, que resultou na transfer??ncia e na adapta????o bem sucedidas de cursos e metodologias canadenses, o novo projeto extrapola o n??cleo do servi??o p??blico em Bras??lia, alcan??ando escolas de governo em regi??es brasileiras em situa????o de desvantagem. ?? semelhan??a do papel da CSPS no primeiro projeto, a ENAP fortalecer?? a capacidade das escolas parceiras regionais para capacitar servidores p??blicos envolvidos na presta????o de servi??os aos brasileiros. O interc??mbio estruturado entre Minist??rios dos Governos canadense e brasileiro tamb??m aplicar?? a aprendizagem mais diretamente a quest??es de pol??ticas e programas sociais do Brasil. O desafio assumido neste Projeto ?? a adapta????o de conhecimentos e aprendizagem, com vistas a melhorar a implementa????o de pol??ticas e programas sociais. Para tanto, a CSPS e a ENAP introduzir??o novos cursos nos curr??culos das escolas parceiras e incorporar??o novos m??todos e tecnologias de aprendizagem como, por exemplo, comunidades de pr??tica virtuais e um componente de tutoria (mentoring) envolvendo o Human Resources and Skills Development Canada e o Minist??rio do Desenvolvimento Social e Combate ?? Fome do Brasil. Seis institui????es da Rede Nacional de Escolas de Governo do Brasil e do Programa de Parceria da ENAP foram selecionadas e convidadas a se unir ?? CSPS e ?? ENAP nesse novo Projeto: a Universidade Federal do Par?? (UFPA), de Bel??m (estado do Par?? ??? regi??o Norte); a Funda????o Joaquim Nabuco (FUNDAJ), de Recife (Pernambuco ??? Nordeste); a Universidade Corporativa do Servi??o P??blico / Secretaria de Administra????o do Estado da Bahia (UCS/SAEB), Salvador (Bahia ??? Nordeste); a Escola de Governo do Mato Grosso do Sul (ESCOLAGOV), Campo Grande (estado do Mato Grosso do Sul ??? Centro-Oeste); a Escola Nacional de Ci??ncias Estat??sticas / Instituto Brasileiro de Geografia e Estat??stica (ENCE/IBGE), Rio de Janeiro (estado do Rio de Janeiro ??? Sudeste); e o Instituto Municipal de Administra????o P??blica (IMAP) de Curitiba (Paran?? ??? Sul). Essas escolas de refer??ncia foram escolhidas segundo sua capacidade de trabalhar como p??los de pr??ticas inovadoras em pol??ticas p??blicas e disseminar os benef??cios do Projeto para outras escolas em suas regi??es, por meio da Rede Nacional coordenada pela ENAP. O objetivo dessa parceria ?? fortalecer as escolas de governo locais, para que estas desenvolvam, por meio de eventos de aprendizagem, compet??ncias em servidores p??blicos, a fim de aumentar a capacidade do governo na implementa????o e gest??o de pol??ticas p??blicas. O Plano de Implementa????o do Projeto (PIP) descreve o trabalho a ser realizado por essas institui????es nos pr??ximos 30 meses, ao tempo em que serve de guia para os Parceiros do Projeto no que se refere ??s a????es e aos recursos necess??rios para a obten????o dos resultados acordados. Na medida em que o Projeto estiver em andamento e os parceiros iniciarem um interc??mbio produtivo de conhecimentos, o Plano de Trabalho Anual ser?? atualizado e revisto por meio de reuni??es anuais de avalia????o e encontros do Comit?? Diretor do Projeto, com vistas a assegurar que os resultados descritos no PIP sejam alcan??ados com sucesso

Measuring data transfer in heterogeneous IoT environments

Dissertação para obtenção do Grau de Mestre em Engenharia Electrotécnica e de Computadores

Knowledge repository framework for crowdsourcing innovation intermediary: a proposal

A crowdsourcing innovation intermediary performs mediation activities between companies that have a problem to solve or that seek a business opportunity, and a group of people motivated to present ideas based on their knowledge, experience and wisdom, taking advantage of technology sharing and collaboration emerging from Web2.0. As far as we know, most of the present intermediaries don´t have, yet, an integrated vision that combines the creation of value through community development, brokering and technology transfer. In this paper we present a proposal of a knowledge repository framework for crowdsourcing innovation that enables effective support and integration of the activities developed in the process of value creation (community building, brokering and technology transfer), modeled using ontology engineering methods.

High-Content Analysis of Breast Cancer Using Single-Cell Deep Transfer Learning

High-content analysis has revolutionized cancer drug discovery by identifying substances that alter the phenotype of a cell, which prevents tumor growth and metastasis. The high-resolution biofluorescence images from assays allow precise quantitative measures enabling the distinction of small molecules of a host cell from a tumor. In this work, we are particularly interested in the application of deep neural networks (DNNs), a cutting-edge machine learning method, to the classification of compounds in chemical mechanisms of action (MOAs). Compound classification has been performed using image-based profiling methods sometimes combined with feature reduction methods such as principal component analysis or factor analysis. In this article, we map the input features of each cell to a particular MOA class without using any treatment-level profiles or feature reduction methods. To the best of our knowledge, this is the first application of DNN in this domain, leveraging single-cell information. Furthermore, we use deep transfer learning (DTL) to alleviate the intensive and computational demanding effort of searching the huge parameter's space of a DNN. Results show that using this approach, we obtain a 30% speedup and a 2% accuracy improvement.

The Impact of Higher Education Institution-Firm Knowledge Links on Firm-level Productivity in Britain

This paper estimates whether both sourcing knowledge from and/or cooperating on innovation with HEIs (Higher Education Institutions)1 impacts on establishment-level total factor productivity (TFP) using a dataset created by merging the UK government’s Community Innovation Survey (CIS) with the Annual Respondents Database (ARD). It also considers whether higher graduate employment (as a measure of human capital) also impacts positively on TFP at the establishment-level. Many studies have investigated the relationship between university-firm knowledge links and innovation (see, for example, Mansfield, 1991; Becker, 2003; Thorn et al, 2007). Most of these studies find a positive impact. Fewer studies have investigated the impact of university-firm knowledge links on productivity. Belderbos et al. (2004), using the Dutch CIS, find that cooperation with universities has no statistically significant impact on the growth of labour productivity. Medda et al. (2005) find no statistically significant effect of collaborative research undertaken by Italian manufacturing firms and universities on the growth of TFP. Arvanitis et al. (2008), using Swiss data, show that university-firm knowledge and technology transfer has both a direct impact on labour productivity and an indirect impact through its positive impact on innovation. In sum, there is as yet no clear consensus as to the impact of university-firm knowledge links on productivity.

Regulation of horizontal gene transfer of the catabolic island ICEclc

ICEclc is a mobile genetic element found in two copies on the chromosome of the bacterium Pseudomonas knackmussii B13. ICEclc harbors genes encoding metabolic pathways for the degradation of chlorocatechols (CLC) and 2-aminophenol (2AP). At low frequencies, ICEclc excises from the chromosome, closes into a circular DNA molecule which can transfer to another bacterium via conjugation. Once in the recipient cell, ICEclc can reintegrate into the chromosome by site-specific recombination. This thesis aimed at identifying the regulatory network underlying the decisions for ICEclc horizontal transfer (HGT). The first chapter is an introduction on integrative and conjugative elements (ICEs) more in general, of which ICEclc is one example. In particular I emphasized the current knowledge of regulation and conjugation machineries of the different classes of ICE. In the second chapter, I describe a transcriptional analysis using microarrays and other experiments to understand expression of ICEclc in exponential and stationary phase. By overlaying transcriptomic profiles with Northern hybridizations and RT- PCR data, we established a transcription map for the entire core region of ICEclc, a region assumed to encode the ICE conjugation process. We also demonstrated how transcription of the ICEclc core is maximal in stationary phase, which correlates to expression of reporter genes fused to key ICEclc promoters. In the third chapter, I present a transcriptome analysis of ICEclc in a variety of different host species, in order to explore whether there are species-specific differences. In the fourth chapter, I focus on the role of a curious ICEclc-encoded TetR-type transcriptional repressor. We find that this gene, which we name mfsR, not only controls its own expression but that of a set of genes for a putative multi-drug efflux pump (mfsABC) as well. By using a combination of biochemical and molecular biology techniques, I could show that MfsR specifically binds to operator boxes in two ICEclc promoters (PmfsR and PmfsA), inhibiting the transcription of both the mfsR and mfsABC-orf38184 operons. Although we could not detect a clear phenotype of an mfsABC deletion, we discuss the implications of pump gene reorganizations in ICEclc and close relatives. In the fifth chapter, we find that mfsR not only controls its own expression and that of the mfsABC operon, but is also indirectly controlling ICEclc transfer. Using gene deletions, microarrays, transfer assays and microscopy-based reporter fusions, we demonstrate that mfsR actually controls a small operon of three regulatory genes. The last gene of this mfsR operon, orf17162, encodes a LysR-type activator that when deleted strongly impairs ICEclc transfer. Interestingly, deletion of mfsR leads to transfer competence in almost all cells, thereby overruling the bistability process in the wild-type. In the final sixth chapter, I discuss the relevance of the present thesis and the resulting perspectives for future studies.