Vocabulary Transfer and Knowledge Distillation for Language Model Compression

Natural Language Processing (NLP) has seen tremendous improvements over the last few years. Transformer architectures achieved impressive results in almost any NLP task, such as Text Classification, Machine Translation, and Language Generation. As time went by, transformers continued to improve thanks to larger corpora and bigger networks, reaching hundreds of billions of parameters. Training and deploying such large models has become prohibitively expensive, such that only big high tech companies can afford to train those models. Therefore, a lot of research has been dedicated to reducing a model’s size. In this thesis, we investigate the effects of Vocabulary Transfer and Knowledge Distillation for compressing large Language Models. The goal is to combine these two methodologies to further compress models without significant loss of performance. In particular, we designed different combination strategies and conducted a series of experiments on different vertical domains (medical, legal, news) and downstream tasks (Text Classification and Named Entity Recognition). Four different methods involving Vocabulary Transfer (VIPI) with and without a Masked Language Modelling (MLM) step and with and without Knowledge Distillation are compared against a baseline that assigns random vectors to new elements of the vocabulary. Results indicate that VIPI effectively transfers information of the original vocabulary and that MLM is beneficial. It is also noted that both vocabulary transfer and knowledge distillation are orthogonal to one another and may be applied jointly. The application of knowledge distillation first before subsequently applying vocabulary transfer is recommended. Finally, model performance due to vocabulary transfer does not always show a consistent trend as the vocabulary size is reduced. Hence, the choice of vocabulary size should be empirically selected by evaluation on the downstream task similar to hyperparameter tuning.

Communicative and psycholinguistic abilities in children with phenylketonuria and congenital hypothyroidism

The Neonatal Screening for Inborn Errors of Metabolism of the Association of Parents and Friends of Special Needs Individuals (APAE) - Bauru, Brazil, was implanted and accredited by the Brazilian Ministry of Health in 1998. It covers about 286 cities of the Bauru region and 420 collection spots. Their activities include screening, diagnosis, treatment and assistance to congenital hypothyroidism (CH) and phenylketonuria (PKU), among others. In 2005, a partnership was established with the Department of Speech-Language Pathology and Audiology, Bauru School of Dentistry, University of São Paulo, Bauru, seeking to characterize and to follow, by means of research studies, the development of the communicative abilities of children with CH and PKU. OBJECTIVE: The aim of this study was to describe communicative and psycholinguistic abilities in children with CH and PKU. MATERIALS AND METHODS: Sixty-eight children (25 children aged 1 to 120 months with PKU and 43 children aged 1 to 60 months with CH) participated in the study. The handbooks were analyzed and different instruments were applied (Observation of Communication Behavior, Early Language Milestone Scale, Peabody Picture Vocabulary Test, Gesell & Amatruda's Behavioral Development Scale, Portage Operation Inventory, Language Development Evaluation Scale, Denver Developmental Screening Test, ABFW Child Language Test-phonology and Illinois Test of Psycholinguistic Abilities), according to the children's age group and developmental level. RESULTS: It was observed that the children with PKU and CH at risk for alterations in their developmental abilities (motor, cognitive, linguistic, adaptive and personal-social), mainly in the first years of life. Alterations in the psycholinguistic abilities were also found, mainly after the preschool age. Attention deficits, language and cognitive alterations were more often observed in children with CH, while attention deficits with hyperactivity and alterations in the personal-social, language and motor adaptive abilities were more frequent in children with PKU. CONCLUSION: CH and PKU can cause communicative and psycholinguistic alterations that compromise the communication and affect the social integration and learning of these individuals, proving the need of having these abilities assisted by a speech and language pathologist.

Magnetic properties of Fe(3)O(4) nanoparticles coated with oleic and dodecanoic acids

Magnetic nanoparticles (NP) of magnetite (Fe(3)O(4)) coated with oleic acid (OA) and dodecanoic acid (DA) were synthesized and investigated through transmission electron microscopy (TEM), magnetization M, and ac magnetic susceptibility measurements. The OA coated samples were produced with different magnetic concentrations (78%, 76%, and 65%) and the DA sample with 63% of Fe(3)O(4). Images from TEM indicate that the NP have a nearly spherical geometry and mean diameter similar to 5.5 nm. Magnetization measurements, performed in zero-field cooled (ZFC) and field cooled processes under different external magnetic fields H, exhibited a maximum at a given temperature T(B) in the ZFC curves, which depends on the NP coating (OA or DA), magnetite concentration, and H. The temperature T(B) decreases monotonically with increasing H and, for a given H, the increase in the magnetite concentration results in an increase in T(B). The observed behavior is related to the dipolar interaction between NP, which seems to be an important mechanism in all samples studied. This is supported by the results of the ac magnetic susceptibility chi(ac) measurements, where the temperature in which chi' peaks for different frequencies follows the Vogel-Fulcher model, a feature commonly found in systems with dipolar interactions. Curves of H versus T(B)/T(B) (H=0) for samples with different coatings and magnetite concentrations collapse into a universal curve, indicating that the qualitative magnetic behavior of the samples may be described by the NP themselves, instead of the coating or the strength of the dipolar interaction. Below T(B), M versus H curves show a coercive field (H(C)) that increases monotonically with decreasing temperature. The saturation magnetization (M(S)) follows the Bloch's law and values of M(S) at room temperature as high as 78 emu/g were estimated, a result corresponding to similar to 80% of the bulk value. The overlap of M/M(S) versus H/T curves for a given sample and the low H(C) at high temperatures suggest superparamagnetic behavior in all samples studied. The overlap of M/M(S) versus H curves at constant temperature for different samples indicates that the NP magnetization behavior is preserved, independently of the coating and magnetite concentration. (C) 2010 American Institute of Physics. [doi: 10.1063/1.3311611]

Identification and Validation of Brazilian Chronic Pain Descriptors

Appropriate pain assessment is very important for managing chronic pain. Given the cultural differences in verbally expressing pain and in psychosocial problems, specific tools are needed. The goal of this study was to identify and validate Brazilian pain descriptors. A purposive sample of health professionals and chronic pain patients was recruited. Four studies were conducted using direct and indirect psychophysical methods: category estimation, magnitude estimation, and magnitude estimation and tine-length. Results showed the descriptors which best describe chronic pain in Brazilian culture and demonstrated that there is not a significant correlation between patients and health professionals and that the psychophysical scale of judgment of pain descriptors is valid, stable, and consistent. Results reinforced that the translations of word descriptors and research tools into another language may be inappropriate, owing to differences in perception and communication and the inadequacy of exact translations to reflect the intended meaning. Given the complexity of the chronic pain, personal suffering involved, and the need for accurate assessment of chronic pain using descriptors stemming from Brazilian culture and language, it is essential to investigate the most adequate words to describe chronic pain. Although it requires more refinement, the Brazilian chronic pain descriptors can be used further to develop a multidimensional pain assessment tool that is culturally sensitive. (C) 2009 by the American Society for Pain Management Nursing

Role of price and enforcement in water allocation: Insights from Game Theory

As many countries are moving toward water sector reforms, practical issues of how water management institutions can better effect allocation, regulation, and enforcement of water rights have emerged. The problem of nonavailability of water to tailenders on an irrigation system in developing countries, due to unlicensed upstream diversions is well documented. The reliability of access or equivalently the uncertainty associated with water availability at their diversion point becomes a parameter that is likely to influence the application by users for water licenses, as well as their willingness to pay for licensed use. The ability of a water agency to reduce this uncertainty through effective water rights enforcement is related to the fiscal ability of the agency to monitor and enforce licensed use. In this paper, this interplay across the users and the agency is explored, considering the hydraulic structure or sequence of water use and parameters that define the users and the agency`s economics. The potential for free rider behavior by the users, as well as their proposals for licensed use are derived conditional on this setting. The analyses presented are developed in the framework of the theory of ""Law and Economics,`` with user interactions modeled as a game theoretic enterprise. The state of Ceara, Brazil, is used loosely as an example setting, with parameter values for the experiments indexed to be approximately those relevant for current decisions. The potential for using the ideas in participatory decision making is discussed. This paper is an initial attempt to develop a conceptual framework for analyzing such situations but with a focus on the reservoir-canal system water rights enforcement.

On the solubility of proteins as a function of pH: Mathematical development and application

Thermodynamic relations between the solubility of a protein and the solution pH are presented in this work. The hypotheses behind the development are that the protein chemical potential in liquid phase can be described by Henry`s law and that the solid-liquid equilibrium is established only between neutral molecules. The mathematical development results in an analytical expression of the solubility curve, as a function of the ionization equilibrium constants, the pH and the solubility at the isoelectric point. It is shown that the same equation can be obtained either by directly calculating the fraction of neutral protein molecules or by integrating the curve of the protein average charge. The methodology was successfully applied to the description of the solubility of porcine insulin as a function of pH at three different temperatures and of bovine beta-lactoglobulin at four different ionic strengths. (C) 2011 Elsevier B.V. All rights reserved.

Using Reverberation to Improve Range and Elevation Discrimination for Small Array Sound Source Localization

Sound source localization (SSL) is an essential task in many applications involving speech capture and enhancement. As such, speaker localization with microphone arrays has received significant research attention. Nevertheless, existing SSL algorithms for small arrays still have two significant limitations: lack of range resolution, and accuracy degradation with increasing reverberation. The latter is natural and expected, given that strong reflections can have amplitudes similar to that of the direct signal, but different directions of arrival. Therefore, correctly modeling the room and compensating for the reflections should reduce the degradation due to reverberation. In this paper, we show a stronger result. If modeled correctly, early reflections can be used to provide more information about the source location than would have been available in an anechoic scenario. The modeling not only compensates for the reverberation, but also significantly increases resolution for range and elevation. Thus, we show that under certain conditions and limitations, reverberation can be used to improve SSL performance. Prior attempts to compensate for reverberation tried to model the room impulse response (RIR). However, RIRs change quickly with speaker position, and are nearly impossible to track accurately. Instead, we build a 3-D model of the room, which we use to predict early reflections, which are then incorporated into the SSL estimation. Simulation results with real and synthetic data show that even a simplistic room model is sufficient to produce significant improvements in range and elevation estimation, tasks which would be very difficult when relying only on direct path signal components.

Environmental, land-use and economic implications of Brazilian sugarcane expansion 1996-2006

Governments are promoting biofuels and the resulting changes in land use and crop reallocation to biofuels production have raised concerns about impacts on environment and food security. The promotion of biofuels has also been questioned based on suggested marginal contribution to greenhouse gas emissions reduction, partly due to induced land use change causing greenhouse gas emissions. This study reports how the expansion of sugarcane in Brazil during 1996-2006 affected indicators for environment, land use and economy. The results indicate that sugarcane expansion did not in general contribute to direct deforestation in the traditional agricultural region where most of the expansion took place. The amount of forests on farmland in this area is below the minimum stated in law and the situation did not change over the studied period. Sugarcane expansion resulted in a significant reduction of pastures and cattle heads and higher economic growth than in neighboring areas. It could not be established to what extent the discontinuation of cattle production induced expansion of pastures in other areas, possibly leading to indirect deforestation. However, the results indicate that a possible migration of the cattle production reached further than the neighboring of expansion regions. Occurring at much smaller rates, expansion of sugarcane in regions such as the Amazon and the Northeast region was related to direct deforestation and competition with food crops, and appear not to have induced economic growth. These regions are not expected to experience substantial increases of sugarcane in the near future, but mitigating measures are warranted.

Morphosyntactic and syntactic priming: an investigation of underlying processing mechanisms and the effects of Parkinson’s disease

There is now considerable evidence to suggest that non-demented people with Parkinson's disease (PD) experience difficulties using the morphosyntactic aspects of language. It remains unclear, however, at precisely which point in the processing of morphosyntax, these difficulties emerge. The major objective of the present study was to examine the impact of PD on the processes involved in accessing morphosyntactic information in the lexicon. Nineteen people with PD and 19 matched control subjects participated in the study which employed on-line word recognition tasks to examine morphosyntactic priming for local grammatical dependencies that occur both within (e.g. is going) and across (e.g. she gives) phrasal boundaries (Experiments 1 and 2, respectively). The control group evidenced robust morphosyntactic priming effects that were consistent with the involvement of both pre- (Experiment 1) and post-lexical (Experiment 2) processing routines. Whilst the participants with PD also recorded priming for dependencies within phrasal boundaries (Experiment 1), priming effects were observed over an abnormally brief time course. Further, in contrast to the controls, the PD group failed to record morphosyntactic priming for constructions that crossed phrasal boundaries (Experiment 2). The results demonstrate that attentionally mediated mechanisms operating at both the pre- and post-lexical stages of processing are able to contribute to morphosyntactic priming effects. In addition, the findings support the notion that, whilst people with PD are able to access morphosyntactic information in a normal manner, the time frame in which this information remains available for processing is altered. Deficits may also be experienced at the post-lexical integrational stage of processing.

Hemispheric contributions to lexical ambiguity resolution: Evidence from individuals with complex language impairment following left-hemisphere lesions

Nine individuals with complex language deficits following left-hemisphere cortical lesions and a matched control group (n 5 9) performed speeded lexical decisions on the third word of auditory word triplets containing a lexical ambiguity. The critical conditions were concordant (e.g., coin–bank–money), discordant (e.g., river–bank–money), neutral (e.g., day–bank– money), and unrelated (e.g., river–day–money). Triplets were presented with an interstimulus interval (ISI) of 100 and 1250 ms. Overall, the left-hemisphere-damaged subjects appeared able to exhaustively access meanings for lexical ambiguities rapidly, but were unable to reduce the level of activation for contextually inappropriate meanings at both short and long ISIs, unlike control subjects. These findings are consistent with a disruption of the proposed role of the left hemisphere in selecting and suppressing meanings via contextual integration and a sparing of the right-hemisphere mechanisms responsible for maintaining alternative meanings.

Discourse priming of homophones in individuals with dominant subcortical lesions, cortical lesions, and Parkinson’s disease

An on-line priming experiment was used to investigate discourse-level processing in four matched groups of subjects: individuals with nonthalamic subcortical lesions (NSL) ( n =10), normal control subjects ( n =10), subjects with Parkinsons disease (PD) ( n =10), and subjects with cortical lesions ( n =10). Subjects listened to paragraphs that ended in lexical ambiguities, and then made speeded lexical decisions on visual letter strings that were: nonwords, matched control words, contextually appropriate associates of the lexical ambiguity, contextually inappropriate associates of the ambiguity, and inferences (representing information which could be drawn from the paragraphs but was not explicitly stated). Targets were presented at an interstimulus interval (ISI) of 0 or 1000ms. NSL and PD subjects demonstrated priming for appropriate and inappropriate associates at the short ISI, similar to control subjects and cortical lesion subjects, but were unable to demonstrate selective priming of the appropriate associate and inference words at the long ISI. These results imply intact automatic lexical processing and a breakdown in discourse-based meaning selection and inference development via attentional/strategic mechanisms.

One size doesn't fit all: Research methodologies in a language variation study of Sudanese teens

The research reported here draws on a study of five teenagers from a Dinka-speaking community of Sudanese settling in Australia. A range of factors including language proficiency, social network structure and language attitudes are examined as possible causes for the variability of language use. The results and discussion illustrate how the use of a triangular research approach captured the complexity of the participants' language situation and was critical to developing a full understanding of the interplay of factors influencing the teens' language maintenance and shift in a way that no single method could. Further, it shows that employment of different methodologies allowed for flexibility in data collection to ensure the fullest response from participants. Overall, this research suggests that for studies of non-standard communities, variability in research methods may prove more of a strength that the use of standardised instruments and approaches.

Parameter setting and statistical learning

Three main models of parameter setting have been proposed: the Variational model proposed by Yang (2002; 2004), the Structured Acquisition model endorsed by Baker (2001; 2005), and the Very Early Parameter Setting (VEPS) model advanced by Wexler (1998). The VEPS model contends that parameters are set early. The Variational model supposes that children employ statistical learning mechanisms to decide among competing parameter values, so this model anticipates delays in parameter setting when critical input is sparse, and gradual setting of parameters. On the Structured Acquisition model, delays occur because parameters form a hierarchy, with higher-level parameters set before lower-level parameters. Assuming that children freely choose the initial value, children sometimes will miss-set parameters. However when that happens, the input is expected to trigger a precipitous rise in one parameter value and a corresponding decline in the other value. We will point to the kind of child language data that is needed in order to adjudicate among these competing models.

Demonstrators with flags and banners during Moratorium in Brisbane 1970

Demonstrators in Brisbane, Australia with banners and flags during Moratorium march 1970. Anne Berquier can be seen fourth from the left.

Demonstrators and Semper press car during Moratorium in Brisbane 1970

Demonstrators in Brisbane, Australia with banners and flags during Moratorium march 1970. Semper press car (Volkswagon Kombi) can be seen. Protestors are outside the Presbyterian Church of Queensland (now Uniting Church) in Ann Street, Brisbane. In the background, scaffolding can be seen at the construction site of the Crest Hotel. Semper Floreat is the student newspaper of the University of Queensland.