Vocabulary Transfer and Knowledge Distillation for Language Model Compression

Natural Language Processing (NLP) has seen tremendous improvements over the last few years. Transformer architectures achieved impressive results in almost any NLP task, such as Text Classification, Machine Translation, and Language Generation. As time went by, transformers continued to improve thanks to larger corpora and bigger networks, reaching hundreds of billions of parameters. Training and deploying such large models has become prohibitively expensive, such that only big high tech companies can afford to train those models. Therefore, a lot of research has been dedicated to reducing a model’s size. In this thesis, we investigate the effects of Vocabulary Transfer and Knowledge Distillation for compressing large Language Models. The goal is to combine these two methodologies to further compress models without significant loss of performance. In particular, we designed different combination strategies and conducted a series of experiments on different vertical domains (medical, legal, news) and downstream tasks (Text Classification and Named Entity Recognition). Four different methods involving Vocabulary Transfer (VIPI) with and without a Masked Language Modelling (MLM) step and with and without Knowledge Distillation are compared against a baseline that assigns random vectors to new elements of the vocabulary. Results indicate that VIPI effectively transfers information of the original vocabulary and that MLM is beneficial. It is also noted that both vocabulary transfer and knowledge distillation are orthogonal to one another and may be applied jointly. The application of knowledge distillation first before subsequently applying vocabulary transfer is recommended. Finally, model performance due to vocabulary transfer does not always show a consistent trend as the vocabulary size is reduced. Hence, the choice of vocabulary size should be empirically selected by evaluation on the downstream task similar to hyperparameter tuning.

Distinção entre cachaças destiladas em alambiques e em colunas usando quimiometria

One hundred fifteen cachaça samples derived from distillation in copper stills (73) or in stainless steels (42) were analyzed for thirty five itens by chromatography and inductively coupled plasma optical emission spectrometry. The analytical data were treated through Factor Analysis (FA), Partial Least Square Discriminant Analysis (PLS-DA) and Quadratic Discriminant Analysis (QDA). The FA explained 66.0% of the database variance. PLS-DA showed that it is possible to distinguish between the two groups of cachaças with 52.8% of the database variance. QDA was used to build up a classification model using acetaldehyde, ethyl carbamate, isobutyl alcohol, benzaldehyde, acetic acid and formaldehyde as chemical descriptors. The model presented 91.7% of accuracy on predicting the apparatus in which unknown samples were distilled.

Teores de carbamato de etila em aguardentes de cana e mandioca: parte II

Herein, we report the concentration of ethyl carbamate (EC) and copper in 380 samples of sugar-cane spirit and 45 samples of manioc spirit as determined by GC-MS and FAAS respectively. The cyanide content determined spectrophotometrically is reported for the manioc spirit. Sugar cane spirit produced by alembic distillation (70,0 µg L-1) shown a lower content of EC than samples produced by column distillation (270 µg L-1). No simple correlation between the content of EC and copper for sugar cane spirit as well among the concentration of EC, copper, and cyanide for manioc spirit could be observed.

Formação de carbamato de etila em aguardentes recém-destiladas: proposta para seu controle

The commercial sugar cane spits redistillation decreased up to 92,5% their ethyl carbamate (EC) original content. Quantitative analysis of EC in 15 samples of sugar cane spirit (alembic and column), fresh distilled and collected in situ demonstrated that the urethane is formed mostly after distillation. The average time to achieve the complete EC formation is independent of the diffuse light presence and of the distillation apparatus used. The k obs for urethane formation at 25 ºC was calculate as (3,3 ± 0,5) x 10-5/s and the activation parameters are: ΔH‡ 34 kcal/mol; ΔS‡ - 69 cal/mol K; and ΔG‡ 54 kcal/mol.

Ácidos orgânicos em aguardentes produzidas em alambique e em coluna

The concentration of 14 organic acids of 50 sugarcane spirits samples was determined by gas chromatography using flame ionization detection. The organic acids analytical quantitative profile in stills and column distilled spirits from wines obtained from the same must were compared. The comparison was also carried in "head", "heart" and "tail fractions of stills distilled spirits. The experimental data were analyzed by Principal Components Analysis (PCA) and pointed out that the distillation process (stills and column) strongly influences the lead spirits' organic acid composition and that producers' operational "cuts off" to produce "tail", "heart" and "head", fractions should be optimized.

Produção de óleo essencial e balanço nutricional em Corymbia citriodora adubado com lodo de esgoto em diferentes espaçamentos

The Corymbia citriodora is one of the most cultivated species in Brazil for wood and essential oil production. The oil is obtained by the leaves distillation and the yield is around 2%. For the sustainability of leaves extraction is necessary a good nutritional replacement, because the leaves contain a large quantity of nutrients. The sewage sludge application in Corymbia plantations is a way to apply organic fertilizer and is also a good alternative for the final disposal of this residue. In this context, the objective of this study was to evaluate the sewage sludge use as fertilizer for the essential oil production in two different Spacing. The experiment was constituted by 4 blocks and the treatments were: i) Close spacing (1m x 1m) with sewage sludge + K and B; ii) Close spacing with mineral fertilization: iii) Regular spacing (3m x 1m) with sewage sludge + K and B; and iv) Regular spacing with mineral fertilization. When the C. citriodora plots were 12, 18, 24 and 30 months old, leaves and branches biomass and also essential oil production were quantified. It was observed that the system of close planting totalized a higher biomass, 3.8 times more than the conventional system, providing 3 times more essential oil production (363 l ha(-1) at the close spacing and 116 l ha(-1) at the regular Spacing during 30 months). No difference was found in the leaves or oil production with the use of sewage sludge or mineral fertilizer any of spacing studied The use of sewage sludge as fertilizer for Corymbia citriodora plantations may be an interesting alternative. because it supplies large quantities of nutrients and ensures a positive budget of nutrients in the ecosystem.

INFLUENCE OF ALTITUDE, AGE AND DIAMETER ON YIELD AND ALPHA-BISABOLOL CONTENT OF CANDEIA TREES (Eremanthus erythropappus)

The heartwood of candeia tree is a source of essential oil rich in alpha-bisabolol, a substance widely used in the cosmetic and pharmaceutical industry. Bearing in mind the economic importance of alpha-bisabolol, this work aimed to evaluate the influence of tree age on the yield and content of alpha-bisabolol present in essential oil from candeia, considering two distinct reliefs and three diameter classes, in Aiuruoca region, south Minas Gerais state. The two distinct reliefs correspond respectively to one section of the stand growing at 1,000m of altitude (Area 1) and another section growing at 1,100m of altitude (Area 2). In each section, 15 trees were felled from among 3 different diameter classes. Discs were removed from the base of each tree to estimate their age by doing growth ring count. Soil samples were taken and Subjected to physical and chemical analysis. The logs were reduced into chips and random samples were taken for distillation to extract essential oil. The method used was steam distillation at a pressure of 2 kgf/cm(2)/2.5 h. The chemical analysis was performed in a gas chromatograph (GC) based on the alpha-bisabolol standard reference. The yield of essential oil from trees in Area I was higher than that from trees in Area 2, with the same pattern of influence for older trees. In Area 2, the alpha-bisabolol content was higher in younger trees. No differences were found between the relevant parameters in relation to diameter classes.

A Simple and Fast Method for the Production and Characterization of Methylic and Ethylic Biodiesels from Tucum Oil via an Alkaline Route

A simple, fast, and complete route for the production of methylic and ethylic biodiesel from tucum oil is described. Aliquots of the oil obtained directly from pressed tucum (pulp and almonds) were treated with potassium methoxide or ethoxide at 40 degrees C for 40 min. The biodiesel form was removed from the reactor and washed with 0.1 M HCl aqueous solution. A simple distillation at 100 degrees C was carried out in order to remove water and alcohol species from the biodiesel. The oxidative stability index was obtained for the tucum oil as well as the methylic and ethylic biodiesel at 6.13, 2.90, and 2.80 h, for storage times higher than 8 days. Quality control of the original oil and of the methylic and ethylic biodiesels, such as the amount of glycerin produced during the transesterification process, was accomplished by the TLC, GC-MS, and FT-IR techniques. The results obtained in this study indicate a potential biofuel production by simple treatment of tucum, an important Amazonian fruit.

Ethanol production process from banana fruit and its lignocellulosic residues: Energy analysis

Tropical countries, such as Brazil and Colombia, have the possibility of using agricultural lands for growing biomass to produce bio-fuels such as biodiesel and ethanol. This study applies an energy analysis to the production process of anhydrous ethanol obtained from the hydrolysis of starch and cellulosic and hemicellulosic material present in the banana fruit and its residual biomass. Four different production routes were analyzed: acid hydrolysis of amylaceous material (banana pulp and banana fruit) and enzymatic hydrolysis of lignocellulosic material (flower stalk and banana skin). The analysis considered banana plant cultivation, feedstock transport, hydrolysis, fermentation, distillation, dehydration, residue treatment and utility plant. The best indexes were obtained for amylaceous material for which mass performance varied from 346.5 L/t to 388.7 L/t, Net Energy Value (NEV) ranged from 9.86 MJ/L to 9.94 MJ/L and the energy ratio was 1.9 MJ/MJ. For lignocellulosic materials, the figures were less favorable: mass performance varied from 86.1 to 123.5 L/t, NEV from 5.24 10 8.79 MJ/L and energy ratio from 1.3 to 1.6 MJ/MJ. The analysis showed, however, that both processes can be considered energetically feasible. (C) 2010 Elsevier Ltd. All rights reserved.

Energy and exergy analysis of ethanol production process from banana fruit

An alternative for ethanol production, is the use of vegetable waste, such as excess of banana production, that are evaluated in 2,400,000 t/year, which includes: residual banana fruit and lignocellulosic material. This paper analyzes the energetic and exergetic behavior to carry the process developed at laboratory scale to a plant processing of banana for the ethanol production, involving: growing and transport of the vegetable material, hydrolysis of banana fruit, sugar fermentation, ethanol distillation and utility plant. Finally, energy and exergy indicators are obtained. The results show a positive energy balance when banana fruit is used for ethanol production, but some process modification must be done looking for improving the exergetic efficiency in ethanol production.

MPC for tracking zone regions

This paper deals with the problem of tracking target sets using a model predictive control (MPC) law. Some MPC applications require a control strategy in which some system outputs are controlled within specified ranges or zones (zone control), while some other variables - possibly including input variables - are steered to fixed target or set-point. In real applications, this problem is often overcome by including and excluding an appropriate penalization for the output errors in the control cost function. In this way, throughout the continuous operation of the process, the control system keeps switching from one controller to another, and even if a stabilizing control law is developed for each of the control configurations, switching among stable controllers not necessarily produces a stable closed loop system. From a theoretical point of view, the control objective of this kind of problem can be seen as a target set (in the output space) instead of a target point, since inside the zones there are no preferences between one point or another. In this work, a stable MPC formulation for constrained linear systems, with several practical properties is developed for this scenario. The concept of distance from a point to a set is exploited to propose an additional cost term, which ensures both, recursive feasibility and local optimality. The performance of the proposed strategy is illustrated by simulation of an ill-conditioned distillation column. (C) 2010 Elsevier Ltd. All rights reserved.

Stable Model Predictive Control for Integrating Systems with Optimizing Targets

This paper concern the development of a stable model predictive controller (MPC) to be integrated with real time optimization (RTO) in the control structure of a process system with stable and integrating outputs. The real time process optimizer produces Optimal targets for the system inputs and for Outputs that Should be dynamically implemented by the MPC controller. This paper is based oil a previous work (Comput. Chem. Eng. 2005, 29, 1089) where a nominally stable MPC was proposed for systems with the conventional control approach where only the outputs have set points. This work is also based oil the work of Gonzalez et at. (J. Process Control 2009, 19, 110) where the zone control of stable systems is studied. The new control for is obtained by defining ail extended control objective that includes input targets and zone controller the outputs. Additional decision variables are also defined to increase the set of feasible solutions to the control problem. The hard constraints resulting from the cancellation of the integrating modes Lit the end of the control horizon are softened,, and the resulting control problem is made feasible to a large class of unknown disturbances and changes of the optimizing targets. The methods are illustrated with the simulated application of the proposed,approaches to a distillation column of the oil refining industry.

Replacement of Trays by Packing To Increase the Absorption Capacity of Acetone during Cellulose Acetate Spinning

All textile uses of cellulose acetate involve acetone recovery, which, because of safety issues, results in large installations, in order to work with dilute streams. This compromises the efficiency of all of the involved unit operations, in this case, acetone absorption in cold water, acetone distillation, and water chilling, making them more expensive. The present article proposes the improvement of the absorption of acetone in water, traditionally performed with sieve trays, by using structured packing instead. The advantageous implementation was enabled through the utilization of a calculation methodology based on concepts of thermodynamic equilibrium of the binary acetone/water system and empirical relations that allow the evaluation of the hydrodynamics of the proposed modification.

Comparison of the chemical composition of the essential oil and the water soluble oil of Baccharis dracunculifolia DC. (Asteraceae)

This study aimed to evaluate the chemical composition of Baccharis dracunculifolia essential oil and the water soluble oil obtained by steam distillation that were analyzed by GC and GUMS. in the first hour of distillation, B. dracunculifolia aerial parts yielded 0.08% oil and in the second hour, 0.27%. The oil recovered from the distillate water yielded 0.18 g/L in the first hour and 0.44 g/L in the second hour of distillation. The main volatile compounds identified in the distillate water were aromatic compounds and sesquiterpene alcohols.

Thermal Cycling Stability of Silica Membranes for Gas Separation

Hydrogen is being seen as an alternative energy carrier to conventional hydrocarbons to reduce greenhouse gas emissions. High efficiency separation technologies to remove hydrogen from the greenhouse gas, carbon dioxide, are therefore in growing demand. Traditional thermodynamic separation systems utilise distillation, absorption and adsorption, but are limited in efficiency at compact scales. Molecular sieve silica (MSS) membranes can perform this separation as they have high permselectivity of hydrogen to carbon dioxide, but their stability under thermal cycling is not well reported. In this work we exposed a standard MSS membrane and a carbonised template MSS (CTMSS) membrane to thermal cycling from 100 to 450°C. The standard MSS and carbonised template CTMSS membranes both showed permselectivity of helium to nitrogen dropping from around 10 to 6 in the first set of cycles, remaining stable until the last test. The permselectivity drop was due to small micropore collapse, which occurred via structure movement during cycling. Simulating single stage membrane separation with a 50:50 molar feed of H2:CO2, H2 exiting the permeate stream would start at 79% and stabilise at 67%. Higher selectivity membranes showed less of a purity drop, indicating the margin at which to design a stable membrane separation unit for CO2 capture.