Repairs of databases with null values

Dissertação para obtenção do Grau de Mestre em Engenharia Informática

Scaling In-Memory databases on multicores

Current computer systems have evolved from featuring only a single processing unit and limited RAM, in the order of kilobytes or few megabytes, to include several multicore processors, o↵ering in the order of several tens of concurrent execution contexts, and have main memory in the order of several tens to hundreds of gigabytes. This allows to keep all data of many applications in the main memory, leading to the development of inmemory databases. Compared to disk-backed databases, in-memory databases (IMDBs) are expected to provide better performance by incurring in less I/O overhead. In this dissertation, we present a scalability study of two general purpose IMDBs on multicore systems. The results show that current general purpose IMDBs do not scale on multicores, due to contention among threads running concurrent transactions. In this work, we explore di↵erent direction to overcome the scalability issues of IMDBs in multicores, while enforcing strong isolation semantics. First, we present a solution that requires no modification to either database systems or to the applications, called MacroDB. MacroDB replicates the database among several engines, using a master-slave replication scheme, where update transactions execute on the master, while read-only transactions execute on slaves. This reduces contention, allowing MacroDB to o↵er scalable performance under read-only workloads, while updateintensive workloads su↵er from performance loss, when compared to the standalone engine. Second, we delve into the database engine and identify the concurrency control mechanism used by the storage sub-component as a scalability bottleneck. We then propose a new locking scheme that allows the removal of such mechanisms from the storage sub-component. This modification o↵ers performance improvement under all workloads, when compared to the standalone engine, while scalability is limited to read-only workloads. Next we addressed the scalability limitations for update-intensive workloads, and propose the reduction of locking granularity from the table level to the attribute level. This further improved performance for intensive and moderate update workloads, at a slight cost for read-only workloads. Scalability is limited to intensive-read and read-only workloads. Finally, we investigate the impact applications have on the performance of database systems, by studying how operation order inside transactions influences the database performance. We then propose a Read before Write (RbW) interaction pattern, under which transaction perform all read operations before executing write operations. The RbW pattern allowed TPC-C to achieve scalable performance on our modified engine for all workloads. Additionally, the RbW pattern allowed our modified engine to achieve scalable performance on multicores, almost up to the total number of cores, while enforcing strong isolation.

Automatic learning for the classification of chemical reactions and in statistical thermodynamics

This Thesis describes the application of automatic learning methods for a) the classification of organic and metabolic reactions, and b) the mapping of Potential Energy Surfaces(PES). The classification of reactions was approached with two distinct methodologies: a representation of chemical reactions based on NMR data, and a representation of chemical reactions from the reaction equation based on the physico-chemical and topological features of chemical bonds. NMR-based classification of photochemical and enzymatic reactions. Photochemical and metabolic reactions were classified by Kohonen Self-Organizing Maps (Kohonen SOMs) and Random Forests (RFs) taking as input the difference between the 1H NMR spectra of the products and the reactants. The development of such a representation can be applied in automatic analysis of changes in the 1H NMR spectrum of a mixture and their interpretation in terms of the chemical reactions taking place. Examples of possible applications are the monitoring of reaction processes, evaluation of the stability of chemicals, or even the interpretation of metabonomic data. A Kohonen SOM trained with a data set of metabolic reactions catalysed by transferases was able to correctly classify 75% of an independent test set in terms of the EC number subclass. Random Forests improved the correct predictions to 79%. With photochemical reactions classified into 7 groups, an independent test set was classified with 86-93% accuracy. The data set of photochemical reactions was also used to simulate mixtures with two reactions occurring simultaneously. Kohonen SOMs and Feed-Forward Neural Networks (FFNNs) were trained to classify the reactions occurring in a mixture based on the 1H NMR spectra of the products and reactants. Kohonen SOMs allowed the correct assignment of 53-63% of the mixtures (in a test set). Counter-Propagation Neural Networks (CPNNs) gave origin to similar results. The use of supervised learning techniques allowed an improvement in the results. They were improved to 77% of correct assignments when an ensemble of ten FFNNs were used and to 80% when Random Forests were used. This study was performed with NMR data simulated from the molecular structure by the SPINUS program. In the design of one test set, simulated data was combined with experimental data. The results support the proposal of linking databases of chemical reactions to experimental or simulated NMR data for automatic classification of reactions and mixtures of reactions. Genome-scale classification of enzymatic reactions from their reaction equation. The MOLMAP descriptor relies on a Kohonen SOM that defines types of bonds on the basis of their physico-chemical and topological properties. The MOLMAP descriptor of a molecule represents the types of bonds available in that molecule. The MOLMAP descriptor of a reaction is defined as the difference between the MOLMAPs of the products and the reactants, and numerically encodes the pattern of bonds that are broken, changed, and made during a chemical reaction. The automatic perception of chemical similarities between metabolic reactions is required for a variety of applications ranging from the computer validation of classification systems, genome-scale reconstruction (or comparison) of metabolic pathways, to the classification of enzymatic mechanisms. Catalytic functions of proteins are generally described by the EC numbers that are simultaneously employed as identifiers of reactions, enzymes, and enzyme genes, thus linking metabolic and genomic information. Different methods should be available to automatically compare metabolic reactions and for the automatic assignment of EC numbers to reactions still not officially classified. In this study, the genome-scale data set of enzymatic reactions available in the KEGG database was encoded by the MOLMAP descriptors, and was submitted to Kohonen SOMs to compare the resulting map with the official EC number classification, to explore the possibility of predicting EC numbers from the reaction equation, and to assess the internal consistency of the EC classification at the class level. A general agreement with the EC classification was observed, i.e. a relationship between the similarity of MOLMAPs and the similarity of EC numbers. At the same time, MOLMAPs were able to discriminate between EC sub-subclasses. EC numbers could be assigned at the class, subclass, and sub-subclass levels with accuracies up to 92%, 80%, and 70% for independent test sets. The correspondence between chemical similarity of metabolic reactions and their MOLMAP descriptors was applied to the identification of a number of reactions mapped into the same neuron but belonging to different EC classes, which demonstrated the ability of the MOLMAP/SOM approach to verify the internal consistency of classifications in databases of metabolic reactions. RFs were also used to assign the four levels of the EC hierarchy from the reaction equation. EC numbers were correctly assigned in 95%, 90%, 85% and 86% of the cases (for independent test sets) at the class, subclass, sub-subclass and full EC number level,respectively. Experiments for the classification of reactions from the main reactants and products were performed with RFs - EC numbers were assigned at the class, subclass and sub-subclass level with accuracies of 78%, 74% and 63%, respectively. In the course of the experiments with metabolic reactions we suggested that the MOLMAP / SOM concept could be extended to the representation of other levels of metabolic information such as metabolic pathways. Following the MOLMAP idea, the pattern of neurons activated by the reactions of a metabolic pathway is a representation of the reactions involved in that pathway - a descriptor of the metabolic pathway. This reasoning enabled the comparison of different pathways, the automatic classification of pathways, and a classification of organisms based on their biochemical machinery. The three levels of classification (from bonds to metabolic pathways) allowed to map and perceive chemical similarities between metabolic pathways even for pathways of different types of metabolism and pathways that do not share similarities in terms of EC numbers. Mapping of PES by neural networks (NNs). In a first series of experiments, ensembles of Feed-Forward NNs (EnsFFNNs) and Associative Neural Networks (ASNNs) were trained to reproduce PES represented by the Lennard-Jones (LJ) analytical potential function. The accuracy of the method was assessed by comparing the results of molecular dynamics simulations (thermal, structural, and dynamic properties) obtained from the NNs-PES and from the LJ function. The results indicated that for LJ-type potentials, NNs can be trained to generate accurate PES to be used in molecular simulations. EnsFFNNs and ASNNs gave better results than single FFNNs. A remarkable ability of the NNs models to interpolate between distant curves and accurately reproduce potentials to be used in molecular simulations is shown. The purpose of the first study was to systematically analyse the accuracy of different NNs. Our main motivation, however, is reflected in the next study: the mapping of multidimensional PES by NNs to simulate, by Molecular Dynamics or Monte Carlo, the adsorption and self-assembly of solvated organic molecules on noble-metal electrodes. Indeed, for such complex and heterogeneous systems the development of suitable analytical functions that fit quantum mechanical interaction energies is a non-trivial or even impossible task. The data consisted of energy values, from Density Functional Theory (DFT) calculations, at different distances, for several molecular orientations and three electrode adsorption sites. The results indicate that NNs require a data set large enough to cover well the diversity of possible interaction sites, distances, and orientations. NNs trained with such data sets can perform equally well or even better than analytical functions. Therefore, they can be used in molecular simulations, particularly for the ethanol/Au (111) interface which is the case studied in the present Thesis. Once properly trained, the networks are able to produce, as output, any required number of energy points for accurate interpolations.

Efficient middleware for database replication

Dissertação de Mestrado em Engenharia Informática

Dpws middleware to support agent-based manufacturing control and simulation

Dissertação apresentada na Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa para obtenção do grau de Mestre em Engenharia Electrotécnica e de Computadores

Aplicações de sistemas de informação geográfica em arqueologia

Dissertação apresentada como requisito parcial para obtenção do grau de Mestre em Ciência e Sistemas de Informação Geográfica

Cooperative memory and database transactions

Dissertação apresentada na Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa para a obtenção do Grau de Mestre em Engenharia Informática

Developing Libraries Using Software Transactional Memory

Software transactional memory is a promising programming model that adapts many concepts borrowed from the databases world to control concurrent accesses to main memory (RAM) locations. This paper discusses how to support apparently irreversible operations, such as memory allocation and deallocation, within software libraries that will be used in (software memory) transactional contexts, and propose a generic and elegant approach based on a handler system, which provide the means to create and execute compensation actions at key moments during the life-time of a transaction.

Meta-analysis of gender and science research - Country report - Portugal

In Portugal, especially starting in the 1970s, women’s studies had implications on the emergency of the concept of gender and the feminist criticism to the prevailing models about differences between sexes. Until then, women had been absent from scientific research both as subject and as object. Feminism brought more reflexivity to the scientific thinking. After the 25th of April 1974, because of the consequent political openness, several innovating themes of research emerged, together with new concepts and fields of study. However, as far as gender and science relationship is concerned, such studies especially concentrate on higher education institutions. The feminist thinking seems to have two main objectives: to give women visibility, on the one hand, and to denunciate men’s domain in the several fields of knowledge. In 1977, the “Feminine Commission” is created and since then it has been publishing studies on women’s condition and contributing to the enhancement of the reflection of female condition at all levels. In the 1980s, the growing feminisation of tertiary education (both of students and academics), favoured the development of women’s studies, especially on their condition within universities with a special focus on the glass ceiling, despite the lack of statistical data by gender, thus making difficult the analysis of women integration in several sectors, namely in educational and scientific research activities. Other agglutinating themes are family, social and legal condition, work, education, and feminine intervention on political and social movements. In the 1990s, Women Studies are institutionalised in the academic context with the creation of the first Master in Women Studies in the Universidade Aberta (Open University), in Lisbon. In 1999, the first Portuguese journal of women studies is created – “Faces de Eva”. Seminars, conferences, thesis, journals, and projects on women’s studies are more and more common. However, results and publications are not so divulgated as they should be, because of lack of comprehensive and coordinated databases. 2. Analysis by topics 2.1. Horizontal and vertical segregation Research questions It is one of the main areas of research in Portugal. Essentially two issues have been considered: - The analysis of vertical gender segregation in educational and professional fields, having reflexes on women professional career progression with special attention to men’s power in control positions and the glass ceiling. - The analysis of horizontal segregation, special in higher education (teaching and research) where women have less visibility than men, and the under-representation of women in technology and technological careers. Research in this area mainly focuses on description, showing the under-representation of women in certain scientific areas and senior positions. Nevertheless, the studies that analyze horizontal segregation in the field of education adopt a more analytical approach which focuses on the analysis of the mechanisms of reproduction of gender stereotypes, especially socialisation, influencing educational and career choices. 1

Unsupervised automatic music genre classification

Trabalho apresentado no âmbito do Mestrado em Engenharia Informática, como requisito parcial para obtenção do grau de Mestre em Engenharia Informática

A Leishmaniose visceral em Portugal Continental : 1999-2009

RESUMO - As alterações climáticas alteraram a incidência e distribuição mundial de zoonoses, ao modificarem o perfil epidemiológico dos seus vectores. A leishmaniose visceral é reemergente na bacia mediterrânica, sendo o seu impacto real subestimado. Em Portugal, é endémica em três regiões, de declaração obrigatória desde 1948 e o reservatório é o canídeo. O aumento da incidência da doença no cão e a escassez de informação epidemiológica tornou pertinente investigar a realidade nacional. A partir das bases de dados das notificações e dos grupos de diagnósticos homogéneos hospitalares, foram identificados todos os casos e, consultados todos os processos clínicos dos doentes com episódios de internamento nos hospitais do continente entre 1999-2009. Ocorreram 730 internamentos para 375 indivíduos na maioria: homens, eurocaucasianos, com em média, 27 anos e, residência em Lisboa e Vale do Tejo. A sintomatologia e comorbilidades dos doentes vão de encontro ao descrito internacionalmente. A doença foi subnotificada, com uma demora média de 19 dias. A letalidade foi de 5%. A taxa de incidência média do continente foi de 0,294/100000 habitantes, sem padrão de sazonalidade. O corredor endémico de Bortman construído apresentou picos com amplitudes de 2-3 anos. O mapeamento dos doentes evidenciou casos em regiões não endémicas acompanhando a distribuição da leishmaniose canina. Seria pertinente que futuras investigações construíssem uma modelação matemática que confirmasse a tendência do corredor endémico (pico em 2011?) para accionar um sistema de alerta nos Serviços de Saúde. Seria também útil a avaliação das condições geoclimáticas das localidades com casos para evidenciar possíveis similitudes no território. -------ABSTRACT - Climate changed the incidence and worldwide distribution of zoonosis while the epidemiological profile of their vectors was changing. Visceral leishmaniasis is reemerging in the Mediterranean basin and its real impact underestimated. In Portugal, it’s endemic in three regions; the notification occurs since 1948 and dog is the reservoir. The increased incidence of the canines’ disease and the scarcity of epidemiological information relevant investigate the national reality. From Notifications and Homogeneous’ Diagnostics Groups system databases, all cases were identified and also analyze all clinical processes of inpatients’ hospitals in 1999- 2009 in Portugal. 730 admissions occurred for 375 patients. In most they were men, Caucasians, with an average of 27 years and residency in Lisboa e Vale do Tejo. The symptoms and comorbilidades patient go against described internationally. The disease was under notified, with an average delay of 19 days. Lethality was 5%. The incidence rate was 0,294/100000 inhabitants, without seasonality. The endemic’s Bortman corridor presents peak amplitudes of 2-3 years. Mapping patient’s residency shows that cases’ distribution is similar to endemic canine leishmaniasis. It would be appropriate a research to build a mathematical modeling up to confirm the trend of corridor endemic (peak in 2011?), to trigger an alert system for health services. It would also be useful to evaluate the geo-climatics conditions of localities with cases to highlight possible similarities in the territory.

A new model for EEZ surveillance and management in Portugal

Management Information Systems 2000, p. 103-111

Geospatial database generation from digital newspapers: use case for risk and disaster domains.

Dissertation submitted in partial fulfilment of the requirements for the Degree of Master of Science in Geospatial Technologies.

Artificial Intelligence in geospatial analysis: applications of self-organizing maps in the context of geographic information science.

A thesis submitted in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Information Systems.

O financiamento por capitação ajustada pelo risco em contexto de integração vertical de cuidados de saúde : a utilização dos consumos com medicamentos como proxy da carga de doença em ambulatório

RESUMO - Introdução - Com o presente projecto de investigação pretendeu-se estudar o financiamento por capitação ajustado pelo risco em contexto de integração vertical de cuidados de saúde, recorrendo particularmente a informação sobre o consumo de medicamentos em ambulatório como proxy da carga de doença. No nosso país, factores como a expansão de estruturas de oferta verticalmente integradas, inadequação histórica da sua forma de pagamento e a recente possibilidade de dispor de informação sobre o consumo de medicamentos de ambulatório em bases de dados informatizadas são três fortes motivos para o desenvolvimento de conhecimento associado a esta temática. Metodologia - Este trabalho compreende duas fases principais: i) a adaptação e aplicação de um modelo de consumo de medicamentos que permite estimar a carga de doença em ambulatório (designado de PRx). Nesta fase foi necessário realizar um trabalho de selecção, estruturação e classificação do modelo. A sua aplicação envolveu a utilização de bases de dados informatizadas de consumos com medicamentos nos anos de 2007 e 2008 para a região de Saúde do Alentejo; ii) na segunda fase foram simulados três modelos de financiamento alternativos que foram propostos para financiar as ULS em Portugal. Particularmente foram analisadas as dimensões e variáveis de ajustamento pelo risco (índices de mortalidade, morbilidade e custos per capita), sua ponderação relativa e consequente impacto financeiro. Resultados - Com o desenvolvimento do modelo PRx estima-se que 36% dos residentes na região Alentejo têm pelo menos uma doença crónica, sendo a capacidade de estimação do modelo no que respeita aos consumos de medicamentos na ordem dos 0,45 (R2). Este modelo revelou constituir uma alternativa a fontes de informação tradicionais como são os casos de outros estudos internacionais ou o Inquérito Nacional de Saúde. A consideração dos valores do PRx para efeitos de financiamento per capita introduz alterações face a outros modelos propostos neste âmbito. Após a análise dos montantes de financiamento entre os cenários alternativos, obtendo os modelos 1 e 2 níveis de concordância por percentil mais próximos entre si comparativamente ao modelo 3, seleccionou-se o modelo 1 como o mais adequado para a nossa realidade. Conclusão - A aplicação do modelo PRx numa região de saúde permitiu concluir em função dos resultados alcançados, que já existe a possibilidade de estruturação e operacionalização de um modelo que permite estimar a carga de doença em ambulatório a partir de informação relativa ao seu perfil de consumo de medicamentos dos utentes. A utilização desta informação para efeitos de financiamento de organizações de saúde verticalmente integradas provoca uma variação no seu actual nível de financiamento. Entendendo este estudo como um ponto de partida onde apenas uma parte da presente temática ficará definida, outras questões estruturantes do actual sistema de financiamento não deverão também ser olvidadas neste contexto. ------- ABSTRACT - Introduction - The main goal of this study was the development of a risk adjustment model for financing integrated delivery systems (IDS) in Portugal. The recent improvement of patient records, mainly at primary care level, the historical inadequacy of payment models and the increasing number of IDS were three important factors that drove us to develop new approaches for risk adjustment in our country. Methods - The work was divided in two steps: the development of a pharmacy-based model in Portugal and the proposal of a risk adjustment model for financing IDS. In the first step an expert panel was specially formed to classify more than 33.000 codes included in Portuguese pharmacy national codes into 33 chronic conditions. The study included population of Alentejo Region in Portugal (N=441.550 patients) during 2007 and 2008. Using pharmacy data extracted from three databases: prescription, private pharmacies and hospital ambulatory pharmacies we estimated a regression model including Potential Years of Life Lost, Complexity, Severity and PRx information as dependent variables to assess total cost as the independent variable. This healthcare financing model was compared with other two models proposed for IDS. Results - The more prevalent chronic conditions are cardiovascular (34%), psychiatric disorders (10%) and diabetes (10%). These results are also consistent with the National Health Survey. Apparently the model presents some limitations in identifying patients with rheumatic conditions, since it underestimates prevalence and future drug expenditure. We obtained a R2 value of 0,45, which constitutes a good value comparing with the state of the art. After testing three scenarios we propose a model for financing IDS in Portugal. Conclusion - Drug information is a good alternative to diagnosis in determining morbidity level in a population basis through ambulatory care data. This model offers potential benefits to estimate chronic conditions and future drug costs in the Portuguese healthcare system. This information could be important to resource allocation decision process, especially concerning risk adjustment and healthcare financing.