Using data mining techniques to support breast cancer diagnosis

More than ever, there is an increase of the number of decision support methods and computer aided diagnostic systems applied to various areas of medicine. In breast cancer research, many works have been done in order to reduce false-positives when used as a double reading method. In this study, we aimed to present a set of data mining techniques that were applied to approach a decision support system in the area of breast cancer diagnosis. This method is geared to assist clinical practice in identifying mammographic findings such as microcalcifications, masses and even normal tissues, in order to avoid misdiagnosis. In this work a reliable database was used, with 410 images from about 115 patients, containing previous reviews performed by radiologists as microcalcifications, masses and also normal tissue findings. Throughout this work, two feature extraction techniques were used: the gray level co-occurrence matrix and the gray level run length matrix. For classification purposes, we considered various scenarios according to different distinct patterns of injuries and several classifiers in order to distinguish the best performance in each case described. The many classifiers used were Naïve Bayes, Support Vector Machines, k-nearest Neighbors and Decision Trees (J48 and Random Forests). The results in distinguishing mammographic findings revealed great percentages of PPV and very good accuracy values. Furthermore, it also presented other related results of classification of breast density and BI-RADS® scale. The best predictive method found for all tested groups was the Random Forest classifier, and the best performance has been achieved through the distinction of microcalcifications. The conclusions based on the several tested scenarios represent a new perspective in breast cancer diagnosis using data mining techniques.

Characterisation of noisy speech channels in 2G and 3G mobile networks

As the wireless cellular market reaches competitive levels never seen before, network operators need to focus on maintaining Quality of Service (QoS) a main priority if they wish to attract new subscribers while keeping existing customers satisfied. Speech Quality as perceived by the end user is one major example of a characteristic in constant need of maintenance and improvement. It is in this topic that this Master Thesis project fits in. Making use of an intrusive method of speech quality evaluation, as a means to further study and characterize the performance of speech codecs in second-generation (2G) and third-generation (3G) technologies. Trying to find further correlation between codecs with similar bit rates, along with the exploration of certain transmission parameters which may aid in the assessment of speech quality. Due to some limitations concerning the audio analyzer equipment that was to be employed, a different system for recording the test samples was sought out. Although the new designed system is not standard, after extensive testing and optimization of the system's parameters, final results were found reliable and satisfactory. Tests include a set of high and low bit rate codecs for both 2G and 3G, where values were compared and analysed, leading to the outcome that 3G speech codecs perform better, under the approximately same conditions, when compared with 2G. Reinforcing the idea that 3G is, with no doubt, the best choice if the costumer looks for the best possible listening speech quality. Regarding the transmission parameters chosen for the experiment, the Receiver Quality (RxQual) and Received Energy per Chip to the Power Density Ratio (Ec/N0), these were subject to speech quality correlation tests. Final results of RxQual were compared to those of prior studies from different researchers and, are considered to be of important relevance. Leading to the confirmation of RxQual as a reliable indicator of speech quality. As for Ec/N0, it is not possible to state it as a speech quality indicator however, it shows clear thresholds for which the MOS values decrease significantly. The studied transmission parameters show that they can be used not only for network management purposes but, at the same time, give an expected idea to the communications engineer (or technician) of the end-to-end speech quality consequences. With the conclusion of the work new ideas for future studies come to mind. Considering that the fourth-generation (4G) cellular technologies are now beginning to take an important place in the global market, as the first all-IP network structure, it seems of great relevance that 4G speech quality should be subject of evaluation. Comparing it to 3G, not only in narrowband but also adding wideband scenarios with the most recent standard objective method of speech quality assessment, POLQA. Also, new data found on Ec/N0 tests, justifies further research studies with the intention of validating the assumptions made in this work.

A regulação endócrina dos balanços de cálcio

RESUMO: As concentrações circulantes de cálcio são notavelmente constantes a despeito das variações diárias na absorção intestinal e na eliminação renal deste elemento. A regulação da calcémia é um sistema complexo que compreende vários factores controladores (a calcémia, a fosforémia, as concentrações circulantes de paratormona (PTH) e calcitriol além de muitos outros factores como hormonas esteróides em geral, outros iões como o magnésio e outros factores hormonais) e vários órgãos alvo (glândulas paratiroideias, osso, rim e intestino). As respostas dos órgãos alvo também são muito variadas. No caso mais simples, a cristalização de sais de cálcio corresponde a uma mudança de fase em que participam moléculas orgânicas que a iniciam, aceleram ou inibem. Em geral a combinação de um factor controlador com o respectivo receptor de membrana (para polipeptídeos ou iões) ou intracelular (hormonas esteróides) é apenas o primeiro passo de uma cadeia bioquímica que introduz uma enorme amplificação na resposta. A esta variedade de mecanismos de resposta correspondem grandes diferenças nos tempos de resposta que podem ser de minutos a semanas. É hoje possível “observar” (medir) com apreciável rigor nos líquidos biológicos (sangue, urina, fezes, etc.) os factores mais importantes do sistema de regulação da calcémia (cálcio, fósforo, paratormona e calcitriol) assim como administrar estes factores em experiências agudas. Esta possibilidade reflecte – se na literatura neste campo que tem vindo a crescer. O advento das técnicas da biologia molecular tem permitido a caracterização molecular de algumas das disfunções da homeostase do cálcio e é de esperar um diagnóstico fisiopatológico cada vez mais rigoroso dessas disfunções. Com o avanço dos conhecimentos nesta área que não cessa de aumentar temos cada vez maiores capacidades para fazer diagnósticos e é cada vez mais difícil interpretar com rigor os correspondentes quadros metabólicos. A análise ou síntese de sistemas complexos é a actividade mais nobre dos engenheiros que lhes permite desenhar pontes, diques, barcos, aviões ou automóveis. Com o aparecimento de computadores de médio ou grande porte foi – lhes possível utilizar descrições matemáticas não só para desenhar sistemas como ainda para interpretar eventuais falhas na sua operação. Essas descrições matemáticas consistem numa sequência de operações realizadas num computador segundo um “programa informático” que receberam a designação genérica de modelos, por analogia com as famosas leis (equações) da física que foram deduzidas a partir de um certo número de postulados e que permitem representar matematicamente processos físicos. As famosas leis de Newton são talvez os exemplos mais famosos de “modelos” de sistemas físicos. A introdução de modelos matemáticos em biologia e particularmente em medicina só se deu recentemente.MÉTODOS No trabalho que aqui se apresenta construiu - se um modelo simplificado da homeostase do cálcio destinado ao cálculo de variáveis observáveis (concentrações de cálcio, fósforo, PTH e calcitriol) de modo a poderem comparar-se valores calculados com valores observados. A escolha dos componentes do modelo foi determinada pela nossa experiência clínica e pela informação fisiopatológica e clínica publicada. Houve a preocupação de construir o modelo de forma modular de modo a ser possível a sua expansão sem grandes transformações na descrição matemática (e informática) já existente. Na sua fase actual o modelo não pode ser usado como instrumento de diagnóstico. É antes uma ferramenta destinada a esclarecer “em princípio” mecanismos fisiopatológicos. Usou – se o modelo para simular um certo número de observações publicadas e para exemplificar a sua eventual aplicação clínica na simulação de situações hipotéticas e na análise de possíveis mecanismos fisiopatológicos responsáveis por situações de hipo ou hipercalcémias. Simultaneamente fez – se uma análise dos dados acumulados relativos a doentes vistos no Serviço de Endocrinologia do Instituto Português de Oncologia de Francisco Gentil – Centro Regional Oncológico de Lisboa, S.A. CONCLUSÕES Numa população de 894 doentes com patologias variadas do Instituto Português de Oncologia de Lisboa os valores da calcémia tiveram uma distribuição normal unimodal com uma média de 9.56 mg/dl, e um erro padrão de 0.41 mg/dl. Estas observações sugerem que a calcémia está sujeita a regulação. A partir dos resultados publicados em que o metabolismo do cálcio foi perturbado por infusões de cálcio, calcitriol ou PTH, de estudos bioquímicos e fisiológicos sobre os mecanismos de acção de factores controladores da calcémia e do estudo do comportamento de órgãos alvo (paratiroideias, intestino, osso e rim) foi possível construir um modelo matemático de parâmetros concentrados do sistema de regulação da calcémia. As expressões analíticas usadas foram baseadas na cinética enzimática de modo a que os seus parâmetros tivessem um significado físico ou fisiológico simples. O modelo revelou apreciável robustez e flexibilidade. É estável quando não perturbado e transita entre estados estacionários quando perturbado. Na sua forma actual gera simulações que reproduzem satisfatoriamente um número apreciável de dados experimentais colhidos em doentes. Isto não significa que possa ser usado como instrumento de diagnóstico aplicável a doentes individuais. O desenho do modelo comporta a adição posterior de novas relações quando surgirem situações para as quais se revele insuficiente. A utilização exaustiva do modelo permitiu explicitar aspectos do metabolismo do cálcio que ou não estão contidas na sua formulação actual – o aparecimento de hipertrofia ou de adenomas das paratiroideias e as alterações na estrutura óssea , a participação de outros factores controladores – magnésio, ou estão insuficientemente descritas – alterações do metabolismo do fósforo nos hipoparatiroidismos. A análise dos dados relativos aos doentes do Serviço de Endocrinologia do IPO permitiu o início da caracterização dos tipos de patologia que representam e de possíveis mecanismos fisiopatológicos subjacentes. Estas observações são o ponto de partida para análises futuras. São exemplos das relações encontradas: a distribuição dos doentes por dois grandes grupos conforme a calcémia é determinada pelas concentrações circulantes de PTH ou estas são determinadas pela calcémia; a distribuição sazonal das concentrações de Vit. D25. no sangue; a correlação negativa entre estas e as concentrações de PTH no sangue. Também foi possível extrair a cinética do controlo da PTH sobre a síntese de calcitriol. O estudo dos níveis circulantes de PTH no pós-operatório imediato de doentes paratiroidectomizados permitiu determinar as suas taxas de degradação metabólica. O modelo permitiu simular as relações Ca/PTH no sangue, Ca/Fracção excretada da carga tubular, Ca/P no sangue para valores normais ou altos de Ca. Foram feitas simulações de situações fisiopatológicas (em “doentes virtuais”): infusões crónicas de cálcio, PTH e calcitriol; alterações no comportamento de receptores. Estas simulações correspondem a experiências que não podem ser realizadas em humanos. São exemplos da utilização do modelo na exploração de possíveis mecanismos fisiopatológicos através da observação de resultados quantitativos inacessíveis à intuição. O modelo foi útil em duas fases do trabalho: Primeiro, durante a sua síntese implicou uma escolha criticamente selectiva de informação, sua análise quantitativa e processamento, uma explicitação rigorosa (analítica) das relações funcionais entre os controladores e as variáveis e da sua integração numa estrutura global; Segundo, a simulação de situações experimentais ou clínicas (dados do Serviço de Endocrinologia do IPO) em doentes obrigou a explicitar raciocínios fisiopatológicos habitualmente formulados em bases puramente intuitivas. Esta prática revelou comportamentos óbvios após as simulações – acção reduzida das infusões PTH (simulação de hiperparatiroidismos primários) enquanto não há inibição total da respectiva secreção, necessidade de aumento da massa secretora da paratiroideia nas insuficiências renais avançadas, etc. A síntese e utilização do modelo não implicaram uma preparação matemática avançada e foram possíveis mercê da disponibilidade de “software” interactivo especificamente desenhado para a simulação de sistemas dinâmicos em que os programas se escrevem em inglês usando a simbologia simples da álgebra elementar. A função nobre de modelos desta natureza é semelhante à dos modelos usados pelos físicos desde o século XVII: permitir explicações de carácter geral funcionando como uma ferramenta intelectual para manipulação de conceitos e para a realização de “experiências pensadas” (“thought experiments”) respeitando certos princípios físicos (princípios de conservação) que estabelecem as fronteiras da realidade. -------ABSTRACT: Calcium blood levels are remarkably constant despite great variations in calcium daily intake, intestinal absorption and renal excretion. The regulation of the calcium concentration in the blood is achieved by a complex system that includes several controller factors (mainly the serum levels of calcium, phosphorus, parathyroid hormone (PTH) and calcitriol but also of steroid hormones, ions such as magnesium and other hormonal factors) and several target organs (parathyroid glands, bone, kidney and intestine). The functional response to the controlling factors obeys a variety of kinetics. The precipitation of calcium salts is a simple phase transition in which organic molecules may provide nucleation centres or inhibit the process. The combination of a controller factor with its receptor located in the cell membrane (for peptides or ions) or in the nucleus (for steroid hormones) is only the first step of a biochemical chain that introduces a huge amplification in the response. To this great variability of response we have to add the times of response that vary from minutes to weeks. It is possible to “observe” (measure) with great accuracy in biological fluids (blood, urine, faeces, etc.) the most important factors intervening in the calcium regulation (calcium, phosphorus, PTH and calcitriol). The response of the system to acute infusions of the controlling factors has also been studied. Using molecular biology techniques it has been possible to characterize some calcium homeostasis dysfunctions and better physiopathological diagnosis are expected. With the increasingly new knowledge in this area we have better capacity to diagnose but it is harder to explain correctly the underlying metabolic mechanisms. The analysis or synthesis of complex systems is the noble activity of engineers that enables them to draw bridges, dams, boats, airplanes or cars. With the availability of medium-large frame computers it was possible to use mathematical descriptions not only to draw systems but also to explain flaws in its operations. These mathematical descriptions are generally known as models by analogy with the laws (equations) of physics that allow the mathematical description of physical processes. In practice it is not possible to find general solutions for the mathematical descriptions of complex systems but (numeric) computations for specific situations can be obtained with digital computers. The introduction of mathematical models in biology and particularly in medicine is a recent event. METHODS In this thesis a simplified model of calcium homeostasis was built that enables the computation of observable variables (concentrations of calcium, phosphorus, PTH and calcitriol) and allows the comparison between the simulation values and observed values. The choice of the model’s components was made according to our clinical experience and to the published clinical and physiopathological data. The model has a modular design that allows future expansions with minor alterations in its structure. In its present form the model cannot be used for diagnosis. It is a tool designed to enlighten physiopathological processes. To exemplify its possible clinical application in the simulation of hypothetical situations and in the analysis of possible mechanisms responsible for hypo or hypercalcemias the model was used to simulate a certain number of published observations. An analysis of clinical and laboratory data from the Endocrinology Department of the Portuguese Cancer Institute (I.P.O.F.G.-C.R.O.L.,S.A.) is also presented. CONCLUSIONS In a population of 188 patients without an identifiable disease of the calcium metabolism at the Portuguese Cancer Institute the calcemia levels had a unimodal distribution with an average of 9.56 mg/dL and a S.E.M of 0.41 mg/dL. This observation confirms that serum calcium is regulated. Using published data; in which calcium metabolism was disrupted by calcium, PTH or calcitriol infusions; from biochemical and physiological studies of the action of controller factors on the calcemia; in which the response of target organs (parathyroid glands, intestine, bone, kidney) was studied it was possible to build a mathematical model of concentrated parameters of the calcium homeostasis. Analytical expressions used were based on enzymatic kinetics. The model is flexible and robust. It is stable when not disturbed and changes between steady states when disturbed. In its present form it provides simulations that reproduce closely a number of experimental clinical data. This does not mean that it can be used as a diagnostic tool for individual patients. The exhaustive utilisation of the model revealed the need of future expansions to include aspects of the calcium metabolism not included in its present form –hypertrophy or adenomas of the parathyroid glands, bone structure changes, participation of other controller factors such as magnesium – or insufficiently described – phosphate metabolism in hypoparathyroidism. The analysis of the data collected from the I.P.O.’s Endocrinology Department allowed the initial characterization of the different pathologies represented and of their possible physiopathological mechanisms. These observations are a starting point for future analysis. As examples of the relations found were: the distribution of patients in two groups according to the dependency of calcium by PTH levels or PTH levels by calcium concentration; the seasonal distribution of the serum concentrations of D25; its negative correlation with PTH concentration. It was also possible to extract the kinetics of the control of the synthesis of calcitriol by PTH. The analysis of immediate post-surgical levels of PTH in parathyroidectomized patients allowed the determination of its metabolic clearance. The model also allowed the simulation of the relations between Ca/PTH in blood, serum Ca/Fraction of tubular load excreted and Ca/P in blood for normal and high values of calcium. Simulations were made of pathological situations (in “virtual patients”): chronic infusions of calcium, PTH and calcitriol; changes in the characteristics of receptors. These simulations are not possible in real persons. They are an example of the use of this model in exploring possible mechanisms of disease through the observation of quantitative results not accessible to simple intuition. This model was useful in two phases: Firstly, its construction required a careful choice of data, its quantitative analysis and processing, an analytical description of the relations between controller factors and variables and their integration in a global structure. Secondly, the simulation of experimental or clinical (I.P.O.’s Endocrinology Department) data implied testing physiopathological explanations that previously were based on intuition. The construction and utilisation of the model didn’t demand an advanced mathematical preparation since user-friendly interactive software was used. This software was specifically designed for the simulation of dynamic systems. The programs are written in English using elementary algebra symbols. The essential function of this type of models is identical to that of those used by physicists since the XVII century which describe quantitatively natural processes and are an intellectual tool for the manipulation of concepts and the performance of “thought experiments” based in certain physical principles (conservation principles) that are the frontiers of reality.------------------RESUMÉE: Les concentrations circulantes de calcium sont constantes même pendant des variations de l’absorption intestinale et de l’élimination rénale de cet élément. La régulation de la calcémie est un système complexe qui comprend plusieurs éléments contrôleurs (la calcémie, la phosphorémie, les concentrations circulantes de l’hormone parathyroïdienne (PTH) e du calcitriol et d’autres comme les hormones stéroïdes ou des ions comme le magnésium) et plusieurs organes (glandes parathyroïdiennes, l’os, le rein et l’intestin). Les réponses de ces organes sont variées. Dans le cas plus simple, la cristallisation des sels de calcium correspond à un changement de phase dans lequel y participent des molécules organiques que la débutent, l’accélèrent ou l’inhibent. Généralement la combinaison d’un élément contrôleur avec leur récepteur de membrane (pour les peptides ou les ions) ou intracellulaire (pour les hormones stéroïdes) n’est que le premier pas d’une chaîne biochimique qu’introduit une grande amplification de la réponse. A cette variété de réponses correspondent des grandes différences des temps de réponses qu’y vont des minuits a semaines. Il est possible « observer » (mesurer) dans les fluides biologiques (sang, urine, fèces, etc.) les éléments plus importants du système de régulation de la calcémie (calcium, phosphate, PTH et le calcitriol) et les administrer en expérimentes aigus. Cette possibilité est visible dans la littérature publiée dans ce domaine qui est en croissance permanente. L’avenir des techniques de biologie moléculaire a permis caractériser des nombreuses dysfonctions de la régulation de la calcémie et on attend un diagnostique physiopathologique de ces dysfonctions chaque fois plus rigoureuses. Les connaissances dans ce domaine s’agrandissent et on a de plus de capacités pour faire des diagnostiques et il est chaque fois plus difficile les interpréter. L’analyse ou synthèse de systèmes complexes est l’activité plus noble des ingénieurs qui les permit dessiner des ponts, bateaux, avions ou automobiles. Avec des ordinateurs de médium ou grand port il les est possible utiliser descriptions mathématiques pour dessiner les systèmes et interpréter des éventuelles fautes d’opération. Ces descriptions mathématiques sont une séquence d’opérations réalisées dans un ordinateur selon « un programme informatique » qui ont reçu la désignation générique de modèles, pour analogie avec les équations de la physique qui ont été déduits d’un nombre de postulées et qu’ont permit représenter des processus physiques en équations mathématiques. Les fameuses équations de Newton sont peut-être les exemples plus connus des systèmes physiques. L’introduction des modèles mathématiques en biologie et en particulier en médecine est un évènement récent. Dans ce travaille, on a construit un modèle simplifié de l’homéostasie du calcium pour calculer les variables observables (concentrations de calcium, phosphate, PTH et calcitriol) pour les comparer. Les choix des components a été déterminés par notre expérience clinique et par l’information physiopathologique et clinique publiée. Le modèle a été construit de façon modulaire ce que permit leur postérieur expansion sans des grandes altérations dans la description mathématique et informatique déjà existante. Dans cette forme le modèle ne peut être utilisé comme un instrument de diagnostique. Il est un outil pour éclairer la physiopathologie. Le modèle a été utilisé pour simuler un certain nombre d’observations publiées et pour exemplifier leur possible utilisation clinique dans la simulation des hypothèses et de la physiopathologie des situations d’hypo ou hypercalcémie. On a fait une analyse des éléments des procès cliniques des malades observées dans le Service d’Endocrinologie de l’IPOFG-CROL, SA. Dans une population de 894 malades avec des différentes pathologies les valeurs de calcémie on une distribution uni modale avec une Médie de 9.56 mg/dL et une erreur standard de 0.41 mg/dL. Ces observations suggèrent que la calcémie soit sujette de régulation. En utilisant des résultats de travaux publiés dans lesquels le métabolisme du calcium a été changé par des infusions de calcium, calcitriol ou PTH, des études biochimiques et physiologiques sur des mécanismes d’action des éléments contrôleurs de la calcémie et de l’étude du comportement des organes cible (parathyroïdes, intestin, rein, os), il a été possible de construire un modèle mathématique de paramètres concentrés du système de régulation de la calcémie. Les expressions analytiques utilisées ont été basées sur la cinétique enzymatique de façon à que les paramètres aient eu une signification physique ou biologique. Le modèle est stable quand il n’est pas perturbé et transit entre états stationnaires quand il est sujet a des perturbations. A ce moment il fait des simulations qui reproduisent de façon satisfaisant un nombre d’observations expérimentales. La construction du modèle permit l’addiction de nouvelles relations dans les cas ou il est insuffisant. L’utilisation exhaustive du modèle a permit expliciter des aspects du métabolisme du calcium qui y ne sont pas compris – l’hyperplasie ou la formation des adénomes des parathyroïdes, les altérations de la structure des os, la participation d’outres éléments régulateurs (magnésium), ou sont insuffisamment décrites – les altérations du métabolisme des phosphates dans l’hypoparathyroidism. L’analyse de l’information des malades du Service d’Endocrinologie a permit caractériser les pathologies représentées et leurs possibles mécanismes physiopathologiques. Ces observations sont le point de départ pour les analyses futures. Sont des exemples des relations trouvées: la distribution des malades par deux groupes: ceux dans lequel la calcémie est déterminée par la PTH ou ceux dans lesquels la PTH est déterminée par la calcémie; la distribution sazonale de la concentration de la vitamine D; la corrélation négative entre la vitamine D et la PTH. On a eu la possibilité de déduire la cinétique de control de la PTH sur la synthèse du calcitriol. L’étude des niveaux circulants de PTH sur des sujets parathyroidectomisées a permit déduire leur taux de dégradation métabolique. Le modèle a permit simuler les relations Ca/PTH dans le sang, Ca/fraction éliminée par le rein, Ca/P dans le sang pour des valeurs normales ou hautes de calcium. On a fait des simulations de situations physiopathologiques (dans “malades virtuelles”): Infusions chroniques de calcium, PTH ou calcitriol; altérations des récepteurs. Ces simulations ne peuvent pas être réalisées dans les humains. Sont des exemples d’utilisation du modèle dans l’exploration des possibles mécanismes de la physiopathologie en observant des résultats quantitatifs inaccessibles à l’intuition. Le modèle a été utile pendant deux étapes des travaux: La première, dans sa construction on a choisi l’information disponible, son analyse quantitative, l’explicitation rigoureuse (analytique) des relations fonctionnelles entre les contrôleurs et les variables et sa intégration dans une structure globale. La deuxième, la simulation de situations expérimentales ou cliniques (du Service d’Endocrinologie) a obligé d’expliciter des raisonnements physiopathologiques généralement formulés utilisant l’intuition. Cette pratique a montré des comportements – action réduite des infusions de PTH (jusqu’à l’inhibition totale de leur respective sécrétion), nécessité d’augmenter la masse sécréteuse de la parathyroïde dans les insuffisants rénales, etc. La synthèse et utilisation du modèle n’ont pas besoin d’une formation avancée en mathématique et sont possibles grâce à un programme interactif qui a été conçu pour la simulation des systèmes dynamiques dans lesquels le programme se construit en anglais en utilisant la symbolique élémentaire de l’algèbre. La fonction noble de ces modèles est semblable à celles des physiques du XVII siècle: Permettre établir explications générales en fonctionnant comme un outil intellectuel pour manipuler des concepts et pour la réalisation d’expérimentes pensées en respectant certains principes de la physique (principe de la conservation) qu’établissent les frontières de la réalité.

A Data Mining Framework for Electric Load Profiling

This paper consists in the characterization of medium voltage (MV) electric power consumers based on a data clustering approach. It is intended to identify typical load profiles by selecting the best partition of a power consumption database among a pool of data partitions produced by several clustering algorithms. The best partition is selected using several cluster validity indices. These methods are intended to be used in a smart grid environment to extract useful knowledge about customers’ behavior. The data-mining-based methodology presented throughout the paper consists in several steps, namely the pre-processing data phase, clustering algorithms application and the evaluation of the quality of the partitions. To validate our approach, a case study with a real database of 1.022 MV consumers was used.

Scenarios Generation for Multi-Agent simulation of Electricity Markets based on Intelligent Data Analysis

This document presents a tool able to automatically gather data provided by real energy markets and to generate scenarios, capture and improve market players’ profiles and strategies by using knowledge discovery processes in databases supported by artificial intelligence techniques, data mining algorithms and machine learning methods. It provides the means for generating scenarios with different dimensions and characteristics, ensuring the representation of real and adapted markets, and their participating entities. The scenarios generator module enhances the MASCEM (Multi-Agent Simulator of Competitive Electricity Markets) simulator, endowing a more effective tool for decision support. The achievements from the implementation of the proposed module enables researchers and electricity markets’ participating entities to analyze data, create real scenarios and make experiments with them. On the other hand, applying knowledge discovery techniques to real data also allows the improvement of MASCEM agents’ profiles and strategies resulting in a better representation of real market players’ behavior. This work aims to improve the comprehension of electricity markets and the interactions among the involved entities through adequate multi-agent simulation.

Data Extraction Tool to Analyse, Transform and Store Real Data from Electricity Markets

The study of electricity markets operation has been gaining an increasing importance in the last years, as result of the new challenges that the restructuring process produced. Currently, lots of information concerning electricity markets is available, as market operators provide, after a period of confidentiality, data regarding market proposals and transactions. These data can be used as source of knowledge to define realistic scenarios, which are essential for understanding and forecast electricity markets behavior. The development of tools able to extract, transform, store and dynamically update data, is of great importance to go a step further into the comprehension of electricity markets and of the behaviour of the involved entities. In this paper an adaptable tool capable of downloading, parsing and storing data from market operators’ websites is presented, assuring constant updating and reliability of the stored data.

Automatic Electricity Markets Data Extraction for Realistic Multi-agent Simulations

Electricity markets worldwide suffered profound transformations. The privatization of previously nationally owned systems; the deregulation of privately owned systems that were regulated; and the strong interconnection of national systems, are some examples of such transformations [1, 2]. In general, competitive environments, as is the case of electricity markets, require good decision-support tools to assist players in their decisions. Relevant research is being undertaken in this field, namely concerning player modeling and simulation, strategic bidding and decision-support.

A Data-mining-based Methodology to support MV Electricity Customers' Characterization

This paper presents an electricity medium voltage (MV) customer characterization framework supportedby knowledge discovery in database (KDD). The main idea is to identify typical load profiles (TLP) of MVconsumers and to develop a rule set for the automatic classification of new consumers. To achieve ourgoal a methodology is proposed consisting of several steps: data pre-processing; application of severalclustering algorithms to segment the daily load profiles; selection of the best partition, corresponding tothe best consumers’ segmentation, based on the assessments of several clustering validity indices; andfinally, a classification model is built based on the resulting clusters. To validate the proposed framework,a case study which includes a real database of MV consumers is performed.

An automatic tool to Extract, Transform and Load data from real electricity markets

The study of Electricity Markets operation has been gaining an increasing importance in the last years, as result of the new challenges that the restructuring produced. Currently, lots of information concerning Electricity Markets is available, as market operators provide, after a period of confidentiality, data regarding market proposals and transactions. These data can be used as source of knowledge, to define realistic scenarios, essential for understanding and forecast Electricity Markets behaviour. The development of tools able to extract, transform, store and dynamically update data, is of great importance to go a step further into the comprehension of Electricity Markets and the behaviour of the involved entities. In this paper we present an adaptable tool capable of downloading, parsing and storing data from market operators’ websites, assuring actualization and reliability of stored data.

Analysis of Consumption Data to Detect Commercial Losses Using Performance Evaluation Methods in a Smart Grid

Electric power networks, namely distribution networks, have been suffering several changes during the last years due to changes in the power systems operation, towards the implementation of smart grids. Several approaches to the operation of the resources have been introduced, as the case of demand response, making use of the new capabilities of the smart grids. In the initial levels of the smart grids implementation reduced amounts of data are generated, namely consumption data. The methodology proposed in the present paper makes use of demand response consumers’ performance evaluation methods to determine the expected consumption for a given consumer. Then, potential commercial losses are identified using monthly historic consumption data. Real consumption data is used in the case study to demonstrate the application of the proposed method.

Multi-scale vegetation classification using earth observation data of the Sundarban mangrove forest, Bangladesh.

Dissertation submitted in partial fulfilment of the requirements for the Degree of Master of Science in Geospatial Technologies.

Integration of forest fire management with SDI: user requirements.

Dissertation submitted in partial fulfilment of the requirements for the Degree of Master of Science in Geospatial Technologies.

Algorithms for time series clustering applied to biomedical signals

Thesis submitted in the fulfillment of the requirements for the Degree of Master in Biomedical Engineering

Towards a unified European electricity market: The contribution of data-mining to support realistic simulation studies

Worldwide electricity markets have been evolving into regional and even continental scales. The aim at an efficient use of renewable based generation in places where it exceeds the local needs is one of the main reasons. A reference case of this evolution is the European Electricity Market, where countries are connected, and several regional markets were created, each one grouping several countries, and supporting transactions of huge amounts of electrical energy. The continuous transformations electricity markets have been experiencing over the years create the need to use simulation platforms to support operators, regulators, and involved players for understanding and dealing with this complex environment. This paper focuses on demonstrating the advantage that real electricity markets data has for the creation of realistic simulation scenarios, which allow the study of the impacts and implications that electricity markets transformations will bring to the participant countries. A case study using MASCEM (Multi-Agent System for Competitive Electricity Markets) is presented, with a scenario based on real data, simulating the European Electricity Market environment, and comparing its performance when using several different market mechanisms.

Data Mining Approach to support the Generation of Realistic Scenarios for Multi-Agent simulation of Electricity Markets

This paper presents the Realistic Scenarios Generator (RealScen), a tool that processes data from real electricity markets to generate realistic scenarios that enable the modeling of electricity market players’ characteristics and strategic behavior. The proposed tool provides significant advantages to the decision making process in an electricity market environment, especially when coupled with a multi-agent electricity markets simulator. The generation of realistic scenarios is performed using mechanisms for intelligent data analysis, which are based on artificial intelligence and data mining algorithms. These techniques allow the study of realistic scenarios, adapted to the existing markets, and improve the representation of market entities as software agents, enabling a detailed modeling of their profiles and strategies. This work contributes significantly to the understanding of the interactions between the entities acting in electricity markets by increasing the capability and realism of market simulations.