An architecture for an effective usage of data mining in business intelligence systems

Business Intelligence (BI) is one emergent area of the Decision Support Systems (DSS) discipline. Over the last years, the evolution in this area has been considerable. Similarly, in the last years, there has been a huge growth and consolidation of the Data Mining (DM) field. DM is being used with success in BI systems, but a truly DM integration with BI is lacking. Therefore, a lack of an effective usage of DM in BI can be found in some BI systems. An architecture that pretends to conduct to an effective usage of DM in BI is presented.

Data mining contributions to characterize MV consumers and to improve the suppliers-consumers settlements

This paper deals with the establishment of a characterization methodology of electric power profiles of medium voltage (MV) consumers. The characterization is supported on the data base knowledge discovery process (KDD). Data Mining techniques are used with the purpose of obtaining typical load profiles of MV customers and specific knowledge of their customers’ consumption habits. In order to form the different customers’ classes and to find a set of representative consumption patterns, a hierarchical clustering algorithm and a clustering ensemble combination approach (WEACS) are used. Taking into account the typical consumption profile of the class to which the customers belong, new tariff options were defined and new energy coefficients prices were proposed. Finally, and with the results obtained, the consequences that these will have in the interaction between customer and electric power suppliers are analyzed.

Data mining techniques contributions to support electrical vehicle demand response

The introduction of Electric Vehicles (EVs) together with the implementation of smart grids will raise new challenges to power system operators. This paper proposes a demand response program for electric vehicle users which provides the network operator with another useful resource that consists in reducing vehicles charging necessities. This demand response program enables vehicle users to get some profit by agreeing to reduce their travel necessities and minimum battery level requirements on a given period. To support network operator actions, the amount of demand response usage can be estimated using data mining techniques applied to a database containing a large set of operation scenarios. The paper includes a case study based on simulated operation scenarios that consider different operation conditions, e.g. available renewable generation, and considering a diversity of distributed resources and electric vehicles with vehicle-to-grid capacity and demand response capacity in a 33 bus distribution network.

Data mining techniques to support the classification of MV electricity customers

This paper describes a methodology that was developed for the classification of Medium Voltage (MV) electricity customers. Starting from a sample of data bases, resulting from a monitoring campaign, Data Mining (DM) techniques are used in order to discover a set of a MV consumer typical load profile and, therefore, to extract knowledge regarding to the electric energy consumption patterns. In first stage, it was applied several hierarchical clustering algorithms and compared the clustering performance among them using adequacy measures. In second stage, a classification model was developed in order to allow classifying new consumers in one of the obtained clusters that had resulted from the previously process. Finally, the interpretation of the discovered knowledge are presented and discussed.

Using data mining techniques to support DR programs definition in smart grids

In recent years, Power Systems (PS) have experimented many changes in their operation. The introduction of new players managing Distributed Generation (DG) units, and the existence of new Demand Response (DR) programs make the control of the system a more complex problem and allow a more flexible management. An intelligent resource management in the context of smart grids is of huge important so that smart grids functions are assured. This paper proposes a new methodology to support system operators and/or Virtual Power Players (VPPs) to determine effective and efficient DR programs that can be put into practice. This method is based on the use of data mining techniques applied to a database which is obtained for a large set of operation scenarios. The paper includes a case study based on 27,000 scenarios considering a diversity of distributed resources in a 32 bus distribution network.

A data-mining based methodology for wind forecasting

In many countries the use of renewable energy is increasing due to the introduction of new energy and environmental policies. Thus, the focus on the efficient integration of renewable energy into electric power systems is becoming extremely important. Several European countries have already achieved high penetration of wind based electricity generation and are gradually evolving towards intensive use of this generation technology. The introduction of wind based generation in power systems poses new challenges for the power system operators. This is mainly due to the variability and uncertainty in weather conditions and, consequently, in the wind based generation. In order to deal with this uncertainty and to improve the power system efficiency, adequate wind forecasting tools must be used. This paper proposes a data-mining-based methodology for very short-term wind forecasting, which is suitable to deal with large real databases. The paper includes a case study based on a real database regarding the last three years of wind speed, and results for wind speed forecasting at 5 minutes intervals.

A data-mining-based methodology for transmission expansion planning

In recent decades, all over the world, competition in the electric power sector has deeply changed the way this sector’s agents play their roles. In most countries, electric process deregulation was conducted in stages, beginning with the clients of higher voltage levels and with larger electricity consumption, and later extended to all electrical consumers. The sector liberalization and the operation of competitive electricity markets were expected to lower prices and improve quality of service, leading to greater consumer satisfaction. Transmission and distribution remain noncompetitive business areas, due to the large infrastructure investments required. However, the industry has yet to clearly establish the best business model for transmission in a competitive environment. After generation, the electricity needs to be delivered to the electrical system nodes where demand requires it, taking into consideration transmission constraints and electrical losses. If the amount of power flowing through a certain line is close to or surpasses the safety limits, then cheap but distant generation might have to be replaced by more expensive closer generation to reduce the exceeded power flows. In a congested area, the optimal price of electricity rises to the marginal cost of the local generation or to the level needed to ration demand to the amount of available electricity. Even without congestion, some power will be lost in the transmission system through heat dissipation, so prices reflect that it is more expensive to supply electricity at the far end of a heavily loaded line than close to an electric power generation. Locational marginal pricing (LMP), resulting from bidding competition, represents electrical and economical values at nodes or in areas that may provide economical indicator signals to the market agents. This article proposes a data-mining-based methodology that helps characterize zonal prices in real power transmission networks. To test our methodology, we used an LMP database from the California Independent System Operator for 2009 to identify economical zones. (CAISO is a nonprofit public benefit corporation charged with operating the majority of California’s high-voltage wholesale power grid.) To group the buses into typical classes that represent a set of buses with the approximate LMP value, we used two-step and k-means clustering algorithms. By analyzing the various LMP components, our goal was to extract knowledge to support the ISO in investment and network-expansion planning.

Power systems reliability calculation based on fuzzy data mining

This paper presents a methodology supported on the data base knowledge discovery process (KDD), in order to find out the failure probability of electrical equipments’, which belong to a real electrical high voltage network. Data Mining (DM) techniques are used to discover a set of outcome failure probability and, therefore, to extract knowledge concerning to the unavailability of the electrical equipments such us power transformers and high-voltages power lines. The framework includes several steps, following the analysis of the real data base, the pre-processing data, the application of DM algorithms, and finally, the interpretation of the discovered knowledge. To validate the proposed methodology, a case study which includes real databases is used. This data have a heavy uncertainty due to climate conditions for this reason it was used fuzzy logic to determine the set of the electrical components failure probabilities in order to reestablish the service. The results reflect an interesting potential of this approach and encourage further research on the topic.

Data mining applications in power systems - case-studies and future trends

Presently power system operation produces huge volumes of data that is still treated in a very limited way. Knowledge discovery and machine learning can make use of these data resulting in relevant knowledge with very positive impact. In the context of competitive electricity markets these data is of even higher value making clear the trend to make data mining techniques application in power systems more relevant. This paper presents two cases based on real data, showing the importance of the use of data mining for supporting demand response and for supporting player strategic behavior.

Zonal prices analysis supported by a data mining based methodology

A methodology based on data mining techniques to support the analysis of zonal prices in real transmission networks is proposed in this paper. The mentioned methodology uses clustering algorithms to group the buses in typical classes that include a set of buses with similar LMP values. Two different clustering algorithms have been used to determine the LMP clusters: the two-step and K-means algorithms. In order to evaluate the quality of the partition as well as the best performance algorithm adequacy measurements indices are used. The paper includes a case study using a Locational Marginal Prices (LMP) data base from the California ISO (CAISO) in order to identify zonal prices.

Basic dyestuffs removal from textile effluents using feathers: equilibrium, kinetic and column studies

This research work has been focused in the study of gallinaceous feathers, a waste that may be valorised as sorbent, to remove the Dark Blue Astrazon 2RN (DBA) from Dystar. This study was focused on the following aspects: optimization of experimental conditions through factorial design methodology, kinetic studies into a continuous stirred tank adsorber (at pH 7 and 20ºC), equilibrium isotherms (at pH 5, 7 and 9 at 20 and 45ºC) and column studies (at 20ºC, at pH 5, 7 and 9). In order to evaluate the influence of the presence of other components in the sorption of the dyestuff, all experiments were performed both for the dyestuff in aqueous solution and in real textile effluent. The pseudo-first and pseudo-second order kinetic models were fitted to the experimental data, being the latter the best fit for the aqueous solution of dyestuff. For the real effluent both models fit the experimental results and there is no statistical difference between them. The Central Composite Design (CCD) was used to evaluate the effects of temperature (15 - 45ºC) and pH (5 - 9) over the sorption in aqueous solution. The influence of pH was more significant than temperature. The optimal conditions selected were 45ºC and pH 9. Both Langmuir and Freundlich models could fit the equilibrium data. In the concentration range studied, the highest sorbent capacity was obtained for the optimal conditions in aqueous solution, which corresponds to a maximum capacity of 47± 4 mg g-1. The Yoon-Nelson, Thomas and Yan’s models fitted well the column experimental data. The highest breakthrough time for 50% removal, 170 min, was obtained at pH 9 in aqueous solution. The presence of the dyeing agents in the real wastewater decreased the sorption of the dyestuff mostly for pH 9, which is the optimal pH. The effect of pH is less pronounced in the real effluent than in aqueous solution. This work shows that feathers can be used as sorbent in the treatment of textile wastewaters containing DBA.

Removal of sulfamethoxazole from solution by raw and chemically treated walnut shells

Purpose: The sorption of sulfamethoxazole, a frequently detected pharmaceutical compound in the environment, onto walnut shells was evaluated. Methods: The sorption proprieties of the raw sorbent were chemically modified and two additional samples were obtained, respectively HCl and NaOH treated. Scanning electron microscopy, Fourier transform infrared spectroscopy, X-ray photoelectron spectroscopy, and thermogravimetric (TG/DTG) techniques were applied to investigate the effect of the chemical treatments on the shell surface morphology and chemistry. Sorption experiments to investigate the pH effect on the process were carried out between pH 2 and 8. Results: The chemical treatment did not substantially alter the structure of the sorbent (physical and textural characteristics) but modified the surface chemistry of the sorbent (acid–base properties, point of zero charge—pHpzc). The solution pH influences both the sorbent’s surface charge and sulfamethoxazole speciation. The best removal efficiencies were obtained for lower pH values where the neutral and cationic sulfamethoxazole forms are present in the solution. Langmuir and Freundlich isotherms were applied to the experimental adsorption data for sulfamethoxazole sorption at pH 2, 4, and 7 onto raw walnut shell. No statistical difference was found between the two models except for the pH 2 experimental data to which the Freundlich model fitted better. Conclusion: Sorption of sulfamethoxazole was found to be highly pH dependent in the entire pH range studied and for both raw and treated sorbent.

Previsão de vento baseado em técnicas de Data Mining

Mestrado em Engenharia Electrotécnica – Sistemas Eléctricos de Energia

Application of wetland systems in the treatment of acid rock mine drainage

The presented work was conducted within the Dissertation / Internship, branch of Environmental Protection Technology, associated to the Master thesis in Chemical Engineering by the Instituto Superior de Engenharia do Porto and it was developed in the Aquatest a.s, headquartered in Prague, in Czech Republic. The ore mining exploitation in the Czech Republic began in the thirteenth century, and has been extended until the twentieth century, being now evident the consequences of the intensive extraction which includes contamination of soil and sub-soil by high concentrations of heavy metals. The mountain region of Zlaté Hory was chosen for the implementation of the remediation project, which consisted in the construction of three cells (tanks), the first to raise the pH, the second for the sedimentation of the formed precipitates and a third to increase the process efficiency in order to reduce high concentrations of metals, with special emphasis on iron, manganese and sulfates. This project was initiated in 2005, being pioneer in this country and is still ongoing due to the complex chemical and biological phenomenon’s inherent to the system. At the site where the project was implemented, there is a natural lagoon, thereby enabling a comparative study of the two systems (natural and artificial) regarding the efficiency of both in the reduction/ removal of the referred pollutants. The study aimed to assist and cooperate in the ongoing investigation at the company Aquatest, in terms of field work conducted in Zlaté Hory and in terms of research methodologies used in it. Thereby, it was carried out a survey and analysis of available data from 2005 to 2008, being complemented by the treatment of new data from 2009 to 2010. Moreover, a theoretical study of the chemical and biological processes that occurs in both systems was performed. Regarding the field work, an active participation in the collection and in situ sample analyzing of water and soil from the natural pond has been attained, with the supervision of Engineer, Irena Šupiková. Laboratory analysis of water and soil were carried out by laboratory technicians. It was found that the natural lagoon is more efficient in reducing iron and manganese, being obtained removal percentages of 100%. The artificial lagoon had a removal percentage of 90% and 33% for iron and manganese respectively. Despite the minor efficiency of the constructed wetland, it must be pointed out that this system was designed for the treatment and consequent reduction of iron. In this context, it can conclude that the main goal has been achieved. In the case of sulphates, the removal optimization is yet a goal to be achieved not only in the Czech Republic but also in other places where this type of contamination persists. In fact, in the natural lagoon and in the constructed wetland, removal efficiencies of 45% and 7% were obtained respectively. It has been speculated that the water at the entrance of both systems has different sources. The analysis of the collected data shows at the entrance of the natural pond, a concentration of 4.6 mg/L of total iron, 14.6 mg/L of manganese and 951 mg/L of sulphates. In the artificial pond, the concentrations are 27.7 mg/L, 8.1 mg/L and 382 mg/L respectively for iron, manganese and sulphates. During 2010 the investigation has been expanded. The study of soil samples has started in order to observe and evaluate the contribution of bacteria in the removal of heavy metals being in its early phase. Summarizing, this technology has revealed to be an interesting solution, since in addition to substantially reduce the mentioned contaminants, mostly iron, it combines the low cost of implementation with an reduced maintenance, and it can also be installed in recreation parks, providing habitats for plants and birds.

Estudo da tratabilidade laboratorial e optimização experimental do tratamento de águas residuais de indústrias lácteas

O presente trabalho experimental teve como objectivos estudar a tratabilidade dum efluente lácteo utilizando a técnica de coagulação/floculação e avaliar a possibilidade de utilização do carvão activado granulado (CAG) Aquasorb 2000 como adsorvente para a remoção de compostos orgânicos presentes nos efluentes lácteos pré–tratados por coagulação/floculação, funcionando como um tratamento de polimento. No estudo da tratabilidade do efluente por coagulação/floculação investigou-se a influência de determinadas variáveis como o tipo e dose de coagulante e pH, a fim de encontrar as melhores condições operatórias. A utilização da referida técnica visou a redução do valor de concentração de alguns parâmetros: carência química de oxigénio (CQO); fósforo total e turvação, tendo sido utilizados efluentes desta indústria recolhidos em diferentes datas e após tratamento biológico, designados por A, B, C e D apresentando características diferentes. Sendo que o efluente A apresentava valores de CQO, fósforo total e turvação de 500 mg O2/L;32 mg P/L e 40 NTU respectivamente, o efluente B 1400 mg O2/L; 120 mg P/L e 80 NTU respectivamente, o efluente C 12300 mg O2/L; 87 mg P/L e 350 NTU respectivamente e o efluente D 340 mg O2/L; 33 mg P/L e 42 NTU respectivamente. Os coagulantes estudados foram hidróxido de cálcio (HC), sulfato de alumínio (SA) e tricloreto de ferro (TF). Verificou-se que o coagulante com maior eficácia nos efluentes estudados foi o TF. As maiores remoções de CQO, fósforo total e turvação, 89%, 99,9% e 99%, respectivamente, foram obtidas para o efluente C, com uma dosagem de TF de 4 g/L e com um pH entre 6 e 7. Entre os efluentes estudados este era o que apresentava valores iniciais mais elevados para qualquer um destes parâmetros. As melhores percentagens de remoção obtidas com o HC, para a CQO, fósforo total e turvação foram de 59%, 99% e 91%, respectivamente, com uma dosagem de HC de 1 g/L e com um pH entre 10 e 11,5 foram conseguidas no tratamento do efluente D, com o qual se alcançaram também as melhores remoções de CQO, fósforo total e turvação de 65%, 99% e 87%, respectivamente, quando se utilizou o coagulante SA, com uma dosagem de 2 g/L e com um pH entre 7 e 7,5. Relativamente ao volume de lamas produzido neste processo pela utilização dos diferentes coagulantes no tratamento dos efluentes referidos concluiu-se que o coagulante que gera menor volume de lamas é o HC, sendo o SA aquele que origina um maior volume. Submeteu-se posteriormente o efluente D, pré-tratado por coagulação/floculação, a um processo de adsorção em batch utilizando o CAG Aquasorb 2000, onde se conseguiu uma remoção de CQO de 48%, alcançando para este parâmetro o valor de 63 mg O2/L, nas condições operatórias que correspondem a uma massa de CAG de 12,5 g/L e um tempo de contacto de 3 horas. Quanto aos custos associados com os coagulantes, o que menores custos apresenta é o HC (150 €/ton), seguido pelo TF (250 €/ton) e por ultimo o SA (340 €/ton). Sendo que o efluente quando tratado com TF e SA é necessário uma correcção do pH do meio para que estes coagulantes actuem eficazmente, em que essa correcção de pH é realizada com hidróxido de sódio (540 €/ton). Realizaram-se ainda estudos de equilíbrio de adsorção com o carvão activado referido e o azul-de-metileno usando diferentes concentrações deste (50 mg/L; 100 mg/L e 200 mg/L) e diferentes massas de CAG (0,1g; 0,2g; 0,3g; 0,4g e 0,5g). A temperatura a que se realizaram estes ensaios foi de 28,7ºC e o volume de azul-de-metileno foi de 200 mL. Verificou-se que os melhores resultados obtidos foram para uma concentração de adsorvato de 100 mg/L. Ajustaram-se os modelos de Langmuir e Freundlich às isotérmicas obtidas tendo correlações mais elevadas para a concentração de 100 mg/L de corante (azul de metileno), sendo o modelo de Freundlich aquele que melhor se ajustou apresentando uma correlação quadrática de 0,9744 e os seguintes parâmetros Kf = 6,59 e n = 5,33, enquanto que o de Langmuir apresentou uma correlação quadrática de 0,9583 e os seguintes parâmetros qmáx = 83,3 mg/g de adsorvente e K = 20 L/mg de adsorvato.. Verificou-se que a capacidade de adsorção promovida pelo CAG, em relação ao azul-demetileno, obtida experimentalmente, 83,3 mg/g, é muito inferior à capacidade de adsorção teoricamente prevista pela ficha técnica deste carvão, 280 mg/g a uma temperatura de 25ºC, o que pode indiciar que o carvão utilizado não estaria nas melhores condições.