New kernel functions and learning methods for text and data mining

Recent advances in machine learning methods enable increasingly the automatic construction of various types of computer assisted methods that have been difficult or laborious to program by human experts. The tasks for which this kind of tools are needed arise in many areas, here especially in the fields of bioinformatics and natural language processing. The machine learning methods may not work satisfactorily if they are not appropriately tailored to the task in question. However, their learning performance can often be improved by taking advantage of deeper insight of the application domain or the learning problem at hand. This thesis considers developing kernel-based learning algorithms incorporating this kind of prior knowledge of the task in question in an advantageous way. Moreover, computationally efficient algorithms for training the learning machines for specific tasks are presented. In the context of kernel-based learning methods, the incorporation of prior knowledge is often done by designing appropriate kernel functions. Another well-known way is to develop cost functions that fit to the task under consideration. For disambiguation tasks in natural language, we develop kernel functions that take account of the positional information and the mutual similarities of words. It is shown that the use of this information significantly improves the disambiguation performance of the learning machine. Further, we design a new cost function that is better suitable for the task of information retrieval and for more general ranking problems than the cost functions designed for regression and classification. We also consider other applications of the kernel-based learning algorithms such as text categorization, and pattern recognition in differential display. We develop computationally efficient algorithms for training the considered learning machines with the proposed kernel functions. We also design a fast cross-validation algorithm for regularized least-squares type of learning algorithm. Further, an efficient version of the regularized least-squares algorithm that can be used together with the new cost function for preference learning and ranking tasks is proposed. In summary, we demonstrate that the incorporation of prior knowledge is possible and beneficial, and novel advanced kernels and cost functions can be used in algorithms efficiently.

Inter organizational learning within and by innovation networks

The thesis deals with the phenomenon of learning between organizations in innovation networks that develop new products, services or processes. Inter organizational learning is studied especially at the level of the network. The role of the network can be seen as twofold: either the network is a context for inter organizational learning, if the learner is something else than the network (organization, group, individual), or the network itself is the learner. Innovations are regarded as a primary source of competitiveness and renewal in organizations. Networking has become increasingly common particularly because of the possibility to extend the resource base of the organization through partnerships and to concentrate on core competencies. Especially in innovation activities, networks provide the possibility to answer the complex needs of the customers faster and to share the costs and risks of the development work. Networked innovation activities are often organized in practice as distributed virtual teams, either within one organization or as cross organizational co operation. The role of technology is considered in the research mainly as an enabling tool for collaboration and learning. Learning has been recognized as one important collaborative process in networks or as a motivation for networking. It is even more important in the innovation context as an enabler of renewal, since the essence of the innovation process is creating new knowledge, processes, products and services. The thesis aims at providing enhanced understanding about the inter organizational learning phenomenon in and by innovation networks, especially concentrating on the network level. The perspectives used in the research are the theoretical viewpoints and concepts, challenges, and solutions for learning. The methods used in the study are literature reviews and empirical research carried out with semi structured interviews analyzed with qualitative content analysis. The empirical research concentrates on two different areas, firstly on the theoretical approaches to learning that are relevant to innovation networks, secondly on learning in virtual innovation teams. As a result, the research identifies insights and implications for learning in innovation networks from several viewpoints on organizational learning. Using multiple perspectives allows drawing a many sided picture of the learning phenomenon that is valuable because of the versatility and complexity of situations and challenges of learning in the context of innovation and networks. The research results also show some of the challenges of learning and possible solutions for supporting especially network level learning.

Conditions for EFL learning and professional development : Finland-Swedish learner and teacher perspectives

Engelskans dominerande roll som internationellt språk och andra globaliseringstrender påverkar också Svenskfinland. Dessa trender påverkar i sin tur förutsättningarna för lärande och undervisning i engelska som främmande språk, det vill säga undervisningsmålen, de förväntade elev- och lärarroller, materialens ändamålsenlighet, lärares och elevers initiala erfarenheter av engelska och engelskspråkiga länder. Denna studie undersöker förutsättningarna för lärande och professionell utveckling i det svenskspråkiga nybörjarklassrummet i engelska som främmande språk. Utgångsläget för 351 nybörjare i engelska som främmande språk och 19 av deras lärare beskrivs och analyseras. Resultaten tyder på att engelska håller på att bli ett andraspråk snarare än ett traditionellt främmande språk för många unga elever. Dessa elever har också goda förutsättningar att lära sig engelska utanför skolan. Sådan var dock inte situationen för alla elever, vilket tyder på att det finns en anmärkningsvärd heterogenitet och även regional variation i det finlandssvenska klassrummet i engelska som främmande språk. Lärarresultaten tyder på att vissa lärare har klarat av att på ett konstruktivt sätt att tackla de förutsättningar de möter. Andra lärare uttrycker frustration över sin arbetssituation, läroplanen, undervisningsmaterialen och andra aktörer som kommer är av betydelse för skolmiljön. Studien påvisar att förutsättningarna för lärande och undervisning i engelska som främmande språk varierar i Svenskfinland. För att stöda elevers och lärares utveckling föreslås att dialogen mellan aktörer på olika nivå i samhället bör förbättras och systematiseras.

Kernel-Based Ranking. Methods for Learning and Performance Estimation

Machine learning provides tools for automated construction of predictive models in data intensive areas of engineering and science. The family of regularized kernel methods have in the recent years become one of the mainstream approaches to machine learning, due to a number of advantages the methods share. The approach provides theoretically well-founded solutions to the problems of under- and overfitting, allows learning from structured data, and has been empirically demonstrated to yield high predictive performance on a wide range of application domains. Historically, the problems of classification and regression have gained the majority of attention in the field. In this thesis we focus on another type of learning problem, that of learning to rank. In learning to rank, the aim is from a set of past observations to learn a ranking function that can order new objects according to how well they match some underlying criterion of goodness. As an important special case of the setting, we can recover the bipartite ranking problem, corresponding to maximizing the area under the ROC curve (AUC) in binary classification. Ranking applications appear in a large variety of settings, examples encountered in this thesis include document retrieval in web search, recommender systems, information extraction and automated parsing of natural language. We consider the pairwise approach to learning to rank, where ranking models are learned by minimizing the expected probability of ranking any two randomly drawn test examples incorrectly. The development of computationally efficient kernel methods, based on this approach, has in the past proven to be challenging. Moreover, it is not clear what techniques for estimating the predictive performance of learned models are the most reliable in the ranking setting, and how the techniques can be implemented efficiently. The contributions of this thesis are as follows. First, we develop RankRLS, a computationally efficient kernel method for learning to rank, that is based on minimizing a regularized pairwise least-squares loss. In addition to training methods, we introduce a variety of algorithms for tasks such as model selection, multi-output learning, and cross-validation, based on computational shortcuts from matrix algebra. Second, we improve the fastest known training method for the linear version of the RankSVM algorithm, which is one of the most well established methods for learning to rank. Third, we study the combination of the empirical kernel map and reduced set approximation, which allows the large-scale training of kernel machines using linear solvers, and propose computationally efficient solutions to cross-validation when using the approach. Next, we explore the problem of reliable cross-validation when using AUC as a performance criterion, through an extensive simulation study. We demonstrate that the proposed leave-pair-out cross-validation approach leads to more reliable performance estimation than commonly used alternative approaches. Finally, we present a case study on applying machine learning to information extraction from biomedical literature, which combines several of the approaches considered in the thesis. The thesis is divided into two parts. Part I provides the background for the research work and summarizes the most central results, Part II consists of the five original research articles that are the main contribution of this thesis.

Playing to Learn: Business Simulation Games as Leadership Learning Environments

The aim of this dissertation is to investigate if participation in business simulation gaming sessions can make different leadership styles visible and provide students with experiences beneficial for the development of leadership skills. Particularly, the focus is to describe the development of leadership styles when leading virtual teams in computer-­supported collaborative game settings and to identify the outcomes of using computer simulation games as leadership training tools. To answer to the objectives of the study, three empirical experiments were conducted to explore if participation in business simulation gaming sessions (Study I and II), which integrate face-­to-­face and virtual communication (Study III and IV), can make different leadership styles visible and provide students with experiences beneficial for the development of leadership skills. In the first experiment, a group of multicultural graduate business students (N=41) participated in gaming sessions with a computerized business simulation game (Study III). In the second experiment, a group of graduate students (N=9) participated in the training with a ‘real estate’ computer game (Study I and II). In the third experiment, a business simulation gaming session was organized for graduate students group (N=26) and the participants played the simulation game in virtual teams, which were organizationally and geographically dispersed but connected via technology (Study IV). Each team in all experiments had three to four students and students were between 22 and 25 years old. The business computer games used for the empirical experiments presented an enormous number of complex operations in which a team leader needed to make the final decisions involved in leading the team to win the game. These gaming environments were interactive;; participants interacted by solving the given tasks in the game. Thus, strategy and appropriate leadership were needed to be successful. The training was competition-­based and required implementation of leadership skills. The data of these studies consist of observations, participants’ reflective essays written after the gaming sessions, pre-­ and post-­tests questionnaires and participants’ answers to open-­ ended questions. Participants’ interactions and collaboration were observed when they played the computer games. The transcripts of notes from observations and students dialogs were coded in terms of transactional, transformational, heroic and post-­heroic leadership styles. For the data analysis of the transcribed notes from observations, content analysis and discourse analysis was implemented. The Multifactor Leadership Questionnaire (MLQ) was also utilized in the study to measure transformational and transactional leadership styles;; in addition, quantitative (one-­way repeated measures ANOVA) and qualitative data analyses have been performed. The results of this study indicate that in the business simulation gaming environment, certain leadership characteristics emerged spontaneously. Experiences about leadership varied between the teams and were dependent on the role individual students had in their team. These four studies showed that simulation gaming environment has the potential to be used in higher education to exercise the leadership styles relevant in real-­world work contexts. Further, the study indicated that given debriefing sessions, the simulation game context has much potential to benefit learning. The participants who showed interest in leadership roles were given the opportunity of developing leadership skills in practice. The study also provides evidence of unpredictable situations that participants can experience and learn from during the gaming sessions. The study illustrates the complex nature of experiences from the gaming environments and the need for the team leader and role divisions during the gaming sessions. It could be concluded that the experience of simulation game training illustrated the complexity of real life situations and provided participants with the challenges of virtual leadership experiences and the difficulties of using leadership styles in practice. As a result, the study offers playing computer simulation games in small teams as one way to exercise leadership styles in practice.

Learning to Understand the Retailers - The New Approach of a Paperboard Manufacturer

The aim of the study is to expand networking between a packaging material manufacturer and retailers in order to develop products which appeal to brand owners and their customers. The in-built targets are to understand the retailer’s role in the value chain, clarify who makes packaging decision of private label products, and canvass the importance of sustainability. The present value chain of the packaging material manufacturer is reviewed first. It is assumed that sustainability could be a common interest, and The Consumer Goods Forum’s “A Global Language for Packaging and Sustainability” report is shortly discussed. The presentation of the most common packaging materials is based on a guide called “Packaging in the Sustainability Agenda: A Guide for Corporate Decision Makers”. The terms manufacturer’s brand and private label are defined. A retail value chain with emphasis on the role of customers as partners is introduced. The study area is the Nordic countries, and the information about Nordic retailers was provided first by desk research. The interviews were made in Finland, Sweden, Norway and Denmark. The study method is qualitative: the intention was to get initial insights, ideas and understandings. The results are compiled under the headings: sustainability, private labels, cooperation and packaging development. Also the reasons for good profitability of private labels are explained. Sustainability or responsibility is a key driver for innovation in the retail sector. Private labels have become brands. The ways of cooperation between a packaging material manufacturer and a retailer could be education and training. Packaging development is of great interest to retailers and they are willing to contribute.

Biomedical Event Extraction with Machine Learning

Biomedical natural language processing (BioNLP) is a subfield of natural language processing, an area of computational linguistics concerned with developing programs that work with natural language: written texts and speech. Biomedical relation extraction concerns the detection of semantic relations such as protein-protein interactions (PPI) from scientific texts. The aim is to enhance information retrieval by detecting relations between concepts, not just individual concepts as with a keyword search. In recent years, events have been proposed as a more detailed alternative for simple pairwise PPI relations. Events provide a systematic, structural representation for annotating the content of natural language texts. Events are characterized by annotated trigger words, directed and typed arguments and the ability to nest other events. For example, the sentence “Protein A causes protein B to bind protein C” can be annotated with the nested event structure CAUSE(A, BIND(B, C)). Converted to such formal representations, the information of natural language texts can be used by computational applications. Biomedical event annotations were introduced by the BioInfer and GENIA corpora, and event extraction was popularized by the BioNLP'09 Shared Task on Event Extraction. In this thesis we present a method for automated event extraction, implemented as the Turku Event Extraction System (TEES). A unified graph format is defined for representing event annotations and the problem of extracting complex event structures is decomposed into a number of independent classification tasks. These classification tasks are solved using SVM and RLS classifiers, utilizing rich feature representations built from full dependency parsing. Building on earlier work on pairwise relation extraction and using a generalized graph representation, the resulting TEES system is capable of detecting binary relations as well as complex event structures. We show that this event extraction system has good performance, reaching the first place in the BioNLP'09 Shared Task on Event Extraction. Subsequently, TEES has achieved several first ranks in the BioNLP'11 and BioNLP'13 Shared Tasks, as well as shown competitive performance in the binary relation Drug-Drug Interaction Extraction 2011 and 2013 shared tasks. The Turku Event Extraction System is published as a freely available open-source project, documenting the research in detail as well as making the method available for practical applications. In particular, in this thesis we describe the application of the event extraction method to PubMed-scale text mining, showing how the developed approach not only shows good performance, but is generalizable and applicable to large-scale real-world text mining projects. Finally, we discuss related literature, summarize the contributions of the work and present some thoughts on future directions for biomedical event extraction. This thesis includes and builds on six original research publications. The first of these introduces the analysis of dependency parses that leads to development of TEES. The entries in the three BioNLP Shared Tasks, as well as in the DDIExtraction 2011 task are covered in four publications, and the sixth one demonstrates the application of the system to PubMed-scale text mining.

Sentence Compression with Supervised Machine Learning

The subject of the thesis is automatic sentence compression with machine learning, so that the compressed sentences remain both grammatical and retain their essential meaning. There are multiple possible uses for the compression of natural language sentences. In this thesis the focus is generation of television program subtitles, which often are compressed version of the original script of the program. The main part of the thesis consists of machine learning experiments for automatic sentence compression using different approaches to the problem. The machine learning methods used for this work are linear-chain conditional random fields and support vector machines. Also we take a look which automatic text analysis methods provide useful features for the task. The data used for machine learning is supplied by Lingsoft Inc. and consists of subtitles in both compressed an uncompressed form. The models are compared to a baseline system and comparisons are made both automatically and also using human evaluation, because of the potentially subjective nature of the output. The best result is achieved using a CRF - sequence classification using a rich feature set. All text analysis methods help classification and most useful method is morphological analysis. Tutkielman aihe on suomenkielisten lauseiden automaattinen tiivistäminen koneellisesti, niin että lyhennetyt lauseet säilyttävät olennaisen informaationsa ja pysyvät kieliopillisina. Luonnollisen kielen lauseiden tiivistämiselle on monta käyttötarkoitusta, mutta tässä tutkielmassa aihetta lähestytään television ohjelmien tekstittämisen kautta, johon käytännössä kuuluu alkuperäisen tekstin lyhentäminen televisioruudulle paremmin sopivaksi. Tutkielmassa kokeillaan erilaisia koneoppimismenetelmiä tekstin automaatiseen lyhentämiseen ja tarkastellaan miten hyvin erilaiset luonnollisen kielen analyysimenetelmät tuottavat informaatiota, joka auttaa näitä menetelmiä lyhentämään lauseita. Lisäksi tarkastellaan minkälainen lähestymistapa tuottaa parhaan lopputuloksen. Käytetyt koneoppimismenetelmät ovat tukivektorikone ja lineaarisen sekvenssin mallinen CRF. Koneoppimisen tukena käytetään tekstityksiä niiden eri käsittelyvaiheissa, jotka on saatu Lingsoft OY:ltä. Luotuja malleja vertaillaan Lopulta mallien lopputuloksia evaluoidaan automaattisesti ja koska teksti lopputuksena on jossain määrin subjektiivinen myös ihmisarviointiin perustuen. Vertailukohtana toimii kirjallisuudesta poimittu menetelmä. Tutkielman tuloksena paras lopputulos saadaan aikaan käyttäen CRF sekvenssi-luokittelijaa laajalla piirrejoukolla. Kaikki kokeillut teksin analyysimenetelmät auttavat luokittelussa, joista tärkeimmän panoksen antaa morfologinen analyysi.

Language-specific aspects of reading acquisition: the case of Russian

We have investigated Russian children’s reading acquisition during an intermediate period in their development: after literacy onset, but before they have acquired well-developed decoding skills. The results of our study suggest that Russian first graders rely primarily on phonemes and syllables as reading grain-size units. Phonemic awareness seems to have reached the metalinguistic level more rapidly than syllabic awareness after the onset of reading instruction, the reversal which is typical for the initial stages of formal reading instruction creating external demand for phonemic awareness. Another reason might be the inherent instability of syllabic boundaries in Russian. We have shown that body-coda is a more natural representation of subsyllabic structure in Russian than onset-rime. We also found that Russian children displayed variability of syllable onset and offset decisions which can be attributed to the lack of congruence between syllabic and morphemic word division in Russian. We suggest that fuzziness of syllable boundary decisions is a sign of the transitional nature of this stage in the reading development and it indicates progress towards an awareness of morphologically determined closed syllables. Our study also showed that orthographic complexity exerts an influence on reading in Russian from the very start of reading acquisition. Besides, we found that Russian first graders experience fluency difficulties in reading orthographically simple words and nonwords of two and more syllables. The transition from monosyllabic to bisyllabic lexical items constitutes a certain threshold, for which the syllabic structure seemed to be of no difference. When we compared the outcomes of the Russian children with the ones produced by speakers of other languages, we discovered that in the tasks which could be performed with the help of alphabetic recoding Russian children’s accuracy was comparable to that of children learning to read in relatively shallow orthographies. In tasks where this approach works only partially, Russian children demonstrated accuracy results similar to those in deeper orthographies. This pattern of moderate results in accuracy and excellent performance in terms of reaction times is an indication that children apply phonological recoding as their dominant strategy to various reading tasks and are only beginning to develop suitable multiple strategies in dealing with orthographically complex material. The development of these strategies is not completed during Grade 1 and the shift towards diversification of strategies apparently continues in Grade 2.