Search Interfaces on the Web: Querying and Characterizing

Current-day web search engines (e.g., Google) do not crawl and index a significant portion of theWeb and, hence, web users relying on search engines only are unable to discover and access a large amount of information from the non-indexable part of the Web. Specifically, dynamic pages generated based on parameters provided by a user via web search forms (or search interfaces) are not indexed by search engines and cannot be found in searchers’ results. Such search interfaces provide web users with an online access to myriads of databases on the Web. In order to obtain some information from a web database of interest, a user issues his/her query by specifying query terms in a search form and receives the query results, a set of dynamic pages that embed required information from a database. At the same time, issuing a query via an arbitrary search interface is an extremely complex task for any kind of automatic agents including web crawlers, which, at least up to the present day, do not even attempt to pass through web forms on a large scale. In this thesis, our primary and key object of study is a huge portion of the Web (hereafter referred as the deep Web) hidden behind web search interfaces. We concentrate on three classes of problems around the deep Web: characterization of deep Web, finding and classifying deep web resources, and querying web databases. Characterizing deep Web: Though the term deep Web was coined in 2000, which is sufficiently long ago for any web-related concept/technology, we still do not know many important characteristics of the deep Web. Another matter of concern is that surveys of the deep Web existing so far are predominantly based on study of deep web sites in English. One can then expect that findings from these surveys may be biased, especially owing to a steady increase in non-English web content. In this way, surveying of national segments of the deep Web is of interest not only to national communities but to the whole web community as well. In this thesis, we propose two new methods for estimating the main parameters of deep Web. We use the suggested methods to estimate the scale of one specific national segment of the Web and report our findings. We also build and make publicly available a dataset describing more than 200 web databases from the national segment of the Web. Finding deep web resources: The deep Web has been growing at a very fast pace. It has been estimated that there are hundred thousands of deep web sites. Due to the huge volume of information in the deep Web, there has been a significant interest to approaches that allow users and computer applications to leverage this information. Most approaches assumed that search interfaces to web databases of interest are already discovered and known to query systems. However, such assumptions do not hold true mostly because of the large scale of the deep Web – indeed, for any given domain of interest there are too many web databases with relevant content. Thus, the ability to locate search interfaces to web databases becomes a key requirement for any application accessing the deep Web. In this thesis, we describe the architecture of the I-Crawler, a system for finding and classifying search interfaces. Specifically, the I-Crawler is intentionally designed to be used in deepWeb characterization studies and for constructing directories of deep web resources. Unlike almost all other approaches to the deep Web existing so far, the I-Crawler is able to recognize and analyze JavaScript-rich and non-HTML searchable forms. Querying web databases: Retrieving information by filling out web search forms is a typical task for a web user. This is all the more so as interfaces of conventional search engines are also web forms. At present, a user needs to manually provide input values to search interfaces and then extract required data from the pages with results. The manual filling out forms is not feasible and cumbersome in cases of complex queries but such kind of queries are essential for many web searches especially in the area of e-commerce. In this way, the automation of querying and retrieving data behind search interfaces is desirable and essential for such tasks as building domain-independent deep web crawlers and automated web agents, searching for domain-specific information (vertical search engines), and for extraction and integration of information from various deep web resources. We present a data model for representing search interfaces and discuss techniques for extracting field labels, client-side scripts and structured data from HTML pages. We also describe a representation of result pages and discuss how to extract and store results of form queries. Besides, we present a user-friendly and expressive form query language that allows one to retrieve information behind search interfaces and extract useful data from the result pages based on specified conditions. We implement a prototype system for querying web databases and describe its architecture and components design.

Search Engine Optimization Methods & Search Engine Indexing for CMS Applications

Search engine optimization & marketing is a set of processes widely used on websites to improve search engine rankings which generate quality web traffic and increase ROI. Content is the most important part of any website. CMS web development is now become very essential for most of organizations and online businesses to develop their online system and websites. Every online business using a CMS wants to get users (customers) to make profit and ROI. This thesis comprises a brief study of existing SEO methods, tools and techniques and how they can be implemented to optimize a content base website. In results, the study provides recommendations about how to use SEO methods; tools and techniques to optimize CMS based websites on major search engines. This study compares popular CMS systems like Drupal, WordPress and Joomla SEO features and how implementing SEO can be improved on these CMS systems. Having knowledge of search engine indexing and search engine working is essential for a successful SEO campaign. This work is a complete guideline for web developers or SEO experts who want to optimize a CMS based website on all major search engines.

Optimization as a method of search engine marketing

This study is dedicated to search engine marketing (SEM). It aims for developing a business model of SEM firms and to provide explicit research of trustworthy practices of virtual marketing companies. Optimization is a general term that represents a variety of techniques and methods of the web pages promotion. The research addresses optimization as a business activity, and it explains its role for the online marketing. Additionally, it highlights issues of unethical techniques utilization by marketers which created relatively negative attitude to them on the Internet environment. Literature insight combines in the one place both technical and economical scientific findings in order to highlight technological and business attributes incorporated in SEM activities. Empirical data regarding search marketers was collected via e-mail questionnaires. 4 representatives of SEM companies were engaged in this study to accomplish the business model design. Additionally, the fifth respondent was a representative of the search engine portal, who provided insight on relations between search engines and marketers. Obtained information of the respondents was processed qualitatively. Movement of commercial organizations to the online market increases demand on promotional programs. SEM is the largest part of online marketing, and it is a prerogative of search engines portals. However, skilled users, or marketers, are able to implement long-term marketing programs by utilizing web page optimization techniques, key word consultancy or content optimization to increase web site visibility to search engines and, therefore, user’s attention to the customer pages. SEM firms are related to small knowledge-intensive businesses. On the basis of data analysis the business model was constructed. The SEM model includes generalized constructs, although they represent a wider amount of operational aspects. Constructing blocks of the model includes fundamental parts of SEM commercial activity: value creation, customer, infrastructure and financial segments. Also, approaches were provided on company’s differentiation and competitive advantages evaluation. It is assumed that search marketers should apply further attempts to differentiate own business out of the large number of similar service providing companies. Findings indicate that SEM companies are interested in the increasing their trustworthiness and the reputation building. Future of the search marketing is directly depending on search engines development.

Hakukoneoptimointi internetmarkkinoinnin tukena

Yhä useampi etsii nykyään tietoa tuotteista ja palveluista internetin kautta. Vastapainoisesti lähes jokainen yritys käyttää internetsivujaan markkinointikanavana. Mietittäessä markkinoinnin peruskysymyksiä kuten kohdesegmentin saavuttamista tai kampanjan tuottoastetta ei vastausta usein osaa internetsivujen osalta antaa niin markkinointiosasto kuin IT-osastokaan. Hakukoneoptimointi on yksi hakukonemarkkinoinnin muoto, jonka avulla internetsivujen saavutettavuutta voidaan parantaa. Kehityksen toteamiseksi on oltava mittareita, joina internetsivuilla voidaan käyttää internetsivuille tarkoitettuja kävijäseurantaohjelmistoja. Tässä työssä käsitellään hakukoneoptimointia ja sen mahdollisuuksia parantaa sivustojen näkyvyyttä internetin hakukoneissa. Hakukoneoptimoinnilla tarkoitetaan sivustojen teknisen toteutuksen muokkaamista hakukoneystävälliseksi ja sisällön muokkaamista niin, että sivustotsijoittuvat halutuin hakusanoin hakutulosten kärkipäähän. Onnistumisen mittaamiseksi työssä perehdytään kävijäseurannan mahdollisuuksiin ja toteutukseen. Työn tavoitteena oli tuoda Primesoft Oy:lle riittävä tietotaito hakukoneoptimoinnista, toteuttaa hakukoneoptimointipalvelu ja muokata yrityksen ohjelmistot hakukoneoptimointia tukeviksi. Työn tavoitteet saavutettiin pääosin ja tutustuminen hakukoneoptimointiin avasi portin koko internetmarkkinoinnin maailmaan. Palvelun toimivuutta testattiin Primesoftin omilla sivuilla ja tulokset osoittautuivat varsin rohkaiseviksi. Jatkossa hakukoneoptimointia voidaan tarjota palveluna asiakkaille.

Elasticsearch as a new search engine for Invenio and more

Presentation at Open Repositories 2014, Helsinki, Finland, June 9-13, 2014

Tag Cloud of Search Queries for Repository System

Poster at Open Repositories 2014, Helsinki, Finland, June 9-13, 2014

Text Classification and Indexing in IP Networks

Internet on elektronisen postin perusrakenne ja ollut tärkeä tiedonlähde akateemisille käyttäjille jo pitkään. Siitä on tullut merkittävä tietolähde kaupallisille yrityksille niiden pyrkiessä pitämään yhteyttä asiakkaisiinsa ja seuraamaan kilpailijoitansa. WWW:n kasvu sekä määrällisesti että sen moninaisuus on luonut kasvavan kysynnän kehittyneille tiedonhallintapalveluille. Tällaisia palveluja ovet ryhmittely ja luokittelu, tiedon löytäminen ja suodattaminen sekä lähteiden käytön personointi ja seuranta. Vaikka WWW:stä saatavan tieteellisen ja kaupallisesti arvokkaan tiedon määrä on huomattavasti kasvanut viime vuosina sen etsiminen ja löytyminen on edelleen tavanomaisen Internet hakukoneen varassa. Tietojen hakuun kohdistuvien kasvavien ja muuttuvien tarpeiden tyydyttämisestä on tullut monimutkainen tehtävä Internet hakukoneille. Luokittelu ja indeksointi ovat merkittävä osa luotettavan ja täsmällisen tiedon etsimisessä ja löytämisessä. Tämä diplomityö esittelee luokittelussa ja indeksoinnissa käytettävät yleisimmät menetelmät ja niitä käyttäviä sovelluksia ja projekteja, joissa tiedon hakuun liittyvät ongelmat on pyritty ratkaisemaan.

Learning Preferences with Kernel-Based Methods

Learning of preference relations has recently received significant attention in machine learning community. It is closely related to the classification and regression analysis and can be reduced to these tasks. However, preference learning involves prediction of ordering of the data points rather than prediction of a single numerical value as in case of regression or a class label as in case of classification. Therefore, studying preference relations within a separate framework facilitates not only better theoretical understanding of the problem, but also motivates development of the efficient algorithms for the task. Preference learning has many applications in domains such as information retrieval, bioinformatics, natural language processing, etc. For example, algorithms that learn to rank are frequently used in search engines for ordering documents retrieved by the query. Preference learning methods have been also applied to collaborative filtering problems for predicting individual customer choices from the vast amount of user generated feedback. In this thesis we propose several algorithms for learning preference relations. These algorithms stem from well founded and robust class of regularized least-squares methods and have many attractive computational properties. In order to improve the performance of our methods, we introduce several non-linear kernel functions. Thus, contribution of this thesis is twofold: kernel functions for structured data that are used to take advantage of various non-vectorial data representations and the preference learning algorithms that are suitable for different tasks, namely efficient learning of preference relations, learning with large amount of training data, and semi-supervised preference learning. Proposed kernel-based algorithms and kernels are applied to the parse ranking task in natural language processing, document ranking in information retrieval, and remote homology detection in bioinformatics domain. Training of kernel-based ranking algorithms can be infeasible when the size of the training set is large. This problem is addressed by proposing a preference learning algorithm whose computation complexity scales linearly with the number of training data points. We also introduce sparse approximation of the algorithm that can be efficiently trained with large amount of data. For situations when small amount of labeled data but a large amount of unlabeled data is available, we propose a co-regularized preference learning algorithm. To conclude, the methods presented in this thesis address not only the problem of the efficient training of the algorithms but also fast regularization parameter selection, multiple output prediction, and cross-validation. Furthermore, proposed algorithms lead to notably better performance in many preference learning tasks considered.

Kävijätietojen kerääminen ja hyödyntäminen osana verkkopalveluita

Kävijätietojen keräys ja tiedon hyödyntäminen ovat monelle yritykselle yksi lisäkeino saavuttaa kohdeasiakkaansa sekä tarjota heille uusia lisäpalveluita. Kävijätietojen seuraamiselle sekä näiden tietojen hyödyntämiselle löytyy lukuisia käyttökohteita aina hakukoneoptimoinnista uusien asiakkaiden etsimiseen. Moni yritys onkin löytänyt tästä itselleen uuden toimialan. Työn tavoitteena on toteuttaa IP-pohjainen kävijätietojen keräämiseen soveltuva ohjelma, jonka tietoja pystytään hyödyntämään yrityksessä, jonka tuotekonseptiin kuuluu tarjota loppukäyttäjälle maksuttomia hakemisto- ja yhteystietopalveluita. Työssä keskitytään erityisesti ylläpidettävän kävijätietorekisterin luomiseen, mutta esitellään myös, kuinka esimerkiksi yrityksen hakutuloksia voidaan kerätyillä kävijätiedoilla tehostaa sekä tarjota palveluja käyttäville asiakkaille uusia lisäpalveluita. Työssä pyritään hyödyntämään tarjolla olevia avoimiin lisensseihin pohjautuvia ratkaisuja. Tiedonkeruuta toteutettaessa on jo alussa otettava huomioon, että tiedosta ei saada täysin eksaktia, vaan suuntaa-antavaa. Tämä ei kuitenkaan poista sitä, ettei tietoa voisi hyväksikäyttää.

Kernel-Based Ranking. Methods for Learning and Performance Estimation

Machine learning provides tools for automated construction of predictive models in data intensive areas of engineering and science. The family of regularized kernel methods have in the recent years become one of the mainstream approaches to machine learning, due to a number of advantages the methods share. The approach provides theoretically well-founded solutions to the problems of under- and overfitting, allows learning from structured data, and has been empirically demonstrated to yield high predictive performance on a wide range of application domains. Historically, the problems of classification and regression have gained the majority of attention in the field. In this thesis we focus on another type of learning problem, that of learning to rank. In learning to rank, the aim is from a set of past observations to learn a ranking function that can order new objects according to how well they match some underlying criterion of goodness. As an important special case of the setting, we can recover the bipartite ranking problem, corresponding to maximizing the area under the ROC curve (AUC) in binary classification. Ranking applications appear in a large variety of settings, examples encountered in this thesis include document retrieval in web search, recommender systems, information extraction and automated parsing of natural language. We consider the pairwise approach to learning to rank, where ranking models are learned by minimizing the expected probability of ranking any two randomly drawn test examples incorrectly. The development of computationally efficient kernel methods, based on this approach, has in the past proven to be challenging. Moreover, it is not clear what techniques for estimating the predictive performance of learned models are the most reliable in the ranking setting, and how the techniques can be implemented efficiently. The contributions of this thesis are as follows. First, we develop RankRLS, a computationally efficient kernel method for learning to rank, that is based on minimizing a regularized pairwise least-squares loss. In addition to training methods, we introduce a variety of algorithms for tasks such as model selection, multi-output learning, and cross-validation, based on computational shortcuts from matrix algebra. Second, we improve the fastest known training method for the linear version of the RankSVM algorithm, which is one of the most well established methods for learning to rank. Third, we study the combination of the empirical kernel map and reduced set approximation, which allows the large-scale training of kernel machines using linear solvers, and propose computationally efficient solutions to cross-validation when using the approach. Next, we explore the problem of reliable cross-validation when using AUC as a performance criterion, through an extensive simulation study. We demonstrate that the proposed leave-pair-out cross-validation approach leads to more reliable performance estimation than commonly used alternative approaches. Finally, we present a case study on applying machine learning to information extraction from biomedical literature, which combines several of the approaches considered in the thesis. The thesis is divided into two parts. Part I provides the background for the research work and summarizes the most central results, Part II consists of the five original research articles that are the main contribution of this thesis.

2.6 MW:n tuulivoimakonvertterin nestejäähdytysjärjestelmän komponentit ja hinnat

Tässä tutkimuksessa selvitettiin nestejäähdytteisen 2.6 MW:n tuulivoimakonvertterin jäähdytysjärjestelmään kuuluvat komponentit ja niiden hinnat. Tällä pyritään helpottamaan uutta teknologiaa hyödyntävien jäähdytysjärjestelmien kehittäjiä kaupallistamaan tuotteensa selvittämällä heille nykyisten jäähdytysjärjestelmien hintaluokka. Työ tehtiin kirjallisuuskatsauksena ja lähteinä käytettiin pääasiassa ABB:lta saatavia julkisia materiaaleja. Komponenttien hinnat haettiin internetin hakukoneiden avulla, sekä lähettämällä tarjouspyyntöjä myyjille, jotka eivät hintojaan olleet suoraan ilmoittaneet. Hinnat komponenteista kerättiin 1, 10, 50 ja 100 kappaleen toimituserille, jolloin voitiin vertailla myös kuinka paljon edullisemmaksi sarjatuotanto tulisi, kuin yksittäisen tuotteen valmistaminen. Tuloksista huomattiin, että konvertterikaapiston sisäiset jäähdytysjärjestelmään kuuluvat komponentit muodostavat vain pienen osan siitä hinnasta, mikä aiheutuu ulkopuolisesta jäähdytysyksiköstä, johon kaapisto on kytkettävä. Käytettäessä yksittäistä jäähdytysyksikköä on koko jäähdytysjärjestelmän hinta 14-15 000 €, josta ulkoisen jäähdytysyksikön osuus on 12 000 €, eli yli 80 % kokonaishinnasta. Sarjatuotannossa hinta putoaa etenkin jäähdytysyksikön osalta huomattavasti. Uusia teknologioita kaupallistettaessa on niiden kokonaisuudessaan oltava tätä summaa halvempia, johon paras tapa päästä on keskittää huomionsa ulkoisen jäähdytysyksikön kustannusten laskemiseen. Toinen vaihtoehto kaupalliseen menestykseen, on tehdä samanhintainen jäähdytysjärjestelmä, mutta jäähdytysteholtaan huomattavasti tehokkaampi, sekä kooltaan pienempi.

Repository content - lost in Discovery?

Presentation at Open Repositories 2014, Helsinki, Finland, June 9-13, 2014

Mobile E-commerce business model – A value web based approach to business models in mobile gaming industry

Tämän tutkimuksen tavoitteena oli tutkia langattomien internet palveluiden arvoverkkoa ja liiketoimintamalleja. Tutkimus oli luonteeltaan kvalitatiivinen ja siinä käytettiin strategiana konstruktiivista case-tutkimusta. Esimerkkipalveluna oli Treasure Hunters matkapuhelinpeli. Tutkimus muodostui teoreettisesta ja empiirisestä osasta. Teoriaosassa liitettiin innovaatio, liiketoimintamallit ja arvoverkko käsitteellisesti toisiinsa, sekä luotiin perusta liiketoimintamallien kehittämiselle. Empiirisessä osassa keskityttiin ensin liiketoimintamallien luomiseen kehitettyjen innovaatioiden pohjalta. Lopuksi pyrittiin määrittämään arvoverkko palvelun toteuttamiseksi. Tutkimusmenetelminä käytettiin innovaatiosessiota, haastatteluja ja lomakekyselyä. Tulosten pohjalta muodostettiin useita liiketoimintakonsepteja sekä kuvaus arvoverkon perusmallista langattomille peleille. Loppupäätelmänä todettiin että langattomat palvelut vaativat toteutuakseen useista toimijoista koostuvan arvoverkon.

Influence of Link Structure on Search Engine Result Pages of Online Nursery Shops

The primary objective of this thesis is to assess how the backlink portfolio structure and off site Search Engine Optimisation (SEO) elements influence ranking of UK based online nursery shops. The growth of the internet use demanded significant effort from companies to optimize and increase their online presence in order to cope with the increasing online competition. The new e-Commerce technology - called Search Engine Optimisation - has been developed that helped increase website visibility of companies. The SEO process involves on site elements (i.e. changing the parameters of the company's website such as keywords, title tags and meta descriptions) and off site elements (link building and social media marketing activity). Link Building is based on several steps of marketing planning including keyword research and competitor analysis. The underlying goal of keyword research is to understand the targeted market through identifying relevant keyword queries that are used by targeted costumer group. In the analysis, three types (geographic, field and company’s strategy related) and seven sources of keywords has been identified and used as a base of analysis. Following the determination of the most popular keywords, allinanchor and allintitle search has been conducted and the first ten results of the searches have been collected to identify the companies with the most significant web presence among the nursery shops. Finally, Link Profiling has been performed where the essential goal was to understand to what extent other companies' link structure is different that the base company's backlinks. Significant difference has been found that distinguished the top three companies ranking in the allinanchor and allintitle search. The top three companies, „Mothercare”, „Mamas and Papas” and „Kiddicare” maintained significantly better metrics regarding domain and page authority on the main landing pages, the average number of outbound links for link portfolio metric and in number of backlinks. These companies also ranked among the highest in page authority distribution and followed external linking.

A RecommendationSystem for a Web Shop

Tämän työn tarkoituksena on käytännöllisen suositusjärjestelmäratkaisun kehittäminen verkkokauppaympäristöön olemassaolevaa teoriatietoa käyttäen. Työn ensimmäisessä osiossa tarkastellaan ensin tapoja lähdetiedon keräämiseksi järjestelmää varten. Tämän jälkeen käydään läpi eri menetelmiä suosituksen toteuttamiseksi. Lisäksi tutustutaan yleisiin ongelmiin eri menetelmien kanssa. Seuraavaksi tutkitaan miten järjestelmän käyttämään suositustietoa voidaan ryhmitellä. Tämänjälkeen arvioidaan esitettyjä menetelmiä yleisesti tunnettujen kriteerien perusteella. Suositusjärjestelmän toteutustyö on kuvattuna työn toisessa osiossa. Toteutettu ohjelmisto on asennettu kahteen erilliseen toimintaympäristöön.