Word Segmentation for Chinese Wikipedia Using N-Gram Mutual Information

In this paper, we propose an unsupervised segmentation approach, named "n-gram mutual information", or NGMI, which is used to segment Chinese documents into n-character words or phrases, using language statistics drawn from the Chinese Wikipedia corpus. The approach alleviates the tremendous effort that is required in preparing and maintaining the manually segmented Chinese text for training purposes, and manually maintaining ever expanding lexicons. Previously, mutual information was used to achieve automated segmentation into 2-character words. The NGMI approach extends the approach to handle longer n-character words. Experiments with heterogeneous documents from the Chinese Wikipedia collection show good results.

Thai word segmentation with hidden Markov Model and decision tree

The Thai written language is one of the languages that does not have word boundaries. In order to discover the meaning of the document, all texts must be separated into syllables, words, sentences, and paragraphs. This paper develops a novel method to segment the Thai text by combining a non-dictionary based technique with a dictionary-based technique. This method first applies the Thai language grammar rules to the text for identifying syllables. The hidden Markov model is then used for merging possible syllables into words. The identified words are verified with a lexical dictionary and a decision tree is employed to discover the words unidentified by the lexical dictionary. Documents used in the litigation process of Thai court proceedings have been used in experiments. The results which are segmented words, obtained by the proposed method outperform the results obtained by other existing methods.

Improvement to Chinese information retrieval by incorporating word segmentation and query expansion

The increasing diversity of the Internet has created a vast number of multilingual resources on the Web. A huge number of these documents are written in various languages other than English. Consequently, the demand for searching in non-English languages is growing exponentially. It is desirable that a search engine can search for information over collections of documents in other languages. This research investigates the techniques for developing high-quality Chinese information retrieval systems. A distinctive feature of Chinese text is that a Chinese document is a sequence of Chinese characters with no space or boundary between Chinese words. This feature makes Chinese information retrieval more difficult since a retrieved document which contains the query term as a sequence of Chinese characters may not be really relevant to the query since the query term (as a sequence Chinese characters) may not be a valid Chinese word in that documents. On the other hand, a document that is actually relevant may not be retrieved because it does not contain the query sequence but contains other relevant words. In this research, we propose two approaches to deal with the problems. In the first approach, we propose a hybrid Chinese information retrieval model by incorporating word-based techniques with the traditional character-based techniques. The aim of this approach is to investigate the influence of Chinese segmentation on the performance of Chinese information retrieval. Two ranking methods are proposed to rank retrieved documents based on the relevancy to the query calculated by combining character-based ranking and word-based ranking. Our experimental results show that Chinese segmentation can improve the performance of Chinese information retrieval, but the improvement is not significant if it incorporates only Chinese segmentation with the traditional character-based approach. In the second approach, we propose a novel query expansion method which applies text mining techniques in order to find the most relevant words to extend the query. Unlike most existing query expansion methods, which generally select the highly frequent indexing terms from the retrieved documents to expand the query. In our approach, we utilize text mining techniques to find patterns from the retrieved documents that highly correlate with the query term and then use the relevant words in the patterns to expand the original query. This research project develops and implements a Chinese information retrieval system for evaluating the proposed approaches. There are two stages in the experiments. The first stage is to investigate if high accuracy segmentation can make an improvement to Chinese information retrieval. In the second stage, a text mining based query expansion approach is implemented and a further experiment has been done to compare its performance with the standard Rocchio approach with the proposed text mining based query expansion method. The NTCIR5 Chinese collections are used in the experiments. The experiment results show that by incorporating the text mining based query expansion with the hybrid model, significant improvement has been achieved in both precision and recall assessments.

A Boundary-Oriented Chinese Segmentation Method Using NGram Mutual Information

This paper describes our participation in the Chinese word segmentation task of CIPS-SIGHAN 2010. We implemented an n-gram mutual information (NGMI) based segmentation algorithm with the mixed-up features from unsupervised, supervised and dictionarybased segmentation methods. This algorithm is also combined with a simple strategy for out-of-vocabulary (OOV) word recognition. The evaluation for both open and closed training shows encouraging results of our system. The results for OOV word recognition in closed training evaluation were however found unsatisfactory.

Traduction statistique vers une langue à morphologie riche : combinaison d’algorithmes de segmentation morphologique et de modèles statistiques de traduction automatique

Les systèmes statistiques de traduction automatique ont pour tâche la traduction d’une langue source vers une langue cible. Dans la plupart des systèmes de traduction de référence, l'unité de base considérée dans l'analyse textuelle est la forme telle qu’observée dans un texte. Une telle conception permet d’obtenir une bonne performance quand il s'agit de traduire entre deux langues morphologiquement pauvres. Toutefois, ceci n'est plus vrai lorsqu’il s’agit de traduire vers une langue morphologiquement riche (ou complexe). Le but de notre travail est de développer un système statistique de traduction automatique comme solution pour relever les défis soulevés par la complexité morphologique. Dans ce mémoire, nous examinons, dans un premier temps, un certain nombre de méthodes considérées comme des extensions aux systèmes de traduction traditionnels et nous évaluons leurs performances. Cette évaluation est faite par rapport aux systèmes à l’état de l’art (système de référence) et ceci dans des tâches de traduction anglais-inuktitut et anglais-finnois. Nous développons ensuite un nouvel algorithme de segmentation qui prend en compte les informations provenant de la paire de langues objet de la traduction. Cet algorithme de segmentation est ensuite intégré dans le modèle de traduction à base d’unités lexicales « Phrase-Based Models » pour former notre système de traduction à base de séquences de segments. Enfin, nous combinons le système obtenu avec des algorithmes de post-traitement pour obtenir un système de traduction complet. Les résultats des expériences réalisées dans ce mémoire montrent que le système de traduction à base de séquences de segments proposé permet d’obtenir des améliorations significatives au niveau de la qualité de la traduction en terme de le métrique d’évaluation BLEU (Papineni et al., 2002) et qui sert à évaluer. Plus particulièrement, notre approche de segmentation réussie à améliorer légèrement la qualité de la traduction par rapport au système de référence et une amélioration significative de la qualité de la traduction est observée par rapport aux techniques de prétraitement de base (baseline).

COGNITIVE AND LINGUISTIC FACTORS OF READING ARABIC: THE ROLE OF MORPHOLOGICAL AWARENESS IN READING

Scientific reading research has produced substantial evidence linking specific reading components to a range of constructs including phonological awareness (PA), morphological awareness, orthographic processing (OP), rapid automatized naming, working memory and vocabulary. There is a paucity of research on Arabic, although 420 million people around the world (Gordon, 2005) speak Arabic. As a Semitic language, Arabic differs in many ways from Indo-European languages. Over the past three decades, literacy research has begun to elucidate the importance of morphological awareness (MA) in reading. Morphology is a salient aspect of Arabic word structure. This study was designed to (a) examine the dimensions underlying MA in Arabic; (b) determine how well MA predicts reading; (c) investigate the role of the standard predictors in different reading outcomes; and (d) investigate the construct of reading in Arabic. This study was undertaken in two phases. In Phase I, 10 MA measures and two reading measures were developed, and tested in a sample of 102 Grade 3 Arabic-speaking children. Factor analysis of the 10 MA tasks yielded one predominant factor supporting the construct validity of MA in Arabic. Hierarchical regression analyses, controlling for age and gender, indicated that the MA factor solution accounted for 41– 43% of the variance in reading. In Phase II, the widely studied predictor measures were developed for PA and OP in addition to one additional measure of MA (root awareness), and three reading measures In Phase II, all measures were administered to another sample of 201 Grade 3 Arabic-speaking children. The construct of reading in Arabic was examined using factor analysis. The joint and unique effects of all standard predictors were examined using different sets of hierarchical regression analyses. Results of Phase II showed that: (a) all five reading measures loaded on one factor; (b) MA consistently accounted for unique variance in reading, particularly in comprehension, above and beyond the standard predictors; and (c) the standard predictors had differential contributions. These findings underscore the contribution of MA to all components of Arabic reading. The need for more emphasis on including morphology in Arabic reading instruction and assessment is discussed.

Kiinan kielen automaattinen käsittely ja kieliteknologia

Tutkielma käsittelee kiinan kielen automaattista käsittelyä ja kieliteknologiaa. Kieliteknologian osa-alueista keskitytään kiinan kielelle tyypilliseen sanarajatunnistus- eli segmentointiongelmaan, joka kumpuaa kiinan kielen kirjoitusjärjestelmän erityispiirteistä. Tutkielma on aihepiiriä esittelevä pilottitutkimus, jonka tarkoitettu lukijaryhmä on kiinan kieliteknologisesta tutkimuksesta kiinnostuneet opiskelijat ja tutkijat. Lähdemateriaali koostuu englannin- ja kiinankielisestä kirjallisuudesta, lähinnä konferenssiartikkeleista. Tutkielma esittelee kiinan kirjoitusjärjestelmää automaattisen käsittelyn näkökulmasta, käsittelee perinteisten ja yksinkertaistettujen merkkien eroja, merkkikoodauksia sekä erilaisia lähestymistapoja käyttäviä syöttöjärjestelmiä. Kirjoitusjärjestelmän esittely tarjoaa esitietoja kielen rakenteen ymmärtämiseksi sekä rakentaa pohjaa sanarajatunnistusta käsitteleviä osuuksia varten. Sanarajatunnistus- eli segmentointiongelma johtuu kiinan kirjoitusjärjestelmästä, jossa sanojen välejä ei merkitä välilyönneillä. Kielen kieliteknologista käsittelyä varten sanojen rajat tulee kuitenkin selvittää. Sanarajatunnistusjärjestelmät ovat tietokoneohjelmia, jotka etsivät ja merkitsevät nämä rajat automaattisesti. Tehtävä ei kuitenkaan ole yksinkertainen kielen monitulkintaisuuksien ja ns. tuntemattomien sanojen vuoksi. Joissain tilanteissa ei ole olemassa yksiselitteisen oikeaa segmentointia. Tutkielmassa esitellään kaksi segmentointijärjestelmää, keskittyen erityisesti niiden toiminnan kuvaukseen lukijalle ymmärrettävässä muodossa. Tärkeää on menetelmien ymmärtäminen, ei tekniset yksityiskohdat. Lopuksi paneudutaan segmentointijärjestelmien evaluaation ongelmiin. Sanarajatunnistusta suorittavien ohjelmien vertailu on usein hankalaa, koska monissa tapauksissa järjestelmät eivät tuota yhteismitallisia tuloksia. Tutkielmassa esitellään yritys saada aikaan yhteismitallisia evaluaatiomenetelmiä segmentointiohjelmien Chinese Word Segmentation Bakeoff -kilpailujen muodossa. Tutkielmassa todetaan sanarajatunnistusongelman olevan tärkeä tutkimuskohde. Ratkaisemattomia ongelmia on kuitenkin edelleen, tärkeimpänä evaluaatio. Avainsanat – Nyckelord – Keywords kiinan kieli, sanarajatunnistus, segmentointi,kirjoitusmerkit, merkkikoodaukset, kiinan syöttötavat

Machine recognition of printed Kannada text

This paper presents the design of a full fledged OCR system for printed Kannada text. The machine recognition of Kannada characters is difficult due to similarity in the shapes of different characters, script complexity and non-uniqueness in the representation of diacritics. The document image is subject to line segmentation, word segmentation and zone detection. From the zonal information, base characters, vowel modifiers and consonant conjucts are separated. Knowledge based approach is employed for recognizing the base characters. Various features are employed for recognising the characters. These include the coefficients of the Discrete Cosine Transform, Discrete Wavelet Transform and Karhunen-Louve Transform. These features are fed to different classifiers. Structural features are used in the subsequent levels to discriminate confused characters. Use of structural features, increases recognition rate from 93% to 98%. Apart from the classical pattern classification technique of nearest neighbour, Artificial Neural Network (ANN) based classifiers like Back Propogation and Radial Basis Function (RBF) Networks have also been studied. The ANN classifiers are trained in supervised mode using the transform features. Highest recognition rate of 99% is obtained with RBF using second level approximation coefficients of Haar wavelets as the features on presegmented base characters.

Recognition of open vocabulary, online handwritten pages in Tamil script

In this work, we describe a system, which recognises open vocabulary, isolated, online handwritten Tamil words and extend it to recognize a paragraph of writing. We explain in detail each step involved in the process: segmentation, preprocessing, feature extraction, classification and bigram-based post-processing. On our database of 45,000 handwritten words obtained through tablet PC, we have obtained symbol level accuracy of 78.5% and 85.3% without and with the usage of post-processing using symbol level language models, respectively. Word level accuracies for the same are 40.1% and 59.6%. A line and word level segmentation strategy is proposed, which gives promising results of 100% line segmentation and 98.1% word segmentation accuracies on our initial trials of 40 handwritten paragraphs. The two modules have been combined to obtain a full-fledged page recognition system for online handwritten Tamil data. To the knowledge of the authors, this is the first ever attempt on recognition of open vocabulary, online handwritten paragraphs in any Indian language.

Estratégias de construção textual do chat escrito em espanhol com língua estrangeira

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)

Online Handwritten Kannada Word Recognizer with Unrestricted Vocabulary

In this paper, we propose a novel heuristic approach to segment recognizable symbols from online Kannada word data and perform recognition of the entire word. Two different estimates of first derivative are extracted from the preprocessed stroke groups and used as features for classification. Estimate 2 proved better resulting in 88% accuracy, which is 3% more than that achieved with estimate 1. Classification is performed by statistical dynamic space warping (SDSW) classifier which uses X, Y co-ordinates and their first derivatives as features. Classifier is trained with data from 40 writers. 295 classes are handled covering Kannada aksharas, with Kannada numerals, Indo-Arabic numerals, punctuations and other special symbols like $ and #. Classification accuracies obtained are 88% at the akshara level and 80% at the word level, which shows the scope for further improvement in segmentation algorithm

NESP: Nonlinear enhancement and selection of plane for optimal segmentation and recognition of scene word images

In this paper, we report a breakthrough result on the difficult task of segmentation and recognition of coloured text from the word image dataset of ICDAR robust reading competition challenge 2: reading text in scene images. We split the word image into individual colour, gray and lightness planes and enhance the contrast of each of these planes independently by a power-law transform. The discrimination factor of each plane is computed as the maximum between-class variance used in Otsu thresholding. The plane that has maximum discrimination factor is selected for segmentation. The trial version of Omnipage OCR is then used on the binarized words for recognition. Our recognition results on ICDAR 2011 and ICDAR 2003 word datasets are compared with those reported in the literature. As baseline, the images binarized by simple global and local thresholding techniques were also recognized. The word recognition rate obtained by our non-linear enhancement and selection of plance method is 72.8% and 66.2% for ICDAR 2011 and 2003 word datasets, respectively. We have created ground-truth for each image at the pixel level to benchmark these datasets using a toolkit developed by us. The recognition rate of benchmarked images is 86.7% and 83.9% for ICDAR 2011 and 2003 datasets, respectively.

Word boundary modelling and full covariance gaussians for Arabic Speech-to-Text systems

This paper describes recent improvements to the Cambridge Arabic Large Vocabulary Continuous Speech Recognition (LVCSR) Speech-to-Text (STT) system. It is shown that wordboundary context markers provide a powerful method to enhance graphemic systems by implicit phonetic information, improving the modelling capability of graphemic systems. In addition, a robust technique for full covariance Gaussian modelling in the Minimum Phone Error (MPE) training framework is introduced. This reduces the full covariance training to a diagonal covariance training problem, thereby solving related robustness problems. The full system results show that the combined use of these and other techniques within a multi-branch combination framework reduces the Word Error Rate (WER) of the complete system by up to 5.9% relative. Copyright © 2011 ISCA.