The challenge of validation: assessing the performance of a test of productive vocabulary

This paper assesses the performance of a vocabulary test designed to measure second language productive vocabulary knowledge.The test, Lex30, uses a word association task to elicit vocabulary, and uses word frequency data to measure the vocabulary produced. Here we report firstly on the reliability of the test as measured by a test-retest study, a parallel test forms experiment and an internal consistency measure. We then investigate the construct validity of the test by looking at changes in test performance over time, analyses of correlations with scores on similar tests, and comparison of spoken and written test performance. Last, we examine the theoretical bases of the two main test components: eliciting vocabulary and measuring vocabulary. Interpretations of our findings are discussed in the context of test validation research literature. We conclude that the findings reported here present a robust argument for the validity of the test as a research tool, and encourage further investigation of its validity in an instructional context

Testing Turing's five minutes, parallel-paired imitation game

Purpose – The purpose of this paper is to consider Turing's two tests for machine intelligence: the parallel-paired, three-participants game presented in his 1950 paper, and the “jury-service” one-to-one measure described two years later in a radio broadcast. Both versions were instantiated in practical Turing tests during the 18th Loebner Prize for artificial intelligence hosted at the University of Reading, UK, in October 2008. This involved jury-service tests in the preliminary phase and parallel-paired in the final phase. Design/methodology/approach – Almost 100 test results from the final have been evaluated and this paper reports some intriguing nuances which arose as a result of the unique contest. Findings – In the 2008 competition, Turing's 30 per cent pass rate is not achieved by any machine in the parallel-paired tests but Turing's modified prediction: “at least in a hundred years time” is remembered. Originality/value – The paper presents actual responses from “modern Elizas” to human interrogators during contest dialogues that show considerable improvement in artificial conversational entities (ACE). Unlike their ancestor – Weizenbaum's natural language understanding system – ACE are now able to recall, share information and disclose personal interests.

Heritage speaker competence differences, language change and input type: inflected infinitives in Heritage Brazilian Portuguese

It has been argued that colloquial dialects of Brazilian Portuguese (BP) have undergone significant linguistic change resulting in the loss of inflected infinitives (e.g., Pires, 2002, 2006). Since BP adults, at least educated ones, have complete knowledge of inflected infinitives, the implicit claim is that they are transmitted via formal education in the standard dialect. In the present article, I test one of the latent predictions of such claims; namely, the fact that heritage speakers of BP who lack formal education in the standard dialect should never develop native-like knowledge of inflected infinitives. In doing so, I highlight two significant implications (a) that heritage speaker grammars are a good source for testing dialectal variation and language change proposals and (b) incomplete acquisition and/or attrition are not the only sources of heritage language competence differences. Employing the syntactic and semantic tests of Rothman and Iverson (2007), I compare heritage speakers' knowledge to Rothman and Iverson's advanced adult L2 learners and educated native controls. Unlike the latter groups, the data for heritage speakers indicate that they do not have target knowledge of inflected infinitives, lending support to Pires' claims, suggesting that literacy plays a significant role in the acquisition of this grammatical property in BP.

Testing the Revised Hierarchical Model: evidence from word associations

The main purpose of the work described in this paper is to examine the extent to which the L2 developmental changes predicted by Kroll and Stewart's (1994) Revised Hierarchical Model (RHM) can be understood by word association response behaviour. The RHM attempts to account for the relative “strength of the links between words and concepts in each of the bilingual's languages” (Kroll, Van Hell, Tokowicz & Green, 2010, p. 373). It proposes that bilinguals with higher L2 proficiency tend to rely less on mediation, while less proficient L2 learners tend to rely on mediation and access L2 words by translating from L1 equivalents. In this paper, I present findings from a simple word association task. More proficient learners provided a greater proportion of collocational links, suggesting that they mediate less when compared to less proficient learners. The results provide tentative support for Kroll and Stewart's model

Typographic layout and first impressions: testing how changes in text layout influence reader's judgments of documents

This study explores how the typographic layout of information influences readers' impressions of magazine contents pages. Thirteen descriptors were used in a paired comparison procedure that assessed whether participants' rhetorical impressions of a set of six controlled documents change in relation to variations in layout. The combinations of layout attributes tested were derived from the structural attributes associated with three patterns of typographic differentiation (high, moderate, and low) described in a previous study (see Moys, 2014). The content and the range of stylistic attributes applied to the test material were controlled in order to focus on layout attributes. Triangulation of the quantitative and qualitative data indicates that, even within the experimental confines of limited stylistic differentiation, the layout attributes associated with patterns of high, moderate, and low typographic differentiation do influence readers' rhetorical judgments. In addition, the findings emphasize the importance of considering inter-relationships between clusters of typographic attributes rather than testing isolated variables.

Elephants in pyjamas: testing the weak central coherence account of autism spectrum disorders using a syntactic disambiguation task

According to the weak central coherence (CC) account individuals with autism spectrum disorders (ASD) exhibit enhanced local processing and weak part-whole integration. CC was investigated in the verbal domain. Adolescents, recruited using a 2 (ASD status) by 2 (language impairment status) design, completed an aural forced choice comprehension task involving syntactically ambiguous sentences. Half the picture targets depicted the least plausible interpretation, resulting in longer RTs across groups. These were assumed to reflect local processing. There was no ASD by plausibility interaction and consequently little evidence for weak CC in the verbal domain when conceptualised as enhanced local processing. Furthermore, there was little evidence that the processing of syntactically ambiguous sentences differed as a function of ASD or language-impairment status.

Syntactic complexity in the comprehension of wh-questions and relative clauses in typical language development and autism

This study investigates effects of syntactic complexity operationalised in terms of movement, intervention and (NP) feature similarity in the development of A’ dependencies in 4-, 6-, and 8-year old typically developing (TD) French children and children with Autism Spectrum Disorders (ASD). Children completed an off-line comprehension task testing eight syntactic structures classified in four levels of complexity: Level 0: No Movement; Level 1: Movement without (configurational) Intervention; Level 2: Movement with Intervention from an element which is maximally different or featurally ‘disjoint’ (mismatched in both lexical NP restriction and number); Level 3: Movement with Intervention from an element similar in one feature or featurally ‘intersecting’ (matched in lexical NP restriction, mismatched in number). The results show that syntactic complexity affects TD children across the three age groups, but also indicate developmental differences between these groups. Movement affected all three groups in a similar way, but intervention effects in intersection cases were stronger in younger than older children, with NP feature similarity affecting only 4-year olds. Complexity effects created by the similarity in lexical restriction of an intervener thus appear to be overcome early in development, arguably thanks to other differences of this intervener (which was mismatched in number). Children with ASD performed less well than the TD children although they were matched on non-verbal reasoning. Overall, syntactic complexity affected their performance in a similar way as in their TD controls, but their performance correlated with non-verbal abilities rather than age, suggesting that their grammatical development does not follow the smooth relation to age that is found in TD children.