Using constructed-response and multiple-choice questions in undergraduate examinations

This paper looks at the use of constructed-response and multiple-choice questions in first year management examinations involving 2364 students over a three semester period in 2008-9. It also compares student’s performance in research and analytical assignments with their results in Multiple-Choice tests. The results show that students who perform poorly in non-multiple-choice tests are slightly advantaged by multiple-choice tests. However, students who display high distinction scores in other pieces of assessment do not receive any advantage from multiple-choice testing. Interestingly, the research also highlighted that female students perform better in their articulation of management techniques overall, but do not gain the same comparative advantage from multiple choice testing. The analysis of the assessment methods was extremely useful in predicting the level of failures for the unit in the subsequent year. Finding the right balance of assessment tasks in first year university business courses is extremely difficult, using multiple-choice tests can be useful tool as part of the assessment mix in management units.

Cluster housing and the freedom of choice: a response to Emerson (2004)

Comments on the study of people with an intellectual disability. Inferiority of the standard of various objectively measured variables for the people living in cluster housing; Behavior of people living in cluster housing; Consideration of a study that compared pensioners with socialites on the extent of their social networks.

An Australian study comparing the use of multiple-choice questionnaires with assignments as interim, summative law school assessment

o the author’s knowledge, this is the first Australian study to empirically compare the use of a multiple-choice questionnaire (MCQ) with the use of a written assignment for interim, summative law school assessment. This study also surveyed the same student sample as to what types of assessments are preferred and why. In total, 182 undergraduate property law students participated in this study. Results showed that scores for the MCQ (assessing five topics) and assignment (assessing one topic) followed a similar distribution. This indicates that an MCQ does not necessarily skew students towards higher grades than an assignment. Results also showed significant but low correlations of test scores across instruments. When asked which instrument best assessed their knowledge of property law, students expressed a strong preference for an assignment over an MCQ or examination. Comments revealed a strong belief that, because lawyers write, law schools must assess legal writing – a skill not captured by MCQs. This study is important as many Australian law schools face increasing marking loads due to higher student numbers and compulsory mid-term assessments. This article endorses the use of MCQs but only as part of a diverse suite of law school assessment.

Assessment method difference : comparisons between international and domestic students within a first year undergraduate management course

This paper investigates the assessment methods and student results within a first year undergraduate management course offered within the business faculty of an Australian university. This course is compulsory for those studying for a commerce or management degree. The assessment results of full fee paying international students were compared with those of domestic students, during four teaching semesters in 2009 and 2010. Analysis compares 2,682 students’ numerical results for two constructed response assignments to their results for an examination comprising both multiple choice questions and constructed response questions. It also compares the results of international and domestic students across metropolitan, regional and rural campuses. However due to little comparison data for multiple campuses, findings are consolidated by domestic and international students, university-wide. International students were found to achieve lower results than domestic students for constructed response assessment tasks, but higher results than domestic students for multiple choice question assessments. These findings have implications for instructors eager to provide a level assessment playing field for both domestic and international students, enabling both groupings to take advantage of existing strengths but also to improve their weaknesses. This research led to a restructuring and rescheduling of assessment tasks for the 2012 academic year.

Design, format, validity and reliability of mutiple choice questions for use in nursing research and education

Multiple choice questions are used extensively in nursing research and education and play a fundamental role in the design of research studies or educational programs. Despite their widespread use, there is a lack of evidence-based guidelines relating to design and use of multiple choice questions. Little is written about their format, structure, validity and reliability of in the context of nursing research and/or education and most of the current literature in this area is based on opinion or consensus. Systematic multiple choice question design and use of valid and reliable multiple choice questions are vital if the results of research or educational testing are to be considered valid. Content and face validity schould be established by expert panel review and construct validity should be established using ‘key check’, item discrimination and item difficulty analyses. Reliability measures include internal consistency and equivalence. Internal consistency should be established by determination of internal consistency using reliability coefficients while equivalence should be established using alternate form correlation. This paper reviews literature related to the use of multiple choice questions, current design recommendations and processes to establish reliability and validity, and discusses implications for their use in nursing research and education.

Response rates in Australian market research

RICA commissioned Deakin University to “establish whether response rates are in decline in the Australian market research industry and to identify, as far as possible, the reasons for these declines if they exist. This is likely to involve a review of previous research, a literature review and collection of data on response rates provided on a confidential basis and with the assistance of AMSRO to facilitate data provision.”

Attempts were made to contact all listed market research companies in Australia, including all major internet panel companies. While industry co-operation was not high with the study, sufficient data was provided to depict current response rates and to show how they had declined over time. Because of the low contactability issues, this Report proposes the use of better methods to compute the reliability of survey estimates by taking account of past survey results.

The literature review revealed a wealth of recent studies, with the main emphasis being on studies of telephone and internet surveys. This review of the research produced 34 evidence-based guidelines for social researchers. While some of these reflect current practice, the emergence of the internet as the main survey method raises a number of disclosure and sampling issues. Esomar (2012) has produced 28 issues to be raised with providers, which sets the basis for good industry practice. This suggests an opportunity for the industry to adopt these standards as its own and to conduct training courses for major clients and suppliers. There are many panel providers, some of whom are not AMSRO members. AMSRO may need to examine what role it can play in mandating or encouraging adherence to these standards as a way of promoting the industry.

Talks with key industry people, as well as the literature, have revealed the importance of blended surveys, where multiple contact and response mechanisms are used. Particularly where an internet panel is used as one source, this poses representativeness and weighting issues which are difficult to resolve. The Report recommends that where blended survey methods are used, measures be taken to measure contactability in the other contact media, along with more sophisticated weighting schemes. The industry should examine its training courses to ensure that industry expertise keeps pace with these developments.
Summary of Results

The results focus on two main collection methods – the telephone and the internet. As far as the telephone is concerned response rates have been in a gradual decline the last decade. This outcome is hard to detect because the data show considerable fluctuations from one survey wave to the next. Among cold-calling surveys, telephone response rates are typically below 10%, for a range of topics and survey types. Co-operation rates, (the ratio of obtained interviews to refusals) are typically below 0.2 (that is below one interview to five refusals). Telephone interviews with clients have a higher response rate – typically above 20% with co-operation rates above 1.0. It would appear that some topics, such as financial services, may induce a lower level of co-operation. Government sponsored surveys have higher response rates, at times over 50%, but even here a sharp decline in response rates over time for one long running monitor was observed. Co-operation rates were also higher in government sponsored surveys.

One long data series from a telephone omnibus suggested that the “Do Not Call Register” which began in May, 2007 had some positive effects for the industry. Initially there was a spike in both response rates and co-operation. Although this was relatively short-lived, response rates thereafter declined more slowly and co-operation rates were somewhat higher and remained stable. These conclusions should be regarded as tentative as more data series would really be necessary to see if similar trends occurred elsewhere.

As far as the internet is concerned, panel response rates are around the 20% mark and appear to be relatively stable over the last few years. In this case, the gross response rate is the number of interviews divided by the number of invitations sent. As the number of invitations may be a function of the need to fill a survey quickly, it should be considered a gross indicator of response. In order to capture this phenomenon, a further measure has been devised, termed the “attempt rate” which measures the percentage of people who attempt to participate once sent the invitation. The available data suggests that it is relatively stable. However, it is also somewhat susceptible to the time the survey was left open. Finally, a co-operation rate was also calculated. It measures the ratio of completed to terminated interviews, typically at least five interviews to each termination, but often much higher. This measure is not directly comparable with the co-operation rate in telephone surveys because it cannot take account of the number of eligible people on the panel who open the invitation, see the company doing the survey or its length and decide not to take the survey. For internet client studies, response rates were typically somewhat higher than shown for the panels, but there was marked variability.

There was only one study provided of intercept interviews. It showed response rates of over 60% and co-operation rates of nearly 2 interviews per refusal. A strength and a weakness of intercept interviewing is the ability to be selective in who is asked to participate. As for mail, one government sponsored mail survey from 2010 is reported, with a response rate over 50%. The previous review contains more data, as mail appears to be infrequently used within the industry for commercial surveys.

While surveys remain a major and highly effective tool for the industry and its clients, issues with contactability and co-operation mean that even closer attention is needed to survey design, sampling, weighting and analysis than was previously the case.

Is demography destiny? Application of machine learning techniques to accurately predict population health outcomes from a minimal demographic dataset

For years, we have relied on population surveys to keep track of regional public health statistics, including the prevalence of non-communicable diseases. Because of the cost and limitations of such surveys, we often do not have the up-to-date data on health outcomes of a region. In this paper, we examined the feasibility of inferring regional health outcomes from socio-demographic data that are widely available and timely updated through national censuses and community surveys. Using data for 50 American states (excluding Washington DC) from 2007 to 2012, we constructed a machine-learning model to predict the prevalence of six non-communicable disease (NCD) outcomes (four NCDs and two major clinical risk factors), based on population socio-demographic characteristics from the American Community Survey. We found that regional prevalence estimates for non-communicable diseases can be reasonably predicted. The predictions were highly correlated with the observed data, in both the states included in the derivation model (median correlation 0.88) and those excluded from the development for use as a completely separated validation sample (median correlation 0.85), demonstrating that the model had sufficient external validity to make good predictions, based on demographics alone, for areas not included in the model development. This highlights both the utility of this sophisticated approach to model development, and the vital importance of simple socio-demographic characteristics as both indicators and determinants of chronic disease.

Fusing data mining, machine learning and traditional statistics to detect biomarkers associated with depression

BACKGROUND: Atheoretical large-scale data mining techniques using machine learning algorithms have promise in the analysis of large epidemiological datasets. This study illustrates the use of a hybrid methodology for variable selection that took account of missing data and complex survey design to identify key biomarkers associated with depression from a large epidemiological study.

METHODS: The study used a three-step methodology amalgamating multiple imputation, a machine learning boosted regression algorithm and logistic regression, to identify key biomarkers associated with depression in the National Health and Nutrition Examination Study (2009-2010). Depression was measured using the Patient Health Questionnaire-9 and 67 biomarkers were analysed. Covariates in this study included gender, age, race, smoking, food security, Poverty Income Ratio, Body Mass Index, physical activity, alcohol use, medical conditions and medications. The final imputed weighted multiple logistic regression model included possible confounders and moderators.

RESULTS: After the creation of 20 imputation data sets from multiple chained regression sequences, machine learning boosted regression initially identified 21 biomarkers associated with depression. Using traditional logistic regression methods, including controlling for possible confounders and moderators, a final set of three biomarkers were selected. The final three biomarkers from the novel hybrid variable selection methodology were red cell distribution width (OR 1.15; 95% CI 1.01, 1.30), serum glucose (OR 1.01; 95% CI 1.00, 1.01) and total bilirubin (OR 0.12; 95% CI 0.05, 0.28). Significant interactions were found between total bilirubin with Mexican American/Hispanic group (p = 0.016), and current smokers (p<0.001).

CONCLUSION: The systematic use of a hybrid methodology for variable selection, fusing data mining techniques using a machine learning algorithm with traditional statistical modelling, accounted for missing data and complex survey sampling methodology and was demonstrated to be a useful tool for detecting three biomarkers associated with depression for future hypothesis generation: red cell distribution width, serum glucose and total bilirubin.

Does school health and home economics education influence adults' food knowledge?

Home economics and health teachers are to be found in many parts of the world. They teach students about food in relation to its nutritional, safety and environmental properties. The effects of such teaching might be expected to be reflected in the food knowledge of adults who have undertaken school education in these areas. This study examined the food knowledge associations of school home economics and health education among Australian adults. Two separate online surveys were conducted nationwide among 2022 (November 2011) and 2146 Australian adults (November-December 2012). True/false and multiple choice questions in both surveys were used to assess nutrition, food safety and environmental knowledge. Knowledge scores were constructed and compared against respondents' experience of school health or home economics education via multiple regression analyses. The results from both studies showed that home economics (and similar) education was associated with higher levels of food knowledge among several age groups. The associations of home economics education with food knowledge differed across several Australian states and recall of home economics themes differed across the age groups. These findings suggest that home economics education may bring about long-lasting learning of food knowledge. Further research is required, however, to confirm the findings and to test the causal influence of home economics education on adults' food knowledge.

A survey of first-year university students`ability to use spreadsheets

Universities are using more information and communication technologies (ICT) in their teaching and learning environments. An anonymous multiple-choice survey self-assessed the spreadsheet skills of students enrolled in first-year units at the beginning of 2003. The results of the survey indicate significant deficiencies in the use of spreadsheets. There is a significant proportion of students who are unable to use spreadsheets as part of their education at the start of their university studies. The implications for tertiary education are discussed.

Learning approaches: examination type, discipline of study, and gender

A sample of 248 Australian university students participated in a study that investigated the influence of assessment type and discipline of study on students' learning approaches. Participants (93 psychology and 155 business students) responded to the Study Process Questionnaire in a context of preparation for a hypothetical examination. The hypothetical examination is in either a multiple-choice or an essay format. Results indicated that assessment type had no significant influence on how students approach their learning, while discipline did have significant influence on student learning.

Accessible telecommunications information for people with communication impairment

Study investigating the telecommunication information needs of people with communication disabilities in Australia. Participants were informed about the project through flyers, letters, e-mail, disability agency contacts, and Web sites. A survey with multiple choice and closed and open-ended items was developed. People with communication disabilities used a hard-copy survey format to facilitate completion. Sixty-five participants age eighteen and over from Victoria, Tasmania, South Australia, and Queensland completed the survey. Preliminary results of the study indicated that the participants requested six text-based adaptations: (1) make information clear and easy to read and understand, (2) use larger print, (3) highlight the key points, (4) use dot points, (5) use visual information, such as photos and communication symbols, and (6) provide a range of oral/audio and visual formats for information. The accessibility characteristics requested by the participants called for the development of text and Web-based formats, and for the development of inclusive design guidelines. The authors concluded that further investigation was required to determine the best possible method of making the information accessible.

Mathematical knowledge of some entrants to a pre-service education course

Standards for teaching emphasise the need for teachers to have deep content knowledge. To assess the mathematical knowledge of students enrolling in its B.Ed. program, the University of New England has introduced a mathematics diagnostic test. This work is the first stage of an ongoing research project into the numeracy needs of students entering the B.Ed. program. The test is a pen-andpaper test that replaces previous on-line, multiple-choice tests. This paper reports on the test results, discusses some common errors made by students and outlines the future direction of the research.