Sensitive Questions in Online Surveys: An Experimental Evaluation of the Randomized Response Technique and the Crosswise Model

Self-administered online surveys provide a higher level of privacy protection to respondents than surveys administered by an interviewer. Yet, studies indicate that asking sensitive questions is problematic also in self-administered surveys. Because respondents might not be willing to reveal the truth and provide answers that are subject to social desirability bias, the validity of prevalence estimates of sensitive behaviors from online surveys can be challenged. A well-known method to overcome these problems is the Randomized Response Technique (RRT). However, convincing evidence that the RRT provides more valid estimates than direct questioning in online surveys is still lacking. A new variant of the RRT called the Crosswise Model has recently been proposed to overcome some of the deficiencies of existing RRT designs. We therefore conducted an experimental study in which different implementations of the RRT, including two implementations of the crosswise model, were tested and compared to direct questioning. Our study is a large-scale online survey (N = 6,037) on sensitive behaviors by students such as cheating in exams and plagiarism. Results indicate that the crosswise-model RRT---unlike the other variants of RRT we evaluated---yields higher prevalence estimates of sensitive behaviors than direct questioning. Whether higher estimates are a sufficient condition for more valid results, however, remains questionable.

Using Factorial Surveys and Stated Choice Experiments to Investigate Discriminatory Attitudes and Preferences

Empirical research on discriminatory attitudes and behaviour grapples with the social undesirability of its object. In many studies using regular survey methods, estimates are biased, and the social context of discrimination is not taken into account. Several methods have been developed, especially to deal with the first problem. In this regard, the estimation of the ‘true value’ of discriminatory attitudes is at the centre of interest. However, methodological contributions focusing on the social context of attitude communication and discriminatory behaviour, as well as the correlation between both, are rare. We present two experimental methods which address those issues: factorial surveys and stated choice experiments. In a first study, the usefulness of factorial surveys is demonstrated with data on German anti-Semitism (N=279). We show that the rate of approval with anti-Semitic statements increases if (a) respondents are told that the majority of fellows agree with such statements, (b) the term “Jews” is replaced by the term “Israelis”, and (c) reference to the Holocaust is made. Apart from the main effects of these experimental factors, significant interaction effects regarding the political attitudes and social status of respondents are observed. In a second study, a stated choice experiment on the purchase of olive oil and tomatoes was conducted in Germany (N=440). We find that respondents prefer Italian and Dutch products (control treatment) compared to Israeli and Palestinian ones (discrimination treatments). There are no significant differences between preferences for a so called ‘Peace product’ (which is produced jointly by Israelis and Palestinians) and products from Italy as well as the Netherlands. Yet, taking discriminatory attitudes (anti-Semitic and anti-Arabic attitudes) into account, a strong correlation between those attitudes and stated behaviour (purchase of Israeli, Palestinian and jointly produced products) can be found. This adds support to the hypothesis that discriminatory attitudes hold behavioural consequences.

Does the Use of Mobile Devices (Tablets and Smartphones) Affect Stated Choice Behavior in Web Surveys? An Application to Environmental Valuation

Web surveys are becoming increasingly popular in survey research. Compared with face-to-face, telephone and mail surveys, web surveys may contain a different and new source of measurement error and bias: the type of device that respondents use to answer the survey questions. To the best of our knowledge, this is the first study that tests whether the use of mobile devices affects survey characteristics and stated preferences in a web-based choice experiment. The web survey was carried out in Germany with 3,400 respondents, of which 12 per cent used a mobile device (i.e. tablet or smartphone), and comprised a stated choice experiment on externalities of renewable energy production using wind, solar and biomass. Our main finding is that survey characteristics such as interview length and acquiescence tendency are affected by the device used. In contrast to what might be expected, we find that, compared with respondents using desktop computers and laptops, mobile device users spent more time to answer the survey and are less likely to be prone to acquiescence bias. In the choice experiment, mobile device users tended to be more consistent in their stated choices, and there are differences in willingness to pay between both subsamples.

The consequences of social intolerance on non-violent protest behavior

This paper scrutinizes the impact of intolerance toward diverse ethnic, religious, and cultural groups on an individuals willingness to actively engage in non-violent protest. Following new insights, we examine the individual as well as the ecological effect of social intolerance on protest behavior. Drawing from insights of social psychology and communication science, we expect that the prevalence of intolerance reinforces the positive effect of individual-level intolerance on protest participation. From a rational choice perspective, however, a negative moderating effect is expected, as the expression of opinions becomes redundant for intolerant individuals in an intolerant society. We base our multilevel analyses on data from the World Values Surveys covering 32 established democracies. Our results reveal that intolerance leads to more non-violent protest participation. This relationship, however, is strongly influenced by the prevalence of intolerance in a country.

Three Experimental Approaches to Measure the Social Context Dependence of Prejudice Communication and Discriminatory Behavior

Empirical research on discrimination is faced with crucial problems stemming from the specific character of its object of study. In democratic societies the communication of prejudices and other forms of discriminatory behavior is considered socially undesirable and depends on situational factors such as whether a situation is considered private or whether a discriminatory consensus can be assumed. Regular surveys thus can only offer a blurred picture of the phenomenon. But also survey experiments intended to decrease the social desirability bias (SDB) so far failed in systematically implementing situational variables. This paper introduces three experimental approaches to improve the study of discrimination and other topics of social (un-)desirability. First, we argue in favor of cognitive context framing in surveys in order to operationalize the salience of situational norms. Second, factorial surveys offer a way to take situational contexts and substitute behavior into account. And third, choice experiments – a rather new method in sociology – offer a more valid method of measuring behavioral characteristics compared to simple items in surveys. All three approaches – which may be combined – are easy to implement in large-scale surveys. Results of empirical studies demonstrate the fruitfulness of each of these approaches.

Does the use of mobile devices (tablets and smartphones) affect survey quality and choice behaviour in web surveys?

Web surveys are becoming increasingly popular in survey research including stated preference surveys. Compared with face-to-face, telephone and mail surveys, web surveys may contain a different and new source of measurement error and bias: the type of device that respondents use to answer the survey questions. This is the first study that tests whether the use of mobile devices, tablets or smartphones, affects survey characteristics and stated preferences in a web-based choice experiment. The web survey on expanding renewable energy production in Germany was carried out with 3182 respondents, of which 12% used a mobile device. Propensity score matching is used to account for selection bias in the use of mobile devices for survey completion. We find that mobile device users spent more time than desktop/laptop users to answer the survey. Yet, desktop/laptop users and mobile device users do not differ in acquiescence tendency as an indicator of extreme response patterns. For mobile device users only, we find a negative correlation between screen size and interview length and a positive correlation between screen size and acquiescence tendency. In the choice experiment data, we do not find significant differences in the tendency to choose the status quo option and scale between both subsamples. However, some of the estimates of implicit prices differ, albeit not in a unidirectional fashion. Model results for mobile device users indicate a U-shaped relationship between error variance and screen size. Together, the results suggest that using mobile devices is not detrimental to survey quality.

Sensitive Questions in Online Surveys: Experimental Results for the Randomized Response Technique (RRT) and the Unmatched Count Technique (UCT)

Gaining valid answers to so-called sensitive questions is an age-old problem in survey research. Various techniques have been developed to guarantee anonymity and minimize the respondent's feelings of jeopardy. Two such techniques are the randomized response technique (RRT) and the unmatched count technique (UCT). In this study we evaluate the effectiveness of different implementations of the RRT (using a forced-response design) in a computer-assisted setting and also compare the use of the RRT to that of the UCT. The techniques are evaluated according to various quality criteria, such as the prevalence estimates they provide, the ease of their use, and respondent trust in the techniques. Our results indicate that the RRTs are problematic with respect to several domains, such as the limited trust they inspire and non-response, and that the RRT estimates are unreliable due to a strong false "no" bias, especially for the more sensitive questions. The UCT, however, performed well compared to the RRTs on all the evaluated measures. The UCT estimates also had more face validity than the RRT estimates. We conclude that the UCT is a promising alternative to RRT in self-administered surveys and that future research should be directed towards evaluating and improving the technique.

Sensitive Questions in Online Surveys: Experimental Results for the Randomized Response Technique (RRT) and the Unmatched Count Technique (UCT)

Gaining valid answers to so-called sensitive questions is an age-old problem in survey research. Various techniques have been developed to guarantee anonymity and minimize the respondent's feelings of jeopardy. Two such techniques are the randomized response technique (RRT) and the unmatched count technique (UCT). In this study we evaluate the effectiveness of different implementations of the RRT (using a forced-response design) in a computer-assisted setting and also compare the use of the RRT to that of the UCT. The techniques are evaluated according to various quality criteria, such as the prevalence estimates they provide, the ease of their use, and respondent trust in the techniques. Our results indicate that the RRTs are problematic with respect to several domains, such as the limited trust they inspire and non-response, and that the RRT estimates are unreliable due to a strong false "no" bias, especially for the more sensitive questions. The UCT, however, performed well compared to the RRTs on all the evaluated measures. The UCT estimates also had more face validity than the RRT estimates. We conclude that the UCT is a promising alternative to RRT in self-administered surveys and that future research should be directed towards evaluating and improving the technique.