1 Comment
Guest *Julia Silge* @ 2018-02-22 18:50:53 originally posted:
The options were randomized in order for every respondent! So we have that at least. I agree that this is almost too many options for a ranking question, and that it might be better have respondents instead choose most/least from such a list.
And certainly with surveys like these it is important to reflect what kind of sample we are likely measuring. Is it all developers? Is it our users? Our heavy users? In what ways are those different? I think about this in the context of, for example, women's participation. This is very similar to many of the challenges of the "big data" era. Sure, the sample is big, but what is the sample measuring?
yihui 2022-12-17 03:09:32Wow! Nice job! It is great to know that you have thought about both issues (randomization of choices and sample representation) so carefully.
It just occurred to me this morning that I actually struggled even more with the last year's R Consortium Survey : https://www.r-consortium.org/blog/2017/06/29/take-the-r-consortiums-survey-on-r They said it would only take about 10 minutes, but some questions were very difficult to me, and even if I ignored quite a few questions, it still took me more than 20 minutes. I didn't give up and made it to the end only because I was "loyal" to R. I imagine there should be a fair amount of respondents who were like me: they filled out the questionnaire because they cared deeply about R, so presumably we would lose information about those who don't really care and were scared away by the difficult questionnaire.
Perhaps sometimes we are just too ambitious when designing surveys and want to squeeze every bit of information out of the respondents :)
Originally posted on 2018-02-22 19:31:21
Sign in to join the discussion
Sign in with GitHub