Best practice in questionnaire design

Agree-disagree response options may introduce acquiescence bias, which is the tendency to agree with an item regardless of its content (Wright, 1975). Asking respondents to rate their level of agreement to different statements can be cognitively demanding, which increases respondent error and reduces respondent effort (Fowler, 2009). Instead, use verbally labelled response options that reinforce the underlying topic (e.g., the responses for “How happy are you?” would be not at all happy, slightly happy, somewhat happy, quite happy, extremely happy). Empirical evidence demonstrates that agree-disagree response options diminish item quality (Saris, Revilla, Krosnick, Schaeffer, & Shaeffer, 2010), and are among the “worst ways to present items” (Gehlbach & Artino Jr., 2018, p. 361).

Use verbal labels for each response option, rather than labelling only the end points of the response options or labelling with both numbers and verbal labels. This helps to focus the attention of the respondent and reduce measurement error (Artino, Jr. & Gehlbach, 2012).

Ask about one idea at a time rather than using double-barrelled items, which ask about two or more ideas in the same question (e.g., instead of asking, “How happy and engaged are you?” ask two questions, one about happiness and one about engagement). If you use double-barrelled items, you risk students responding to only one part of that item (Dillman, Smyth, & Christian, 2014)

Phrase questions with positive language rather than using reverse scored or negative language, which students tend to have trouble understanding. Negative words are more difficult to process cognitively, which leads these items to take longer to answer and leads to misresponses (Swain, Weathers, & Niedrich, 2008).

Use at least five response options per scale to capture a wider range of perceptions. Research indicates that the “sweet spot” of the number of response anchors is about five (Weng, 2004; Nielsen, Makransky, Vang, & Danmeyer, 2017). A five-item scale that assesses a representative cross-section of a student’s experience should improve measurement (Gehlbach & Artino Jr., 2018).

Maintain equal spacing between response options, and use additional space to visually separate non-substantive response options. This will reinforce the notion that conceptually, there is equal distance between each response option, which yields less biased responses. Moreover, this will help align the visual midpoint with the conceptual midpoint, reducing measurement error (Artino, Jr. & Gehlbach, 2012). This is especially important if you are administering your questionnaire on paper. Electronic questionnaire administrators such as Qualtrics will space response options equally, and you will have to be aware to add an extra space to separate non-substantive response options (e.g., ‘N/A’). To see some examples, check out the resources for evaluating self-efficacy and take a look at this visual guide).

This will increase the likelihood that respondents will answer these questions whilst they are focused and have energy.

Make sure that item content applies to each respondent and is worded accessibly, or you risk alienating respondents to whom the item does not apply (Dillman, Smyth, & Christian, 2014).

Scales bolster accuracy, with each question addressing a representative cross-section of the experience.

This helps maintain clarity for the respondent, who can then complete the questionnaire more efficiently.

Respondents will feel more comfortable sharing this information later on in the questionnaire.