Questionnaire Design — Order Effects and Order Bias
Order Effects
The question sequence must be
carefully considered to avoid order effects — biases caused by the
sequencing of the questions. Order effects occur when a question earlier in the
study has an influence on the response to one or more questions that follow.
Take for example, the sequence of questions listed below:
- Please rate the colour of this drink.
- Please rate the smell of this drink.
- Please rate the amount of cocoa in this drink.
- Please rate the amount of sugar in this drink.
- Please rate the size of chocolate chips in this drink.
- What is your overall rating of this drink?
Placing it at the end, a common error, biases the
response to the question on “overall rating”. If for instance, respondents have
a positive (or a negative) feeling for the attributes colour, smell, amount of
cocoa, etc. it will influence their response on overall rating. For this reason,
questions on overall rating of product, service or experience should precede
questions about the various attributes that influence the overall rating.
We also need to be mindful of halo effects. If the
respondent has a positive (or a negative) feeling for the first one or two
attributes (colour, smell), this might cause a halo affecting her response to
some of the subsequent attributes (amount of cocoa, amount of sugar, etc.). The
halo effect may be contained by randomizing the sequence of the questions that
appear on attribute rating.
Order effects can also occur when the text of a
question reveals previously unknown facts to respondents. For example, the
question “Do you use Timotei shampoo?” makes the respondents aware of the brand
Timotei. Hence unaided brand awareness questions
are placed near the start of the questionnaire, and questions on factual and
behavioural information go before questions relating to attitudes and opinions.
For instance, in advertising tracking, brand awareness questions precede questions
on purchase behaviour and brand image, and these are followed by questions
pertaining to advertising awareness and
diagnostics.
Order Bias
Besides the sequencing of questions,
the sequencing of choices may also generate bias. This is referred to as order
bias — it may occur when a respondent is given the opportunity to select
answers in multiple choice questions. For instance, given a choice of flavours,
colours, ingredients and so on, respondents have the tendency to select items
that appear at the top or the bottom of the list. Order bias may be eliminated
by randomizing the sequence of the items in the list so that each has the same
likelihood of being in any given position in the list.
