In a recent episode we talked through designing a decent survey, but of course that is only one part of the bigger picture. To boost the number of responses to our surveys, should we offer an incentive to our potential survey respondents? If we were to offer an incentive, what should it be? In this episode we’ll explore what the research says about this!
There has actually been some useful research on this recently. When we did a quick search looking for journal articles to see what had been published, a few studies popped up, and specifically ones focused on the farming community. One study we found (Avemegah et al., 2021) looked at three different surveys sent to agricultural producers in South Dakota in 2018 and 2019. The researchers tried a few different approaches. In one survey, they randomly sent half the survey recipients a $2 note with the survey. In another, they followed up with two email reminders, as well as experimenting with whether to have an official logo on the contact material. In the third survey, they randomly selected half the recipients to receive both an emailed and a mailed survey—so some farmers would have received both a hard copy and an electronic version. The other half of the respondents received the mail survey and email at different times. What did they find? Well only the $2 note actually resulted in a significantly different response rate!
We think this is fascinating! In a similar recent study, other researchers in the US carried out just the experiment of mailing $2 with a survey. This was a survey of farmers in the mid-West region. They set this up by mailing out a letter in advance, followed by the mail survey, a reminder postcard, a second mailed survey, and finally a third survey with a postcard. Respondents also had the option to complete it online. They randomly included a $2 note in half of the surveys. And what did they find? When they crunched the numbers, they found that those who received the money were significantly more likely to have completed the survey! You can read the details in the paper by Glas et al. (2019) which we have added in the references below.
The other recent paper we found on this topic, highlighted the decline in response rates of agricultural and rural surveys. A Norwegian group of researchers (Zahl-Thanem et al., 2021) quote a decline of nearly 1% for each year from 1971 to 2017. No one is quite sure why, but it does start ringing alarm bells for those of us who rely on responses to surveys for our work! These researchers also point out the bias that could creep into results because of this decline and they spend some time exploring this. We are n’t going to focus on response bias in this post but if you are interested in that detail, have a read of their paper.
Those authors were not as focused on incentives but they did explore the difference in response from mail based surveys versus email based surveys with farmers. What they did was to follow their standard practice of sending out mail-based surveys to one group. Simultaneously, they also sent out the same number of surveys by email, with an online survey link, to a different group of farmers. When they received the surveys back, they compared response rates—and there was quite a difference. The response rate for the mail survey was 41% but the email survey was only 21%. They concluded that it is worth continuing with mail surveys for the foreseeable future, given that they would have to send out more and more emails to get a sufficient number of responses.
So what have we learnt so far? We can conclude that monetary incentives can be effective, and a mail survey will yield a higher response rate, at least for farmers. So what about non-monetary incentives? Well we are glad you asked, because there is a study that investigated the impact of providing chocolate (yes, chocolate!) as an addition to a mail survey. [Denise thinks this is one of the best studies she has read about!] Fairweather (2010), a New Zealand researcher, went to the trouble of exploring this. Unfortunately, the results indicate that providing chocolate did not impact response rates. We think this means that if you are going to use chocolate when researching, save it for the researchers when they are analysing the survey responses!
The final bit of research we want to highlight was done back in 2002. Pennings et al. (2002), US-based researchers again, experimented with farmers, trying a range of different methods to understand the response rates they received. They highlighted a couple of key points. Firstly, there are good times and bad times to send surveys to farmers. If it is the time of the year when farmers are particularly busy, it really does not matter what you do to try and increase survey response rates, as responding to a survey just is not high on their priority list! Secondly, they found that farmers were only willing to spend a short time filling out a survey. About ten minutes was the limit! Interestingly, they also explored monetary incentives and found that $15 was the sweet spot for influencing response rates.
Now you may have noticed that we have focused on research that involved farmers. There is a huge amount of literature exploring incentives and surveys, but we wanted to start with some that was particularly relevant to enablers of change. Hopefully we have covered some useful ideas in this post. We have highlighted that monetary incentives do help increase response rates, but that there are limits to this if our survey gets too long!
Well, you have read our thoughts, now we would like to hear yours! Add a comment below the blog post and tell us about your experiences with surveys and getting good response rates, including any tips and further ideas about it. We do not want this to be just a one-way conversation—join in by sharing your thoughts and ideas with us!
Thanks folks for reading this Enablers of change blog post. Remember to subscribe to our newsletter if you would like to know when new episodes are available. And if you liked what you heard, please tell your friends so they can join the conversation!
Avemegah, E., Gu, W., Abulbasher, A., Koci, K., Ogunyiola, A., Eduful, J., … & Ulrich-Schad, J. D. (2021). An Examination of Best Practices for Survey Research with Agricultural Producers. Society & Natural Resources, 34(4), 538-549. Available online.
Fairweather, J. R. (2010). The effectiveness of a chocolate incentive in a mail survey of New Zealand farmers. Marketing Bulletin, 21. Available online.
Glas, Z. E., Getson, J. M., Gao, Y., Singh, A. S., Eanes, F. R., Esman, L. A., … & Prokopy, L. S. (2019). Effect of monetary incentives on mail survey response rates for midwestern farmers. Society & Natural Resources, 32(2), 229-237. Available online.
Pennings, J. M., Irwin, S. H., & Good, D. L. (2002). Surveying farmers: A case study. Applied Economic Perspectives and Policy, 24(1), 266-277. Available online.
Zahl-Thanem, A., Burton, R. J., & Vik, J. (2021). Should we use email for farm surveys? A comparative study of email and postal survey response rate and non-response bias. Journal of Rural Studies, 87, 352-360. Available online.