How many survey responses do you need?

There are a lot of questions you can have about survey responses. Are you receiving enough? Do you need more? Are you receiving fewer than expected? Or do you even have too much data?

The short answer to all those questions is "It depends", but even so, we have done our best to answer them below.

How many responses do you need?

Well, that depends on how you are going to use the data. If you're mainly interested in getting a general feel for the user experience on the website, you should have at least a few hundred responses for the length of time you're interested in. If you're only interested in data per year, that means a few hundred answers per year. If you're interested in data per month, that means a few hundred per month.

However, if you're also interested in creating segments for certain visitor categories, for example, you should instead ideally have at least a couple of hundred answers per selected visitor category for the length of time you want to look at. Which means that you'll need more answers. For example, you may be interested in the visitor category "Customers", which is 10% of the respondents. If you want to have 200 responses for "Customers", you'll need 2,000 responses in total.

If you're unable to reach a couple of hundred responses for the visitor category during the selected time period, 50 responses will at least show you the tendencies.

How many responses can you expect?

Based on the many surveys we at Extellio have worked with, a general survey with an introduction page asking for participation has a response rate of about 4% and a general survey without the introduction page has a response rate of about 10%. If you have 10,000 visitors per month, of which 60% remain when the survey is triggered (some visitors will bounce or leave sooner), and 10% of them answer the survey, you'll get 600 responses per month.

For websites with little traffic, we therefore recommend not to have a separate introduction page (the default setting when creating a survey is to not have an introduction page).

How do you encourage more responses?

The main factor controlling the number of survey responses is the number of visitors on the website, but there are still a few things you can do to increase the number of responses.

Remove the introduction page

If you have an introduction page, the easiest way to increase the number of responses (and usually double them) is to remove the introduction page. When you remove the introduction page, the potential respondents are able to see all the questions (except for dependent follow-up questions) directly, which makes them more likely to answer the survey.

Reduce the trigger time

A shorter trigger time means that fewer website visitors will have left the website, and therefore that more visitors will be asked to answer the survey. However, if you have questions such as "Did you find what you were looking for?", a shorter trigger time will most likely decrease the share of "Yes" as the visitors won’t have had sufficient time to find what they were looking for before the survey was triggered. In other words, a shorter trigger time may give more responses but also lower quality data (depending on what questions you ask).

Use a different survey type

An embedded targeted survey can have four times the response rate of a general survey. It can therefore be an option if you need more answers. However, you shouldn't ask as many questions in an embedded targeted survey as you can in a general survey. This change will therefore also affect the data, but not the quality. Therefore, it may be better as a complement than a full replacement.

Can you collect too many responses?

No, you can't collect too many responses. But collecting every answer you can may be unnecessary. It can be tempting to think that the data quality increases with more answers, but as soon as your survey reaches a representative sample of the population (e.g. website visitors), the difference between 3,000 responses and 30,000 responses may be negligible. If 3,000 and 30,000 give the same results, you only need to collect 3,000.

Why shouldn't you collect every answer?

There are a few reasons for why you shouldn't collect every answer you can get. The first reason is mentioned above - more answers do not necessarily increase the data quality. Second, it can make the analysis more difficult. Categorizing thousands of text answers is very time consuming, and a waste of time if you would've gotten the same results from half as many responses.

Furthermore, some website visitors will find the survey annoying, so to reduce the portion of visitors that trigger the survey means annoying fewer visitors. But it's also polite of you to not ask for your visitors' time when you don't need it.

When should you reduce the trigger portion?

If you haven't started your survey yet, use the calculation example above to calculate how many responses your survey should receive. Of course, this will only be an estimation. Reduce the portion if it seems you will receive more answers than needed.

You can also change the trigger portion at any point (but if you do, make a note of it! It can affect the data, and then you'll want to know when this change was made). That means that you can also let the survey collect data for a month before deciding if you should reduce the portion.

It will also be easier to determine whether a smaller portion is needed when you have familiarized yourself with your data (depending on how used you are to working with statistical data, that may be a couple of months or a year later). When you know what to expect from the data (especially from text answers) it will be easier to tell if you collect more data than you need.