 |
 |
Statistical Significance
Name: armin Brott
Status: N/A
Age: N/A
Location: N/A
Country: N/A
Date: N/A
Question:
Is there a basic percentage of population one needs to
have in
order for survey results to be statistically significant?
In other words, if there are a million people visiting my website,
how many surveys would I have to send out and what kind of response
rates would I need to get any meaningful results?
Would those numbers be the same if the population is smaller?
Replies:
Eric Tolman
Computer Scientist
I can not give you a very definitive answer such as you need to send surveys
to 0.0001% of the people to get significant results. That generally is not
true.
There are a few guidelines that can be helpful however.
Generally, the number of surveys you collect is not going to be the most
important factor in determining if your survey is accurate.
More important factors include:
1. Who is represented by the surveys?
2. How was the information collected?
3. Are the questions clear and unbiased?
1. For example, suppose you collect 1000 surveys, but 90% of the surveys
are from old, single females, but most of the people who visit your web site
really are teen age males. Your survey will tell you nothing useful at all.
2. How the information was collected is also very deterministic. If you
ask for volunteers, you are only going to get people who like to voluteer
for things. If you offer a free prize you are going to get a lot of people
only interested in prizes. It is a litter harder to track what kind of
personality the survey represents, but this is generally represented by how
the survey was collected. You want to find a way that gets as many
different kinds of personalities to respond--people that you are interested
in. Many compainies have a survey as part of the buying, registration, or
warranty process because these are the people they are most interested
in--people who are making money for them.
By making it part of the process to get a warranty or registration, people
tend to fill out the survey because they feel they would miss out on a vital
part of their service if they don't complete the process.
3. Finally, the questions have to be clear and neutral. If you want useful
information, you can not ask questions such as, What about myproduct do you
feel makes it superior to competingproduct? This does not allow the person
to answer: Your product really needs improvement!
Ask questions that are meant to just try and get information, good and bad
from people. Don't try and force them into any other answer.
The number of surveys should be sufficient to give you a good cross-section
of people, generally a several hundred to a few thousand should be
sufficient. Unless all your surveys are computer tabulated so there are no
errors, and you can guarantee you are getting a good cross section, it
becomes difficult to keep the surveys accurate with larger numbers--because
you will either get errors in tabulation, or you will have too many people
from a specific group. For example, if you have a question that allows the
person to give a suggestion, and you have 200,000 surveys, how do you make
sure everyone gives a valid suggestion. With a few hundred surveys, it is
much easier to make sure all the questions are answered.
Hope this helps.
--Eric Tolman
This is a subject that can be debated. Though I will not try to truly
answer your question, I would suggest that there are many good books that
have chapters on the subject - sampling theory. You will be surprised how
small the sample needs to be, but the catch is that it must be random (or in
come cases representative). A survey of visitors to your web sight might
well not satisfy this and therefore be of limited use.
Larry Krengel
Click here to return to the Mathematics Archives
| |
Update: June 2012
|
|
