Department of Energy Argonne National Laboratory Office of Science NEWTON's Homepage NEWTON's Homepage
NEWTON, Ask A Scientist!
NEWTON Home Page NEWTON Teachers Visit Our Archives Ask A Question How To Ask A Question Question of the Week Our Expert Scientists Volunteer at NEWTON! Frequently Asked Questions Referencing NEWTON About NEWTON About Ask A Scientist Education At Argonne Statistical Significance
Name: armin Brott
Status: N/A
Age: N/A
Location: N/A
Country: N/A
Date: N/A 


Question:
Is there a basic percentage of population one needs to have in order for survey results to be statistically significant?

In other words, if there are a million people visiting my website, how many surveys would I have to send out and what kind of response rates would I need to get any meaningful results?

Would those numbers be the same if the population is smaller?



Replies:
Eric Tolman
Computer Scientist

I can not give you a very definitive answer such as you need to send surveys to 0.0001% of the people to get significant results. That generally is not true.

There are a few guidelines that can be helpful however.

Generally, the number of surveys you collect is not going to be the most important factor in determining if your survey is accurate.

More important factors include:
1. Who is represented by the surveys?
2. How was the information collected?
3. Are the questions clear and unbiased?

1. For example, suppose you collect 1000 surveys, but 90% of the surveys are from old, single females, but most of the people who visit your web site really are teen age males. Your survey will tell you nothing useful at all.

2. How the information was collected is also very deterministic. If you ask for volunteers, you are only going to get people who like to voluteer for things. If you offer a free prize you are going to get a lot of people only interested in prizes. It is a litter harder to track what kind of personality the survey represents, but this is generally represented by how the survey was collected. You want to find a way that gets as many different kinds of personalities to respond--people that you are interested in. Many compainies have a survey as part of the buying, registration, or warranty process because these are the people they are most interested in--people who are making money for them.

By making it part of the process to get a warranty or registration, people tend to fill out the survey because they feel they would miss out on a vital part of their service if they don't complete the process.

3. Finally, the questions have to be clear and neutral. If you want useful information, you can not ask questions such as, What about myproduct do you feel makes it superior to competingproduct? This does not allow the person to answer: Your product really needs improvement! Ask questions that are meant to just try and get information, good and bad from people. Don't try and force them into any other answer.

The number of surveys should be sufficient to give you a good cross-section of people, generally a several hundred to a few thousand should be sufficient. Unless all your surveys are computer tabulated so there are no errors, and you can guarantee you are getting a good cross section, it becomes difficult to keep the surveys accurate with larger numbers--because you will either get errors in tabulation, or you will have too many people from a specific group. For example, if you have a question that allows the person to give a suggestion, and you have 200,000 surveys, how do you make sure everyone gives a valid suggestion. With a few hundred surveys, it is much easier to make sure all the questions are answered.

Hope this helps.

--Eric Tolman


This is a subject that can be debated. Though I will not try to truly answer your question, I would suggest that there are many good books that have chapters on the subject - sampling theory. You will be surprised how small the sample needs to be, but the catch is that it must be random (or in come cases representative). A survey of visitors to your web sight might well not satisfy this and therefore be of limited use.

Larry Krengel



Click here to return to the Mathematics Archives

NEWTON is an electronic community for Science, Math, and Computer Science K-12 Educators, sponsored and operated by Argonne National Laboratory's Educational Programs, Andrew Skipor, Ph.D., Head of Educational Programs.

For assistance with NEWTON contact a System Operator (help@newton.dep.anl.gov), or at Argonne's Educational Programs

NEWTON AND ASK A SCIENTIST
Educational Programs
Building 360
9700 S. Cass Ave.
Argonne, Illinois
60439-4845, USA
Update: June 2012
Weclome To Newton

Argonne National Laboratory