How can a survey of 1,000 people tell us what the nation thinks?

Abel Gustafson & Matthew Goldberg
august 2024

New polling results and the findings from consumer research studies are often portrayed as being an accurate depiction of the beliefs or behaviors of a large population, such as an entire country. But the data typically comes from studying only around 1,000 people. How is this possible? How could the views of such a small group reliably tell us about the beliefs of millions? 

These questions are central to the science of statistics, which—when applied correctly—allow researchers to draw accurate conclusions about very large populations based on relatively small samples. In this article, we’ll explore how and why a well-conducted survey of 1,000 people can indeed provide a reliable snapshot of the opinions and behaviors of an entire country. The answers are found in some simple logic… and in a pot of soup.

What’s a Sample?

When researchers want to know what the general public thinks about a particular issue, product, or policy, it’s usually impractical—and often impossible—to ask literally everyone in the country. So instead, they only study a smaller group out of the population (called a “sample”). But the important thing here is to make sure that the sample is representative: that it does a good job of representing the full population.

Ideally, the sample is a miniature replica of the population. So if the population you’re studying is 49% male and 51% female, then the sample needs to be too. If the population is 11% African American, 23% with a household income greater than $150k, and 38% with a bachelor’s degree, then the sample should be just like that too.

Although the sample is much smaller than the population in absolute size, it is made up of the same types of people—each in the same proportional sizes. Good sampling is the building of a scale model or microcosm of the population. The precision required in this process is one of the reasons why there is such a difference in reliability and accuracy between survey research done by amateurs versus experts.

Why is 1,000 Usually Enough?

Imagine your friend has just made a large pot of soup, and you are going to try it to see if you like it. To find out what the soup tastes like, you don’t have to eat the entire pot. In fact, you don’t even need a whole bowl. Just a little sip from a spoon will usually tell you all you need to know. This analogy tells us a lot about how surveys work when we study a small sample.

We can even take this analogy a little further. If the soup is homogeneous (like a tomato bisque), there is not very much variation throughout the pot. Any spoonful will be the same as every other. In this case, you can taste a very small bit (sample) and still be very confident—due to the lack of variation across the pot (population)—that you have the full picture. There will be no surprises. However, if the soup is heterogeneous (e.g., chicken noodle), then the accuracy of your taste-test might depend on how big of a spoonful you take. You still probably don’t need to eat an entire bowl, but if you happen to take too small of a spoonful you risk missing some of the ingredients that are actually included in the rest of the pot. This is why the size of a sample matters for accuracy. When the population is heterogeneous (e.g., filled with many different types of people), you need to have a large enough sample so that you are capturing as much of those diverse and varying characteristics as possible. 

So how big of a spoonful do we need when doing national surveys? One of the most surprising aspects of statistical sampling is that there are diminishing returns for sample size. The reason behind this is rooted in the concept of sampling error, which is the difference between the results obtained from the sample and the true values in the population. Sampling error decreases as the sample size increases, but it does so at a diminishing rate. For example, increasing your sample size from 100 to 1,000 substantially reduces sampling error, but increasing it further from 1,000 to 10,000 only reduces error by a little more. Statisticians have found that for many types of surveys, a sample size of around 1,000 people is the sweet spot—regardless of if the population size is 100,000 or 100M. This sample size strikes a balance between cost and accuracy. It is large enough to minimize sampling error but small enough to be practical in terms of time and resources. With a sample size of 1,000, the margin of error—a measure of the uncertainty in the results—typically falls around ±3 percentage points, which is usually small enough for practical purposes.

The Power of Randomness

One of the most crucial elements in ensuring that a sample is representative is randomness. Random sampling means that every individual in the population has an equal chance of being selected to participate in the survey. This randomness is the cornerstone of survey sampling because it helps eliminate bias, ensuring that the survey results are not skewed by the overrepresentation of some groups and underrepresentation of others.

Imagine you wanted to survey the opinions of a classroom of 100 students about their favorite hobbies. If you only ask the 15 students sitting in the front row, your results might be biased because students in the front row might consistently have different hobbies than students who sit in the back. But if you randomly pick 15 students from all over the classroom, you’re more likely to get a mix of opinions that better represents the entire class. 

Similarly with the pot of soup, taking your spoonful from just the top might give you a false impression of what the soup contains, because some ingredients may have sunk to the bottom. For the most “representative” spoonful, you would want to stir up the pot first, or perhaps scoop from a few different places. In survey research, we can maximize randomness through a variety of techniques—such as drawing from many different sources, using random digit dialing for telephone surveys, or drawing from a pool of participants that has already been randomly selected.

This Is Only the Tip of the Iceberg

Sample size is only one of many factors to consider. Even when researchers use the best possible sampling methods, there is still some sampling error due to random chance. We often correct for this in the analysis phase by “weighting” the data to correct for any of these random imbalances. For example, if younger people are overrepresented in the sample, their responses would be given slightly less weight when calculating the results to better reflect their true proportion in the actual population.

There is additional uncertainty even when your sample is a perfect scale model of the population. For example, even if your sample matches the population on all characteristics you care about, there will still be some variability among the groups you have surveyed. So if your survey has the correct proportion of Latino respondents, for example, there is still uncertainty remaining because different subgroups of Latinos are likely to vary in the opinions you are trying to measure. 

Also, even the best sampling methods and the largest samples cannot overcome flaws in the design of the survey itself! Poorly worded or leading questions can skew results by influencing how respondents answer. For example, asking, “Do you agree with the widely supported idea of increasing the minimum wage?” might lead to different responses than a more neutrally worded question like, “Do you support or oppose increasing the minimum wage?”

Further, public opinion is not static; it changes over time and in response to events, media coverage, or new information. Therefore, any one survey represents a snapshot of opinions at a specific moment in time. Repeated surveys, or longitudinal studies, are valuable because they can track how public opinion shifts, providing a more dynamic understanding of the population’s views.

In Summary

Due to advances in statistical science, we see that a survey of 1,000 people (if done well) can indeed provide a reliable picture of public opinion for an entire country. The key is in the principles of random sampling, the careful selection of participants, and the thoughtful interpretation of results. By understanding these principles, we can appreciate how a relatively small sample can offer valuable insights into the collective views of a population of millions.

This process is both an art and a science. It requires not just mathematical precision, but also a deep understanding of human behavior, demographics, and the context in which opinions are formed. When conducted properly, surveys serve as powerful tools for understanding public sentiment, guiding policy decisions, and reflecting the voice of the people in a complex and ever-changing world.

cite this

Gustafson, A. & Goldberg, M. H. (2024). How Can a Survey of 1,000 People Tell Us What the Nation Thinks. XandY. New Haven, CT. Retrieved from: https://www.xandyanalytics.com/survey-of-1000-people