Making the Sample Size Decision in Market Research

by Steve Boespflug

 

“What sample size do you recommend?”  This is one of the most common questions we hear from clients who need quantitative research.  My usual reply includes asking more questions like the following:

  • How large is the population (i.e., the target market) that the sample is intended to represent?
  • What market segments need to be adequately represented?  (e.g., geographic region, age, income, industry, revenue, product used, etc.)
  • In the final analysis, do you want to know the differences between market segments?  (e.g., younger age versus older age product preferences)

The purpose of sampling is to collect, calculate and analyze statistics so  we can make inferences about the actual population.  In the end, we need to be confident that the results adequately represent the population we care about, and that proportion differences between segments are statistically reliable.   The sticking point in the sample size decision is balancing cost and reliability.  In other words, choosing a sample size that’s large enough for data reliability, but not so large that clients spend more than necessary.

Many researchers determine final sample size based on standard error margins only. These are the margin of error or “plus-or-minus” statistics we see in the news, such as when political survey results are made public.  The following matrix shows the potential margins of standard error we can expect from three different sample sizes.

 

 

Sample Size
n=

Margin of Standard Error
Given 50/50 expected proportion
testing at the 95% level of confidence

250

+/- 6.2%

400

+/- 4.9%

650

+/- 3.8%

 

To use an example, let’s say the results of a survey showed 50% of 650 randomly sampled people who own a smart phone use it to purchase products online.  Based on standard error calculations, we can be 95% confident that this proportion is actually between 46.8% and 53.8% of the population of smart phone users.  The most important thing to consider in this statement, however, is whether the survey adequately represented “the actual population of smart phone users.”  Did the sample properly represent the demographics of the population?  Did we consider that the propensity to purchase items using a smart phone may likely differ between younger and older people, or between more wealthy and less wealthy people?

When we start considering the various segments (subgroups) within the population, and differences in how they respond, choosing the right sample size becomes even more important.  We need to consider more than just the margin of standard error.  You see, the standard error calculation is a universally standard way of determining the potential margin of error that will occur by not measuring the entire population.  In reality, it doesn’t tell the whole story. The calculation doesn’t specifically take into account certain biases–ultimately controllable by experienced researchers–that can impact data reliability.  These include:

  1. Sampling error — caused by incorrect sampling. or the sample isn’t truly representative of the population.
  2. Question bias — caused by formulating leading or incomplete questions in the questionnaire, improperly worded questions, and/or neglecting response order or question sequence bias.
  3. Interviewing bias — caused by poor supervision or training of interviewers, or failing to ensure optimal consistency across interviewing styles.
  4. Non-Response error — caused when efforts are not made to reach and engage individuals who initially fail to respond to the survey and/or are never available.  This would happen, for example, if we accepted all the easy-to-reach individuals such as homemakers or the elderly who are usually at home and answer the telephone.

We must all be aware there is more to consider than total sample size and the “plus-or-minus” margins of standard error.  When designing a quantitative study, choosing a “total” sample size requires a more detailed discussion than most people realize.  You must account for the important segments (subgroups) including, at the very least, the demographics (consumers) and firmographics (businesses) of your target markets.

Good researchers control and minimize biases and error types not necessarily accounted for in the “standard margin of error” calculations.  Make sure you are, or your researcher is, doing the same.

Making the right sample size decision is important.  Good researchers will ensure clients understand it thoroughly and will help them choose the right sample size for maximum reliability without spending more than is necessary.

ViewHide Comments (0)

Leave a Reply

YEP. YOU’VE SEEN IT ALL

Use one of the options below to get in touch with us to start a conversation with Pivot about your marketing, research, or web needs. We are standing by.

Pivot Group, LLC | Mail: PO Box 1326, Wilsonville, OR 97070 | Visit: 7405 SW Tech Center Drive, Building B, Suite 140, Portland, OR 97223