CIV Survey Sample Size

Can a sample size of 300 people give accurate results?

In a sample survey, only a part of the total population is approached for information on the topic under study.  These data are then ‘expanded’ or ‘weighted’ to represent the target population as a whole.  The sample size chosen for a survey is a balance between contradictory factors:  higher sample sizes give results with a greater level of accuracy, but require more resources to conduct the survey and may lead to higher burden being placed on respondents.  This is particularly important if there are other surveys collecting information from the same target population.

Estimates that are based on information from a sample are subject to sampling variability; that is they may differ from the figures that would have been obtained had the entire population been surveyed.  A large sample is more likely than a small sample to produce results that closely resemble those that would be obtained if a census was conducted.

Standard Error

The difference between survey results based on a random sample and results from a census can be measured by the standard error.  The standard error is used to construct a confidence interval which is expected to include the 'true value'.  A 95% confidence interval is approximately equivalent to the survey estimate plus or minus two times the standard error of the estimate.  In practice, this means that in 95 out of 100 samples the confidence interval is expected to contain the ‘true value’ for the target population.

Figure 1 shows the 95% confidence intervals for various proportions estimated from a sample of 300 respondents.  As an example of interpreting the confidence interval, consider a survey result that 50% of respondents are in favour of a proposal.  The standard error of the estimate is 2.9%, and we can be 95% confident that the ‘true value’ lies between 44.3% and 55.7%.  The further away from 50% the survey estimate is, the narrower the corresponding confidence interval.

Whether an estimate is ‘reliable’ depends to some extent on the purpose for which the estimate is being used. In some instances, a higher degree of confidence on an estimate (i.e. a narrower confidence interval) may be required than the survey is able to give, but in other instances this may not be a concern.  An increase in the sample size reduces the standard error and hence the range of the confidence interval. Figure 2 shows how, for a population of equivalent size to a large LGA, the 95% confidence interval for a survey result of 50% changes as the sample size increases. For the largest LGA, a sample of 300 represents about 0.2% of the adult population, but the confidence interval depicted below is only marginally bigger than that for the smallest LGA (where the sample represents about 11.5% of the adult population).

Figure 2 also shows that any detailed results obtained from the survey must have a large enough sample size in the sub-group of interest (e.g. total females; males aged 18-24) for reliable estimates to be produced for that sub-group.