What Is a Statistically Valid Community Survey?
By Dr. Adam Probolsky
You can call a community survey statistically valid only if it is designed so that the results accurately represent the overall demographics of the resident population of a city, town, county, or other defined community, like special district. For local governments, the only way to achieve statistical validity is to work with a pollster – a survey firm that can carefully recruit residents to participate in the right demographic proportions. A survey firm will also ensure a transparent methodology and guarantee responses from an appropriate number of residents – usually 300 – 400 completed interviews. This yields accurate, decision-ready data.
Things to Keep in Mind
- A statistically valid resident survey is built to represent the whole city, county, district, or service area.
- An open online survey can be useful for engagement, but it should not be treated as representative.
- Sample size (number of completed surveys) matters, but it is not the biggest factor. Getting the right mix of demographics is paramount.
- Staff, city councils, and other government boards should check the representativeness of a survey before relying on the data for decision-making like budgets, strategic plans, service changes, or revenue measures.
Useful Definitions
| Term | What it means for local government |
|---|---|
| Resident survey | A survey of people who live in a defined political subdivision or service area. |
| Statistically valid survey | A survey designed so findings can be generalized to the larger population with a high degree of certainty. |
| Representative sample | A respondent group that reflects the population on key characteristics such as age, geography, gender, race/ethnicity, home type, language, or other demographics. |
| Margin of error | A mathematical calculation estimating the chance of fluctuation in the results. |
| Weighting | A statistical adjustment that aligns survey responses with known population benchmarks when some groups are over- or underrepresented. NOTE: Weighting is never a good idea for community surveys. Agencies should demand a fully representative sample with real residents, not made up results. |
| Open link survey | A survey anyone can choose to take, often through a public link posted on a website or social media. They are great for getting feedback and demonstrating an interest in the the public’s input, but should not be characterized as accurate or relied upon for making decisions. NOTE: There are a few exceptions to this rule. Consult a qualified researcher to explain how to make an open link survey more valid. |
Why Statistical Validity Matters for Community Surveys
Statistical validity matters because public agencies often use survey findings to make decisions that affect everyone, including the majority of residents who never attend a meeting, email council, or comment online.
When a city asks, “What do residents think about public safety, housing, parks, streets, taxes, or trust in local government?” staff should be able to tell policymakers that the survey results are from a fully representative cross-section of real residents.
That distinction is especially important when survey results support critical government functions.
| Decision area | Why representativeness matters |
|---|---|
| Budget priorities | Leaders need to know whether preferences reflect broad resident priorities. |
| Strategic planning | Long-term goals should be grounded in communitywide needs and expectations. |
| Revenue measures | Tax, bond, millage, fee, and rate research requires defensible public opinion data so that governing bodies have the information they need prior to the go/no go moment of asking residents, ratepayers, or voters to support a new measure. |
| Service satisfaction | Satisfaction with roads, sidewalks, parks, public safety, libraries, utilities, or code enforcement may vary sharply by neighborhood or demographic group. |
| Communications | Agencies need to know which messages residents understand, trust, or reject, who needs more outreach, and the ideal ways to reach them. |
What Makes a Community Survey Statistically Valid?
A resident survey becomes statistically valid through design, not through a large number of responses.
Accurate resident surveys include these four elements.
| Element | What to look for |
|---|---|
| Defined population | The survey includes a defined population, like adult residents, registered voters, utility customers, business owners, parents, or some other group. |
| Precise geography | The geographic boundaries of who can participate should be specific. GIS mapping should be used to ensure precision. |
| Modes that match how the population communicates | Respondents are recruited in multiple modes that match the communication style of residents – mail, telephone, online (email- and text-to-web). Sometimes door-to-door and in-person intercepts. |
| An appropriate number of completed interviews | For most cities, counties, or other local agencies, 300 – 400 completes is the right number. There is no harm in collecting more completes, but it’s usually unnecessary. |
Context on sample size
A statistically valid survey in a large state might include 1,000 – 1,500 responses, 700 – 900 in a medium size state, and 500 – 600 in a small state. These surveys predict election outcomes – a measure that tests accuracy, even for non-election related research.
Insights from Dr. Adam Probolsky A science-based methodology makes survey results relevant and reliable. It gives policymakers the knowledge that they are making decisions that align with the public.
Weighting is Absolutely Manipulating the Data
Weighting is a standard survey practice used by some research organizations instead of doing the hard work of reaching residents in the right proportions based on Census or voter data.
Example: If renters are underrepresented in a survey and older homeowners are overrepresented (something that is almost always the case), some survey companies may weight the data – assigning a higher value to the limited number of renters in the sample. They failed to get the right mix of residents, usually because they focused on reaching people in just one mode such as mail. This will never yield accurate results because most people communicate online, or on their phones.
Weighting is a lazy approach that is almost never necessary.
Questions We Hear
Is my city’s online-only survey run by staff statistically valid?
No. An online-only survey run by staff will never be representative of all the relevant demographic groups. There are still reasons to do them, but not to inform policy.
Can we really use survey results if only 400 residents respond?
Yes. Virtually any agency can use 300 to 400 responses if the survey connects with residents using multiple modes (i.e., email- and text-to-web, mail, and telephone) and includes a representative group of residents in the right demographic proportions based on Census or voter data.
Shouldn’t we collect thousands of responses?
No. There is no science-based reason to collect thousands of responses. There is also no downside to getting a greater number of responses, assuming the ones you have are fully representative of the demographics and geographic location of residents. It’s just not necessary, and it can be cost prohibitive.
Is margin of error the same thing as survey accuracy?
No. It’s a popular statistic that has limited utility in community surveys. It can be seen as the outer limits of how the result %s might fluctuate.
Should cities publish open-ended survey comments?
Cities should publish open-ended comments only when personally identifiable information is stripped away. Any research firm will do this before delivering the results. Open-end survey comments are useful for context. But take caution not to assume one comment represents more than that one person’s ideas.
What is the difference between a resident survey and community engagement?
A resident survey measures public opinion through a structured research process, yielding representative results. Community engagement creates opportunities for residents to discuss, provide feedback, and participate in their government. There is value in doing both. But the two are not interchangeable.
Examples of Community Survey Media Coverage
- Edgewater, Colorado, Neighborhood Gazette
- City of Falls Church, Virginia, Falls Church Pulse
- Rochester Hills, Michigan, C & G News
Sources You Might Look To
- American Association for Public Opinion Research, Standard Definitions
- AAPOR, Understanding a Credibility Interval and How It Differs From the Margin of Sampling Error
- U.S. Census Bureau, Sampling Estimation and Survey Inference
How Probolsky Research can help your public agency
About Dr. Adam Probolsky
Dr. Adam Probolsky is President of Probolsky Research where he has conducted over 1,000 polls and surveys for local governments across the U.S. Adam has also served in government roles at the city, county, and state levels where he made and oversaw policy related to finance, parks, planning and land use, transportation, and waste & recycling. He was also a sheriff’s department information officer. He is a Senior Research Fellow for the Drucker School of Management at Claremont Graduate University.
About Probolsky Research
Probolsky Research conducts public opinion research for corporate, election, government, and nonprofit clients. The firm works for public agencies in 29 states from offices in Dallas, Denver, Newport Beach, Pasadena, San Francisco, and Washington DC.
