Should Your Community Survey Be Weighted?
No. Survey weighting is a tool that should be used sparingly. In almost all situations it is unnecessary because your research firm should match demographics by getting responses from real people. By using stratified random sampling and multimode outreach, your research firm can get responses from the right mix of residents, customers, or voters during data collection.
This article is for city managers, county administrators, special district executives, advocacy, communications, and marketing teams, procurement staff, and other leaders who need accurate and defensible survey results.
Important takeaways
A pollster’s first responsibility is to collect accurate findings from a representative group of real people.
Stratified random sampling and persistent outreach eliminate the need for post-survey demographic weighting in almost any city, county, town, or special district.
Weighting does not create additional interviews. Weighting makes some existing respondents count more, and others count less.
Weighting creates a hole in the data because it creates synthetic responses that do not include the rich qualitative open-ended answers to important questions. These voice-of-the-resident answers (or customer or voter) are how pollsters uncover the underlying emotional drivers and discover new concerns, not on the radar of leaders.
Routine weighting has become a lazy substitute for doing the hard work of making additional calls, sending more mail, reminder emails and text invitations, and sometimes even making the real effort of door-to-door outreach.
What does weighting really do?
Post-survey demographic weighting makes up for the failings of researchers to match the target population.
Weighting changes each respondent’s influence until selected characteristics of the analyzed sample align with population benchmarks. A respondent may count as 0.6 of an interview, 1.0 interview, or 2.3 interviews in the calculated percentages.
Does weighting make up respondents?
Sort of. Weighting creates synthetic statistical influence by making some interviews count more and others count less.
A respondent with a final weight of 2 contributes roughly twice as much to a calculated percentage as a respondent with a weight of 1.
Is the weighting of survey data really a lazy practice?
Yes. Weighting is used when a researcher chooses not to do the data collection properly — reaching real people, in the right proportion — and instead chooses to manufacture the outcome. It is taking the easy path by collecting responses from people who answer quickly, closing the survey, and adjusting the demographic totals in a spreadsheet rather than real data.
The right path is to identify which groups remain underrepresented and keep working to include them. Persistent outreach costs more time and effort, but it produces accurate results.
One of the most egregious uses of weighting is when a researcher quits before reaching harder-to-reach communities like younger residents, people of color, and lower-income neighborhoods.
The American Association for Public Opinion Research‘s best practices advise researchers to think carefully about sample design so the sample matches the population and requires fewer adjustments later. Probolsky Research takes that principle seriously by doing the work during collection.
What goes missing when researchers rely on weighting?
When survey participants answer open-ended questions, they reveal their raw feelings and a view into their minds that would never be seen without a survey. But this cannot happen with statistically manufactured results. Weighting means that you lose some of the most useful information in a community survey — residents explain why they support, oppose, trust, fear, and their aspirations for their community.
A respondent with a weight of 2.3 still provides one comment. The dataset does not contain 2.3 explanations, 2.3 personal stories, or 2.3 emotional reactions. Weighting results also gives one person’s comment more numerical influence.
Public agency leaders deserve to see the words behind the percentages.
Can demographic weighting still produce inaccurate results?
Pew Research Center‘s study of online opt-in samples found that basic demographic weighting sometimes made bias worse.
Both approaches require methodological decisions, but post-survey weighting introduces additional assumptions about whether the respondents who participated can represent people who did not.
How does stratified random sampling produce better data?
Stratified random sampling produces better data by building representation into the collection process. The researcher divides the target population into relevant demographic and geographic groups, randomly selects potential participants within those groups, and monitors completed interviews throughout the data collection process.
These factors include age, gender, party registration, race, rent or own, college or no college, council district or policing zone, etc. The exact targets should reflect the study population and the decisions the research must support.
Probolsky Research keeps fielding until the completed sample contains the right mix of people. Targeted outreach reaches groups that respond at different rates. The resulting dataset contains interviews from those groups rather than heavier mathematical influence assigned to easier-to-reach respondents.
Stratified collection = Representation is built during fielding = Real voices are captured
Why does this matter for public agencies?
Survey RFPs sometimes use “weighted” as a general term for ensuring that results represent the community. But in research methodology, weighting refers to the specific practice of making statistical adjustments after responses have been collected.
Public agencies should instead require that survey and poll participants closely match the target population.
How Probolsky Research can help
Probolsky Research designs statistically valid public opinion surveys for cities, counties, towns, special districts, state agencies, and other organizations. Each project begins with the questions the research must answer, the population that must be represented, and the subgroup analysis the client needs.
Our team develops the sample, establishes demographic and geographic targets, designs the questionnaire, and fields the study through multiple modes like telephone, mail, email, text-to-web, and in-person when necessary. We monitor respondent representation throughout data collection and keep working to obtain the right mix of participants.
Probolsky Research delivers findings based on real constituents and their actual words. Clients receive accurate results, meaningful open-ended insights, and a clear understanding of how the survey was conducted.
About Dr. Adam Probolsky
Dr. Adam Probolsky is President of Probolsky Research where he has conducted over 1,000 polls, surveys, focus groups, and workshops for local governments across the U.S. Adam has served in government roles at the city, county, and state levels where he made and oversaw policy related to finance, parks, planning and land use, transportation, and waste & recycling. He was also a sheriff’s department public information officer. He is a Senior Research Fellow for the Drucker School of Management at Claremont Graduate University.
About Probolsky Research
Probolsky Research conducts public opinion research for corporate, election, government, and nonprofit clients. The firm works for public agencies in 29 states from offices in Dallas, Denver, Newport Beach, Pasadena, San Francisco, and Washington DC.
