Posts

What Is a Statistically Valid Community Survey?

/in /by

By Dr. Adam Probolsky

You can call a community survey statistically valid only if it is designed so that the results accurately represent the overall demographics of the resident population of a city, town, county, or other defined community, like special district. For local governments, the only way to achieve statistical validity is to work with a pollster – a survey firm that can carefully recruit residents to participate in the right demographic proportions. A survey firm will also ensure a transparent methodology and guarantee responses from an appropriate number of residents – usually 300 – 400 completed interviews. This yields accurate, decision-ready data.

Things to Keep in Mind

  • A statistically valid resident survey is built to represent the whole city, county, district, or service area.
  • An open online survey can be useful for engagement, but it should not be treated as representative.
  • Sample size (number of completed surveys) matters, but it is not the biggest factor. Getting the right mix of demographics is paramount.
  • Staff, city councils, and other government boards should check the representativeness of a survey before relying on the data for decision-making like budgets, strategic plans, service changes, or revenue measures.

Useful Definitions

TermWhat it means for local government
Resident surveyA survey of people who live in a defined political subdivision or service area.
Statistically valid surveyA survey designed so findings can be generalized to the larger population with a high degree of certainty.
Representative sampleA respondent group that reflects the population on key characteristics such as age, geography, gender, race/ethnicity, home type, language, or other demographics.
Margin of errorA mathematical calculation estimating the chance of fluctuation in the results.
WeightingA statistical adjustment that aligns survey responses with known population benchmarks when some groups are over- or underrepresented. NOTE: Weighting is never a good idea for community surveys. Agencies should demand a fully representative sample with real residents, not made up results.
Open link surveyA survey anyone can choose to take, often through a public link posted on a website or social media. They are great for getting feedback and demonstrating an interest in the the public’s input, but should not be characterized as accurate or relied upon for making decisions. NOTE: There are a few exceptions to this rule. Consult a qualified researcher to explain how to make an open link survey more valid.

Why Statistical Validity Matters for Community Surveys

Statistical validity matters because public agencies often use survey findings to make decisions that affect everyone, including the majority of residents who never attend a meeting, email council, or comment online.

When a city asks, “What do residents think about public safety, housing, parks, streets, taxes, or trust in local government?” staff should be able to tell policymakers that the survey results are from a fully representative cross-section of real residents.

That distinction is especially important when survey results support critical government functions.

Decision areaWhy representativeness matters
Budget prioritiesLeaders need to know whether preferences reflect broad resident priorities.
Strategic planningLong-term goals should be grounded in communitywide needs and expectations.
Revenue measuresTax, bond, millage, fee, and rate research requires defensible public opinion data so that governing bodies have the information they need prior to the go/no go moment of asking residents, ratepayers, or voters to support a new measure.
Service satisfactionSatisfaction with roads, sidewalks, parks, public safety, libraries, utilities, or code enforcement may vary sharply by neighborhood or demographic group.
CommunicationsAgencies need to know which messages residents understand, trust, or reject, who needs more outreach, and the ideal ways to reach them.

What Makes a Community Survey Statistically Valid?

A resident survey becomes statistically valid through design, not through a large number of responses.

Accurate resident surveys include these four elements.

ElementWhat to look for
Defined populationThe survey includes a defined population, like adult residents, registered voters, utility customers, business owners, parents, or some other group.
Precise geographyThe geographic boundaries of who can participate should be specific. GIS mapping should be used to ensure precision.
Modes that match how the population communicatesRespondents are recruited in multiple modes that match the communication style of residents – mail, telephone, online (email- and text-to-web). Sometimes door-to-door and in-person intercepts.
An appropriate number of completed interviewsFor most cities, counties, or other local agencies, 300 – 400 completes is the right number. There is no harm in collecting more completes, but it’s usually unnecessary.

Context on sample size
A statistically valid survey in a large state might include 1,000 – 1,500 responses, 700 – 900 in a medium size state, and 500 – 600 in a small state. These surveys predict election outcomes – a measure that tests accuracy, even for non-election related research.

Insights from Dr. Adam Probolsky A science-based methodology makes survey results relevant and reliable. It gives policymakers the knowledge that they are making decisions that align with the public.

Weighting is Absolutely Manipulating the Data

Weighting is a standard survey practice used by some research organizations instead of doing the hard work of reaching residents in the right proportions based on Census or voter data.

Example: If renters are underrepresented in a survey and older homeowners are overrepresented (something that is almost always the case), some survey companies may weight the data – assigning a higher value to the limited number of renters in the sample. They failed to get the right mix of residents, usually because they focused on reaching people in just one mode such as mail. This will never yield accurate results because most people communicate online, or on their phones.

Weighting is a lazy approach that is almost never necessary.

Questions We Hear

Is my city’s online-only survey run by staff statistically valid?

No. An online-only survey run by staff will never be representative of all the relevant demographic groups. There are still reasons to do them, but not to inform policy.

Can we really use survey results if only 400 residents respond?

Yes. Virtually any agency can use 300 to 400 responses if the survey connects with residents using multiple modes (i.e., email- and text-to-web, mail, and telephone) and includes a representative group of residents in the right demographic proportions based on Census or voter data.

Shouldn’t we collect thousands of responses?

No. There is no science-based reason to collect thousands of responses. There is also no downside to getting a greater number of responses, assuming the ones you have are fully representative of the demographics and geographic location of residents. It’s just not necessary, and it can be cost prohibitive.

Is margin of error the same thing as survey accuracy?

No. It’s a popular statistic that has limited utility in community surveys. It can be seen as the outer limits of how the result %s might fluctuate.

Should cities publish open-ended survey comments?

Cities should publish open-ended comments only when personally identifiable information is stripped away. Any research firm will do this before delivering the results. Open-end survey comments are useful for context. But take caution not to assume one comment represents more than that one person’s ideas.

What is the difference between a resident survey and community engagement?

A resident survey measures public opinion through a structured research process, yielding representative results. Community engagement creates opportunities for residents to discuss, provide feedback, and participate in their government. There is value in doing both. But the two are not interchangeable.

Examples of Community Survey Media Coverage

Sources You Might Look To

How Probolsky Research can help your public agency

About Dr. Adam Probolsky

Dr. Adam Probolsky is President of Probolsky Research where he has conducted over 1,000 polls and surveys for local governments across the U.S. Adam has also served in government roles at the city, county, and state levels where he made and oversaw policy related to finance, parks, planning and land use, transportation, and waste & recycling. He was also a sheriff’s department information officer. He is a Senior Research Fellow for the Drucker School of Management at Claremont Graduate University.

About Probolsky Research

Probolsky Research conducts public opinion research for corporate, election, government, and nonprofit clients. The firm works for public agencies in 29 states from offices in Dallas, Denver, Newport Beach, Pasadena, San Francisco, and Washington DC.

The Voter – Policymaker Divide on Water

/in /by

A poll conducted in November discovered that California voters have a distinctly different view from environmental and political leadership on how to address the future of water.

Dr. Adam Probolsky, president of Probolsky Research presented the poll results at the California Foundation on the Environment and the Economy (CFEE) Water Conference in Indian Wells which included some of the state’s top elected officials and advocacy groups leadership.

Surface storage (dams and reservoirs) are supported by seventy-two percent of voters.

Ocean water desalination is supported by seventy-seven percent of the voters.

The real number is closer to 80% of the state’s water supply going to agricultural use.

All this adds up to a challenge for the environmental community that is largely opposed to dams and desalination. It may also be an opportunity for supporters of new water infrastructure. They may use these voter sentiments in how they approach funding.

A more complete look at the polling data on water policy can be found here.

The Next Step After Conducting an Employee Survey: Turning Results into Action

/in /by

Every year, millions of employees are asked to fill out surveys. They share their opinions on everything from organizational culture to feedback on their direct supervisor to what training would help them do their job better. But a crucial question often goes unanswered: Does this feedback actually lead to change?

The answer is perfectly illustrated by Peoria, Arizona Police Chief Thomas Intrieri during a recent city council meeting. He was making the pitch for additional funding for the police department and cited how their employee survey found that officers were concerned about equipment. His use of the data to make his point and drive home support from policymakers is a testament to the principles recommended by organizational leadership experts and researchers.

Instead of relying only on anecdotes, Chief Intrieri presented hard data from an employee engagement survey. He pointed to a specific finding, noting that the equipment he was seeking funds for was among the top three concerns expressed by employees. Good staff work and the forethought to have done the employee survey led to a 7-to-0 yes vote for the $884,192 budget adjustment.

Why This Is a Textbook Example of Effective Leadership

It Proves to Employees That They Were Heard

The single most important outcome of a survey is for employees to know their feedback wasn’t ignored. By bringing the item before the city council and publicly citing the survey results, the Chief showed every officer in his department that their collective voice was the foundation of his request. This is the first and most vital step in building trust.

It Turns Subjective Wants into Objective Evidence

A leader’s request for “better equipment” can be dismissed as a routine departmental wish. But a request based on a formal, organization-wide survey becomes a business case – informed by data. It provides decision-makers (like a city council or a management team) with objective evidence to justify allocating funds and making a change.

It Creates a Positive Cycle of Improvement

When employees see a direct line between the feedback they gave and a positive change in their workplace, they become more invested in the organization’s success. They are more likely to participate thoughtfully in future surveys, which provides leaders with even better data. This creates a powerful loop: employees give quality feedback, leadership acts on it, and the organization gets better as a result.

The alternative: It’s easy for employees to feel like their feedback goes into a black box. This can lead to distrust in the process and diminishing response rates to future employee surveys. Don’t let this happen in your organization.

Frequently Asked Questions About Employee Surveys (FAQ)

Q1 Why should an organization conduct an employee survey?

Organizations use surveys to get an honest, confidential look at what’s working and what isn’t from the employees’ perspective. It’s one of the most effective ways to identify hidden problems, measure morale, and find opportunities to improve the workplace. Hint: Using an independent research firm gives assurance to employees that their responses will remain anonymous.

Q2 What is the most important step after a survey is completed?

The most critical step is action. Leaders analyze the results, share them with employees, and demonstrate a clear plan to address the key findings. Even small, visible changes show that the feedback was taken seriously.

Q3 How does acting on survey feedback help a business or organization?

Acting on feedback directly impacts organizational success. It boosts employee morale and engagement, which leads to higher productivity, better customer service, and lower employee turnover. A workplace that listens and responds is a workplace where people want to be.

Dr. Adam Probolsky is president of Probolsky Research, which conducts employee surveys for corporate, government, and nonprofit clients.

Politics ≠ Political

/in /by

Local government staff have a dual role – their functional job and the job of keeping their residents happy. No one else has to work under those condition. Sure, corporate executives answer to shareholders, but not with the intensity and proximity that residents have to City Hall.

Yet many agencies instruct staff to leave politics outside the building – not just their personal politics. My new piece for the International City/County Managers Association (ICMA), “A Strict Politics – Policy Divide Is Holding Your Agency Back,” shows why limiting staff’s ability to watch and consider the politics of the day is bad for the agency.

The article explains how politics works as an early warning system.

I outline a routine that requires no extra headcount: track narrative shifts, assign a lead, and circulate a one-page memo each week. This way staff gain situational awareness.

Probolsky Research follows the same approach when guiding agencies through surveys, focus groups, and message testing. We help staff see the story behind the numbers, then translate that story into policy choices that withstand public scrutiny.

Read the full article on the ICMA blog: https://icma.org/blog-posts/strict-politics-policy-divide-holding-your-agency-back.

Or read it here:

A Strict Politics–Policy Divide Is Holding Your Organization Back

If your organization is facing a political threat, you do not fight it with silence—you fight it with strategy.

By Dr. Adam Probolsky, president of Probolsky Research and senior research fellow with Claremont Graduate University’s Drucker School of Management

I have yet to meet a local government staff member that could not read the room. But when their hands are tied behind their backs—told to avoid politics at all costs—they are not operating at peak performance. Staff are not confused about the differences between political and public policy disciplines. They understand the unique dynamics between thinking politically and acting politically.

They know it means anticipating what might trigger backlash before it shows up on an agenda.

For example, flagging a new fee before it appears on an agenda where activists are likely to label it a “tax” is thinking politically and giving everyone involved the opportunity to prepare. Staff are not going to seek an endorsement of the fee from the local county party chairman.

Politics is about power: getting it, keeping it, and using it to win. Public policy is about outcomes: what laws, regulations, or programs actually say and do. The two often intersect. Both are grounded in research, data, and stakeholder input—we hope.

Politics as an Early Warning System

What is often overlooked in the public policy realm is the anticipatory value of politics.

Political systems theory frames politics as a connected feedback loop. Citizens express needs and preferences (inputs) through elections, political speech in all its forms, and media consumption patterns. Politicians interpret those signals and attempt to translate them into public policy (outputs). When expectations are not met, the public responds through disengagement, outrage, or votes.
 

What This Means for Staff

Staff’s focus on statutory obligations or program design is important. But dismissal of the broader political environment is causing your agency to miss early signals of change. This is not about partisanship. It is about situational awareness. Politics can tell us what people believe is happening, even if it is not true.

Staff may pride themselves on evidence-based thinking, but politics operates on emotion, identity, and perception. Research in behavioral science consistently shows that people make judgments based on recent, vivid events, not long-term data. Policy professionals who do not account for this disconnect risk being right but irrelevant, sharing the truth but still losing trust.
 

Monitoring, Not Reacting

The answer is not for staff to become political operatives. It is to treat politics like a real-time sensor network. This means tracking the origin of negative public comments—sometimes valid concerns, sometimes personal frustration. Monitoring social media not just for sentiment, but for narrative shifts. Knowing which local or regional issues are heating up, even if they are not “yours” yet.
 

Taking Action

Every public agency needs a top-down directive that makes caring about, learning about, and strategizing over politics’ impact on local policy the norm, not a taboo activity to be hidden.

Having a clear, consistent approach to tracking the political environment does not require a new hire. It requires assigning responsibility and making political awareness part of normal operations.

That might mean:
 

  • Reading statewide and regional political blogs, not just waiting for association updates.
  • Monitoring legislative agendas and committee activity, not just relying on lobbyist summaries.
  • Watching for local versions of national debates before they surface at your public meetings.
     

This work can live within your public affairs or public information offices, or be assigned to a management analyst intern. The point is: it needs to be someone’s job, and everyone’s shared awareness.

Consider creating a short political signals memo. Think one page, once a week. No spin, just intelligence. A few bullet points on:
 

  • Community sentiment shifts.
  • Stakeholder activity worth watching.
  • Viral local issues or narratives.
  • Legislative developments that could ripple your way.
     

Some agencies already have a version of this, but keep it confined to the executive team. That is a mistake. Siloed awareness delays coordinated response. Bring a broader group into the picture. Give them the foresight they need to avoid surprises and plan smart.

This is not about playing into the headlines. It is about anticipating impact. The goal is to be aware. Strategic. Less reactive.

Politics is not always rational. But it is rarely random. People expressing fear, anger, or distrust are data points. If your agency is facing a political threat, you do not fight it with silence, you fight it with strategy.

Local Government Insights Podcast Features Adam Probolsky

/in /by

Firm president, Adam Probolsky joined the Local Government Insights podcast, episode 27, to talk about how government uses public opinion research in the process of making public policy.

Listen here:

Informing Policy Decisions Through Opinion Research

Making Infrastructure Relatable

/in /by

Originally published in PublicCEO – written by Adam Probolsky

If your kid came home with a D+ on her report card, you would start asking questions. How did things get this bad? Why didn’t we catch the problem sooner? And what’s the name of a good tutor?

Americans are in a similar predicament with our failing infrastructure. According to the latest report card from the American Society of Civil Engineers, the country’s infrastructure gets a D+. That’s a marginally passing grade for our bridges, dams, highways and water systems.

For decades, civil engineers have lamented the lack of public support for a real infrastructure fix. There’s no question that Americans want to drive on safe roads, access reliable water sources and keep the lights on. We want our country to thrive, and understand that it costs money to rebuild necessary systems. Why, then, has rebuilding America’s infrastructure languished?

To understand current attitudes toward infrastructure, you can to go back to the country’s last great big investment: President Dwight D. Eisenhower’s Interstate Highway System. While today we think of highways as a necessity for trade and commerce, it wasn’t the only selling point. Part of the pitch for the National System of Interstate and Defense Highways was to ready the nation “in case of atomic attack on our key cities, the road net [would] permit quick evacuation of target areas.”

Highway advocates understood that framing transportation infrastructure in the context of the Cold War would increase the public’s intensity for the issue. It delivered a compelling narrative with a well-understood sense of urgency because it related to something on everyone’s mind.

In a similar fashion, there is a relatable message for supporting today’s infrastructure that should be tapped into, another layer of infrastructure we should all be concerned with but rarely talk about: software. A recent Bloomberg Businessweek article highlights the shocking age of software that many cities and regional government agencies rely upon for major systems like billing, emergency communications and transit. Some of us have a nostalgia for 1980s software like the DOS prompt or Commodore 64 commands. But no one would be comfortable with either as the backbone for police and fire radios or water treatment plants in 2019. That is essentially though the reality for far too governments in the U.S. today.

In a recent Probolsky Research survey, more than half (54%) of Americans say they support upgrading decades-old software that is running critical infrastructure, “even if it costs millions.” How could it be that the same Americans who are wholly weary of most government spending, would open public coffers to upgrade software that is still working? It’s worth noting that 31% of Americans said just that, “If it ain’t broke, don’t fix it.”

Perhaps support for spending on software updates is rooted in the fact that they upgrade the software on their smart phones on what seems like a monthly basis. And when you turn on an iPhone from just a few years ago, you can see the dramatic difference in look and feel of the operating system, but more importantly, the difference in speed. Americans know that software powers our world. And innovations happen every day.

Policymakers should not be afraid to support spending money on infrastructure. And they should know that it will be easy to make the case for upgrading and maintaining the underlying software that runs critical government systems.

 

California Special Districts Association – June 2018 Leadership Summit presentation

/in /by

Thank you to the California Special Districts Association for inviting Adam Probolsky to present new statewide polling data at their Leadership Summit in Squaw Creek last week. Adam recorded a brief overview of the presentation for those who missed the event or for those who attended and want to share what they learned during the discussion. We welcome inquiries on the data or for help with research for specific agencies or regions.

Here you can see just the slides:

Probolsky on Polling Process and Recommended Outreach for City Sales Tax Measure

/in /by

There is no formula that is set in stone for how to conduct research on any subject matter. But when a local government is looking to pursue new voter-approved revenues, we have identified a process we follow to help the city, county or other agency understand their constituency and do the necessary outreach to communicate the need and purpose.

We welcome the chance to share this process and review the needs of any community and explain how we can help.

Here is a brief video of Adam Probolsky speaking to the Dixon, CA City Council about a poll our firm conducted on a possible special sales tax measure to fund transportation and infrastructure needs:

 

WATCH: CALAFCO Conference 2017 Poll Presentation

/in , /by

We conducted a poll for CALAFCO which represents all 58 California Local Agency Formation Commissions. And presented the findings at their October 25, 2017 in San Diego. The presentation and Q&A took over an hour, but I wanted to give people the chance to watch in less than 10 minutes. Click to watch and listen to the run-through below.

Contact our San Francisco office at 415-870-8150 if you would like more detailed results or a personalized briefing.

Click here for the the presentation posted at the CALAFCO.