It is entirely possible that the adult population of Utah has changed its views on gay marriage and civil unions over the last two years. But directly comparing the recent SurveyUSA poll commissioned by the Salt Lake Tribune1 to polling done over the last decade by Brigham Young University’s Center for the Study of Elections and Democracy (CSED)2 creates a false impression about the speed of public opinion change on the issue of gay marriage. The two polls survey different groups using different survey questions. Be cautious about comparing the two surveys without recognizing their differences.
Like everyone else, the research team here at Y2 Analytics is interested in knowing if recent events have moved Utah public opinion on gay marriage, but we took a different approach than the Tribune. We replicated the question and the population asked in the CSED polls. Using a statistical model derived from the publicly-available Utah voter file, we sampled likely voters for the 2014 General Election. Then we had live interviewers call sampled voters and ask the following question:
This question wording is identical to the one used in the CSED poll and follows question wording pioneered by the New York Times/CBS News poll; respondents are given all three options at the same time and are asked to select their preference. Our interviewers talked to 500 Utah voters.
The results are very similar to the 2010 Utah Colleges Exit Poll results3 once we account for the margin of error of each poll.4 While it’s possible that the general adult population of Utah may have moved toward supporting gay marriage over the last few years, among voters attitudes have not changed significantly during the same period. Attitudes about gay marriage in Utah may be changing, but to see the change it’s important to compare surveys of the same population with the same question over time.
To interpret a poll, it’s important to understand the “target population” or the group that the poll is trying to measure. The Tribune poll measured a representative sample of Utah adults.5 The CSED polls measured a representative sample of Utah voters.6 Voters and non-voters are different in ways that affect their views on political issues.
For example, according to the 2010 U.S. Census, 29% of Utahns are between the ages of 18 and 29. But only 19% of the electorate in 2012 was between the ages of 18 and 29, per exit polling done by the Utah Colleges Exit Poll, a difference of 10 points. A review of polling on gay marriage reveals a substantial age gap; younger adults are much more supportive than previous generations.7 It follows that the electorate is less supportive of gay marriage than the adult population at large. Voters also tend to be more educated and are more likely to be married than non-voters. And these differences are greater during election years without a presidential election.
Whenever you are reading the results of a poll, consider which population the poll is sampling. There are legitimate reasons for polling all adults just like there are defensible reasons for sampling voters. Just be careful when comparing the two.
Additionally, these two polls use different survey questions. The Tribune’s SurveyUSA poll offers each same-sex relationship recognition option in separate questions: first an up-down approval question of gay marriage, then the same for civil unions, then a question about opposing all forms of legal recognition:
Do these types of question differences really matter? They do.8 Question wording and question ordering in a survey can affect the mindset of the respondent and change his or her response. Many pollsters insist on keeping question wording consistent across their polls to make direct comparisons possible, as the CSED researchers have done on their surveys about gay marriage since 2004.
Is one of these question formats preferable to the other? Not necessarily. The Tribune wording and ordering allows respondents to weigh each option independently from the others, while the CSED wording asks respondents to choose in context of the others. If this were a public opinion campaign leading up to a vote – like we witnessed in Maine, Washington, and Minnesota in 2012 – an independent option mirroring the ballot would make the most sense. But in a public opinion discussion without electoral context, asking respondents to consider all options at once is perfectly valid too.
Since these polls are different in important ways, we should shy away from making direct comparison of their results and then making inferences about changes in public opinion. For example, over the last few days well-meaning commentators have asserted that support for gay marriage in Utah has grown by 20 percentage points. These polls do not justify that conclusion. If you do happen to mention both the Tribune poll and the CSED polls in the same discussion, make sure to highlight their key differences.
(Edit) If you are interested in digging into the details of our poll, take a look at the topline.
– Scott Riding
- See the Tribune poll write-up here.
- Read the CSED review of their polling over the last decade here.
- We compare our sample to the 2010 UCEP poll because they are both samples of voters in non-presidential off-year elections.
- Our survey has a margin of error of +-4.38, while the CSED polls have margins of error that vary. See their write-up for details.
- As far as we know they are adults. SurveyUSA conducts the majority of its interviews via an automated recording. Critics of autodial polling point out that it’s impossible to verify the demographics of the respondents, among other criticisms. Many news organizations refuse to publish autodialed polls as a policy.
- Even the Utah Voter Poll, an online panel, is recruited by BYU Exit Poll interviewers who invite voters to join the panel as part of the exit poll questionnaires.
- The Fix at the Washington Post has a good article on this gap in a recent national poll.
- The Pew Research Center does a good job showing one case where question wording mattered a great deal.