Who Ya Gonna Call?

We are quickly becoming a cell phone society. A recent Centers for Disease Control and Prevention (CDC) survey reports that the number of U.S. household that use only cell phones has surpassed the number that use only traditional landline phones. According to the report, nearly 20% of U.S. households no longer have a landline phone, and that number is rapidly rising. At the current pace, in less than 10 years the majority of U.S. households will not have a landline phone.

What this means to researchers is that traditional telephone sampling methods are becoming less representative of the general population. In fact, Marketing Systems Group, a leader in telephone survey samples recently reported:

“The unfolding changes in US telephony have introduced new sources of undercoverage in traditional RDD samples with magnitudes that are no longer ignorable.”

In addition to RDD samples, listed telephone samples face the same challenges in that there are not yet and are not likely soon to be directories of personal cell phone numbers. Unfortunately, we have gotten to a point where using any traditional telephone sampling methodology subjects a survey to the risk of exclusion bias.

As we move forward, sampling strategy will play an increasingly important role in research design. As reliable sources of telephone samples fade, new sources of online samples are emerging, not only with traditional opt-in panels but in other areas such as within social networks. Many of these communication outlets are popular with the cell phone only set. Combining various sources of survey respondents can often generate a sample that is more representative than any single source alone. While there will never be a way to get a perfectly representative sample, there will always be a way to improve the representativeness, and a good researcher should be able to advise you how.

The CDC report also highlights important demographic differences between the growing breed of cell phone only households and those with traditional landline service – differences that researchers must address.

Who has gone wireless?

• 39% of renters
• 10% of those who own their home
• 33% aged 18-24
• 3% aged 65 and older
• 21% who are living in the South
• 17% who are living in the West
• 20% of men
• 17% of women

Do You Get A Round?

Rounding of numbers has been second nature to all of us since early in our school years. Saying that 96.5517241% of our customers are satisfied is cumbersome if not confusing. Instead we say 96.6% are satisfied or even more simply 97%. It’s such a simple calculation that we do it without a second thought.

99.9% of the time, rounding follows very strict rules. If the part of the number you are “chopping off” starts with five or more you round the last remaining digit up; if it’s less you chop off the unneeded digits and leave the last remaining digit as it is. It would be nice if this simple rule worked 100% of the time, but there is actually a bit more to rounding than meets the eye.

We were taught to always round 5’s up. 2.755 becomes 2.76; 7.5 becomes 8, and so on. For most applications this treatment of 5’s works fine but for some it can introduce a small bias into your findings.

Let’s consider a poll where we ask 200 people how many dogs and cats they own. Suppose the averages are as follows:

Dog – 1.25 per person
Cat – 0.75 per person

In our report, we are rounding everything to one decimal, so we say that people on average own 1.3 dogs and 0.8 cats. The trouble with this is that if we add the two mean scores together, we conclude that the average person owns 2.1 pets (1.3 + 0.8) when in fact they own on average exactly 2 pets (1.25 + 0.75). This is only a small difference, but if we were to expand the study to include rabbits, fish, ferrets, gerbils, guinea pigs, hamsters, and iguanas, there are many more numbers that could potentially land on a five. If we round them all in the same direction, we quickly begin to overstate the average number of pets that people own.

To correct for this bias, when a number ends in five, you should round it up half of the time and round it down the other half. The best way to do this from a mathematical perspective is to randomly round up half of the time and down the other half of the time. Unfortunately, this doesn’t work in practice because each time you calculate a percentage or mean you could be rounding to a different number. 55 out of 200 responses is 27.5%. You can’t call it 27% sometimes and 28% other times. It needs to be one or the other.

The accepted method of rounding fives is to round to the nearest odd number. 27.5% would always get rounded to 27%. 28.5% would always get rounded to 29%. Many research tabulation programs have an option for employing this method of rounding.

Another common mistake of rounding is to round numbers twice. Standard market research data tables typically display percentages to one decimal place. Keep in mind that those numbers are already rounded. If you round them again to whole number percentages in your report, you could be making a mathematical error. Suppose there is a response of 120 out of a base of 320. This evaluates to 36.4741641…%. On a data table, this would round to 36.5%. You might then round that number up to 37%, when in fact the closest integer to the actual percentage is 36%.

Just some little things to keep in mind next time you have data around.

Why Do They Sell Vanilla and Chocolate Ice Cream?

Total Unduplicated Reach and Frequency (TURF) analysis is a statistical technique used by researchers to determine the smallest set of product or service characteristics that satisfy the largest audience.

Suppose you are asking:

1. In which media outlets should we advertise our restaurant to reach the greatest number of people and stay within our advertising budget?
2. Which flavors of ice cream should we offer if we only have space for two?
3. What would be the incremental value of adding a third flavor?
   

By analyzing both the frequency and overlap of respondent preference from a set of available options, TURF analysis can help answer these questions.

Consider this simple example:

Q: Which of the following flavors of ice cream would you buy?

1. Vanilla
2. Chocolate
3. Strawberry
   

Then suppose we asked ten people this question and got the following replies:

Respondent	Vanilla	Chocolate	Strawberry
1		X
2	X		X
3	X
4	X
5		X
6	X		X
7			X
8	X	X	X
9		X
10	X		X

In this example, vanilla is the most common answer with six responses, strawberry is next with five, and finally chocolate with four. Based on this simple frequency analysis, if you were limited to only offering two of these flavors, you would chose vanilla and strawberry since those two flavors received the greatest number of mentions in your poll.

However, upon examining the responses further you see that there is a large overlap between people who would buy vanilla and people who would buy strawberry. In fact, four of the five people who would buy strawberry would also buy vanilla if strawberry weren’t an option. There is very little overlap between vanilla and chocolate. Only one of the four people who would buy chocolate also say they would buy vanilla.

TURF analysis examines all of the possible combinations of flavors and calculates how many people would buy at least one flavor from each set. Here are the full results:

People who would buy…	Frequency
Vanilla	6
Strawberry	5
Chocolate	4
Vanilla or chocolate	9
Chocolate or strawberry	8
Vanilla or strawberry	7
Vanilla or chocolate or strawberry	10

The first observation that can be made is that any combination of two flavors will satisfy more people than offering any single flavor. A second, more interesting observation is that the combination of vanilla and chocolate will satisfy more people than any other two-flavor combination even though individually, chocolate was not one of the two most preferred flavors.

The reason for this is that if you are already offering vanilla, the incremental value of also offering strawberry is low. In fact, only one person in the poll who would buy strawberry says he would not buy vanilla.

Ideally we would offer all three flavors of ice cream; however we stated in our initial question that we only have space for two. Suppose that we have only two dispensers, but have ample space in our freezer to store strawberry ice cream in the event that either chocolate or vanilla runs out. This data shows that it would be more detrimental if chocolate ran out than vanilla since the combination of chocolate and strawberry reaches more customers than the combination of vanilla and strawberry.

None of these new findings could be known by looking at frequencies alone. Understanding TURF analysis helps uncover new and interesting things in our data and makes it clear why they sell both vanilla and chocolate ice cream.

Will You Engage Me?

The emergence of online research has created an illusion that researchers now have an endless pool of survey respondents. But the competition for a share of a person's online time is fierce. There are far more enjoyable things to do online than answer long, monotonous, and poorly worded surveys.

According to Simon Chadwick, CEO of online sample provider Peanut Labs, the average response rate for email-based survey invitations has dropped from 20% five years ago to only 4% today. Attrition rates on many panels are now reaching 50% per year. If we as researchers do not engage respondents by offering an enjoyable survey-taking experience, we run the risk of this pool drying up.

Last week, Survey Sampling International (SSI) announced its Respondent Preservation Initiative; an industry wide call to action to preserve and protect the most valuable resource to researchers - the respondent.

"The Initiative is a full-scale program to educate, support, and direct the industry. It places the people who give their time and opinions to marketing research front and center because it is people who, by sharing their insights, help businesses stay informed and reduce risk in decision-making."

Many online sample providers are now beginning to measure their panel members' experience with particular surveys and are offering discounts to clients whose surveys are engaging and enjoyable. This, of course, is a euphemism for saying that they will charge you more if your survey is confusing, boring, or uninteresting.

Advantage Research supports SSI in the Respondent Preservation Initiative and encourages the rest of the research community to do so as well.

Ink? With a "k"?

Advantage Research Ink (with a “k”) is the official blog of Advantage Research, Inc. (with a “c”). If you’ve worked with the people at Advantage Research, Inc. (with a “c”), you might enjoy reading Advantage Research Ink (with a “k”) for our refreshing thoughts and insights on the marketing research industry. If you haven’t worked with Advantage Research, Inc. (with a “c”) we’d invite you to read along as well to learn more about us. We’d love to help you with your next research study and have you become one of our klients.