FAQs About Conducting ResearchHow much should I worry that my research won’t reach cell-phone only households?
The answer to this question is dependent on several factors. - How important is it to include information from a younger, mobile, renter and more often minority population?
- Are you basing your research on a single method of data collection?
Gilmore has been working with clients in the field of health trends and behaviors since the mid 1980s. One of the major projects, the “Behavioral Risk Factor Surveillance System” is the world’s largest, on-going telephone health survey system, tracking health conditions and risk behaviors in the United States. Because this is a comprehensive tracking study, the sampling design for it is critical.
As the number of reported “cell phone only” households has increased, more concerns have been raised about the information that might be missing in the annual BRFSS research. At the June, 2008 MRA Conference, Dr. Jeanne Wintz from Gilmore Research delivered the results of a special study comparing health data from cell phone only households to those in the general household sample. Information from this special research indicates that the cell phone only household population is definitely a particular segment of the general population and not simply a cross-section of all consumers. Research that does not address this issue may indeed systematically miss an entire segment of people.
Marketers and researchers who want to measure any behavior or opinion patterns among the population should consider the potential impact of “cell phone only” households. For a copy of the slides Dr. Wintz used in her presentation at the national MRA conference, go to Cell Phone Only Research Slides - MRA 2008. Back to Top When should I have quantitative research conducted and when should it be qualitative?
In the simplest of terms, quantitative research is conducted when you want to count something (“How many potential buyers recognize my brand name?” / “What is my brand’s share of market?” / “How many registered voters are concerned about government spending?”)
Qualitative research is more often your choice when you want to investigate or understand people’s opinions or perceptions about something (“Why do consumers prefer family size portions of the competition and single serving sizes of my brand?” / “What are the main reasons alumni join the Alumni Association at their alma mater?” / “What changes in service would increase patient satisfaction with a healthcare provider?”)
The most thorough approach to conducting research on an important topic might involve both qualitative and quantitative methods. Depending on the particular topic being studied, the order in which the methods are used may be different. For example, a qualitative approach might be used to identify the attitudes or issues that are involved followed by a quantitative method to measure the prevalence of those attitudes. “First, identify the major reasons why athletic club members choose to belong to a particular facility and then survey the membership to find out how many of them joined for each of those reasons.”
In other cases, a survey may bring up issues that need to be understood at a deeper level which can be accomplished with a more qualitative method. “The historic market leader loses market share mainly among young consumers in a quantitative survey. Focus group sessions or in-depth interviews with such consumers allow the investigation of assumptions they have regarding that leading brand resulting in the identification of previously unrecognized vulnerabilities.” Back to Top How can I select the best research partner for my needs?
A great deal of the success you experience in selecting the right research partner for your projects has to do with the determination of what it is that you really need. In some cases, these needs may change from one project to another. In other situations, once you find a research partner who meets or exceed your needs for one project, you may be able to incorporate them into your planning process for other projects – especially if your projects often require similar capabilities.
In general, however, there are three steps to consider when you begin the process of finding a research partner. Back to Top When should I have a professionally respected “third party” conduct the research I need instead of my in-house staff?
One of the reasons for having an in-house staff design and conduct research for your product or company is the efficiency that comes with a high level of familiarity with the topics being investigated or measured. However, at times, that familiarity can be perceived as having a potential impact or bias on the results.
In the case where there may be differences of opinion among either internal constituencies or external audiences, the clearest way to provide an assurance of objectivity in research results is to bring in a respected third party to conduct the study. Such a firm can offer not only impartiality in analyzing data and making recommendations but also “fresh eyes” in the design of the research. Where familiarity increases efficiency, a new view of the situation may include investigative directions that otherwise might be overlooked. Back to Top Is a random probability sample all that different from other samples?
A population that we may want to study with research refers to all individuals with certain common characteristics (residents of the United States, people who have purchased an automobile within the past six months, etc.). A sample is a subset of that population. The goal of a research study is to obtain information about certain characteristics of the sample, that will provide us with an estimate of the prevalence of the same characteristics in the entire population from which the sample was drawn. The conclusions and estimates resulting from the analysis of the research are only as good as the sample selection process will allow.
One of the most reliable methods of sample selection is the random sample. If a sufficiently large sample is chosen using truly random methods* (and the response rate is sufficiently high), then the estimates based on the sample results can be interpreted within a predictable "error range" (or margin of error). Within that range, they are then referred to as being "projectable" or representative of the entire population.
Once the random sample has been selected, it is important that all members of the sample be given ample opportunity to be included in the results. If response patterns are biased, the value of the random sample is diminished. An online survey of consumers can only be projectable to others who have online access. A telephone survey conducted during the day can only be projectable to those who are available to take a daytime phone call.
For this reason, responsible researchers try reach as many members of the sample as possible. Online surveys may be supplemented using other data collection modes such as mail or telephone. In a telephone survey, it is preferable to call potential respondents at different times of the day and different days of the week to reach people with different schedules. Each number selected should be called a minimum of five or six times at different times and days before that number is discarded and replaced with a new randomly selected number.
Response rates are useful measures of the quality of a sample. These compare the total number of responses to the number of eligible members of the sample. A high response rate indicates a completed sample that will be more representative of the population. When response rates are not as high as we would like, we may conduct a non-response bias analysis to determine if the characteristics of those who did respond to the research effort are significantly different from those of the non-responders. This may consist of comparing known characteristics of the non-responders to those of the responders or conducting additional, often costly, analysis to collect data from non-responders.
The quick answer as to whether or not a random sample is “all that different” from other forms of sampling is – “Yes.” It is the sampling method most likely to produce results that can be projected to a total population within a measurable range. CAUTION: Just because research results include the “+/- 5% at the 90% Confidence Level” statement does not mean the sample necessarily has the elements that make it a random sample. Too many studies today include these measurements when they are not reporting results based on random sampling methods. Back to Top What is involved in moving my tracking study from telephone or direct mail to online?
There are many reasons why changing methods of collecting data for an ongoing research measurement can be an appropriate course of action. Often, changing budgets and time frames are the cause of such revised designs. In the strictest sense, statistical comparisons of tracking study results conducted using different methods of data collection are measurements of the difference of those data collection methods rather than of the measurements across time. Later, subsequent measurements using the new method can then be compared and changes in study metrics can again be tracked.
To create as smooth a transition to a new method as possible, there are several areas on which to focus. Determine the best match from a sampling perspective. (That is, keep the sampling design as consistent as possible. For example, if you are moving to online surveys, remember that you will have the additional limitation of only contacting those for whom you have current and accurate email addresses.) If you are transitioning to an online panel as your sample source, be sure to fully understand the source of the panel members. The most dramatic differences may occur in transitioning studies that measure patterns among the general population to online panel memberships. That is, there is a higher likelihood of differences between “random samples of the general market” and members of an online panel than between samples of your customers however they have been identified. Be aware that differences in the delivery of questions to the respondent as well as the entire respondent “experience” may well impact the way they answer questions. (For example, many self-administered methods allow respondents to view the entire survey before answering any questions – virtually eliminating any “unaided” measurements. If you are moving from a telephone method, that will has not been the case in results from your previous studies.)
Back to Top When is it better to use focus groups than in-depth one-on-one interviews?
The major question to ask when deciding whether group sessions or IDIs are better for your research study is “How important will it be to have the impact of group dynamic in your investigation?” There are actually two sides of this question which represent the positive and potentially negative impacts of group discussion.
A negative impact of discussing your topic in a group could be related to it being too personal for group members to be comfortable discussing with others. With a trained moderator, many topics which may seem fairly sensitive can be addressed successfully in focus groups (from alcohol consumption to personal hygiene to satisfaction with funeral services.) However, when the discussion of the topic in a group might lead to distorted or concealed information, using individual interviews may be a much better course of action.
The positive impacts of group sessions can be seen in almost any session that is appropriately conducted. Group discussion allows you to investigate how easily participants can be moved from one position on an issue to another without sacrificing rapport with group members. That is, when the moderator encourages the identification of differing opinions in a session, he or she creates an acceptance of challenges between group members. If the method is one-on-one interviews, that challenge would have to come from the interviewer, potentially degrading the relationship and reducing the information that might be generated. In these cases, focus group sessions will provide the insights you seek without impacting the effectiveness of the moderator. Back to Top How projectable is a sample from an online panel?
In today’s world of tight budgets and time lines, the speed and often lower costs of a web survey make it a popular choice for many studies. When the population that you are interested in studying is appropriate for an internet methodology, web surveys, especially among members of established panels, can be a quick and easy way to conduct research.
It is important, however, to understand how the panel as the source of your sample should be viewed. Because a panel consists of individuals who have chosen to join the panel (and not those who elected not to join or never had the opportunity to do so), survey results from a sample of those individuals are only directly projectable to the panel not the total population. Because respondents from a web panel sample do not fit the requirements of a random probability sample, statistically speaking, survey results should not be evaluated using error range calculations.
This, however, is not to say that results from a web survey of panel members do not reflect the total population you are targeting. Characteristics of your total population such as demographics or product usage patterns can be compared to the sample of panel members who completed your survey. If there are differences in critical elements between the profile of panel respondents and the overall population, weights can be applied to balance those differences. Back to Top Are online panels all about the same?
The quick answer to this important question is “NO.” In the last few years, the increase in the number of online panels has certainly suggested high interest in the internet survey method for marketing research. Unfortunately, more does not necessarily translate to better.
As with any sample, the source of individuals who are members of a panel is critical. Important questions to ask when you are investigating a panel include whether members are invited to join (that is, are known to the panel before being invited) or if they have actively pursued membership (an indication of potential problems presented by “professional respondents.”) Incentive policies also affect the quality of panels in the impact they have on member loyalty and / or retention rates. Some methods actually may encourage falsification of responses in order for members to qualify for more surveys and maximize the incentives they receive.
Another issue that is becoming more common is that of the “brokered panel.” That is, some providers are touting huge panel sizes due mainly to their brokering of other panel memberships. Simply brokering access to a number of different panels does not guarantee the quality of responses you might experience when surveying a sample of its “members.” More reputable panel firms have methods of cross-comparing members of the panels they broker in order to minimize duplication between the panels. Even though members who use different computers for different panels may not be easily identified, some panel brokers have technology that can identify response patterns to further reduce the number of such respondents.
An excellent method of fully understanding the quality of a panel you are considering is to ask some or all of the questions suggested by ESOMAR (a respected worldwide association of opinion and marketing research firms). Truthful responses to these questions will provide valuable insights for you to consider when making your panel selection. (www.esomar.org) For further discussion, see "How to Select the Best Online Panel for Your Research." Back to Top What can I do to increase response rates to a mail survey?
Highly experienced in the design and implementation of direct-mail methodologies, Gilmore understands the factors that impact return rates, the most important consideration for this type of research. A cover letter accompanies the self-administered questionnaire to provide the respondent with an understanding of why it is important to fully respond to all questions. For some studies, incentives are offered to encourage participation. The questionnaire itself is designed to facilitate responses and minimize the potential for leading the respondent. A convenient method for returning completed questionnaires also helps to ensure the desired rate of return. At Gilmore, we frequently utilize the Total Design Method (TDM) pioneered by Prof. Don Dillman, which is very effective at achieving high response rates with mail surveys. The Total Design Method involves a systematic approach to contacting respondents and a subsequent follow-up procedure. Attaining the highest response rate possible is essential for the success of a mail survey and the TDM helps ensure that successful outcome.
|
Quick Reference Center