|
|
|
| Online Surveys
FAQ |
|
|
|
|
Our
online survey response rates are not consistent. They
are excellent on one survey and poor on another. What
can we do to assure adequate response rates, especially
from specific segments?
How
do you determine the survey sample size and segment
sample sizes needed for a valid scientific sample of
the population?
How
do I conduct a SMART survey when only some of our target
population have provided e-mail addresses and others
may not have access to e-mail or the Internet?
Isn't
there a problem with online surveys of a lack of control
in maintaining a legitimate random sample when people
can self-select to take the survey?
I
have a concern about people taking our surveys on the
Web site who are not authorized. How do I know which
survey data is legitimate and which is not?
What
about the confidentiality of people's identity and responses
in e-mail surveys? When we use e-mail, our company can
read it if they want to. Can they read our survey responses
as well?
We
do not have the internal expertise and resources to
conduct surveys of any kind let alone SMART surveys.
Do you have any suggestions?
I
have heard that SMART surveys cost less and are faster
than print surveys. Is this really true, and if so,
how much faster and cheaper are they?"
|
|
|
|
|
Good consistent response rates result from doing all
the basics correctly (and aggressively). It starts with
a survey design that offers a true value proposition
to the participants. To develop a true value proposition,
you need to talk with, or meet with, a few groups of
prospective participants and explore their concerns
and issues and the various types of value options (survey
content, incentives, confidentiality, etc.) which would
appeal to them. Then test the value proposition with
one or two more groups.
With a tested value proposition, you next have to communicate
the proposition ahead of time via e-mail (and print
if appropriate) to create awareness and acquire buy-in.
The cover message which accompanies the questionnaire
must resell the value proposition and lead quickly to
the act of clicking open the questionnaire. The opening
view of the questionnaire page must be simple, inviting
and friendly with the first couple of responses right
at hand to begin their engagement with the questionnaire
quickly.
If you are going to analyze your findings by segments,
your communications and value proposition should be
tailored if possible to each segment's interests. Also,
if possible, use custom e-mail cover messages for each
segment with their value proposition clearly explained
in the invitation. If your address list is large enough,
it is a good idea to have a secondary list of additional
e-mail addresses of people in your segments in reserve
which can be used on an "as needed" basis.
Give people adequate notice and deadlines. Although
many responses, as many as 50% of what you need may
come in within 24-48 hours, many SMART survey respondents
tell us they would like a week to ten days to complete
a survey. Responses should be tracked on a daily or
every other day basis (both overall and by segment).
At the halfway mark, targeted e-mail message reminders
should be sent to non-responders. This practice should
be completed every couple of days through the end of
the survey.
Deadline extensions of 24-48 hours should be planned
into the schedule in advance as a precaution. If needed,
a deadline extension announcement should be sent in
the afternoon of the last day to non-responders. Also,
if needed, send invitations to some or all of the people
in your reserve database.
Incentives can also help increase response rates. It
is best to make them substantial and to introduce them
right at the beginning so that the motivational value
can be incorporated into the value proposition and applied
as part of the overall frequency of communications during
the survey process.
As a last resort, if you still have not achieved a required
sample size, you can employ a telephone survey facility
to conduct a limited number of outbound interviews to
complete your sample. There are also two things you
can do with an undersize sample at the statistical analysis
level. One is to carefully weight the sample overall
and by segments to adjust the database to reflect each
population's size in the sample - which you should do
with every survey sample - before beginning the data
analysis. The other is to use analytic software such
as SPSS Exact Tests, which analyzes and adjusts statistical
reports to account for small sample sizes. Both of these
techniques should be done by a research professional.
|
|
| Return
to Top |
|
|
|
There are pre-calculated tables available for this as
well as software such as SPSS's Sample Power. We have
created two free,
easy-to-use tools for this that are available here
on the Web site at free research tools.
One is a table in Microsoft Word97, which tells you
the sample sizes required for populations up to 10,000
at various confidence levels and margins for error.
The other is a software applet for SPSS users, which
will calculate the margin for error of a given sample
for any size population and will also calculate the
exact sample size required for any population.
|
|
| Return
to Top |
|
|
|
The answer may be for you to consider a dual-media survey,
which uses both an electronic survey to quickly, and
inexpensively reach those who are online and another
survey method (print, telephone, etc.) to collect data
from those who are not online.
We do this routinely for many of our clients. The world
is in transition with many people already working and
relying on the online world and many who are not yet.
The dual-media survey reflects the reality of the world
we live in. It may sound more complicated than it is,
but most professional survey firms working with online
surveys should be able to do it for you. There will
be some extra costs involved because two questionnaires
will have to be created, one for each media, and there
may also be some additional costs for database merging.
Outside of these two functions, costs should not be
any different than conducting a survey in a single media.
|
|
| Return
to Top |
|
|
|
This is no more of a problem than with traditional paper
surveys. With traditional print surveys, the development
of a random sample list of every "nth" name is done
before the invitations and questionnaires are sent out.
The same is true of e-mail address lists for online
surveys. Keep in mind that no matter how you randomize
your lists, survey participants always self-select as
to whether to participate after receiving the questionnaire.
The same is true of people visiting a Web site and encountering
a survey offering. It is the same as if they encountered
a survey offering in the mail or in a publication.
If you're fortunate enough to have a larger sample size
than you need, you can always randomize the completed
data records you have if you wish, however, most researchers
are delighted to have a larger sample than they need
because it reduces the margin for error on the survey.
For those who are very concerned about this, you can
conduct outbound telephone interviews with non-responders
to determine whether their responses are statistically
different from those who responded and factor that information
into the analysis. Pilot surveys can also help in this
regard.
|
|
| Return
to Top |
|
|
|
If you are conducting a public survey which is available
to any visitor on your Web site, you must design your
survey with qualifying filter questions built-in so
that inappropriate records can be identified and discarded
during the database cleaning process.
If you are working with a defined list of prospective
survey participants who are being invited to participate
in your survey, you can send them qualifying passwords
and individual survey ID codes to use when taking the
SMART survey. These authorize their access to the questionnaire
and validate their responses. This assures there will
be no unauthorized participants or multiple responses
from the same person.
If you don't want participants to deal with passwords
and survey ID codes, we can program our e-mail invitation
lists with special individualized codes which will be
automatically checked against an authorized database
when a participant begins an SMART survey questionnaire
on our Web site. This is transparent to the participant
and eliminates the need for participants to deal with
entering passwords, etc.
|
|
| Return
to Top |
|
|
|
The issue is really more about people's concern about
confidentiality than the reality of the confidentiality
because the reality is that SMART surveys can be made
quite confidential.
It is true that the law says that e-mail on company
systems is company property and can be reviewed; however,
we use a number of safeguards to protect people's confidentiality
on SMART surveys. First, response data entered in our
questionnaire software cannot be saved on a PC's hard
drive. Only the blank questionnaire itself can be saved.
The responses only exist during the time they are being
entered. Once sent, there is no record on the respondent's
hard drive.
As far as the e-mail file being captured in transit,
only the data is transmitted so it is relatively meaningless
to a reader who does not have the translation software
program plus the individual custom program created to
read the data from that specific survey.
It is also highly unlikely that a corporation's IT resources
would be directed away from mission critical tasks to
try to capture mostly undecipherable raw data from a
survey.
In addition, if a client desires, we can apply SSL (Socket
Security Layer) which encrypts each file from the browser
to the destination server in the same way credit card
transaction information is protected on the Internet.
Plus, we use a third party's neutral server on the Internet
to strip out any e-mail address identification, which
might be present in the file, before the response file
is forwarded to us. When we receive the file, it really
is anonymous except for data entered voluntarily by
the respondent.
What is important is responding to people's real concerns
about confidentiality in advance to allay their concerns
as much as possible. This should be done by reassuring
them in every communication that the survey is confidential
along with a brief explanation of the safeguards in
place to protect the confidentiality of their responses.
Additionally, we find that most of our clients prefer
to have the Web site survey hosted on our password protected
Web site because people are often more comfortable interacting
electronically with an independent third party firm
that guarantees to protect the confidentiality of their
responses.
It is generally considered true that higher response
rates can be achieved by using a third party research
firm's Web site to collect confidential data.
|
|
| Return
to Top |
|
|
|
Like any business task, you have to assign resources
to get it done. If the information a survey can provide
is important to your success, put some resources against
it. There are some basic survey software packages which
can be self-taught, and Microsoft FrontPage 2000 has
form functionality to create questions and collect information
from Web site visitors. These can be useful for short
polls and simple surveys if someone in your organization
has the time to master the learning curve.
However, if your needs for sample design, questionnaire
construction, question writing, data collection, data
management, data analysis and report preparation are
anything more than rudimentary, I would strongly suggest
calling in a professional resource, either an individual
consultant or a research firm, to help you successfully
complete your survey project.
|
|
| Return
to Top |
|
|
|
It is true that SMART surveys can cost less and deliver
findings faster than print surveys. They also generally
deliver higher response rates and better quality comments
on open-end questions.
How much "faster and cheaper" depends on the questionnaire
and the participants. Shorter SMART survey questionnaires
deliver quicker responses than lengthier, complex questionnaires.
They also take less time to produce and deploy. Some
populations are more high-energy cultures than others
and respond more quickly.
In general, it takes anywhere from a day to a week to
program and deploy an SMART survey once the questions
are in hand. There is no mailing time involved. Most
responses will come back within a few days with the
rest coming in over a week to 10 days.
Reports can be ready within hours or days after the
responses are collected depending upon the complexity
of the reports because the data is already in electronic
form. And the open-end comments are already typed, so
reporting of these findings is eminently quicker and
easier. Because people are typing their comments, they
tend to write more complete ideas and expand on them.
The whole process can be as short as 48-72 hours or
as long as several weeks depending on the nature of
the survey.
Regarding cost savings, there are no printing or postage
costs, and the communications and reminders can be sent
within minutes and include another copy of the questionnaire
so you are assured everyone has a copy during the entire
period of the survey. And, of course, with a Web-based
survey, the questionnaire is always available. There
are also no data entry costs and no transcription costs
for open-end comments.
A general rule of thumb is that SMART survey data collection
can cost a third to half of what print survey data collection
costs. There is no way to estimate the benefit and opportunity
costs saved by having information weeks or months sooner
than from a print survey, but these can be considerable
in a competitive, demanding environment.
|
| Return
to Top |
|
|
| |
|
|
