CONDUCTING AND ADMINISTERING SURVEYS

 

              (Sources:  Schuh and Upcraft, Assessment Practice in Student Affairs, “Designing and Selecting Quantitative Instruments; Giraud, G. National Wellness Conference, 2001; and the Office of Assessment of Programs and Services in Student Affairs, University of Illinois at   Champaign-Urbana, http://www.apssa.uiuc)

 

                               

 

I.                   Step One: Decide what information is needed. Take the time to clarify purpose

 

A.    Determine the purpose of the survey. Are you trying to determine:

 

1.      What is optimal?

2.      What is actual?

3.      What feelings/priorities do respondents have?

4.      What causes those feelings?

5.      What problems exist?

6.      What are possible solutions?

 

B.     Consult with the Institutional Review Board (IRB) of the University

 

1.      Dr. Sandra Holmes of the Psychology Department is chair of the IRB

2.      She will provide information about ensuring that your study meets the guidelines protection of human subjects in research

 

II.                Step Two: Decide what population(s) to survey.

A.    Determine (in general terms) who is most likely to possess the information you are seeking (may be multiple audiences)

 

III.             Step Three: Establish your sample. Get the right people. Who do you want to generalize the results to?

 

A.    Determine the precise target population for this specific survey

B.     Determine what portion of the target population is accessible for this survey

C.    Identify or generate a sampling frame

D.    Determine what sampling strategy to use

E.     Draw the sample

 

 

IV.             Step Four: Deciding on Measurement Scales

A.    Categorical Scales: Nominal Scales (categories are mutually exclusive and a case goes into one category or another)

 

1.      These scales categorize objects: they “name” them

2.      gender, ethnicity,  and religious preference are nominal variables

3.      A trait is in a category or not

4.      Examples:

 

                                          a. I like Diet Pepsi                               ____Yes          ____No

                                          b. Please indicate your gender            _____Male       ____Female

                                  5. Another form: check all those that apply

 

a.       Please indicate your ethnicity (check all that apply)

 

____African American   ____Hispanic    ____White

 

____Asian American      ____Native American

 

B.     Continuous measures: Interval Scales (Example: Likert Scales- Likert items are statements, or questions, single words, or phrases on a continuum)

 

1.      Use of a scale to indicate the strength of the reaction to the statement

2.      The scale could indicate agreement/disagreement, like/dislike, important/not important, relevant/not relevant, etc.

3.      Range of responses using a numerical scale

4.      Common ranges are 5, 7, and 9 point scales, with a no opinion or don’t know option; a middle of the scale option could be neither agree or disagree

 

C.    Sources for Likert Type Statements

 

1.      Researcher’s interests

2.      Interviews with persons holding views of interest to the researcher

3.      Review of relevant literature

 

D.    Suggestions for Writing Likert Items: The suggestions are aimed at making the items as clear and free of obvious bias

 

1.      Avoid statements that refer to the past rather than the present or future

2.      Avoid statements that can have more than one interpretation

3.      Avoid statements that are irrelevant to the attitude being measured

4.      Avoid statements that are likely to be endorsed (or rejected) by almost everyone or almost no one

5.      Select statements that cover the entire range of the attitude or emotion

6.      Keep the language clear, simple, and direct

7.      Statements should be short, rarely exceeding 20 words

8.      Each statement should contain only one complete thought

9.      Statements containing words such as “all, always, none, and never” should be avoided

10.  Words such as “only, just, merely” are leading and should be used with care

11.  Whenever possible, statements should be in simple sentences rather than compound or complex sentences

12.  Avoid use of vocabulary that might not be understood by respondents

13.  Avoid the use of negatively phrased statements

14.  Include equal numbers of positively and negatively oriented statements to reduce response bias

15.  Statements should be clearly negatively or positively oriented

16.  Avoid emotional words and phrases

17.  Draft questions that do not presume a particular answer

18.  Provide information necessary to answer the question

 

E.     Ordinal Scales

1.      Rank order objects according to the amount they possess of a certain trait

2.      Example: the order of finish in a horse race is an ordinal variable

3.      Another example: Please rank order your on-campus living preference for next year: ___single room    ____double room   ___suite

 

F.     Ratio Scales

 

1.      They have all the ordering characteristics of nominal, ordinal, and interval scales, but have an empirically meaningful zero (i.e.family income, age, years of formal schooling completed, number of times a student meets with an advisor).

2.      Ratio scales can be converted to ordinal scales for analysis

 

a.       age can be in categories, such as under 18, 18-22, 23-25, and over 25.

b.      Thus, ratio scales have some strong benefits

 

G.    Choosing A Scale: (refer to Quantitative Methods in Assessment)

 

1.      Types of statistical analyses are related to the choice of a scale

2.      If you want to find out means, you need to use interval or ratio scales

 

H.    Constructing Scales

 

1.      Scale: a set of items put together to measure a more complex idea

2.      Constructs of interest usually have several components

3.      Scales and subscales: A scale on job satisfaction (two items on coworkers and two on supervisors as potential subscales)

 

1.      I like my coworkers

2.      I feel comfortable around my coworkers

3.      I feel my supervisor supports my work efforts

4.      I trust my supervisor

 

4.      Other issues in scale construction

 

1.      don’t overdo it: carefully write items that address areas of interest

2.      limit the number of items

3.      the length of items ensures reliability and validity, while not be so long that respondents are annoyed with the length

 

                  F. Writing Demographic and Informational Items

 

1.      Gain information about personal characteristics of respondents, workplaces, etc.

a. .Use of forced choice responses:

                                                                                             (1)      What is your income?

(a)    0-10,000

(b)   10,000-20,000

(c)    20,000-30,000

(d)   30,000 or more

 

 

                                                                                             (2)      Benefit: ease of use

                                                                                             (3)      Drawback: may limit potential responses

 

                                                                                             (4)      Use more choices rather than fewer

                                                                                             (5)      Use responses that are likely to fit the respondents

                                                                                             (6)      In general, tailor your data gathering to questions you want answered

                                                                                             (7)      Carefully consider whether open-ended or forced choice responses are best for your survey

                                                                                             (8)      Carefully consider the range of responses appropriate for your question and the respondents

 

                      G. Determine the Sequencing of the Questions

 

1.      Start with questions that are easy to answer

2.      Place sensitive information late in the instrument

3.      Pose related questions together

4.      Follow a logical sequence of questions

5.      Use filter or screen questions very carefully, i.e. if you answered yes to the previous question, respond to the next series of questions

6.      Consider reliability checks: use of the same question later in the questionnaire

7.      Ask demographic questions last: keep the questions to a minimum and put them last so as to engage the respondent in the questions in the early part of the questionnaire

 

                                                                

 

                              

 

H.    Survey Formats

 

1.      Surveys should be as “clean” as possible, free of distracting graphics and fonts.

2.      Instructions should be clear

3.      Instructions should be repeated at the top of each page when groups of like items require more than one page

4.      Use of scan sheets or electronic methods (email, web sites) are helpful with large number of subjects

5.      When using telephone or personal interviews, rehearse the interviewers using the script that covers all likely situations related to the survey

 

V.                Step Five: Plan the distribution, return, and follow-up procedures

 

A.    Distribution of the Sample

 

1.      Hand out in meetings or on-site

2.      Deliver through internal mail systems

3.      Deliver through intermediaries (batch mail)

4.      Mail directly to individuals

5.      Electronic distribution: e-mail, website, or use of list server

 

B.     Return

1.      Return envelop addressed and stamped/franked

2.      Return envelope addressed only

3.      Return to collection box or individual

 

VI.             Step Six: Write a good cover letter. A good cover letter will include the following information:

 

A.    Why they received the survey

B.     The purpose of the survey

C.    Why they should respond (using either personal or professional/institutional appeal)

D.    How and when to respond

E.     Whether responses will be anonymous or confidential

F.     Your appreciation for their participation

G.    Sponsorship- who is underwriting this

H.    Incentives for responding

 

VII.          Step Seven: Pilot test the survey. Determine that the questions “work” with a small sample

 

A.    Select an accessible pilot sample from the target population

B.     Administer the survey and cover letter

 

1.      Initially, try out a few people

2.      Interview each person to identify needed revisions

3.      After the pilot test, field test with a full mail procedure

 

C.    Revise, as necessary

D.    Repeat above, as necessary

 

VIII. Step Eight: Determine the questionnaire’s technical accuracy

 

A.    Internal validity: reflective of the truth or reality

 

B.     Reliability: consistency or steadiness

 

1.      Are factual questions repeated?

 

                  

IX. Step Nine: Improving response rates

 

A.    Relevancy

 

1.      Ensure that the topic is relevant to the respondents

2.      Articulate the What’s In It For Me (WIFM) in the cover letter

3.      Articulate any sponsorship

4.      Inform participants that a survey is coming (pre-notification)

5.      Encourage participants with a promised reward (money, gift certificate, etc.)

 

B.     Immediacy

                                     1.Maximize timing of issues and survey distribution

 

C.    Convenience

 

1.Make the questionnaire as brief and easy as possible

                                    2.Allow sufficient time for response

 

D.    Face Validity (Cosmetics)

 

            1.Make the survey look relevant and “official”

   2. Ensure that it is visually pleasing and free of typo’s

                                    3.Follow-up: 1st follow-up- include full survey in the follow-up; in the second follow-up, use post card or letter only

X. Summary

 

A.    Advantages of Surveys

 

1.      Self-administered, allowing time for the respondent to consider answers

2.      Anonymity can cause more honest responses

3.      Relatively economical

4.      Easy to score and analyze (if properly constructed)

5.      Easy to seek respondents’ reactions to content that may be difficult for them to react to with other data collection methods

 

B.     Disadvantages of Surveys

 

1.      Frequent low percentage of returns

2.      No assurance that the intended respondent understands the questions

3.      No assurances that the intended respondent actually completed the form

4.      No opportunity to interact with the respondent to clarify, probe, or seek substantiation