Survey Research

•      Results of surveys are used to describe people’s opinions, attitudes, and preferences.

•      Survey results are used to make predictions about people’s behavior.

•      In typical survey research, a sample of people completes a questionnaire(s); responses from the sample are used to describe the population.

•      The survey involves using a predetermined set of questions.

Correlational Research

•      Correlational research: Assess relationships among naturally occurring variables.

For example: Attitudes, preferences, intelligence, personality traits, feelings, age, sex

 

•      Researchers calculate correlation coefficients to determine the strength and direction of a predictive relationship between two variables.

       -1.00   to   0   to   +1.00

(negative)                    (positive)

Obtaining a Sample

•      Researchers are not interested simply in the responses of those surveyed

     — they seek to describe the larger population from which the sample was drawn.

•      Careful selection of a survey sample allows researchers to generalize the findings from the sample to the population.

Basic Terms of Sampling

•      Population: set of all cases of interest. For example:

–   current students at your institution

–   current residents of your city

–   citizens of the United States

–   citizens in North America

•      Sampling Frame: list of the members of a population.

–   For example, registrar’s list of all currently registered students

•      Sample: subset of the population used to represent the population.

–   Students in your class as a sample of current students at your institution (or your city, United States, North America)

•      Element: each member of the population.

•      Goal: Sample should represent the population.

–   Characteristics of participants in the sample should be similar to those of the entire population.

–   Example: Which sample represents a population that is 30% freshman, 30% sophomore, 20% junior, 20% senior?

 

Sample A                                             Sample B   

30 freshmen, 30 sophomores,   60 freshmen, 60 sophomores,

20 juniors, 20 seniors                40 juniors, 40 seniors

 

Both! But note: The samples are representative on one feature only!

•      A biased sample occurs when the characteristics of the sample differ systematically from those of the target population.

–  A sample may under-represent a segment of the population, or

–  over-represent a segment of the population.

•   For example, most samples in psychology research overrepresent college students and underrepresent individuals who are not in college.

•   Most research underrepresents individuals from diverse cultures.

•      Two sources of biased samples:

 

–  Selection bias occurs when the researcher’s procedures for selecting a sample result in one or more segments of the population being under- or over-represented.

•   Example: Researcher places sign-up sheets for a research study in a Psychology Department. Psychology students are likely to be over-represented because of the selection procedure.

–  Response bias occurs when individuals selected for the initial sample do not complete and return the survey.

•   Example: People who receive the survey aren’t interested, they’re worried about privacy, have vision or other problems, don’t have time, etc.

•   Final sample will only represent the population of people who are interested, not worried, have good vision, time, etc.

 

Approaches to Sampling

•      Sampling refers to the procedures used to obtain a sample.

 

•      Two basic approaches to sampling are

 

–  nonprobability sampling, and

–  probability sampling.

•      Nonprobability sampling: No guarantee that each member of the population has an equal chance of being included in the sample.

–  “Convenience sampling” occurs when the researcher selects individuals who are available and willing to respond to the survey.

•   Examples: Magazine surveys, Internet surveys

–  Lots of psychological research uses convenience samples (but this can be OK).

•      Probability sampling: All members of a population have an equal chance of being selected for the survey (this is called a “simple random sample”).

 

–  Need to have a sampling frame (list) of people in the population, or

–  use random-digit dialing (but not all members of the population may be included).

•      Stratified Random Sample: The population is divided into subpopulations called “strata.”

•      Random samples are then drawn from the strata.

–   For example, strata from a university population potentially include freshmen, sophomores, juniors, seniors, graduate students, faculty, staff, administrators.

•      Stratified random sampling increases the likelihood that the sample will represent the population.

Survey Methods

•      Four methods for obtaining survey data are:

–  mail surveys,

–  personal interviews,

–  telephone interviews, and

–  Internet surveys.

•      Each method has advantages and disadvantages.

•      Researchers choose depending on the nature of their research question.

•      Mail surveys

–  quick and convenient, self-administered, best for highly personal or embarrassing topics.

–  may have the problem of response bias when people selected for the survey sample don’t complete and return their survey.

–  Due to response bias, the final sample may not be representative of the population.

–  Because mailed surveys are self-administered, respondents are free to interpret questions as they see fit, leading to possible differences in how people respond to questions.

•      Personal Interviews

–  are costly, but researchers gain more control over how the survey is administered, and how people interpret survey questions.

–  Interviewers can seek clarification of answers.

–  potential problem: interviewer bias.

–  Interviewer bias occurs when interviewer records only selected portions of respondents’ answers, or interviewer words questions differently to fit particular respondents.

–  Interviewers must be highly motivated, carefully trained, and supervised.

 

•      Telephone Interviews

–  brief surveys can be completed efficiently and with greater access to the population.

–  Random-digit dialing technology allows researchers to select random samples.

–  Interviewers can be supervised easily from one location.

–  Potential problems include selection bias (only people with phones can be included), response bias (people may refuse solicitations to complete surveys over the phone), and interviewer bias.

•      Internet Surveys

–  The Internet allows for efficient, low-cost means to survey very large samples.

–  Samples can be very diverse and access typically underrepresented samples.

–  Potential problems include selection bias (access to computers and Internet required), response bias, and lack of control over the research environment.

•      Ways to increase survey response rate (and lessen problems associated with response bias):

–  Questionnaire has a “personal touch” (e.g., respondent are addressed by name and not simply “Dear student”)

–  Responding requires a minimum of effort

–  Topic of survey is intrinsically interesting to respondent

–  Respondent identifies with the organization or researcher who is sponsoring the survey.

 

Survey Research Designs

•      A research design is a plan for conducting a research project.

–   It’s the research method the psychologist chooses to best answer his/her research question.

•      There are three types of survey research designs:

–   Cross-sectional design

–   Successive independent samples design

–   Longitudinal design

•      The survey design researchers choose depends on their research question.

•      Cross-sectional Survey Design

–  A sample is selected from one or more populations at one time.

•   Researchers choose the population(s) they would like to describe.

•   They use either probability or convenience sampling — probability sampling will lead to a more representative sample.

•   Respondents complete a survey.

–  The responses are used to describe and make predictions for the population at that point in time.

•   Descriptive statistics and correlations are used.

–  If two or more samples are drawn from different populations, the populations can be compared.

–  Researchers cannot assess changes over time with cross-sectional designs.

•      Successive Independent Samples Design

–  A series of cross-sectional surveys over time.

–  A different sample of people completes the survey each time.

–  Each sample is selected from the same population(s).

–  Responses from the sample are used to describe the population at each point in time.

–  Researchers can compare the survey responses from each sample to see how the population changes over time.

•   Successive independent samples designs don’t tell us whether individuals change over time (because different individuals complete the survey each time).

–  Problem of noncomparable samples:

•   If different populations are sampled at each time, we don’t know if responses differ because of true changes over time, or because different populations were sampled.

•      Longitudinal Research Designs

–  The same sample of individuals completes the survey at different points in time.

–  This allows researchers to assess how individuals change over time.

–  Responses from the sample of respondents are generalized to describe changes over time in the population from which the sample was drawn.

•      Problems with longitudinal designs:

–  Just because people change over time, surveys can’t tell us why people change.

•   no causal inferences with correlational data

–  Attrition occurs when people drop out of the study.

•    the sample no longer represents the population from which it was selected

–  Reactivity: Respondents may strive to be consistent or become sensitized to the topic.

Measures in Correlational Research: Questionnaires

•      Survey researchers most frequently use questionnaires to gather information.

•      Psychologists measure different types of variables:

–  demographic variables (e.g., age, sex, race, socioeconomic status)

–  preferences and attitudes

•   most often these are measured with self-report scales

•   participants respond on rating scales (assume interval level of measurement)

Reliability and Validity of Self-Report Measures

•      All measurement must be reliable and valid.

•      Reliability refers to the consistency of measurement.

–  Test-retest reliability: Administer measure two times to the same sample. Individuals’ scores should be consistent over time.

•   A high correlation between the two scores indicates good test-retest reliability (r >.80).

•   Individuals need only to maintain their relative position in the distribution; they do not need the same score on each administration.

–  Note: We don’t expect some measures to produce consistent scores over time.

–  When people change on a particular variable over time, we expect the measure to have low test-retest reliability.

•   Would you expect these measures to have high or low test-retest reliability?

–  Depression symptoms

–  Personality traits

–  Intelligence

•      How do we improve reliability?

–  More items improves reliability.

•   If I use only one item to measure your research methods knowledge, the test probably wouldn’t be reliable.

–  There should be great variability on the factor being measured among the tested individuals.

•   When the sample is diverse, the measure will more reliably discriminate high and low individuals.

–  Reliability improves when the testing situation is free of distractions and instructions are clear.

•   Anything that decreases opportunities for errors improves the reliability of the measure.

–  Reliable measures make us more confident that we are consistently measuring a concept within a sample, but are reliable measures truthful?

•   I could reliably measure your research methods knowledge by measuring your height. The taller you are, the better your score.

•   Is this a truthful or accurate measure of research methods knowledge?

•      Validity refers to the truthfulness of a measure.

 

•      A valid measure assesses what it is intended to measure.

 

–  Construct Validity: Does an instrument measure the theoretical construct (concept) it was designed to measure?

•      Establishing the construct validity of a measure depends on

–  convergent validity and

–  discriminant validity.

•      Convergent validity refers to the extent to which two measures of the same construct are correlated (go together).

•      Discriminant validity refers to the extent to which two measures of different constructs are not correlated (do not go together).

Constructing a Questionnaire

•      The best choice for selecting a questionnaire is to use one that already has been established as reliable and valid.

•      If a suitable measure cannot be found, researchers choose to create their own questionnaire.

•      It may seem easy, but a lot goes into developing a reliable and valid questionnaire.

•      Important steps for preparing a questionnaire:

1. Decide what information should be sought.

2. Decide what type of questionnaire should be used (e.g., will it be self-administered?).

3. Write a first draft of the questionnaire.

4. Reexamine and revise the questionnaire after it is reviewed by experts.

5. Pretest the questionnaire using a sample of respondents under conditions similar to the planned administration of the survey.

6. Edit the questionnaire, and specify the procedures for its use.

•      Next steps include establishing reliability and validity of the questionnaire.

Guidelines for Writing Survey Questions

•     Write clear and specific questions:

•      Avoid double-barreled questions (e.g., “Do you support capital punishment and abortion?”).

•      Place any conditional phrases at the beginning of the question (e.g., “If you were forced to leave your current city, where would you live?”

•      Avoid leading questions (e.g., “Most people favor gun control; what do you think?”).

•      Avoid loaded (emotion-laden) questions (e.g., “People who discriminate are racist pigs: T or F”).

Thinking Critically About Survey Research

•      Correspondence Between Reported and Actual Behavior

–  People’s responses on surveys may not be truthful.

•   Reactivity: People sometimes don’t report truthful responses, because they know the information is being recorded.

•   Social Desirability occurs when people respond to surveys as they think they “should,” rather than how they actually feel or believe.

 

–  Generally, researchers accept people’s responses as truthful, unless there’s reason to suspect otherwise

•   For example, responses aren’t consistent or visual pattern of responses forms a picture.

–  Because behavior doesn’t always match verbal reports of behavior, the multimethod approach to answering questions in psychology is best.

•      Correlation and Causality

–  “Correlation does not imply causation.”

–  Example: Correlation between being socially active (outgoing) and life satisfaction

–  Three possible causal relationships:

•   A causes B (being outgoing causes people to be more satisfied with their life)

•   B causes A (being more satisfied with life causes people to be more outgoing)

•   C causes A and B

Some third variable may be responsible for the relationship between social activity and life satisfaction.

For example, having more friends (a third variable) may cause people to be more outgoing and to be more satisfied with their life.

•   A correlation that can be explained by a third variable is called a “spurious relationship.”