QUANTITATIVE METHODS IN ASSESSMENT
(Sources: Schuh and Upcraft, “Using Quantitative Methods” in Schuh and Upcraft, Assessment in Student Affairs; and Sommer, B. and Sommer, R. A., “Descriptive Statistics” and Inferential Statistics” in Sommer and Sommer, A Practical Guide To Behavioral Research)
I.
Definition:
Quantitative methods involve assigning numbers to objects, events, or observations
according to some rule
II.
Step
One: Define the Problem
A. Example of a theoretical problem: determine
why recruiters have mixed satisfaction with Placement Services
III. Step Two: Determine the Purposes of the Study
A. Determine how recruiters
rated various placement services
B. Determine
if recruiter responses varied by gender, ethnicity, size of company, and number
of previous visits to the placement service.
IV.
Step
Three: Determine the Appropriate Assessment Approach
A.
Factors:
purpose of the study (find out which group of recruiters were not as happy with
the placement services
1. size of the population (2,500 recruiters)
2. time limitations (needed results fairly quickly to correct potential problems)
3. resource limitations (not a large staff to conduct individual interviews with a large number of recruiters)
B.
Decision:
quantitative assessment, using a representative sample
V.
Step
Four: Define the Population
A.
Use
of sampling techniques
1. simple random sampling
a. use of table of random numbers
2. systematic sampling
a. everyone in the population is placed on a list and every fourth person is chosen
b. people are not chosen independently of each other as in random sampling
3. stratified sampling
a. selection ensures that certain subgroups will be represented in the sample
b. helpful when subgroups are important to be studied
4. cluster sampling
a. pick groups, rather than individuals for the sample
b. in this example, draw a random sample of companies and then include all recruiters from that company in the sample
B.
Decision:
Stratified random sampling chosen
1. need to study subgroups
C.
Sample
size: related to the type of statistical analysis to be done
1. Consider sample error: possible difference between the sample results and the results if studied the entire population
Random
Sample Size Sample
Error
196 7 percent
294 6 percent
384 5 percent
600 4 percent
1,067 3 percent
2,401 2 percent
9,604 1 percent
2. in this example, if 35% of a sample of 1,100 said the service was outstanding, with a sample error of 3%, we could be 95% sure that between 32% & and 38% of all respondents thought the service was outstanding.
3. Subgroup statistical analysis: need at least 30 in each of 12 subgroups (subgroups are based on gender, size of company, and ethnicity)
4. Want a sample error no larger than 5%
5. Estimated return rate: 40%
6. Sample size of 1000 was chosen to get a 5% sample error rate, with at least 30 in 12 subgroups for statistical analysis, with an anticipated return of 400 with a 40% return rate
VI.
Step
Five: Determine the Instrument To Be Used
A.
Instrument
available or self-constructed?
B.
Self-constructed
instruments
1. Types of Measurement Scales (See Conducting and Administering Surveys)
a. categorical scales: nominal scale categorizes objects (gender, ethnicity, type of institution)
b. ordinal scale: rank-orders according to how much of a variable they possess- finish of a horse race
c. continuous scales: interval scales (i.e. Likert-type scale with rating along a continuum from strongly agree to strongly disagree
d. ratio scales: can order variables, plus have a meaningful zero
2. Types of scales chosen for this hypothetical study
a. Demographic information: Categorical scales (nominal scales- gender, ethnicity, and company size)
b. Ratings of quality of placement services: continuous scales (interval scales- a Likert-type scale)
c. Number of times recruiter had visited previously: ratio scale
VII.
Step
Six: Determine Types of Statistical Analyses: Descriptive and Inferential
A.
Two
types of statistics
1. Descriptive statistics
a. measures of central tendency
(1) mean: average of scores
(2) median: middle score in the rank order of scores
(3) mode: most frequently occurring score
b. measures of spread of dispersion:
(1 ) range: highest to lowest numbers
(2) standard deviation: sum of scores’ dispersal around the mean
c. measures of relative position
(1) percentile rank
d. measures of relationships
(1) correlations: values from -1.0 to +1.0.
(2) correlation does not imply cause and effect
(3) Use of the coefficient of determination (square of the correlation coefficient)
(a) if the correlation coefficient between grades and SAT scores is 50%, then 25% (.50 squared) of the variability in SAT scores and college GPA can be predicted or explained by aptitude or achievement
(b) with a correlation coefficient of .30 (.09 squared), 91% of the variability of GPA is unrelated to academic aptitude
(c) in general, correlations under .30 indicate little relationships
Correlation Coefficient Coefficient Determination Strength of Relationship
.00 to .20 .00 to .04 Negligible
.21 to .40 .05 to .16 Low
.41 to .60 .17 to .36 Moderate
.61 to .80 .37 to .64 Substantial
.81 to 1.0 .65 to 1.0 High to very high
e. showing results
What You Want to
Do Kind of Data Use this Method
Examine responses
Of one group to one question Interval, ratio, Frequency distribution,
Or ordinal bar graph/line graph
Nominal Frequency distribution,
Bar graph
Compare the responses
Of two groups to one
Question Any kind Paired frequency distributions
Or graphs
Compare the responses
of
One group to two questions Interval or ratio Scattergram
Ordinal or nominal Cross-tabulation table
2. Inferential Statistics: compare the results with chance expectations
a. introduction to inferential statistics
(1) you would like to generalize from the sample to the population
(2) you would like to generalize from the situation studied to situations not studied
(3) you would like to know if your results are a chance finding or are they significant because of your program
(4) If you conducted a similar program again, would you get similar results?
(5) Answers to these questions can be found by using inferential statistics and tests of significance
(6) Two types of inferential statistics
(a) test relationship between two groups (bivariate analysis): chi-squares, t-tests, and one-way analysis of variance
(b) test relationship between multiple groups (multivariate analysis): analysis of variance (ANOVA)
(7) Statistical significance
(a) level of statistical significance: probability that the difference between the means of two groups is due to chance rather than a “real” difference between the two groups
(b) Levels of significance are commonly set at the .05 level or higher
(c) The .05 level of significance means that one would find a difference between two means under investigation only 5 times in a hundred
(d) This is expressed as “p>.05”
(e) “P>.001” means that the difference between the two means would occur only once in a thousand times
(f) Statistical significance doesn’t speak to the strength of the difference between the two means, but to the possibility of the difference occurring by chance
(g) This difference is inferred to be “real” and due to your program because of there being so little probability the difference occurred by chance
(h) Statistical significance is influenced by the size of the sample
(i) a sample size of 2,000 will get a statistically significant difference at the p>.05 level with a very small statistical difference
(ii) in that type of situation, it is better to increase the level of significance to at least .01.
Using Descriptive Statistics
to Describe Results
What is Described?
What Kind of Data?
Use this Statistic
The average response
To a question Interval or ratio Mean or median
Ordinal Semi-interquartile range
Nominal Mode
The spread or
variability
Of responses to a Interval or Ratio Standard Deviation
Single question
Ordinal Semi-interquartile range
Nominal Proportion falling outside
Mode
The degree of
relationship
Between responses to
Two questions Interval or Ratio Pearson’s Product-Moment
Correlation Coefficient
Ordinal Spearman’s rank-order
Correlation Coefficient
Nominal Cramer’s index of contingency
The degree of
relationship
Among responses to
three
Or more questions Interval or ratio Multiple correlation or
Partial correlation
Ordinal
Correlation
Using Statistical Analysis to Test
Differences Among Groups
What kind of
data? How many
subgroups? Use this analysis
Interval or ratio Two t-test for two independent means
Three or more One-way analysis of variance
Ordinal Two Mann-Whitney U-test
Three or more Kruskal-Wallis one-way
Analysis of variance
Nominal Two t-test for two proportions
Two of more Chi-square test of association
Using Statistical
Analyses To Test Differences in Response
What kind of data? How many responses to compare? Use this analysis
Interval or ratio Two t-tests for matched pairs
Three or more One-way analysis of variance
Ordinal Two Sign test
Three of more Friedman two-way
Analysis of variance
Nominal Two McNemar test for
Significance of change
Three or more Cochran Q-test
VIII. Step Seven: Data Collection Plan
A. In the hypothetical case of placement services, a 1000 surveys were
mailed out to recruiters using a stratified sample
IX.
Step
Eight: Record The Data In Usable Form
A.
Machine
Readable Instrument Forms Are Very Useful With Larger Groups
B.
These
Forms Were Used in the Recruiter Study
X.
Step
Nine: Conduct the Appropriate Analyses: Statistical Packages for the Social
Services (SPSS) is a commonly used computer software package
A.
Results
from the Recruiter Survey
1. Using the chi-square test of goodness of fit, the sample was representative of the population.
2. There were at least 35 usable responses in each of the subgroups
3. Using the Kruskal-Wallis one-one analysis of variance to analyze the rank ordering of reasons for choosing the institution to interview, no differences were found among the subgroups
4. Using the Pearson product-moment correlation to analyze the ratio scale of number of previous visits to the institution, no differences were found.
5. Using the mean responses to each of the services rated, the highest and lowest rated service areas were found
6. Using one-way analysis of variance, differences were found by gender and size of company.
7. Using a one-way analysis of variance, no differences in ratings of services were found in the number of previous visits to the institution.
XI.
Step
Eleven: Evaluation, or the Meaning of Analyses for Policy and Practice
A.
Use
of the Results to Increase Satisfaction
with Services
XII.
Step
Twelve: Strategy For Use of Results
A.
Follow
up with focus groups and interviews
B.
Implement
Changes In Services
C.
Re-do
the study in two years
XIII. Common Statistical Notations
A.
< less than
B.
> greater than
C.
N Number of scores
D.
df degrees of freedom
E.
p probability
that the results of a statistical test are due to chance
F.
ns Not significant
G.
M Mean (Arithmetic average)
H.
X Individual score
I.
Y Individual
score (used in a second set of scores)
J.
SD Standard deviation, a descriptive statistic
that indicates variability or dispersion from the mean
K.
SS Sum of squares, refers to the sum of
squared deviations from the mean
L.
MS Mean square: SS divided by df.
M.
T t ratio, an inferential statistic used for contrasting two
means
N.
ANOVA Analysis of Variance
O.
F F ratio, an inferential statistic for testing significant
differences among two or more means.
Calculated by ANOVA
P.
X
Squared: Chi-Square, an inferential
statistic used for analyzing categorical scores
Q.
r: Pearson
product-moment coefficient; a measure of correlation for continuous data