QUANTITATIVE METHODS IN ASSESSMENT

(Sources: Schuh and Upcraft, “Using Quantitative Methods” in Schuh and Upcraft, Assessment in Student Affairs; and Sommer, B. and Sommer, R. A., “Descriptive Statistics” and Inferential Statistics” in Sommer and Sommer, A Practical Guide To Behavioral Research)

I.                   Definition: Quantitative methods involve assigning numbers to objects, events, or observations according to some rule

II.                Step One: Define the Problem

A.    Example of a theoretical problem: determine why recruiters have mixed satisfaction with Placement Services

III. Step Two: Determine the Purposes of the Study

A. Determine how recruiters rated various placement services

B. Determine if recruiter responses varied by gender, ethnicity, size of company, and number of previous visits to the placement service.

IV.             Step Three: Determine the Appropriate Assessment Approach

A.      Factors: purpose of the study (find out which group of recruiters were not as happy with the placement services

1.      size of the population (2,500 recruiters)

2.      time limitations (needed results fairly quickly to correct potential problems)

3.       resource limitations (not a large staff to conduct individual interviews with a large number of recruiters)

B.      Decision: quantitative assessment, using a representative sample

V.                Step Four: Define the Population

A.      Use of sampling techniques

1.      simple random sampling

a.       use of table of random numbers

2.      systematic sampling

a.       everyone in the population is placed on a list and every fourth person is chosen

b.      people are not chosen independently of each other as in random sampling

3.      stratified sampling

a.       selection ensures that certain subgroups will be represented in the sample

b.      helpful when subgroups are important to be studied

4.      cluster sampling

a.       pick groups, rather than individuals for the sample

b.      in this example, draw a random sample of companies and then include all recruiters from that company in the sample

B.      Decision: Stratified random sampling chosen

1.      need to study subgroups

C.      Sample size: related to the type of statistical analysis to be done

1.      Consider sample error: possible difference between the sample results and the results if studied the entire population

Random Sample Size                       Sample Error

196                                                                                            7 percent

294                                                                                            6 percent

384                                                                                            5 percent

600                                                                                            4 percent

1,067                                           3 percent

2,401                                           2 percent

9,604                                           1 percent

2.      in this example, if 35% of a sample of 1,100 said the service was outstanding, with a sample error of 3%, we could be 95% sure that between 32% & and 38% of all respondents thought the service was outstanding.

3.      Subgroup statistical  analysis: need at least 30 in each of 12 subgroups (subgroups are based on gender, size of company, and ethnicity)

4.      Want a sample error no larger than 5%

5.      Estimated return rate: 40%

6.      Sample size of 1000 was chosen to get a 5% sample error rate, with at least 30 in 12 subgroups for statistical analysis, with an anticipated return of 400 with a 40% return rate

VI.             Step Five: Determine the Instrument To Be Used

A.      Instrument available or self-constructed?

B.      Self-constructed instruments

1.      Types of Measurement Scales (See Conducting and Administering Surveys)

a.       categorical scales: nominal scale categorizes objects (gender, ethnicity, type of institution)

b.      ordinal scale: rank-orders according to how much of a variable they possess- finish of a horse race

c.       continuous scales: interval scales (i.e. Likert-type scale with rating along a continuum from strongly agree to strongly disagree

d.      ratio scales: can order variables, plus have a meaningful zero

2.      Types of scales chosen for this hypothetical study

a.       Demographic information: Categorical scales (nominal scales- gender, ethnicity, and company size)

b.      Ratings of quality of placement services: continuous scales (interval scales- a Likert-type scale)

c.       Number of times recruiter had visited previously: ratio scale

VII.          Step Six: Determine Types of Statistical Analyses: Descriptive and Inferential

A.      Two types of statistics

1.      Descriptive statistics

a.       measures of central tendency

(1) mean: average of scores

(2) median: middle score in the rank order of scores

(3) mode: most frequently occurring score

b.      measures of spread of dispersion:

(1 ) range: highest to lowest numbers

(2) standard deviation: sum of scores’ dispersal  around the  mean

c.       measures of relative position

(1) percentile rank

d.      measures of relationships

(1)      correlations: values from -1.0 to +1.0.

(2)      correlation does not imply cause and effect

(3)      Use of the coefficient of determination (square of the correlation coefficient)

(a)    if the correlation  coefficient between grades and SAT scores is 50%, then 25% (.50 squared) of the variability in SAT scores and college GPA can be predicted or explained by aptitude or achievement

(b)   with a correlation coefficient of .30 (.09 squared), 91% of the variability of GPA is unrelated to academic aptitude

(c)    in general, correlations under .30 indicate little relationships

Correlation Coefficient          Coefficient  Determination      Strength of Relationship

.00 to .20                                   .00 to .04                                  Negligible

.21 to .40                                    .05 to .16                                  Low

.41 to .60                                    .17 to .36                                  Moderate

.61 to .80                                    .37 to .64                                  Substantial

.81 to 1.0                                    .65 to 1.0                                  High to very high

e.       showing results

What You Want to Do            Kind of Data                  Use this Method

Examine responses

Of one group to one question        Interval, ratio,             Frequency distribution,

Or ordinal                   bar graph/line graph

Nominal                      Frequency distribution,

Bar graph

Compare the responses

Of two groups to one

Question                                        Any kind                       Paired frequency distributions

Or graphs

Compare the responses of

One group to two questions         Interval or ratio              Scattergram

Ordinal or nominal         Cross-tabulation table

2.      Inferential Statistics: compare the results with chance expectations

a.       introduction to inferential statistics

(1)      you would like to generalize from the sample to the population

(2)      you would like to generalize from the situation studied to situations not studied

(3)      you would like to know if your results are a chance finding or are they significant because of your program

(4)      If you conducted a similar program again, would you get similar results?

(5)      Answers to these questions can be found by using inferential statistics and tests of significance

(6)      Two types of inferential statistics

(a)    test relationship between two groups (bivariate analysis): chi-squares, t-tests, and one-way analysis of variance

(b)   test relationship between multiple groups (multivariate analysis): analysis of variance (ANOVA)

(7)      Statistical significance

(a)    level of statistical significance: probability that the difference between the means of two groups is due to chance rather than a “real” difference between the two groups

(b)   Levels of significance are commonly set at the .05 level or higher

(c)    The .05 level of significance means that one would find a difference between two means under investigation only 5 times in a hundred

(d)   This is expressed as “p>.05”

(e)    “P>.001” means that the difference between the two means would occur only once in a thousand times

(f)    Statistical significance doesn’t speak to the strength of the difference between the two means, but to the possibility of the difference occurring by chance

(g)   This difference is inferred to be “real” and due to your program because of there being so little probability the difference occurred by chance

(h)   Statistical significance is influenced by the size of the sample

(i)     a sample size of 2,000 will get a statistically significant difference at the p>.05 level with a very small statistical difference

(ii) in that type of situation, it is better to increase the level of significance to at least .01.

Using Descriptive Statistics to Describe Results

What is Described?                  What Kind of Data?                  Use this Statistic

The average response

To a question                            Interval or ratio                          Mean or median

Ordinal                                        Semi-interquartile range

Nominal                                      Mode

Of responses to a                     Interval or Ratio                          Standard Deviation

Single question

Ordinal                                         Semi-interquartile range

Nominal                                       Proportion falling outside

Mode

The degree of relationship

Between responses to

Two questions                         Interval or Ratio                           Pearson’s Product-Moment

Correlation Coefficient

Ordinal                                         Spearman’s rank-order

Correlation Coefficient

Nominal                                        Cramer’s index of contingency

The degree of relationship

Among responses to three

Or more questions                  Interval or ratio                             Multiple correlation or

Partial correlation

Ordinal                                           Kendall’s partial rank

Correlation

Using Statistical Analysis to Test Differences Among Groups

What kind of data?                   How many subgroups?          Use this analysis

Interval or ratio                          Two                            t-test for two independent means

Three or more            One-way analysis of variance

Ordinal                                       Two                            Mann-Whitney U-test

Three or more            Kruskal-Wallis one-way

Analysis of variance

Nominal                                      Two                            t-test for two proportions

Two of more               Chi-square test of association

Using Statistical Analyses To Test Differences in Response

What kind of data?                How many responses to compare?     Use this analysis

Interval or ratio                          Two                                          t-tests for matched pairs

Three or more                           One-way analysis of variance

Ordinal                                       Two                                           Sign test

Three of more                            Friedman two-way

Analysis of variance

Nominal                                     Two                                            McNemar test for

Significance of change

Three or more                             Cochran Q-test

VIII.       Step Seven: Data Collection Plan

A. In the hypothetical case of placement services, a 1000 surveys were mailed out to recruiters using a stratified sample

IX.             Step Eight: Record The Data In Usable Form

A.      Machine Readable Instrument Forms Are Very Useful With Larger Groups

B.      These Forms Were Used in the Recruiter Study

X.                Step Nine: Conduct the Appropriate Analyses: Statistical Packages for the Social Services (SPSS) is a commonly used computer software package

A.      Results from the Recruiter Survey

1.      Using the chi-square test of goodness of fit, the sample was representative of the population.

2.      There were at least 35 usable responses in each of the subgroups

3.      Using the Kruskal-Wallis one-one analysis of variance to analyze the rank ordering of reasons for choosing the institution to interview, no differences were found among the subgroups

4.      Using the Pearson product-moment correlation to analyze the ratio scale of number of previous visits to the institution, no differences were found.

5.      Using the mean responses to each of the services rated, the highest and lowest rated service areas were found

6.      Using one-way analysis of variance, differences were found by gender and size of company.

7.      Using a one-way analysis of variance, no differences in ratings of services were found in the number of previous visits to the institution.

XI.             Step Eleven: Evaluation, or the Meaning of Analyses for Policy and Practice

A.      Use of the Results to Increase Satisfaction with Services

XII.          Step Twelve: Strategy For Use of Results

A.      Follow up with focus groups and interviews

B.      Implement Changes In Services

C.      Re-do the study in two years

XIII.       Common Statistical Notations

A.      <  less than

B.      >  greater than

C.      N  Number of scores

D.      df  degrees of freedom

E.      p    probability that the results of a statistical test are due to chance

F.      ns   Not significant

G.      M   Mean (Arithmetic average)

H.      X     Individual score

I.      Y     Individual score (used in a second set of scores)

J.      SD    Standard deviation, a descriptive statistic that indicates variability or dispersion from the mean

K.      SS    Sum of squares, refers to the sum of squared deviations from the mean

L.      MS  Mean square: SS divided by df.

M.       T    t ratio, an inferential statistic used for contrasting two means

N.      ANOVA   Analysis of Variance

O.      F   F ratio, an inferential statistic for testing significant differences among  two or more means. Calculated by ANOVA

P.      X Squared:  Chi-Square, an inferential statistic used for analyzing categorical scores

Q.      r: Pearson product-moment coefficient; a measure of correlation for continuous data