OUTCOMES ASSESSMENT: LEADERSHIP DEVELOPMENT
(Schuh and Upcraft,
“Assessing Learning Outcomes”, in Schuh and Upcraft, Assessment Practice in
Student Affairs)
I.
Introduction
A. This simulated case study demonstrates both
a quantitative outcomes assessment process and a qualitative outcomes
assessment process
B. The outcomes assessment example focuses on
a theoretical two credit leadership course taught by the student activities
staff and business department faculty
C. First a quantitative outcomes approach will
be presented and then a qualitative approach
D. The approach will follow Terenzini and
Upcraft’s 12 step outcomes assessment model (1996)
II.
A
Quantitative Outcome Model for Assessing Leadership Development
A. Step One: Define the Problem
1. Provide credible empirical evidence that the Leadership Development Course is making a difference
2. This information is to be presented to the Director of Student Activities and the Chair of the Business Department
B. Step Two: Determine the Purpose of the
Study (leadership improvement due to the course)
C. Step Three: Determine the Appropriate
Assessment Approach
1. Use of Astin’s Input-Environment-Outcome (I-E-O model): see Terenzini and Upcraft “Assessing Program and Service Outcomes”, in Schuh and Upcraft’s Assessment in Student Affairs.
2. Show impact of an environmental intervention (leadership development) on a selected outcome (leadership development skills); take into account initial differences and other environmental experiences
D. Step Four: Determine the Outcomes and Objectives
(Based on leadership skills by Kouzes and Posner, (1990), “The Leadership
Practices Inventory”, San Francisco: Pfeiffer) on which the course was based)
1. Outcomes
a. Challenging the process
b. Inspiring a shared vision
c. Enabling others to act
d. Modeling the way
e. Encouraging the heart
2. Objectives
a. The Leadership Development course participants will demonstrate skills on leadership development within the concept of Kouzes and Posner’s “challenging the process” by performing at a statistically significant higher level than the control group on the Leadership Practices Inventory pre- and post-tests, which will be taken at the beginning and end of the Leadership Development course.
b. The Leadership Development course participants will demonstrate skills on leadership development within the concept of Kouzes and Posner’s “inspiring a shared vision” by performing at a statistically significant higher level than the control group on the Leadership Practices Inventory pre- and post-tests, which will be taken at the beginning and end of the Leadership Development course.
c. The Leadership Development course participants will demonstrate skills on leadership development within the concept of Kouzes and Posner’s “enabling others to act” by performing at a statistically significant higher level than the control group on the Leadership Practices Inventory pre- and post-tests, which will be taken at the beginning and end of the Leadership Development course.
d. The Leadership Development course participants will demonstrate skills on leadership development within the concept of Kouzes and Posner’s “modeling the way” by performing at a statistically significant higher level than the control group on the Leadership Practices Inventory pre- and post-tests, which will be taken at the beginning and end of the Leadership Development course.
e. The Leadership Development course participants will demonstrate skills on leadership development within the concept of Kouzes and Posner’s “encouraging the heart” by performing at a statistically significant higher level than the control group on the Leadership Practices Inventory pre- and post-tests, which will be taken at the beginning and end of the Leadership Development course.
E. Step Five: Identify the Input or Control
Variables
1. race
2. gender
3. class standing
4. pre-course leadership skills measure
F. Step Six: Identify Some Environmental
Variables
1. past and current involvement in student leadership positions
2. place of residence
3. number of hours worked per week
4. grade point average
G. Step Seven: Select the Measurement
Instruments
1. Leadership Practices Inventory, by Kouzes and Posner
H. Step Eight: Determine the Population to be
Studied and the Sample to Be Drawn
1. Experimental Group: all students in the Leadership Development class (300)
2. Control Group: a similar size group (300) of undergraduates matched to the control group on gender, age, and class standing
3. The population is all undergraduate students
I. Step Nine: Determine the Modes of
Statistical Analyses
1. Based on Astin’s Input-Environment-Outcomes (I-E-O model), a multivariate analysis was chosen
J. Step Ten: Develop and Implement a Plan for
Data Collection
1. Data on background characteristics, the Leadership Practices Inventory, and prior leadership experiences were collected in the first class period
2. Data from the control group was collected in a mailed survey
3. The Leadership Practices Inventory was given to both groups at the end of the semester
4. Data was collected on the environmental variables
K. Step Eleven: Record the Data in Usable Form
1. Data were recorded in an electronic database for analysis
L. Step Twelve: Conduct the Appropriate
Analyses
1. Data was analyzed using the SPSS statistical package to run multiple regression analyses.
2. The results were examined on the five dimensions of leadership development
III.
Steps
in the Qualitative Outcomes Assessment Program
A. Step One: Define the Problem
1. Systematic evidence that this leadership development course has resulted in improved leadership skills
2. Same as in quantitative assessment
B. Step Two: Determine the Purpose of the Study
1. Did students improve their leadership skills?
2. Did the course help the development of leadership skills?
3. How could the course be improved?
4. Did the students recommend making the course permanent?
5. The last two statements are not strictly speaking outcomes, but were goals for the assessment
C. Step Three: Determine the Appropriate
Assessment Approach
D. Step Four: Determine the Outcomes
1. Outcomes were similar, but gained insight on the outcomes from the quantitative assessment.
E. Step Five: Select the Assessment Instrument
1. An individual interview protocol was selected, since students were asked to reflect on their experiences in the class, based on course content and activities
2. This would gain greater detail
F. Step Six: Determine the Population to Be
Studied and the Sample To Be Drawn
1. The sample of 30 students out of 300 was drawn using a stratified, purposeful sampling technique based on grades students received of above average, average, and below average grades
2. Ten students were drawn from each of those groups who earned an A, a B, and a C in the course
G. Step Seven: Determine How the Information Will Be Codified
H. Step Eight: Conduct the Appropriate
Analyses
1. There is not a set way of analyzing the data, similar to quantitative data
2. The researchers codified the most meaningful data
IV.
Evaluation:
Two Additional Steps
A. Step One: Evaluate the Analyses for Policy
and Practice Implications
B. Step Two: Report the Results Effectively