During Faculty Week at Plymouth State University in late summer, Gloria Rogers, Associate Executive Director of ABET, Inc., spent a day working with faculty and staff on program assessment. Her presentation was entitled, “Promise and Pitfalls of Assessing Student Learning”. I thought of our Pakistani Educational Leadership Institute participants hosted this summer, 2007, who focused on preparing evaluation plans to assess the impact of past institutes. I wanted to share some of what I thought you might find useful or at least what I plan to incorporate into my educational work. Paraphrasing from one of the tables she showed us, for any program, one has to decide, in the following order:
- what are your objectives (broad statements that describe the program goals;
- what are your outcomes (statements that describe what participants should be able to do as a result of participating in the program;
- what are the specific performance(s) required to meet the above stated outcomes that are confirmable through evidence;
- what are the processes that identify, collect, use and prepare data that can be used to evaluate achievement; and lastly,
- how is the information obtained from the processes in the step above interpreted, valued, and used to take action.
While these steps don’t seem novel, I was glad to be reminded in a straight forward manner of what’s important in assessing or evaluating any kind of program- whether it be the student’s work in my classes or the professional development I offer to international educational visitors at PSU- like our Pakistani guests. What was really nice is that Gloria emphasized that more data isn’t necessarily better, and in fact, she has a rule of thumb that one shouldn’t specify more than four performance criteria for a particular outcome. So, for example, in her model, if the desired educational objective is, “work effectively with others,” then, the outcome could be, “ability to function on multidisciplinary team,” and the performance criteria could be, “1) researches and gathers information; 2) fulfills duties of team roles, 3) shares work equally, and 4) listens to other teammates.” As a science educator, I might add that Gloria had particular credibility with me because she worked at a small, higher education institution that primarily focused on graduating competent engineering students, before becoming a consultant on program evaluation. It was interesting that the examples of objectives, outcomes, and performance critieria she used for students at her former institution were so generic and transferable- valuable skills for most anyone in the workforce today.
Gloria also shared tidbits of wisdom, what she called “things I wish I had known” about program evaluation and her list included, “Capitalize on what you are already doing; You don’t have to measure everything all the time; Take advantage of local resources; and, Don’t wait for perfection.” She had many other gems of advice (http://www.abet.org/).
Gloria provided a list of assessment methods: “written surveys and questionnaires, exit and other interviews, standardized exams, locally developed exams, archival records, focus groups, portfolios, simulations, performance appraisal, external examiner, oral exams, and behavioral observations.” She also emphasized that it is crucial to use a multi-method/multi-source approach to maximize validity and reduce bias of any one approach. I’m finally getting around to sharing this information with you, which I’ve been meaning to do for several weeks, because the external evaluator for this summer’s Pakistani Institute, sent me an email asking for more information and for a time to set up a phone interview. So, I’m pleased to say, PSU’s Pakistani Institute is following Gloria’s recommendation of using multiple assessment methods.
I’m still reflecting on my work with you this summer, asking what could I have done better and recognizing that providing a syllabus at the onset which clearly articulated my goals, objectives, outcomes, and performance criteria would have been best. I knew about these best practices, but now this succinct chart/list is prominently displayed on the bulletin board in my office to serve as a beacon and framework for all my future curricular work with students and international invited guests.
I hope you all are doing well!
Best,
Mary Ann
