Feature Article 1

Assessment of Student Learning: Direct and Indirect Methods that Work

By Charles W. McGlothin, Jr., Ph.D, P.E.

Author Biography

Charles W. McGlothlin, Jr., Ph.D., P.E.
Program Director & Assistant Professor
Occupational Safety & Health
School of Health Sciences
Oakland University
Rochester, MI

Charles W. McGlothlin, Jr. is Assistant Professor and Program Director of the Occupational Safety and Health Program at Oakland University in Rochester, Michigan. He has work as an engineer, operations manager, and safety educator for over 40 years. Professor McGlothlin received BS and MS degrees in Mining Engineering from West Virginia University and his Ph.D. in Education and Human Resource Studies at Colorado State University. He has served as the corporate officer responsible for operations and environmental, health, and safety compliance at mining operations for Bethlehem Steel in Pennsylvania, Atlantic Richfield and Kaiser Coal corporations in Utah, and KN Energy in Colorado. In addition, he has worked as a Registered Professional Engineer in Pennsylvania, Utah, and Colorado and continues to work as a safety and health consultant domestically and internationally.

Dr. McGlothlin is a professional member of the American Society of Safety Engineers and a member of the National Safety Management Society, the Midwest Cooperative Education and Internship Association, and the National Safety Council. He serves on the ASSE Education Standards Committee and the Executive Committee of the ABET Applied Science Accreditation Commission. In 2006 Dr. McGlothlin was recognized as the ASSE Educator of the Year.

Abstract

The purpose of this presentation is to explain the principles of best practice for assessing student learning and describe methods to implement an effective assessment program. Assessment is not an end in itself but a means for continuous training and educational improvement. As such an effective assessment program becomes the basis for continuous improvement in a company’s training program or in a university’s educational efforts. The assessment program can provide a means to enhance employees and student involvement and ownership in the training and educational process. This stakeholder involvement is required of ABET accredited universities offering safety-related degree programs and is recommended for any company or college that is striving to improve their training or educational efforts.

The following nine principles of good practice for assessing student learning will be discussed:

  1. Assessment of student learning begins with training/educational values.
  2. Assessment is most effective when it reflects understanding of learning as multidimensional, integrated, and revealed in performance over time.
  3. Assessment works best when the program it seeks to improve have clear, explicitly stated purposes.
  4. Assessment requires attention to outcomes but also and equally to the experiences that lead to those outcomes.
  5. Assessment works best when it is ongoing not episodic.
  6. Assessment fosters wider improvement when representatives for across the full range of stakeholders are involved.
  7. Assessment makes a difference when it begins with issues of use and illuminates questions that people really care about.
  8. Assessment is most likely to lead to improvement when it is part off a larger set of conditions that promote change.
  9. Through assessment, trainers and educators meet responsibilities to students and to the public.

Trainee/student learning outcomes, which are the subject of assessment, are usually divided into categories of knowledge, skills, values or attitudes, or job/career success of trainee or graduate. In each of these outcomes categories, measures could focus on either the improvement in students or students achieving certain standards/objectives of knowledge, skills, values, or attitudes. Specific learning objectives can be developed that reflect what is expected of the trainee or college graduate relative to the training program desired outcomes or stakeholder expectations of the college graduate. These outcomes are assessed through either direct or indirect measures of performance.

At least one direct measure of student outcomes is required in an effective assessment plan. Direct measures of student learning outcomes include:

  1. Locally-developed test
  2. Licensure or nationally-normed subject content exams
  3. Graded assignments
  4. Portfolios with contents assessed by multiple raters
  5. Research papers of projects assessed by multiple raters
  6. Writing samples
  7. Capstone courses with project or paper assessed by multiple raters
  8. Simulations/Demonstrations/Role-playing
  9. Performance in supervised internships
  10. Performance or presentations
  11. Oral exam
  12. Pre/Post-test

Indirect measures of student learning outcomes will provide additional information as to how the training or educational program is meeting curricular and learning objectives. Indirect measures address student, alumni, and employer perceptions of the effectiveness of the training or education. These measures include:

  1. Graduate and alumni surveys
  2. Employer surveys
  3. Focus Groups of students, advisory boards, or other stakeholders
  4. Exit interviews or surveys
  5. Reports from students after internship experience
  6. Graduate school placement

Finally each of the selected direct and indirect measures should build upon the other. There should be a logical flow from training program or educational goals to student learning objectives to the direct and indirect measures used for assessment of student learning. Once the assessment process is complete and information has been gathered and analyzed the key question becomes what now and how will the information be used. The continuous improvement program is that follows from an effective assessment program is the key to success and becomes part of a continuous improvement process. Stakeholder involvement is essential for continuous improvement and key elements of this process will be addressed in the presentation.

Introduction

The purpose of this presentation is to explain the generally accepted principles of good practice for assessing student learning and describe methods to implement an effective assessment program. However, assessment is not an end in itself but more importantly a means for continuous educational improvement. As such, an effective assessment program becomes the basis for continuous improvement in our higher educational efforts. In addition, an effective assessment program can provide a means to enhance stakeholder involvement and ownership in our educational process. Although this stakeholder or constituency involvement is required in universities offering ABET accredited safety-related degree programs, it is highly recommended for any institution of higher education interested in improving their educational efforts.

On the downside, an assessment program requires considerable effort and can be a significant burden on program faculty and administrators without generating any positive results. To generate desired results, the assessment data must be collected, analyzed, evaluated, and used in decision-making as part of a continuous improvement process. Accepted principles of good practice for assessing student learning must be followed and effective assessment methods used for the data to be useful for fact-based program improvements. The following principles of good assessment practice are distilled from work developed under the auspices of the American Association for Higher Education Assessment Forum.

Principles of Good Assessment Practice

The first essential principle of good practice is that effective assessment must be driven by the institution’s educational mission and core objectives to ensure a process for improving what we truly value. More specifically educational outcomes associated with curricular course work must relate directly back to the safety degree program mission and associate educational objectives. Development of educational outcomes and longer range educational objectives requires involvement and insights from employers, industry advisors, alumni, faculty, internal and external program evaluators, and our professional society. The role and responsibly of each group must be defined and incorporated into a comprehensive assessment plan for continuous improvement.

The next principle is based on the fact that learning is multidimensional, integrated, and revealed in improved performance over time of matriculation. Learning involves not only knowledge and ability to apply that knowledge but values and attitudes that affect academic success and job performance beyond the classroom. Assessment should reflect this understanding by employing diverse methods over time of matriculation for a more complete and accurate picture of student learning. Effective assessment works best when it is ongoing, not episodic. Although an isolated, one-shot assessment is better than nothing, improvement over time is the intent of an effective continuous improvement program. Effective assessment planning is essential to gaining insights into student learning and development.

Another principle of good practice recognizes that assessment of student learning is not limited to a small group of academic experts but must be a collaborative activity by all parties with a stake in the educational process; students, faculty, alumni, employers, and industry advisors. To effectively gain assessment insight from this diverse group of stakeholders requires a diverse set of assessment tools and methods. Carrying out assessment activities with this diverse group requires evaluation of various aspect of an educational program from assessment of classroom learning and program outcomes to evaluation of longer range program educational objectives. Development of an effective assessment plan is essential to ensure manageable, sustainable, and effective data collection processes.

Finally it should be recognized that effective assessment is about quality assurance. We as educators have a responsibility to students, parents, and employers that support or depend on us to provide an educational experience that meets their collective needs. Educators can not satisfy this responsibility without close collaboration with recent graduates and employers that hire these graduates. We must involve these constituents in the assessment process. An effective assessment program involving these stakeholders is the basis for the continuous improvement we need to stay abreast of our ever changing global society and associated educational needs.

Data Collection Methods

The question now becomes what methods are we going to use to collect meaningful data that satisfies these principles of good practice for assessing student learning outcomes and longer range educational objectives? Most assessment experts agree that a variety of assessment methods must be used to get adequate feedback to measure student learning and educational program success. There are basically two types of assessment methods, direct and indirect. Direct methods of assessment require the student to produce evidence of learning that a reviewer can assess how well the student meets expectations. Indirect methods of assessment allow the student, or others associated with the student, to reflect on the student’s level of learning and provide the reviewer with their perceptions of learning attainment. At least one direct measure of assessment is considered by many, including ABET, to be essential to measure learning outcomes. Indirect methods are often helpful in interpreting the findings of the direct methods. Following are some of the more common examples of direct and indirect methods of assessment:

Direct Methods of Assessment

Indirect Methods of Assessment

Although all of these direct and indirect methods work, some may work better than others for any given situation. The most effective direct methods this author has used are addressed as follows. Evaluation of course assignments and research papers is essential when attempting to relate student learning to identified educational program relative to identification, evaluation, and control of workplace hazards. Tests are commonly used to determine lower levels of learning at the knowledge, comprehension, and application levels. A comprehensive, program developed test can be designed in conjunction with a senior capstone course or internship to address educational education outcomes. Course-embedded assessments can be used throughout the educational experience to simplify and structure the assessment process somewhat by identifying specific assignments, papers or exam questions that address specific outcomes. Direct methods can also be developed to address oral and written communication skills using criteria or rubrics to ensure consistent evaluations. Rubrics to address oral and written assignments are ready available on several internet sites easily assessable through Google.

ABET Requirements

Internships
An experiential learning component is required in an ABET accredited safety-related program and is required in most other higher education safety-related degree programs. A properly structured internship can very effectively serve as a capstone experience for direct assessment of student development and educational program outcomes. An internship portfolio, including a reflective paper on the student entire college experience as it relates to the intended program outcomes and the internship work experience, can provide the basis for direct assessment of the student’s educational experience. Since documentation of the internship experience is usually required, this direct assessment adds little additional faculty work load. With some specific direction to the intern, the portfolio can be structured to document student learning over the course of the entire educational experience.

Peer Review Process
External peer review is a widely accepted indirect method for assessing both short term educational program outcomes and longer term educational objectives. Both regional and program specific accrediting bodies rely on the peer review process to access student learning as it relates to established outcomes and university/departmental missions. The ABET safety-related program accreditation process has proven effective in identifying program strengths and weakness and contributing to educational program structural changes leading to continuous improvement of the delivered educational product. The ABET program evaluation team is composed of peer educators and industry safety professionals that help ensure the educational outcomes are meeting constituencies’ needs and thereby ensuring a quality educational experience for both student and employer.

Surveys

Student, alumni, and employer surveys are becoming increasingly important in understanding student educational needs and overall effectiveness of the educational experience. Student exit surveys and interviews provide important insights into student perceptions of their various courses, instructional approaches, the learning environment, curricular strengths and weakness, and their overall satisfaction with their college experience. Alumni surveys are useful in determining preparation for professional practice, curricular relevance to job requirements, and achievement of both educational outcomes and longer range educational objectives. These stakeholders in the educational process are particularly important to the educational program’s continuous improvement process. Employer surveys provide critical insights relative to the educational program’s ability to graduate students that meet ever changing employer needs. These surveys assess employer perceptions of graduates’ knowledge and skills as well as programmatic characteristics needed for a successful educational experience. Employer assessment is another critical component in the continuous improvement process. These surveys coupled with regular curricular reviews by faculty and industry advisors ensures courses are revised to meet changing needs in educational outcomes to ensure a quality educational experience.

Conclusion
An assessment process built on generally accepted principles of good practice for assessing student learning that incorporates these direct and indirect methods can yield significant insights into student learning and professional development. Applying knowledge gained in this assessment process should result in the continuous improvement that ensures the quality higher education required to meet the demands of the 21st century safety and health professional.