Knowledge Assessment Instrument – Rationale Report

PIDP 3230 – Evaluation of Learning
Vancouver Community College
Kathryn Truant
June 4, 2017

Knowledge Assessment Instrument – Oral & Maxillofacial Surgery Module for Certified Dental Assistants (CDA’s): Unit One – The Perioperative Period
Rationale Report

What is the purpose of my Oral & Maxillofacial Surgery Module for CDA’s: Unit One – The Perioperative Period knowledge assessment and what claims of validity and reliability can be made?

The purpose of this Knowledge Assessment Instrument (KAI) is a formal assessment of the cognitive knowledge of learners in the theory portion of Unit One – The Perioperative Period, from my DACUM Assignment in PIDP 3210 (Truant, 2017). The course is based on the Competency Based Education (CBE) approach; it introduces new skills, procedures and systems. The items in this KAI are valid – they reflect the lessons taught, and reliable – they reflect the learning outcomes (Vancouver Community College, 2017). The curriculum is based on professional experience, and information found in A Manual of Oral and Maxillofacial Surgery for Nurses (Yates, 2000). The website Quizlet (2017) was effective in helping me construct items consistent to my lessons as well. My KAI also demonstrates reliable items that are consistent to what needs to be assessed; evaluation relies on the most important aspects of the lessons, and more importantly, what a CDA needs to know in a clinical environment.

 What goals and outcomes are going to be assessed by this cognitive assessment exam?

I based my KAI on the first goal of my DACUM from PIDP 3210. It is the theory component of Unit One; it is important for my students to be able to receive cognitive knowledge of the subject prior to application and evaluation in a clinical setting. My students will be able to identify and define the three phases of perioperative care of a patient, and recognize the importance of their role in the process.

Table of Specifications

Course: Oral & Maxillofacial Surgery Module for CDA’s
Unit One: The Perioperative Period Theory Component
Course Value: 5%
Unit Outcomes Cognitive Level Exam
Recall
LOW
1 mark each
Application
MEDIUM
2 marks each
Synthesis
HIGH
3 marks each
# of items % of exam
Describe the Perioperative Period AR 2 MC 1 RR 1 4 14%
Identify the Preoperative Phase AR 2

MI 2

RR 3 7 26%
Identify the Intraoperative Phase AR 2

MI 2

MC 5 RR 1 10 34%
Identify the Postoperative Phase AR 2

MI 4

MC 2 RR 1 9 26%
TOTAL 16 8 6 30 100%
Marks Awarded 16 16 18 / 50

Table 1
SUMMARY:
AR – Alternate Response       8 (16%)
MI –  Matching Items               8 (16%)
MC – Multiple Choice              8 (32%)
RR –  Restricted Response    6 (36%)
TOTAL: 30 items (% that items are worth by importance and value)

How did the above Table of Specifications (Table 1) impact the construction of the exam?

Each individual Unit One theory outcome requires varying emphasis and cognitive knowledge. For example, “describing” the Perioperative Period is important, but more emphasis in instruction is placed on “identifying” the role of the CDA in the three phases of perioperative care. The Table of Specifications was helpful in balancing what the important outcomes of my course goals are. The low and medium cognitive levels contain explicit content; the evidence of learning focuses on knowledge. This document follows Bloom’s Taxonomy through the cognitive levels of knowledge (Forbes, 2015, para. 5,6). I want my students to remember facts, and to understand, apply and analyze what they have been taught. The high cognitive level contains explicit content, and implicit content by determining how well the learner can articulate what they’ve learned. In summary, the table of specifications was instrumental in determining important unit outcomes AND in determining the item types necessary to assess cognitive knowledge.

 How much time is allocated for the exam and why?

To determine time allocation for the KAI, I used the online reading “Time Allocation for Various Item Types” found in the PIDP 3230 course material (Vancouver Community College, 2017). The document is beneficial in choosing how much time I will provide for each item based on the complexity of the item. I looked at the mental process involved with each item to decide whether to allow the maximum time suggested. I allowed additional time for the students to complete the Restricted Response Essay items because they are 3-part items. I feel a 30-minute time limit correlates with a KAI worth 5% of the total mark for the course (refer to Table 2 below), but I will allocate 45 minutes to write the KAI because I want the evaluation to be a positive learning experience, allowing time for my students to reflect on their responses.

Item Type Suggested Time Allocation for Student Responses
Alternate Response 10-15 seconds x 8 items = 80-120 seconds (max = 2 min.)
Matching 10-15 seconds x 8 items = 80-120 seconds (max = 2 min)
Multiple Choice 30-60 seconds x 8 items = 240-480 seconds (max = 8 min)
Restricted Essay 45-120 seconds x 6 items = 270-720 seconds (max = 12 min)

Table 2
Summary:
Alternate Response   2 minutes
Matching                        2 minutes
Multiple Choice           8 minutes
Restricted Essay         18 minutes
TOTAL: 30 minutes (Time allocated = 45 minutes)

How did I determine the value of the exam?

I determined the value of the exam on the theory component of Unit One at 5% of the total course mark (the clinical component of this unit is worth 10%). There are 6 units in the course, and Unit One is the introductory portion. Although the course value of the exam is only 5%, the passing grade for the exam is 70%; I want the exam to provide feedback on what the students know before moving on to the clinical portion of Unit One.

I determined the value of each item of the exam by referring to the “Item Type and Corresponding Cognitive Level” found in the online course readings for PIDP 3230 (Vancouver Community College, 2017). My Table of Specifications (Table 1) places Multiple Choice items at a medium cognitive level because I purposely constructed these items to challenge cognitive ability, and Restricted Response Essay items are higher on the cognitive level of knowledge because they are three-part items. I want my students to remember, understand, apply and analyze the material taught in the theory portion of Unit One in my course so I created challenging items and placed higher value on them to reflect important required outcomes (refer to Summary Table 1).

 Why did I pick the various item types? Are they valid and reliable?

The items types that I chose for my KAI were directed by the “Item Type and Corresponding Cognitive Level” chart, and from the “Best Practices” item types from the PIDP 3230 course online readings (Vancouver Community College, 2017).  I applied best practices when choosing my item types so that they would reflect content validity – focused on my curriculum, and process validity – reliable items that reflect consistent learning outcomes (Fenwick & Parsons, 2009). The required learning outcomes of my students in the theoretical portion of Unit One of my course correlate to the following items:

Alternate Response – remembering and understanding. I chose this type of item because it assesses verbal knowledge without having to allocate a lot of time on the exam. The Alternate Response items on my KAI are valid because they reflect my lessons, however, the student has a 50/50 chance of selecting the correct response so are not completely reliable for evaluation unless they’re combined with other types of items.

Matching – remembering and understanding. I chose this type of item to assess comprehension of the goal of my course. These items require little space on my KAI, and they also can be completed in a short amount of time. The Matching Response items on my KAI are valid because they reflect the key lessons of my course. They are reliable items as well because I can evaluate my student’s overall understanding of the goal of Unit One.

Multiple Choice – remembering, understanding, applying, an analyzing. I chose this type of item because I want to assess knowledge of principles and procedures. The Multiple Choice items on my KAI are valid because the questions refer to important facts and concepts of my curriculum.  The items are reliable because I designed my incorrect alternative answers to be slightly plausible so that the student would consider the best and correct answer among the alternatives so that important learning outcomes are reflected.

Restricted Response Essay – remembering, understanding, applying, and analyzing. Alternate Response, Matching and Multiple Choice are selected response items; the correct response is provided, and the student must choose the answer. While this is a good representation of what the student has learned, constructed response items like the Restricted Response Essay are more indicative of what the student knows and what they will need to know to meet prescribed outcomes. My Restricted Response Essay items are valid because the learner must be able to explain in an articulate manner key components of the Perioperative Phase. These items are reliable because they will imitate learning outcomes; for example, in a clinical setting, a CDA will be able to select the correct perioperative protocol or procedure if they truly understand the important outcomes of the unit.

 What best practices did I use when creating the layout of my KAI?

When constructing my KAI, I referred to the PIDP 3230 course reading “Arranging the Items” (Vancouver Community College 2017). I used best practices by arranging the item types in sections from simple to complex. The items within each section follow the sequence of instruction through Unit 1 – The Perioperative Period as well as events in a clinical environment, but there is no detectable pattern of responses.

When constructing my KAI, I used best practices by following the “Format the Exam” document found in the PIDP 3230 online readings (Vancouver Community College, 2017). There is a cover page stating the name of the exam, a place for the student to enter their name and student number. The name of the course is clear and the topic of the exam is stated. The total mark, and the passing grade is specified. The cover page also contains a list of instructions explaining what section of the course they are being assessed on, the percentage that the exam is worth, a scoring guide, suggested time allocations for each section, and the total time allocation. There are instructions on how to record answers, and a page count on the cover page as well.

Exam instructions, item instructions, and items are not split between pages. Item types are grouped in separate sections with plenty of space between items. The items are numbered consecutively from the beginning of the exam until the end. The end of the exam is clearly marked. Finally, I had a colleague review my KAI for errors in content, and readability.

 What factors did I consider when creating the scoring guide?

The value of my KAI is outlined in my Table of Specifications (Table 1); the importance of each item reflects cognitive outcomes (quantifying the importance of items as they pertain to my lessons and curriculum). However, when creating my scoring guide, I referenced my instructor’s YouTube video “Creating the Scoring Guide” link found in the PIDP 3230 course (Vancouver Community College, 2017). The factors that I considered are accountability to my Table of Specifications (Table 1), and assisting my students with the following information about the KAI: how many sections there are, how many items in each section, what the items are worth, and the time that they should allow themselves to complete each section. These factors ensure process validity (the ability to obtain consistent results). My scoring guide is clearly displayed on the cover page of my KAI.

 How will I prepare the learners for the exam and provide feedback to them after the exam?

The KAI for Oral & Maxillofacial Surgery Module for CDA’s Unit One – The Perioperative Period is a summative evaluation; it is the final representation of a student’s success and competency in the first unit of the module (Fenwick & Parsons, 2009). I will prepare my learners by providing them with the following explicit written and verbal information at the onset of the entire course:

– a list of all exams and the values

– the content of the exam

– the value of the exam in the overall course mark

– the passing grade (percentage)

– the types and numbers of items

– the time allocation for the exam

– when to expect their results of the exam (the exams will be graded and distributed back to the students at the beginning of the next session)

– the students will have opportunity to provide feedback to the instructor regarding the KAI (this will take place once the exams have been given back to the students. It will be group discussion and there will be a time limit)

– my contact information (I will make myself available to the students for one-on-one questions and feedback)

It is important that students understand that “one course should create a grade from a blending of learner assessments” (Fenwick & Parsons, 2009, p. 142). Preparing them for an evaluation that will contribute to professional growth will enhance their success.

@ Please refer to my Resources page for works cited