Undergraduate Nursing Students’ Perceptions of Computer Based Testing

by Jennifer Hooper, MN, RN

School of Nursing and Midwifery
Mount Royal University

Abstract

Background & Purpose:The National Council Licensure Examination for Registered Nurses (NCLEX-RN) is a computer adaptive licensure examination that nursing students are eligible to write upon completion of their undergraduate nursing degree. Success on this exam is a requirement for Registered Nurse practice. Historically, the Canadian licensure examination was a paper-based exam. However, in 2015 the NCLEX-RN was adopted. Initially Canadian pass rates declined and nursing schools have been seeking strategies to better prepare students for this exam. Practice with computer based testing (CBT) may be one approach.  However, CBT has not been widely used and many nursing programs continue with paper-based exams. Recently CBT was integrated into a third year undergraduate nursing course in Alberta, Canada. The purpose of this qualitative study was to gain an understanding of undergraduate nursing student experiences with and perceptions of CBT. Specifically, the goals were to inform personal teaching and evaluative practices, positively contribute to changes to future course testing approaches, and share knowledge with other educators who may be considering the integration of CBT into their courses. Sharing this knowledge may contribute to the body of literature and serve as a foundation on which further research on this topic area could be conducted. 

Method: Through purposive sampling, data was collected from 38 students who completed a set of reflective questions. Data was analyzed by thematic analysis. 

Results: Four themes were generated: immediacy matters, distrust of self, navigating the new, and high stakes on the horizon. Findings including the benefits, challenges, and recommendations are discussed. 

Conclusion: There are advantages and disadvantages to CBT. Providing CBT opportunities may encourage students to reflect on their learning, test-preparation, and test-taking strategies promoting reflection upon how they prepare for and engage in future computerized exams.

Keywords: Computer Based Testing, Test Taking Strategies, NCLEX-RN

Introduction

The National Council Licensure Examination for Registered Nurses (NCLEX-RN) is a computer adaptive licensure examination that nursing students are eligible to write upon completion of their undergraduate nursing degree. Success on this exam is a requirement for registered nurse (RN) practice. Historically, the Canadian licensure examination was a paper-based exam. However, in 2015 the NCLEX-RN was adopted for its computer adaptive format (CNA, 2015a). The NCLEX-RN exam is different than the previous Canadian licensure exam in a variety of ways including differences in content tested, style of questions (alternate format), and delivery format (computerized). Upon initial implementation of this exam, Canadian pass rates declined by over 10 % compared to previous pass rates (CNA, 2015b) and since then, nursing schools have been seeking strategies to better prepare students for this exam. Some literature suggests that practice with computer based testing (CBT), whether adaptive or linear, may be one approach (CASN, 2014.; Dosch, 2012, Okunji & Hill, 2014). However, CBT has not been widely used in Canadian undergraduate nursing programs due to issues with computer availability, IT support, and faculty and student comfort levels with technology. Therefore, many nursing programs continue with paper-based exams. There is limited literature surrounding Canadian undergraduate nursing students’ experiences with and perceptions of CBT. Some literature specifically supports the use of Blackboard (a learning management system) to increase student familiarity with CBT in preparation for future computerized exams including the NCLEX-RN (Okunji & Hill, 2014). Nursing educators have a duty to prepare students for the NCLEX-RN exam, implement successful preparation strategies into undergraduate programs, and to collect data regarding these strategies. While empirical data (strategies and pass rates) are critical, data on student perceptions of interventions must also be gained (Quinn, Smolinski, & Peters, 2018).

Background

In the four-year Mount Royal University Bachelor of Nursing program, strategies have been implemented to prepare students for the NCLEX- RN including preparatory modules and assignments, and the incorporation of alternate format questions on tests. With the assistance of the university’s academic development centre, this author recently converted all paper-based exams to CBT in a compulsory third year nursing course (maternal-newborn nursing) using the institution’s Blackboard system. Blackboard is used extensively throughout the nursing program in other capacities (e.g. course notes, communication, etc.). Therefore, students were familiar with this learning management system prior to this course however this is the first time they used it for testing. The author’s intent was to increase student exposure to alternate (non-paper based) testing approaches. Prior to this course in the program, students had no evaluative (exams associated with grades) CBT exposure and exams remained paper-based. Of note, many alternate format questions (e.g. ordering, matching, hot-spot) cannot be performed at all or in the same way on paper-based exams as they can with CBT. Students in this course are exposed to CBT on three occasions: the practice exam (non-evaluative), and the midterm and final exam (both evaluative). While the practice exam can be performed repeatedly and from any computer, the midterm and final exams are performed and invigilated in computer labs on campus. After students submit their midterm answers, they immediately receive their grade and have the opportunity to review their exam under direct supervision (many questions contain rationale for correct and incorrect answers). In contrast, the final exam does not provide the same review opportunity as it is considered strictly evaluative. Students are made aware during course orientation and in the course syllabus, that the course will use CBT approaches via Blackboard including the opportunity to do the practice exam prior to the midterm and final exam. This author has no product (commercial or proprietary) interest, potential conflicts of interest, financial, or otherwise to disclose.

Aim

The aim of this qualitative study was to gain an understanding of undergraduate nursing student experiences with and perceptions of CBT. Specifically, the goals were to inform personal teaching and evaluative practices, positively contribute to changes of future course testing approaches, and share knowledge with other educators who may be considering the integration of CBT into their classroom and undergraduate course settings. Sharing the knowledge gained may contribute to the body of literature and serve as a foundation on which further research on this topic area could be conducted.

Method

Reflective questions were distributed to assess student experiences with and perceptions of CBT in a third year undergraduate nursing course. Data collection focused on three main areas including the identification of challenges, advantages, and future recommendations for this testing approach. Besides being the author and researcher, I was also the faculty member teaching the course, to ensure no potential or perceived conflict of interest, the study was not implemented until after the course completion date and all final grades had been entered. Furthermore, the research study was described and data was collected by an external faculty member who did not teach in the third year of the nursing program nor held any other teaching or supervisory relationship with these students at that time. This study was approved by the university’s human research ethics board.

Data Collection

Through purposive sampling, a set of five reflective questions were distributed to students and were completed anonymously. Participation in this study was voluntary and the study was open to all students who completed the course in the winter 2019 semester, where CBT was the compulsory testing modality.

The research was guided by the following questions:

• What did students learn from engaging in CBT?
• What did students like/dislike about CBT?
• What would students recommend to enhance the experience of CBT?

Data Analysis

Data was collected and analyzed by thematic analysis. This was performed in a three phased process: 1) Reading for resonance: familiarizing self with the content 2) Coding: identifying words, phrases, and passages of text that are meaningful to the research question and 3) Categorization into themes: inductive codes were reviewed, and subsequent themes were generated (Jirojwong, Johnson, & Welch, 2014).

Results

Within one course, all 38 students participated and completed the anonymous set of reflective questions and all were included in the analysis. Four main themes were generated:

• immediacy matters,
• distrust of self,
• navigating the new, and
• high stakes on the horizon.

Immediacy Matters

In a world where technology enables instant response, the concept of ‘immediacy’ does not limit itself from CBT. Many students commented upon the speed of the test (not having to fill out a Scantron and overall decreased completion time), immediacy of grades (not having to wait for the professor to grade and then post results), and the immediacy of feedback (being able to review their exam immediately after submission without having to make a separate appointment with their professor). While immediacy was an overall positive theme, it uncovered the following second identified theme.

Distrust of Self

Students revealed what was deemed to be a lack of trust in oneself due to the speed of test completion. Students believed that they were not reading the questions carefully enough during the exam because they perceived it took them less time to read and answer each question compared to prior paper-based testing experiences. Furthermore, many indicated because it was faster, it was easy to second guess themselves and change answers:

“It was easy to second guess myself… I felt like I wasn’t being as careful because it didn’t take as long but it helped me realize that it was just the mode of testing… [after the midterm] I knew for the final that I should not second guess [myself]…it taught me how to trust my initial instincts”. 

Navigating the New

After experiencing CBT, students revealed their learning needs and that a new mode of testing requires the need to learn new test-taking strategies. They identified that they could not use previous strategies that they used for paper-based testing including circling or highlighting important information, crossing out answers or writing notes on the exam itself:

“I found it harder to think through the questions and wasn’t able to cross answers out…not being able to write directly on the test is difficult for me…I couldn’t highlight/underline parts of the question to avoid mistakes…I had to change my test-taking strategy to accommodate the different format… I would [like] strategies offered to help with online test taking as strategies for paper testing can’t be transferred”.

Students identified that after the practice and midterm they felt more prepared for the final exam:

“I knew what I was going into and how to navigate the exam… it prepared me with some strategies for making notes during the exam without directly writing on the question”.

Another aspect of navigation was student learning of how to do computerized alternate format questions as many of these questions cannot be performed in the same way (or at all) on a paper exam. One student wrote:

“I liked how it was more visual testing of knowledge of models/anatomy…[it] provided an idea of how to answer some of the question formats…I was able to get a feel for different questions that could be asked (matching, selecting from a picture)”.

High Stakes on the Horizon

The final major theme revealed in the data was students’ perceptions of preparation strategies for future CBTs, specifically the NCLEX-RN. They viewed the CBT experience as a preparation opportunity and evidenced that the high stakes NCLEX-RN is at the forefront of their minds. They identified the importance of, and their desire to have, increased NCLEX-RN preparation strategies in place at the undergraduate level before they encounter the licensure exam:

“I preferred it over paper tests and I wish more courses did it to prepare me for the NCLEX…gave an idea of how the NCLEX will feel… I felt it prepared me for future CBTs like the NCLEX…it was nice to experience a computer based exam before the NCLEX…I would like to see this implemented for ALL nursing courses”.

Discussion

The concept of CBT is not new. However, there is a lack of current literature surrounding CBT and effective NCLEX-RN preparation strategies in Canadian undergraduate nursing programs. Nursing educators should support students by implementing successful preparation strategies into undergraduate programs (Quinn, Smolinski, & Peters, 2018). Suggested strategies include posing higher-level and alternate format style of questions on exams and increasing student exposure to computerized exams (CASN, 2014; Clariana & Wallace, 2002; Dosch, 2012, Okunji & Hill, 2014, Pawasauskas, Matson, & Youssef, 2014).

There are many advantages to CBT including immediate exam scoring and feedback, reduced scoring errors, more effective use of large question banks, reduced cost and time, potential for adaptive testing, and enhanced analysis of individual student performance. Disadvantages however include increased student anxiety levels attributed to student anxiety about technical problems, inability to write on the exam itself, inability to change answers or skip and return to previous questions, or concern that concentration may be broken even when allowed to write on a scratch piece of paper (Bloom, Rich, Olson, & Adams, 2018; Mason, Patry & Berstein, 2001; Vrabel, 2004). The students’ comments reflected many of these advantages and disadvantages as noted in the literature. While immediacy of receiving grades was important, immediacy of feedback too was considered advantageous. They deemed CBT as a learning tool as they had the ability to review questions and rationale for both correct and incorrect answers, thus revealing individual knowledge strengths and deficits.

Major perceived disadvantages also surrounded test-taking strategies. While in this study students did have scratch paper to make notes upon during the exams (similar to the NCLEX-RN format), but they still perceived a loss of their usual test-taking strategies and learned that different modes in testing required the development of new strategies. This was a significant finding and this author will consider how to enhance student recognition of this need. This could be achieved by further emphasizing the notion in the course orientation and syllabus. Also, providing direction within the practice CBT instructions that encourage students to use scratch paper and reflect upon test-taking strategies while they practice could be considered. Since the time of data collection, this author has increased the number of practice CBTs in the course so this could serve as increased opportunities for students to reflect on their test-taking strategies. Another consideration may be to collaborate with the university’s student learning services program to develop test-taking strategy resources and workshops to support students not only in the nursing program, but also those that are exposed to CBT in other programs across the institution. 

As evidenced in the data, students perceived that these CBT opportunities provided NCLEX-RN preparation in the context of testing mode and exposure to computerized alternate format styles of questions. While students were aware that the CBTs were not the same as the NCLEX-RN in some ways (e.g. non-adaptive), they still perceived this as preparation. They consistently used word choices of ‘glimpse’ ‘reflects’ and ‘taste’. 

“I liked that we were practicing how it would sort of be like for the NCLEX exam…provides a glimpse of what the NCLEX might look like…it allowed me to get a taste of what the NCLEX exam would look like”. 

Furthermore, and surprisingly, there were student suggestions to make the CBTs even more similar to the NCLEX-RN with recommendations to make each exam question appear one at a time on the screen. This author developed the CBTs to display all questions at once allowing students to easily scroll back and forth, review questions and make any changes to their answers prior to submitting their exam for scoring. The hypothesis was that it might decrease anxiety as this was their first experience with CBT, however this was not the case for all students.

While students learned that test-taking strategies need to be altered for different modes of testing, some identified test preparation as a concept. Many educators are curious about how students prepare for and study for their exams. The literature described both ineffective (e.g. re-reading course textbooks and re-reading class notes) and effective (e.g. flash-cards and practice tests) strategies. However, students tended to underuse practice tests and place too much emphasis on re-reading (Blasiman, Dunlosky & Rawson, 2017; Janes, Dunlosky, & Rawson, 2018; Morehead, Rhodes, & Delozier, 2015). While substantive literature exists regarding effective test preparation strategies in general, there is little that specifically targets those for CBT. This may be an area for future research.While in this study both advantages and disadvantages of CBT were identified, overall, feedback was positive. From these findings, this author is motivated to continue offering CBT and provide additional test preparation and test taking resources to enhance student support.

Limitations

A limitation of this study includes the small sample size (n=38) and that the data gained from this purposive sample reflects only those perceptions of students in a single third year undergraduate nursing course. Also, as with any self-reported data, bias may contribute to findings. While these findings are not generalizable, it may be worthwhile for other educators to consider how the information gained may contribute to the integration of potential supportive strategies in other nursing courses and programs that may be considering using CBT.

Conclusions

Nursing educators must continue to consider, implement, and evaluate appropriate strategies to support students in NCLEX-RN preparation. This may include increasing the number of practice tests, increasing the number of higher level and alternate format questions on exams, and exposing students to CBT. Providing these opportunities may encourage students to reflect on their learning, specifically surrounding test-preparation and test-taking strategies promoting reflection upon how they prepare for and engage in future computerized exams. 

References

Blasiman, R., Dunlosky, J.  & Rawson, K. (2017).The what, how much, and when of study strategies: comparing intended versus actual study behaviour,Memory,25(6),784-792.

Bloom, T., Rich, W., Olson, S. & Adams, M. (2018). Perceptions and performance using computer-based testing: One institution’s experience. Currents in Pharmacy Teaching  and Learning,10, 235-242.

Canadian Association of Schools of Nursing (CASN), (2014). Transitioning to the NCLEX-RN:               Briefing Note: Repertoire of Strategies. Retrieved from https://casn.ca/wp-     content/uploads/2014/12/TransitioningtotheNCLEX.pdf

Canadian Nurses Association (CNA) (2015a). Nursing Profession Concerned with First NCLEX-RN Results.Retrieved from https://cna-aiic.ca/news-room/news-releases/2015/nursing-profession-concerned-with-first-nclex-rn-results

Canadian Nurses Association (CNA). (2015b). NCLEX-RN Roundtable: Student Supports. Retrieved from https://www.cna-aiic.ca/-/media/cna/page-content/pdf-en/nclex-rn-roundtable_student-supports-report.pdf

Clariana, R. & Wallace, P. (2002). Paper-based versus computer-based assessment: key factors  associated with the test mode effect. BJET, 33, 593-602.

Dosch, M. (2012). Practice in Computer-Based Testing Improves Scores on the National Certification Examination for Nurse Anesthetists. American Association of NurseAnesthetists Journal, 80 (4), S60-S66.

Janes, J., Dunlosky, J. & Rawson, K. (2018). How Do Students Use Self-Testing Across Multiple Study Sessions When Preparing for a High-Stakes Exam? Journal of Applied  Research in Memory and Cognition, 7,230-240. 

Jirojwong, S., Johnson, M., & Welch, A. (2014). Research methods in Nursing and Midwifery: Pathways to evidenced-based practice(2^nded). Oxford University Press.

Mason, B., Patry, M., & Bernstein, D. (2001). An Examination of the Equivalence between Non-Adaptive Computer-Based and Traditional Testing. Journal of Educational Computing Research,24(1), 29–39. https://doi.org/10.2190/9EPM-B14R-XQWT-WVNL

Morehead, K., Rhodes, M., & Delozier, S. (2015). Instructor and student knowledge of study strategies. Memory,10, 1-15. 

Okunji, P. & Hill, M. (2014). Technology integration in undergraduate traditional nursingprograms: Students online testing experience. Canadian Journal of Nursing Informatics, 9 ,(1-2). Retrieved from:         http://search.proquest.com/docview/1698427884?accountid=28844

Pawasauskas, J, Matson, K., & Youssef, R. (2014). Transitioning to Computer-Based Testing. Currents in Pharmacy Teaching and Learning, 6,289-297. 

Quinn, B., Smolinski, M. & Peters, A. (2018). Strategies to improve NCLEX-RN success.  Journal of Teaching and Learning in Nursing, 13, 18-26.

Vrabel, M. (2004). Computerized Versus Paper-and-Pencil Testing Methods for a Nursing Certification Examination. Computers, Informatics, Nursing, 22, 94-8.

AUTHOR BIO

Jennifer Hooper, RN BScN, MN

Jennifer has completed her undergraduate Bachelor of Science in Nursing degree through St. Francis Xavier University in Nova Scotia and her graduate Master of Nursing degree at University of Calgary in Alberta. She is an Associate Professor at Mount Royal University who teaches across the Bachelor of Nursing program as well as in the Bridge to Canadian Nursing program. She has a strong clinical focus in maternal-newborn nursing and continues to practice in this setting alongside her educator role. Jennifer is actively involved in the Institute for the Scholarship of Teaching and Learning and one of her key research and scholarly pursuits surrounds higher education and supporting student learning and transitioning to practice.