by Priscilla O. Okunji, Ph.D., RN-BC
and Mary H. Hill, DSN., RN
Howard University, College of Nursing and Allied Health Sciences,
Division of Nursing
E-mail Address: email@example.com
Introduction: This article describes a pilot study conducted to survey senior nursing students’ responses to online testing preference and the efficiency of the Blackboard® system to offer such testing.
Methods: Preparation for online testing was done in phases starting with the planning for the testing space and computer availability. This was followed by selection and uploading of appropriate questions on the course Blackboard® site for student access. The questions were then exported to the testing port of the course web page with certain embedded restrictions. Finally, the survey was developed with seventeen items based on a review of the literature and implemented. Data analysis was performed on student responses on the survey, and results indicated moderate to high support for online testing.>
Findings: The survey analysis indicated that more than half of the students preferred online exams over traditional testing, indicating likability and convenience over traditional scantron paper exams. Most students showed a preference for online testing and recommended that test results be available for immediate review. One advantage to introducing online testing was to foster familiarity with computerized testing as preparation for the NCLEX-RN® licensure exam. However, a disadvantage was that the Blackboard® system lacked the capacity for comprehensive statistical analysis for test item discrimination.
Key Words: Online Education, Testing, Learning, Technology, Online Tests, Blackboard®
Many institutes of higher learning are integrating technology and informatics to promote innovation in the process of teaching and learning. Instructors are looking for efficient ways to deliver content and assessment. Advances in educational technologies are increasing at a rapid pace. The technology explosion creates a challenge for nursing faculty to stay current on existing and emerging technological developments. Today’s students expect information to be timely and engaging, and resources availed through technology provide opportunities to enhance learning environments (Price, Handley, Millara & O’Donovana, 2010). The numbers of online programs continue to expand (Hart & Morgan, 2010). According to Salamonson and Lantz (2005), more programs in higher education are adopting hybrid formats of course delivery and learning. The hybrid online learning with traditional classroom teaching offer effective and flexible course delivery without the complete loss of face-to-face contact. Online testing (use of information technology for any assessment-related activity) is another opportune application of technology, and readiness for online testing before integrating in nursing courses is an important consideration.
The technology explosion of e-learning may have major benefits for nursing education, especially with the current shortage of faculty (Neuman, 2006). Online teaching and learning are being increasingly adopted in both medical and nursing educational programs (Cook et al., (2008); Lewis, Davies, Jenkins & Tait, (2001); Reime, Harris, Aksnes & Mikkelsen, (2008); Tegtmeyer, Ibsen & Goldstein, (2001)). According to Ormrod (2011), course factors of relevance and design influence a learner’s decision to persist or drop a course even after enrollment. Nurse educators today face many challenges including how to design imaginative and innovative ways to educate future nurses (Rich & Nugent, 2010). Integration of technology in the classroom is becoming common with many colleges introducing technologies such as smart stations (electronics fitted with a built-in microprocessor); audience response tools (such as Clickers) and smart phones (e.g. Blackberry, iPhone, Android). Web 2.0 (defined as a new version of World Wide Web that allows users to interact and collaborate with each other in a social media dialogue) applications such as blogs, forums, and wikis are also being used as components of Course Management Systems (CMS).
Blackboard® is a popular Web-based server CMS that features convenient course management, customizable open architecture, and scalable design, allowing for integration with student information systems and authentication protocols. Simulations like SimMan (medical simulation mannequins, models or related artifacts) and virtual patients (interactive computer simulations used in health care education) are also in use in some nursing schools.
According to Chaffin and Maddux (2004), the evolution of technology has vitally influenced nursing education and is continually gaining greater importance. The Internet enables the students to learn in the classroom, remotely, or at home. Some nurse educators are now adapting traditional content with ease by using alternative teaching methods that integrate Internet technology (Chaffin & Maddux, 2004). Nelson et al, (2006) highlighted potential benefits of vital educational information systems for nurse faculty. The authors reported that although faculty manage large amounts of data, that only a few automated systems have been created to assist faculty to improve teaching and learning via the management of information related to individual students, the curriculum, educational programs, and program evaluation.
Fetter (2009) described how substantial evidence links information technology (IT) with improved patient safety, care quality, access, and efficiency, but cautioned how nurses must demonstrate competencies in computers, informatics, and information literacy in order to use technology effectively for practice, education, and research. The author further reported that the profession has established technology competencies for both beginner and experienced nurses and newly revised standards have been articulated for advanced practice nurses. Unfortunately, there is a critical concern that many nursing students do not have these requirements and that nurse educators are not prepared to remedy this.
The literature suggests that technology competency is still at novice skill levels for most faculty and students. In the USA, deficits in technological competencies are a significant concern, because of the federal government mandate of full implementation of Electronic Health Records (EHR) by 2014. It is a known fact t that EHR will require all nurses to use technology to deliver, document, and obtain reimbursement for patient care. This is the rational for the introduction of SimChart in many undergraduate schools to simulate EHRs before students graduate and join the paperless healthcare work environment.
Creedy et al, (2007) conducted a study to explore graduating Bachelor of Nursing (BN) students’ perceptions of a Web-enhanced learning environment, their computer literacy skills, and use of technology, and how these influenced their satisfaction. The results showed a 64% (n = 170) response rate. The authors provided Web-enhanced learning opportunities by integrating online activities and content such as quizzes, videos, and virtual laboratories to augment on-campus and off-campus learning approaches. The authors reported that more than half (61.4%) of the students reported having competent skills, while the quality and usefulness of the Web-enhanced material was rated as fair to above average. However, the students’ perceptions of technical and faculty support for Web-enhanced learning was rated low (Creedy et al, 2007). Overall the authors reported that satisfaction with the Web-enhanced program was associated with level of technology skills and perceived quality and usefulness of the online material. The authors reported that statistical analysis of the factors contributing to the students’ overall satisfaction of a Web-enhanced learning environment included literacy skills, access, perceived quality, usefulness, and support) and accounted for 18.5% of the variance. The authors concluded that as more nursing programs use Web-based resources, greater attention should be given to the initial assessment and development of students’ information literacy skills. As well, students with good technology skills are more likely to perceive Web-enhanced material as useful.
Kock, et al (2010) reported that tailoring information to the needs of the students is an critical strategy in today’s modern classroom. The authors described how web-based learning support, informed by multimedia theory, including interactive quizzes, glossaries with audio, short narrated Power Point presentations, animations and digitized video clips were introduced in a first year Bachelor of Nursing biological sciences course at an university in metropolitan Sydney. The authors enrolled all students and invited them to access the site then recorded the number of hits to the site using the student tracking facility available on WebCT (the course management system used in their program). The authors showed that 80% of students enrolled in the subject accessed the learning support site and that the students’ perception of the value of the learning support site was assessed using a web-based survey. The survey was completed by 123 participants, representing a response rate of 22%. Three themes emerged from the qualitative survey data concerning nursing students’ perception of the web-based activities: ‘enhances my learning’, ‘study at my own pace’, and ‘about the activities: what I really liked/disliked’. The authors concluded that web-based interventions, supplementing a traditionally presented nursing science course were perceived by students to be beneficial in both learning and language development.
Another study by Rouse (2007) entailed the adoption of a computerized database for testing and analysis to promote and evaluate the nursing student’s critical thinking skills and prepare them to write the National Council Licensure Examination (NCLEX) exam after graduation. According to Schmidt & Stewart (2009), faculty members are challenged to create meaningful learning activities that enhance online nursing education. The authors discussed the implementation and usage of Second Life as an innovative Internet-based strategy to engage students in active learning. As well, Campbell and colleagues (2008), concluded that a research methods course could be web-based and found that increased online activity was associated with higher assignment marks. The authors highlighted how new opportunities for educational research have emerged through the use of virtual learning environments such as the Blackboard® system to routinely record the activities of learners and instructors. Many schools with online programs use Web 2.0 applications, such as forums, wikis, blogs, and virtual classrooms to enhance online interaction with students. Other sophisticated social media are being used by some, such as Second Life (an online virtual world that enables users, called Residents, to interact with each other through avatars) to provide real- time (synchronous) interaction.
Online testing is another example of technology integration in nursing education. Some authors argue that information should be tailored to a learner’s unique needs for effective and flexible learning (Park & Choi (2009); Koch et al. (2010); Glinkowski & Ciszek,(2007); Mangunkusumo et al. (2007); Ruiz, Mintzer & Leipzig (2006)). Online testing as a method of assessment has not been widely adopted in traditional programs yet due to space, computer availability, technical concerns, security and reliability of the testing system, and student and instructor comfort levels. Instructors have contemplated about whether students should write exams in classrooms due to concerns about space. Yet at the same time instructors worry about the integrity of tests if students write them online without supervision, or need technical assistance.
Authors such as Wu et al., (2006) warn about unsuccessful online learning, but several other studies (Kock, (2010), Salamonson and Lantz, (2005), Bloomfield, Roberts & White, (2010); Cook et al., (2008) and Moule, (2006)) have demonstrated some positive outcomes of e-learning and testing. Online tests offer numerous advantages including access and convenience which encourage student time management skills. Students also instantly see the result of their test because the results are automatically generated to give immediate feedback. Furthermore, after the tests are scored, the data statistics are immediately loaded in the grading folder for instructor access and interpretation. Another major benefit of online testing is the time savings for grading compared to traditional pencil and paper shading and scantron grading (Newman, 2000). Many schools are going digital due to time management, accuracy cost, and access of digitally stored grades
Traditionally, the Division of Nursing in our institution used the CMS for student communication and grading but few had incorporated instructional technology for research, classroom assignments or quizzes in general. Preparation for online testing was done in phases starting with the planning for the testing space and computer availability. This was followed by selection and uploading of appropriate questions on the course Blackboard® site for student access. The questions were then exported to the testing port of the course web page with certain embedded restrictions. Finally, the survey was developed with seventeen items based on a review of the literature and implemented. Data analysis was performed on student responses on the survey, and results indicated moderate to high support for online testing.
The Division of Nursing expanded the use of the CMS recently by incorporating a new online program, Blackboard® that included synchronized and a-synchronized teaching methods. The University recently adopted this new CMS to met our unique teaching and learning needs. Blackboard® offers custom applications to help one adapt the Blackboard Learn™ platform to meet unique needs and preferences and to and create tailored applications for students and educators.
According to the Blackboard Learn™ plus (2011), custom applications feature CMS building blocks™ development to deliver a custom software solution that was reported to have been professionally designed and tested. This new version of CMS enhances and extends learning to meet course specific requirements. It also includes the necessary information, technical exercises, development examples, and documentation to allow administrators to effectively manage the application. As well, the new integration and customization maintenance tools provide the reliability of ongoing support for the custom application (Blackboard Learn™ plus, 2011). Unfortunately, this CMS platform does not compute test item analysis. This component is necessary because it enables instructors to recognize, focus on and teach content that students are weak on.
Class evaluation can be done online in both traditional and online program but whether the current Blackboard® CMS version is efficient in reflecting test results and student friendly is not clear. This study focused on a pilot experience of recent online testing in an undergraduate traditional nursing course to determine if Blackboard® CMS is an effective tool for online testing.
The online testing was implemented in phases. The initial phase started with the planning and securing space in three separate sites: Health Sciences Stokes Library Informatics room with 15 computers, Division of Nursing Learning Resource Center (LRC) with 14 computers and the remaining students (8) were housed in the lecture room (ISAS 100) with their wireless laptops. This led to item analysis and creation of the questions within the institution`s Blackboard® system. The third phase involved exporting the questions for student use after inserting all the control cues including the timing and password for access. The fourth step was the creation of questions for the survey which generated the data for analysis. The survey questions were coded using a likert scale with points of very high (5), high (4), moderate (3), low (2), very low (1) and not applicable (0). Two exams were given and supervised by instructors at the three separate sites for each exam.
The participants (n = 37) were senior students enrolled in a Nursing Leadership and Management course and the online survey was administered via the Blackboard® CMS (see Table 1). The survey submission was anonymous to avoid identification of participants (blind submission). The survey was voluntary and ethical approval from Howard IRB was obtained prior to the submission of this paper.
Data were collected via the online survey, presented in Blackboard® at the end of the 2011 spring semester and quantitatively analyzed for frequency of response to explore student perceptions and satisfaction related to 17 variables. The survey information was tallied and entered into the variable table as shown in Table 2.
All data were subjected to statistical analysis available in the Blackboard® CMS to ensure meaningful results. General data analyses were conducted in the following order. First, descriptive statistics was used to summarize students’ demographics, experiences, perspectives and satisfaction level. Next, Excel was used to table the baseline characteristics of the students (Table 1) and finally, the students’ perception and satisfaction levels of the online testing were represented as shown in Table 2.
The findings of the study showed that the majority of the students were female, less than 24 years of age and less than half (40.5%) had never had online testing experience.
Of the 37 students surveyed, all completed the questionnaires for a response rate of 100%. As shown in Table 1, 13.5% were males while the remaining were females (86.5%). Many of these students were young adults with ages ranging from < 24 (62.2%), 25 – 35 years (27.0%) and 36 – 46 years old (10.8%). The results also indicated that fifty nine percent of the students had previous online testing while 40.5% had none.
Table 2 presents the results of student perception and satisfaction levels of the online testing which indicate that students have very high perception and satisfaction on “I don’t have to wait until next class to find out what I made on the exam because I can obtain my score immediately after taking the exam,’ (40.5%). This was followed by students’ high perception and satisfaction on, “growth of online exam is a reality for most colleges,” (56.8%) “the system allowing the students exiting a test and continuing the test from the last question they answered,” (51.3%), “Within the exam-taking period, I can break and resume where I stopped,” (45.9%); “being comfortable in computer level” (45.9%) and “Online testing environment is in line with an online “NCLEX” or other proficiency online type exams which is becoming more popular” (37.8%).
The students’ medium/moderate perception and satisfaction were “advantages of the online exam over traditional scantron paper exam,” (54.1%) “online exam likability and convenience over the traditional scantron paper exam,” (48.6%), “The preference of using an online testing system enables students to get used to a similar environment as they would find themselves working in today’s paperless acute care,” (48.6%), “preference of the system showing all the questions at the same time in a window,” (45.9) and “preference of the system showing the questions one at a time in a window,”(45.4%).
Qualitative student comments on how the online testing could be improved for more likeability included:
1. “I like the idea of the test still being proctored, as opposed to taking the exam at home, because that is how NCLEX is.”
2. “The more practice the better. It is good preparation for NCLEX.”
3. “After get use to it maybe it will increase my ability to take on line test.”
4. “More practice and online exams at junior level.”
5. “Having taken exams online in other classes and an online class in the past, I don’t mind online exams.”
6. “I think that the time limit and remaining time should be specified at the top of the screen. Also results should be available immediately. I like the idea of showing fewer questions on the screen as well (maybe 10 questions instead of all 50).”
7. “Finding my score out right away after I take it. And doing it on working computers.”
8. “As long as we can go back after completing exam to recheck our answers. Taking exam online is not a bad idea; I have to get use to it.”
9. “It’s okay how it is now.”
10. “Given adequate time, and opportunity for a break if needed.”
11. “Practice assignments online.”
12. “It is an innovative step and prepares us for the real world, which is progressing rapidly
from paper based to information technology.”
13. “More time to take the exam.”
14. “Ability to take a break.”
15. “Ability to take the exam at my own time.”
Key benefits of online testing identified by the students centered on the relevance of online testing to other online tests they must take, including NCLEX and preparation for work environments tests, and being able to see their results immediately after the test. Our results showed promising responses to the online testing, indicating high likeability and positive student perceptions and satisfaction (Chen and Chuang, 2012).. These positive effects are in keeping with other reported studies (Kock, (2010), Salamonson & Lantz, (2005), Bloomfield et al., (2010); Cook et al., (2008) and Moule, (2006)).
However, the main concern for future use of the Blackboard® CMS for online testing was its limitations for comprehensive statistical analysis of grades and test item responses. The system’s statistical result output only gives basic statistical analysis of frequency, mean, mode, median, and standard deviation with no test item analysis for each question.
As well, students commented that it could be more user-friendly if the CMS settings/controls could be adjusted by the test-taker to present a determined number of questions on the screen (such as all questions, one question, and so on) and if the test-taking could be paused so they could go for break as needed, then resume where they left off without loss of allotted test time.
The results indicate that students engaged favourably with the online testing and wanted more in other courses. Based on this result, it could be concluded that the system is well suited for certain tests such as quizzes and other homework assignments that some instructors have already initiated online. However, exams with many questions (50 or more) that require item discrimination for students benefit may need a more robust system. Hopefully, Blackboard® will evolve by redesigning system components that could feature expanded statistically- driven results that are required in today’s innovative summative program evaluations.
A limitation of this study is that a small sample of students (n=37) and only the senior students were used. A second limitation was that the survey questions may have been biased and leading. The validity of the study tool could have been increased if the students’ perspective and satisfaction survey was delivered by another faculty other than the course instructor and was examined by experts for validation prior to administration.
We shall continue to use the Blackboard® platform for both online and traditional testing but will continue to search for supportive exam software that would complement Blackboard® and accomplish the goal of our technology integration into the curriculum with needed test item analysis outcomes. Repetition of this pilot study will utilize a larger sample size from different classes and courses and the survey questions would be validated and delivered by a neutral individual to avoid bias.
Blackboard Learn plus. (2011). Custom applications, Retrieved May 19, 2011 from http://www.blackboard.com/Platforms/Learn/Services/Implementation-Services/Custom-Applications.aspx
Bloomfield, J., Roberts, J. & While, A. (2010). The effect of computer-assisted learning versus conventional teaching methods on the acquisition and retention of handwashing theory and skills in pre-qualification nursing students: a randomized controlled trial. International Journal of Nursing Studies, 47 (3), 287-294
Campbell M, Gibson W, Hall A, Richards D, Callery P. (2008). Online vs. face-to-face discussion in a Web-based research methods course for postgraduate nursing students: a quasi-experimental study. International Journal of Nursing Studies, 45(5):750-9.
Chaffin A. & Maddux C. (2004). Internet teaching methods for use in baccalaureate nursing education. Computers and Informatics in Nursing, ;22(3):132-42.
Chen, H. Y. & Chuang, C. (2012). The learning effectiveness of nursing students using online testing as an assistant tool: A cluster randomized controlled trial. Nurse Education Today 32 (3):208-213.
Cook, D.A., Levinson, A.J., Garside, S.,Dupras, D.M. Erwin, P.J & Montori, V.M. (2008). Internet-based learning in the health professions: a meta-analysis. Journal of the American Medical Association, 300 (10), 1181-1196.
Creedy DK, Mitchell M, Seaton-Sykes P, Cooke M, Patterson E, Purcell C. & Weeks P. (2007).Evaluating a Web-enhanced bachelor of nursing curriculum: perspectives of third-year students. Journal of Nursing Education,;46(10):460-7.
Fetter M. (2009). Improving information technology competencies: implications for psychiatric mental health nursing, Issues in Mental Health Nursing, 30(1):3-13.
Glinkowski, W., Ciszek, B.(2007). WWW-based e-teaching of normal anatomy as an introduction to telemedicine and e-health, Telemedicine and e-Health 13(5), 535-544.
Hart L. & Morgan L. (2010). Academic integrity in an online registered nurse to baccalaureate in nursing program. Journal of Continuing Education in Nursing, 41(11):498-505.
Koch J, Andrew S, Salamonson Y, Everett B, Davidson P. (2010). Nursing students’ perception of a Web-based intervention to support learning. Nurse Education Today, 30(6):584-90.
Lewis, M.J., Davies. R. Jenkins, D. Tait, M.I. (2001). A review of evaluative studies of computer-based learning in nursing education. Nurse Education Today, 21(1), 26-37.
Mangunkusumo, R.T., Brug, J., Duisterhout, J.S., de Koning, H.J., Raat, H. (2007). Feasibility, acceptability, and quality of Internet-administered adolescent health promotion in a preventive-care setting. Health Education Research, 22(1), 1-13.
Moule, P. (2006). E-learning for healthcare students: developing the communities of practice framework. Journal of Advanced Nursing, 54(3), 370-380.
Nelson R, Meyers L, Rizzolo MA, Rutar P, Proto MB, Newbold,S.( 2006). The evolution of educational information systems and nurse faculty roles. Nursing Education Perspectives, 27(5):247-53.
Neuman, L. (2006). Creating new futures in nursing education: Envisioning the evolution of e-nursing education. Nursing Education Perspectives, 27(1), 12-15.
Newman, C., 2000. Online testing rated. Advertising-Age, 71(20), p 64.
Ormrod, J.E.(2011). Social cognitive views of learning. In Smith, P.A. Educational psychology: developing learners (pp.352-354). Boston, MA: Pearson Education, Inc.
Park, J. H., & Choi, H. J. (2009). Factors Influencing Adult Learners’ Decision to Drop Out or Persist in Online Learning. Journal of Educational Technology & Society, 12(4), 207- 217. Retrieved from Academic Search Premier database on May 19, 2011.
Price, M., Handley, K., Millara, J., & O’Donovana, B (2010). Assessment & Evaluation in Higher Education, 35 (3), 277-289. DOI:10.1080/02602930903541007
Reime, M. H., Harris, A., Aksnes, J., Mikkelsen. J. (2008). The most successful method in teaching nursing students infection control–E-learning or lecture? Nurse Educator Today, 28 (7), 798-806.
Rich KL, Nugent K. (2010). A United States perspective on the challenges in nursing education. Nurse Education Today, 30(3):228-32.
Rouse DP. (2007). Implementation of a computer database testing and analysis program. Computers and Informatics in Nursing,25(5):273-80.
Ruiz, J.G., Mintzer, M.J., Leipzig, R.M.(2006).The impact of e-learning in medical education. Academic Medicine, 81(3), 207-212
Salamonson Y, Lantz J.(2005). Factors influencing nursing students’ preference for a hybrid format delivery in a pathophysiology course. Nurse Education Today, 25(1):9-16.
Schmidt B, Stewart S. (2009).Implementing the virtual reality learning environment: Second Life. Nurse Educator, 34(4):152-5.
Tegtmeyer, K., Ibsen, L., Goldstein, B. (2001). Computer-assisted learning in critical care: from ENIAC to HAL. Critical Care Medicine, 29 (8), N177-N182 suppl.
Wu, J.P., Tsai, R.J., Chen, C.C., Wu, Y.C. (2006). An integrative model to predict the continuance use of electronic learning systems: hints for teaching, International Journal on E-learning, 5(2), 287-302.
Dr. Priscilla Okunji has been on nursing faculty at Howard University since August of 2009. She has a Bachelor and Master’s degrees in Medical Microbiology and a Post graduate degree in Education before relocating to join her husband in United States. Dr. Okunji is the Coordinator of the first and only RN-BSN Online which she developed and implemented at Howard University in summer 2010. She is the current winner of the 2011 Center for Excellence in Teaching, Learning and Assessment (CETLA) Award for, “Teaching with Technology,” at Howard University. Dr. Okunji also has a Master’s degree in Nursing Informatics from the University of Maryland and recently earned her doctoral degree in Health Sciences with a specialization in International Health and Research.
She held several managerial positions as Clinical Coordinator, Director for Nursing Education and Quality Assurance. Her professional background merges over sixteen years of teaching, clinical and research experience in Microbiology, Nursing and Informatics. Dr. Okunji has board certification in Medical Surgical Nursing and her scholarly paper at the University of Maryland inspired her into using the Agency for Health Research and Quality (AHRQ) large database to study the “Outcomes of Diabetic Myocardial Infarction Inpatients: Patient and Hospital Characteristics” as her Ph.D. dissertation. Dr. Okunji has presented and published in peer review journals such as Circulation and Diabetes. Dr. Okunji is currently a Principal Investigator of a funded grant—“Comparative Health Disparity Outcomes of Inpatient Care”, supported in part by NIH, National Center for Minority Health and Health Disparities.
Mary H. Hill is the Interim Dean and Professor at College of Nursing and Allied Health Sciences, Howard University in the Washington District of Columbia. Dr. Hill obtained her Baccalaureate in Nursing from Tuskegee University, Masters in Nursing from University of Maryland and Doctor of Science in Nursing University of Alabama at Birmingham. Prior to joining Howard University she held many administrative positions in many colleges and universities. Dr. Hill’s Research interest includes Breast Cancer among African American Women, Obesity, Diabetes Consumer Education, Organizational Development, Cultural Diversity in Organizations and Outcomes Assessment and Evaluation. She has authored and coauthored selected peer reviewed publications. She is a renowned advocate for disadvantaged students as evidenced in her serving as a project director for many funded grants on the Scholarships for Disadvantaged Students at Howard University. Dr. Hill has had many awards including a visionary leader award which correlates to her believes that technology integration in the curriculum would lead nursing profession to a new horizon, hence the implementation of the online RN-BSN at Howard University Division of Nursing.