by
Jacqueline Ellis, RN, PhD,
University of Ottawa,
School of Nursing and Children’s Hospital of Eastern Ontario,
Chronic Pain Service
jellis@uottawa.ca

Avi Parush, PhD,
Monika Taing, BA,
Catherine Campbell, P.Eng. M.Des,
Charissa Chiu, RN, MScN,
Susan Eldred, RN, MScN,
Anthony Whitehead, PhD,
Danica Irwin, B. Pharm,
Regis Vaillancourt, Pharm D, FCSHP,
Brenda Martelli, RN(EC) MEd,
Debby Voskamp, RN, BScN,
Pat Elliott-Miller, RN, MScN,
Ken Farion MD, FRCPC.

Abstract

Dosing errors are the most common type of pediatric drug errors, with over-dose outnumbering under-dose errors. Weight-based calculations are essential for proper dosing but complex in pediatric settings where patient weights may vary from 0.5 kg for a premature newborn to over 100 kg for an obese adolescent. A computer application called iDoseCheck (iDC) was developed to assist nurses in the calculation and verification component of the double check for administering intravenous morphine.

The purpose of this study was to assess whether a graphic dose calculator, in comparison to standard paper/pencil and calculator (PPC), can support the double check process and reduce the rate of potential errors with high alert drugs.

Results indicate that the over-all usability of the iDC was high (91.1) as compared to a benchmark of 68. Participants detected errors equally with the PPC and the iDC but completed the calculation slightly faster with the iDC when the scenario contained an error. Participants preferred the iDC as compared to the PPC and were more confident performing the dose calculation without hesitation.

The iDC proved to be safe and effective as a support for drug calculations when used in a simulated clinical environment. The results of this study enabled us to plan a follow up field study to test the iDC in a pediatric emergency department and post surgical care unit.

Introduction

Technology in general, and computer-assisted computations in particular, can mitigate the disruptive impact of problematic work conditions, such as the interruptions and distractions that are a common occurrence in a busy clinical environment (Sharp, Preece, & Rogers, 2006). A computer application called iDoseCheck (iDC) was developed to assist nurses in the calculation and verification component of the double check for administering intravenous morphine (see Figure 1). The iDC was developed in collaboration with professionals from nursing, pharmacy, information technology and human factors engineering. The iDC provides nurses with several visual cues to ascertain the correct mg/kg dose of morphine and the correct volume of drug to be withdrawn from the morphine vial. Phase 1 testing of the iDC took place in a nursing simulation center with third and fourth year baccalaureate nursing students. Phase 2 testing will take place in a pediatric hospital on a post surgical unit and the emergency department. The focus of this paper is the testing and results of the simulation study.

 

 Background

Dosing errors are the most common type of pediatric drug errors, with over-dose outnumbering under-dose errors (Crowley, Williams, & Cousins, 2001; Kaushal, Bates, Landrigan et al., 2001; Lesar, Mitchell, & Sommo, 2006). Calculation errors, including 5 and 10-fold errors range from 6% to 15% of reported pediatric drug errors (Cousins, Sabatier, Begue, Schmitt, & Hoppe-Tichy, 2005; Ross, Wallace, & Payton, 2000). Morphine is the high alert drug most frequently implicated in pediatric drug errors (ISMP, 2009; Wong, Ghaleb, Franklin, & Barber, 2004). High alert drugs are those that bear a heightened risk of errors causing significant harm (ISMP, 2008).  Pediatric drug calculations are typically weight-based and nurses administering medications ascertain the accuracy and appropriateness of the dose using the patient’s weight (Conroy, Sweis, Planner, et al. (2007). Weight-based calculations are essential for proper dosing but complex in pediatric settings where patient weights may vary from 0.5 kg for a premature newborn to over 100 kg for an obese adolescent (American Academy of Pediatrics, 2003).

The independent double check has been proposed as an intervention that can effectively detect and avert medication errors at the point-of-care. The Institute for Safe Medication Practices (ISMP) defines the independent double check as a process whereby two nurses independently verify the patient, drug concentration, drug, dose, time and route. Research supports the importance of the independent double check, however, under ideal conditions it fails to detect 5% -10% (Grasha, 2000) of errors and under conditions of moderate stress this increases to 20% (Gertman, Blackman, Marble et al., 2005; System Reliability Center, 2005). The purpose of this study was to assess whether a graphic dose calculator, in comparison to standard paper and pencil/calculator calculations, can support the independent double check process and reduce the rate of potential errors with high alert drugs.

 METHODS

Design

The simulation experiment involved 18 paediatric case scenarios which focused on postoperative care and pain management. For each scenario the participant was directed to prepare a dose of IV bolus morphine. The iDC was used for the calculation for 9 scenarios and the standard paper and pencil/calculator (PPC) method was used for the remaining 9 scenarios. Six of the 18 scenarios had errors that were either an under-dose or an overdose, and 6 of the scenarios included interruptions to the process of dose calculation. The remaining 6 scenarios had no interruptions and no errors (control).

A within subjects crossover design was used with an experimental group (iDC) and a standard practice group (PPC), with three sets of conditions (errors, with interruptions, and control). A counter-balanced design was used to ensure each participant received the same scenarios with dosing errors, interruptions, or neither (control) in a different order. The dependent variables consisted of speed, error recognition, recovery after interruptions, system usability, perceived helpfulness, preference and confidence levels with the iDC versus the PPC method.

 Measures

Questionnaire-based Measures

            Demographic and background information.

Information about participant age, gender, and year of study in the nursing program was collected. The interface of the iDC uses blue, green and red to convey important dose information so participants were asked if they had problems distinguishing colours and if they were ever diagnosed with colour blindness. Further information was gathered about computer usage and degree of comfort with computer technology. Participants were asked about their degree of confidence in drug calculations, degree of anxiety with drug calculations, and perceived difficulty with drug calculations.

            System Usability Scale (SUS)

The SUS is a 10-item scale that assesses subjective perception of usability (Brooke, 1996). Questions address two key components: Usability and Learnability. Items are scored on a 5 point Likert scale with descriptors strongly disagree and strongly agree. The scores for each question are multiplied by 2.5 and final SUS score can range from 0 to 100, where higher scores indicate better usability. The SUS has good face validity and correlates highly (Cronbach’s alpha 0.91) with longer usability scales (Bangor, Kortum, & Miller, 2008; 2009).

            Perceived helpfulness of iDC features

A subjective measure of perceived helpfulness of the visual aids, features and functions of the iDC was developed by the study authors. The tool was based on a questionnaire addressing specific features and functions of the iDC on a scale from 1 (not helpful) to 5 (very helpful). Participants were asked to respond to 5 questions about the helpfulness of visual aids such as the height of the patient, standard dose scale, written calculation, error warnings, and images of the vial(s) and syringe. Two questions about ease of error recovery and the iDC as an accurate method to prepare IV bolus morphine were also scored on a 1 to 5 scale.

             Confidence

Participants’ confidence in the accuracy of drug dose calculations and preparation was measured with a 6-item questionnaire.  Responses were on a 5-point Likert scale and ranged from strongly disagree to strongly agree.

            Preference

Preference for the iDC or paper and pencil/calculator were ascertained with a 5 item questionnaire that focused on support for dose calculations specifically and support for the double check generally, speed, accuracy, and error detection. The response scale was coded 1 (preference for PPC), to 5 (preference for iDC).

Observation-based Measures

            Speed/efficiency

The experimenter activated a timer the moment the participant started reading the case scenario and stopped the timer when participants placed the morphine syringe on the medication administration tray indicating that it was ready to be administered to the patient. During error scenarios, the simulation ended when the participant detected the error and verbally stated that they needed to consult a physician about the prescribed dosage. However, if the error went undetected, the scenario ended when the participant placed the syringe on the medication tray.

            Error recognition

Six patient scenarios contained under or over-dose errors.  Participants were expected to detect these errors and choose not to administer the dosage to the patient until they consulted with the physician (similar to what they would do in a real clinical setting). The number of errors detected was recorded.

            Recovery from interruption

Six scenarios had interruptions which were introduced by the research assistant. While participants were engaged in the drug calculation they were interrupted with a question specific to their patient or the clinical environment (e.g. “When will you be free to assist the physiotherapist with your patient in room 4 bed 2?”). Participants were required to read a small text from a nearby booklet when interrupted. Afterwards the participants were allowed to continue on with their scenario. A score of 0 was noted if they returned to the calculation without obvious difficulty and 1 if they restarted their calculations after the interruption.

 Apparatus

           

The iDC was installed on two tablet computers to be used by the participant and the ‘double checker’. Information on the iDC interface is organized into three panels designed to be viewed from left to right. Patient weight and morphine dose are entered into two text boxes at the far left of the screen (refer to figure 1). The dose entered is compared to the standard dose which is based on .05 to .10 mg/kg of body weight (Lexicomop/Micromedix, 2011). The center panel provides information about the appropriateness of the dose as compared to the standard and about the size of the child based on the weight entered and compared to a background image of 180 cm. The third panel provides a visual image of the number of vials of morphine to use and a visual image of the correct amount of drug in the appropriate syringe. Participants did not receive training with the program and were given directions to simply use it to the best of their abilities

Participants were provided with a pencil, blank sheets of white paper and a calculator for the standard practice method of drug calculation. The formula for calculating the proper dose was posted on the wall directly in front of the medication preparation station.

 Setting

Data collection for this study took place in the nursing simulation lab which provided an environment similar to a hospital room with beds, cribs, manikins, automated medication dispensing cabinet and other medical equipment. Participants were provided with all the equipment and supplies needed to prepare a dose of IV bolus morphine which took place at the medication dispensing cabinet.

 Procedures

After approval of the study protocol by the University Research Ethics Board, students were recruited into the study. Upon arrival to the simulation lab, participants signed an informed consent and were administered the demographic questionnaire. Participants were then provided with a training scenario that served as a refresher on how to prepare IV bolus morphine and perform the double check. Study personnel included a psychology student who acted as the observer and the ‘interrupter’, and a nursing master’s student who acted as the ‘double checker’ and provided debriefing to the participant

Participants prepared the morphine dose just as they would in a clinical situation and the master’s student simulated performing the independent double check. When participants detected an error they would stop and provide instructions as to what should happen next (e.g. call the doctor or call the pharmacy) and the scenario was considered to be completed. If the scenario had an interruption, the observer introduced the interruption in the form of a question while the participant was performing the dose calculations. The participant moved away from the medication preparation station and consulted a booklet for the answer to the question then resumed drug preparation. Post-test questionnaires and debriefing was completed when all 18 scenarios were finished. The debriefing involved an overview of participants’ performance and feedback about any performance errors that may have occurred during the simulation in order to clarify misunderstandings and to take advantage of a ‘teachable moment’.

 RESULTS

Demographics

Twenty-nine nursing students participated in the study: 16 third year students (55.2%) and 13 fourth year students (44.8%) with an average age of 23 years. The sample consisted of 26 females (90%) and 3 males (10%). No participants reported being diagnosed with color blindness nor had difficulty distinguishing color. On a scale of 1 (very uncomfortable) to 5 (very comfortable) participants reported being very comfortable with computers (M = 4.34, SD = .67). Participants were relatively confident in their ability to perform calculations and scored on average 3.66 (SD = .81) on a 5-point scale which ranged from ‘not confident at all’ to ‘very confident’. Generally, participants did not find drug calculations difficult (M = 3.62, SD = .90) nor were they overly anxious (M = 3.41, SD = .83). Both questions had a 5-point response scale similar to previous questions.

 System Usability

The overall SUS score is scored out of 100 and has two components: usability and learnability. In addition, this SUS score is compared with a benchmark which indicates whether the score is above or below the average. By assessing 446 studies and over 5000 individual SUS responses in previous studies, a global benchmark of 68 was suggested by Sauro (2011). The over-all usability score for the iDC on the SUS was 91.1 (SD = 9.5). The two components were also examined, with 8 out of 10 questions assessing usability (M = 90.3, SD = 11.0) and 2 questions assessing learnability (M = 94.4, SD = 12.1). A one-sample t-test was used to compare the overall usability score with the benchmark value of 68. The overall usability score of the iDC was significantly above the benchmark, t (28) = 13.48, p< .001. There was no difference in SUS scores between third and fourth year students (p = .55).

 Helpfulness of iDC visuals

Visual aids such as the images of the patient’s height, the standard dose and therapeutic range, error warnings and the images of the vials and syringe were scored for helpfulness on a 1 to 5 scale and compared to a neutral score of 3. All components tested were rated significantly better than neutral (see Table 1).

 

 

 Confidence

Participants were equally confident using the iDC or the PPC method to do the calculation components of the double check. However, participants were more confident performing the calculation without hesitation when using the iDC as compared to the PPC method, t (28) = -3.88, p< .001, iDC (M = 4.17, SD = .76), PPC (M = 3.55, SD = .91).

 Preference

The preference questionnaire was analyzed using a one-sample t-test. When analyzing the preferred dose calculation method, there was an overall significant preference for the iDC (p< .001) relative to a neutral rating of no preference. Participants preferred the iDC as a general support for dose calculations, for the speed of dose calculations, as an accurate measure for dose calculations, as a support when detecting errors, and as a support during the double check (see Table 2).

 

 

Speed

A repeated measures ANOVA was used to compare the time it took to do the double check when using the iDC or the standard method. Over-all there was no difference between the two methods. The interaction effect for method (iDC versus PPC) and scenario type (error, interruptions, and control) approached significance, F(2, 54) = 3.145, p = .051. Specifically it was slightly faster to do the calculations using the iDC (M = .83, SE = .05) as compared to PPC (M = .99, SE = .08) when there were errors in the scenario.

 Error recognition

No significant differences were found when comparing the amount of errors detected when using the iDC versus PPC.

 Recovery from Interruption

Participants scored 0 if they needed to redo the calculation after the interruption and 1 if they did not need to start again. There was no difference in recovery post interruption when using the iDC versus the PPC method, F (1, 54) = .06, p = .81. The mean score for the iDC was .6 (SD = .50) and .62 (SD = .49) for the PPC. In addition there was no difference between third and fourth year students with respect to their ability to recover from interruptions.

 

DISCUSSION

A working prototype of the iDC, a graphic dose calculator, was designed to assist nurses with the calculation and verification components of preparing IV bolus morphine for pediatric patients. Integrating the iDC into the independent double check process represents a practice change with the potential to introduce error. Consequently, phase one testing was completed in a simulated patient setting to ensure that the iDC is at least as safe and effective as the standard method of drug calculation which is typically a calculator, paper and pencil. In addition, the study results provided us with information about the applicability of the iDC for use with nursing students.

The primary design principle of the iDC was to present the required information in a visual, graphic format and not rely solely on alphanumeric data items as would be seen on a calculator. The presentation of graphic information, in addition to accurate alphanumeric presentation, is known to support user tasks (Gurushanthaiah, Weiner, & England, 1995; Westenskow, 1997; Blike, Durgenor, & Whalen, 1999; Zhang, Drews, Westenskow, et al. 2002).This is particularly relevant in work contexts in which there are many distractions and interruptions, while the user is still required to perform the task accurately and efficiently. This visual graphic approach has been implemented in many other High Reliability Organizations such as aviation and the nuclear power industry. Aircraft cockpit displays and nuclear power control rooms incorporate multimodal display information to facilitate better monitoring, control, and decision making (Wickens, Lee, Liu, & Gordon-Becker, 2004).

Users responded favorably to the iDC over-all and to the specific graphical components of the interface. The over-all usability of the iDC as indicated by the SUS score (91.1/100) was quite high as compared to the benchmark of 68 (Sauro 2011). In addition, the learnability subscale score was high (94.4/100) which is of particular interest since participants were not trained to use the iDC. They were instructed to ‘talk aloud’ as they learned to use the iDC. Participant’s positive response to the iDC visuals were reflected both in their comments and in their scores on the helpfulness questionnaire. The specific visual components of the iDC were all rated by the participants as helpful.

A primary aim of the simulation study was to demonstrate that the iDC was equally as safe and accurate as the standard PPC method of calculating a pediatric morphine dose. Results of this study indicate there were no significant differences in the number of errors detected when comparing the iDC to the PPC. In addition, participants were more confident performing the calculation without hesitation using the iDC and the iDC proved to be more efficient as reflected in the shorter duration of scenarios with errors. The iDC is highly usable (‘user friendly’) and was preferred by participants over the standard PPC method of calculation. Taken together, these results indicate that the iDC is at least as safe and accurate as the standard method and is ready to be field tested in a clinical setting.

Furthermore, the importance of providing adequate support to nurses performing drug calculations was illustrated by the initial difficulties students experienced performing the independent double check. During the pilot testing it became clear that participants were uncomfortable or unable to calculate the morphine dose. In order to proceed with the study, extensive training related to calculating the mg/kg dose and the mg/ml morphine dose was needed. Students were unprepared to do the drug calculations, despite lab training and experience in a clinical setting with drug calculations and the independent double check.

Even when provided with the formula to use for the calculations many students were hesitant to proceed with the calculation. Other students calculated the mg/kg dose and then confused this with the mg/ml dose that is derived from the morphine concentration of 2mg/ml. A post-study debriefing was carried out to highlight errors and clarify the calculation steps.

The study involved 18 patient scenarios and the act of doing these calculations in succession over a 45-60 minute period was a useful exercise for the students. Since the students spoke aloud during the simulations we were able to gain insight into how they were problem solving, recovering from interruptions, detecting errors and generally gaining confidence, speed and comfort with all components of the independent double check.

  Strengths and limitations

The experimental design and setting for the study were strengths. By conducting the study in a nursing simulation lab, we were able to manipulate the independent double check by introducing errors and interruptions without comprising patient safety. Testing the usability and the attributes of the iDC in an environment similar to a healthcare setting provided insight into the usability and applicability of the tool in high stress environments where nurses work. Limitations include the potential for limited generalizability as the participants were students and had very limited experience with medication administration and preparation of high alert drugs. The usability of the iDC and the helpfulness of the particular components of the interface might be different when used by experienced nurses in a clinical environment.

The iDC proved to be safe and effective as a support for drug calculations when used in a simulated clinical environment. The results of this study enabled us to plan a follow up field study to test the iDC in a pediatric emergency department and post surgical care unit.

References

 American Academy of Pediatrics (2003). Policy statement: Prevention of medication errors in the pediatric inpatient setting. Pediatrics, 112(2), 431-436.

Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the System Usability Scale. International Journal of Human-Computer Interaction, 24(6), 574-594.

Bangor, A., Kortum, P. T., & Miller, J. T. (2009). Determining what individual SUS scores mean: adding an adjective rating scale. Journal of Usability Studies, 4(3), 114-123.

Blike, G., Surgenor, S. D., & Whalen, K. (1999). A graphic object display     improves an anesthesiologist’s performance in a simulated diagnostic  task. Journal of Clinical Monitoring, 15, 37–44.

Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & I. L. McClelland (Eds.), Usability  evaluation in industry (pp. 189–194). London: Taylor & Francis.

Conroy, S., Sweis, D., Planner, C., Yeung, V., Collier, J., Haines, L., & Wong,  I. C. (2007). Interventions to reduce dosing errors in children: a systematic  review of the literature. Drug Safety, 30, 1111-1125.

Cousins, D. H., Sabatier, B., Begue, D., Schmitt, C., & Hoppe-Tichy, T. (2005). Medication errors in intravenous drug preparation and administration: a multicentre audit in the UK, Germany and France. Quality and Safety in Health Care, 14, 190-195.

Crowley, E., Williams, R., & Cousins, D. (2001). Medication errors in children: a descriptive summary of medication error reports submitted to the United States Pharmacopeia. Current Therapy and Research, 26, 627-640.

Gertman, D., Blackman, H. Marble, J. et al. (2005). The SPAR-H human reliability analysis method. Prepared for the Division of Risk Analysis and Applications, Office of Nuclear Regulatory Research, U.S. Nuclear Regulatory Commission, Washington, D.C.

Grasha, A. A. (2000). A cognitive systems perspective on human performance in the pharmacy: implications for accuracy, effectiveness and job satisfaction. Executive Summary Report, Report No. 062100, Alexandria, VA: National Association of Chain Drug Stores.

Gurushanthaiah, K., Weinger, M. B., & Eglund, C. E. (1995). Visual display  format affects the ability of anesthesiologists to detect acute physiologic  changes. Anesthesiology, 83, 1184–1193.

ISMP (2008). ISMP’s list of high-alert medications. Retrieved from http://www.ismp.org/Tools/highalertmedications.pdf

ISMP (2009). National collaborative: Top 5 drugs reported as causing harm through medication error in paediatrics. ISMP Canada Safety Bulletin,  9(6), 1-4.

Kaushal, R., Bates, D. W., Landrigan, C. et al. (2001). Medication errors and adverse drug events in pediatric inpatients. JAMA, 285, 2114-2120.

Lesar, T. S., Mitchell, A., & Sommo, P. (2006). Medication safety in critically ill children. Clinical Pediatric Emergency Medicine, 7, 215-225.

Lexicomop/Micromedix, and CPS Online – Lexicomp, retrieved September  2011.

Ross, L. M., Wallace, J., & Payton, Y. (2000). Medication errors in a pediatric teaching hospital in the UK: five years of operational experience. Archives of Disease in Childhood, 83(6), 492-497.

Sauro, J. (2011). A practical guide to the system usability scale (SUS).  Measuring usability, Denver, USA

Sharp, H., Preece, J., & Rogers, Y. (2006). Interaction Design: Beyond human-computer interaction. Hoboken, NJ: John Wiley & Sons, Ltd.

System Reliability Center (2005). Technique for human error rate prediction (THERP). Rome, NY: Alion Science and Technology.

Westenskow, D. R. (1997). An integrated graphic display improves detection and identification of critical events during anesthesia. Journal of Clinical Monitoring, 13, 249–259.

Wickens, C. D., Lee, J. D., Liu, Y., & Gordon-Becker, S. (2004). An introduction to human factors engineering. Upper Saddle River, NJ: Pearson Prentice Hall.

Wong, I. C., Ghaleb, M. A., Franklin, B. D., & Barber, N. (2004). Incidence and nature of dosing errors in paediatric medications: a systematic review. Drug Safety, 27, 661-70.

Zhang, Y., Drews, F. A., Westenskow, D. R., Foresti, S., Agutter, J., Bermudez,    J. C., Blike, G., & Loeb, R. (2002). Effects of Integrated Graphical Displays on Situation Awareness in Anaesthesiology. Cognition, Technology &    Work, 4, 82–90.

 Author Details

Jacqueline Ellis, RN, PhD, University of Ottawa, School of Nursing and Children’s Hospital of Eastern Ontario, Chronic Pain Service jellis@uottawa.ca

Avi Parush, PhD, Carleton University, Department of Psychology, avi_parush@carleton.ca

Monika Taing, BA, Carleton University, Department of Psychology, mtaing@connect.carleton.ca

Catherine Campbell, P.Eng. M.Des, Children’s Hospital of Eastern Ontario, Information Systems Department, ccampbell@cheo.on.ca

Charissa Chiu, RN, MScN, University of Ottawa, School of Nursing, cchiu044@uottawa.ca

Susan Eldred, RN, MScN, University of Ottawa, School of Nursing, seldred@uottawa.ca

Anthony Whitehead, PhD, Carleton University, School of Computer Science, Anthony_Whitehead@carleton.ca

Danica Irwin, B. Pharm, Children’s Hospital of Eastern Ontario, Pharmacy, Irwin@cheo.on.ca

Regis Vaillancourt, Pharm D, FCSHP, Children’s Hospital of Eastern Ontario, Director of Pharmacy, rvaillancourt@cheo.on.ca

Brenda Martelli, RN(EC) MEd, Children’s Hospital of Eastern Ontario, Integrated Pain Service, martelli@cheo.on.ca

Debby Voskamp, RN, BScN, Children’s Hospital of Eastern Ontario, Emergency Department, dvoskamp@cheo.on.ca

Pat Elliott-Miller, RN, MScN, Children’s Hospital of Eastern Ontario, VP Patient Services and Chief Nurse Executive, pelliottmiller@cheo.on.ca

Ken Farion MD, FRCPC, Children’s Hospital of Eastern Ontario, Emergency department, farion@cheo.on.ca

 CJNI Editors

• Mary Eileen MacPhail
• Paulette Lacroix
• Teresa Birznieks