Dennis T. Costakos, MD, Mengyi Zha, MD, Lynn R. Dahlen, RN

Abstract:

The present study evaluated end-user experience for effectiveness, efficiency, and satisfaction of a mobile application (app) to estimate the risk of rebound hyperbilirubinemia within 72 hours after initial phototherapy ends for infants at 35 weeks’ gestation or longer and age younger than 14 days. This prospective parallel study was performed at one medical center. The 26 participants are neonatology and pediatric and family physicians, neonatal and family nurse practitioners, and family practice residents, who were asked to estimate the risk of rebound hyperbilirubinemia when phototherapy is stopped for a fictional newborn. Participants in Group 1 (no app) were compared with Group 2 (with app). Satisfaction measured from 1 (0%) to 7 (100%). Group 1 had fewer correct answers than Group 2 (38.5% vs. 84.6%; P=.04). The mean satisfaction score for Group 1 was 2, lower than the score for Group 2, 6.6 (P<.001). This provider-developed app is effective and receives high end-user satisfaction.

Keywords: jaundice; neonate; phototherapy; predictive; smartphone

Introduction:

Worldwide, hyperbilirubinemia is a leading cause of neonatal morbidity and death.(1,2) The treatment goal of neonatal jaundice is the prevention of kernicterus, and newborn phototherapy is effective. Every treatment has associated adverse events, however, and this includes phototherapy of newborns for hyperbilirubinemia. Phototherapy can disrupt breastfeeding and infant-mother bonding. Hospitalization for phototherapy can be burdensome for families. Emerging evidence shows that DNA damage with phototherapy may cause melanocytic nevi and, potentially, infantile cancer.(3,4) Newborns who receive phototherapy have a small increased risk of childhood seizures, noted even after adjusting for bilirubin levels, at 4 per 1,000 excess risks over ten years.(5)

In a large study, Chang et al(6) investigated the timing of stopping f irst-time inpatient newborn phototherapy for late-preterm and term infants younger than 14 days. They reported that “34% of infants may have been able to discontinue inpatient phototherapy a day earlier with < 4% risk of rebound hyperbilirubinemia.”

In the United States and the United Kingdom, 6% to 9% of babies at 35 weeks’ gestation or more are treated with phototherapy, but hyperbilirubinemia and the use of phototherapy are a global phenomenon. We believe that optimizing the length of initial inpatient phototherapy by doing an evidence-based estimate of bilirubin rebound is especially important not to over- or undertreat babies with phototherapy. Hence, estimation is important for the risk of rebound hyperbilirubinemia if phototherapy is discontinued to help decrease the need for over- or undertreatment.(7) This may be even more important during the time of coronavirus disease 2019 (COVID-19) because risk assessment of hyperbilirubinemia rebound has implications for hospital length of stay and readmission of babies.

Chang et al(6) evaluated a large cohort of 7,048 infants treated within patient phototherapy from a population of 105,808 infants from the Kaiser Permanente Northern California hospitals and addressed the timing for stopping phototherapy for babies at 35 weeks’ gestation or more and before 14 days of age. The derived algorithm was based on the three clinical variables that best predicted the risk of rebound hyperbilirubinemia that would require further treatment within 72 hours if the initial phototherapy is stopped: the age when phototherapy is initiated, gestational age, and the American Academy of Pediatrics (AAP) threshold serum bilirubin (TSB) for phototherapy at the age when phototherapy is discontinued. The algorithm generates a score, corresponding to a percentage of risk of rebound. The score performed well, as evidenced by the receiver operating characteristics. Score = 15 (if gestational age is <38 weeks) – (7 × age [days at initiation]) –(4 × [AAP threshold TSB – infant’s TSB at discontinuation of phototherapy]) + 50.

In our experience, we have found it cumbersome and time-consuming to calculate this score without a digital tool in real-time in the nurseries. Thus, we, as health care providers, developed a mobile application (app), with no malfunctions or crashes, to help clinicians perform this calculation. We created the app after a search for an available webbased or mobile app. We knew that Chang et al(6) recommended that a web-based calculator be created, but we did not find one in the Apple App Store or through Google (Alphabet Inc), and Bing (Microsoft Corp) searches. The purpose of the present study was to evaluate end-user experience for effectiveness, efficiency, and satisfaction of the app, which provides decision support and does the calculation to estimate the risk of rebound hyperbilirubinemia if phototherapy is stopped for infants born at 35 weeks’ gestation or greater and before age 14 days.

Methods:

The Mayo Clinic Institutional Review Board approved this minimal risk, prospective, parallel study. We invited pediatricians, neonatologists, neonatal, and family nurse practitioners, family physicians, and resident physicians and providers who worked for Mayo Clinic and made decisions about stopping phototherapy to participate in this study. For the study, they were asked to estimate the risk of rebound hyperbilirubinemia if phototherapy was stopped in a published, fictional newborn case.(8)

The fictional newborn is described as being born at 37 weeks’ gestation and weighing 3,150 g. The infant is in the well nursery and receives phototherapy at 30 hours of age because of a TSB of 11 mg/dL. At 48 hours of age, the TSB is 10 mg/dL, and the mother is to be discharged from the hospital. No blood group incompatibility is known (mother has blood type A+). The infant is breastfeeding well, has had acceptable urine and stool output for age, and the mother believes her milk is coming in. Question 8 in the published article asks, “Of the following if phototherapy is discontinued at this time, the predicted risk of rebound hyperbilirubinemia to a concentration meeting the criteria for the reinitiation of phototherapy is CLOSEST to: A. 5%, B. 10%, C. 20%, or D. 40%?” The correct answer is D. Historically, the control respondents answered this question correctly 14% of the time.

In the present study, all-volunteer participants read a peer review paper about the phototherapy stop algorithm and viewed a brief (1 minute 52 seconds) video of the new app. The evaluated app was developed by two investigators (D.T.C. and M.Z.) in 2018 using Xcode (Apple Inc). The app has a user-friendly interface (i.e., what the user sees on the screen of a smartphone or digital tablet), and the user interacts by entering patient clinical data step by step as prompted by the app. A response (percentage risk of rebound admission) based on the algorithm published in the study by Chang et al(6) is generated for decision support.

This app does not ask for the name, sex, or health record number of the patient. It does not identify the patient or the user, and no personal information is stored.

The first interaction of the app has the user confirm answers to 5 patient-related questions with the use of a quick checkoff list (Figure 1). The app user advances to the next screen if the answers to the checkoff list are “yes” and enters the answers for four additional questions (Figure 2), including a query about AAP’s threshold for initiating phototherapy. The AAP threshold is calculated with the free website (https://www.bilitool.org or https:// peditools.org/bili/) in mg per dL or mmol per L. (The app does not provide linkage to either website.) These data are often readily available for the clinician and sometimes provided by nursing staff.

Additionally, clinicians commonly use hours as the unit for a newborn’s age. However, the algorithm by Chang et al(6) asks for fractional days as the input. This entry can create errors. For end-users, our app converts the more familiar and readily available unit of hours to the required unit of the algorithm, thereby reducing the risk of errors.

After the user has entered the data, the app displays a risk percentage of rebound hyperbilirubinemia within 72 hours (Figure 3). This answer is generated quickly, 22 seconds from start to f inish of all data.

In the present study, participants were entered sequentially. Control Group 1 were the odd-numbered participants, and Group 2, the even-numbered participants. The 2 groups differed only in that Group 2 used the app as a simulation on a laptop computer.

The groups were compared for the proportion of correct answers and time to the correct answer. No provider in the study had ever used this app. Provider satisfaction was measured on a Likert scale of 1 to 7 (1, 0% satisfaction; 7, 100% satisfaction).

In our statistical analysis, the t-test was used for continuous variables. χ2 tests and Fisher exact tests were used for categorical variables. A P value less than or equal to .05 was considered statistically significant. Sample size calculations predicted that 20 volunteers would be needed for the study, with the assumption that the chance was 80% for missing the difference if Group 2 (with the app) answered correctly 90% of the time and Group 1 answered correctly 20% of the time.(9)

Results:

Twenty-six volunteers (average work experience, 16.4 years) participated in the study. In Group 1, the average work experience was 18.7 years vs. 12.2 years in Group 2. This difference was not statistically significant (P=.13, t-test) (Table). Group 1 had proportionally fewer correct answers (Figure 4). Time to the correct answer was 6.67 minutes for Group 1 vs. 4.89 minutes for Group 2 (P=.55, t-test). The mean satisfaction score was 2 for Group 1 vs. 6.6 for Group 2 (P<.001) (Figure 5).

Table: Professional Roles of the 26 Volunteer Study Participants

Provider Role	Participants, No. (%)
Neonatologist	1 (4)
Neonatal nurse practitioner	5 (19)
Family nurse practitioner	1 (4)
Pediatrition	7 (27)
Family physician	7 (27)
Pediatrician	5 (19)

Legends Discussion:

The 100 most popular medical mobile apps, paid or free, in the Google and Apple stores cover 27 medical topics.(10) However, 44% of the apps did not provide the content’s source in their provision of medical information to their target users. More than half of the apps did not provide general terms and conditions of use. Only 56% of the apps that collect user health data require the explicit consent of their users’ precollection.(10)

In addition, because end users do not always have a privacy policy or context source, they must always wonder whether accuracy or effectiveness has been studied. The authors of 1 study tested the accuracy of 2 oxygen saturation, smartphoneintegrated baby monitors not regulated by the US Food and Drug Administration (FDA) compared with an FDA-regulated pulse oximeter.(11) The authors reported that one monitor detected hypoxemia but performed inconsistently. The other monitor never detected hypoxemia and displayed pulse rates that were lower than a reference monitor. The authors of that study advise that physicians and parents should exercise caution when incorporating data from the smartphone-integrated consumer baby monitors that they tested. This outcome is in contrast to a different study of smartphone apps and wearable devices for tracking physical activity data.(12) The investigators observed that many of the devices were accurate for tracking step counts.

We believe that good medical and health mobile apps will transform health care delivery. Similar to food in the United States, medical and health mobile apps might have a label that grades safety, evidence basis, and app interoperability in the future.(13) Therefore, in the development of our app, we focused its design on ease of use, prevention of errors, maintenance of user and patient privacy, and, most importantly, evidence-based conclusions and high end-user satisfaction.(14-16)

In the present study of our app, we found that the number needed to treat (NNT) is 2.2 for one infant to benefit from its use.(17) In a cohort study of 204,485 infants, the NNT through the use of the Kaiser Permanente online early-onset sepsis calculator was 41 infants compared with the infants who received no antibiotics in the first 24 hours if their provider used the Kaiser sepsis app. (18) This was the primary question of that study. Its secondary question asked about a reduction in blood cultures in the first 24 hours. With the use of the Kaiser sepsis calculator, the NNT was ten infants. The providers who used our app had never seen the mobile app before. We speculate that if we allowed them to practice with our app, the proportion of correct answers could be close to 100%, thus decreasing the NNT even more. Our data showed that the use of the app did not improve the time to a correct answer significantly. However, familiarity with the app will likely change that outcome.

In 2020, we did find a free web-based rebound bilirubin calculator through the Bing search engine (https://jscalc.io/ calc/68NNiFfS7iTMZhZY). This free calculator is also based on the study by Chang et al(6) and uses the same algorithm. It worked well when answers were compared with the fictional patients to the mobile medical app that we created. However, no reminders (i.e., decision support) are provided that warn users not to use the calculator for babies born at less than 35 weeks’ gestation, that the baby in question should be age 14 days or less, and that this is the first time phototherapy is not reinitiated for the baby. In addition, physicians should make sure the privacy policy of the app is clear when they enter patient data into a mobile app.

In our opinion, the next step for our app should be the integration of its use into electronic health records. Although mobile apps are a relatively new development in medicine and health, they are used more and more frequently in medical practice. For instance, the Kaiser early-onset sepsis calculator is routinely used in the assessment of the risk of neonatal sepsis in many medical centers. (19-21) In fact, some centers have integrated it into electronic health records because it has helped reduce neonatal antibiotic treatment with no adverse events.(22) We believe our app also can benefit patients through its evidence-based design and our current study that found it effective with high end-user satisfaction. At the minimum, the provider is more efficient, and the digital tool should perform the math quicker, allowing easy checking. On many days, more than one baby on rounds will benefit in a busy newborn practice.(23)

We conclude that our mobile medical app and this study demonstrated that providers have an important role in transforming medical care with the leveraging of digital tools and data and in checking the quality of any medical software.(24) We were not surprised that—after about 5 months to make our mobile app, a year to design this study, the time to receive Mayo Clinic Institutional Review Board approval, and then the time to execute this study and interpret the study results—a separate physiciancreated, web-based bilirubin rebound calculator appeared online by a different group. This development indicates the importance of the algorithm in clinical practice.(22,25) The new digital tool week we developed and tested as a mobile application was made available in the Apple store as BiliStop on June 12, 2020.

References:

1. Sankar MJ, Natarajan CK, Das RR, Agarwal R, Chandrasekaran A, Paul VK. When do newborns die? A systematic review of timing of overall and cause-specifc neonatal deaths in developing countries. J Perinatol. 2016;36 Suppl 1:S1-S11.
2. Olusanya BO, Teeple S, Kassebaum NJ. The contribution of neonatal jaundice to global child mortality: fndings from the GBD 2016 study. Pediatrics. 2018;141(2).
3. Newman TB, Wickremasinghe AC, Walsh EM, Grimes BA, McCulloch CE, Kuzniewicz MW. Retrospective cohort study of phototherapy and childhood cancer in northern California. Pediatrics. 2016;137(6).
4. Wickremasinghe AC, Kuzniewicz MW, Grimes BA, McCulloch CE, Newman TB. Neonatal phototherapy and infantile cancer. Pediatrics. 2016;137(6).
5. Newman TB, Wu YW, Kuzniewicz MW, Grimes BA, McCulloch CE. Childhood seizures after phototherapy. Pediatrics. 2018;142(4).
6. Chang PW, Kuzniewicz MW, McCulloch CE, Newman TB. A clinical prediction rule for rebound hyperbilirubinemia following inpatient phototherapy. Pediatrics. 2017;139(3).
7. Rennie JM, Beer J, Upton M. Learning from claims: hyperbilirubinaemia and kernicterus. Arch Dis Child Fetal Neonatal Ed. 2019;104(2):F202-F204.
8. NeoReview Plus; Question 8. https://pedialink.aap.org/ped?lens=ped. Published 2017. Accessed.
9. Hulley SB, Cummings SR, Browner WS, Hearst N, Grady D, Newman TB. Designing clinical research : an epidemiologic approach. 2nd ed. Philadelphia: Lippincott Williams & Wilkins; 2001.
10. Fougerouse PA, Yasini M, Marchand G, Aalami OO. A crosssectional study of prominent US mobile health applications: evaluating the current landscape. AMIA Annu Symp Proc. 2017;2017:715-723.
11. Bonafde CP, Localio AR, Ferro DF, et al. Accuracy of pulse oximetry-based home baby monitors. JAMA. 2018;320(7):717-719.
12. Case MA, Burwick HA, Volpp KG, Patel MS. Accuracy of smartphone applications and wearable devices for tracking physical activity data. JAMA. 2015;313(6):625-626.
13. Bates DW, Landman A, Levine DM. Health apps and health policy: what is needed? JAMA. 2018;320(19):1975-1976.
14. Nundy S, Montgomery T, Wachter RM. Promoting trust between patients and physicians in the era of artifcial intelligence. JAMA. 2019.
15. Wallis CJD, Detsky AS, Fan E. Establishing the efectiveness of procedural interventions: the limited role of randomized trials. JAMA. 2018;320(23):2421-2422.
16. Gawande A. Why doctors hate their computers. The New Yorker 2018(November 12).
17. Saver JL, Lewis RJ. Number needed to treat: conveying the likelihood of a therapeutic efect. JAMA. 2019;321(8):798-799.
18. Kuzniewicz MW, Puopolo KM, Fischer A, et al. A quantitative, isk-based approach to the management of neonatal
    early-onset sepsis. JAMA Pediatr. 2017;171(4):365-371.
19. Puopolo KM, Benitz WE, Zaoutis TE, Committee On F, Newborn, Committee On Infectious D. Management of neonates born at >/=35 0/7 weeks’ gestation with suspected or proven early-onset bacterial sepsis. Pediatrics. 2018;142(6).
20. Puopolo KM, Lynfeld R, Cummings JJ, Committee On F, Newborn, Committee On Infectious D. Management of infants at risk for Group B streptococcal disease. Pediatrics. 2019;144(2).
21. Goel N, Shrestha S, Smith R, et al. Screening for early onset neonatal sepsis: NICE guidance-based practice versus projected application of the Kaiser Permanente sepsis risk calculator in the UK population. Arch Dis Child Fetal Neonatal Ed. 2020;105(2):118-122.
22. Stipelman CH, Smith ER, Diaz-Ochu M, et al. Early-onset sepsis risk calculator integration into an electronic health record in the nursery. Pediatrics. 2019;144(2).
23. Costakos DT. Of lobsters, electronic medical records, and neonatal total parenteral nutrition. Pediatrics. 2006;117(2):e328-332.
24. Costakos DT, Love LA, Kirby RS. The computerized perinatal database: are the data reliable? Am J Perinatol.
    1998;15(7):453-459.
25. Liu Y, Chen PC, Krause J, Peng L. How to read articles that use machine learning: users’ guides to the medical literature. JAMA. 2019;322(18):1806-1816.

Acknowledgements:

We acknowledge the work of Carolyn R. Flock, BS, who helped assist with the Mayo Clinic Institutional Review Board application and Dianne M. Kreibich, who assisted with manuscript preparation.

Declaration of Conflicts of Interest:

DTC and MZ are the developers and owners of the new mobile medical app (BiliStop) discussed in this manuscript and studied. The authors acknowledge that they are the owners of the limited liability company Evergreen Bioinformatics, formed to allow further development of the mobile app by making it available to other health care providers

Funding:

The authors received no financial support for the research, authorship, and/or publication of this article.

Author Contributions:

DTC: conceptualized and designed the study, coordinated and supervised data collection, participated in data collection and data analysis, and critically reviewed the manuscript for important intellectual content. Dr Costakos has approved the final manuscript as submitted and agrees to be accountable for all aspects of the work.

MZ: designed the data collection instruments, collected data, carried out the initial analyses, and reviewed and revised the manuscript. Dr Zha has approved the final manuscript as submitted and agrees to be accountable for all aspects of the work.

LRD: designed the data collection instruments, collected data, carried out the initial analyses, and reviewed and revised the manuscript. Ms. Dahlen has approved the final manuscript as submitted and agrees to be accountable for all aspects of the work.

Presented at the Southern Regional Meeting of the American Federation for Medical Research, New Orleans, Louisiana, February 15, 2020, and published in abstract form: J Investig Med. 2020;68(2):656.

Potential Conflict of interest: Dennis T. Costakos MD and Mengyi Zha, MD, have intellectual property rights to the BiliStop as owners of Evergreen Bioinformatics LLC, one of the digital tools discussed in this paper. The BiliStop is sold in the Apple Inc. app store by Apple Inc. and Evergreen Bioinformatics LLC.