REINFORCEMENT LEARNING IN CARDIOVASCULAR THERAPY PROTOCOL: A NEW PERSPECTIVE
Abstract
Two examples of how RL, a technique in cardiovascular therapy that UK scientists are studying to enhance existence and lessen its number of negative effects, could advance therapy plans and momentous decision-making. In this paper, an attempt is made to look at the application of RL in cardiovascular medicine and how and in what ways it can be used to define the need for the treatment plan molding and drug dosage optimization as well as the patient outcome predictions based on actual-time data. Moreover, application of RL in Cardiovascular Disordered Increases patient rehabilitation utilizing other AI approaches like Computer vision and Predictive Analytics. However, there is a list of challenges that need to be addressed before this technology may be implemented in clinical practice such as data quality, interpretability and integration with the existing workflow. To ensure that the use of AI in healthcare is safe and permits fairness in its application some of the ethical issues include: informed permission, data privacy, and regulation as well as accountability for bias in algorithms. Potential future possibilities of RL in cardiovascular therapy include higher patient benefits, real-life adjustment of treatments plans, and advancements in algorithm design. Making sense of RL’s potential and creating the environment to close the gap between the individual patient’s, efficient, and equitable cardiovascular treatment will involve overcoming its ethical and technological challenges.
References
Liu V, Escobar GJ, Greene JD, Soule J, Whippy A, Angus DC, et al. Hospital deaths in patients with sepsis from 2 independent cohorts. Jama. 2014; 312(1):90–92. https://doi.org/10.1001/jama.2014. 5804 PMID: 24838355
Rhee C, Dantes R, Epstein L, Murphy DJ, Seymour CW, Iwashyna TJ, et al. Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014. Jama. 2017; 318(13):1241–1249. https://doi.org/10.1001/jama.2017.13836 PMID: 28903154
Paoli CJ, Reynolds MA, Sinha M, Gitlin M, Crouser E. Epidemiology and Costs of Sepsis in the United States-An Analysis Based on Timing of Diagnosis and Severity Level*. Critical Care Medicine. 2018; 46 (12):1889–1897. https://doi.org/10.1097/CCM.0000000000003342 PMID: 30048332
Marik P. The demise of early goal-directed therapy for severe sepsis and septic shock. Acta Anaesthesiologica Scandinavica. 2015; 59(5):561–567. https://doi.org/10.1111/aas.12479 PMID: 25656742
R. Pirracchio, M. L. Petersen, M. Carone, M. R. Rigon, S. Chevret, and M. J. v. d. Laan, “Mortality prediction in intensive care units with the super ICU learner algorithm (SICULA): A population-based study,” Lancet Respir. Med., vol. 3, pp. 42–52, Jan. 2015.
Arif, A., Khan, A., & Khan, M. I. (2024). Role of AI in Predicting and Mitigating Threats: A Comprehensive Review. JURIHUM: Jurnal Inovasi dan Humaniora, 2(3), 297-311.
L. A. Celi, R. G. Mark, D. J. Stone, and R. A. Montgomery, ““Big Data” in the intensive care unit. Closing the data loop,” Amer. J. Respir. Crit. Care Med., vol. 187, pp. 1157–1160, Jun. 2013.
B. Gholami, W. M. Haddad, and J. M. Bailey, “AI in the ICU: In the intensive care unit, artificial intelligence can keep watch,” IEEE Spectr., vol. 55, no. 10, pp. 31–35, Oct. 2018.
D. Shillan, J. A. C. Sterne, A. Champneys, and B. Gibbison, “Use of machine learning to analyse routinely collected intensive care unit data: A systematic review,” Crit. Care, vol. 23, pp. 1–11, Aug. 2019.
Khan, M. I., Arif, A., & Khan, A. R. A. (2024). AI-Driven Threat Detection: A Brief Overview of AI Techniques in Cybersecurity. BIN: Bulletin Of Informatics, 2(2), 248-261.
J. C. Y. Seah, J. S. N. Tang, A. Kitchen, F. Gaillard, and A. F. Dixon, “Chest radiographs in congestive heart failure: Visualizing neural network learning,” Radiology, vol. 290, pp. 514–522, Feb. 2019.
L. Li et al., “Using artificial intelligence to detect COVID-19 and community-acquired pneumonia based on pulmonary CT: Evaluation of the diagnostic accuracy,” Radiology, vol. 296, pp. E65–E71, Aug. 2020
L. Chen, O. Ogundele, G. Clermont, M. Hravnak, M. R. Pinsky, and A. W. Dubrawski, “Dynamic and personalized risk forecast in step-down units. Implications for monitoring paradigms,”Ann. Amer. Thoracic Soc., vol. 14, pp. 384–391, Mar. 2017.
S. Nemati, A. Holder, F. Razmi, M. D. Stanley, G. D. Clifford, and T. G. Buchman, “An interpretable machine learning model for accurate prediction of sepsis in the ICU,” Crit. Care Med., vol. 46, pp. 547–553, Apr. 2018.
C. W. Seymour et al., “Derivation, validation, and potential treatment implications of novel clinical phenotypes for sepsis,” JAMA, vol. 321, pp. 2003–2017, 2019.
C. S. Calfee et al., “Acute respiratory distress syndrome subphenotypes and differential response to simvastatin: Secondary analysis of a randomised controlled trial,” Lancet Respir. Med., vol. 6, pp. 691–698, Sep. 2018.
S. Liu, K. C. See, K. Y. Ngiam, L. A. Celi, X. Sun, andM. Feng, “Reinforcement learning for clinical decision support in critical care: Comprehensive review,” J. Med. Internet Res., vol. 22, Jul. 2020, Art. no. e18477.
W.-H. Weng, M. GAO, Z. He, S. Yan, and P. Szolovits, “Representation and reinforcement learning for personalized glycemic control in septic patients,” 2017. [Online]. Available: http://arxiv.org/abs/1712.00654
N. Prasad, L. Cheng, C. Chivers, M. Draugelis, and B. E. Engelhardt, “A reinforcement learning approach to weaning of mechanical ventilation in intensive care units,” 2017, arXiv:1704.06300.
D. Lopez-Martinez, P. Eschenfeldt, S. Ostvar, M. Ingram, C. Hur, and R. Picard, “Deep reinforcement learning for optimal critical care pain management with morphine using dueling double-deep Q networks,” in Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., 2019, pp. 3960–3963.
A. Raghu, M. Komorowski, L. A. Celi, P. Szolovits, and M. Ghassemi, “Continuous state-space models for optimal sepsis treatment: A deep reinforcement learning approach,” in Proc. 2nd Mach. Learn. Healthcare Conf., 2017, pp. 147–163.
M. Komorowski, L. A. Celi, O. Badawi, A. C. Gordon, and A. A. Faisal, “The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care,” Nature Med., vol. 24, pp. 1716–1720, Oct. 2018.
B. K. Petersen et al., “Deep reinforcement learning and simulation as a path toward precision medicine,” J. Comput. Biol., vol. 26, pp. 597–604, Jun. 2019. [20] S. Dugani, J. Veillard, and N. Kissoon, “Reducing the global burden of sepsis,” Can. Med. Assoc. J., vol. 189, pp. E2–E3, Jan. 2017.
C. M. Torio and B. J. Moore, “National inpatient hospital costs: The most expensive conditions by payer, in Healthcare Cost and Utilization Project (HCUP) Statistical Briefs [Internet], Rockville MD, USA: Agency for Healthcare Research and Quality (US), Feb. 2006. [Online]. Available: https://www.ncbi.nlm.nih.gov/books/NBK368492/
J. E. Gotts and M. A. Matthay, “Sepsis: Pathophysiology and clinical management,” BMJ, vol. 353, May 2016, Art. No. i 1585.
M. M. Levy, L. E. Evans, and A. Rhodes, “The surviving sepsis campaign bundle: 2018 update,” Intensive Care Med., vol. 44, pp. 925–928, Apr. 2018.
M. Radovanovic, A. Nanopoulos, and M. Ivanovic, “Hubs in space: Popular nearest neighbors in high-dimensional data,” J. Mach. Learn. Res., vol. 11, no. 86, pp. 2487–2531, 2010.
A. E. Johnson et al., “MIMIC-III, a freely accessible critical care database,” Scientif Data, vol. 3, May 2016, Art. No. 160035.
R. P. Dellinger et al., “Surviving sepsis campaign: International guidelines for management of severe sepsis and septic shock,” Intensive Care Med., vol. 39, pp. 165–228, Jan. 2013. DRUDI et al.: REINFORCEMENT LEARNING MODEL FOR OPTIMAL TREATMENT STRATEGIES IN INTENSIVE CARE 815
Khan, M. I., Arif, A., & Khan, A. (2024). AI's Revolutionary Role in Cyber Defense and Social Engineering. International Journal of Multidisciplinary Sciences and Arts, 3(4), 57-66.
E. P. Raith et al., “Prognostic accuracy of the SOFA score, SIRS criteria, and qSOFA score for in-hospital mortality among adults with suspected infection admitted to the intensive care unit,” JAMA, vol. 317, Jan. 2017, Art. no. 290.
AF. S. V. d. Oliveira et al., “Positive fluid balance as a prognostic factor for mortality and acute kidney injury in severe sepsis and septic shock,” J. Crit. Care, vol. 30, pp. 97–101, Feb. 2015.
J. Waechter et al., “Interaction between fluids and vasoactive agents on mortality in septic shock,” Crit. Care Med., vol. 42, pp. 2158–2168, Oct. 2014
Davenport EE, Burnham KL, Radhakrishnan J, et al: Genomic landscape of the individual host response and outcomes in sepsis: A prospective cohort study. Lancet Respir Med 2016; 4:259–271
Arif, A., Khan, M. I., & Khan, A. (2024). An overview of cyber threats generated by AI. International Journal of Multidisciplinary Sciences and Arts, 3(4), 67-76.
Orde S, Slama M, Yastrebov K, et al; College of Intensive Care Medicine of Australia and New Zealand [CICM] Ultrasound Special Interest Group [USIG]: Subjective right ventricle assessment by echo qualified intensive care specialists: Assessing agreement with objective measures. Crit Care 2019; 23:70
Shimabukuro DW, Barton CW, Feldman MD, et al: Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: A randomised clinical trial. BMJ Open Respir Res 2017; 4:e000234
Adams R, Henry KE, Sridharan A, et al: Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nat Med 2022; 28:1455–1460
Boussina A, Shashikumar SP, Malhotra A, et al: Impact of a deep learning sepsis prediction model on quality of care and survival. NPJ Digit Med 2024; 7:14
Catling FJR, Nagendran M, Festor P, et al: Data-driven decision support for individualised cardiovascular resuscitation in sepsis: A scoping review and primer for clinicians. medRxiv 2023.08.26.23294666
Lin P-C, Huang H-C, Komorowski M, et al: A machine learning approach for predicting urine output after fluid administration. Comput Methods Programs Biomed 2019; 177:155–159
Tang Y, Brown S, Sorensen J, et al: Reduced rank least squares for real-time short term estimation of mean arterial blood pressure in septic patients receiving norepinephrine. IEEE J Transl Eng Health Med 2019; 7:4100209
DeMerle KM, Angus DC, Baillie JK, et al: Sepsis subclasses: A framework for development and interpretation. Crit Care Med 2021; 49:748–759 21. Zhang Z, Zhang G, Goyal H, et al: Identification of subclasses of sepsis that showed different clinical outcomes and responses to amount of fluid resuscitation: A latent profile analysis. Crit Care 2018; 22:347
Bhavani SV, Semler M, Qian ET, et al: Development and validation of novel sepsis subphenotypes using trajectories of vital signs. Intensive Care Med 2022; 48:1582–1592
Seymour CW, Kennedy JN, Wang S, et al: Derivation, validation, and potential treatment implications of novel clinical phenotypes for sepsis. JAMA 2019; 321:2003–2017
Komorowski M, Celi LA, Badawi O, et al: The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care. Nat Med 2018; 24:1716–1720
Roggeveen L, El Hassouni A, Ahrendt J, et al: Transatlantic transferability of a new reinforcement learning model for optimizing haemodynamic treatment for critically ill patients with sepsis. Artif Intell Med 2021; 112:102003
Böck M, Malle J, Pasterk D, et al: Superhuman performance on sepsis MIMIC-III data by distributional reinforcement learning. PLoS One 2022; 17:e0275358
Wu X, Li R, He Z, et al: A value-based deep reinforcement learning model with human expertise in optimal treatment of sepsis. NPJ Digit Med 2023; 6:15
Joshi S, Parbhoo S, Doshi-Velez F: Learning-to-defer for sequential medical decision-making under uncertainty. arXiv:2109.06312
Merouani M, Guignard B, Vincent F, et al: Norepinephrine weaning in septic shock patients by closed loop control based on fuzzy logic. Crit Care 2008; 12:R155
Ghosh S, Li J, Cao L, et al: Septic shock prediction for ICU patients via coupled HMM walking on sequential contrast patterns. J Biomed Inform 2017; 66:19–31
TW, Zhang H, Subramanian J, et al: An empirical study of representation learning for reinforcement learning in healthcare. arXiv [cs.LG] Preprint posted online November 23, 2020. doi: 10.48550/arXiv.2011.11235
Belogolovsky S, Korsunsky P, Mannor S, et al: Inverse reinforcement learning in contextual MDPs. Mach Learn 2021; 110:2295–2334
Seymour CW, Liu VX, Iwashyna TJ, et al: Assessment of clinical criteria for sepsis: For the third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 2016; 315:762–774
Kamran F, Tjandra D, Heiler A, et al: Evaluation of sepsis prediction models before onset of treatment. NEJM AI 2024; 1 35. Johnson AEW, Pollard TJ, Shen L, et al: MIMIC-III, a freely accessible critical care database. Sci Data 2016; 3:160035
World Health Organization: Global Report on the Epidemiology and Burden of Sepsis: Current Evidence, Identifying Gaps and Future Directions. 2020. Available at: https://reliefweb.int/ report/world/global-report-epidemiology-and-burden-sepsiscurrent-evidence-identifying-gaps-and. Accessed January 16, 2024
Maximous S, Brotherton BJ, Achilleos A, et al; COVID-LMIC Task Force and the Mahidol-Oxford Research Unit (MORU): Pragmatic recommendations for the management of COVID19 patients with shock in low- and middle-income countries. Am J Trop Med Hyg 2020; 104:72–86
Sheller MJ, Edwards B, Reina GA, et al: Federated learning in medicine: Facilitating multi-institutional collaborations without sharing patient data. Sci Rep 2020; 10:12598
Khan, M. I., Arif, A., & Khan, A. R. A. (2024). The Most Recent Advances and Uses of AI in Cybersecurity. BULLET: Jurnal Multidisiplin Ilmu, 3(4), 566-578.
Gupta A, Lash MT, Nachimuthu SK: Optimal sepsis patient treatment using human-in-the-loop artificial intelligence. Expert Syst Appl 2021; 169:114476 41. Xu Y, Xiao C, and Sun J: DeepRite: Deep recurrent inverse treatment weighting for adjusting time-varying confounding in modern longitudinal observational data. ArXiv:2010.15028
Jeter R, Lehman L-W, Josef C, et al: Learning to treat hypotensive episodes in sepsis patients using a counterfactual reasoning framework. medRxiv 2021.03.03.21252863
Kausik C, Lu Y, Tan K, et al: Offline policy evaluation and optimization under confounding. arXiv:2211.16583
Chase JG, Preiser J-C, Dickson JL, et al: Next-generation, personalised, model-based critical care medicine: A state-of-the art review of in silico virtual patient models, methods, and cohorts, and how to validation them. Biomed Eng Online 2018; 17:24
Raghu A, Komorowski M, Ahmed I, et al: Deep reinforcement learning for sepsis treatment. arXiv:1711.09602
Bologheanu R, Kapral L, Laxar D, et al: Development of a reinforcement learning algorithm to optimize corticosteroid therapy in critically ill patients with sepsis. J Clin Med 2023; 12:1513
Vasey B, Nagendran M, Campbell B, et al; DECIDE-AI expert group: Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat Med 2022; 28:924–933
Madras D, Pitassi T, Zemel R: Predict responsibly: Improving fairness and accuracy by learning to defer. ArXiv: 1711.06664
Wilson FP, Martin M, Yamamoto Y, et al: Electronic health record alerts for acute kidney injury: Multicenter, randomized clinical trial. BMJ 2021; 372:m4786
Festor P, Jia Y, Gordon AC, et al: Assuring the safety of AI-based clinical decision support systems: A case study of the AI clinician for sepsis treatment. BMJ Health Care Inform 2022; 29:e100549
Nagendran M, Festor PGRE, Komorowski M, et al: Clinical Evaluation Framework Using Behavioural & Visual Attention Read-Outs for Explainable AI (XAI). 2023. Available at: https:// ssrn.com/abstract=4496127. Accessed August 18, 2023
Sivaraman V, Bukowski LA, Levin J, et al: Ignore, trust, or negotiate: Understanding clinician acceptance of AI-based treatment recommendations in health care. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. New York, NY, Association for Computing Machinery, 2023, pp 1–18
Wong A, Otles E, Donnelly JP, et al: External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med 2021; 181:1065–1070
Feng J, Phillips RV, Malenica I, et al: Clinical artificial intelligence quality improvement: Towards continual monitoring and updating of AI algorithms in healthcare. NPJ Digit Med 2022; 5:66
Henry KE, Adams R, Parent C, et al: Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing. Nat Med 2022; 28:1447–1454
