Hospitals must establish safety guardrails before deploying AI

0:00Hi, it's Kevin. Partner with me on a KevinMD platform. With over 3 million monthly readers and half a million social media followers, I give you direct access to the doctors and patients who matter most. Whether you need a sponsored article, email campaign, video interview, or a spot right here on the podcast, I offer the trusted space your brand deserves to be heard. Let's work together to tell your story. Visit KevinMD.com and contact me today. And now on to the show. From KevinMD, I'm Dr. Kevin Poe, and this is the podcast by KevinMD. Welcome to the podcast by KevinMD, the only daily medical podcast where we share the stories of the many who intersect with our healthcare system but are rarely heard from. Now, here's your host, Dr. Kevin Poe. Hello. Hi, and welcome to the show.

1:02Subscribe at KevinMD.com slash podcast. Today, we welcome back Harvey Castro, emergency medicine physician and healthcare consultant. Today's KevinMD article is ChatGPT Health in Hospitals, five essential safety protocols. Harvey, welcome back to the show. Thanks for having me, buddy. All right. So, let's talk about your latest article about ChatGPT Health. It's been all over the news. What's this what about and what led you to write it? Yeah. So, big picture. Back in the year 22, I wrote the first book. How do we use this thing called ChatGPT in healthcare and I chuckle now because everybody knows what that is. And the more and more I study this, the more and more I realize, hey, governance and the right way of doing this is crucial because if we get it wrong, there's too much at stake. It's not like we pick a bad stock because AI told us to buy that stock, like literally people can die. So ChatGPT Health, for those who aren't familiar with it, it was recently introduced within the last few months. What exactly is it in 30 seconds? Yeah, 30 seconds is kind of interesting. Basically, your ChatGPT algorithm now allows you to upload your wearables, your Apple

2:07information, health. And then as a result, you can go through and go back and forth with a chat and say, hey, my hemoglobin A1C is X, what does that mean? And so, for non-physicians, it's helpful, but at the same time for us physician, it kind of gives us a little scaryness because this is correct. This is hallucinating. You know, what's happening here? So basically, patients can upload so much of their personal data. You mentioned things like Apple Watches, but also their entire health record as well. And then they can use ChatGPT to query it and ask it any information. That's correct? Correct. Yeah. And it's kind of scary because you don't know what's in your medical records at time, right? Most patients, they think they know, but they really don't know what's all in there. It's for it to give that information up to ChatGPT. From a privacy point of view, that's a flag. And then the second is, even if you're okay with that, what if it misinterprets what's there? And as we know, medical records has so many errors in it already. So from the hospital standpoint, what are some of the red flags that they need to be

3:09careful of as more and more patients use ChatGPT health? Yeah. So, I broke it down into five pillars in the article and invited everybody to make sure they read it. That's when obviously it's a data privacy. If we get this one wrong, it's game over. If your patients don't trust us, you know they're not coming back. And so we must make sure that obviously the hit gets hit but compliant, obviously that it's encrypted, but not only that the information is safe, every time we move the data is the data safe as well, meaning if the nurse is saying the hospital system, the AI is going slow and she's like, I'm just going to use my phone and use my public AI account, obviously we can't do that. But there's people that say, you know what, it's going to be okay. The answer is no. We need to make sure we do this correctly. Now, in terms of the privacy risk from the patient standpoint, what are the things that they need to look out for? ChatGPT health encourages patients to upload all this health data there. What are some of the privacy safeguards with that tool?

4:09Well, number one, obviously they're going to make sure that that platform is hit by compliant, meaning there's no person out there that's trying to hack into that bridge of you connecting up to ChatGPT, make sure that that's taking your data, so that's obviously number one. Number two, that other data sets in the sense that when your encryption going back and forth is connected and then other things on their side, that their servers, you're actually at the mercy of their servers, we've heard it multiple times in the news saying, oh, at now this company of that one got hacked, how can we make sure that ChatGPT doesn't get hacked? But they're taking that information. Now is that happening? Are we confident in the security of patient privacy from the ChatGPT perspective? Oh, that's a tough question. I would imagine with the billions of dollars I stake that they're doing everything possible, but at the end of the day, we all know that nothing's 100%. I haven't read anything that they've been hacked, but I would imagine that there's tons of people out there lining up trying to break that. And again, going back to the hospital standpoint, do they have to take any specific measures

5:14above what they're already doing as more and more patients are using AI and using ChatGPT health to analyze their data? Now, you see, that's really good question. Number one is the diversity in the data itself, right? The hospital just can't say, hey, Harvey just came in and is trying to sell us this product. No, let's look at the data. Does that data represent the population that that hospital is serving? So that's a really important question that the hospital must look at. Other things that they must also look at is when we're using the AI model and the geeky side, we call it the data drift, meaning I'm asking a question every day and it's doing what I meant to ask, but with time, the data, the model will drift to a certain point where it stops doing that. So we need to, what I'm adding now is we need to make sure that the hospital administrators know that we must have a data scientist in the loop to make sure that it's checking the data to make sure that the data looks good and the model hasn't changed or drifted to something that it wasn't meant to be. Now, today, just give us an update where this episode will go out probably sometime in March, how are hospitals and health systems currently using large language models?

6:19What's the state of the art? That's a good question. The nice thing is we're seeing the bellcard, hospital systems that have a little bit more resources are different leveraging AI to the point where they're bringing models in house, training the data of their patients in their hospital system, making sure that it's secure. But second, they're able to use this data for predictive analytics for different portions of like radiology, finding there's different studies that have been showing, like, for example, the human can see certain things, but the AI is able to not get tired and see certain things, patterns that we may miss because we're tired. And so the hospital systems are adding this as part of their protocols. What I find fascinating is we're starting to see a two-divide system where hospitals that are going real strong with AI doing it correctly, doing the governance. Then we're seeing the other side where they have no idea or they don't have the resources to be able to get that. So those hospitals that don't have those resources, are they just using these commercial models? Are they just being more lax with security and compliance?

7:20What are these lower resource hospitals doing? Unfortunately, they are just trying to keep their doors open. So a lot of times they're not even adding the extra expense because they can't afford it. They're totally understanding that point of view. And so what I foresee happening is this is my bias opinion. I foresee the bigger hospitals, stronger, healthier coming in and taking over the other hospitals that don't have those resources and then using that as a competitive advantage to possibly close those other hospitals or just take over them, which is, I know the last thing we want to hear as a doctor is that we're closing hospitals. Now as we both know, a lot of these health AI startups are selling their products to these hospitals. Now as hospitals evaluate these tools, you mentioned your framework, but let's go into more detail. What are some of the questions they need to ask these new companies to ensure that not only their tools are compliant, but effective as well. Yeah. So number one, the data is the data junk data is a correct give you a quick example. I was in Europe giving a talk in dermatology and it dawned on me.

8:23It's like, wait a second. When I went to med school, all the books that I read, all the populations was a certain type of population and I hadn't seen rashes in different populations, same analogy. So ask the vendor, hey, let's look at the data set. Is this representative of my data? And if it isn't, how are you making sure that it is? And then as I start feeding it, is there someone in the company that's going to be in your side of the AI side that's going to check for that model drift that I'm talking about? Because most of the times they're going to sell it to you, they're going to walk away and then it changes for what you spend all this money on and it's not there. And so obviously on the internal side, can we handle the electricity bandwidth? A lot of people don't think about that, but we are using AI. We are going to start using more electricity and if it is true that some of the hospitals are just barely making it, can we expend that much more money and energy? Now I was reading article that a lot of these AI companies, because it's so easy to code, to vibe code, so to speak, that they're coming up with all these products, but they

9:24don't have any physician or clinician looking at how this product would really impact hospitals. So now hospitals are just inundated with a lot of these vendors that really have no basis in terms of how medicine's really practiced. Are you finding something similar to your observations? Yeah, totally agree with you and that's why I say it takes the village, it takes people like you, me, but the funny thing is when I consult startups, I ask them, do you have a patient as part of your board and they're like, no, I said, do you have a doctor in the board? No, and I'm thinking, oh my gosh, they have part of the ecosystem and they're trying to create a solution, but they don't have the full team to represent that solution. And so obviously, to your point, we need to make sure with that, we have the doctor in the loop, the patient in the loop, but not just the doctor, the whole staff, because as you know, the team, how many times has our nurses saved us from X or Y or the front desk that, hey, Doc, did they tell you X? She's like, whoa, I didn't even know that. So we need everybody included. Now for these health IT companies that you consult with, so when you say, hey, you need a patient,

10:24you need a clinician in the loop, do they actually listen to you and make those changes? Actually they do. We've been able to successfully create certain boards for this and I've been able to look at the model and look at their framework and how they're doing it. At the end of the day, my personal goal is the patient's safety and I have to add this because I want to make sure we get this in there. We've got to make sure that our hospital systems are transparent with their policy so that if AI is being used, our patients know that it's being used. And I know that sounds like, oh, no, they just registered, no, no, no, we need to be transparent because again, it's all about trust. If we lose this trust in this fight of AI and healthcare, we're going to lose this battle and work. Now for the patient standpoint, what are some common scenarios where AI may be used that they may not initially be aware of? I'm kind of lacks on this department, but I hope patients realize even down to your appointment now, a lot of that information now, it's going, it's creating, it's looking for a slide and giving you that information, there's more and more models that as we know, the ambient listening, a lot of hospital systems are using that and for better or worse depending

11:26which side you're on, you want to make sure that it's transparent so everybody knows. And then on that side, I really think we need to educate both sides, the position at the hospital side, but the patient side of how this tool is being used. And then I know this is very controversial, but we should give an apt out button. If the patient's like, look, I don't want this AI as part of my healthcare, we need to be respectful of that. And so you know what, that's fine. Just know that this is the different framework when we're working with you. Oh, it's so interesting. So you ever see a point where hospitals are going to be so intertwined with AI that if patients opt out of that, that hospital system simply can't serve them appropriately? Yes, unfortunately, I see that they coming. I see a day that patients, like you said, it's going to be everywhere. It's going to be in our appointments, predictive analytics and our x-rays, imaging labs in our EMR. It gets to a point where, you know, and that's why I say at the end of the day, it's education. Let's educate the patients on the good, the bad, the unknown, that way they understand this product, that way they don't fear the unknown and they're more likely to accept it.

12:30So as part of your framework, you talk about human interloop as hospitals adopt. Some of these AI tools, what would that look like? Yeah, great question. So, for example, many people don't know this. But the first ShedGB 3.5 that came out, third world countries were saying yes or no to the output, meaning they were not doctors. There were some other countries saying, yeah, this looks like something right. And then people are using that data. So the human interloop, the reason that's so important is not just any human, it's a doctor. And that's just any doctor, if you're asking an ER question, then make sure you have an ER doctor in the loop, meaning if it hallucinates, if it makes an error, if it's saying something that is totally off, or if the guidelines have changed. Example, if AI was built until a certain date, and the guideline came out on that day, but a day after new guideline came out, the AI has no idea. It's technically correct, but it's no longer accurate. That's where the human comes in and looks at it and it's like, no, no, this is wrong. And that's why we need to have that.

13:31And the evaluation of these tools doesn't just stop once the hospital adopts them. There needs to be continuous monitoring after adoption as well. So talk more about that. Yeah, we need to make sure that thing I mentioned earlier about the data drift, that makes sure that we have someone auditing it. And then make sure that it's a 360 again, so that the patients can say, you know what, that seems off, or the doctor, the front office. That way, the data scientist is able to get that information and look at it and say, oh yeah, this is drifting. But more importantly, hopefully the data scientist that's working in this saying, wait, the model is shifting. We need to pause it for a minute. Let us fix this before we use it anymore. So that is really an important. Make sure that it's tracking the trends. Make sure that it's accepting the clinical feedback. And obviously, make sure that we're doing regular audits on the system. Now, you're consulting a lot of healthcare institutions. Of course, regarding AI adoption, what are you seeing? Are there a majority of these medical institutions adhering to this framework that we're talking about today? Yeah, I see it happening. What fascinates me is I'm getting a lot of calls from the Middle East,

14:33from Singapore, and they're doing things way differently. And I think it comes down to politics. If it's pushing down from the government down saying, hey, one electron a medical record, one data set, one AI, things are going to happen. Here in the United States, we got fragmentation, we got lobbyists, we got different points of view, so things are a little bit slower. So I find it really interesting, I hate making prediction, but I predict we're going to start seeing some really interesting models outside of the US and the use of the outside. Example, Middle East is using digital twins. I'm like, oh my gosh, being able to have your own digital twin and practice medicine, what drugs work, and what not, and then tell the patient, okay, this is a drug that I know is going to work best for you. And I'm thinking, I'm getting a little phomo. I was like, I want to have that here in the United States. We're going to Harvey Castro, emergency department physician, health care consultant. Today's Kevin M. DiArticle is CHI-GPD health in hospitals, five essential safety protocols. Harvey let them with some take-home messages they want to leave with the Kevin M. Di audience. Yeah, at the end of the day, think of AI like riding a bike.

15:36When you first started riding a bike, you were scared, you were like, I'm going to fall, and before you know it, you started riding a bike, or not using your hands. Same thing with AI, use the tool, know the good, the bad, the ugly, and the more you use it, you more you'll realize how more efficient I can be, and how I can help my patients more importantly. And thank you so much for coming. Harvey, thank you again for sharing your perspective and insight. Thanks again for coming back on the show. Thank you. Thank you for listening to the podcast by Kevin M. Di. To share your story and appear on the show, visit kevinmd.com.