Clinical Informatics for Surgeons, Episode 3: AI in the OR and Beyond; Machines That See, Read, and Act

In the not so distant future, just imagine three machines in your operation. That first one's watching you. It's watching that laparoscopic feed, robotic feed maybe during a gallbladder case, quietly drawing out what should you be paying attention to. audio1270906071: Is this bit of tissue, it's safe to divide? Is this an unsafe zone in which you're dissecting? That other machine, reading the chart, right? So by the time you've finished that operation, op note's done. There's a plain language summary for the patient audio1270906071: that audio1270906071: that patient will actually understand. And that third machine in the room has been quietly acting in the background. audio1270906071: Case went a little long. Some of the data coming in from anesthesia suggests that they're gonna need maybe a little bit more monitoring, so it's already got that ICU bed request queued up and flagged to the rest of the perioperative team to have a bigger discussion about goals of care.

And notice what none of those three machines did. audio1270906071: None of them handed you a percentage. audio1270906071: Right. I mean, audio1270906071: if you think about really audio1270906071: sort of audio1270906071: like for the last decade, maybe two decades, when we think about AI audio1270906071: or, audio1270906071: or surgical AI, it's largely been in the context of risk scores, right? audio1270906071: Like, audio1270906071: what's the risk of this complication or that complication? audio1270906071: and Really with the advent of some of these newer technologies and newer algorithms, that era of sort of cold and a little impersonal risk calculation is audio1270906071: kind of audio1270906071: coming to a close as we start to think about these models that can see, that can read, audio1270906071: and, audio1270906071: maybe scary or not so scary as we may talk about, machines that act. audio1270906071: And of those three categories you described, one of them is real, and it's already in operating rooms right now. One's arriving fast, and one is mostly promise, and it's got serious safety problems attached. And today we'll tell you which is which Welcome back to Behind the Knife. I'm Mason

Pearce, general surgery resident, clinical informatics fellow, and BTK education fellow. audio1270906071: This is the final episode of our Clinical Informatics for Surgeons series. we close the loop with the frontier: computer vision, large language models, and agentic AI. audio1270906071: And I wanna start with some vocabulary first so that we're all on the same page. There are three different kinds of AI here. Predictive AI, which gives you the number, like a risk of a complication. Generative AI, the large language models, reads and writes language, like notes, summaries, recommendations. audio1270906071: And then we have agentic AI, which doesn't just answer, it acts. It plans a task, uses tools, and takes steps on its own. See, read, act. We have two of the best people to walk us through this, Dr. Tyler Loftus, trauma surgeon at UF, board-certified in clinical informatics, past chair of the ACS Health IT Committee, and actually builds decision support using risk calculators. audio1270906071: And

Dr. Dan Hashimoto, associate professor of surgery and director of surgical informatics at Duke, edits the textbook on AI and surgery, sits on the AI task forces at ACS, ASE, and SAGES. Thank you both for closing this out with me Thank you, Mason. Thanks for having us. Really happy to be here and excited for this discussion. audio1270906071: Same. This is my probably favorite topic ever. Maybe second only to endoscopy, so excited to dive right in. All right. Let's briefly review surgical risk calculators and prediction. Dr. Loftus, could you give us a quick overview? Yeah, it'd be my pleasure. So the NSQIP risk calculator that you've all used started in 2013 as a logistic regression calculator trained on 1.4 million patients. audio1270906071: And the ACS has since rebuilt it on machine learning, specifically gradient boosting, and this is described in work described by Lu and colleagues in 2023. A tool called POTTER

showed that surgical risk is not linear, that trees capture the interactions regression flattens with the mortality C-statistic around 0.92. audio1270906071: Our group's model, MySurgeryRisk, went from a single University of Florida build to video1270906071: external audio1270906071: validation across 14 Florida health systems. More than half a million encounters with discrimination of up to about 0.95 for in-hospital mortality described by Ren and colleagues in work led by Dr. Azra Bihorac, which was recently published in JAMA Surgery. audio1270906071: Okay, so it sounds like the prediction grew up. That's right. Prediction grew up. But a number you look at does not change care by itself. What changes care is a system that can see what audio1270906071: you see, audio1270906071: explain what it thinks, and take the next step with you. That's the frontier, and that's where Dan lives. All right. audio1270906071: Dr. Hashimoto, let's start with the basics. What can a computer do with surgical video today? Yeah, that's a great question. audio1270906071: You know, audio1270906071: I think the starting point really is thinking about

recognition. What's audio1270906071: kind of audio1270906071: wild is when we think about the current sort of state of AI and things like that, right? audio1270906071: we think of this as a fairly recent phenomenon, but actually one of the very first papers on recognition and computer vision was in 2004, if I'm not mistaken, from the Nassir Navab group in Munich. audio1270906071: So this is actually 20 years' worth of work specifically in surgery that's been building up to this. our own group, from the time that I was at Mass General, with Ozanan Meireles, Daniela Rus, audio1270906071: and, audio1270906071: and Guy Rosman, was, first built around sleeve gastrectomy video audio1270906071: when bariatric surgery was still fairly popular. audio1270906071: but we built a computer vision model that could automatically label the operative steps of that case. and at the time, we were achieving about 82% audio1270906071: accuracy or so, average video1270906071: across audio1270906071: seven steps. but this was on video that the model itself had really never seen, right? and, we were fortunate enough to publish that video1270906071: the Annals audio1270906071: of Surgery back in, in 2019, which feels like ages ago. audio1270906071: But, audio1270906071: you know, audio1270906071: as, as academic audio1270906071: as it may sound to think of, okay, audio1270906071: well, audio1270906071: so you can identify the step of video1270906071: a surgery, audio1270906071: what's the big deal? I

think it's important to know that step recognition is what gives you context, right? It gives you context for what part of the case you're in, what you're trying to achieve. audio1270906071: And if you want a computer vision system that can warn you about what's gonna happen next or what you should or should not think about as you're moving to the next step, you need it to first recognize what's happening right now. That's really interesting. Like you said, from recognizing the steps to flagging danger, how does it do that? audio1270906071: Yeah, audio1270906071: I mean, audio1270906071: and this is audio1270906071: kind of audio1270906071: where the biggest questions really arise, right? It's like, how does it... how do I know this machine, audio1270906071: is seeing something that is actually meaningful? video1270906071: So audio1270906071: when we think about cholecystectomy, right? What is that complication that we all fear as surgeons? It's a bile duct injury. audio1270906071: when we think about the traditional ways of building computer vision models to recognize anatomy, we're trying to recognize things like the cystic duct, the cystic artery, the common bile duct, right? But sometimes it's, by the time you get to the point that you can recognize those structures, the injury has already occurred. audio1270906071: So in work that I was fortunate enough to

collaborate on with Amin Madani at the University of Toronto, we actually thought about how could we build models that instead of trying to identify the specific anatomy, how can we have these models, video1270906071: quote-unquote, audio1270906071: "see things" in the way that we see things as surgeons, which is where is it safe to video1270906071: dissect audio1270906071: and where is it not safe to dissect, what we call the go and the no-go zones respectively. audio1270906071: And that work we published in the Annals of Surgery in 2022, where we were actually able to demonstrate that these models would be able to highlight the this is not safe area, the no-go zone, with a, to 95% accuracy when we compared it to experienced surgeons, who a- annotated the same thing. And we've done multiple follow-up validation studies, with SAGES that tested that platform a- and replicated these results. audio1270906071: so really now we have this, machine learning model that can see the operative field and really start to process it in the same way video1270906071: that audio1270906071: we as surgeons would, and give us that visual feedback. That's very cool. And is any of

this actually in the OR today, or does it still live in the lab? Yeah. audio1270906071: Well, audio1270906071: that's a great question. audio1270906071: I mean, audio1270906071: actually, this go, no-go zone project in gallbladder, Dr. Madani is wrapping up a trial on that in the operating room at the University of Toronto, so I'm excited to see what those results look like. we've also seen similar examples of this coming from outside the US. audio1270906071: there's a group in China that developed a navigation system for laparoscopic gastrectomy. and we've also seen the group at the University of Strasbourg who's done a critical view of safety, checkpoint in live surgery actually several years ago, that they've demonstrated at the EAES meeting live, which was fascinating to see. audio1270906071: And here I really wanna highlight and applaud that group of investigators that Dan is highlighting, including his own work with, Amin Madani, because they're using audio1270906071: real-world data. And so it's not, an algorithm that's trained on nice, tidy anatomy, and typical textbook presentations. audio1270906071: It's real-world data, so it's gonna have

bloody, atypical fields that you might see at 2:00 in the morning, and we can expect those models to maintain high performance then when they are translated broadly into clinical practice. Yeah. audio1270906071: and, audio1270906071: that's the part that is, probably the most onerous and least glamorous when we do these types of studies. audio1270906071: It's back-breaking annotation, right? And, when we first started this, the paradigm was very much somebody had to sit there and do thousands and thousands of frames. audio1270906071: I mean, audio1270906071: it's actually hundreds to thousands of hours of labeling, to make sure that we had a valid ground truth that really synced up and matched with how surgeons see things. audio1270906071: And that was the, audio1270906071: sort of audio1270906071: driving force behind the SAGES Annotation Framework that was published in 2021. and it was meant to help standardize how we label these videos so that data sets from across multiple institutions, could work together, audio1270906071: and could be audio1270906071: collated together across different data sets. audio1270906071: Really, we wanted to avoid this whole sort of GIGO, the garbage in, garbage out,

element of data that we think about, traditionally that exists just as much if not more when we think about video data. Very true. Where do kinematics fit in, the instrument motion side of things? Yeah. So that part's super exciting, right? audio1270906071: it's, audio1270906071: it's fascinating to see, audio1270906071: like, you know, audio1270906071: how the world sort audio1270906071: of audio1270906071: comes back around in academia because kinematics was such a big part of the simulation revolution and craze, in the mid-2000s, where we were really thinking about, audio1270906071: well, audio1270906071: how can measuring path length and economy of motion and whatnot help us predict expertise, audio1270906071: or the development of competency in VR surgical simulation models? audio1270906071: And now that we have these computer vision models, now that we have robotic surgery, machines, we can actually start to capture those kinematics. So obviously in robotic systems, some of those vendors do provide some of the kinematic, ground truths. so you'll know the XYZ position of all the instruments in space. audio1270906071: and if you can't get those, there are certain ways to derive

those or infer those from two-dimensional video. So really that's audio1270906071: kind of like audio1270906071: been my holy grail in terms of my research career when I started in simulation, back in 2006, so 20 years ago. thinking about, audio1270906071: you know, audio1270906071: if you can quantify motion, can you use that quantification to give more specific feedback, right? audio1270906071: So can you enable deliberate practice, by having these quantifiable changes in your technique so that rather than just being like, "Good job. Next time sew better," right? Or, "Next time," audio1270906071: like, "be audio1270906071: faster." audio1270906071: Like, audio1270906071: what does that even mean to be faster to sew better, right? Can we quantify that audio1270906071: and, audio1270906071: and make it where you can actually level up on your performance just like a professional athlete might, right? audio1270906071: and really when we start to think about how we can daisy chain those tasks and events together, now you start having this almost story of an operation through the movements that you're engaging in. and that allows you to think about a traditional story structure. And just like a traditional story structure has a beginning, middle, and end and you wanna

get to the fairy tale ending, it's the same kind of thing with these computer vision models, right? audio1270906071: So if you can, get a sense of the statistical distribution of these events and tasks and movements, can you start to predict what may happen next? that's Some of the work that, Yutong Ban, who was one of my former postdocs, had done, with great success using generative adversarial networks to try to predict out in the future about 15, at the time, 15 seconds in advance what was gonna be the next step and what is the likelihood of an injury if you were to proceed down that chain video1270906071: of, audio1270906071: anticipated events. audio1270906071: Very cool. And I think I speak for all trainees when I say we would greatly appreciate that detailed feedback rather than the "do better." So we appreciate that work. and the big shift I keep hearing about, foundation models. What does that mean in this context? Yeah, audio1270906071: the foundation model, sort of paradigm has really thrown things for a loop. audio1270906071: Before we really started having these foundation models, you would take these specific tasks, and you'd have to highly train audio1270906071:

on these specific things, and one model was audio1270906071: kind of audio1270906071: stuck with doing one thing. audio1270906071: You know, audio1270906071: these foundation models are trained on an incredibly large corpus of data, such that you ideally don't have to retrain from scratch every single model to do every single little thing. audio1270906071: the analogy that I often give is, audio1270906071: when you're learning how to suture, you may first learn how to suture on a skin pad, then you're gonna learn how to do audio1270906071: a subcuticular stitch, and then eventually you're throwing knots and things audio1270906071: on portal vein, right? but every time you move to a different type of tissue, you don't totally need to relearn how to suture audio1270906071: or, audio1270906071: or tie knots. audio1270906071: Those basics are still ingrained in you from other contexts. And that's really audio1270906071: kind of audio1270906071: the idea of how we wanna think about utilizing foundation models. These models may be trained in cars, birds, dogs, cats, but it gives you important priors about things like color, motion, contrast. These are all things that are not unique to surgical video but can apply to surgical video, and then you can fine-tune on the more

specific elements that you may wanna know about surgery Those are some outstanding points about transferable skills and a framework for thinking about this. audio1270906071: I'd say for listeners who want to learn more about how a surgeon should consume this kind of information, as you might imagine, Dan wrote the guide on this. It's a framework for critically evaluating a surgical AI paper. it's written in the European Journal of Surgical Oncology. So when a vendor or an abstract throws a number at you, that framework will help you decide whether to believe it and how to use it. audio1270906071: Okay, great. Let's switch machines. Large language models. Dr. Loftus, level set us, because people conflate this with everything. What is an LLM actually doing? So at its core, a large language model is predicting the next word over and over. It's doing an extremely good job at that, a better job than any automated or computational process before ever has done, because it's using a lot of prior knowledge and a lot

of context, when it's deciding what that next word is. audio1270906071: So when you put it together audio1270906071: word after audio1270906071: word after word, these complex and sort of brilliant patterns can emerge from it. But it still is just predicting the next word, and, and I think one of the criticisms of language models is that they're just a fancy auto-complete. I'd say we have to also ground ourselves in the reality that it's a, if it's a fancy auto-complete, it can also take standardized tests, as well or better than a group of experts. audio1270906071: and so although that's all audio1270906071: it's doing, audio1270906071: it's doing that audio1270906071: really, audio1270906071: really well. but at its core, it's native task is language, reading it and producing it, and not calculating a predicted probability. And so its strengths are summarizing, drafting, explaining, translating, or what, data scientists might call semantic search. and the failure mode, is that it can be fluent and confident and yet wrong. And it may not tell you when it's wrong, because it's trained often to be

confident and assertive and helpful. so that leads to the hallucination problem, and that fluency is not accuracy. So where does that talent help a surgeon? audio1270906071: I mean, audio1270906071: I think audio1270906071: the immediate thing that jumps out to me is, audio1270906071: like, audio1270906071: paperwork, right? I think as surgeons, I can tell you that is not my favorite thing in the world to be doing, right? I'd rather be operating. but it really is thinking about probably the place where most people have seen it now is ambient documentation, right? audio1270906071: So when you think about taking an ambient scribe into your clinic visits or incorporating some of these elements into the EHR. Some of these EHRs now have these really audio1270906071: incredible summarization elements that can go through a very complex history and try to pull out the key things based on your specific specialty, that you may be seeing patients for. audio1270906071: And so we really think about how that can, also make things more accessible for patients because, obviously as, as clinicians, we have a certain jargon, and a certain style of

communicating with each other, but that may not be the best way to communicate with a patient. So seeing how some of these technologies can take our clinical documentation and, quote unquote, "translate it" into a version that is much more accessible to a patient in their after-visit summary, I think has a lot of value. audio1270906071: I think one of the biggest questions is how much value does it actually bring to an individual surgeon versus a health system, right? I think some of the, cynical aspects are, audio1270906071: well, like, audio1270906071: if you can do these things faster, does that mean you can see more patients? there are some pretty interesting data that came out of Mass General earlier this spring that looked at an average cost savings of about 16 minutes, not per patient, but per an eight-hour clinic block. audio1270906071: and so it's not like you can fit in a whole bunch of new patients. However, interestingly, the physicians had a decrease in perceived burnout by saving just those 16 minutes. So there's something about using these

technologies that is-- maybe this might be a stretch too far, this isn't what was in that paper, but audio1270906071: I, audio1270906071: think about my own experiences as bringing a little bit of that joy back to being able to talk to patients instead of sitting in front of my computer typing, every last word they say, so then I can actually focus more on that part of surgery that I think, drives what a lot of us went to it for, which is that interpersonal interaction. audio1270906071: Mm-hmm. audio1270906071: Absolutely. And I share Dan's optimism absolutely about that, kind of humanistic aspect audio1270906071: of the potential for using, models in that way. I'd say just to mention another potential danger here, I think a concern that I've heard raised even recently is that surgeons have witnessed CMS telling us that we've become more efficient in performing procedures. audio1270906071: And so now, audio1270906071: you know, audio1270906071: work RVs for procedures are going down. audio1270906071: Like, audio1270906071: you should be able to do more work because you can do it faster and better now. So work harder for less. And, there is a danger that as we can see more

patients and these tools make us more efficient, audio1270906071: that instead of translating to more time at the bedside with our patients, just translates into higher productivity thresholds. audio1270906071: And so it's something for us to think about and actively manage, audio1270906071: as we implement. and I'd say in addition to those things, we should notice that, it's also about being wired into the workflow, rather than just a chat window audio1270906071: or note-writing. There's a study this year out of Stanford where, a language model integrated into electronic health record and identified surgical patients, who might benefit from co-management and performed this task fairly well with high sensitivity, around ninety-four percent. audio1270906071: And this was using real patients from our EHR, and it was running prospectively. and I think audio1270906071: that audio1270906071: that's a really important future direction video1270906071: a audio1270906071: model living inside the workflow and building into existing digital workflows rather than in a separate browser tab or adding a new process to existing processes. audio1270906071: Absolutely. Dr. audio1270906071: Loftus, this next one

is where your research thrives, the consent and decision support side. video1270906071: This is audio1270906071: the one that I care most about. So picture the patient who cannot consent because they've lost decision-making capacity, and you're asking the family member, what that patient would have wanted if they could speak for themselves. audio1270906071: And the uncomfortable data point here is that human proxies match the patient's own choice only about 45% of the time on average, and that's been reproduced several times. and these are very high stakes decisions. These are decisions about whether someone's life is gonna be over and end a certain way under their terms audio1270906071: or not, and whether we'll do everything possible short of futility, audio1270906071: for this patient with therapeutic intent. audio1270906071: So in our study, which was published in New England Journal of Medicine AI this year, we gave a language model, a structured profile, the patient's own stated values, and we asked, that patient to choose the treatments that they would choose given a

scenario, and then asked their surrogate what they would choose for that patient. audio1270906071: and, the surrogates did about as well as surrogates always have, which is around 44, 45% concordance audio1270906071: with the patient. But when the model was enriched with a description of the patient's values, what really matters to them in life, the model selected the same treatment as the patient about 73% of the time, which was a substantial improvement. audio1270906071: That sounds almost too good. Where's the catch? So there's two catches. when we tried to have the model infer the patient's values from the chart or the electronic health record instead of from the patient directly, the accuracy falls off predictably and reliably, I think. And audio1270906071: we've learned that we should not use the electronic health record to infer a patient's values. audio1270906071: There's too much noise, and we need to be very confident when we're inferring patient's values, and I audio1270906071: would, audio1270906071: recommend against that approach. So this only works if the patient documents what matters to them while they still can document what matters

to them. And the second catch is automation bias. audio1270906071: If the screen says 73%, this is what the patient would want, a frightened and tired team audio1270906071: or audio1270906071: a family member or loved one might stop arguing with it and just accept it. And this is a non-maleficence problem, about first not doing any harm. video1270906071: Model audio1270906071: has to augment the human judgment and, not synthesize it. audio1270906071: In our view of AI decision support in surgery, that's the recurring theme. The goal is a better conversation rather than replacing the human conversation. And I feel like that part, I think as a computer vision person, I think about at least in, audio1270906071: in my spectrum of things, we have these metrics that we can see audio1270906071: like, audio1270906071: oh, how much does this image match up with that one? audio1270906071: But for this problem, it seems a lot harder, right? Like with these risk scores, you can check to see like audio1270906071: your observed minus expected, comparison. But like when you look at a paragraph of like clinical reasoning, like how do you benchmark that, right? Like how video1270906071: do audio1270906071: you interpret that, audio1270906071: and try to get a sense of audio1270906071: like audio1270906071: how are we as a field gonna

come together and determine how these audio1270906071: are working, right? audio1270906071: And then what's gonna be an acceptable, level of performance from these models? audio1270906071: Right, audio1270906071: and that makes everyone a little cautious. let's move on to the third machine, and the one that is in every headline right now, agentic AI. Dr. Hashimoto, what makes something an agent versus just a very good chatbot? audio1270906071: Yeah, it usually starts with, audio1270906071: like, audio1270906071: a double O number in the front, and then, audio1270906071: No, audio1270906071: no. So bad dad jokes aside, audio1270906071: when we think about chatbots, audio1270906071: you know, audio1270906071: the traditional sense or conversational agents, you might copy or paste something in there, it would spit something out. You would copy and paste that back into, audio1270906071: your IDE or, audio1270906071: your programming interface. audio1270906071: but now with agentic systems, you are audio1270906071: allowing these models to not just give you an answer back, but then start to act on it, So you give an agent a goal. It can make a plan. It can pull in tools. It can call in other software, query databases. Sometimes it can, subcontract out

its tasks to other agents, audio1270906071: And then take steps from there. it may have another agent that checks its work along the way to make sure that things are getting done in the way that they were intended. so you can audio1270906071: sort of audio1270906071: imagine how a traditional large language model might be a, quote-unquote, "brain" that you can talk to in sort of video1270906071: this audio1270906071: disembodied sense. audio1270906071: but the agent does now give that brain a set of hands to start doing things, in a software kind of way. audio1270906071: audio1270906071: and that's really where you start to get that leap and starts to bring into mind, audio1270906071: like, audio1270906071: okay, audio1270906071: well, audio1270906071: what's the next step for, audio1270906071: quote-unquote, "physical AI," right? And how does this translate into surgical practice? audio1270906071: So let's take, for example, postoperative triage decisions. The non-agentic version shows you a risk number, and perhaps some of the variables that the model was paying attention to when it produced that number. The agentic version, meaning that the model itself can use tools and gather resources and perform a more complex task,

in series, will notice the high-risk patient, pull the lab values, maybe the vital signs and ventilator data too from during the operation. audio1270906071: Could draft a triage note, put in a tentative ICU bed request for a high-risk patient that was destined for the floor that might have been untriaged, and could message the team that this patient meets criteria for the ICU. This is why we think patient meets criteria for the ICU, and do you approve this? audio1270906071: So this is the difference between a prediction on a screen and work that's moving and being done with greater autonomy. Yeah, and I think there's some interesting work that is suggesting that taking an agentic approach, is likely better at some forms of complex reasoning than thinking about a raw model, quote unquote, audio1270906071: or like a single foundation model type approach that we may traditionally be used to using in medicine. audio1270906071: we think about, audio1270906071: you know, audio1270906071: oncology, actually, even in scientific discovery, it was published in Nature earlier this year, video1270906071: and audio1270906071: you can see multiple agentic

systems working together, and they outperform general purpose language models on things like treatment recommendations for certain types of cancers, or even coming up with de novo experiments, running those experiments, analyzing, interpreting those experiments, and writing the paper, audio1270906071: It's really turning on its head the traditional academic paradigm of productivity. Yeah. audio1270906071: I'm glad you brought up that paper. It really is changing the paradigm. but when you look closer at that paper, even the best agentic system was judged ready for clinical use without editing only about one quarter of the time. audio1270906071: So it's better, but it's still not something that we think we should be ready to turn loose. Which gets at the thing that scares people, autonomy. How much should one of these be allowed to do on its own? Yeah, that's like the multi-billion dollar question coming up these days, right? audio1270906071: so there is consensus pretty much that especially in healthcare, I think agentic systems all land on this idea that you gotta keep a

human in the loop. audio1270906071: but the bigger question I think even, one step further than that is, audio1270906071: well, what does audio1270906071: that even mean, right? Like, where in the loop do you put that human, and which human do you wanna put where? and so we really wanna define exactly which steps we're going to allow agents to do versus which may need some sort of, approval audio1270906071: or, audio1270906071: or signature from a human, how you're gonna build in verification. audio1270906071: can it be an automated verification? Does it have to be human-based verification? and how do you prevent errors from audio1270906071: chaining together and compounding, right? if an agent can take 10 steps and was wrong at step two, and then eight more steps get built on top of that, audio1270906071: well, audio1270906071: now you've got a potential disaster. audio1270906071: And you think about that from a surgical perspective. Think about when you're doing an operation. You get into the wrong plane early, and the downstream consequences of that become a nightmare. and it's the same kind of thing when you think about these agents. Absolutely. And that's exactly why an agent that recommends a consent

choice or fires off triage orders without a human owning each step or at least being on the loop, is not something that governance exists for yet. audio1270906071: So the technology is running ahead of the guardrails, and that brings us to the closing segment. So we have three machines at three different stages. How does a typical surgeon stay in control of it all? I recommend that you interrogate all three the same way, and it's the same checklist from the whole series. audio1270906071: Where did the data come from? Was it validated on patients or videos like my patients? how's it gonna drift? Can it explain itself? And the bottom line is, does it change what I do, and does it allow the patient to have a better outcome or improve something for the surgical workforce? Most surgical AI has not been measured on that last bar. audio1270906071: I'd add that although this is described as a checklist, I think there's a lot to be said for the qualitative assessment of just

asking the users themselves and the people affected by this, "Does this make sense? Is this something we should use?" And, actually more of a vibe check on the technology before rolling it out. audio1270906071: I think Karandeep Singh has spoken about this and, through his own work as a, a chief health AI officer, has learned that both parts are important. We should have the checklist, and we should also ask ourselves, stepping back, how do we see this working, and what are the issues with it? Yeah, and you don't need, audio1270906071: like, audio1270906071: a PhD in machine learning or data science, computer science to be able to ask these questions. audio1270906071: In fact, we already possess so many of the tools that are needed because at the end of the day, it's just critical thinking, right? And thinking about how we wanna apply evidence. So yes, a little bit of vocabulary is helpful to understand certain terms. You wanna know about validation, calibration. You wanna know what happens audio1270906071: when data can drift and impact a model. audio1270906071: But in reality, for example, you would never just

prescribe a new drug because you saw a commercial for it, right? You would look into it. You would look into the trial, the efficacy. You would try to get a better understanding as to whether this drug fits the patient that you're thinking about, and do they fit the profile of the study population on which that drug was, studied, right? audio1270906071: And it's the same thing with these AI tools, We wanna think about them in the same critical way that we've been taught to think about things from sort of a biostatistics and epidemiology type perspective. And that is actually also why I think as surgeons, we should feel empowered to weigh in on some of our hospital's governance committees. audio1270906071: And so thinking about joining either AI committees or clinical decision support committees, your voice is valuable because you are the practicing clinician that is taking care of patients and using these technologies, The last thing that you want is a bunch of bureaucrats making decisions that have never sat in front of a patient. audio1270906071: and so we need to bring our surgical voice to the table, and that's something that the American College of Surgeons has been, really

supportive of. It's also the American Surgical Association and SAGES. These are all organizations that have really stressed the importance of the surgeon being an informed steward of this kind of technology. audio1270906071: Awesome. This has been a really great discussion. Let's give our listeners three things to walk away with. Number one? so we definitely wanna think about these sort of three modes of interacting with data or a machine interacting with these datas, right? Like seeing, quote-unquote, "seeing the data," reading data, and acting upon data. audio1270906071: I think a lot of the confusion about how AI may apply in surgery is really just audio1270906071: mixing up how those things may come together audio1270906071: or may work independently. Great. Second, The value is never in the output alone. It's whether the tool changes a decision and improves an outcome for our patients without burying you in clicks. audio1270906071: So we should demand that evidence before implementing. And our last takeaway. Yeah, I think the more autonomous a tool, the more a surgeon has to be involved, right? Gotta be in that

room, either in the operating room or in the committee writing those rules about how to use these safely and effectively. audio1270906071: And I think having that understanding is what's gonna empower you to steward that technology safely. All right, and that's a wrap on Clinical Informatics for Surgeons. from the surgeon informatician all the way to the machine that sees, reads, and acts. Huge thanks to Dr. Dan Hashimoto and Dr. audio1270906071: Tyler Loftus for closing it out. Check out our references in the show notes. Bring questions to your next AI committee. Stay curious and dominate the day audio1270906071: