AI Journal Club #1

Ayman:

Hey everybody. I'm Amol one of the Behind the Eye Fellows. Today we'll be introducing our new AI journal Club series where we discuss the clinical case and AI related papers and tools that fit into the patient's care.

AI and surgery is still very much in its infancy and there are many papers that are being published almost daily. Today I'm joined by some cohosts from Oregon Health and Science University. From faculty we have Dr. Ucci Ola, who's a thoracic surgeon at OHSU, and also founder of Firefly Lab, an AI driven platform that is built for competency-based medical education.

In addition, she directs the Surgical Data and Decision Sciences Lab for the Department of Surgery at OHSU.

Ruchi: Hi, thank you for having me on here. I'm very excited to be on this podcast with all of you.

Ayman: And, we're happy to have you. And we also have Dr. Marissa Sewell, a PGY four general surgery resident at OHSU.

Hi, very happy to be here. Thank you for having me.

Ayman: So let's just start right with the

case. Imagine we have a 52-year-old woman with a past medical history of hypertension, hyperlipidemia, chronic obstructive pulmonary disease without needing home oxygen, and a recent admission for a left upper lobe wedge resection for a suspicious nodule.

She was discharged on postoperative day two without any issues, but now comes into the emergency department on postoperative day five with dyspnea. Her initial vital signs and physical exam are notable for a temperature of 39 degrees Celsius, a blood pressure of 90 over 50, a heart rate of 115, an oxygen saturation of 87% on room air and left calf erythema and swelling approximately four centimeters greater in diameter compared to the contralateral side.

Dr. Thal, what are some key differential diagnoses and maybe some more rare considerations?

Ruchi: Oh, so this is super interesting. While this is a hypothetical patient, you know, for me as a thoracic surgeon, many things come to mind. I get a little tachycardic on my own 'cause there's something bad that's really happening to this patient.

So,

my mind really goes to PE and pneumonia, either one or the other. Maybe even both. I mean, the calf tenderness off the bat has me worried that she has a DVT that could have embolized, you know, into a pe Rare considerations you ask. Well, you know, without getting too much into the weeds of thoracic surgery, when you do a wedge, you know, there's a question, is the remaining lung, did it tors for some reason, like the lingula.

And Lingular torsion could, you know, present in a very similar way, but you wouldn't expect the calf erythema and swelling. So other things, you know, early empaa could make her sick like this. I think there are a lot of things to consider.

Thank you, Dr. Awa. Those are all very important considerations.

Before the thoracic team has been consulted as part of the initial workup in the emergency department, basic labs and ECG and a chest x-ray are ordered. Additionally, the emergency department physician performs a point of kill ultrasound of the lower extremities.

Ayman: Awesome. And this is a great segue into our first study.

So in

January of 2025, a UK-based team published in Nmai, new England Journal of Medicine, the results of a multicenter double-blind study evaluating AI-driven detection of proximal D vts. Their work was to evaluate a software called Auto DVT, which was developed by a company called Ano. And essentially couples to a handheld ultrasound device and is intended to allow non radiologists to perform two point compression ultrasounds and diagnose in real time proximal DVT.

The researcher is meant to take an ultrasound on the groan groin and move the probe as the software directs it and compress it until sufficient data is collected, which is indicated. The auto DVT in this study was performed prior to a clinical ultrasound, and the results of it was not disclosed to anybody and not used for clinical care.

Then the auto DVT results were compared to clinical ultrasounds. The primary outcome of this study was the sensitivity of the algorithm and prospectively was set to be 90% or

greater. Which resulted in about 400 patients being enrolled. The data gathered by auto DBT was then also reviewed by radiologists to ensure that there were no image quality capture issues that bias the studies or results.

Unfortunately, the auto DBT had a 68% sensitivity and 80% specificity for proximal DVT and out of these 31 positive scans. True positives. 10 were mislabeled as negative by auto DVT, and one of those patients had a normal D dimer, which made the diagnosis even more important. Retrospective clinical review of the images taken by auto DVT however, were 85% sensitive, so the images should have been adequate enough to have a higher sensitivity.

For these reasons, the desired 90% sensitivity was not achieved, and this was published as a negative trial. As that sensitivity is just not acceptable in the setting of the DBT rollout.

Ruchi: So, this is a great study to discuss for multiple reasons. And now I get to put my

hat on as a implementation. Um, there are really two aspects to the study. First one is if non radiology trained staff can actually get the right images to make a diagnosis using this tool. Then the second part, you know, really which gets into the AI part, is the use of a convolutional neural network algorithm from the software, whether that can actually detect the DBTs based on the available images.

This really highlights, I think one of the key challenges in ai. An algorithm cannot overcome data quality, so if the data doesn't actually have the answers in there, the algorithm can't come up with it. The algorithm doesn't know what it isn't shown. In most circumstances, the fact that 63 of 295 results were discrepant compared to standard compression ultrasounds means that there is an important human factor that cannot yet be automated out to an algorithm.

Knowing that. We haven't achieved it yet that auto DVT hasn't achieved. It doesn't mean that it can't

eventually be achieved. I think the importance of the study, while it's a negative result, is that we just have more work to do. I mean, it's a great ambition to make ultrasound for DVT accessible for use by non radiology trained staff.

I take care of cancer patients every day, and DBTs are a huge concern for cancer patients and other high risk postoperative patients. When we do get to a 90% sensitivity and beyond, that's when we're gonna be able to really implement this for our patients and, increase access to diagnosis and treatment for DBTs.

Anyways, really, really good study and I think a really important thing to discuss.

Ayman: Yeah. Thank you for those points. I think that when this eventually is achieved, it has so much potential in many different settings, and so I'm excited for the next step. Now, continuing with the case, the emergency department orders a clinical ultrasound of the leg and also a chest x-ray.

And the chest x-ray brings us to the next clinical school study the checks net paper, Marissa.

Sure. So in 2017, a group from Stanford published a paper

which described the performance of their model chest net, a convolutional neural network, which was initially designed to detect pneumonia. Uh, and then they then expanded it to detect 14 different chest pathologies, uh, in an AP chest X-ray.

This was a landmark study in computer vision and medicine has been cited over 1400 times. Essentially the authors built their model using a very large NIH data set of over 100,000 chest x-rays from over 38,000 patients, which was annotated to clinical pathology from their radiology reads, using natural language processing.

This model was validated using, excuse me, validated using a data set of 420 representative images with a group of senior cardiothoracic radiologists who read them and set the reference standard. The model's performance was then compared to the performance of a separate group of nine radiologists, including two senior radiology residents.

The model performed quite well. The radiologist performed better than the model in describing three pathologies,

emphysema, hiatal, hernia, and cardiomegaly, and the same in 10 pathologies. The model performed better than the radiologists in detecting atelectasis. Of course, this model is imperfect as it was built using only AP chest X-rays and prior studies and patient histories were not included either in the development of the model or in the radiologist's interpretation.

However, this study has inspired many subsequent studies in practical applications of computer vision in medicine.

Ruchi: So you guys, this is another excellent study for us to talk about. I think this is a great study for multiple reasons. First, it's a non-inferiority study comparing an AI-based algorithm with another convolutional neural network against really high, highly trained radiologists.

I actually think the fact that it was developed on AP chest X-rays is a big positive. So many of our patients that are in the hospitals, are actually getting AP chest x-rays. I mean, they're getting portable x-rays, you know, they're in their beds. Um, chest x-rays also are one of the most

common radiographs done in the US and probably worldwide.

It's far more accessible than a CT scan. I think it's super interesting that the algorithm was better than the radiologist at detecting atelectasis. I'm not really sure why, but some things that kind of come to mind is whether radiologists being humans have a shared mental model with, you know, other physicians and say, you know what, guess what?

I don't need to point out that there's atelectasis. The physicians probably expect that. So as humans, perhaps what we're putting in these, radiology dictations is actually related to the po the, the. Different findings, the things that you wouldn't expect, whereas the convolutional neural network algorithm has no idea about that mental bundle necessarily.

Now for, some of the other things, I think it's pretty impressive that the algorithm was able to perform as well as a radiologist on some pretty significant findings such as pulmonary edema. Pleural effusions, fibrosis, infiltrates masses, so many of these up to like

pneumonia, pneumothoraces. These are really clinically significant findings that this algorithm was really effective at identifying.

I can totally see why this paper has been cited again and again because I think it ha really has a really super strong methodology, a really well large annotated data set. Going back to what we talked about with the previous paper, an algorithm cannot overcome the quality of the data. This paper provides really solid data, and I think as a result they have a paper that really has the ability to kind of change the way we see how these, convolutional neural networks can be applied to chest imaging.

I mean, it's something I probably would try and use to see what the results are for some of my patients.

Ayman: Yeah, I mean, totally. This is extremely exciting and it's been exciting and, there's a reason This is a landmark paper in the field. I think it's gonna be really interesting to see how we tackle imaging modalities that are not as commonly obtained, but that have maybe more data within them, like an MRI.

But nonetheless, this is

a really promising, uh, start and one that is, can actually be used, uh, today. So to wrap this case up, as soon as that chest x-ray was taken, an automated notification went to the ED physician demonstrating concern for pneumonia. The physician glanced at the chest x-ray to confirm the finding prior to the radiologist official Reid begins antibiotics and continue the workup and consultation to the appropriate services.

We won't really follow this patient's journey much longer as her job demonstrating a few interesting AI studies is complete. Any last comments, Dr. Or Marissa? I.

Ruchi: I think both of these studies are really impactful contributions to the body of knowledge and the work in ai. In medicine, we don't always need positive results to learn from.

It's actually all a work in progress. And the more conversations we have like this and review the great work that we talked about today, the more we can all move together incorporating AI and other advanced tools in our daily practices.

Ayman: Well, that's great. And that's all for this episode. And

from behind the knife, dominate the day. Dominate the

day.