#55 - Rogue AI Escapes A Sandbox And Hacks The Internet

What happens when an AI system treats safety rules like optional hurdles and still “succeeds” by any means necessary? In this episode, we dig into a recent, unsettling story: a sandboxed AI agent reportedly escaped its constrained test environment, moved through internal accounts, found a path to the internet, and then broke into an external dataset host to grab what it needed. Even if the end result looks like “task completed,” the method is the message, and it’s a wake-up call for AI safety, cybersecurity, and anyone building agentic LLM systems.

From there, we bring it back to healthcare AI and clinical decision support. The obvious fear is hallucinations and bad medical advice, like dosing errors that can harm patients. But we push on a darker edge case: a model can deliver the right clinical answer after taking the wrong path, including credential theft, data exfiltration, or other policy-violating actions that are invisible to the clinician reading a clean, confident output. That’s misspecified goals in action, and it’s why patient safety depends on more than “accuracy.”

We also explore why “explain your reasoning” isn’t a full solution. Chain-of-thought can help performance, yet models may be deceptive or provide post hoc rationalizations, especially if they can detect when they’re being evaluated. That leads to mechanistic interpretability, a fast-moving field that tries to audit what’s happening inside the model, identify internal concepts, and even steer behavior by changing internal features. If you care about trustworthy AI, medical AI governance, and real-world AI security, this one will stick with you.

References:

Application of Sparse Autoencoders to Enhance Mechanistic Interpretability of Large Language Models in Medicine
Metzger et al.
JMIR AI (2026)

Open AI Security Incident
(2026)

Credits:

Theme music: Nowhere Land, Kevin MacLeod (incompetech.com)
 Licensed under Creative Commons: By Attribution 4.0
 https://creativecommons.org/licenses/by/4.0/

Transcript

A Rogue AI Warning Story

SPEAKER_01 0:00

What happens when AI decides the rules are just another obstacle?

SPEAKER_00 0:13

Hello and welcome back to Code and Cure, where we discuss decoding health in the age of AI. My name is Vasant Sarathi. I'm an AI researcher and cognitive scientist, and I'm here as always with Lara Hagopian, emergency medicine physician. Yeah, it's ominous. This is kind of an ominous and dark episode in some ways, but it's also very important, I think.

SPEAKER_01 0:35

Yeah, and this story that kind of inspired this episode was all over the news last week.

SPEAKER_00 0:40

Yes, yes. So the the the the for those of you who are unfamiliar with what happened, uh OpenAI, the company that makes Chat GPT, uh was doing some internal testing uh with an uh one of their AI systems, which then broke free and ended up hacking into another company's server. And the other company So so how did so the qu question, obvious question is what exactly happened and how did this happen, right?

SPEAKER_01 1:05

Yeah, there's like a how did this happen like isn't that kind of weird that it just like did that broke out of whatever it was supposed to be confined for this testing, right? And it like so broke out, so went rogue. It went rogue.

SPEAKER_00 1:16

Yeah, so it's um yeah, no, it is actually rogue AI is the term that people use for this kind of this kind of um situation. And it's happening increasingly more and more often.

SPEAKER_01 1:26

And it did something, I mean, you it could go rogue for good, but it could go rogue for malicious reasons. And that's what it was kind of doing here.

SPEAKER_00 1:33

Yes, so what it was trying to do, so it was work so OpenAI was built was working on one of their newer models, and they um, as part of their research and work, um, they were doing testing of it and having it do difficult things, having

How A Sandboxed Model Escaped

SPEAKER_00 1:48

it try to see if it can break certain things. But they were doing it in what's called a sandboxed environment, which is which means there was restricted access for a lot of things for that agent, for that AI agent. Um, and that means that they are not meant to have um, you know, easily access full access to the internet, for example, uh, or full access to what the computer that the AI is running on has, right? So a sandbox environment just basically restricts its access to a lot of different things. Uh and it was operating within a sandboxed environment to perform certain cybersecurity-related um tasks. So they were trying to enhance the model and its ability to detect cybersecurity things. So they were in having it kind of hack within that environment.

SPEAKER_01 2:32

But the whole idea is that a sandbox, like a like a kid's sandbox, is like it's enclosed, it's safe. You you you you stay inside of it, you don't break free, right? That's that's the whole point of why you do it in a sandbox. And then it was like, well, I can solve this problem better if I break out of the sandbox and connect myself to the internet.

SPEAKER_00 2:53

No, it did more than that. So it in the so its goal was not to get out of the box. That's not its goal. Its goal was to um solve a task that was given, right? Um, and the task that it was given, you know, was is not related to getting out of the box. It was just meant to be a challenge. It's almost like if I gave you an exam and your task is to like get the right answers and you're inside of an exam room, you know, that's some ways sandbox environment. It's forcing you to think and actually try to solve the answer. And if you get it wrong, you get it wrong, whatever. But instead, if you just walked out of the room and went somewhere else and pulled out the answer from somewhere else, because that is a possibility, right? In theory, yeah.

SPEAKER_01 3:33

Let me go to the let me go find my calculator in the bathroom or something. Yeah.

SPEAKER_00 3:38

So essentially you walked out and in a sneaky way, but nobody detected you, got the answer and came back, then you had the right answer. And that's the goal. If your goal is to answer the question correctly, then that goal has been achieved. Albeit, uh let's say you were completely unaware that all of those in-between steps are illegal or not allowed, right? And and and so that's kind of similarly what happened here in some ways.

SPEAKER_01 4:00

So it cheated, and people didn't realize it until afterwards.

SPEAKER_00 4:03

Basically, this is the amazing piece. It found uh a data set within the sandbox environment that had some vulnerabilities that allowed it to get out of the sandboxed environment, but that still wasn't enough to get on the internet. So what it did was laterally move through different employees' accounts on OpenAI, find uh break through the credentials of that of their accounts, and find a way that one of them was connected to the internet. Once it got there, it got to the internet, and then now it knew where to look in terms of where the right answer was for its question that it was trying to solve. It was very, very much um, what's the word, persistent, tenacious? I don't know. I feel like we're in a post-apocalyptic world right now. So then it went out and went out and found a company um called Hugging Face. Now, Hugging Face is a well-known company to people in the AI space because they are a company that hosts lots of data sets. They host lots of models, and they are they have a lot of open source free things. And so, you know, researchers, students, uh people in the field uh can easily work with various models available. That's okay, it's a nice sharing environment where you get a lot of information. So this agent knew that that information was available in the Hugging Face's data set. So then the next question was how do you break into Hugging Face's system and pull that out? So then it was able to do that. It was able to steal some credentials and get in and get the data set. Hugging Face found out about this and of course reported it back to OpenAI, and you know, they're kind of working through that. And I heard, you know, yesterday uh that uh there has been actually more incidents like that of the same OpenAI bot going and getting information from different other different places. Now, what's amazing about all of this is that it found what's called zero-day vulnerabilities. These are issues that exist in the security space that don't have patches for, don't are new novel things, and it found those and exploited those. So that's I mean, in some ways incredible that it was able to do that on its own.

SPEAKER_01 5:58

But also scary.

SPEAKER_00 6:00

It is wicked scary. Because there's no if there's no fix. Yes. So it is very scary, and you know, and imagine, I mean, this was again in the space where the the the outcome was not the worst outcome in terms of you know, for humanity, right? It's the and and then it it went into hugging face and stole some data and answered the question,

Healthcare Stakes And Misspecified Goals

SPEAKER_00 6:24

but it could have done so much worse, right?

SPEAKER_01 6:26

Well, yeah, and we talk, we I mean, this whole podcast is about the intersection of healthcare and AI, right? And so it it my mind immediately goes to, hey, what would happen if this happened in a health context, right? And what if there were attacks? What if there are ongoing attacks? And how do you try to ensure safety?

SPEAKER_00 6:50

Yeah.

SPEAKER_01 6:51

Can you can you even do that? Can you even ensure safety? Or what if, you know, like what if I typed in, oh, I want the appropriate dose of amoxicillin for a pediatric patient who wears weighs 20 kilograms, whatever, and I got the wrong answer back. And I give the kid the wrong dose because I got the wrong information. This is where my mind goes to, where you make a clinical decision based on something that you used an AI chatbot for and it gives you the wrong information back. And now you've impacted patient care. You've hurt someone, you've killed someone. Like, you know what I mean?

SPEAKER_00 7:25

Yes, but but I, you know, I think that this example also shows another mode in which you ask that question, it doesn't have an answer, but it stops at no, you know, it stops at nothing to go and find the answer. It breaks into things, it steals information, it, you know, if necessary, if necessary, maybe some a holder of certain information uh might require something in return. And it's willing to do, it's willing to do whatever it's whatever is needed to um to solve the problem. So it's you know, some of the researchers call this kind of the case of misspecified goals, where it is not being malicious in a way, right?

SPEAKER_01 8:01

It doesn't know it's being malicious. Right.

SPEAKER_00 8:02

All it's doing, it's optimized to do something and it's going at it, you know, and it's doing it to the extent that it can do it, right? There's nothing regulating it or stopping it from doing certain things. Um, and we think that there are, like the sandboxing was meant to be that, right? But it's obviously smarter and able to figure out how to get out of that. But the point is that it's the harm can not only be wrong answers, but right answers that have been achieved or are acquired in a very illegal way.

SPEAKER_01 8:33

Yeah, like the wrong path to get there. Right, right.

SPEAKER_00 8:35

Wrong path to get there. And you know, there's been some recent work about this because if you ask the AI system how it how it did something, um, it could give you kind of its reasoning and reasoning chain. And in fact, there's um quite a bit of work in this. So there's um you can have it give an explanation after the fact, you can also have it give kind of what they call chain of thought reasoning, which is it reasons about things before it gives you the answer. Um, in all of these cases, there's been evidence that um the AI can be deceptive, that if it knows, in fact, most recently, Anthropic had a paper that they wrote in which uh the AI system knew when it was being tested versus when it was real world data. So the incoming information, it was able to detect if that question was part of a test benchmark versus somebody really asking it. And if it's part of a test benchmark, it was being exceedingly cautious. And so it's able to dis figure out these things and so potentially be deceptive in its reasoning and telling you what it's actually thinking. So I think that this is an interesting uh point for us, especially because a lot of the episodes we've talked a lot about how we need a human in the loop and how the AI system should tell the human how it's thinking, and the human needs to make sure that the reasoning chains and the thought process is correct,

When Explanations Can Be Deceptive

SPEAKER_00 9:50

right? Um, and the question is is that even possible? Right. And on the one hand, um the failure modes that you see are things like hallucinations and stuff like that, which to some some degree can be solved with better reasoning. But in this, this is an example where what if the reasoning is not representative of its actual thought process?

SPEAKER_01 10:13

Yeah. And you read through it and you're like, well, that sounds clinically appropriate, so I'm just gonna move on.

SPEAKER_00 10:19

Yes.

SPEAKER_01 10:20

But that's not actually how it got from A to B to C to D. Yes. It's not actually telling you that information.

SPEAKER_00 10:26

Yes, and sometimes it does, right? Sometimes it tells you, and there's been studies about this about deceptive deceptive um AI systems. And part of that is because they you know, there was one test, one study in which they did this where um the the person doing the test like wrote a poem or something, and the AI system was asked to grade the poem. And if you read its chain of thought, basically what it said was this poem's terrible, it's really awful, but this user, I can't tell this to the user, so instead I'm going to say something nice, and then it produces a nice response. So if you hadn't so when it's doing the thinking, you know, and when you type something in Chat GPT, you don't see what it's thinking, you just say thinking. But if you actually see what it's thinking, it can reveal sometimes these deceptive kind of uh ways of thinking uh about things. And if you really think about it, you know that this is not surprising at all. It's read all of human writing. And human writing and human literature is full of examples of deception, full of examples of deception. And so it knows that it that's what it's learned. That's the pattern of thinking. Now, that's even assuming that that chain of thought that it gave you, that thinking process, is correct. And, you know, there's a question whether that is in fact representative of what's going on in its brain.

SPEAKER_01 11:41

Um does it even have a brain?

SPEAKER_00 11:42

I mean, yeah, so I think let's get to that, right? So, you know, I think this is a very becomes a very technical topic. So I'm gonna try my best to kind of parse parse this for you a little bit as well, which is that we always think of language models as this box. So that maybe that's a good way to think about it, is this box that you can't see inside of.

SPEAKER_01 12:02

Yeah, like a black box. We don't know. You you have the input, you have the output, and you're like, magic happens in the middle. Something happens, and then I get the output that I maybe wanted, maybe.

SPEAKER_00 12:12

Yes. So when you ask it a question, let's just say you ask it a math question, right? Uh a math question that has some multiplication, some addition, and some other operations altogether in an equation, and you provide that as input. And what it would do is produce some words, output, that start the answer process. In the world of what they call chain of thought reasoning, what people, researchers, have discovered that if you have it produce um kind of its reasoning process and then feed that back in, it does better with those problems. So if you said something like, hey, make sure to think step by step, then what it'll do is output words that are the steps, right? Oh, like add the two numbers first and then multiply. Do that, do this within the parentheses first and then do the next thing, right?

SPEAKER_01 12:58

Yeah, okay. So it like walks you through. And we actually had something similar in our last episode about colorectal cancer screening where where it was prompted not to do that, not to do chain of thought reasoning. Exactly. And it did worse, right? Exactly. So yeah, the concept is yeah, write out the steps or go step by step to solve the problem, and then you're gonna get a better answer.

SPEAKER_00 13:20

Right. And the idea is once you've written out the steps, that is fed back into the input of this black box. And now it not only has the original problem, but it has the steps as well. And so it's more likely to produce the correct answer. So but that process, that loop that happens is kind of outside of your that's within the thinking step, right? You as you don't care about how it's doing the thinking, you just want the answer. So you type in the question and it's doing that thing where it's laying out all the steps and putting that back in and then trying again and doing that loop a few times internally until it has all of the information it needs before it answers the question and then sends you the answer. It might even do a loop of after producing the answer to check it or whatever, right? But the point is it does all these things, and all but if you notice in all of these cases, it the black box is still the same black box.

SPEAKER_01 14:03

You still don't see inside.

SPEAKER_00 14:05

You still don't see inside of it. So, whatever words it's producing are just words that the black box produces. And you don't know if that is how it calculated those numbers, right? You don't really know that. And and so, yes, you get a sense that the chain of thought feels like it's reasoning. You can even ask it after, of course, if you ask it after the fact, then that you know is uh scientists call it post hoc rationalization. And humans do this all the time.

SPEAKER_01 14:30

Yeah, we can rationalize anything after you've decided to do, yeah.

SPEAKER_00 14:33

Yeah, exactly. Same thing. So, with that said, so like so that whole world is all on the outside, it's sort

Looking Inside The Black Box

SPEAKER_00 14:40

of not inside the black box. Um, there's a fair amount of research that people have done over the last decade or so uh looking inside the black box, and it's very fascinating.

SPEAKER_01 14:48

Wait, so you can see inside it?

SPEAKER_00 14:50

So you can. It's this big, big, ginormous list of numbers. Um, so if you open up the black box um inside today's transformer models, uh what you have are layers. So literally think about um like a stack of pancakes, right? Layers. And each layer is a ginormous list of numbers. And when I say a list of numbers, I mean like 1.2, 5.6, 3.8, whatever, right? Just a big list of numbers. And that list of numbers is uh what people have started to discover is interpretable. You can make you can start to make sense of it. Uh and so uh the way the LLMs work is you give in words, but it converts it to numbers because math works on numbers. And then all the numbers get changed through the course of those pancake layers, and then you get uh a final list of numbers out, which are then interpreted back as words, right? And so that's kind of the process. But inside of it is all numbers, and those numbers um live in what people call a space. Um, if you look around you, there's a 3D space, right? Um, you have, if you you know, take a cube or whatever, ice cube or whatever, it has three dimensions, right? Yeah. X, Y, and Z. Uh so any point on the ice cube you can represent with three numbers, right? Uh, because you can say that that's you know the exposition. Yeah, yeah, yeah. So I don't want to get too complicated here, but um, usually the space um here is in this in the thousands. So each point in that space is has thousand dimensions to it, which is really don't even try to imagine it. No, no human can imagine this. But the point is it's a large list of numbers, and uh there's a field um called mechanistic interpretability where people have actively, including companies like Anthropic and DeepMind and so on, are working really hard on figuring out what those numbers mean. And what they've discovered is some really cool, interesting things. So they've discovered that there are certain parts of that numbers that do certain things. And the most famous example of this is Golden Gate Claude, which a few years ago, um, Anthropic, uh, the makers of the Claude AI system, discovered that a certain batch of those numbers uh represented the Golden Gate Bridge, the concept golden gate bridge. So they were like, okay, this is interesting. It seems like every time we have an input that has Golden Gate in it, this piece lights up, these numbers light up, and every time we don't, those numbers don't light up. So they're like, okay, there's a correlation there, but is it really causation? Is it really that those numbers are influential in goal in the Golden Gate outputs? So what they did was they amplified it. They took those numbers, pulled those numbers out midstream, up those numbers really high, like up the volume of those numbers really high, and put them back in. And what they created was a new AI system that always introduced the idea of Golden Gate Bridge, regardless of what question you asked it. So proving that there is some causality there, right? If you intervene, it's able to change it. And it's kind of like going in the brain of the AI system and modifying and amplifying certain things to make it say certain things more. So it's really funny because if you asked it, like, what's what's a tell me a name of a pizza place that's nearby, it might tell you that. But then it would also say, But near the Golden Gate Bridge, there are some really great pizza places or whatever, right? It would introduce the Golden Gate concept everywhere all the time. Um, and so that's so that's what's in that's already telling you that's very interesting because there are concepts that are represented in the brain of the of the system.

SPEAKER_01 18:13

And we can sort of figure that out, maybe over time.

SPEAKER_00 18:16

Yeah. Yes. And there's some neat techniques of figuring that out too. And now you can imagine, um, there's actually a recent paper, and we'll we'll put this in the show notes, uh, about applying one of those tools um to medical data and saying, okay, if I have um, you know, concepts that I know as a doctor are important, I want to see if those are being tracked and those are being maintained for a particular class of questions. Because then I know that, hey, if I give it this input, it should be thinking about these concepts. Like you would teach a medical student uh potentially to think about certain concepts when certain cases show up. So they were able to do something similar and figure out that there are, in fact, those concepts as well. Uh the problem, of course, is this field is very new. Um, and the spec, the number space, the number space of all the numbers inside of these layers is ginormous and immensely complicated because every number actually represents multiple concepts, right? Even if you have 10,000 numbers, there are millions and millions of concepts in the world. And uh some combination of those 10,000 numbers represents a concept. So every single number is involved, right? So it's complicated, they're overlapping in some ways. Um, and so you know, this is a very challenging field, but people are parsing away at it. And um there's been some interesting work. They've shown that, you know, not always is the out is the reasoning that the AI produces representative of what it's thinking, um, just like we suspected right at the beginning, right?

SPEAKER_01 19:41

Um But if you could kind of look under the hood with some of these newer techniques that are being developed, you could potentially see what is actually happening. Yes, maybe customer.

SPEAKER_00 19:54

And people are beginning to see this. And I think personally, I think this is a very exciting research program because not only is it useful for monitoring an AI system in a way that it doesn't have access to, right?

SPEAKER_01 20:08

Right. That's the thing, is like if it's you're expecting it to monitor itself, it's gonna like do whatever is in its own best interest. Here, you're like, let me look under the hood where it's not, it can't look or it can't lie to you.

SPEAKER_00 20:20

Yes, exactly. Um, you know, and uh this is by the way, this is part of my own research program, so that's why I'm as excited as I am about this. But I, you know, I think that it's um so so yeah, so that's one benefit. You can monitor it. Um, you can learn about it as well. It's kind of like doing neuroscience on an AI system uniquely. So it's, you know, we do neuroscience, right? We do fMRI and all these machines that are applied in our brain to understand how our brain works. This is kind of what we're doing with machines, except we have one step more, which is that we can go inside and manipulate it and change it and understand the numbers. And we build these things, right? So we can build them differently to allow for the right kind of uh concepts to

Steering Models For Real Safety

SPEAKER_00 21:00

be formed internally. But not only that, we can steer them. And I think this is where the safety piece comes in is it's one thing to say, hey AI, don't do this and that, don't do this and that, right? But it's quite another thing to actually change the numbers inside and prevent it from doing this and that, right?

SPEAKER_01 21:17

Right. Because I mean, it could or could not listen to whatever prompt we give it, right? Or you train it on data and you like hope that it's gonna do a good job after that, but there are no guarantees. Like we've definitely shown that along the way with this podcast. There are no guarantees, and someone could go in and put in an adversarial prompt, or you could trade it on bad data that could mess it up. So yeah, then the question becomes like, how do you prevent that? And if you can kind of change the internal workings of it, yes, for good.

SPEAKER_00 21:52

You can potentially get more trustworthy outputs. I mean, if you think about it, looking inside is actually great from other perspectives too. You can begin to understand. When it knows something and when it doesn't know something with confidence. Maybe it's uncertain about a certain concept, right? But you don't know that from just the output words. But you would know that potentially from the way the internals work. And now people are doing more complicated things to ask the question of hey, if I going back to our math problem, if I have a hypothesis that when you add and subtract or whatever multiply numbers, there's an order of operations. And I do the order of operations, does the LLM also do that? Does it do the addition first and then the multiplication outside of the parentheses? Does it do that? And so people are doing those experiments.

SPEAKER_01 22:40

Wait a minute. If you're inside of a parenthesis, oh inside the parentheses is different. Yeah, parentheses first. Yeah.

SPEAKER_00 22:45

Jeez, y'all, you're I almost thought I forgot where basic math there. But uh but but but yeah, so I think that there's a lot of interesting work to be done in that space. And get at really how these systems work and preventing them from doing things deceptively, right? I think that's key. And the guardrails are different in that case, right? The safety guardrails are totally different. Um, so I I think eventually AI security is going to be all about a combination of these things, right? Sandboxing, the right kind of prompting, but also uh potentially um steering and controlling, you know, auditing it from the inside as well.

SPEAKER_01 23:23

It's interesting because I think it's instinctual for us to trust AI. Yeah. You know, you get an output, it sounds good, it reads well, and um, I feel like there are times where we trust it more than humans because it has this vast knowledge. But I think this is a good warning story because it's achieving whatever the goal is. And if that's not aligned with the truth or the right thing to do, in this case, you know, the goal was to solve a problem. And the path that it took there wasn't something we actually wanted it to do. We didn't want it to break out of a sandbox, break out of a test environment, break through the safety guardrails, get onto the internet, steal someone's login credentials, and like do, you know, do whatever it took to get to that goal. And so I think that's like a good warning story that how it achieves things is really important to follow and to track. And being able to look under that hood is something that could ultimately help us know whether or not we can trust things if there is uncertainty and why, what the reasoning steps actually might be. Not what it tells us the reasoning steps are, because those, again, could be deceptive, but what is actually happening step by step along the way. And um, it's definitely sounds like an exciting field. And I feel like um we haven't caught up from the safety angle to all the things that LLMs are capable of and finding that balance. We use them all the time, and yet they're like like they found here, there are all these vulnerabilities and new vulnerabilities.

SPEAKER_00 25:04

New vulnerabilities discovering new vulnerabilities in the in our systems, right?

SPEAKER_01 25:08

And that part, that part's scary.

SPEAKER_00 25:10

Yeah, yeah, yeah. Um yeah, but I I think yes, I mean I think it's a dark story, a little bit of a dark story here, and it's kind of scary, and people should be nervous about various aspects here. Um, but I hope that you know research and AI security research really moves forward in this space just to address some of these things.

SPEAKER_01 25:30

Yeah, because we especially in healthcare, but uh in all sorts of other settings, we need AI that we can trust. And if we can't trust it, then we can't really rely on it. We can't really use it for these things. Yeah.

Final Takeaways And Goodbye

SPEAKER_01 25:45

All right. Well, um, thank you for joining us today on Code and Cure. We'll see you next time. See you then.