[2:50] Juan Andres Guerrero-Saade of SentinelOne

Eric: I have to announce something to our audience. Aside from our top 30 Federal Influencers Award three years in a row. By the way, subscribe, tell your friends and please leave us some feedback. We get so little feedback through the podcast platform and the mechanisms provided so leave us some feedback.

We have a guest from SentinelOne. My wife works at SentinelOne as some people may know. She was nominated for the CyberScoop 50 recently and she was put in as Most Inspiring Up & Comer. Who do we have today from SentinelOne?

Rachael: We have Juan Andres Guerrero-Saade.

Juan: It's so complicated that most of the people I know in the industry have decided to go by Jags.

Rachael: Jags is a Principal Threat Researcher at SentinelOne. He's also an adjunct professor of strategic studies at Johns Hopkins School of Advanced International Studies. He worked as a senior cyber security and national security advisor to the government of Ecuador. His joint work on Moonlight Maze is now featured in the International Spy Museum's permanent exhibit in Washington, DC.

Eric: So Jags, you're in the Spy Museum.

Juan: Yes, me and Thomas Rid, and a couple of other folks. We're actually holograms in the Spy Museum, just trying to explain code similarity and all this attribution stuff. They did a great job, it's very cool.

Eric: I was working with a coworker years ago and there's an InfoSec guy out there called Jester. He had to transport Jester's laptop to the Spy Museum because nobody knows who he is. It's a big thing to be in the Spy Museum and they just redid it in DC.

Cyber Espionage Campaign

Juan: If you haven't visited, the new building is fantastic.

Rachael: Admittedly, I had to look up Moonlight Maze. 1996, first widely known cyber espionage campaign in world history. The value of the information stolen according to congressional testimony was in the hundreds of millions of dollars. If you were to print out and stack all of the information that had been taken, that paper stack would be three times the height of the Washington Monument. This is a big deal.

Eric: Nowhere close to OPM or Sunburst or the like, but go on.

Juan: Well, it's a different era. The internet of the '90s is a very different landscape.

Eric: Let's go back to the cold war of the '80s. What would it take, let's just pick on an American spy, to carry that type of content out of Russia? The height of the stack of paper.

Rachael: It was three times the height of Washington Monument and the Washington Monument is 555 feet tall.

Eric: So we're talking almost 1800 feet of paper, eight and a half and 11, stacked up. Think about how long it would take a spy organization to walk that out of a building. That's a lot of content.

Juan: The closest thing would have been the Mitrokhin Archive. It's an interesting story totally unrelated to this. One of the guys that worked at the KGB archives was taking notes of everything that he was transcribing. He saved it all for 20 years until they finally got him out of Russia. There's no corollary as far as cyber goes. The amount of information that you can steal in one go is astounding.

Eric: Just a click of a button.

Juan Andres Guerrero-Saade Talks About Hands-on Keyboard

Juan: In the '90s it was a little more involved. It’s really interesting to go antagonistically in the research. We work on all these APTs, all these cyber espionage campaigns, all this recent stuff. You get used to a certain level of automation.

You’ll get used to all the facilities that come with the modern internet. You're looking at an operation that's evolving from its infancy in the late '90s. It looked very different. You're talking about a hands-on keyboard.

There's no command and control servers sort of orchestrating the whole thing. They're trying to code their way through this. You see broken tools getting deployed all the time then trying to grow as they go along. It’s like watching the birth of a threat actor.

Eric: I bet most of our listeners have no idea what Moonlight Maze was. They weren't in InfoSec at the time. A good percentage may not have even been alive at the time. It's like 9/11 or how long we've been in Afghanistan. You go back and it's a generation at least at this point.

Juan: With your Fed Award, you might actually have a good amount of the people that dealt with Moonlight Maze from the Air Force and NSA. But for the most part, it is a forgotten phase of “cybe war”.

Eric: The beginning, yes. If anybody wants to understand the time period, there's a book, The Cuckoo's Egg by Clifford Stoll. A DOE IT person essentially. I've talked about it on the show. The Cuckoo's Egg was the late '80s, early '90s, when Clifford wrote the book. I don't even know what he's doing, he might be retired at this point.

Juan: He is.

[08:36] Massive and Unexpected Attacks

Eric: It'll give you some framework for the time which is very different from now. These types of attacks were massive and not expected, like they're almost expected every month at this point. We're recording the week that T-Mobile lost 50 million personnel records or something?

Rachael: Something around that.

Eric: Nobody even cares anymore.

Rachael: I know. I'm a T-Mobile customer and I'm like, yes, whatever.

Eric: What are you going to do?

Juan: It's the breach of the week.

Eric: Can you put us in that timeframe? Obviously you were a little younger back then, the world was different, InfoSec, IT, you name it. There wasn't really much cybersecurity out of NSA and a couple of companies.

Juan: It's a really interesting landscape. There are two ways to approach this that are kind of foreign to most of us. One of them is the state of the internet at the time. You're talking about mostly university research centers, military computers, that kind of stuff. Then some early adopters tried to get into the scene.

But it's not at all like what the internet looks like now in its proliferation and its number of users nor in its uses. For the most part, you're storing research and databases and government stuff. So in itself, it's a very different target environment and from the cyber espionage side, it’s entirely undeveloped.

We have rumors of early Israeli operations, we believe the NSA was already operating at that time. I like to call it the League of Titans. We've got the folks that were doing Moonlight Maze which has a connection to a modern threat actor called Turla.

Juan Andres Guerrero-Saade Recalls the Folks From the Equation Group

Juan: Then you've got the folks from the Equation Group which we've come to know as some function within NSA that were also around in 1995. So you basically have a drastically under-populated threat actor menagerie.

Eric: When did it start? What year really? I know '98, '99 is a big year.

Rachael: Wikipedia said '96 but I don't know if that's accurate.

Juan: Moonlight Maze started somewhere in '96, as far as we can tell.

Eric: So Windows 95 is out, most people are still on Windows 3.1 maybe, we're talking NT 4.0, NT 3.5. Most of our users haven't heard of this stuff.

Juan: That was our initial assumption, it gets more obscure. Our ability to do this research, I kind of have to tell a bit of the backstory. It comes mostly from the doggedness of Thomas Rid. He is a full-time professor over at SAIS and a brilliant researcher, a fantastic author. And he and I talked years back.

He was very much focused on this idea of, what happened with Moonlight Maze? Why have we never seen anything technical come out about Moonlight Maze? He started filing FOIA requests and trying to follow up with everybody involved. Just kept pressing until he found a bit of a redaction error.

One of the documents, basically, redacted the name of a company that had been compromised. But it didn't redact the name of the person managing it, it's one of those two. It's either the company or the person, prints his name and Thomas was able to contact this man.

Juan Andres Guerrero-Saade Tells the Story of David Hedges

Juan: An older gentleman called David Hedges, a super nice guy who had been managing this system for a UK company that got compromised. It was being used to route part of the attack to the United States. As luck would have it, he had the machine, he still had the machine under his desk.

It was basically his willingness to hold onto all this stuff allowed us to do all this research. He’d been asked by the FBI at the time whether he would be willing to let the hack continue and essentially watch everything that went through there. He did but he also didn't get rid of any of it afterwards.

Eric: Nobody thought to ask him?

Juan: Well, yes, you know, the FBI didn't do their homework on that one. One of the tragic things of this is there's a notice that Thomas uncovered first. It says as part of standard procedures after a certain amount of time, we have destroyed all of the evidence that we had collected.

It was a gut punch for us in the early days of our research. We were like, unless you're in the NSA or GCHQ we're not going to get anything. Then Thomas stumbles upon David who was just sitting on this treasure trove of fossils. That we could essentially reconstruct a good portion of the attack from.

Eric: When were you doing this with the construction piece?

Juan: I'm going to have to think back, all of time has sort of blended into one giant everyday. But I believe we were doing the research around 2016, 2017. I might actually have to Google it myself.

After the Cold War Ended

Eric: No worries. But, this is all happening after the wall fell, five years after the cold war ended. It's still underway. I suspect that most people in government weren't thinking about cybersecurity back then. We've got to protect this info. People can walk through our walls and just get in here from keyboard strokes.

Juan: It was a rude awakening on a variety of levels. For one, it essentially kicks off establishing things like JTF and other functions within the US government to respond to this. This is a big wake-up call. It’s also because someone eventually decides to brief Congress.

Of course it leaks and it becomes the first rally cry, including a Newsweek article that said, we are in a cyber war. It was the beginning of that cyber Pearl Harbor hyperbole style of taking on these things. But it's also a really interesting time.

You mentioned Cliff Stoll and what Cliff was onto. He's kind of the patron saint of threat hunters because it's the late '80s, or early '90s. More than anything, he doesn't have any of the tools available that were used. You're not talking about firewall logs or SIMs or AV or EDR, nothing.

Eric: If you read the book he's got the CIA involved but they really don't care or aren't doing anything. They're not sharing with him. He's a government employee, he's at the department of energy. Was he at Lawrence Livermore?

Juan: Yes.

Eric: In Berkeley, he was in that area. He had nowhere to go but he's watching this behavior, it's never explained. That was a pretty good book, I have to admit, but it's a decent read. It's a little detailed, but yes, it's a different time.

[15:54] Juan Andres Guerrero-Saade Points to SANS CTI

Juan: It's really interesting. For folks who have not been exposed to it or even for big fans of The Cuckoo's Egg, I would actually point you to a more recent talk. SANS CTI in 2015, 2016, Cliff Stoll came back and he did a keynote talk for this conference. I had the pleasure of being in the crowd, he's an incredibly animated speaker. To the point where he was jumping around he basically disconnected the projector but he showed up.

Eric: Can you imagine him back then when he couldn't get anybody to listen to the fact that there's people inside the energy?

Juan: The amount of energy this man has at his age is fantastic. He literally showed up with the same slides. The old-timey projector slides that he used to explain to the NSA what was happening. He just pulled them back out and kind of went through them.

Eric: The flip chart projector slides.

Juan: I don't even know what you call them, but yes, it was fantastic.

Eric: What was that called Rachael? The overhead projectors. Remember you would put the film-based slides down, you can write on them with the right markers.

Juan: He still has them. I don't know where they sourced this projector for him. So I would point to that as a must-see, it's probably the best keynote talk I've ever seen.

Eric: Back to Moonlight Maze, you're doing all this work, you hit the mother load. Where do you go from there?

Juan Andres Guerrero-Saade Closes Off the Thing With Cliff

Juan: To close off the thing with Cliff, the reason I brought it up is, we didn't understand at the time that he was seeing these German hackers. Who were stealing American documents to sell them to the KGB for some combination of drugs and money. At the time we were not really cognizant that this could happen.

With Moonlight Maze it comes at a time when the US is already in a very covert fashion taking on that same activity. Someone in Russia figures out, why not go for this ourselves? So what we see and you asked, what did we feel at the time? The idea of getting our hands on this material was, if there’s such a thing as a miracle in threat intel I think this is it.

We found more detailed information for that incident than we usually get for most modern investigations. You had on keyboard logs, all kinds of tools, you could see how they were deploying things to the different victims. Danny Moore who worked with us on this, he's over at Facebook now. He was able to reconstruct this whole cloud of all the IPs connecting to each other.

Figuring out how they were routing themselves through these different systems. Costin Raiu and I spend our time reverse-engineering the different samples. I’ve told you that it was a little more obscure than Windows NT and whatnot. These were actually SPARCstation, Solaris systems, ERIX systems from back in the day.

Eric: SGI, so I can hang Rachael.

Juan: I was seven years old when this stuff was being coded, it was entirely new.

How Does Juan Andres Guerrero-Saade Take Solaris

Eric: I guess you've been in this one. How do you take Solaris? Back then it's probably Solaris 7 or Solaris 8. How do you take IRIX and actually even do anything with it?

Juan: Thankfully, IDA Pro will battle anything you throw at it.

Eric: That's a tool you're using for reverse engineering?

Juan: That's the tool. Until Ghidra came out it was the tool and I think that it still is. But essentially, the harder issue was not disassembling these things. I was entirely foreign to this type of assembly. I’ve had to sit down and basically learn a whole new form of assembly to understand these different binaries. Try to figure out what it is that they're doing.

Thankfully I had Costin Raiu who has always been a mentor. He's much more experienced in these sorts of things to help guide me. But we had a ton of stuff to reverse, so it took us at least six months just to deal with the samples. Figure out how that toolkit was being iteratively developed, what it was that they were trying to do, what was going on.

The greatest finding of the whole Moonlight Maze parallel construction that we got to do was realizing that these guys who for all intents and purposes were skiddies, were script kiddies at the time. They were just testing out different tools and what they could get their hands on.

Eventually, they start to catch their stride and develop one set of tools that really worked for them. Developed it better and got closer to what we now would think of as a malware family. The interesting thing was they built on top of a publicly available backdoor called LOKI2.

Build, Build, Build

Juan: We saw them start to iterate on that, strip aspects of it, improve on certain aspects of it. Build, build, build, and then our visibility ends. There's a period when this leaks out of Congress and the Newsweek story comes out. They freak out and burn all of their infrastructure, including the server that we'd gotten access to. So at that point, we got cut off.

Eric: So they reach into the server which is in somebody's house?

Juan: It was in an HR company in the UK.

Eric: It's part of their infrastructure though and they basically burn it all down.

Juan: They burned everything.

Eric: They, being the Russians?

Juan: Yes, for all intents and purposes. We had these connections going back to Cityline which was a Russian ISP at the time, everything pointed. They tried using proxies, that's what this company in the UK was. They hacked this company and used it to route themselves so that the attacks would look like they were coming from the UK rather than Russia.

But eventually, that mask falls apart. Where it gets interesting is, that tool that we were watching get developed doesn't disappear. As a testament to the compatibility of POSIX systems and Linux and still working on the same elements. It looks like they continued to use that same source code up to now.

Eric: Why would you recreate it if you don't need to?

Juan: I was in disbelief to consider that you could have a malware family working 20 years later. In Windows it would be impossible. But in Linux, they took the same source code that they had continued to develop over the years for these Solaris systems and recompiled it for Linux eventually.

[22:56] Juan Andres Guerrero-Saade Has Already Seen It

Juan: We had already seen it, we just didn't know what it was, we didn't know how to connect it. It's something that researchers at Kaspersky discovered around 2015 called Penguin Turla. You might know Turla, a really well-known cyber espionage Russian actor. They've been behind a lot of very notable attacks including DoD systems, military systems.

Eric: Yes, a lot of governments.

Juan: They are very much an old school proper espionage organization. You've got the Bears that come around like bulls in a China shop, like Sofacy, APT28, Fancy Bear, whatever you want to call them. Then you've got the pros that are actually just stealthily watching embassies, watching different ministries of state, and so on.

Eric: It sends a little respect there.

Juan: My blog is named after them. I love these guys, they just do fascinating work. But they used something called Penguin Turla around 2015 and they continue to use it sparingly over the years. What we figured out was, when they were having a hard time with an intrusion, whenever somebody was starting to clean them out of a network, they would grab a Linux server somewhere on that enterprise.

Hide this little backdoor and they would get cleaned out. They would wait three months or whatever. Then they would just come right back in through that Linux backdoor that most folks didn't catch. They would just repopulate.

That Linux backdoor was compiled from the same source code that we were seeing develop from Moonlight Maze. You have this perfect connection of 20 some years from Moonlight Maze to the modern Turla that we continue to deal with, which is just mind-blowing.

The Operating Systems that We Work With

Eric: If you look at macOS it comes from NeXT OS which comes from Unix. If you look at Windows, what are we up to now? Windows 11?
Juan: Soon to be 11.

Eric: I can still see in Windows 10, which I don't do a lot of, remnants of DOS, and early Windows 95 and Windows 3.1. The operating systems that we work with still go back 20, 30 plus years. In the case of Unix, we're probably talking close to 50, 60 years now. Unix was the late '60s if I had to take a guess. The code still works. You don't hear about this often but why wouldn't you just keep using it if it works? If nobody shuts you down, why not keep using it? We do it on the operating system.

Juan: I say that this is more possible in Linux where POSIX standards are much more important. Folks are continuing to maintain the same. OpenSSL has been around for a billion years. You just kind of iterate on it which is why it's a freaking mess but it continues to work.

You couldn't do that on Windows. Windows has a lot of things that continue to look like their old versions. But if I took malware from the early 2000s and tried to run it on Windows 10, chances are it's just going to crap. The DLLs aren't going to work the same way, the services don't work the same way.

Eric: No. When I go to edit the registry it's like back in the day when I was an MCSE on Windows NT 3.5, 3.51. It's still the registry.

Juan: Yes, the structure is still there.

A Cat and Mouse Game

Eric: Terminal, Command Prompt, there's a lot. I've got macOS books, Unix books downstairs that still work surprisingly because I forgot everything. You can still run VI. I forgot it all.

Juan: Well, another version of that. Speaking more to the security industry and the way that it's evolved over the past 10, 15 years. There's been a lot more of a cat and mouse game on Windows. There was a greater consciousness about viruses on Windows. Then the antivirus industry started to evolve from the great figures that we've known from back in the day.

Eugene Kaspersky was there and the folks from McAfee. I won't say that John, rest in peace, as a figure, he didn't sort of withstand the test of time. But we have these sort of luminaries that started the AV industry. It was all about a new virus that has come out. All these different folks around the world are doing their best to best it.

That evolves into the industry that we know now, where you have hundreds of thousands of unique samples coming in all the time. We've tried to develop more automated systems that deal with them. All of that is largely rooted in the Windows battlefield. Linux and macOS have kind of flown under the radar.

Not because there aren't threats for either of them but rather from a lack of visibility, from a lack of adoption. Honestly, some snobbery on the part of Linux administrators. Who seem to think that these things can't affect them even though it's quite clear that they do. In a sense, the evolution of security tooling under the hood of Windows has been battle-tested.

Juan Andres Guerrero-Saade Shares What Happened Between the Predator and the Prey

Juan: It's been this natural evolution that's happened between predator and prey. Whereas, Linux is really lagging behind in that sense. They adopt security measures just out of, they want to.

Eric: You're saying on the defensive side, the white hat side. But really the bad guys, the adversary, they don't care. They'll pick whatever platform works for them.

Juan: Whichever they have to. If I know that I want to target you and you've got an iPhone then we know what the stakes are now. I'm going to go to NSO, I'm going to pay them a million dollars, and boom we've got Eric. I am not going to spend all my time trying to figure out Windows malware if you don't use it.

Eric: Now, I've been in the industry 20 years and it's like Linux doesn't have a big enough footprint. The address on Mac is too small, we're not going to have a Linux client capability. It's like leaving two windows in a house open but everything else is totally bolted down.

Juan: It's absolutely ridiculous.

Eric: There's an 18% number which is where a lot of the adversaries look for mass attacks. When a platform goes above 18%, I'm sure that number changes. It becomes attractive from a monetization perspective. That's not a nation-state, that's hacktivists, people out there for money.

Juan: It's a very outdated way of thinking about things. If every server and cloud system on earth essentially is built on Linux in some form or another and the idea of monetization has changed drastically. What has fueled the ransomware epidemic but the ability to exchange value through cryptocurrency. You can mine cryptocurrency.

[30:09] Can Juan Andres Guerrero-Saade Deploy Crypto Miners?

Juan: The only reason that you shouldn't mine cryptocurrency at home is that it's inefficient. You don't want to pay the light bill. But if I can deploy crypto miners to a bunch of AWS instances then, what do I care? There's definitely a whole side of that that we're ignoring and that's the large scale.

On the smallest scale it's like, my router runs Linux. Then there's these Mirai botnets that at times have taken down entire swaths of the internet because of the lack of security on those things. So yes, we treat them like edge cases but it's kind of ridiculous because it supports our whole infrastructure.

Eric: But from corporate America's perspective, it's hard to monetize in many cases. There just aren't as many nodes out there, if you will, systems.

Juan: Some folks are getting ahead of that. I would also expect or hope that customers get a little more savvy in what they ask of their vendors. I try to urge customers to ask for this, ask for something better. Look at the DNC. It's such a contentious issue to talk about what happened in the summer of Sofacy 2016.

But the DNC, if you read the CrowdStrike report carefully, realized that APT28 is there or Fancy Bear or Tsar Team, whatever you want to call them. They have a million names. They’ve realized APT28 is there and they clean all the Windows machines.

They don't realize that there's an X agent sample on a Linux machine. And they repopulate exactly the same way that we were talking about with Turla. Let's not cross trek in particular. Most folks in the industry just aren't paying attention to Linux the same way that they should.

The Chink in the Armor

Juan: It's situations like that where you see the chink in the armor. Where just one machine sitting there is enough to keep that beachhead going. Keep that infection going for way longer and then we see the effects that that has in horrible ways.

Eric: You can be 99% perfect but that 1%, that one machine, you have to have perfection in many cases. You're doing Moonlight Maze, you've done the research, where does it end up? How do you end the story? We have another amazing story coming.

Juan: There's quite a few. I've had the privilege to work on a lot of interesting cases in my career. We can talk about them for as long as you want. Moonlight Maze, I was really happy to see how it ended up.  I got to, first of all, go on stage at SAS which is one of my favorite conferences with my friends. My co-researchers at the time, Thomas Rid, Danny Moore, and Costin Raiu, we got on stage there together. Had a drink together over the machine and got to tell the story.

Better yet, the Spy Museum as they were doing this sort of redesign. They got that brand new building, an amazing site in DC. They’ve dedicated a whole section to cyber espionage, cyber war, sort of the development of things in the cyber domain. Apart from giving us an opportunity to explain some difficult concepts as wonky holograms, they actually took the server.

David Hedges was kind enough to ship them the original command and control server from Moonlight Maze. It's up there in the exhibit. If folks ever get to escape COVID madness, I definitely recommend you go see this machine that filled a thousand hacks.

Rachael: What a great story. With that, let's call it the end of part one and bring people back next week for part two. And if you subscribe, you get it directly in your email box. That's right, on Tuesday.

About Our Guest

Juan Andrés is a Principal Threat Researcher at SentinelOne and an Adjunct Professor of Strategic Studies at Johns Hopkins School of Advanced International Studies (SAIS). He was Chronicle Security’s Research Tsar, founding researcher of the Uppercase team.

Prior to joining Chronicle, he was Principal Security Researcher at Kaspersky’s GReAT team focusing on targeted attacks and worked as Senior Cybersecurity and National Security Advisor to the Government of Ecuador. His joint work on Moonlight Maze is now featured in the International Spy Museum’s permanent exhibit in Washington, DC.