[0:37] Getting Into the Software Vulnerability and Supply Chain

Petko: Today, we're going to be talking about, I guess my favorite topics, which is just supply chain in general. I think we underestimate how much the supply chain could impact us, and we always think of the supply chain as physical. But in this case, we're going to talk about the virtual supply chain and how it applies.

Rachael: Yes. And we're going to talk about breaking things too. So without further ado, please welcome to the podcast Jonathan Knudsen. He's the head of global research for the Cybersecurity Research Center at Synopsys. 

Jonathan: Thank you, Rachael. Thank you, Petko. I'm glad to be here.

Rachael: Now, I mean, where to even start? Because we're going to talk software, we're going to talk supply chain, we're going to talk breaking things. But should we start at the beginning, Petko?

Petko: What's the beginning? I mean, we got March Madness right now. Can we start there? That's the beginning, right? Everything starts with sports.

Rachael: That's a really good point. We are in the middle of sports betting mania, and I think you guys recently put out our research support on such things. Yes?

Jonathan: We did. I was going to say, speaking of broken, how about everybody's brackets? But yes, we'll leave that alone. Yes, we did a research report. We looked at sports and betting apps in the Android store, and the findings were pretty interesting.

I guess before we do that, maybe we should talk about how people make software and then that'll maybe help us understand what the supply chain problems are.

Rachael: Absolutely. Because I think it'll frame up just how horrifying some of these findings were as well.

 

The Open Source Software

Jonathan: Yes, that's right. So software, no matter what kind of software it is, it could be a mobile app, it could be in a nuclear power plant, or it could be in your car, wherever it is, people write software in fundamentally the same way.

The way that they do it is, they don't write a lot of it, they borrow a lot of it using open-source components. So these are little pieces of software, or sometimes little, sometimes big, that are developed by a community of volunteers most of the time. 

And they are made available to the world under certain terms, under a license. The reason people love to use these to make software is that it helps them get to market faster. It helps them build the thing that they're trying to build faster because they're using these pre-built components and stacking them up. And then they write a little bit of code that glues everything together and it's the specific parts of that piece of software.

So that's fine. Open source is amazing. There are amazing projects out there, and everybody does this. Synopsys has this. OSCAR report about open source every year, and they find, I don't remember the exact number, but it's like high 80s or 90% of every single code base that they looked at has an open source in it.

It might even be higher than that, I should check that. But basically, everybody does this. And the actual percentage of each project that is open-source code is also very high.

 

Avoiding Software Vulnerability Through Software Management

Jonathan: So that's cool that you can do that, but also, you have to manage your components, your open source properly. Sometimes we talk about making software in terms of automobile manufacturing or airplane manufacturing. And if you think about an airplane, it's got a gazillion parts in it.

It's got engines and it's got wheels and seats and rivets and all this stuff. And if you are building a piece of software and you're just throwing these open-source packages into your software. 

It's kind of like if Boeing or somebody found a bunch of parts by the side of the road and they're like, "Hey, these look cool, let's put them in the airplane." So you have to manage your components right. And that's what we talk about as the supply chain, the software supply chain.

That's actually part of it. The whole supply chain is even bigger and includes the tools that you use to assemble the software and wherever you're deploying it. However you're packaging it, and it's the whole schlemiel. But a lot of it is about these open-source components. So that was a lot.

Rachael: Yes, I think the idea of Frankenstein kind of came to mind.

Petko: I'm just thinking about having parts laying on the side that you just find in some warehouse-like, "Oh, this one looks close enough, let me just put this in." And honestly, I think software developers do that. They're like, "Well, I need something that works. Let me try this. Oh, it fits just right as a screw."

 

Why Software Vulnerability Proves to Be a Challenge

Jonathan: Yes, that's totally fair. And when you first think about it, you're like, "Wow, those developers, they're really irresponsible," but they're not. They just don't have the mindset for it because somebody has told them, "Go build this thing that does this stuff," and that's what they do.

So that's part of the challenge with the supply chain and with security in general, is that the people building this stuff don't necessarily have the motivation to do all the security things. Or look at the whole supply chain or whatever you need to do. 

So I think that's the fundamental answer to why software security is such a challenge all the time, is that you want to build the thing, you want it to do something, and you're going as fast as you can.

And if you're going to make it safer or more secure, you have to have the security baked into this entire process. And that's another place where airplane manufacturing is a great analogy because nobody builds an airplane without safety being part of the whole process. 

So from the very beginning when you're making designs to when you're selecting your components, when you're testing your components and vetting them and making sure they're good. When you're assembling the thing, every step of the way you're thinking about safety.

And security in software is the same way, or it has to be the same way. It's not, but nobody should ever talk about software development without talking about security at the exact same time.

 

[06:46] The Challenges of Dealing with Software Vulnerability

Petko: I would argue it's not just security, it's legal. And the reason I say that is, when you look at open source, some of them say, "Hey, you can't copy this unless you open source all your code." Imagine one little developer that says, "Oh, I took this, didn't read the license, and then I put in my code."

And then down the chain, maybe a couple of years later they realize, "Oh wait, that open source wasn't just normal open source. It was actually something that requires us to say we can't even use it without open-sourcing our whole solution."

Jonathan: That's right. They call those licenses copyleft or sometimes viral licenses. So managing your open source, managing your supply chain, step one is you have to know what you're using. Sometimes that's a challenge, but that's step one.

So you use a tool, software composition analysis tool, SCA. And it will go pawing through your project and figure out what components you're using.

And then to manage risk from open source properly, you have to look at the licenses. That's one aspect. And you also have to look at known vulnerabilities. 

So each of these components can have its own vulnerabilities in it, which are tracked as CBEs. A good SCA tool will tell you what you have, it'll give you that visibility first of all. And then it'll tell you which components have vulnerabilities and which might have licenses that aren't compatible with how you're trying to release your software.

So you have to manage both things.

Rachael: Is there a threshold like, "I'm okay with 70% software vulnerability in my stack"? Is there something like that out lingering where people are putting things together?

 

Managing the Risk of Software Vulnerability

Jonathan: There is, and I guess we would call that a policy. If I'm building, I don't know, like a piece of software for a traffic light. I'm going to have some policy about how I build it, what kinds of testing I do, what kinds of testing results are acceptable for me to release that software and so on.

So policy's going to vary depending on the person that's building it or the organization, but also the specific application.

So in a larger sense, this is all about managing risk, which is one of those things that's really important, but it's really hard to nail down. Like when you buy a car, you're like, "Oh, I want to buy a safe car. Is it safe?" And that's a nonsensical question.

There are more safe cars and less safe cars. There's more secure software and less secure software, but you're never actually safe. Insurance actuaries know this. They know all the ways you can die and what the probabilities are. And you can do things that make you less likely to die sooner, like you can not smoke. But you can never eliminate risk.

So it's the same thing with software where it's up to the organization and the specific application to say, "Okay, here's how we're going to do it. Here's the testing we're going to do. Here's the results that are acceptable."

Petko: From a safety standpoint, aren't cars today safer than they used to be 20 years ago?

Jonathan: That's what I've been told.

Petko: But would you say that software is more secure than it was 20 years ago?

Jonathan: Geez, I don't know.

 

The Changes in Software Vulnerability

Petko: Internet-connected software, because I think 20 years ago some of it was disconnected.

Jonathan: Some of it was disconnected. Yes, I would say in the world in general, software is better and harder to break. And I think really big names that make software and that's their whole business, they're mostly doing things right. They know how to do it right and they got the resources to make it happen.

But it's so easy for things to go wrong. And then you think to yourself, "Well, why are cars safer?" 

And some of it has to do with government regulation, and the government's trying to come to terms with software. I mean, they've been trying for a while to set down guidelines and maybe make rules, but they never quite get to making rules because it's just so hard to define.

Petko: Yes. I mean, what I kind of see with software is, we now have more software apps that exist in the last year than we did, just amount of velocity we have in applications. By the nature of it, even if we're getting more secure, the sheer number of new apps that come out constantly, it's hard to keep up with that. And it looks like we're getting less secure globally when you look at the quantity. 

But as a percentage, we're probably getting more secure every day. It's just that there's now more apps. I mean before you just have to worry about applications on your computer, now you have to worry about applications on your phone and everywhere else.

And each of them are different. If you have an Android, it's different than an iPhone, and from an Apple versus Windows, the velocity has definitely increased.

 

The Democratizing Force of Software and Internet

Jonathan: Yes. It's insane. I mean, one of the things I love about software and the internet is a democratizing force. So anyone with a laptop, even a kind of crappy laptop these days and an internet connection, can learn to program and build something.

And that's amazing. But at the same time, it's dangerous. If you build something and it becomes successful and there's significant value in it somewhere, then attackers will come for you.

Petko: I'm thinking of the people who created apps for certain iPhones or Androids, and somehow it just became a hit. And they didn't think about security. It was just a developer saying, "Hey, let me create a game."

And that little game, maybe flying a little bird around, became a huge hit. Not saying there's vulnerabilities there, but just in general, there was copycats that started existing from that and it became harder and harder to tell what's the real one.

Jonathan: Yes. Some years ago we did an analysis on a flashlight app for Android. All it has to do is turn the light on and off, right? And it was connecting to servers all over the internet, I think mostly ad networks, we hope. But that's the thing. Right? 
I mean, when you get stuff for free, you are the product somehow, so either your information's being monetized or something worse.

But yes, Cybersecurity Research Center, we did some disclosures, I think it was last year, about mouse and keyboard apps for Android. The idea is that you can use your Android device as remote mouse or keyboard for your computer.

And to be honest, some of these looked like they might have been student projects or something, but again, democratizing. 

 

[13:33] Lack of Security Causes Software Vulnerability

Jonathan: So people wrote software, they got it into the Android store, and there were like one or two million downloads of these things altogether. And they were totally not written with security in mind at all.

So if you went to the coffee shop and you were trying to use this thing, somebody pretty easily start running commands on your computer. So yes, I don't know what to say. It's tough.

Rachael: It's all about functionality. We just wanted to function right, right?

Jonathan: Right. Just running to get there.

Rachael: Now, can we come back to the research though? Because like I said, I was reading it and my mind was blown. So you guys were looking at sports betting apps, which I think a lot of people are about to be really bummed here.

Particularly with March Madness and Super Bowl and all the other things people are betting on this time of year. What did you guys find? I'd love for you to break that down for our listeners.

Jonathan: So we did a pretty slim slice of analysis. So like you can go to the doctor and they can do MRIs and they can do x-rays and they can do ultrasounds and all this stuff. So we have a tool that does SCA, software composition analysis, except the thing that's cool about this one is we don't need the source code.

We can just run it on the Android app and it figures out what the open-source components are inside. So we ran this tool, and if you're really going to do secure development properly, you've got five different kinds of tools and it's all baked into the process. 

 

Keeping Components up to Date can Lessen Software Vulnerability

Jonathan: But we ran this one tool and we found which open-source components had been used in these apps. And then we talked about that. The way we selected the apps was, we went to the Android store, the Google Play Store, we searched for sports and betting apps. And then we just picked everything that had over 500K downloads, and then we eliminated one because it wasn't really related. So we have these 10 apps, and we ran this tool on all of them, and we found a lot of very old open-source components. 

I think one of them dated all the way back to 2010, which is just in dog years, in internet years it's like a million. And I should say, we talked about open source components having licenses and open source components having known vulnerabilities.

And with the vulnerabilities, usually the open source team will get notified about the software vulnerability, they'll fix it, they'll release a new version. So most of the time what you're trying to do with your open-source supply chain is keep all your components up to date so that they don't have these known vulnerabilities that apply to them.

The other thing to know is that even if a component has a vulnerability, it's not necessarily exposed in the app, in application that it's part of. So a component that dates back to 2010 has a lot of known vulnerabilities in it. It doesn't mean that they're necessarily exposed in the app, but it's definitely an indication that the development team is probably not doing this analysis. Probably not paying attention to their supply chain. 

 

The Lack of Awareness Towards Software Vulnerability

Jonathan: So I think that's the really concerning part. And overall, these 10 apps that we looked at in terms of number of vulnerable components and number of known vulnerabilities associated with those scored really high.

Rachael: Very high. I wrote down the numbers. And I think I wrote down, let's see, average number of components is 125 per app, and then the average number of vulnerabilities per app. I feel like you need a drum roll please, 179.

Jonathan: I know, it's astonishing. And again, not necessarily exposed, but in the report we do actually look at one of the apps that it seems like they are paying attention. And the number of vulnerabilities in that one is in the single digits. It's very low. So it can be done. A lot of times development teams don't know. They're like, "Okay, we'll build stuff. We'll build stuff for you. Let's go." So sometimes it's just an awareness issue.

Rachael: And it's really interesting too because I mean, we see all of these vulnerabilities and they come into the public awareness. some companies seem to be really good at addressing these things, and other companies, not so much, there's no real accountability per se. I mean, it is kind of up to that company and they can like, "Well, you know?" 

Jonathan: Yes.

Petko: I think we'll fix it next release.

Rachael: Yes, exactly.

Jonathan: That's right. Exactly.

Rachael: It's no big deal.

 

The Long-Standing Issue of Software Vulnerability

Jonathan: Nothing to see here, it's just not the software you're looking for. Yes, and that's sort of been a long-standing issue with software, is that from the very beginning I think all the end user license agreements, that stuff that you click through when you install something?

Rachael: Yes.

Jonathan: It disclaims all liability like, "This software is for you. It might work and it might not. Good luck."

Rachael: Yes, exactly.

Petko: I'm thinking back to your flashlight example you had earlier. I mean, I've seen those apps. When you install it, they'll ask you, do you want to give it all these permissions? And it's going to ask for not just your camera, it's going to ask for your photos, your files, everything.

Rachael: Your email, yes.

Petko: And it's like all your apps installed. Well, I don't know if it had email. But I remember all your apps running and one's visibility and you're like, "This is a flashlight app. Why do you need access to all of that?"

Jonathan: Yep. My favorite, fine-grained location. What are you doing?

Petko: I only want the app used at night. So if it could figure out the time of day and everything else and where I'm located.

Jonathan: Makes perfect sense. You're right.

Rachael: That is hilarious. So what are people going to do about this? Because I think about all the apps that I use, Jonathan. I mean, we talked about this, I think last episode. I've got the app Graveyard. I'm an app hoarder on my phone, like Furbo. I don't know how safe Furbo is.

That's a camera in my house, so I could talk to my dogs when I'm away from the house.

 

[19:51] Every Organization Is a Software Organization

Petko: But Rachael, TikTok.

Rachael: I love TikTok. I love it. I mean capybara talk right now, all about capybaras right now. But what are people supposed to do? I mean, what is your lay consumers supposed to do about this?

Jonathan: That's the question. And that's a tough one. So I do PR comments periodically, and that's the same question like, "Okay, you've told me a bunch of interesting stuff and I learned something today, but what do I do?"

Rachael: Yes.

Jonathan: And what do you do? You throw your phone away, you go live in the woods, and go off the grid?

Petko: Hey, that's my plan. Don't take my plan away.

Jonathan: Yes. You're taking your mind and you're going. Well, that's part of the thing, right? Software is everywhere.

One of the things I repeat and people get bored with me about is, software is the infrastructure for all other critical infrastructure. And you cannot overstate the importance of software now. It's everywhere. And in terms of critical infrastructure, it's behind the finance sector and the water sector and the food sector and transportation, anything. And this is not really a new idea. 

Mark Andreessen a dozen years ago said, "Software is eating the world." And then a dozen years before that, Watts Humphrey said, "Every organization is a software organization." And it's just, it's true. so as a consumer, what do you do? You cross your fingers and you hope for the best.

Rachael: So hope is the strategy. That's right.

 

The Deterrents of Software Vulnerability

Jonathan: Yes. Faith-based security, we call it. It's going to be okay because you believe it's going to be okay. And obviously, that doesn't work for everybody. People have problems, need to rebuild credit or their identity or whatever. It's tough.

Petko: Jonathan, can I throw more philosophical conversation about this? Because I'm kind of curious. Like me, I kind of use software. I have a house, I don't always lock my front door, let's say hypothetically. Is it my responsibility to lock my door to prevent folks from coming in?

Or should there be better enforcement on the internet or better policy that has repercussions if you break into someone's house?

Jonathan: Yes, it's a combination of things. So why doesn't something bad happen every time you leave your front door open? It's a combination of things. So there's the risk of the place that you live, and then there are legal deterrents. Everybody knows that if you're caught going into a house and stealing stuff, you're probably going to go to jail. 

So it's a combination of things. Software and the internet makes it so much harder because the best attacks are completely remote. They could be anywhere in the world. And that, in legal terms, involves different jurisdictions. They have different rules.

So the likelihood of being caught or prosecuted or imprisoned is really low. So I think that's part of the problem.

 

Software Security is Everyone’s Responsibility

Petko: So the internet has no jurisdiction is what you just told me.

Jonathan: Yes.

Rachael: No accountability. Yes.

Petko: Or it's difficult to tell where folks are coming from. Was it really Jonathan that broke into my house or was it Rachael? I can't tell because I was just talking to them. That's it.

Jonathan: Yes. Right, exactly. So even if you do know where they're coming from. And if you think of it from law enforcement's point of view, you have to be a little bit sympathetic because of, I call it my local police. I'm like, "Hey, Alexei in Russia just broke into my bank account." They're like, "What are we going to do with that?" It's theoretically possible, but it's tough.

Petko: They're going to probably ask you where's your bank located. And if it's not a jurisdiction, they're going to defer to someone else.

Rachael: Exactly.

Jonathan: Yes, right. That's right. I'm sorry, go call New York. 

Rachael: That's hilarious. And with that, we are going to pause today's discussion with Jonathan Knudsen. We're going to pick it back up next Tuesday, part two, and you don't want to miss it. So to all of our listeners out there, thanks again for joining us this week. Again, a huge thanks to Jonathan for joining us and being part of our two-part episode.

So until next time, be safe.

 

About Our Guest

 

Jonathan Knudsen is currently the Head of Global Research for the CyRC, cybersecurity research center, at Synopsys Inc. In his role, he conducts security and software research, helps people publish their research, and occasionally comments on pressing security topics. His past experiences include being an Adjunct Professor at Duke University for a year. He was also a Principal Security Engineer at Codenomicon for four years, a Principal Technical Writer at Oracle for one year, and a Senior Staff Engineer at Sun Microsystems for nine years.

He enjoys breaking software and teaching others how to make software better. Jonathan is the author of books about 2D graphics, cryptography, mobile application development, Lego robots, and has written more than one hundred articles on a wide range of technical subjects. He lives in Chapel Hill, North Carolina.