Hey, everybody, welcome to another episode of Tech Done Different. As always, I'm your host, Ted Harrington. And with me here today is our special guest, Kyle Tobener. Kyle's the VP and Head of Security & IT at Copado. Kyle, thanks for joining the show.
Happy to be here. Thanks for having me.
I'm excited for us to talk about this enormous topic that impacts most major companies yet, at times, can also sound like a very unsexy topic. And it's this idea of third-party risk management/vendor risk management. So first of all, why don't you maybe just briefly introduce yourself and kind of sum up your background, and then we'll get into a few questions I have?
Sure. So like you mentioned, I'm currently Head of Security at a startup called Copado, kind of a pre-IPO DevOps startup. I just joined Copado, about three months ago, prior to that, I spent 10 and a half years at Salesforce, where I owned their third-party vendor security program, as well as application security and some other things.
Cool. So maybe let's start there. Right, so running a third-party risk management program at a company as large as Salesforce, you know, with deployments and integrations all over the planet, what was the biggest challenge that you saw, trying to deal with the complexity of a problem at that scale?
It's always a balance between the amount of time you can invest to go deep on any given specific vendor versus the scale. You get hundreds of requests a month, and you can only do so much with the resources that you have. So we're constantly trying to walk that line between going deep enough to really understand the impact a new vendor would have on the company versus keeping up with and not delaying the business objectives of the company.
So how does one do that? You don't have to, of course, reveal any proprietary secrets from Salesforce. But you know, what are the principles of how you sort through that trade-off?
First, I really liked building processes, consistent, scalable processes that are effective and kind of support business partners. So I focused a lot on just mapping out the different stages and figuring out how we were going to manage the work within my team. So I would say, you know if you're in a function like that, understanding that the volume will peak, the volume will spike, it'll drop off. And so, you have to have some elasticity to move with the seasons. And so the way I did that is by building in third party risk management or vendor security responsibilities into a broad number of staff who are also responsible for other things like application security, so that when we hit certain cycles, like the end of Q3, for example, the end of Q3 was always a nightmare because everyone had money, they wanted to spend it quickly we'd get a huge spike in volume. And so basically, everyone on the team would kind of drop what they're doing and dogpile into the vendor security process to make sure we could move all those requests efficiently. So that was one way.
Do you find that the different organizations interacting with that whole process have different needs and demands, or I'm assuming that they do? And if that assumption is wrong, poke at it. But do you find that those different competing demands and priorities make it difficult for a process like that to run efficiently?
Yes, oftentimes, you know, they're under pressure for their line of business to deliver whatever it is that they've been asked to deliver by the company. And it's not their job to be experts on the security process. So they just may not know how to interact with the security team, and maybe it's their first time. So oftentimes, timelines clash. Maybe they come to the security team a week before they think they're going to resolve a purchase in 48 hours and realize it can take a little bit longer than that. And that's unfortunate because it's not a position that we want to be in. So I spend a lot of my time there trying to educate people on expectations of how the process could go and how we could partner better. Because being in the security team, you kind of end up in the position of being a blocker. And that's not the position I ever wanted to be in; I was there to support the business and help them achieve their objectives safely. Blocking them from doing something was not my goal.
Yeah, you're definitely hitting a hot button. I think that probably a lot of our listeners have heard security is often seen as the ones who say no, when in fact, we want to say yes but. So how would you go about doing that? In large enterprises, communication, even within a given team or department, can sometimes be complex, let alone across departments or even across business units. How can we foster that education that you're talking about?
It starts in security leadership; oftentimes, leaders may focus on the technical problem they're responsible for and not necessarily look outward to the business problems their colleagues are trying to solve. I spent a lot of time meeting with the business leaders like the people in charge of HR or sales, the ones running the company's various operations. I wanted to understand the problems they were trying to solve and the timeline they were working against to solve those problems.
At the beginning of the year, we had benefits refresh. The benefits department is like we're redoing all the benefits across the entire company this year, knowing that it was like, okay at the beginning of the year. This is a huge priority for the company; anytime we see something from Benefits at the top of the pile, we will make sure that we're there to help them. And they became really good partners and knew how to work with security, because we were proactively showing them that we were willing to support them and help them. I think partnership came down to hey, there's a security problem here; this company that you're looking at isn't as mature as we'd like. They were more than willing to work with us because they saw how willing we were to work with them every step of the way up until that point.
I love this idea you're getting at of establishing these relationships. And it's a theme that does come up quite a bit here on this podcast. Security and tech, in general, are so technical, complicated, scientific, and yet, so often, it boils down to building relationships and how do we use those relationships to further the mission of the business. I'm curious to hear how you think that one can build these relationships. It sounds like the things you were doing were being proactive, setting up time in advance, not being reactive, asking questions, being an active listener. Are those the basic principles for how you, as a security leader, built relationships across the enterprise?
Yes, I would say that, and I always tried to focus on the accessibility of the security team, me, and the resources that worked for me. Often, I think I see security leaders who focus on their process, like I talked a lot about process building before. Sometimes people just focus on their process and get tunnel vision and say, you have to go through my process. If you're not, you're harming my ability to scale. So I won't talk to you. But that can be really frustrating for people who are confused by your process, are new to it, or maybe don't understand it. So I tried to find a bunch of avenues where people could engage our team in scalable ways where perhaps they were confused. Some people just have a better time talking to someone than they do writing an email. So we had a thing we called office hours where we had an app, anyone could hit that app, choose a time during the week for 30 minutes slot and sit down with someone from my team and just have a conversation about what their problem was and what they were trying to achieve. And that was a really effective way, like a lot of people didn't use it. But for the people who needed in-person conversations, it was their favorite thing.
That's really fascinating. So it sounds like the logistical and practical reality that different people communicate in different ways. And you are making yourself available in as many different ways as you can.
Yes, exactly. And then we would have a Slack channel and a self-service knowledge base, and kind of a ticketing system, where if you're good at writing emails back and forth, that was great for you. Engaging in all of these different channels pulls your focus a little bit and requires some resourcing to maintain it. Still, it is much more approachable for our business partners and makes the process easier for them.
I love it. You introduced a problem before that I hear a lot. And it's this idea that the business wants to work with a particular third party, and they want a contract right now. And the security process might take weeks instead of right now. When one runs into that, how do we resolve that problem?
Having a process that communicates early time expectations is important because some of these things do take time. So one of the requirements that we had was for really sensitive vendors, integrations to key systems, we had to do an in-house penetration test. And that takes time to coordinate; it takes time to perform; it really can't just be done instantaneously. So we put a lot of effort into educating people on the timelines for that process so that the minute they talked to a sourcing partner and said, hey, we want to go and look at a solution. The first thing that's said to the partner is okay, let's get this scheduled out. Because we know you're going to need a penetration test in this category, let's get that planned in advance.
COVID is a great example. Because of COVID, we had to respond to emergencies; we needed testing vendors; we needed medical vendors for the pandemic. There was no timeline in which we could move quickly enough. So having a risk acception process, where an executive in the business can say, I understand that I'm bypassing the process, I get it. The risk to the company is much higher if we don't bypass this process and accept the risk, because he's doing something that will buy down risk for the company in a more significant way than going through the process. And then, you know, the process can always catch up later and close out that risk. But that outlet, that kind of emergency outlet, and that flexibility from the security team are also important.
Yeah, I like it. I would imagine that something like that probably winds up getting used a lot, then people become accustomed to this idea that there's always this emergency release valve. Have you found historically that is something that winds up getting abused? I don't mean it in a malicious way, like someone's intentionally circumventing the system. But the way business runs today, everything is an emergency. So how do we help the stakeholders involved in this type of decision distinguish between what's an emergency and something more of, you just want it right now?
Yeah, there is this tendency to gamify the system. If the exception process is seamless, quick, and faster than the more secure process, then everyone's going to go for that. Their incentive is, hey, this one moves much faster. So you have to make the exception process weighty enough that it's not someone's first choice.
The way I like to do that was by requiring kind of an in-depth write-up of the risk scenario that we were going to make an exception for. And then having a VP level executive on the business side and a VP executive on the security side sit down together and discuss that write-up and the ramifications.
If someone more junior in the organization, like a director or a manager is requesting the exception, they know that it's going to go up to their VP who is the one assuming the risk and discussing it with someone from security. That helps them understand, oh, this is actually a pretty impactful decision. I don't want my VP necessarily to feel like he has to sign off on this because I didn't move quickly enough. So that drives the right behavior.
The psychology in that structure is really fascinating. This is to say that, make someone else responsible for this. And in so doing, make sure that person is going to be okay with that and that I can justify why I need to have asked that of them. How did you arrive at the power of that?
I think it was trial and error in a way, but also I like to listen to a lot of podcasts about behavioral economics and learning about incentives and how people respond to them. You kind of look at these systems and security and say, okay, what kind of behavior am I incentivizing? Many processes and security are built so that people naturally want to bypass them. So you need to build them so that it is natural for them to want to go down the best path for you. If you build a process, and this is the thing that I want everyone to follow, you have to find ways to improve it such that the majority of the company is going to want to follow that process versus any other option.
Okay, that's cool. So let's explore that. Because we've just sort of talked about how do you build a process in a way that makes people not always want to trigger the exception, always make everything an emergency? So that's sort of a disincentive? What's an example of a process that you built, even if it's a hypothetical one? How could we build a process, so someone says, I want to do that?
Yeah, so I think I can even drag it back to the sort of the vendor security process when I zoom back in time to earlier Salesforce days when it was more of a startup than a big multibillion-dollar enterprise. But back then, when we had less formal processes in place, one of the ways we got people to engage in our vendor security sort of program was saying, if you follow this process, we'll work with you. We'll figure out a way to de-risk this situation and sort of approve you to move forward as quickly as we can. If you don't follow this process, you own everything about this vendor going forward. So if they have a breach, we are not going to be there to help you because you bypassed our process. And that sort of partnership incentive where they're like, oh, I don't want to be on my own if there's a problem, I want the security team's help. That was not my favorite way of building a process and building an incentive structure. But it worked really well, in getting people to work with us.
I love it, yeah, help them understand what's in it for them, right?
Yeah, exactly.
So we've been talking about this from the perspective within the enterprise itself. Getting the vendors and third parties to participate in these programs, whether it's answering the questionnaires or supplying their security testing reports, or meeting with you, there's often a lot of resistance to that because it feels like more work. It feels like a barrier to procurement, etc. How have you seen that we could build the process in ways that incentivize the third party?
That's a good one. I've been on the purchasing side for many years. And now I've spent a few months on the selling side. And so I've seen both sides of it. And I think on both sides of it, I've kind of come to the same philosophy, which is that transparency is the thing that I want to see more than anything else. It is very difficult to judge the maturity of another company without information, without the details you need. And even when you have details, it's not a science; it's an art. So you want as many data points as you can get.
So for me, when I was at Salesforce, the thing that I would always come at was transparency. When I talked to a vendor or prospective vendor, I would tell them that we're going to ask you for all these details, you may not want to supply them, and that's okay. But I want you to know that when you don't supply them, I'm immediately going to assume you don't have them. And that will be counted against you. And it's kind of up to you how efficiently you want to pass this process. You know, if you think we're going to conclude that you're super mature, without providing us all these details, great, that's what we'll try. But if you don't provide them to us, we have to conclude that you lack these kinds of policies and maturity; you're not fixing your bugs quickly enough. That sort of thing versus like when we would work with the vendor who is super transparent, share everything. It was like a dream. You're just like, I love you. You're amazing. Let's partner together forever.
I think there's some really good examples out there of companies that have gotten much better at transparency. I mean, one of my favorites is GitLab. GitLab is super transparent about everything that they do. All their documentation is published publicly, which I think is amazing.
Yeah, I love this theme that you're keyed in on here about transparency. I wrote a book called Hackable, and it's written for people who build software systems, how to do it securely, but then ultimately, the payoff is now how do you prove that to your customer? And I argue exactly what you said here, which is transparency. And when you're more transparent, it's this fascinating dichotomy, where the more transparent you are, even if you're saying I have problems, look at all these vulnerabilities. And, it's going to take us this long in our roadmap to fix them – that often builds more confidence in the relationship than the person who's like, no, we're good, not applicable. We have military-grade encryption, that kind of stuff, right?
100%, yeah, it sort of feels like the bug bounty revolution, where now many companies are more willing to allow external reporting of vulnerabilities. Acknowledging that we're going to have problems, we need to build processes to recognize what those problems are and get them fixed very quickly. And when we were doing penetration tests against companies, that's what we measured against, it's not whether or not they had problems, but if we identified problems, could they fix them?
And how would you be able to identify whether they could fix them or not?
So we would do a penetration test of a given vendor and keep an excellent technical staff of people who could do good penetration testing. So that the odds of us finding bugs within a vendor were pretty decent, not always like the most mature ones, it was very difficult. But most of the time, we would find a few decent bugs in a software vendor. And then we give them to them and say, here's what we found. And ask them for their SLAs on fixing these types of bugs. And they would share them and say, we'll fix this one within 30 days, we'll fix this within 60 days, great. And then we would let them work against their SLA and prove it to us. And you could immediately tell if a company was not good at fixing bugs against some penalized delay because they feel like we have a lot of priorities. This one's probably going to take us a year to fix, and that was always a very bad sign.
That's the nature of any business, right? We have a lot of priorities. You're saying it takes a year; you're saying it is not a top priority. Is essentially what's being communicated?
Exactly, exactly.
Got it. Now, let's discuss the process itself of managing this. From the vendor having responses to security questionnaires, those questionnaires, whether it's a questionnaire or some other format information being communicated to an enterprise from the security team to the procurement team, legal, business, whatever. Have you seen ways where that whole lifecycle and information work well? And cases where it doesn't work? I know you've given us some tips already on sort of both sides. But if you think about the whole process, breaking it up to this point into two sides, inside the enterprise, and outside the enterprise, but now we think about it as a complete process. Are there examples where people totally bungled it, or examples where it's gone exceptionally well?
I think so. I've spoken to a lot of companies that have vendor security teams; I like to keep in touch with a lot of them. And this is a pretty common problem across all of these companies, which is the information sharing across all these key departments; it can be a struggle. Especially in one company, you may have a really strong vendor security team, someone who gets the process and is super engaged. But another participant in the process may not be as strong, may not have been as funded as well. And so that piece becomes kind of a detractor, the drag on the process, and sometimes with some companies, it's a security team, that's a drag. So I kind of got the sense that wherever you go, there's always a gap here or there in the process, and nobody has it 100% nailed amazingly well. So there's a lot of work to do.
Yeah, agreed. So it seems to me like there's a handful of ways people might solve that complex problem in terms of the tools they use to communicate. On the more sophisticated end, there's actual platforms built to manage this process. And on the lesser sophisticated end, people are using email and spreadsheets. In your experience, do you think that enterprises are largely in the use of tools or non-solutions, like email and spreadsheets? Or is it sort of somewhere in between?
I would say that, among the security programs that I've talked to that focus on third parties, the ones that have a unified sourcing platform are always in the best shape. There isn't a centralized function at some companies that manages and has oversight of procurement; it can happen willy nilly, anywhere. And that makes it hard for the security or legal team or any team to be successful in that process. Because you're just fighting this visibility challenge, you're not seeing everything. And then when you get that centralized procurement function, and hopefully some piece of software that brings together the purchasing process, the main thing you get is visibility. And it may not be the perfect solution from a security perspective or the perfect solution from a legal perspective. But visibility is like the first key battle into really getting your hands around your third-party security. You need to be able to see everything to properly gauge the risk of what's coming into your organization.
I love it. I couldn't agree more. So what have I not asked you about that I should have asked about on this crucial topic for many companies?
I think there's a lot of cool innovation happening in this space, a lot of it in other companies, and smart people are doing it. I really liked Dropbox's program; they were doing many really innovative things, especially their bug bounty for vendors. So they blogged about this in the past, in their bug bounty program, they have third parties included. They will pay researchers to research products that they don't own because they've identified these products as being impactful to the security of Dropbox. And I thought that was a really interesting way to innovate in the third-party security space.
Super cool, awesome. Well, as our time comes to a close here, I definitely appreciate all these insights. I've learned a lot from you as we go through this. And it's great to hear some of the insights that you've shared. Any parting wisdom that you wanted to drop on our audience before we close?
I would just plug third-party security, for some people, it doesn't seem like the sexiest area to work in, in information security. But having done it for a number of years, I felt like it was the thing that prepared me the most for jumping into the seesaw head of security role. It taught me how to work with all these different business units across the company. It taught me how to really oversee all different aspects of security and understand how to gauge a company's security maturity. So if you have designs of becoming a broader security leader in the future, spending some time in the third party risk space can really up-level you; it's kind of like an MBA.
Love it. The security MBA.
Yeah, exactly.
There you go, that's a great insight. Well, Kyle, thank you so much for all the ideas and insights. You've been great. And thanks for being on the show.
It was a pleasure, thank you for having me.
Absolutely, for everybody else, if you want to learn more about the show or request to appear yourself, just go to tedharrington.com/podcast, and we'll catch you next time
