What to Do About Multi-Cloud Audit Log Overload

By Paul South

In an interview with Expel Chief Technology Officer Peter Silberman, we explore one of the biggest issues cybersecurity professionals must overcome in 2021: data overload due to logs generated by multiple cloud platforms.

Most of us are familiar with data sprawl generated by cloud services, but not data overload. Can you discuss how this came to be such a big problem? And just how big of a problem is it now?

SILBERMAN: I think about data overload in two different ways. There’s what we call the control plane logs, which are generated by the activity a user or service is doing within the cloud platform. This all creates logs (data). In addition, you have what you could consider security data, or alerts coming off of their services, or additional third-party security devices and vendors you purchased. All of this combined can cause overload.

While that amount of data can seem like overkill at first glance, you need extensive logging to understand what’s happening in an environment for auditing purposes. When you think about alerts based on user or application activity in the cloud, a challenge can be determining the difference between an active attacker with malicious intent, and a user doing something they shouldn’t (like turning off multi-factor authentication). Detecting suspicious or risky activity can create a lot of noise and overwhelm analysts.

How did this become such a big problem?

SILBERMAN: I think the journey here for the industry at large was probably a long one over time, especially as developers became the ones responsible for defining and creating the audit logs. Engineers have a different perspective than security analysts on what should or shouldn’t be logged. Additionally, though, the move to cloud overall brings many benefits to an organization but also makes security more complex. For example, cloud usage goes up because a company grows, and all of the sudden you have more data. As a result, our reliance on these providers (and their logs) grew over time.

Is interoperability an issue since different cloud platforms use different systems to generate logs?

SILBERMAN: If I had to pinpoint the largest problem facing data overload, it’s that most organizations take a hybrid approach to cloud. They’re not tied to a single provider. And the logs that each cloud generates are very different. They name everything differently, for starters, and each set of logs requires a certain level of expertise.

So, you can’t take an engineer who understands one vendor and assume they'll instinctively know how to handle logs from another. They have to read the documentation and understand the authentication model. There are just so many differences, and this is a fundamental challenge for the industry.

What can happen if a security team doesn’t keep up with the log data coming in, either due to a skills or knowledge gap?

SILBERMAN: The worst-case scenario would be where data is exfiltrated, or in an equally bad scenario, something wasn’t detected and attackers either locked out a company or locked up a company’s production assets (ransomware). Those are your two dreaded scenarios—having data stolen that could impact your business viability, or being locked out, which impacts your business viability.

Sure, there are teams that aren’t aware of how to keep up with the log data and triage it properly, but I also think teams that are capable can experience alert fatigue, and all of a sudden that fatigue is clouding their decision-making.

Sometimes they’re trying to get through the alerts as quickly as possible, so they’re optimizing for speed and not quality of decision. At Expel, we’ve invested a lot in helping our analysts when we or the platform thinks this could be happening. We’ve built processes—both manual and technology-driven—to alert us to this possibility.

What are some solutions to centralizing all this information that’s coming at folks fast and furious?

SILBERMAN: The first thing is that you need to understand the data that you’re interested in and then store it in a way that centralizes it, whether that’s in a SIEM or a data lake that a cloud provider has. But you want to centralize it and make it so that you can query or search the data. That helps with the data sprawl.

But there’s a cost component with data ingest and the volume of data storage that can be non-trivial. That gets you to the point of, I have data somewhere I can work with. And then, what we’d recommend is if we have a skill or knowledge issue, there are opportunities now to do training. There are threat emulations that people have put out there, where you can see a whole scenario of an attack, see what the logs produce and learn from that.

We hear postmortems are generally effective.

SILBERMAN: I think postmortems are critical to improving any part of any business, team or function. There’s also another crucial element to having effective postmortems—being able to own a mistake or discuss [mistakes] without blame is culturally beneficial. There’s a notion of ‘everyone’s improving.’ And that’s been demonstrated time and time again when you do these postmortems. Postmortems also provide visibility into all the different jobs that people have and their different perspectives on why a problem occurred and how to fix it.

Any parting advice to those struggling with cloud data overload?

SILBERMAN: If you’re struggling with alert fatigue, don’t lose sight of what is of strategic importance to your company. Does your company need a 24x7 SOC in-house because of regulations? That would make having a SOC of strategic importance to your business. Understanding your business objective helps you drive toward it by informing the decisions you make on what and who you hire, buy or build.

There’s almost an overwhelming number of vendors that can offer support in various verticals. I would also think really hard about a training budget. It’s one of the most effective ways to grow people, retain talent and build loyalty.

There’s expertise out there that can be tapped into, and there are people out there who think about solving these problems. Not everyone’s journey is going to look the same. And that’s both OK and expected.

Paul South is a freelance writer and past contributor to Cloud Security INSIGHTS.