Cloud-native data warehouse vendor Snowflake and Edge Delta have spun up a new SecOps architecture they claim will broaden the application of analytics in a security information and event management (SIEM) product.
SecOps teams are generally wise to the appeal of computing at the edge – where logs of events on networks or servers may trigger warnings of malicious activity according to some pre-programmed criteria.
Edge Delta is applying machine learning to edge data to pre-process data in parallel before centralising it. But that leaves the question of what to do with all the raw data, which may become useful later.
Users often have to choose between expensive data warehouse products or cheap object storage, which makes further analytics a challenge, according to Snowflake.
“Many teams are forced to dump terabytes of log data into cloud blob storage where they will have a hard time using it for detection or response in the event of a breach,” said Omer Singer, head of cyber security strategy at Snowflake.
“This ‘hope for the best’ strategy is imposed on them because of technological limitations of solutions developed before the cloud.
“Edge Delta shared with us its technology for sending important events to the SIEM while shipping all events to Snowflake and we recognised the impact that a joint solution would have for our customers.”
Using this strategy, SecOps teams can enable real-time monitoring and security alerting, and – according to Singer – achieve cost savings compared with other vendor tech. “One customer went from spending $12m to $2m a year when making the switch,” he claimed.
The lustre of a cluster
Simply dumping all log data into object storage is not helpful because users require a query engine or data warehouse to get anything meaningful from it.
Snowflake said a common approach is for security teams to store compress logs in object storage arranged by time. They can then search a particular day by loading the logs into a search tool, such as Elasticsearch. However, an Elasticsearch cluster has a capacity limit depending on the number of nodes, so searching in the time range needed for many incident response situations would require loading and unloading multiple sets of data from S3.
The advantage of using a combination of vendors is that in the event of a suspected breach, SecOps could spin up, for example, a Snowflake warehouse for the duration of the investigation. The security team then lists a thousand possible command-and-control servers involved and uses an SQL JOIN to compare them with network communication logs that were collected by Edge Delta over the past three years.
Any matches would be returned indicating which internal systems were involved in the breach, he said. Snowflake has cited customers’ approval of the approach, including Amit Mathur, vice president of product engineering, Sinclair Digital, the digital media arm of Sinclair Broadcast Group, the second-largest television station operator in the US.
“The integration between Edge Delta and Snowflake is a new approach that has the potential to fundamentally remove limitations, opening up a whole new set of possibilities,” he said.
Roy Illsley, an analyst at Omdia, said a lot of organisations were doing machine learning at the edge of their networks and infrastructure to detect security anomalies. But the ability to send all logs to a data warehouse at the same time might be interesting.
“The thing to be mindful of with pre-analysis at the edge is it may not detect patterns that can exist between nodes, so the data warehouse with all the data would be useful for that type of analysis,” he said. ®
