In part one of this series on cloud monitoring considerations, we discussed cloud vendors making log data available and the challenges associated with retrieval. In this final part, we will discuss the format of the data once it is retrieved and the data that is contained in the logs.
After the data is retrieved, you have to contend with the format and syntax of the data. The data is often in JSON format, which makes parsing and representing the data very easy. Sometimes it can be hard to escape special characters including white spaces and newlines when including that information in a JSON object. In some instances though, vendors, like Microsoft, will do some interesting things. In example one below, the log is in JSON format but, for this specific type of log, there is raw XML within the object. This raw XML contains valuable information that needs to be parsed.
This poses a challenge when trying to determine how the data should be parsed from the log. While it can be overcome, it is just another hurdle that has to be accounted for. We have also seen JSON, within a JSON object. The appropriate escapes, format readers, and parser have to be set up to be able to read these types of logs. In some instances, the vendor will just have flat text in the log. This may include comma delimited or some other identifier making it easy to distinguish each field. In other instances, the vendor may store the data in a propriety format which only a handful of programs are able to open and interpret. This is the case with Microsoft storing Azure SQL audit logs in storage accounts in EVTX format.
Through all of this we are finally at the data element level. At this point it has to be determined if we have data elements that make sense for what we are trying to accomplish. NTT Security is primarily interested in security monitoring and there are specific data elements we look for to perform this monitoring. It is important to review the data elements that are collected to ensure they will satisfy your requirements, can be used for alerting, and can answer the questions that will be asked.
The other piece of this is ensuring you can easily find and determine the data elements. Not all vendors provide detailed documentation outlining what each data element represents. In example two, we have an Azure SQL log and an Intersect Alliance Snare SQL log. There are certain data elements that are in both logs. One major element that is missing from the Azure SQL log is an identifier for what type of activity is occurring. In the Snare SQL logs, you can see an event ID, highlighted in blue. That ID translates into specific SQL transactions, which makes it much easier to determine what is happening on the system from the log provided.
As you can see, there are a large number of items that must be contended with when monitoring a cloud environment. While the major cloud vendors have made great strides to make the correct information more accessible, there is still progress to be made. The good news is that many of these vendors listen to their clients and work to make the necessary changes.