Logs to Monitor for Security Analytics

By: Evan Klein

DevOps, security, and SOC teams find themselves constantly facing new cyber threats, ever-evolving attackers, and innovative attack vectors. Their challenges range from protecting employees’ mobile devices to preventing malicious parties from accessing an organization’s financial data or customers’ personal information.

Monitor to Protect and Prevent

In every system, there are a handful of defensive endpoints and vectors to defend, monitor, and respond to, and there are never enough resources to protect and contain all the attacks that could happen.

One of the most effective and efficient solutions to this problem is monitoring and tracking log files from all systems and devices. This process unifies most, if not all, required security activities into one.

While it may sound simple and straightforward, producing comprehensive protection for an application and an organization by monitoring all of its relevant logs is not easy. Different attack vectors have different targets and different goals, and, as a result, security efforts must be broadened to include monitoring all possible penetration points.

This article will list the different systems in an application stack and discuss which log files should be monitored. It will also address how to identify and potentially predict a security incident.

Application Logs

Application logs are both the easiest and the hardest to monitor. On the one hand, the R&D team knows them best, since they design and use them. However, even the simplest application writes an enormous amount of logs and events—API logs, logs of services that do not expose an API, and client-side logs that offer a window into the end user experience. Because application logs are unique, there is no generic solution for tracking them. Nevertheless, below are some guidelines that can help identify security events as they occur:

Pay attention to caller IP addresses—Scanning attacks, where all the application APIs are called at the same time, might appear harmless; however, they can end up being the first steps in bigger, more dangerous attacks. Identifying scanning attacks and their sources can eliminate risk.
Suspicious activity—Every application has its own logic and methods. The application logs must be able to provide information when specific activities (the sensitive ones) are being utilized for unauthorized procedures.

Application logs should be able to provide a complete picture of how an application is being used and shed light on activities taking place at any point along the application’s life cycle. That said, protecting application logs is not enough; the application infrastructure must be secured as well.

Infrastructure Logs

Different applications are based on different types of infrastructure; however, they share a few common features that can help you focus on the most relevant monitoring and tracking efforts. Tracking the cloud provider logs supplies a broad view of how the application is taking advantage of its resources. Managed services, virtual machine utilization, IAM, and other modules offered by the cloud provider must all be tracked using records from the provider’s logs.

Every questionable activity—from changing user permissions to VM spin up—that is not approved should generate an alert and provoke an examination of the activity.

Modern IT applications use an orchestrator as the application environment manager. The most common of these is Kubernetes. While the orchestrator is a tool that follows predefined rules and automatically handles incidents, its logs can provide insight into the application’s usage and shed light on the status of the OS and the containers. The orchestrator logs should be maintained separately. This enables you to trace all of the activities that have been done automatically and highlight the important ones that usually require a follow-up.

Another layer of the infrastructure is the OS/containers the application runs on. OS logs can indicate malfunctions as well as offensive usage. Malicious processes, file privilege escalation, or new files instantly appearing on the disk should alarm DevOps and security engineers who should then investigate how they got there.

Database Logs

The database is usually one of the areas of an application most susceptible to attack. Every query or activity done on the database should be monitored. Combined into one clear view, database transaction logs, replication logs, and event logs create a clear picture of the data status, integrity, and confidentiality. When one of these aspects is at risk, database logs can facilitate inquiry and help to find the root cause of an incident. Managed database solutions offered by a cloud provider have all of these logs built into the cloud logging solution. The logs can be easily integrated by an external security system that reviews and monitors them in order to alert the security team when a problem occurs.

Monitoring Systems Logs

The main goal of monitoring tools is to display the application services’ clear status. Monitoring systems and tools can help identify security attacks as they happen and alert the relevant stakeholders to their presence. While this set of tools is out of the attackers’ view, more sophisticated attackers can manipulate the monitoring or logging system to maintain the integrity of their hack. Attackers that are aware of the monitoring system might try to remove indexes or dashboards to ensure that NOC and SOC teams will not be alerted to their efforts. Every monitoring tool has activity logs which should be available. Once an unauthorized change has happened, the monitoring system admin must be made aware of the change and treat it as a possible breach.

Security Systems Logs

Typically, every application that is exposed to the internet is protected by one or more security tools, such as Cloudflare, Encapsula, and Akamai, which are used for WAF, protection against bots, DoS, and other attack types. These systems manage many configurations and rules, and, in the wrong hands, they leave gaping holes that can be infiltrated. Every change in these security tools must be thoroughly reviewed and approved. Therefore, monitoring the audit logs which contain the activity, configuration changes, and traffic of the security tools is an absolute must. This vigilance can help prevent breaches before they occur.

Private Network Security and Logs

Up until this point, we’ve reviewed the different layers of the application infrastructure and explained how every module’s logs can allow you to detect possible security violations. But, the most vulnerable layer is usually the office’s private network. Because there are many different breach points in a private network that can allow attackers to gain access to code, production data, or employees’ personal information, each must be monitored and supervised.

Firewall Logs

Similar to the security system that is used to protect the application, the firewall tool that is used to protect the office network manages a set of configurations that secures the organization’s most important assets—employees, devices, and documentation. Every unsanctioned change in these configurations should be treated as a possible breach and should trigger an inquiry into the change. Firewall audit logs enable you to be aware of such changes.

Mailing System Audit Logs

All attacks that fall under the umbrella of Social Engineering involve communication with victims. This is usually done through email. The mail exchange system, whether it’s a managed server or on-premises, must be constantly reviewed. Any suspicious communication should alert the network admin and security officers. There are many algorithm-based tools that enable you to monitor the mailing system and find noxious content; however, these tools are implemented as warpers around the mail system. This might not satisfy the security requirements. If an attacker manages to infiltrate the mailing system admin console, it can change controls and configurations, allowing social engineering emails to penetrate the protections and achieve their goals. Tapping into the mailing system audit logs (like the office 365 audit logs or the Gsuite audit logs) can enable you to determine if any mail control was edited or if the mailing system itself was hacked. The same logic and process applies to the other modules of the suite, such as Google Drives or Microsoft Sharepoint.

Employees’ Endpoint Antivirus Logs

Antivirus logs and the logs of any other protection software installed on employees’ devices must be part of the monitored stack. Every file, email, process, and configuration can be destructive to the whole organization by exposing valuable information or sabotaging operational systems. Every employee’s OS should be secured by many means, and all of these should report to a central location where the collected data is analyzed. This is where monitoring these logs should take charge. Every suspicious event should raise a red flag for the security officers.

This Is Quite a Lot Of Monitoring…

As this information about logging procedures has shown, a significant amount of effort is required in order to have 360° protection.

The different types of threats that organizations face end up translating to different security systems and prevention and protection methodologies. Keeping track of so many security systems—even without the aspects of operations and quality monitoring—requires either a large team with sophisticated technical skills and knowledge or a system that can collect and aggregate data from the various applications and organization layers and gather it into clear, informative views.

SIEM and Security Analytics systems such as Logz.io Security Analytics provide both aggregation and analysis capabilities, allowing organizations to take a production environment to a new level of confidence. These systems improve the process of attack identification and alerting while also helping to filter out false positives and low severity incidents. Perhaps more importantly, they help organizations save time, preserve their reputation, and keep production and DevOps staff happy and motivated.