Collect, monitor, and process AWS logs and metrics at scale with Cognitive Insights

Originally posted as “How to Collect, Monitor, and Process Logs and Metrics at Scale with Cognitive Insights” on AWS Partner Network (APN) Blog | 05 Feb 2019 | Shashi Raina, AWS Partner Solution Architect at AWS and Daniel Berman, Product Marketing Manager at Logz.io.

Famed management thinker Peter Drucker is often quoted as saying, “You can’t manage what you can’t measure.” Tracking and analyzing data of a system provides metrics to measure, predict, and improve the underlining health of the system. 

Logging data is the simplest act of collecting data for measurement and plays an important role in modern enterprises, as it provides a way to measure the health of hardware devices and software applications alike. Enterprises can be on-premises, on the Amazon Web Services (AWS) Cloud, or a combination of both. Log sources, meanwhile, can be network devices, operating systems, applications, or cloud services, to name a few.

When this log data is large in volume, high in velocity, or has lots of variety across formats, it poses challenges for data storage, processing, and enrichment. It’s not enough to just store the data; you need to react to it. In order to react to data, you need to visualize it and set up alerts to get notified if something of interest happens.

For an alert or notification to be effective, the system should have the ability to reduce the noise by providing unique events and contextualize errors or notifications that need real and immediate attention. The last thing you want is your operators being bombarded by false alerts and getting desensitized to critical errors when the time comes to act immediately.

In their blog post, AWS explores Logz.io features, our Cognitive Insights AI engine for advanced data insights, and how to get started monitoring AWS services. Logz.io is a proud AWS Partner Network (APN) Advanced Technology Partner with AWS Competencies in both Data & Analytics and DevOps. Check out the full post on the AWS Partner Network Blog here and look out for more on Logz.io and AWS!