Why Log Data is Important for Compliance

Over the last few years, we’ve seen a massive shift in the way enterprises consume software. The move from on-premise to software-as-a-service (SaaS) has been unprecedented. However, enterprise and web-scale software requirements have not changed during this time.

Enterprise and web-scale software have always had requirements for:

• High availability
• Built-in preparedness for disaster situations
• Integrity assurances that access to system management functions are limited to a “need to know” basis and that changes to software can be monitored so that no unauthorized changes occur
• Instrumentation and monitoring

It used to be that companies would combine management consoles and application tools to accomplish some of this and assemble other solutions to take care of their enterprise and web-scale systems. All this “assembled infrastructure” software was architected and run by people in the company. It was a lot of extra work!

Welcome to Software as a Service in Which I Don’t Run My Software Anymore

Now, we expect to get both the applications and the infrastructure to come “as a service.” This means that all of the architecting and running of the “assembled infrastructure” is in the hands and depends on the skills of the SaaS operator. But is the SaaS operator doing this correctly? How does one make sure?

Compliance Certifications, Endorsements, and Qualifications Are Everywhere

To answer this question, one can turn to any mature industry and find all kinds of labels and badges and logos on products that indicate compliance with some kind of standardized requirement. In foods, we see indications of “organic,” “kosher,” “non-GMO,” and so on.

Have you looked closely at the logos for compliance certifications, endorsements, and qualifications for electrical equipment lately? Take a look at this computer-charging “brick”:

One can be sure that it meets expectations no matter in which country you are using it!

What is Compliance for Software and Software Services?

The “brick” above has a very well-defined set of logos that signify compliance in various areas. One knows that each logo represents adherence to a particular international organization’s requirement or specification in an area such as safety, heat, energy conservation, materials composition, and lifetime expectation. Buyers of these bricks, especially those who work in security and compliance, know of – and insist on – every one of these logos being on this product before they will buy them.

Over the last few years, similar compliance requirements and certificates have been developed in the SaaS industry. Sophisticated web-scale and enterprise solutions architects know these compliance profiles, and more and more are insisting on them as procurement and operational requirements. But it wasn’t always this way in software compliance.

Compliance Started as a “Built With Quality Processes” Metric

The software is a type of technology that has always been difficult to put rules and regulations around. To see sophisticated and well-defined compliance profiles around software is a very recent phenomenon.

As recently as ten years ago, the first specifications for software construction were just being written. The original ISO9000 family of specifications described methodologies to help developers “build quality in.” Developers were guided to document their systems properly and test each area that was documented. When software solutions were more monolithic and less interconnected, the demands on overall, networked system behavior remained the responsibility of the customer.

Compliance Has Emerged as a “Safety and Reliability” Value

Now, the security and robustness of each element in a network system contributes to or detracts from the overall security and robustness of that total system. A vulnerability in any one element becomes a vulnerability in the total system. As a result, compliance has emerged to specify overall “safety and reliability.”

As all of the various parts in a system come together – applications and infrastructure – one can be sure that the compliance of the platform as a whole will be the sum of the compliance of the parts.

Industries Require Compliance, It’s Not a Choice

An interesting development in compliance is how the practice is “going viral.” Usually, when we say “going viral,” we envision some kind of amazing video or image that is sexy or entertaining. But anyone who has worked in compliance knows that there is nothing sexy or entertaining about compliance. So, what does “compliance going viral” mean?

What is happening is that software companies are building SaaS solutions that are using elements of other third-party solutions — and so on and so forth up and down the supply chain. Every part of the chain needs to be compliant for all parts to be compliant. Customers that ask for a vendor to be compliant will likely find themselves as a vendor one day that also needs to be compliant to satisfy someone else.

As a result, compliance is becoming a “viral” phenomenon. It is becoming “table stakes” – a top-line requirement for SaaS solutions. Procurement departments are making these guidelines prerequisites for purchases.

The Central Role of Logging in Compliance

Applications and solutions – whether on-premise or SaaS – need to have infrastructure that ensures their overall robustness and functionality and provides visibility into what is going on in the system. What components are failing? Are they gone or just overworked? What does IT say is happening?

Logging is one of the most important methods to ensure visibility and obtain insights into one’s infrastructure. This is why every compliance profile requires logging to be an active component in every system. One simply cannot have a total solution that is compliant unless logging is included!

Logging is the Truth of “What Happened”

Why do almost all compliance profiles require logging? When administrators, developers, and security professionals look into a system issue, they turn to the logs for “situational analysis.” Log analysis is the “trusted source of truth” on how software is running and has been recognized as the best method to gain visibility and insight into one’s platform software.

Compliance for the Logging System Itself is Exponentially Important

Log analysis, then, is much like looking at the speedometer of a car. It must be reliable and accurate. During any system incident, problem, or other critical situation — just as during any acceleration or deceleration of a car — the measurement system must be trustworthy, available, and capable of continuing operations.

Therefore, the compliance of the logging system itself is exponentially important. Using a logging system that is not compliant simply does not make sense. When overall system dynamics become unstable or overloaded and malfunctioning subsystems begin to emit vastly increased volumes of error messages, these are precisely the high-demand, high-risk times that one’s logging system must be up to the task.

Using a logging system that is not in itself compliant is like going cheap on car brakes, parachutes, or smoke alarms. As Benjamin Franklin said, an ounce of prevention is worth a pound of cure – do-it-yourself logging without a compliant platform is just an IT disaster waiting to happen.

Compliant Logging Provides Insights You Can Trust

Logging is getting much more sophisticated as Big Data analytics techniques aid in the accelerated understanding of situational analysis. It is no longer just humans who are interpreting the log data; now, the analytics capabilities of logging systems can present insights themselves correlated with information, trends, and predictions for system administrators, security analysts, and network and application engineers.

As the expression goes – “garbage in, garbage out.” If one is relying on a logging system not only to acquire and deliver information but also to intelligently process it using insight and analytics, trusting that system is more important than ever. It is one thing for a logging system to leave system behavior information opaque; it is another to make a wrong prediction.

Tomorrow’s Logging Will Be Built on AI & Machine Learning

As logging systems transform into intelligent analytical systems with predictions and insights, engineers are now enhancing these platforms with machine learning algorithms. In many ways, logging uniquely has both the depth and breadth of system behavior metrics to be a futile ground for machine learning.

Artificial intelligence – scanning the logs in real-time – is also a future direction for logging. Computing systems will soon be able to self-inspect, self-learn, and self-heal — the holy grail of “autonomic systems.” But an autonomic system built without the fundamentals of compliance will not function properly – it will not have high-enough availability, trustworthiness, and integrity to become autonomic.

This only further underscores the need for careful engineering that goes into making a logging system compliant.

Compliance is More Than a Checkmark

Compliance is critical for logging systems. That’s why we at Logz.io have put so much effort into this area. Our platform is SOC-2 Type 2 as well as ISO 27001 compliant and maintains data locality for EU customers with data centers in both the United States and Germany. For more information on the compliance of our log analysis software, download our white paper (PDF). You can also start a free trial of our AI-powered log analysis platform that offers the open source ELK Stack as an enterprise-grade cloud service with machine learning technology.