Why We Built Our Log Analytics SaaS on AWS

Logz.io has launched less than a year ago and already has hundreds of customers from 55 countries across the globe. Our Big Data, microservices infrastructure requires complex networking, a lot of storage, high availability, strong security and complete compliance with international regulatory standards such as SOC 2 and ISO 27001. Trying to build this infrastructure from scratch on a bare-metal infrastructure would have delayed us significantly and made growth a much bigger challenge.

Our platform was designed, developed, and deployed on Amazon Web Services because we knew it could help us to achieve and sustain a high growth rate. Many of our readers and others in the community are considering whether to deploy on the cloud, so I wanted to tell our story in case it will help others with their decisions.

This post explains why we at Logz.io specifically felt that AWS was best-suited for our technical requirements and growth projections. Of course, others might have different opinions.

Scalability and high availability

Logz.io offers enterprises a log analytics SaaS that is based on an open source stack (the ELK Stack) and adds additional capabilities that are based on machine learning and crowdsourcing. Since the base of this stack is Big Data consumption, storage, and analysis, it was clear from the start that our platform would need to store large and ever-growing amounts of data as well as process and search all of it.

Our servers needed to be fault-tolerant and make sure data would always be replicated across the ingestion pipeline. If a failure would occur anywhere due to a bug or a hardware failure, the customer should not experience any degradation of service.

We also needed our platform to respond gracefully to data bursts, which are an all-too-frequent occurrence in log analytics and IT operations. Every layer of the stack required the highest degree of scalability along with the automatic ability to scale out horizontally and scale down when the burst is done.

To maintain an optimized instance availability, we crafted an auto-scaling policy on top of AWS that is constructed of a wide set of rules and conditions that instantiate new servers, re-route data, or move it around in case of errors and high traffic.

Security and compliance

Another crucial part of our SaaS platform is security. In fact, security is one of the main reasons that our users choose us over our competitors.

Logz.io relies on AWS infrastructure to store data across multiple AWS cloud regions and availability zones. AWS makes adherence to industry and government requirements relatively simple and ensures the utmost in data security and protection. For example, AWS infrastructure aligns with IT security best practices and follows a number of compliance standards such as HIPAA, SOC 2, and FISMA. This contributed immensely to Logz.io receiving SOC 2 Type 2 compliance.

The company’s network is segmented using AWS security groups, VPCs, and ACLs. Data is segregated, tagged, and processed on different virtual pipelines per each customer. We leverage EBS encryption to make sure that data is kept safe and have added several additional custom measures to measure usage and detect malicious usage specifically. In addition, a threat-control center is kept up-to-date with security alerts that are analyzed and addressed in real time. Using AWS together with in-depth network monitoring, Logz.io is able to detect critical events and take a more proactive threat-assessment strategy.

Management and monitoring

With higher and higher rates of growth with every month, we knew that any cloud computing solution that we would choose would have to supply the best-in-class management tools and monitoring capabilities.

After all, we needed to make sure that our engineers could focus mainly on the business logic and be improving the product and not on the underlying infrastructure. Obviously, we use Logz.io as a key monitoring tool, but we also use CloudWatch and CloudTrail to carefully monitor and alert on any events that relate to security and access control.

Maturity and feature availability

Another consideration was the overall capabilities and tools offered by AWS. Microsoft Azure and the Google Cloud Platform have both made great progress over the past few years, but we felt that there is still a bit of a gap — especially in relation to high growth and data-intensive systems.

Our application needed to support the daily ingestion of petabytes of data and run complex map reduction and machine learning labeling. So, for large-scale environments such as ours, AWS was better equipped to provide the needed support.

It was also crucial to build our platform on a cloud that had a wide variety of integrations with other technologies that we use (such as DevOps management and configuration tools) as well as programming languages (such as Puppet and Ansible).

Community support was another reason. The AWS ecosystem is huge, and the resources that are available help to minimize the learning curve and reduce the overall cost in terms of time and money.

Our AWS customer base

Knowing that many of our potential customers run their solutions on AWS as well, selecting AWS as our cloud computing solution made even more sense.  Integrating with our product would cost less, both in terms of time and money.

The bonuses

Opting for AWS had some additional perks that are not often mentioned but should nevertheless be discussed because they can help other aspects of your SaaS platform such as marketing.

AWS runs a partner program (the AWS Partner Network) that can assist young companies with marketing campaigns as well as their strategic positioning in the market. While it is somewhat costly, an AWS partnership increases your brand visibility in your field and alongside your competitors.

Our conclusion

Considering all of the above, it was our conclusion that AWS offered us the best solution by providing us with a sophisticated and robust-enough cloud infrastructure that promised to support both our product and the needs of our growing customer base. So far, we are happy with our choice.

Remember — there is no perfect solution, and you will likely find similar success stories involving  Azure and Google Cloud Platform. Ultimately, each company has its own story, with different needs and requirements. This is the Logz.io story.