Form3 Mitigates Issues Faster & Reduces Costs with Logz.io

Form3 helps banks and regulated fintechs move money faster by providing a fully managed payment technology service combining a powerful AWS native-cloud processing platform, multiple payment gateways, workflows and a fully managed 24×7 service model.

Form3 provides a complete end-to-end Payments as a Service for financial institutions and regulated fintechs. Reliable and secure cloud-based access and processing for FPS, BACS, CHAPS, SEPA, SEPA Instant & SWIFT with no hardware or software required by our customers. The service is robust and reliable, trusted by the world’s leading financial institutions.

Form3’s powerful microservices architecture and API’s provide a single point of simple access to the full universe of payment schemes. Future-proofed at the point of delivery for complete peace of mind, the result is a straightforward, easy to use and hassle-free Payments as a Service.

Logging a microservices architecture

Logz.io helps Form3 develop, test and monitor what is a highly distributed architecture deployed on AWS. In total, more than 100 EC2 instances ship various types of infrastructure and application data to Logz.io, helping the team gain insight into how their development, testing and production environments are performing.

Form3’s applications are based on multiple containerized services deployed using Amazon ECS. Using the fluentd Docker logging driver, Form3 ships what are primarily Java and Go JSON-formatted application logs to Logz.io. Metricbeat is used for shipping system metrics for performance monitoring. Using a Lambda function for shipping data from CloudWatch, Form3 also ships ELB and ALB logs to Logz.io.

Parsing is performed automatically by Logz.io for JSON-formatted logs, while other types of logs generated by additional frameworks and tools used by the team, such as Consul, Zipkin and Chaos Monkey — required custom parsing performed in fluentd and withe help of Logz.io’s Support team.

Faster troubleshooting with machine learning

Pulling large amounts of log data from multiple sources, development and operations teams face the challenge of separating the wheat from the chaff. Form3 is no exception to this rule, but using Logz.io’s machine learning features, it has been able to shorten time to resolution in a number of cases.

Using Cognitive Insights — Logz.io’s advanced analysis tool that identifies correlations between log-based events and discussions happening on the web — Form3 was able to quickly identify a database issue caused by a crawl exception.

Once identified, and using the data provided by Logz.io the issue was resolved quickly before having a critical impact on Form3’s transaction services.

Reducing operational costs

Not all logs are born equal. Some log data requires shorter retention periods while other require extended retention. Form3 analyzes specific metrics in the log data, such as transaction volumes, API response time, transaction times, and transaction limit, for the purpose of business intelligence.

This type of analysis requires extended retention which could result in substantial operational costs. However, using Logz.io, Form3 has been able to cut these costs by aggregating and storing these metrics in a Timeless Account —  a dedicated account for indexing data for extended periods of time.

Shipping data into the Timeless Account is done using the Data Optimizer, a set of definitions that define what data to aggregate and collect, at which frequency and which account to ship it to.

A Kibana dashboard was constructed for analyzing and monitoring these metrics. The same dashboard is used for historical trend analysis on the data stored in the Timeless Account as well as for real-time monitoring of the same data in the regular account. This dashboard monitors API response times, transaction volumes, status of transactions, and more.

Advanced monitoring capabilities

Collecting, processing and analyzing log data generated by multiple microservices is a monitoring challenge that many companies find extremely hard to overcome. The amount of data being generated obscures visibility and makes it extremely difficult to identify issues quickly enough to ensure they do not have an impact on business.

Using Logz.io, Form3 have been able to both focus on developing their core business rather than deploying and maintaining a centralized logging system of their own, and shorten the time it takes to identify issues and resolve them. The combination of advanced analysis and visualization features, alerting, machine learning and data optimization, provides Form3 with a powerful monitoring solution for their growing distributed architecture.  

Aggregating logs is an essential part of running a microservices architecture. Logz.io is the perfect partner to get this job done right. We’ve been using the Logz.io platform for many years and have built up a lot of trust in the platform and the team behind it.

Sam Owens, a Senior Engineer at Form3