ED&F Man Capital Pursues Observability and Security

About ED&F Man Capital

ED&F Man Capital Markets is a global financial brokerage business and the financial services division of ED&F Man Group. The company offers direct access to global capital markets via world class IT infrastructure and a full suite of Capital Market products including trade processing, financing, clearing, execution, market making, and agency based electronic and voice brokerage services.

Based in Chicago, the company’s IT infrastructure oversees cloud infrastructure, hardware, software, installations, security, automation, monitoring and much more.

For observability, security and monitoring of their cloud environment, Andrew Girin, Director of US infrastructure at ED&F Man Capital Markets, preferred deploying an ELK-based solution.

“Especially with a smaller team, we embrace open source software because it is easier to work with and analyze the underlying code, while participating in the projects and modifying as needed,” Andrew says. “At my previous organization, we stood up our own ELK stack and built a 22 node cluster on AWS. We added some alerting and I became very familiar with the stack, including also with some of the issues with it around scalability and redundancy.”

At ED&F Man, Andrew knew he would have to avoid those operational issues, as he would oversee a smaller team that needed to focus their resources and energy on IT management and operations–and not standing up their monitoring stack and logging pipeline. 

In search of a managed service for the ELK stack, Andrew found Logz.io and selected the tool to support his team’s efforts in managing, troubleshooting and securing their cloud environment. This use case includes sending their Cloudtrail logs via Amazon S3 and utilizing Logz.io’s pre-built Cloudtrail logging rules to trigger internal alerts, as well as to their MSSP, to perform initial Incident Response on the fired rules. 

“I liked the ability to query the data, and that I didn’t have to do any of the parsing. Logz.io is supportive and does it for us, especially our complex logs. This is a very nice service. I send a sample of the logs, and the next day, they are there.”

Centralizing and Advancing Log Management

With Logz.io, Andrew and his team have better visibility into their cloud environment (AWS) and IT infrastructure. They send a number of logs to the platform, including firewall logs, application logs, linux, and more. On the AWS side, Andrew’s team sends S3 Access logs, as well as AWS-Security-Hub logs, to Logz.io’s Cloud SIEM to enrich the data with threat feeds and rules.

“Without the ability to send and analyze these logs, I feel a little blind. And with Logz.io, I can see errors, warnings, performance issues and other events that happen in the log files. It makes it a lot easier to understand things that are happening and achieve observability–we can see CPU is 100%, and then match it to the log file, troubleshoot and take action, for example.”

After implementing Logz.io, ED&F Man quickly noticed increased scale and visibility into their logs, but also the benefits of the additional features of the managed service, including the pre-made dashboards, AI-powered Insights and alerting functionality.

“We create a lot of our own dashboards, and I really like the feature where I can share the dashboard with people, including the third parties we work with. I create a token and share the dashboard for quick intelligence sharing.

To better understand the root cause of IT incidents and errors, Andrew’s team also uses Logz.io’s Insights capabilities to gain real community-driven contextual intelligence.

“We use Insights within the Operations module because it can be hard to see an error in isolation, but with Insights, Logz.io is sort of doing the research for us. Insights shows us where the errors are, based on correlation with other issues that engineers are facing.”

The team also uses Logz.io’s built-in alerting functionality to define and customize event-driven alerts to email, Slack and even, PagerDuty, for additional escalation across the Infrastructure team. 

The Importance of Data Optimization and Enterprise Support 

Shipping and analyzing so many different log files can be a resource and financial drain on many organizations. To help with data optimization and cost containment, ED&F Man takes advantage of Logz.io’s features to streamline and analyze only the most relevant logs.

“We create and use Drop Filters via Logz.io. We filter out a lot of DNS logs, and Cisco logs, for example, because we get a lot of DNS requests, and this is a lot of redundant information that we just don’t need to see on a regular basis. These logs also take up a lot of our bandwidth, so with Drop Filters we can minimize what we send to reduce noisy logs, decrease our bandwidth and control our costs.”

Another differentiator for Logz.io is the extensive customer support that the company provides.

“No matter the issue, with Logz.io’s customer support team, I can get a response in 5-10 minutes. With other solutions, itcan take forever to parse logs. Logz.io provides log parsing as part of the service, and it has been very valuable. In essence, Logz.io is an extension of my IT team and makes life easier.”

A Final Word on the Value of Open Source in Solving the Observability Challenge

“From my experience, proprietary tools are rich in features and may have a full package, but it’s difficult for these solutions to be good at all the different things we need as a team. That’s why I like open source, because I can pick best of breed solutions, whether it’s ELK, or Grafana, or similar, and piece them together.”

With Open Source, Andrew and his team also like the challenge of working with and customizing the underlying code to meet their needs. “I find it can be easier to modify things. You can plug in, modify, submit an approval request and you feel you are contributing a product. But with proprietary tools, you have to play by the rules.”

“Open Source let’s us complete the challenge of finding the best projects, following them and then using the ones we feel are best. We had that moment when we found ELK. It was exciting to see the early dashboards and ultimately incorporate them it into our stack. It can be an interesting challenge, to analyze and compare products.”

Transforming Security Operations and Intelligence with Cloud SIEM

Currently ED&F’s Security Operations Center is run by third party MSSP. For Andrew, the long term vision is to leverage Logz.io’s solutions to move away from this model and manage security internally. 

That’s one of the reasons ED&F Man deployed Logz.io’s Cloud SIEM solution to validate this approach and begin taking control of their risk intelligence

“We find Cloud SIEM to be a very good tool. We are sending a lot of log files and enriching the data with security intelligence related to IP addresses and security ratings. The mapping feature is also very useful, and helps us find relevant things in our environment.

Andrew prefers to own security within his group because his team already has visibility across the infrastructure, and can drill down into different alerts easily and with the help of the Logz.io support team. 

“With Logz.io, I understand the data I am sending from AWS and other sources, and it’s easier to get intelligence. I understand what to look for. I control the data,, without relying on someone else to monitor our environment.”

A Common Use Case for Investigating a Security Incident

Analyzing security risks from log files is critical to Andrew in his team.On the AWS side, Andrew’s team sends S3 Access logs, as well as AWS-Security-Hub logs, to Cloud SIEM to pair the data with threat feeds and rules for faster detection and response.

“It can be a triggered rule or an alert that notifies us of a threat, and then we look at our logs to get a sense of the issue. Today, for example, we also looked at the Cisco logs and saw that some IPs were marked as dangerous. We then investigated further to determine that some of our external sources, third party appliances were communicating with these dangerous IPs. We fixed this by rerouting the sources to internal IPs.”

The team also finds Logz.io’s threat intelligence feeds valuable as they help Andrew understand the IP’s that they are trying to communicate to, and the potential risks associated with each one. ED&F Man also builds out security dashboards based on Windows events, such as failed log-ins and locked out accounts, to help their IT support group.

“With Logz.io’s Cloud SIEM solution, we can gain better security visibility across our environment without sacrificing control of our data. The same log files we ship to Logz.io to troubleshoot infrastructure issues are enriched with security intelligence, such as identification of malicious IP addresses. With the help of alerts and dashboards, we now have a central system across our team to get real-time visibility into potential issues.” – Andrew Girin, EDF & Man Capital