Debugging and Troubleshooting Faster to Improve the Customer Experience

About Blue dot

Blue dot is an automated, enterprise-wide, cloud-based VAT recovery and compliance solution that has successfully impacted  the global VAT recovery process. Sophisticated AI tools give Blue dot customers complete VAT visibility insights, maximizing VAT returns with an unprecedented 98.7% on average VAT recovery.  Deep learning algorithms help companies achieve the highest levels of compliance – 99% on average – and easily detect incidents of fraud within its T&E expenditures.

Business Drivers for Better Performance Monitoring

Blue dot provides operational technology to help Fortune 500 businesses streamline and cut costs related to travel and entertainment. With increased adoption of their technology, the company’s engineering organization began to prioritize the health of their production environment to ensure their applications were highly available and performant for their customers, with minimal down time or failed requests.

To help achieve production environment resilience, Amit Zohar, Architect for Blue dot emphasized the importance of analyzing logs across production, QA and development in order to debug and troubleshoot issues before they could impact the end user experience. 

With Blue dot’s preference for open source and SaaS solutions, the company originally deployed an in-house ELK stack to help the engineering organization monitor and analyze their logs. However, as the data grew and the  production environment scaled the ELK stack became difficult to to monitor, troubleshoot, and upgrade on a regular basis. In addition, their internally managed ELK stack required constant upgrading to the latest edition of Kibana. On top of that, they had to add engineering resources to oversee log parsing, mapping and performance tuning.

“With ELK, we had issues scaling the solution to meet the increased demand for our service and higher volume of logs. While we didn’t necessarily face a problem with the ELK stack handling an average load of logs, there were always problems handling bursts or sudden increases in load. In those cases it sometimes took hours — even sometimes up to 24 hours — to catch up with the log data,” said Zohar.

With the overall amount of computing resources needed to process all of this traffic overwhelming the organization’s ELK deployment, Zohar knew they had to streamline things. A managed service that could offload the costs of maintaining in-house ELK made more sense to the organization.

Scaling for Success with Logz.io 

With an interest in maintaining their ELK environment but mitigating the time and effort to manage log processing during peaks, Blue dot deployed Logz.io to centralize log management.

“We could not keep dealing with the time constraints and resource drain of ELK, so we knew it was time to call in the experts,” said Zohar.

During deployment, Blue dot immediately recognized value from Logz.io due to the company’s extensive parsing services and dedicated support team.

“With Logz.io, the parsing of our data is taken care of, the fine tuning and mapping of our data is taken care of, and the support and customer success team is a partner that helps us analyze what we send to become more efficient, especially during our bursty periods.”

Blue dot primarily ships application logs to Logz.io, but also some logs are pulled from their cloud infrastructure (AWS), TImeless accounts, Apache Spark and other sources. The logs are shipped and visualized in Kibana to better understand the performance of their environment, debugging and troubleshooting issues as necessary. 

To help take their incident detection and troubleshooting to the next level, Zohar and team use the Log Patterns feature to eliminate unnecessary logs that provide little value. This helps the business stay more efficient during periods of high traffic and excessive reliance on computing resources by controlling data management and cutting costs

“The advanced capabilities beyond the core ELK platform also help Logz.io stand out as a differentiated tool. By helping us better manage and analyze our logs, we feel our team has less of an infrastructure and maintenance load to carry. We can then spend more time building and deploying code to better serve our customers.”