The Central Source of Truth: Fall Guys and Mediatonic

Mediatonic is a sprawling video game studio based in the UK, with a number of successful titles to their name: Heavenstrike Rivals, Gears POP!, and Murder by Numbers among them. In 2020, they struck gold again with Fall Guys: Ultimate Knockout. But this game would be special, and the need of handling these kinds of gaming logs at this kind of scale would be, too.

This battle royal-style fighting game pits 60 players against each other until one reigns supreme. To handle the launch of that sort of multiplayer rumble, repeated tens of thousands of times with players all over the world, they were going to need unprecedented levels of insight into their game’s inner mechanics and gaming insights.

Fall Guys took off in August 2019, but months went into preparing the launch. As you can imagine, monitoring the performance of all this action, especially with the bandwidth stressed state of lockdown-laden 2020, required a heavy amount of lifting. Logging at scale is a challenging prospect.

The technical operations team for Mediatonic, led by Rory Marquis, technical director for Fall Guys, had a trick up its sleeve to monitor the launch of what they knew would be one of the biggest games worldwide from lockdown. It all starts with data.

Spring into Motion: Break Out While in Lockdown

Back in April 2020, they were trying to stack their gaming logs.

“We had a lot of log files and a lot of servers and a lot of data.” But they only had insights and a graph for a single server. “Once you have a visualization of data or just any data, you always want more. And we knew from our beta test that we needed more.”

Rory and co. moved to pull, pack, and pass all the logs they had through a homegrown Python script. They downloaded all the gaming log data from all the servers and managed to get some KPIs out of it. But moving closer to release in August, they decided they needed to look for external answers to their game-logging problems. 

Logging Games with Logz.io

Rory received several recommendations to go with Logz.io. After getting in touch, Rory’s team outlined what they needed: centralized data, a single source, good cost prediction, and saving their engineers time to focus on scaling. Still, he wasn’t optimistic that the complex needs he saw in his game’s beta testing could be met, and wondered if the use case was too complex.

Then he took to Logz.io’s online chat service. With clarity and focus, they identified things right away. Very quickly they were able to identify the couple steps that were missed along the way. Within 10 minutes more of coding they were able to trail all of the data coming in. It was initially overwhelming to have so much data streaming live so quickly, far greater in scale than he thought gaming logs and monitoring would ever allow.

“I realized I was in the Matrix!” he said. “It was pretty simple to do and it basically took about 10 minutes to get a minimally viable connection and product up.”

Gaming Logs Fit to Scale

Next, it was time to build out the visuals. Rory dialed it back and tried to get the hang of things. He managed to set up a counter to look at all the users coming from his personal system. Now, he wanted to get the right guidance to immediately learn the extras. It had to work for all the servers and thousands of players Mediatonic anticipated after launch. But what about the cost?

“I didn’t quite understand how Logz.io could be less than the figure on the website, especially for something with the demands of a gaming company,” he said. “But, amazingly, I learned that you can drop logs that you don’t need, reducing the data Logz.io collects and making the monitoring process all the more efficient.”

In the long run, the ability to drop a log meant that it could be ingested, passed, put through the system, but if the team didn’t want it, Logz.io would drop it for them. That would let them achieve the cost they wanted and actually retain the most important data that they wanted. With the swipe of a toggle, that setting could be activated or deactivated.

Summer Heat: Launch Approaching

In July 2020, they had Logz.io integrated and ready for launch. The server stats looked so, so different at that point.

Everything was now time-serialized, centralized, and persisted, allowing for easy comparison between one moment and another. Even after getting everything they needed and wanted, his team uncovered an unexpected use case.

Receiving a Kibana alert in Slack, they were able to see deeper than they expected. The engineers looked at the hitch summary from the server and were able to filter that down to just the major hitches and see which particular server was the problem, making it easy to identify fast.

Gaming Management Insights in an Instant

“We didn’t expect to be able to click directly to show the logs. It will open Kibana at the exact point, with the exact query information we needed to use…and take that information and 15:20 extrapolate it all out and look at all the servers across the whole network. That was something we could not have done before…It was literally 20 or 30 seconds to do all this.”

In the past, Rory’s team would have downloaded all the logs and passed them through their original Python script. But those days were far in the past. From there, they could break down gameplay by platform, use over time, and region simultaneously. They were able to see how many servers and players were on each platform, in one place, at one time, and display that in the studio on this dashboard.

By August, Logz.io was monitoring hundreds of thousands of users at one time for Mediatonic, recording dips and peaks in activities broken down by country and hour. 

Some might be at the beginning of their monitoring strategy and visualization journey. But, the use cases that Logz.io affords companies are beyond what many anticipate when they start using the platform. Whether it is a specific use case like game monitoring, or larger dev category like server monitoring, Logz.io anticipates the unique needs of each task.

As the only, long-demanded DevOps tool blending ELK, Prometheus, and Jaeger, Logz.io lets teams focus on productivity more than maintenance.