install elk stack google cloud

Trying to find the right cloud provider can be difficult, even when you are looking at it for your log and analytics needs. Here at Logz.io, we use the ELK Stack to monitor our cloud infrastructure as well as offer the log analytics platform as an AI-powered and enterprise-grade cloud service. So, we are always happy to give installation advice whenever we can.

Here, we’ll take a look at installing the ELK Stack on Google Cloud, the cloud infrastructure provided by the popular search engine.

This post is just a proof of concept, but Google Cloud offers a $300 credit for trial users that should be more than enough to deploy an application, get the ELK Stack set up, and prove that it will cover your needs should you want to move forward with Google Cloud.

Logistical Setup

First, you’ll need to set up a Google Cloud Platform account. This is relatively straightforward. Fill out the forms, and you will have your $300/360 Free Trial setup. Setting up your application is easy, it just requires getting the Gcloud SDK setup for CLI deployment.

For this example, we will use a simple ToDo application — the code can be found here. Since the application isn’t the focus here, we can deploy it and hit it a couple of times to ensure that there are some logs to export once we reach this step. The assumption here is that your application will be much more robust and in need of log management.

Getting Started with ELK Setup

Once your application is deployed, logs should start flowing. This means it’s time for the real set up to begin. You begin by setting up Elasticsearch and Kibana on our Google Cloud installation. This is done via the Networking Console in the web interface. To get these pieces in place, you will need to set up firewall rules:

set up google cloud firewall rules

We’ll add Elasticsearch and Kibana with the IP range set to 0.0.0.0/0 and TCP protocols set to 9200 for Elasticsearch and 5601 for Kibana. (Note: Both are all lowercase as the method prescribed by the Google Cloud Firewall Rules interface.) They should appear in the firewall list as follows:

google cloud firewall list

Now, you can begin to set up the actual ELK Stack. The first step is to set up an instance in Google Cloud Platform. Once the instance is setup, you can SSH into it using the following command:

(This assumes that you have gone through the steps with the CLI to establish a connection.)

Here you will begin the installation of Java, Elasticsearch, Logstash, and Kibana. Using the following command sets, each will be installed on your instance.

Install Java

Java installation is simple and straightforward. If you have experience with the programming language, you know to use the following command to make sure that you have the latest, stable version of Java on the instance:

Install Elasticsearch

You will need to adjust the configuration of Elasticsearch so that you can ensure that the network host is correct. To do so, edit the elasticsearch.yml file:

Find the line referring to the network.host portion. It will be commented out. Uncomment the file and make it read network.host “0.0.0.0” — be sure to save the file before exiting.

Back in the console, restart Elasticsearch:

Install Logstash

Your next step is to install Logstash. To do so, use the following commands:

Install Kibana

Finally, do the same for the Kibana service so that you will get nice visualizations on your logs:

Similar to Elasticsearch, Kibana will need some configuration adjustments to work. Take a look at kibana.yml to make these changes:

Find the lines referring to server.port and ensure they say server.port: 5601 and server.host: “0.0.0.0”. It should only be necessary to uncomment these lines. It may also be necessary to adjust the SSL verify line elasticsearch.ssl.verificationMode: none if you are seeing problems once everything is up and running.

Once this is done, you can start the Kibana service:

Success! Well… Mostly

If everything works as planned, you should be able to use the server’s IP with port 5601 specified to see if your setup was successful:

configure index pattern

You’ve come a long way and things are looking up, but you still need to pipe in some logs. For this example, we are going to install MetricBeat to snag system metrics. The installation steps are as follows:

Once MetricBeat is up and running, set it as an index pattern in the Kibana management screen as metricbeat-*. This will begin the shipping of logs, and you will have your ELK Stack set up:

shipping logs to google cloud

There are other beats that can deliver other logs, and you can stack various beats. But just for our purposes here, we have showed you how to successfully run logs on the Google Cloud Platform with the ELK Stack.

Logz.io is an AI-powered log analysis platform based on the open source ELK Stack that can be used to monitor applications and cloud-based infrastructure. Start your free trial or request a free demo today!

PJ Hagerty
PJ Hagerty is a developer, writer, speaker, musician, and developer advocate for Logz.io. He is known to travel the world speaking about programming and the way people think and interact. He is also known for wearing hats. Follow PJ at @aspleenic.