OpenSearch Tutorial: Getting Started with Install and Configuration
OpenSearch is a community response to the recent relicensing of Elasticsearch as a non-Open Source platform. AWS, Logz.io, and a number of partners have been working for months not only to make this merely compatible with Elasticsearch as a functional replacement, but also seeking to create an independent project roadmap.
After forking Elasticsearch and Kibana 7.10.2, Version RC1 (1.0.00 of OpenSearch and OpenSearch Dashboards released on June 7, 2021.RC1 is not considered production-ready, but it is feature-complete and incorporates all former Open Distro plugins (along with a couple of new ones), Docker images, Linux tars, alerting, and event Gantt charts visualization capability (not originally part of the ELK Stack).
The directions to install and configure OpenSearch are going to remain in place for sometime. With that in mind, this tutorial is meant to help you get started with installation and configuration.
OpenSearch Installation
First you’ll need to download OpenSearch for Docker (and obviously have Docker Compose on your machine).
On Mac or Linux, head to Terminal. Pull the Docker images for both 1) OpenSearch AND 2) OpenSearch Dashboards (the equivalent to Kibana).
docker pull opensearchproject/opensearch:1.0.0-rc1
docker pull opensearchproject/opensearch-dashboards:1.0.0-rc1
To move forward, you have to make sure to either remove Elasticsearch or deactivate it. This is because OpenSearch runs on the same default port as Elasticsearch – 9200. A port conflict will prevent you from getting up and running. The same goes for OpenSearch Dashboards and Kibana; both use port 5601 by default.
Next, run the image:
docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" opensearchproject/opensearch:1.0.0-rc1
You should get a message that includes something like this:
[2021-06-28T10:19:29,483][INFO ][o.o.s.c.ConfigurationRepository] [71235t674gby] Node ‘71235t674gby’ initialized
[2021-06-28T10:20:27,525][INFO ][o.o.i.i.ManagedIndexCoordinator] [71235t674gby] Performing move cluster state metadata.
To continue, open a second tab in Terminal. Send requests to verify OpenSearch is running:
curl -XGET https://localhost:9200 -u 'admin:admin' --insecure
Which should return something like this:
{
"name" : "9ae0601b601c",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "KkExYRLvQ1CIPA_hoRBYIA",
"version" : {
"distribution" : "opensearch",
"number" : "1.0.0-rc1",
"build_type" : "tar",
"build_hash" : "26d579287f50bb33e17c8fe1f05ea208d5c64d1f",
"build_date" : "2021-05-28T18:18:49.848386Z",
"build_snapshot" : false,
"lucene_version" : "8.8.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
}
}
To deploy your nodes, create a new docker-compose.yml file. You can use the OpenSearch docker-compose.yml template. Save the file in a place that makes sense. I created my own directory for docker-compose files and a separate subdirectory for each project for the individual yml file to live in. This is the template available in the OpenSearch docs:
version: '3'
services:
opensearch-node1:
image: opensearchproject/opensearch:latest
container_name: opensearch-node1
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node1
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_master_nodes=opensearch-node1,opensearch-node2
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data
#- ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- opensearch-net
opensearch-node2:
image: opensearchproject/opensearch:latest
container_name: opensearch-node2
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node2
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_master_nodes=opensearch-node1,opensearch-node2
- bootstrap.memory_lock=true
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- opensearch-data2:/usr/share/opensearch/data
#- ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml
networks:
- opensearch-net
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:latest
container_name: opensearch-dashboards
ports:
- 5601:5601
expose:
- "5601"
environment:
OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
#volumes:
#- ./custom-opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml
networks:
- opensearch-net
This sample creates the smallest ‘cluster’ possible – two nodes (sure, a single node is technically a cluster, but let’s keep the semantics to a minimum here). It also has a single container to run OpenSearch Dashboards (again, on port 5601).
Finally, run your OpenSearch deployment:
docker-compose up
OpenSearch Configuration
Configuring OpenSearch requires a separate yaml/yml file: opensearch.yml. You can either 1) create this file with the -v command, or 2) within the docker-compose.yml file mentioned above.
Option 1:
docker run \
-p 9200:9200 -p 9600:9600 \
-e "discovery.type=single-node" \
-v /<full-path-to>/custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml \
opensearchproject/opensearch:1.0.0-rc1
Option 2 (within the docker-compose.yml file; you will have to configure this for each node):
version: '3'
services:
opensearch-node1:
image: opensearchproject/opensearch:latest
container_name: opensearch-node1
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node1
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_master_nodes=opensearch-node1,opensearch-node2
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data
#- ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml
To configure OpenSearch Dashboards the same way:
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:latest
container_name: opensearch-dashboards
ports:
- 5601:5601
expose:
- "5601"
environment:
OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
#volumes:
#- ./custom-opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml
networks:
- opensearch-net
Plugins
OpenSearch has built-in plugins, which are carryovers from Open Distro (which originally built unique plugins to be assimilated to service Elasticsearch, but now have been adapted to service OpenSearch). By the way, you can – and should – check out Amitai Stern’s post about building Opensearch plugins.
Our tutorial here covers the all-in-one OpenSearch deployment. If you want, there is a minimum version of OpenSearch without the formerly Open Distro plugins built-in, that you can elect to install yourself.
To set up an image with another plugin, follow this syntax:
FROM opensearchproject/opensearch:1.0.0-rc1
RUN /usr/share/opensearch/bin/opensearch-plugin install --batch <opensearch-plugin-name>
Then run:
docker build --tag=opensearch-custom-plugin
docker run -p 9200:9200 -p 9600:9600 -v /usr/share/opensearch/data opensearch-custom-plugin
Going Forward
Future tutorials will cover more detail, including clusters, mapping, queries, aggregation, and more. If you have questions, please feel free to add comments and request more detailed walkthroughs in the future!