OpenSearch Tutorial: Getting Started with Install and Configuration

OpenSearch is a community response to the recent relicensing of Elasticsearch as a non-Open Source platform. AWS, Logz.io, and a number of partners have been working for months not only to make this merely  compatible with Elasticsearch as a functional replacement, but also seeking to create an independent project roadmap. 

After forking Elasticsearch and Kibana 7.10.2, Version RC1 (1.0.00 of OpenSearch and OpenSearch Dashboards released on June 7, 2021.RC1 is not considered production-ready, but it is feature-complete and incorporates all former Open Distro plugins (along with a couple of new ones), Docker images, Linux tars, alerting, and event  Gantt charts visualization capability (not originally part of the ELK Stack). 

The directions to install and configure OpenSearch are going to remain in place for sometime. With that in mind, this tutorial is meant to help you get started with installation and configuration.

OpenSearch Installation

First you’ll need to download OpenSearch for Docker (and obviously have Docker Compose on your machine). 

On Mac or Linux, head to Terminal. Pull the Docker images for both 1) OpenSearch AND 2) OpenSearch Dashboards (the equivalent to Kibana). 

docker pull opensearchproject/opensearch:1.0.0-rc1

docker pull opensearchproject/opensearch-dashboards:1.0.0-rc1

To move forward, you have to make sure to either remove Elasticsearch or deactivate it. This is because OpenSearch runs on the same default port as Elasticsearch – 9200. A port conflict will prevent you from getting up and running. The same goes for OpenSearch Dashboards and Kibana; both use port 5601 by default. 

Next, run the image:

docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" opensearchproject/opensearch:1.0.0-rc1

You should get a message that includes something like this:

[2021-06-28T10:19:29,483][INFO ][o.o.s.c.ConfigurationRepository] [71235t674gby] Node ‘71235t674gby’ initialized
[2021-06-28T10:20:27,525][INFO ][o.o.i.i.ManagedIndexCoordinator] [71235t674gby] Performing move cluster state metadata.

To continue, open a second tab in Terminal. Send requests to verify OpenSearch is running:

curl -XGET https://localhost:9200 -u 'admin:admin' --insecure

Which should return something like this:

{
  "name" : "9ae0601b601c",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "KkExYRLvQ1CIPA_hoRBYIA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.0.0-rc1",
    "build_type" : "tar",
    "build_hash" : "26d579287f50bb33e17c8fe1f05ea208d5c64d1f",
    "build_date" : "2021-05-28T18:18:49.848386Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  }
}

To deploy your nodes, create a new docker-compose.yml file. You can use the OpenSearch docker-compose.yml template. Save the file in a place that makes sense. I created my own directory for docker-compose files and a separate subdirectory for each project for the individual yml file to live in. This is the template available in the OpenSearch docs:

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
      #- ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml   
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
      #- ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml 
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
    #volumes:
      #- ./custom-opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml
    networks:
      - opensearch-net

This sample creates the smallest ‘cluster’ possible – two nodes (sure, a single node is technically a cluster, but let’s keep the semantics to a minimum here). It also has a single container to run OpenSearch Dashboards (again, on port 5601).

Finally, run your OpenSearch deployment:

docker-compose up 

OpenSearch Configuration

Configuring OpenSearch requires a separate yaml/yml file: opensearch.yml.  You can either 1) create this file with the -v command, or 2) within the docker-compose.yml file mentioned above.

Option 1:

docker run \
-p 9200:9200 -p 9600:9600 \
-e "discovery.type=single-node" \
-v /<full-path-to>/custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml \

opensearchproject/opensearch:1.0.0-rc1

Option 2 (within the docker-compose.yml file; you will have to configure this for each node):

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
      #- ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml

To configure OpenSearch Dashboards the same way:

 opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
    #volumes:
      #- ./custom-opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml
    networks:
      - opensearch-net

Plugins

OpenSearch has built-in plugins, which are carryovers from Open Distro (which originally built unique plugins to be assimilated to service Elasticsearch, but now have been adapted to service OpenSearch). By the way, you can – and should – check out Amitai Stern’s post about building Opensearch plugins.

Our tutorial here covers the all-in-one OpenSearch deployment. If you want, there is a minimum version of OpenSearch without the formerly Open Distro plugins built-in, that you can elect to install yourself.

To set up an image with another plugin, follow this syntax:

FROM opensearchproject/opensearch:1.0.0-rc1

RUN /usr/share/opensearch/bin/opensearch-plugin install --batch <opensearch-plugin-name>

Then run:

docker build --tag=opensearch-custom-plugin

docker run -p 9200:9200 -p 9600:9600 -v /usr/share/opensearch/data opensearch-custom-plugin

Going Forward

Future tutorials will cover more detail, including clusters, mapping, queries, aggregation, and more. If you have questions, please feel free to add comments and request more detailed walkthroughs in the future!