Creating a Docker Image with Elasticsearch, Logstash and Kibana

By | August 23, 2015

In this article I will create a Docker image that contains the Elasticsearch, Logstash and Kibana – more commonly known as the ELK stack. This Docker image will be based on the Docker image with Java 8 on Ubuntu that I created in an earlier article.

I am aware of there being Docker images containing the ELK stack on Docker Hub.
I am also aware of the recommendation to create one separate Docker image for each application, that is one for Elasticsearch, one for Logstash and one for Kibana. For the time being one Docker image with the entire ELK stack is sufficient for my needs. I have considered creating three separate Docker images, but that will have to be the subject of another article.

Prerequisites

In order to be able to build the Docker image created in this article, you will need to have built my Docker image with Java 8 which I named “krizsan/ubuntu1504java8”.
After the last article, the kind people at Docker have released the Docker Toolbox which greatly simplifies installation of Docker and associated utilities. The Docker Toolbox I have used contains Docker 1.8.1.

Layout of the Docker Image

The Docker image has the layout as shown in this figure:

Layout of the Docker image containing the ELK-stack.

Layout of the Docker image containing the ELK-stack.

Configuration files and logs have been located to separate directories in order to allow for convenient mapping of host directories into Docker containers created from this image.

Docker File

The Docker-file for the ELK stack looks like this:

Short Description

The above Docker file prepares an image based on my previous Docker image with Java 8 on Ubuntu 15.04. In this image, Elasticsearch, Logstash and Kibana are downloaded and unpacked.
A special user, “elk”, and a group with the same name as the user are added. Elasticsearch, Logstash and Kibana will all be run by this user rather than the root.
The Kopf plug-in, which is a web administration tool for Elasticsearch, is installed in Elasticsearch. The JMX plug-in is installed in Logstash, as to make it possible to poll Java Virtual Machines for JMX data.
The Kibana configuration is modified as to write log to a file instead of to the console.

Configuration and log files have been placed in dedicated directory structures, as to allow for convenient mapping to host directories.

Longer Description

The Docker file start with some comments on how to access Elasticsearch administration tool, Kibana etc. All URLs assume that you have mapped the appropriate ports from the Docker container to the same ports on the host and that you replace “[docker-container-ip]” with the IP address of the Docker machine.

  • FROM krizsan/ubuntu1504java8:v1
    As in the previous article, this specifies the
    Docker image on which this new Docker image is to be based upon. For those that haven’t read my previous article, the image used is my own Docker image containing Ubuntu 15.04 and Oracle’s Java 8.
    Docker reference.
  • Next follow a number of ENV declarations.
    These are similar to constant declarations, which prevents duplication of, for instance, file paths, in the Docker-file.
    Docker reference.
  • WORKDIR /opt
    Makes /opt the current working directory, similar to the cd shell command.
    Docker reference.
  • RUN groupadd -r elk && useradd -r -g elk elk
    The Docker RUN instruction executes commands. The groupadd shell command creates a system group with the name “elk” and the useradd command creates a system user “elk” belonging to the system group that was just created. Note that the “elk” user will not have a home directory, since it is a system user. This is the user that will run the programs of the ELK stack in the Docker image.
    Docker reference.
  • The next RUN instruction installs gosu, which allows us to start processes that belong to a specific user.
  • COPY ./start-elk.sh /opt/
    Copies the script that is to
    launch the programs of the ELK stack when a Docker container is started from the directory in which the Docker-file is located to the /opt/ directory in the Docker image. We will examine this script more closely in a little while.
    Docker reference.
  • The next Docker instruction is RUN, which is followed by a large number of shell commands.
    As in the earlier article, the reason for executing multiple shell commands with one single RUN instruction is that each time the RUN instruction is invoked, a new layer is created in the Docker image. This will lead to a Docker image which is unnecessarily large.
    Docker reference.

  • The three wget commands will download Elasticsearch, Logstash and Kibana respectively.
    Since the working directory is /opt, the three archives will be downloaded to this directory in the Docker image.

  • I have chosen to place Elasticsearch, Logstash and Kibana in directories that does not contain the version number in order to simplify both the Docker-file and the startup-script.
    The three mkdir commands create the home directory for each of the three components of the ELK stack. The -p flag cause any missing parent directories to be created as well.

  • In the next step the downloaded archives are unpacked using the tar command.
    The x flag tells the tar command to extract files from an archive, the z flag cause unpacking of a gzip-archive and the f flag is used to specify the location of the archive from which to extract files. In addition, the C flag is used, which selects the directory to which the extracted files are to be written. The –strip-components=1 option will cause the first directory component of the files in the archive to be removed. If, for example, the path of a file in the archive is /dir1/dir2/dir3/file.txt the file will be written to dir2/dir3/file.txt in the destination directory when –strip-components=1.

  • Having finished unpacking the Elasticsearch, Logstash and Kibana archives, they are deleted using the rm command.

  • Next up is creation of the directories that are to hold log files for the three applications.
    I have chosen to locate these in a dedicated logs directory in order to be able to mount a host directory as a directory for log files.

  • In a similar fashion, three directories are created that are to hold configuration-files for the three applications.

  • The two cp commands copy configuration files from the application home directories to the external configuration directories just created.
    This creates a default configuration setup in the Docker image. If the user of the Docker image choose to map the entire external configuration directory (/opt/config) to a host directory or individual external configuration directories (for example /opt/config/kibana) the configuration of the entire ELK-stack or of individual applications may be customized.
    The reason for not copying Logstash configuration files is that Logstash does not contain any configuration files.

  • The six chown commands sets the owner of the three log and configuration directories and any files and directories inside these to the elk user created earlier in the Docker-file and the group to the elk group.
    Elasticsearch, Logstash and Kibana will be run by the elk user and in order for the three programs to be able to write log, the elk user needs to have write permissions to the log directories or be the owner of these directories.

  • The next chown command has a similar purpose.
    Elasticsearch will create a data directory in its home directory and thus this directory needs to be writeable by the user running Elasticsearch.

  • ${ELASTICSEARCH_HOME}/bin/plugin -install lmenezes/elasticsearch-kopf
    This line installs the Kopf Elasticsearch plug-in, which is an administration tool for Elasticsearch with a web GUI.

  • ${LOGSTASH_HOME}/bin/plugin install logstash-input-jmx
    Yet another plug-in installation, this time it is the Logstash JMX plug-in which is used to retrieve information from JMX beans exposed in a running JVM periodically.
    I will get back to this plug-in in a subsequent article.

  • chmod +x /opt/start-elk.sh
    Ensure that the start-elk.sh script is executable.

  • sed -i -e”s|\${DIR}/config/kibana.yml|${KIBANA_CONFIG_FILE}|g” ${KIBANA_START_SCRIPT}
    Modifies the Kibana start-script as to use the configuration file in the external (/opt/config/kibana) configuration directory.

  • sed -i -e”s|# log_file: ./kibana.log|log_file: ${KIBANA_LOG_DIRECTORY}kibana.log|g” ${KIBANA_CONFIG_FILE}
    Modifies the Kibana configuration file as to make Kibana write log to a file named kibana.log in the /opt/logs/kibana directory instead of writing log to standard out.

  • sed -i -e”s|#path.logs: /path/to/logs|path.logs: ${ELASTICSEARCH_LOG_DIRECTORY}|g” ${ELASTICSEARCH_CONFIG_FILE}
    In a similar manner, the Elasticsearch configuration file is modified as to redirect log output to the /opt/logs/elasticsearch directory.

  • EXPOSE 5601 9200 5000
    Tells Docker that a container created from this image will expose some kind of service on the ports 5601, 9200 and 5000.
    In the case of this Docker image, the Kibana web GUI will be available on port 5601, the Elasticsearch REST API/Kopf administration tool on port 9200 and Logstash on port 5000.
    Note that the EXPOSE instruction does not automatically expose the listed ports to the host when a container is created. To accomplish that the -p or -P flag must be used when launching the container.
    Docker reference.

  • ENV PATH ${ELASTICSEARCH_HOME}/bin:$PATH
    The above line is the first of three similar lines that adds the bin directories of Elasticsearch, Logstash and Kibana to the path. This enables us to execute the Elasticsearch, Logstash and Kibana binaries in a Docker container created from this image without having to specify an absolute path or changing directory.

  • CMD [“/opt/start-elk.sh”]
    Specifies that, as default, the start-elk.sh script is to be launched whenever a container is created from this Docker image.
    If another executable is provided when launching a container, this script will not be executed.
    Docker reference.

The start-elk.sh Script

The script that starts the ELK stack is quite short, but nevertheless important.

First of all, notice the comment that the script file must not contain any carriage return characters. When first writing this script, I could not get it to execute properly. It took me a while to realize that the carriage return characters in the file were causing trouble.

  • set -e
    If any of the commands in this script terminates with an error status, the shell in which the script is executed will terminate and no further commands from the script will be executed.

  • gosu elk elasticsearch -d -Des.path.conf=/opt/config/elasticsearch/
    Start Elasticsearch running in the background being run with the elk user and use the supplied path (/opt/config/elasticsearch) to the configuration directory.

  • gosu elk logstash -f /opt/config/logstash/ -l /opt/logs/logstash/logstash.log &
    Starts Logstash looking for configuration files in the /opt/config/logstash directory and writing logs to the /opt/logs/logstash/logstash.log directory. As with Elasticsearch, Logstash is also run using the elk user.

  • exec gosu elk kibana
    Starts Kibana. Kibana is also run with the elk user. The exec command tells the shell not to fork a new process for Kibana, but run Kibana in the current process. We need to use the exec command in order for the Docker containers created from this Docker image to continue running. If exec were not used, the process executing the startup-script would terminate and so would the Docker container.

Final Words

I have experienced some problems with Logstash not being able to write log to the logstash.log file if the /opt/logs/logstash directory is a directory shared from the host (using the -v option with docker run). A workaround is to create an empty logstash.log file in the host and set permissions on this file (still in the host) to allow anyone to read and write to it.

I now have a Docker image with the ELK-stack which I will put to use in a future article – stay tuned!

6 thoughts on “Creating a Docker Image with Elasticsearch, Logstash and Kibana

  1. Przemyslaw Ozgo

    Hi,

    Thanks for this nice article. I have created 3 Images separately and I have it in production running. Just never had time to write an article about it. Well done then.

    If you want you could use some info from my images and small descriptions. Please hed to those links to have a look,

    ELK Description https://github.com/pozgo/ELK-CoreOS-Docker
    Elasticsearch docker image https://github.com/million12/docker-elasticsearch
    Logstash docker image https://github.com/million12/docker-logstash
    Kibana4 docker images https://github.com/million12/docker-kibana4

    Reply
    1. Ivan Krizsan Post author

      Here are some more Docker images of the Elasticsearch, Logstash and Kibana for reference:
      https://hub.docker.com/search/?q=elasticsearch&page=1&isAutomated=0&isOfficial=0&starCount=0&pullCount=0
      https://hub.docker.com/search/?q=logstash&page=1&isAutomated=0&isOfficial=0&starCount=0&pullCount=0
      https://hub.docker.com/search/?q=kibana&page=1&isAutomated=0&isOfficial=0&starCount=0&pullCount=0

      With the official Docker images for each product showing up at the top of the list (at least for me).

      Reply
      1. Sam

        Thanks for the blog. It works. However, the web interface http://localhost:/ requiring user login and password by “Shield”. What what the default login/password?

        Reply
      2. Ivan Krizsan Post author

        The problem with the official images and Docker Compose is that if you try to compose the official images, the Kibana container will fail since the Elasticsearch container will not be started fast enough. As of writing this, Docker Compose has no means to assure that container A has been started before starting container B, which depends on container A.
        A workaround is outlined in https://www.ivankrizsan.se/2015/10/19/creating-an-elastalert-docker-image-on-docker-hub/ but I am hoping for a new feature in Docker Compose.

        Reply

Leave a Reply

Your email address will not be published. Required fields are marked *