In this article I will create a Docker image that contains the Elasticsearch, Logstash and Kibana – more commonly known as the ELK stack. This Docker image will be based on the Docker image with Java 8 on Ubuntu that I created in an earlier article.

I am aware of there being Docker images containing the ELK stack on Docker Hub.
I am also aware of the recommendation to create one separate Docker image for each application, that is one for Elasticsearch, one for Logstash and one for Kibana. For the time being one Docker image with the entire ELK stack is sufficient for my needs. I have considered creating three separate Docker images, but that will have to be the subject of another article.

Prerequisites

In order to be able to build the Docker image created in this article, you will need to have built my Docker image with Java 8 which I named “krizsan/ubuntu1504java8”.
After the last article, the kind people at Docker have released the Docker Toolbox which greatly simplifies installation of Docker and associated utilities. The Docker Toolbox I have used contains Docker 1.8.1.

Layout of the Docker Image

The Docker image has the layout as shown in this figure:

Layout of the Docker image containing the ELK-stack.

Configuration files and logs have been located to separate directories in order to allow for convenient mapping of host directories into Docker containers created from this image.

Docker File

The Docker-file for the ELK stack looks like this:

# Docker image containing Elasticsearch, Logstash and Kibana.
#
# Build image with: docker build -t krizsan/elk:v1 .
# Run image with, for example:
# docker run -d -p 5601:5601 -p 9200:9200 -p 5000:5000 -v [path-to-configurations-root-on-host]:/opt/config -v [path-to-logs-root-on-host]:/opt/logs krizsan/elk:v1
#
# View logs in Elasticsearch: curl 'http://[docker-container-ip]:9200/_search?pretty'
# Access Kibana: http://[docker-container-ip]:5601
# Access Kopf Elasticsearch admin GUI: http://[docker-container-ip]:9200/_plugin/kopf/#!/cluster

FROM krizsan/ubuntu1504java8:v1

ENV ELASTICSEARCH_VERSION 1.7.1
ENV LOGSTASH_VERSION 1.5.3
ENV KIBANA_VERSION 4.1.1

ENV CONFIG_ROOT /opt/config
ENV LOG_ROOT /opt/logs

ENV ELASTICSEARCH_HOME /opt/elasticsearch
ENV ELASTICSEARCH_ORIG_CONFIG_DIRECTORY ${ELASTICSEARCH_HOME}/config
ENV ELASTICSEARCH_CONFIG_FILE ${ELASTICSEARCH_ORIG_CONFIG_DIRECTORY}/elasticsearch.yml
ENV ELASTICSEARCH_LOG_DIRECTORY ${LOG_ROOT}/elasticsearch
ENV ELASTICSEARCH_CONFIG_DIRECTORY ${CONFIG_ROOT}/elasticsearch

ENV LOGSTASH_HOME /opt/logstash
ENV LOGSTASH_LOG_DIRECTORY ${LOG_ROOT}/logstash/
ENV LOGSTASH_CONFIG_DIRECTORY ${CONFIG_ROOT}/logstash

ENV KIBANA_HOME /opt/kibana
ENV KIBANA_ORIG_CONFIG_DIRECTORY ${KIBANA_HOME}/config
ENV KIBANA_LOG_DIRECTORY ${LOG_ROOT}/kibana/
ENV KIBANA_CONFIG_DIRECTORY ${CONFIG_ROOT}/kibana
ENV KIBANA_CONFIG_FILE ${KIBANA_CONFIG_DIRECTORY}/kibana.yml
ENV KIBANA_START_SCRIPT ${KIBANA_HOME}/bin/kibana

MAINTAINER Ivan Krizsan, https://github.com/krizsan

WORKDIR /opt

# Create the elk user and elk system group.
RUN groupadd -r elk && useradd -r -g elk elk

# Install gosu as to allow us to run programs with a specific user.
RUN apt-get update && apt-get install -y curl libossp-uuid-dev wget ca-certificates git make && rm -rf /var/lib/apt/lists/* && \
    curl -o /usr/local/bin/gosu -SL 'https://github.com/tianon/gosu/releases/download/1.1/gosu' && \
    chmod +x /usr/local/bin/gosu && \
    apt-get purge -y --auto-remove curl
	
# Copy the script used to launch the ELK-stack when a container is started.
COPY ./start-elk.sh /opt/

# Download Elasticsearch, Logstash and Kibana.
RUN wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-${ELASTICSEARCH_VERSION}.tar.gz && \
    wget https://download.elastic.co/logstash/logstash/logstash-${LOGSTASH_VERSION}.tar.gz  && \
    wget https://download.elastic.co/kibana/kibana/kibana-${KIBANA_VERSION}-linux-x64.tar.gz  && \
# Create home directories.
    mkdir ${ELASTICSEARCH_HOME} && \
    mkdir ${LOGSTASH_HOME} && \
    mkdir ${KIBANA_HOME} && \
# Extract Elasticsearch, Logstash and Kibana to respective directories.
    tar -xzf elasticsearch*.tar.gz  -C ${ELASTICSEARCH_HOME} --strip-components=1 && \
    tar -xzf logstash*.tar.gz  -C ${LOGSTASH_HOME} --strip-components=1 && \
    tar -xzf kibana*.tar.gz  -C ${KIBANA_HOME} --strip-components=1 && \
# Remove archives.
    rm *.tar.gz && \
# Create log directories.
    mkdir -p ${LOGSTASH_LOG_DIRECTORY} && \
    mkdir -p ${ELASTICSEARCH_LOG_DIRECTORY} && \
    mkdir -p ${KIBANA_LOG_DIRECTORY} && \
# Create external configuration directories.
    mkdir -p ${ELASTICSEARCH_CONFIG_DIRECTORY} && \
    mkdir -p ${LOGSTASH_CONFIG_DIRECTORY} && \
    mkdir -p ${KIBANA_CONFIG_DIRECTORY} && \
# Copy configuration to external configuration directories.
    cp ${ELASTICSEARCH_ORIG_CONFIG_DIRECTORY}/*.* ${ELASTICSEARCH_CONFIG_DIRECTORY} && \
    cp ${KIBANA_ORIG_CONFIG_DIRECTORY}/*.* ${KIBANA_CONFIG_DIRECTORY} && \
# Set owner of log directories to the user that runs the applications.
    chown -hR elk:elk ${LOGSTASH_LOG_DIRECTORY}  && \
    chown -hR elk:elk ${ELASTICSEARCH_LOG_DIRECTORY}  && \
    chown -hR elk:elk ${KIBANA_LOG_DIRECTORY}  && \
# Set owner of configuration directories to the user that runs the applications.
    chown -hR elk:elk ${LOGSTASH_CONFIG_DIRECTORY}  && \
    chown -hR elk:elk ${ELASTICSEARCH_CONFIG_DIRECTORY}  && \
    chown -hR elk:elk ${KIBANA_CONFIG_DIRECTORY}  && \
# Set owner of Elasticsearch directory so that data directory can be created.
    chown -hR elk:elk ${ELASTICSEARCH_HOME} && \
# Install Elasticsearch kopf plug-in: https://github.com/lmenezes/elasticsearch-kopf
    ${ELASTICSEARCH_HOME}/bin/plugin -install lmenezes/elasticsearch-kopf && \
# Install Logstash JMX plug-in.
    ${LOGSTASH_HOME}/bin/plugin install logstash-input-jmx && \
# Make the start-script executable.
    chmod +x /opt/start-elk.sh && \
# Modify Kibana start-script as to use external configuration file.
    sed -i -e"s|\${DIR}/config/kibana.yml|${KIBANA_CONFIG_FILE}|g" ${KIBANA_START_SCRIPT} && \
# Modify Kibana configuration as to log to the dedicated logging directory instead of standard out.
    sed -i -e"s|# log_file: ./kibana.log|log_file: ${KIBANA_LOG_DIRECTORY}kibana.log|g" ${KIBANA_CONFIG_FILE} && \
# Modify Elasticsearch configuration to log to the dedicated logging directory.
    sed -i -e"s|#path.logs: /path/to/logs|path.logs: ${ELASTICSEARCH_LOG_DIRECTORY}|g" ${ELASTICSEARCH_CONFIG_FILE}

# Kibana UI port, Elasticsearch REST API/Kopf port and Logstash.
EXPOSE 5601 9200 5000

# Add Elasticsearch, Logstash and Kibana bin directories to path.
ENV PATH ${ELASTICSEARCH_HOME}/bin:$PATH
ENV PATH ${LOGSTASH_HOME}/bin:$PATH
ENV PATH ${KIBANA_HOME}/bin:$PATH

# Launch the ELK stack when a container is started.
CMD ["/opt/start-elk.sh"]

Short Description

The above Docker file prepares an image based on my previous Docker image with Java 8 on Ubuntu 15.04. In this image, Elasticsearch, Logstash and Kibana are downloaded and unpacked.
A special user, “elk”, and a group with the same name as the user are added. Elasticsearch, Logstash and Kibana will all be run by this user rather than the root.
The Kopf plug-in, which is a web administration tool for Elasticsearch, is installed in Elasticsearch. The JMX plug-in is installed in Logstash, as to make it possible to poll Java Virtual Machines for JMX data.
The Kibana configuration is modified as to write log to a file instead of to the console.

Configuration and log files have been placed in dedicated directory structures, as to allow for convenient mapping to host directories.

Longer Description

The Docker file start with some comments on how to access Elasticsearch administration tool, Kibana etc. All URLs assume that you have mapped the appropriate ports from the Docker container to the same ports on the host and that you replace “[docker-container-ip]” with the IP address of the Docker machine.

• FROM krizsan/ubuntu1504java8:v1
  As in the previous article, this specifies the Docker image on which this new Docker image is to be based upon. For those that haven’t read my previous article, the image used is my own Docker image containing Ubuntu 15.04 and Oracle’s Java 8.
  Docker reference.
• Next follow a number of ENV declarations.
  These are similar to constant declarations, which prevents duplication of, for instance, file paths, in the Docker-file.
  Docker reference.
• WORKDIR /opt
  Makes /opt the current working directory, similar to the cd shell command.
  Docker reference.
• RUN groupadd -r elk && useradd -r -g elk elk
  The Docker RUN instruction executes commands. The groupadd shell command creates a system group with the name “elk” and the useradd command creates a system user “elk” belonging to the system group that was just created. Note that the “elk” user will not have a home directory, since it is a system user. This is the user that will run the programs of the ELK stack in the Docker image.
  Docker reference.
• The next RUN instruction installs gosu, which allows us to start processes that belong to a specific user.
• COPY ./start-elk.sh /opt/
  Copies the script that is to launch the programs of the ELK stack when a Docker container is started from the directory in which the Docker-file is located to the /opt/ directory in the Docker image. We will examine this script more closely in a little while.
  Docker reference.

• The next Docker instruction is RUN, which is followed by a large number of shell commands.
  As in the earlier article, the reason for executing multiple shell commands with one single RUN instruction is that each time the RUN instruction is invoked, a new layer is created in the Docker image. This will lead to a Docker image which is unnecessarily large.
  Docker reference.

• The three wget commands will download Elasticsearch, Logstash and Kibana respectively.
  Since the working directory is /opt, the three archives will be downloaded to this directory in the Docker image.

• I have chosen to place Elasticsearch, Logstash and Kibana in directories that does not contain the version number in order to simplify both the Docker-file and the startup-script.
  The three mkdir commands create the home directory for each of the three components of the ELK stack. The -p flag cause any missing parent directories to be created as well.

• In the next step the downloaded archives are unpacked using the tar command.
  The x flag tells the tar command to extract files from an archive, the z flag cause unpacking of a gzip-archive and the f flag is used to specify the location of the archive from which to extract files. In addition, the C flag is used, which selects the directory to which the extracted files are to be written. The –strip-components=1 option will cause the first directory component of the files in the archive to be removed. If, for example, the path of a file in the archive is /dir1/dir2/dir3/file.txt the file will be written to dir2/dir3/file.txt in the destination directory when –strip-components=1.

• Having finished unpacking the Elasticsearch, Logstash and Kibana archives, they are deleted using the rm command.

• Next up is creation of the directories that are to hold log files for the three applications.
  I have chosen to locate these in a dedicated logs directory in order to be able to mount a host directory as a directory for log files.

• In a similar fashion, three directories are created that are to hold configuration-files for the three applications.

• The two cp commands copy configuration files from the application home directories to the external configuration directories just created.
  This creates a default configuration setup in the Docker image. If the user of the Docker image choose to map the entire external configuration directory (/opt/config) to a host directory or individual external configuration directories (for example /opt/config/kibana) the configuration of the entire ELK-stack or of individual applications may be customized.
  The reason for not copying Logstash configuration files is that Logstash does not contain any configuration files.

• The six chown commands sets the owner of the three log and configuration directories and any files and directories inside these to the elk user created earlier in the Docker-file and the group to the elk group.
  Elasticsearch, Logstash and Kibana will be run by the elk user and in order for the three programs to be able to write log, the elk user needs to have write permissions to the log directories or be the owner of these directories.

• The next chown command has a similar purpose.
  Elasticsearch will create a data directory in its home directory and thus this directory needs to be writeable by the user running Elasticsearch.

• ${ELASTICSEARCH_HOME}/bin/plugin -install lmenezes/elasticsearch-kopf
  This line installs the Kopf Elasticsearch plug-in, which is an administration tool for Elasticsearch with a web GUI.

• ${LOGSTASH_HOME}/bin/plugin install logstash-input-jmx
  Yet another plug-in installation, this time it is the Logstash JMX plug-in which is used to retrieve information from JMX beans exposed in a running JVM periodically.
  I will get back to this plug-in in a subsequent article.

• chmod +x /opt/start-elk.sh
  Ensure that the start-elk.sh script is executable.

• sed -i -e”s|\${DIR}/config/kibana.yml|${KIBANA_CONFIG_FILE}|g” ${KIBANA_START_SCRIPT}
  Modifies the Kibana start-script as to use the configuration file in the external (/opt/config/kibana) configuration directory.

• sed -i -e”s|# log_file: ./kibana.log|log_file: ${KIBANA_LOG_DIRECTORY}kibana.log|g” ${KIBANA_CONFIG_FILE}
  Modifies the Kibana configuration file as to make Kibana write log to a file named kibana.log in the /opt/logs/kibana directory instead of writing log to standard out.

• sed -i -e”s|#path.logs: /path/to/logs|path.logs: ${ELASTICSEARCH_LOG_DIRECTORY}|g” ${ELASTICSEARCH_CONFIG_FILE}
  In a similar manner, the Elasticsearch configuration file is modified as to redirect log output to the /opt/logs/elasticsearch directory.

• EXPOSE 5601 9200 5000
  Tells Docker that a container created from this image will expose some kind of service on the ports 5601, 9200 and 5000.
  In the case of this Docker image, the Kibana web GUI will be available on port 5601, the Elasticsearch REST API/Kopf administration tool on port 9200 and Logstash on port 5000.
  Note that the EXPOSE instruction does not automatically expose the listed ports to the host when a container is created. To accomplish that the -p or -P flag must be used when launching the container.
  Docker reference.

• ENV PATH ${ELASTICSEARCH_HOME}/bin:$PATH
  The above line is the first of three similar lines that adds the bin directories of Elasticsearch, Logstash and Kibana to the path. This enables us to execute the Elasticsearch, Logstash and Kibana binaries in a Docker container created from this image without having to specify an absolute path or changing directory.

• CMD [“/opt/start-elk.sh”]
  Specifies that, as default, the start-elk.sh script is to be launched whenever a container is created from this Docker image.
  If another executable is provided when launching a container, this script will not be executed.
  Docker reference.

The start-elk.sh Script

The script that starts the ELK stack is quite short, but nevertheless important.

#!/bin/bash

# Note! This file must not contain any carriage return characters, only line feed characters.

# Exit immediately if a command exits with a non-zero status.
set -e

gosu elk elasticsearch -d -Des.path.conf=/opt/config/elasticsearch/
gosu elk logstash -f /opt/config/logstash/ -l /opt/logs/logstash/logstash.log &
exec gosu elk kibana

First of all, notice the comment that the script file must not contain any carriage return characters. When first writing this script, I could not get it to execute properly. It took me a while to realize that the carriage return characters in the file were causing trouble.

• set -e
  If any of the commands in this script terminates with an error status, the shell in which the script is executed will terminate and no further commands from the script will be executed.

• gosu elk elasticsearch -d -Des.path.conf=/opt/config/elasticsearch/
  Start Elasticsearch running in the background being run with the elk user and use the supplied path (/opt/config/elasticsearch) to the configuration directory.

• gosu elk logstash -f /opt/config/logstash/ -l /opt/logs/logstash/logstash.log &
  Starts Logstash looking for configuration files in the /opt/config/logstash directory and writing logs to the /opt/logs/logstash/logstash.log directory. As with Elasticsearch, Logstash is also run using the elk user.

• exec gosu elk kibana
  Starts Kibana. Kibana is also run with the elk user. The exec command tells the shell not to fork a new process for Kibana, but run Kibana in the current process. We need to use the exec command in order for the Docker containers created from this Docker image to continue running. If exec were not used, the process executing the startup-script would terminate and so would the Docker container.

Final Words

I have experienced some problems with Logstash not being able to write log to the logstash.log file if the /opt/logs/logstash directory is a directory shared from the host (using the -v option with docker run). A workaround is to create an empty logstash.log file in the host and set permissions on this file (still in the host) to allow anyone to read and write to it.

I now have a Docker image with the ELK-stack which I will put to use in a future article – stay tuned!