In this article I will show how I created a Docker image for Elastalert and create an automated build for the image on Docker Hub.
Among other things, I will show how to wait on other services that a service running in a Docker container depends on.
Introduction
For those who are not familiar with Elastalert I can briefly mention that it is a piece of software that can monitor data in an Elasticsearch instance and generate alerts upon detecting a variety of conditions. I were originally looking at Watcher from the makers of Elasticsearch, but it is not open source and after having failed to obtain a price quote over e-mail I gave up and started looking elsewhere.
In a subsequent article I will show how to use the Elastalert Docker image with the ELK Docker image I created in an earlier article.
Dockerfile
The Dockerfile looks like this:
# Elastalert Docker image running on Ubuntu 15.04. # Build image with: docker build -t ivankrizsan/elastalert:v1 . FROM ubuntu:15.04 MAINTAINER Ivan Krizsan, https://github.com/krizsan # URL from which to download Elastalert. ENV ELASTALERT_URL https://github.com/Yelp/elastalert/archive/v0.0.63.zip # Directory holding configuration for Elastalert and Supervisor. ENV CONFIG_DIR /opt/config # Elastalert rules directory. ENV RULES_DIRECTORY /opt/rules # Elastalert configuration file path in configuration directory. ENV ELASTALERT_CONFIG ${CONFIG_DIR}/elastalert_config.yaml # Directory to which Elastalert and Supervisor logs are written. ENV LOG_DIR /opt/logs # Elastalert home directory name. ENV ELASTALERT_DIRECTORY_NAME elastalert # Elastalert home directory full path. ENV ELASTALERT_HOME /opt/${ELASTALERT_DIRECTORY_NAME} # Supervisor configuration file for Elastalert. ENV ELASTALERT_SUPERVISOR_CONF ${CONFIG_DIR}/elastalert_supervisord.conf # Alias, DNS or IP of Elasticsearch host to be queried by Elastalert. Set in default Elasticsearch configuration file. ENV ELASTICSEARCH_HOST elasticsearch_host # Port on above Elasticsearch host. Set in default Elasticsearch configuration file. ENV ELASTICSEARCH_PORT 9200 WORKDIR /opt # Copy the script used to launch the Elastalert when a container is started. COPY ./start-elastalert.sh /opt/ # Install software required for Elastalert. RUN apt-get update && \ apt-get upgrade -y && \ apt-get install -y wget python python-dev unzip gcc && \ rm -rf /var/lib/apt/lists/* && \ apt-get clean && \ # Install pip - required for installation of Elastalert. wget https://bootstrap.pypa.io/get-pip.py && \ python get-pip.py && \ rm get-pip.py && \ # Download and unpack Elastalert. wget ${ELASTALERT_URL} && \ unzip *.zip && \ rm *.zip && \ mv e* ${ELASTALERT_DIRECTORY_NAME} WORKDIR ${ELASTALERT_HOME} # Install Elastalert. RUN python setup.py install && \ pip install -e . && \ # Install Supervisor. easy_install supervisor && \ # Make the start-script executable. chmod +x /opt/start-elastalert.sh && \ # Create directories. mkdir ${CONFIG_DIR} && \ mkdir ${RULES_DIRECTORY} && \ mkdir ${LOG_DIR} && \ # Copy deafult configuration files to configuration directory. cp ${ELASTALERT_HOME}/config.yaml.example ${ELASTALERT_CONFIG} && \ cp ${ELASTALERT_HOME}/supervisord.conf.example ${ELASTALERT_SUPERVISOR_CONF} && \ # Elastalert configuration: # Set the rule directory in the Elastalert config file to external rules directory. sed -i -e"s|rules_folder: [[:print:]]*|rules_folder: ${RULES_DIRECTORY}|g" ${ELASTALERT_CONFIG} && \ # Set the Elasticsearch host that Elastalert is to query. sed -i -e"s|es_host: [[:print:]]*|es_host: ${ELASTICSEARCH_HOST}|g" ${ELASTALERT_CONFIG} && \ # Set the port used by Elasticsearch at the above address. sed -i -e"s|es_port: [0-9]*|es_port: ${ELASTICSEARCH_PORT}|g" ${ELASTALERT_CONFIG} && \ # Elastalert Supervisor configuration: # Redirect Supervisor log output to a file in the designated logs directory. sed -i -e"s|logfile=.*log|logfile=${LOG_DIR}/elastalert_supervisord.log|g" ${ELASTALERT_SUPERVISOR_CONF} && \ # Redirect Supervisor stderr output to a file in the designated logs directory. sed -i -e"s|stderr_logfile=.*log|stderr_logfile=${LOG_DIR}/elastalert_stderr.log|g" ${ELASTALERT_SUPERVISOR_CONF} && \ # Modify the start-command. sed -i -e"s|python elastalert.py|python -m elastalert.elastalert --config ${ELASTALERT_CONFIG}|g" ${ELASTALERT_SUPERVISOR_CONF} && \ # Copy the Elastalert configuration file to Elastalert home directory to be used when creating index first time an Elastalert container is launched. cp ${ELASTALERT_CONFIG} ${ELASTALERT_HOME}/config.yaml && \ # Add Elastalert to Supervisord. supervisord -c ${ELASTALERT_SUPERVISOR_CONF} # Define mount points. VOLUME [ "${CONFIG_DIR}", "${RULES_DIRECTORY}", "${LOG_DIR}"] # Launch Elastalert when a container is started. CMD ["/opt/start-elastalert.sh"]
Some noteworthy things are:
- Version 0.0.63 of Elastalert is used.
At the time of writing, this was the most current tag. - When using the Elastalert Docker image, you have to create a link to the Docker container in which the Elasticsearch instance that Elastalert is to query for data.
The port is assumed to be 9200. - After having installed Elastalert, there is a command “pip install -e .”.
I had some troubles when installing Elastalert and the Elastalert package did not get installed. This command fixed this (big thanks to Qmando for helping me out with this one!). If everything already has bee installed correctly, this will be detected and nothing will happen when this command is executed. - Supervisor is installed.
Supervisor is used to start Elastalert in a Docker container and to redirect log output to files. - The Elastalert and Supervisor configuration files are taken from the Elastalert download and modified slightly.
I originally wanted to create a “elastalert” user in the Docker image and run Elastalert using this user but ran into some problems that caused me to postpone this and settle for letting the root user run Elastalert.
Elastalert Start Script
The Elastalert start script were a little tricky, since Elastalert require one-time initialization and also require that the Elasticsearch instance it is to query to be online.
#!/bin/sh set -e # Wait until Elasticsearch is online since otherwise Elastalert will fail. rm -f garbage_file while ! wget -O garbage_file ${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT} 2>/dev/null do echo "Waiting for Elasticsearch..." rm -f garbage_file sleep 1 done rm -f garbage_file sleep 5 # Check if the Elastalert index exists in Elasticsearch and create it if it does not. if ! wget -O garbage_file ${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/elastalert_status 2>/dev/null then echo "Creating Elastalert index in Elasticsearch..." elastalert-create-index --index elastalert_status --old-index "" else echo "Elastalert index already exists in Elasticsearch." fi rm -f garbage_file echo "Starting Elastalert..." exec supervisord -c ${ELASTALERT_SUPERVISOR_CONF} -n
Checking whether the Elasticsearch instance is online is done with the wget command. The while-loop will wait indefinitely until something answers at the Elasticsearch port and address. I did try the technique employed by docker-wait, but the script hang once the Elasticsearch instance came online.
The wget command is also used to determine if an index with the name “elastalert_status” exists in Elasticsearch. If it does not, the index is created.
Having created these two files and a readme-file, I created a new GitHub project.
Creating an Automated Build on DockerHub
First and foremost: There is no need whatsoever to push an image to DockerHub if you want to create an automated build, in which you display the Dockerfile of your image.
The following steps describes how to create an automated build on DockerHub that will build a new version of your Docker image whenever there is a change in the repository in which the source-files are stored.
- Log in to DockerHub.
- In the Create menu, select Create Automated Build.
- If you haven’t already, you need to link your GitHub account to DockerHub.
Do this in the Link Accounts menu. - Select the repository containing your Dockerfile.
The following screen will appear:
- Now is the last chance to name your Docker image.
If I were to leave the parameters in the above picture unchanged, my Docker image would be named ivankrizsan/elastalert-docker. I did not want that, so I changed “elastalert-docker” to plain “elastalert”. - Enter a short description of your Docker image.
The contents of any readme-file from your GitHub repository will be copied into the long description of the Docker image. - Configure additional tags for your Docker image.
These can be for, for instance, different version of the program that is to be run in Docker containers created from the image. - Decide whether the image is to be public or private.
- Decide whether or not new pushes to the repository are to trigger automatic builds of the image.
- Click the Create button.
Since I have already committed all the files to my GitHub repository, I will need to trigger the first build manually:
- In DockerHub, go to the Docker image.
- Select the Build Settings tab.
- Click the Trigger a Build button.
- Wait.
The Docker image is now alive and kickin’ in DockerHub!
Hi,
Thank you very much for sharing it.
However, I could not get the elastalert docker container running. Here is the error I am getting:
docker run -d –cap-add=sys_nice -e “SET_CONTAINER_TIMEZONE=true” -it –link elasticsearch_host:sebp/elk –name elastalert ivankrizsan/elastalert
docker logs d1279492f73a
/opt/start-elastalert.sh: line 11: setup-timezone: not found
reset adjtime failed: Operation not permitted
constraint certificate verification turned off
adjtimex failed: Operation not permitted
Elastalert index already exists in Elasticsearch.
Starting Elastalert…
2016-04-12 21:33:47,882 CRIT Supervisor running as root (no user in config file)
Unlinking stale socket /var/run/elastalert_supervisor.sock
2016-04-12 21:33:48,192 INFO RPC interface ‘supervisor’ initialized
2016-04-12 21:33:48,192 CRIT Server ‘unix_http_server’ running without any HTTP authentica tion checking
2016-04-12 21:33:48,192 INFO supervisord started with pid 1
2016-04-12 21:33:49,194 INFO spawned: ‘elastalert’ with pid 25
2016-04-12 21:33:49,522 DEBG ‘elastalert’ stderr output:
ERROR:root:No rules loaded. Exiting
None
[edit: removed repeated log messages]
2016-04-12 21:33:56,427 DEBG fd 9 closed, stopped monitoring <POutputDispatcher at 1397391 81785312 for (stder r)>
2016-04-12 21:33:56,427 DEBG fd 7 closed, stopped monitoring <POutputDispatcher at 1397391 82263760 for (stdou t)>
2016-04-12 21:33:56,427 INFO exited: elastalert (exit status 1; not expected)
2016-04-12 21:33:56,427 DEBG received SIGCHLD indicating a child quit
2016-04-12 21:33:57,428 INFO gave up: elastalert entered FATAL state, too many start retri es too quickly
Hello!
It looks as you do not have any rules configured for Elastalert. You have to share the rules directory from the container with the a directory in the host OS which contains Elastalert rules.
For an example, see my example at GitHub: https://github.com/krizsan/alerting-with-elk-and-elastalert
Happy coding!
Hi,
When I run start script, it says timezone is not modified.
It goes in the else block all the time, even if I set timzeone to true in the dockerfile.
Please help on this.
br.
Sunil
Hi Sunil!
You should set the environment variable when you run the Docker image, like in the following example:
docker run -e SET_CONTAINER_TIMEZONE=true [additional parameters] ivankrizsan/elastalert
Happy coding!
Hi, Ivan
after I running you image I see error:
elastalert_1_1 | 2017-08-17 18:23:07,286 DEBG received SIGCLD indicating a child quit
elastalert_1_1 | 2017-08-17 18:23:08,289 INFO spawned: ‘elastalert’ with pid 13
elastalert_1_1 | 2017-08-17 18:23:08,311 DEBG ‘elastalert’ stderr output:
elastalert_1_1 | /usr/bin/python: No module named elastalert
I use default supervisor.conf (but little modify start up command):
[unix_http_server]
file=/var/run/elastalert_supervisor.sock
[supervisord]
logfile=/opt/logs/elastalert_supervisord.log
logfile_maxbytes=1MB
logfile_backups=2
loglevel=debug
nodaemon=false
directory=%(here)s
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/elastalert_supervisor.sock
[program:elastalert]
# running globally
command=python -m elastalert.elastalert –config /opt/config/config.yaml –verbose
#command =
# python elastalert.py
# –verbose
# (alternative) using virtualenv
# command=/path/to/venv/bin/elastalert –config /path/to/config.yaml –verbose
process_name=elastalert
autorestart=true
startsecs=15
stopsignal=INT
stopasgroup=true
killasgroup=true
stderr_logfile=/opt/logs/elastalert_stderr.log
stderr_logfile_maxbytes=5MB
Any ideas?
Hello!
Sorry, I have not encountered those kind of problems myself. Someone mentioned that my Elastalert image sometimes does not build properly. I suggest that you download the sources and try to build it yourself.
I am in the process of contributing the Elastalert Docker image to Yelp in order to have an official Elastalert Docker image, so I won’t make any modifications to it.