Managing Containers – Shipyard

By | July 10, 2016

Update 2017-12-02: It has come to my attention that Shipyard is no longer developed.

I am looking for something to help me manage multiple Docker hosts and the applications running on those hosts. Again, I will use my virtual laboratory, to start exploring the alternatives. In this first article I will look at Shipyard, which is an open-source application for managing a Docker cluster.

Wishlist

Before looking at anything I think I should list some of the qualities I want in the type of management application I am looking for.
Please remember these are my personal preferences and nothing more.

  1. Able to manage multiple Docker hosts.
  2. Require a minimum of additional software on a host that does not contain the management software.
    Obviously Docker needs to be installed on a node, but the less additional software required the better in my opinion.
  3. Provide an API for management.
    I don’t see an immediate need, but I suspect that at some point in time I will want to automate tasks.
  4. Provide a GUI for management.
    While a command-line interface is fine and dandy, a good GUI makes it easier to grasp what is going on and will make it easier to learn.
    In addition, a good GUI will be able to give an overview as well as focus on details when necessary.
  5. Automatic application/container management.
    Examples: Scaling an application depending on its load. Automatic rollback in the case of a failure when deploying a new version of a container.
  6. Facilitate management of applications that consist of multiple Docker containers.
    Ideally I would want to be able to group containers not only depending on how they are to be deployed, scaled etc, which is the regular type of grouping accomplished with Docker Compose, but also group containers depending on metadata on the container.
  7. Free, optionally with commercial support.
    Free in order for me to be able to try out all the features and develop without having to use a trial version or similar. Commercial support because it gives managers and bosses a sense of security when there is a bill to pay, in case I want to introduce it in a customer organisation.
  8. Access control.
    I don’t want anyone to be able to make changes to my applications, so I would like to have some kind of access control where I could limit other people’s access to the applications I configure. Just the traditional stuff with users, groups and, optional but very nice to have, an audit log.
  9. Availability.
    What happens if a node in the cluster or a container goes offline, be it a manager node or a worker node? If it is a manager node, then there should be some kind of failover mechanism so that another node takes on the manager responsibilities. If a container goes offline, then a new instance of it should ideally be started.
  10. Documentation.
    Good documentation is always nice.
  11. Ease of installation.

I suspect there will be additions to this list as the journey unravels and I will return to make changes.
I am aware of Docker Swarm and even of its improvements in Docker 1.12, but my wish is to find something more.

Shipyard

As before, this first post focus on Shipyard which is an application for managing Docker images and containers. Shipyard uses Docker Swarm at its core and have a GUI in form of a web application.

Installation

First of all I will set up Shipyard in my virtual laboratory that, as before, consists of four virtual machines running Ubuntu 16.04 with one manager and three nodes.

Deploying Shipyard in my virtual laboratory.

Deploying Shipyard in my virtual laboratory.

As shown in the figure above, I will run the Shipyard manager on the laboratory1 node and the laboratory2, laboratory3 and laboratory4 nodes will be regular Shipyard nodes. I will use these node-names throughout the example.

Manager Installation

To make it easy for myself during all the installing, testing and re-installing I have created a script that installs and starts the Shipyard manager. Perform the following steps on laboratory1:

  • Create a directory named “shipyard”.
    mkdir shipyard
    cd shipyard
  • Create a file named “start-manager.sh” with the following contents:
    #!/bin/sh
    # Must have root to run Shipyard.
    if [[ $EUID -ne 0 ]]
    then
       echo "This script must be run as root."
       exit 1
    fi
    
    # Download Shipyard deployment script if it isn't already downloaded.
    if [ ! -f deploy ]
    then
        wget https://shipyard-project.com/deploy
        chmod +x deploy
    fi
    
    # Start Shipyard manager.
    ACTION=deploy
    . deploy
  • Make the start-manager script executable.
    chmod +x start-manager.sh
  • Start the Shipyard manager.
    sudo su
    # Enter password.
    ./start-manager.sh

    The log output should end with something along these lines:

    Waiting for Shipyard on 192.168.1.180:8080
    
    Shipyard available at http://192.168.1.180:8080
    Username: admin Password: shipyard
  • In a browser, open the URL shown in the log and log in using the username and password also shown in the log.
    You should see the container list which in my case lists seven containers.

    Container list in the Shipyard manager application.

    Container list in the Shipyard manager application.

  • Click on the Nodes tab.
    You should then see the list of nodes in the Shipyard cluster.

    Node list in the Shipyard manager application.

    Node list in the Shipyard manager application.

We have now successfully installed and started the Shipyard manager. I suggest that you keep the Shipyard manager application open,displaying the node list, since it will make it easy to confirm successful addition of a node to the Shipyard cluster.

Node Installation

The following procedure to install and start a node should be repeated for the number of additional virtual machines in your library – in my case three.

  • Create a directory named “shipyard”.
    mkdir shipyard
    cd shipyard
  • Create a file named “start-node.sh” with the following contents:
    Note that the IP address on the highlighted line needs to be modified to the IP address of the (virtual)computer where your Shipyard manager runs.

    #!/bin/bash
    # Exit upon errors.
    set -e
    
    # Must have root to run Shipyard.
    if [[ $EUID -ne 0 ]]
    then
       echo "This script must be run as root."
       exit 1
    fi
    
    # Download Shipyard deployment script if it isn't already downloaded.
    if [ ! -f deploy ]
    then
        wget https://shipyard-project.com/deploy
        chmod +x deploy
    fi
    
    # Start Shipyard node.
    ACTION=node
    # IMPORTANT! Change this IP address to the IP address where your Shipyard manager runs.
    DISCOVERY=etcd://192.168.1.180:4001
    . deploy
  • Make the start-node script executable.
    chmod +x start-node.sh
  • Start the Shipyard node.
    sudo su
    # Enter password.
    ./start-node.sh

    The log output should end with something similar to this:

    Status: Downloaded newer image for swarm:latest
     -> Starting Swarm Agent
    Node added to Swarm: 192.168.1.181
  • Under the Nodes tab in the Shipyard manager application, click the Refresh button.
    The new node should appear in the list of nodes.

    Node list in the Shipyard manager application with a newly added node.

    Node list in the Shipyard manager application with a newly added node.

Repeat this process until you have added the desired number of nodes to your Shipyard cluster.

Examining Shipyard

Before I examine how well Shipyard matches my requirements, I will very briefly list some of the features of Shipyard.

  • Docker container management.
    Start, stop, pause, remove containers. Examine container statistics (CPU, memory, network). Examine logs. Execute commands in a “terminal window” in the web-application.
  • Docker image management.
    Pull images. Remove images.
  • Node management.
    As we’ve already seen, you can list the current nodes in the swarm.
  • Docker registry management.
    Add and remove registries.
  • User management.
    Add and remove users. Assign access rights to users.
  • View events.
    Lists all the operations performed in Shipyard.
  • Automatical restart.
    Shipyard containers are automatically started when I re-boot a computer on which Shipyard (manager or node) was running at shutdown.

So, how well does Shipyard measure up to the wishes I listed earlier?

Manage Multiple Docker Hosts

Check. Not much to argue about here.

Require a Minimum of Additional Software

The following Docker containers are being run on a non-manager node:

CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS                    NAMES
65d5af34868f        swarm:latest                   "/swarm j --addr 192."   11 minutes ago      Up 11 minutes       2375/tcp                 shipyard-swarm-agent
c1e02c99694f        swarm:latest                   "/swarm m --replicati"   11 minutes ago      Up 11 minutes       2375/tcp                 shipyard-swarm-manager
1bbbf393c4dd        shipyard/docker-proxy:latest   "/usr/local/bin/run"     12 minutes ago      Up 12 minutes       0.0.0.0:2375->2375/tcp   shipyard-proxy
5e1a1fa921fe        alpine                         "sh"                     12 minutes ago      Up 12 minutes                                shipyard-certs

Listing the Docker images required to run a non-manager node, we can see that they are all quite small.

REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
alpine                  latest              4e38e38c8ce0        10 days ago         4.799 MB
swarm                   latest              a180b24e38ed        3 weeks ago         19.34 MB
shipyard/docker-proxy   latest              cfee14e5d6f2        6 months ago        9.468 MB

Looking at the containers that run on a manager node:

CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS                         PORTS                                            NAMES
349c1f27a45e        shipyard/shipyard:latest       "/bin/controller --de"   4 days ago          Restarting (1) 1 seconds ago   0.0.0.0:8080->8080/tcp                           shipyard-controller
19977c4323e1        swarm:latest                   "/swarm j --addr 192."   4 days ago          Up 31 seconds                  2375/tcp                                         shipyard-swarm-agent
e563159bb037        swarm:latest                   "/swarm m --replicati"   4 days ago          Up 31 seconds                  2375/tcp                                         shipyard-swarm-manager
05085154634a        shipyard/docker-proxy:latest   "/usr/local/bin/run"     4 days ago          Up 32 seconds                  0.0.0.0:2375->2375/tcp                           shipyard-proxy
25f8c92f10a9        alpine                         "sh"                     4 days ago          Up 33 seconds                                                                   shipyard-certs
8102f37c016f        microbox/etcd:latest           "/bin/etcd -addr 192."   4 days ago          Up 34 seconds                  0.0.0.0:4001->4001/tcp, 0.0.0.0:7001->7001/tcp   shipyard-discovery
1558913f22ca        rethinkdb                      "rethinkdb --bind all"   4 days ago          Up 33 seconds                  8080/tcp, 28015/tcp, 29015/tcp                   shipyard-rethinkdb

Having a look at the Docker images on the manager node:

REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
alpine                  latest              4e38e38c8ce0        10 days ago         4.799 MB
swarm                   latest              a180b24e38ed        3 weeks ago         19.34 MB
rethinkdb               latest              70aa872c4c97        3 weeks ago         184.2 MB
shipyard/shipyard       latest              ab940e847950        4 weeks ago         58.92 MB
shipyard/docker-proxy   latest              cfee14e5d6f2        6 months ago        9.468 MB
ehazlett/curl           latest              de1763b15312        10 months ago       8.751 MB
microbox/etcd           latest              6aef84b9ec5a        11 months ago       17.87 MB

I am less concerned with the amount of additional software needed to run a manager node. Of course, I will want to have multiple manager nodes for failover in a production environment but nevertheless I suspect there will be a fixed number of manager nodes.
With Shipyard, some additional software is still needed for the non-manager nodes albeit it does look like a bare minimum.
Both the manager and non-manager nodes uses the Docker Swarm image, which is going to be obsolete once Docker 1.12 is released..

Conclusion:
Check. Shipyard has, in my opinion, a reasonable amount of additional software required to run non-manager nodes.

API for Management

Check.
In addition to the functionality provided by the Shipyard web-application, the API also supports:

  • Access to the Docker Remote API.
    At the time of writing, Shipyard supports up to version 1.20 of the Docker Remote API.
  • Management of webhook keys used for deploying to Docker Hub.

The documentation of the Shipyard API is available here.

GUI for Management

We have already seen glimpses of the Shipyard GUI earlier in this article, so: Check.
There are simple filtering-functionality available on the different pages of the web-application. For instance, on the page listing all running containers, you can enter some text and only the containers which entries in the list contain the text, albeit in any column, will be listed.
The GUI is a bit simplistic, but that may not be a bad thing. With four nodes in my swarm, I have no problems finding my way around the different nodes. Haven’t tried 400 nodes yet, so I can’t comment on that.

Automatic Application/Container Management

Apart from the basic Docker functionality to configure a restart policy when creating a new Docker container, I couldn’t find any support for automatic application/container management.

Management of Applications Consisting of Multiple Containers

Management of applications that consist of multiple containers, Docker Compose style, is not available in Shipyard at the moment.

Free with Support Option

Shipyard is free, as far as I have been able to tell, but regretfully there doesn’t seem to be any support option available. I’ll give Shipyard a partial check.

Access Control

As mentioned earlier, Shipyard implements user management with access control. The documentation even claims support for LDAP, though I haven’t tested this.
The events list in Shipyard can also be used as audit log, for which I want to give Shipyard extra credits. So I’ll give Shipyard a double-check, as far as access control is concerned.

Availability

Shipyard acts as a front-end for Docker Swarm, which has support for high availability, and thus does not need to implement these features itself.For those not familiar with Docker Swarm, I will briefly mention that it allows for failover both as far as managers and also as far as containers are concerned. I’ll give Shipyard a check.

If I were to set up a cluster of computers on which to run Docker containers and use Shipyard as the manager front-end, I would first set up a property configured Docker Swarm and then add the Shipyard controller.

Highly available Docker Swarm cluster with a Shipyard controller.

Highly available Docker Swarm cluster with a Shipyard controller.

There is a section in the Shipyard documentation describing how to manually deploy Shipyard if you want to attempt something like this.

Documentation

The Shipyard developers has made an effort, as far as the documentation is concerned. There is documentation not only for the Shipyard GUI but also for the API. Though when I tried out the API I had to take a look at the REST controller before getting it right. Check!

Ease of Installation

As seen in the section on installation above, Shipyard is fairly easy to install – just run the scripts with a few parameters. There are additional features in the Shipyard installation script, such as an uninstall option.

Conclusion

Shipyard seems like a nice, lightweight, Docker cluster manager solution built on top of Docker Swarm. It has most of the features I am looking for but nothing that, in my opinion, stands out. The Shipyard GitHub repository has, at the time of writing, around 40 contributors with two core contributors.
Since Shipyard is the first product I look at in this area, I cannot really compare it to anything else.

Here is a table that summarizes my examination of Shipyard:

Wishlist Item Status
1. Manage multiple Docker hosts. Yes
2. Minimum of additional software. Yes
3. Management API. Yes
4. Management GUI. Yes
5. Automatic application/container management. No
6. Manage groups of containers. No
7. Free, with option for commercial support. Free: Yes. Commercial support: No
8. Access control. Yes, with audit log.
9. Availability. Yes, implemented by Docker Swarm.
10. Documentation. Yes
11. Ease of Installation Yes

Given that Docker Swarm will be incorporated in the Docker Engine 1.12, I would wait for support of this version in Shipyard until I installed it for anything more than experimentation.

Happy coding!

Leave a Reply

Your email address will not be published. Required fields are marked *