Installing grafana and graphite-web + carbon-cache as docker images

Grafana is a powerful and elegant framework allowing easy-to-create dashboards for sharing metric with your team and the world.  The data we gather is collected using collectd, a lightweight agent which collects system performance statistics periodically and stores them locally or to a central repository.  In our case we use graphite’s carbon-cache to collect metrics, and graphite-web to provide a data source for grafana.

Why Docker Docker Docker

During the initial development of browbeat I contributed the initial playbooks to be able to deploy grafana, as well as graphite-web + carbon-cache.   The initial pass at these playbooks were intended to target dedicated systems for these services separate from our openstack deployments due to the possibly intensive resource requirements of carbon-cache (depending on the number of metrics being gathered).  Moreover, deploying grafana, graphite and carbon-cache on newly installed systems dedicated to these services was straightforward and without issue.

Over time my team and I realized that dedicating hardware to these services could be a luxury that cannot be afforded in every environment.  POCs and small lab deployments may in fact wish to deploy these parts of browbeat in the undercloud host managing the overcloud.  The challenge posed by this approach is that the undercloud has a large number of services already running, not to mention the specific python package dependencies that may conflict with grafana, graphite and carbon-cache deployment.

Docker containers solved this problem by providing a simple method of isolating grafana, and graphite-web + carbon-cache into containers.  Since we are using docker, mapping the application data into the host os (undercloud) to maintain persistence was trivial.  Furthermore docker provides an easy to use mechanism to map the application ports to whatever port you need on the host os to avoid conflicting with other service ports.

How to configure containers

There are a small number of settings in the install/group_vars/all.yml file to consider depending on your needs.  Here are the defaults:

########################################
# Docker related
# (use these if deploying graphite/carbon/grafana as containers)
########################################
persistent_carbon_data_path: /data/carbon/whisper
persistent_grafana_data_path: /data/grafana
docker_carbon_cache_port: 2003
docker_graphite_port: 8888
docker_grafana_port: 3000
carbon_cache_docker_image: kambiz/carbon-cache:0.9.15
graphite_web_docker_image: kambiz/graphite-web:0.9.15
grafana_docker_image: grafana/grafana:2.6.0

The persistent_ values are where you would like to store the application data within the container.  These are used to map the docker host’s filesystem into the appropriate locations for the application inside of the container.  The _port settings are straightforward, and will be how you can reach these services on the docker host.  If you are installing on your undercloud host, ensure these ports are available and not in use by other services.  The _image settings are the hub.docker.com published images.

How to install the containers

If you are familiar with ansible, deployment of the docker containers into your target host(s) is trivial.  A prerequisite is a rudimentary familiarity with browbeat.  Inside your ansible/ directory, ensure your hosts file includes a target for these services.  e.g.:

[graphite]
host-01

[grafana]
host-02

Assuming you’ve established ssh access to these target hosts via keys (passwordless remote access), you’re ready to deploy grafana and graphite-web + carbon-cache.

There are  two playbooks you’ll need to use out of browbeat.  install/graphite-docker.yml and install/grafana-docker.yml

Here is a sample deployment:

$ ansible-playbook -i hosts install/graphite-docker.yml 

PLAY [graphite] ****************************************************************

TASK [setup] *******************************************************************
ok: [host-01]

TASK [graphite_docker : Import EPEL GPG Key] ***********************************
changed: [host-01]

TASK [graphite_docker : Check for EPEL repo] ***********************************
changed: [host-01]

TASK [graphite_docker : Install repo file for docker] **************************
changed: [host-01]

TASK [graphite_docker : disable firewalld] *************************************
changed: [host-01]

TASK [graphite_docker : Install docker rpm] ************************************
skipping: [host-01] => (item=[u'docker-engine']) 

TASK [graphite_docker : Setup docker service] **********************************
changed: [host-01]

TASK [graphite_docker : ensure data directory exists] **************************
changed: [host-01]

TASK [graphite_docker : ensure docker overrides for carbon-cache] **************
changed: [host-01]

TASK [graphite_docker : Install carbon.conf] ***********************************
changed: [host-01]

TASK [graphite_docker : Install storage-schemas.conf] **************************
changed: [host-01]

TASK [graphite_docker : check active containers] *******************************
changed: [host-01]

TASK [graphite_docker : start carbon-cache docker container] *******************
failed: [host-01] (item=docker kill carbon-cache) => {"changed": true, "cmd": ["docker", "kill", "carbon-cache"], "delta": "0:00:00.012306", "end": "2016-05-21 08:39:04.228030", "failed": true, "item": "docker kill carbon-cache", "rc": 1, "start": "2016-05-21 08:39:04.215724", "stderr": "Error response from daemon: Cannot kill container carbon-cache: No such container: carbon-cache", "stdout": "", "stdout_lines": [], "warnings": []}
failed: [host-01] (item=docker rm carbon-cache) => {"changed": true, "cmd": ["docker", "rm", "carbon-cache"], "delta": "0:00:00.011853", "end": "2016-05-21 08:39:04.347328", "failed": true, "item": "docker rm carbon-cache", "rc": 1, "start": "2016-05-21 08:39:04.335475", "stderr": "Error response from daemon: No such container: carbon-cache", "stdout": "", "stdout_lines": [], "warnings": []}
changed: [host-01] => (item=sleep 5)
changed: [host-01] => (item=docker run -d --name carbon-cache -p 2003:2003 -v /etc/docker/carbon-cache/carbon.conf:/etc/carbon/carbon.conf -v /etc/docker/carbon-cache/storage-schemas.conf:/etc/carbon/storage-schemas.conf -v /data/carbon/whisper:/var/lib/carbon/whisper kambiz/carbon-cache:0.9.15)
...ignoring

TASK [graphite_docker : start graphite-web docker container] *******************
failed: [host-01] (item=docker kill graphite-web) => {"changed": true, "cmd": ["docker", "kill", "graphite-web"], "delta": "0:00:00.011952", "end": "2016-05-21 08:39:33.322835", "failed": true, "item": "docker kill graphite-web", "rc": 1, "start": "2016-05-21 08:39:33.310883", "stderr": "Error response from daemon: Cannot kill container graphite-web: No such container: graphite-web", "stdout": "", "stdout_lines": [], "warnings": []}
failed: [host-01] (item=docker rm graphite-web) => {"changed": true, "cmd": ["docker", "rm", "graphite-web"], "delta": "0:00:00.010773", "end": "2016-05-21 08:39:33.440686", "failed": true, "item": "docker rm graphite-web", "rc": 1, "start": "2016-05-21 08:39:33.429913", "stderr": "Error response from daemon: No such container: graphite-web", "stdout": "", "stdout_lines": [], "warnings": []}
changed: [host-01] => (item=sleep 5)
changed: [host-01] => (item=docker run -d --name graphite-web -p 8888:80 -v /data/carbon/whisper:/var/lib/carbon/whisper kambiz/graphite-web:0.9.15)
...ignoring

TASK [graphite_docker : Setup graphite-web systemd config] *********************
changed: [host-01]

TASK [graphite_docker : bounce systemd and graphite-web container] *************
changed: [host-01]

TASK [graphite_docker : Setup carbon-cache systemd config] *********************
changed: [host-01]

TASK [graphite_docker : bounce systemd and carbon-cache container] *************
changed: [host-01]

TASK [graphite_docker : Disable EPEL Repo] *************************************
changed: [host-01]

PLAY RECAP *********************************************************************
host-01                    : ok=19   changed=16   unreachable=0    failed=0   

$

The errors regarding failed deletion of carbon-cache and graphite-web can be ignored. The playbook needs to delete the docker containers if they exist before recreating them.  As such we ignore the error condition.

That’s all

You’re done!  Now you can go checkout what you can do with grafana and graphite.  For more information on the projects mentioned above, see:

And of course, to pull a copy of browbeat and checkout the code:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s