Introduction
For now we have setup docker environment, once we move production we may require more containers and machines. All the systems we cannot sit around and monitor one by one. For that here we are going to setup InfluxDB and Grafana using Dockerfile.
Note: you should have a container configured to accept server metrics information from collectd.
What is InfluxDB?
InfluxDB is an open-source time series database developed by InfluxData as part of their time series platform. It is written in Go and optimized for fast, high-availability storage and retrieval of time series data in fields such as operations monitoring, application metrics, Internet of Things sensor data, and real-time analytics.
What is Grafana?
Grafana is a metrics dashboard which plugs into solutions such as Graphite, InfluxDB, and OpenTSDB. You would use Grafana to visualize the various metrics data you are pushing into InfluxDB. In our case the type of metrics we’re preparing for are system metrics we hope to gather using collectd.
The Dockerfile
You can either create a folder or place your Dockerfile and all configuration files in some directory where you are in. we do recommend to create a folder because here we are going to configuration files which required for this setup. It will be easy to maintain.
The first lines of your Dockerfile should always be:
FROM ubuntu
MAINTAINER FoxuTech (mail@foxutech.com)
This indicates what base image you are using and who maintains the Dockerfile.
Next, you’ll want to prepare the environment with all required packages,
RUN \
apt-get update && apt-get -y –no-install-recommends install \
ca-certificates \
software-properties-common \
python-django-tagging \
python-simplejson \
python-memcache \
python-ldap \
python-cairo \
python-pysqlite2 \
python-support \
python-pip \
gunicorn \
supervisor \
nginx-light \
nodejs \
git \
curl \
openjdk-7-jre \
build-essential \
python-dev
Now we installed some required packages, let mention the work directory as /opt and RUN block will install Grafana, InfluxDB, and do some required configuration:
WORKDIR /opt
RUN \
grafana_url=$(curl http://grafanarel.s3.amazonaws.com/latest.json | python -c ‘import sys, json; print json.load(sys.stdin)[“url”]’) && \
curl -s -o grafana.tar.gz $grafana_url && \
curl -s -o influxdb_latest_amd64.deb http://s3.amazonaws.com/influxdb/influxdb_latest_amd64.deb && \
mkdir grafana && \
tar -xzf grafana.tar.gz –directory grafana –strip-components=1 && \
dpkg -i influxdb_latest_amd64.deb && \
echo “influxdb soft nofile unlimited” >> /etc/security/limits.conf && \
echo “influxdb hard nofile unlimited” >> /etc/security/limits.conf
This downloads and installs the latest version of Influx and Grafana (at the time of writing).
Now we need to copy over the configuration files we’ve staged:
ADD config.js /opt/grafana/config.js
ADD nginx.conf /etc/nginx/nginx.conf
ADD supervisord.conf /etc/supervisor/conf.d/supervisord.conf
ADD config.toml /opt/influxdb/current/config.toml
Finally, we map volumes, expose ports, and setup the run command:
VOLUME [“/opt/influxdb/shared/data”]
EXPOSE 80 8083 8086 2003
CMD [“supervisord”, “-n”]
For now we see the complete details about Dockerfile and this is the full copy of Dockerfile,
# vim Dockerfile
FROM ubuntu
MAINTAINER FoxuTech (mail@foxutech.com)
RUN \
apt-get update && apt-get -y –no-install-recommends install \
ca-certificates \
software-properties-common \
python-django-tagging \
python-simplejson \
python-memcache \
python-ldap \
python-cairo \
python-pysqlite2 \
python-support \
python-pip \
gunicorn \
supervisor \
nginx-light \
nodejs \
git \
curl \
openjdk-7-jre \
build-essential \
python-dev
WORKDIR /opt
RUN \
grafana_url=$(curl http://grafanarel.s3.amazonaws.com/latest.json | python -c ‘import sys, json; print json.load(sys.stdin)[“url”]’) && \
curl -s -o grafana.tar.gz $grafana_url && \
curl -s -o influxdb_latest_amd64.deb http://s3.amazonaws.com/influxdb/influxdb_latest_amd64.deb && \
mkdir grafana && \
tar -xzf grafana.tar.gz –directory grafana –strip-components=1 && \
dpkg -i influxdb_latest_amd64.deb && \
echo “influxdb soft nofile unlimited” >> /etc/security/limits.conf && \
echo “influxdb hard nofile unlimited” >> /etc/security/limits.conf
ADD config.js /opt/grafana/config.js
ADD nginx.conf /etc/nginx/nginx.conf
ADD supervisord.conf /etc/supervisor/conf.d/supervisord.conf
ADD config.toml /opt/influxdb/current/config.toml
VOLUME [“/opt/influxdb/shared/data”]
EXPOSE 80 8083 8086 2003
CMD [“supervisord”, “-n”]
You can pull the configuration and Dockerfile in following GIThub. It’s May save your time Docker-Influxdb-Grafana
Application Configuration Files
Grafana Config
The important one that needs to be altered is Grafana’s config.js. You will need to update the HTTP endpoints to be an IP you can reach from the location where you will be using Grafana.
datasources: {
influxdb: {
type: ‘influxdb’,
url: “http://yourpublicIP:8086/db/sampledb”,
username: ‘root’,
password: ‘root’,
},
grafana: {
type: ‘influxdb’,
url: “http://yourpublicIP:8086/db/grafana”,
username: ‘root’,
password: ‘root’,
grafanaDB: true
},
},
In our demo we are connecting to the Grafana via the public Internet.
Grafana is configured to use InfluxDB for its configurtion. While you can use the same db for both it is not recommended.
InfluxDB Config
We have enabled graphite in InfluxDB’s config.toml by doing this:
[input_plugins]
[input_plugins.graphite]
enabled = true
port = 2003
udp_enabled = true
database = “sampledb ”
We tell InfluxDB to start the listener on 2003/UDP and set the database for the metrics.
External Volumes
It is very important to understand the ephemeral nature of containers. This means that data created and stored within the container can disappear if the container restarts or is stopped.
We expose the InfluxDB database path on this line:
VOLUME [“/opt/influxdb/shared/data/db”]
This allows us to pass the -v switch to docker to tell it to map the path in the container to a local path in our host system. When data is written to InfluxDB the it is now not stored within the container itself and is persistent beyond a restart.
Building the Container
Now that you have put your Dockerfile and configuration files together it is time to build your container. You will need to ensure you’re working directory has your Dockerfile and configuration files in it. To build a container from your Dockerfile you would run the following command:
# docker build -t influxdb .
The -t parameter tags the image with the name influxdb. The build will execute the Dockerfile. Inspect the output for any errors. If all looks good you should be ready to start the container up.
Running the Container
Building a container will not automatically start the container. You will need to do that next. Ensure the /opt/influxdb path exists on your host file system then run the following command.
# docker run –name=”influx” –hostname=”influx” -d -v /opt/influxdb/:/opt/influxdb/shared/data -p 80:80 -p 8083:8083 -p 8086:8086 -p 2003:2003/udp influxdb
What’s happening here;
- Start the influx image;
- Name it influx;
- Map the container path of /opt/influxdb/shared/data to your local /opt/influxdb;
- Map your local port 80, 8083, and 8086 to the exposed ports in the container.
- Map port 2003 to UDP. Collectd will use UDP to communicate the data it is collecting.
Once the commend run successfully, you can check the status of the container using below commend,
# docker ps
Accessing Grafana and InfluxDB
You should now be able to reach your Grafana site by your public IP from your browser:
http://yourpublicip
You can reach the InfluxDB management interface by going here:
http://yourpublicip:8083/
The default credential is root with the password root. Once you login make sure you are changing it for security reason.
Before Grafana can pull data out of InfluxDB you will need to first create the databases that were defined in config.js. These would be sampledb and grafana. The Influx interface is relatively easy to use so you should be able to do this pretty quickly.
Reading and Writing Data in InfluxDB
For testing purpose you load some sample data to the DB and you can test.
curl -X POST -d ‘[{“name”:”fox”,”columns”:[“val”],”points”:[[33]]}]’ ‘http://localhost:8086/db/sampledb/series?u=root&p=root’
This data is loading into our sampledb database which is the same one we defined in our Grafana config.js file. You can now define your Grafana queries and dashboard settings.
To validate you can read data from InfluxDB you could do this:
curl -G ‘http://localhost:8086/db/sampledb /series?u=root&p=root’ –data-urlencode “q=select * from fox”